§ feed · storyline
Generative modeling with sparse transformers
OpenAI releases Sparse Transformer, a deep neural network using an improved attention mechanism to model sequences 30 times longer than previously possible across text, images, and sound.
We’ve developed the Sparse Transformer, a deep neural network which sets new records at predicting what comes next in a sequence—whether text, images, or sound.
It uses an algorithmic improvement of the attention mechanism to extract patterns from sequences 30x longer than possible previously.
§ sources1 publication · timeline below