Seedlabs
Source research

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez et al. · 2025 · 6564 citationsRead the paper

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

What the full text shows

Working prototype
  • Achieved state-of-the-art BLEU score on WMT 2014 English-to-German translation, outperforming existing ensembles. — 28.4 BLEU (>2.0 BLEU improvement over previous best) (Transformer (big) model)
  • Established new single-model state-of-the-art BLEU score on WMT 2014 English-to-French translation. — 41.8 BLEU (Transformer (big) model)
  • Reduced training cost for English-to-French translation compared to previous state-of-the-art. — < 1/4 the training cost (Transformer (big) model)
  • Achieved high performance on English constituency parsing in a semi-supervised setting. — 92.7 F1 score (4-layer transformer, semi-supervised on 17M sentences)
  • Achieved competitive performance on English constituency parsing with limited data. — 91.3 F1 score (4-layer transformer, WSJ only (40K sentences))
Setup
Trained on WMT 2014 English-to-German and English-to-French datasets; English constituency parsing on Penn Treebank (WSJ). Hardware used included 8 P100 GPUs.
How
Proposed the Transformer architecture, which replaces recurrent and convolutional layers with stacked multi-head self-attention and position-wise feed-forward networks in an encoder-decoder configuration. Used sinusoidal positional encoding to provide sequence order information.
Limitations
Reduced effective resolution due to averaging attention-weighted positions (mitigated by multi-head attention). · Inference remains auto-regressive (sequential generation).

Summarized by Gemma 4 from the open-access full text · 2 Oct 2026. Read it