---
title: "Attention Is All You Need"
authors: ["Vaswani, Ashish", "Shazeer, Noam", "Parmar, Niki", "Uszkoreit, Jakob"]
year: 2017
venue: "Advances in Neural Information Processing Systems"
volume: "30"
pages: "5998-6008"
arxiv: "1706.03762"
type: "paper"
status: "unread"
added: "2026-02-26"
tags:
  - transformers
  - attention-mechanism
  - self-attention
  - neural-machine-translation
  - encoder-decoder
---

# Attention Is All You Need

**Vaswani, Ashish et al.** • 2017

> [!quote] Memorable Quote
> "The Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output."

## Quick Refresh

This paper introduces the Transformer architecture, which relies entirely on attention mechanisms without recurrence or convolution. The model achieves state-of-the-art results on machine translation tasks while being more parallelizable and requiring less training time.

## Why You Cared

This is the foundational paper for modern language models like GPT and BERT. Understanding the Transformer architecture is essential for working with contemporary NLP systems.

## Key Concepts

`#transformers` `#attention-mechanism` `#self-attention` `#neural-machine-translation` `#encoder-decoder`

## Cites (Key Papers)

- [[Bahdanau, D., Cho, K. & Bengio, Y. (2014) - Neural machine translation by jointly learning to align and ...]]
- [[Bahdanau, D., Cho, K. & Bengio, Y. (2014) - Neural machine translation by jointly learning to align and ...]]
- [[Bahdanau, D., Cho, K. & Bengio, Y. (2014) - Neural machine translation by jointly learning to align and ...]]

## Cited By

*This section will be populated as you process papers that cite this one.*

## Details

**Published:** Advances in Neural Information Processing Systems, Vol. 30, pp. 5998-6008
**arXiv:** [1706.03762](https://arxiv.org/abs/1706.03762)
**PDF:** [[arxiv_1706.03762.pdf]]

## Abstract

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks...

## Full Citation List

1. Bahdanau, D., Cho, K. & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate.
2. Bahdanau, D., Cho, K. & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate.
3. Bahdanau, D., Cho, K. & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate.