Attention Is All You Need

This paper introduces the Transformer, a novel neural network architecture based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.

Key Contributions

1. Self-Attention: Computes representations of a sequence by relating different positions of the same sequence.
2. Multi-Head Attention: Allows the model to jointly attend to information from different representation subspaces.
3. No Recurrence: Enables massive parallelization, leading to the development of LLMs like GPT and BERT.