Attention

Attention Is All You Need

Notes

Introduced the Transformer architecture. The paper that started everything.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Notes

IO-aware attention that is both faster and uses less memory. Essential infrastructure.

Neural Language Models: From RNNs to Transformers

The evolution of neural sequence prediction, and how it connects to classical methods