M4-048M4: Sequences & TransformersEfficient Attention & FlashAttentionEasy
⚡Flashcard Drill➔Mastery:
Online Softmax: Enabling Block-by-Block Incremental Attention Tiling
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Online softmax updates running maximums and normalizers dynamically as blocks arrive, eliminating the traditional requirement of scanning the full sequence before computing exponents.
📌 Key Takeaways
- •Standard Softmax 3-pass limitation: Pass 1 finds global max ; Pass 2 computes sum ; Pass 3 divides by
- •Online Softmax (Milakov & Gimelshein): maintains running state , updating in 1 pass via exponential rescaling factors
- •Tiled Attention integration: dynamically rescales accumulated partial output matrices in SRAM as new key-value blocks are processed
📐 Mathematical Derivations
Mathematical Formulations (Milakov & Gimelshein, 2018; Dao et al., 2022):
Standard 3-pass softmax over vector :
, , . Requires full vector in memory.
Online 2-Pass Softmax Derivation:
Partition vector into two blocks and .
- For block 1: .
- When block 2 arrives: compute local statistics .
- Running Combined Updates:
New global maximum: .
New global normalizer: .
- Output Vector Rescaling in FlashAttention:
Let accumulated output in SRAM from block 1 be . When block 2 arrives, update dynamically:
.
By induction, blocks of arbitrary size can be streamed through SRAM one by one, producing exact final outputs without ever storing the full attention matrix.
Standard 3-pass softmax over vector :
, , . Requires full vector in memory.
Online 2-Pass Softmax Derivation:
Partition vector into two blocks and .
- For block 1: .
- When block 2 arrives: compute local statistics .
- Running Combined Updates:
New global maximum: .
New global normalizer: .
- Output Vector Rescaling in FlashAttention:
Let accumulated output in SRAM from block 1 be . When block 2 arrives, update dynamically:
.
By induction, blocks of arbitrary size can be streamed through SRAM one by one, producing exact final outputs without ever storing the full attention matrix.
🏭 Production Trade-offs
Register pressure: Online softmax requires maintaining running vectors inside GPU register files. Kernel block sizes () must be carefully chosen to avoid register spilling into local memory.
⚠️ Common Interview Pitfalls
- ✕Forgetting to rescale previous partial output accumulators when updating the running maximum
- ✕Assuming online softmax introduces numerical approximation drift; it is mathematically exact down to floating-point rounding
🎯 Interviewer Follow-up Inquiries
- ?How does the exponential rescaling factor guarantee numerical stability in online softmax?
- ?What determines the optimal thread block size in FlashAttention CUDA kernels?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚