M4-049M4: Sequences & TransformersEfficient Attention & FlashAttentionMedium
⚡Flashcard Drill➔Mastery:
FlashAttention Backward Pass: Recomputing Softmax in SRAM Without Saving Intermediate Matrices
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
The backward pass saves only forward normalizers (), recomputing the attention matrix on-the-fly in SRAM blocks to eliminate activation memory completely.
📌 Key Takeaways
- •Memory saving: standard backprop saves attention matrix (); FlashAttention saves only scalar vectors ()
- •Backward recomputation: loads and cached into SRAM, reconstructing attention block in registers
- •Gradient flow: computes locally in fast SRAM and streams accumulated gradients back to HBM
📐 Mathematical Derivations
Mathematical Formulations of FlashAttention Backward Pass:
In standard attention backprop, the gradient formulas require the full forward attention matrix :
, , , , .
Storing requires tens of gigabytes of VRAM for long contexts.
FlashAttention Recomputation Trick:
- Forward Storage: FlashAttention stores ONLY the output and the log-sum-exp normalization vector . Storage footprint is strictly $O(N)$.
- Backward Execution Loop:
1. Load blocks and into fast SRAM.
2. Recompute local attention block in SRAM using cached : .
3. Compute directly in SRAM.
4. Compute , and define .
5. Form .
6. Accumulate and .
All intermediate matrices live exclusively in SRAM registers and are discarded immediately.
In standard attention backprop, the gradient formulas require the full forward attention matrix :
, , , , .
Storing requires tens of gigabytes of VRAM for long contexts.
FlashAttention Recomputation Trick:
- Forward Storage: FlashAttention stores ONLY the output and the log-sum-exp normalization vector . Storage footprint is strictly $O(N)$.
- Backward Execution Loop:
1. Load blocks and into fast SRAM.
2. Recompute local attention block in SRAM using cached : .
3. Compute directly in SRAM.
4. Compute , and define .
5. Form .
6. Accumulate and .
All intermediate matrices live exclusively in SRAM registers and are discarded immediately.
🏭 Production Trade-offs
FLOPs vs Bandwidth Trade-off: Recomputing in backward pass adds more arithmetic FLOPs, but eliminates hundreds of gigabytes of HBM read/write traffic, resulting in an overall faster backward pass.
⚠️ Common Interview Pitfalls
- ✕Attempting to cache intermediate attention matrices during forward pass when using FlashAttention, defeating its memory advantages
- ✕Assuming FlashAttention backward pass is slower due to recomputation; memory bandwidth savings far outweigh recompute FLOPs
🎯 Interviewer Follow-up Inquiries
- ?Why is the vector mathematically equal to in backpropagation?
- ?How does FlashAttention's activation memory enable training with longer context windows?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚