M4-049M4: Sequences & TransformersEfficient Attention & FlashAttentionMedium
⚡Flashcard Drill➔Mastery:

FlashAttention Backward Pass: Recomputing Softmax in SRAM Without Saving Intermediate Matrices

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-050 →
📐 Mathematical Definition
saved: {mi,ℓi,Oi};P=exp⁡(S−m)/ℓ recomputed on the fly\text{saved}:\ \{m_i,\ell_i,O_i\};\quad P=\exp(S-m)/\ell\ \text{recomputed on the fly}
⚡ One-Sentence Core Answer
The backward pass saves only forward normalizers (m,ℓm, \ell), recomputing the attention matrix on-the-fly in SRAM blocks to eliminate O(N2)O(N^2) activation memory completely.

📌 Key Takeaways

  • •
    Memory saving: standard backprop saves N×NN \times N attention matrix (O(N2)O(N^2)); FlashAttention saves only scalar vectors m,ℓ∈RNm, \ell \in \mathbb{R}^N (O(N)O(N))
  • •
    Backward recomputation: loads Qi,Kj,VjQ_i, K_j, V_j and cached (mi,ℓi)(m_i, \ell_i) into SRAM, reconstructing attention block PijP_{ij} in registers
  • •
    Gradient flow: computes dQ,dK,dVdQ, dK, dV locally in fast SRAM and streams accumulated gradients back to HBM

📐 Mathematical Derivations

Mathematical Formulations of FlashAttention Backward Pass:
In standard attention backprop, the gradient formulas require the full forward attention matrix P=softmax(QKT/d)P = \text{softmax}(Q K^T / \sqrt{d}):
dV=PTdOdV = P^T dO, dP=dOVT\quad dP = dO V^T, dS=P⊙(dP−rowsum(dP⊙P))\quad dS = P \odot (dP - \text{rowsum}(dP \odot P)), dQ=1ddSK\quad dQ = \frac{1}{\sqrt{d}} dS K, dK=1ddSTQ\quad dK = \frac{1}{\sqrt{d}} dS^T Q.
Storing P∈RB×H×N×NP \in \mathbb{R}^{B \times H \times N \times N} requires tens of gigabytes of VRAM for long contexts.
FlashAttention Recomputation Trick:
- Forward Storage: FlashAttention stores ONLY the output O∈RN×dO \in \mathbb{R}^{N \times d} and the log-sum-exp normalization vector Li=mi+log⁡(ℓi)∈RNL_i = m_i + \log(\ell_i) \in \mathbb{R}^N. Storage footprint is strictly $O(N)$.
- Backward Execution Loop:
1. Load blocks Qi,Kj,VjQ_i, K_j, V_j and dOidO_i into fast SRAM.
2. Recompute local attention block in SRAM using cached LiL_i: Pij=exp⁡(QiKjTd−Li)P_{ij} = \exp\left( \frac{Q_i K_j^T}{\sqrt{d}} - L_i \right).
3. Compute dVj+=PijTdOidV_j += P_{ij}^T dO_i directly in SRAM.
4. Compute dPij=dOiVjTd P_{ij} = dO_i V_j^T, and define Di=rowsum(dOi⊙Oi)D_i = \text{rowsum}(dO_i \odot O_i).
5. Form dSij=Pij⊙(dPij−Di)d S_{ij} = P_{ij} \odot (d P_{ij} - D_i).
6. Accumulate dQi+=1ddSijKjdQ_i += \frac{1}{\sqrt{d}} d S_{ij} K_j and dKj+=1ddSijTQidK_j += \frac{1}{\sqrt{d}} d S_{ij}^T Q_i.
All intermediate matrices Pij,dPij,dSijP_{ij}, d P_{ij}, d S_{ij} live exclusively in SRAM registers and are discarded immediately.

🏭 Production Trade-offs

FLOPs vs Bandwidth Trade-off: Recomputing PijP_{ij} in backward pass adds ≈15%\approx 15\% more arithmetic FLOPs, but eliminates hundreds of gigabytes of HBM read/write traffic, resulting in an overall 2.5−4×2.5-4\times faster backward pass.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to cache intermediate attention matrices during forward pass when using FlashAttention, defeating its memory advantages
  • ✕
    Assuming FlashAttention backward pass is slower due to recomputation; memory bandwidth savings far outweigh recompute FLOPs
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is the vector Di=rowsum(dOi⊙Oi)D_i = \text{rowsum}(dO_i \odot O_i) mathematically equal to rowsum(dPi⊙Pi)\text{rowsum}(dP_i \odot P_i) in backpropagation?
  • ?
    How does FlashAttention's O(N)O(N) activation memory enable training with 10×10\times longer context windows?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-048: Online Softmax: Enabling Block-by-Block Incremental Attention Tiling📋 Back to BankNext →M4-050: PagedAttention: Eliminating KV-Cache Memory Fragmentation via Virtual Memory Paging