M3-002M3: Deep Learning FoundationsBackprop & AutodiffEasy
⚡Flashcard Drill➔Mastery:

Intermediate Quantities Retained for Backpropagation and Memory Optimization

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-003 →
📐 Mathematical Definition
checkpoint: recompute activations in backward\text{checkpoint}:\ \text{recompute activations in backward}
⚡ One-Sentence Core Answer
Backprop must save intermediate activations whose values appear in local derivative formulas; save memory via activation checkpointing, in-place operations, and fused kernels.

📌 Key Takeaways

  • •
    Saved tensors: linear layers save input activations xx (since ∂L∂W=δ⋅xT\frac{\partial \mathcal{L}}{\partial W} = \delta \cdot x^T); activations like ReLU save boolean masks
  • •
    Activation Checkpointing: discards intermediate activations and recomputes them during backward pass, reducing memory from O(L)O(L) to O(L)O(\sqrt{L})
  • •
    Fused Kernels: FlashAttention fuses attention score calculation to compute softmax on-chip (SRAM) without materializing O(N2)O(N^2) matrices in HBM

📐 Mathematical Derivations

Derivation of saved quantities: For a standard linear layer y=Wx+by = W x + b, the gradient with respect to weights is ∂L∂W=(∂L∂y)xT=δ⋅xT\frac{\partial \mathcal{L}}{\partial W} = \left(\frac{\partial \mathcal{L}}{\partial y}\right) x^T = \delta \cdot x^T. Thus, forward activation xx must be saved in GPU VRAM until the backward step.
For activation functions:
- ReLU(x)=max⁡(0,x)\text{ReLU}(x) = \max(0, x): ∂y∂x=1x>0\frac{\partial y}{\partial x} = \mathbf{1}_{x > 0}. Requires saving a 1-bit boolean mask instead of full 32-bit floats.
- Sigmoid(x)=σ(x)\text{Sigmoid}(x) = \sigma(x): σ′(x)=σ(x)(1−σ(x))\sigma'(x) = \sigma(x)(1 - \sigma(x)). Can save either input xx or output yy. Saving output yy is faster.
Activation Checkpointing (Rematerialization): Partition an LL-layer network into kk segments. Save only the boundary activations (requiring O(k)O(k) memory). During backward pass, recompute the forward pass within each segment on-the-fly (adding ∼20−30%\sim 20-30\% FLOPs overhead while cutting activation memory from O(L)O(L) to O(L)O(\sqrt{L}) for k=Lk=\sqrt{L}).

🏭 Production Trade-offs

Engineering trade-offs: Activation memory dominates model weight memory when sequence length NN or batch size BB is large. In modern LLM training, FlashAttention (tiling and online softmax) and Selective Activation Checkpointing (saving cheap layers, recomputing expensive attention/MLP activations) are standard.
⚠️ Common Interview Pitfalls
  • ✕
    Storing unnecessary intermediate tensors in global variables during training, preventing PyTorch garbage collection
  • ✕
    Applying activation checkpointing to computationally expensive layers that are memory-cheap, creating severe compute waste
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does FlashAttention avoid saving the full N×NN \times N attention matrix while still computing exact gradients?
  • ?
    What is the mathematical threshold where activation memory exceeds model parameter memory in Transformer training?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-001: Computational Graphs and Two Automatic Differentiation Modes: Forward vs Reverse📋 Back to BankNext →M3-003: Gradient Accumulation: Mathematical Equivalence and Implementation