M3-002M3: Deep Learning FoundationsBackprop & AutodiffEasy
⚡Flashcard Drill➔Mastery:
Intermediate Quantities Retained for Backpropagation and Memory Optimization
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Backprop must save intermediate activations whose values appear in local derivative formulas; save memory via activation checkpointing, in-place operations, and fused kernels.
📌 Key Takeaways
- •Saved tensors: linear layers save input activations (since ); activations like ReLU save boolean masks
- •Activation Checkpointing: discards intermediate activations and recomputes them during backward pass, reducing memory from to
- •Fused Kernels: FlashAttention fuses attention score calculation to compute softmax on-chip (SRAM) without materializing matrices in HBM
📐 Mathematical Derivations
Derivation of saved quantities: For a standard linear layer , the gradient with respect to weights is . Thus, forward activation must be saved in GPU VRAM until the backward step.
For activation functions:
- : . Requires saving a 1-bit boolean mask instead of full 32-bit floats.
- : . Can save either input or output . Saving output is faster.
Activation Checkpointing (Rematerialization): Partition an -layer network into segments. Save only the boundary activations (requiring memory). During backward pass, recompute the forward pass within each segment on-the-fly (adding FLOPs overhead while cutting activation memory from to for ).
For activation functions:
- : . Requires saving a 1-bit boolean mask instead of full 32-bit floats.
- : . Can save either input or output . Saving output is faster.
Activation Checkpointing (Rematerialization): Partition an -layer network into segments. Save only the boundary activations (requiring memory). During backward pass, recompute the forward pass within each segment on-the-fly (adding FLOPs overhead while cutting activation memory from to for ).
🏭 Production Trade-offs
Engineering trade-offs: Activation memory dominates model weight memory when sequence length or batch size is large. In modern LLM training, FlashAttention (tiling and online softmax) and Selective Activation Checkpointing (saving cheap layers, recomputing expensive attention/MLP activations) are standard.
⚠️ Common Interview Pitfalls
- ✕Storing unnecessary intermediate tensors in global variables during training, preventing PyTorch garbage collection
- ✕Applying activation checkpointing to computationally expensive layers that are memory-cheap, creating severe compute waste
🎯 Interviewer Follow-up Inquiries
- ?Why does FlashAttention avoid saving the full attention matrix while still computing exact gradients?
- ?What is the mathematical threshold where activation memory exceeds model parameter memory in Transformer training?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚