M3-032M3: Deep Learning FoundationsOptimizers & Second-Order MethodsMedium
⚡Flashcard Drill➔Mastery:
Role of the Epsilon Parameter in Adam and Common Settings in Large Models
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Epsilon prevents division by zero and bounds the maximum effective step size (); LLMs typically use , or in lower-precision (FP16/BF16) training.
📌 Key Takeaways
- •Division protection: avoids division by zero when gradient second moment
- •Maximum step-size bound: when , effective step is bounded by
- •Precision compatibility: in FP32/BF16; increase to or in FP16 to avoid underflow into unnormalized steps
📐 Mathematical Derivations
Mathematical Mechanics:
The update term in Adam is: .
- When , has negligible influence, and update size is scale-invariant: .
- When (e.g., rarely activated embedding tokens or zero-gradient features), the denominator collapses to : .
If , a small non-zero gradient could trigger an explosive update step of magnitude . Thus, functions as an implicit cap bounding maximum parameter displacement.
Positioning of $\epsilon$: vs . PyTorch places outside the square root (
The update term in Adam is: .
- When , has negligible influence, and update size is scale-invariant: .
- When (e.g., rarely activated embedding tokens or zero-gradient features), the denominator collapses to : .
If , a small non-zero gradient could trigger an explosive update step of magnitude . Thus, functions as an implicit cap bounding maximum parameter displacement.
Positioning of $\epsilon$: vs . PyTorch places outside the square root (
eps=1e-8).🏭 Production Trade-offs
Hyperparameter guidance: In large language model pre-training, setting or stabilizes training when using FP16 or BF16 mixed-precision by preventing sudden gradient spikes on rare tokens.
⚠️ Common Interview Pitfalls
- ✕Treating as an irrelevant mathematical artifact and setting it to machine epsilon (), triggering training explosions
- ✕Setting excessively large (), which degenerates Adam into standard unnormalized SGD
🎯 Interviewer Follow-up Inquiries
- ?How does the ratio determine the theoretical upper bound on parameter step size in Adam?
- ?Why is preferred over when training with FP16 mixed precision?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚