M4-002M4: Sequences & TransformersRecurrent Models (RNN/LSTM/GRU)Easy
⚡Flashcard Drill➔Mastery:
LSTM: Three Gating Mechanisms and the Additive Cell State Highway
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
LSTM introduces forget, input, and output gates around a linear additive cell state , providing an uninterrupted gradient highway.
📌 Key Takeaways
- •Additive cell state: linear update avoids repeated matrix multiplication, eliminating gradient vanishing
- •Three gates: Forget gate (what to discard), Input gate (what new information to store), Output gate (what to emit to hidden state )
- •Gating range: Sigmoid activations constrain gates to , acting as continuous soft switches
📐 Mathematical Derivations
Mathematical Formulations (Hochreiter & Schmidhuber, 1997):
At time step with input and previous hidden state :
1. Forget Gate: (fraction of previous memory to retain).
2. Input Gate: (fraction of candidate update to add).
3. Candidate State: .
4. Cell State Update: .
5. Output Gate & Hidden State: , .
Why Gradients Do Not Vanish:
The Jacobian of the cell state is: .
Across time steps, the backward gradient is: .
If the forget gate is near 1 (), the gradient propagates across hundreds of time steps with zero exponential decay!
At time step with input and previous hidden state :
1. Forget Gate: (fraction of previous memory to retain).
2. Input Gate: (fraction of candidate update to add).
3. Candidate State: .
4. Cell State Update: .
5. Output Gate & Hidden State: , .
Why Gradients Do Not Vanish:
The Jacobian of the cell state is: .
Across time steps, the backward gradient is: .
If the forget gate is near 1 (), the gradient propagates across hundreds of time steps with zero exponential decay!
🏭 Production Trade-offs
Initialization trick: Always initialize forget gate bias to a positive constant (e.g., or ). This forces at step 0, ensuring the cell starts by remembering historical information.
⚠️ Common Interview Pitfalls
- ✕Initializing forget gate biases to 0, which sets and attenuates memory retention by per step at initialization
- ✕Assuming LSTM cell state can grow unboundedly; without forget gates (), variance drifts to infinity
🎯 Interviewer Follow-up Inquiries
- ?Why must the forget gate bias be initialized to a positive value (e.g., 1.0)?
- ?How does the additive cell state in LSTM mathematically mirror the identity skip connection in ResNet?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚