M4-002M4: Sequences & TransformersRecurrent Models (RNN/LSTM/GRU)Easy
⚡Flashcard Drill➔Mastery:

LSTM: Three Gating Mechanisms and the Additive Cell State Highway

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-003 →
📐 Mathematical Definition
ct=ft⊙ct−1+it⊙gt;∂ct∂ct−1=ft (no matrix product)c_t=f_t\odot c_{t-1}+i_t\odot g_t;\qquad \frac{\partial c_t}{\partial c_{t-1}}=f_t\ (\text{no matrix product})
⚡ One-Sentence Core Answer
LSTM introduces forget, input, and output gates around a linear additive cell state Ct=ft⊙Ct−1+it⊙C~tC_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t, providing an uninterrupted gradient highway.

📌 Key Takeaways

  • •
    Additive cell state: linear update avoids repeated matrix multiplication, eliminating gradient vanishing
  • •
    Three gates: Forget gate ftf_t (what to discard), Input gate iti_t (what new information to store), Output gate oto_t (what to emit to hidden state hth_t)
  • •
    Gating range: Sigmoid activations constrain gates to (0,1)(0, 1), acting as continuous soft switches

📐 Mathematical Derivations

Mathematical Formulations (Hochreiter & Schmidhuber, 1997):
At time step tt with input xtx_t and previous hidden state ht−1h_{t-1}:
1. Forget Gate: ft=σ(Wf[ht−1,xt]+bf)f_t = \sigma(W_f [h_{t-1}, x_t] + b_f) (fraction of previous memory to retain).
2. Input Gate: it=σ(Wi[ht−1,xt]+bi)i_t = \sigma(W_i [h_{t-1}, x_t] + b_i) (fraction of candidate update to add).
3. Candidate State: C~t=tanh⁡(Wc[ht−1,xt]+bc)\tilde{C}_t = \tanh(W_c [h_{t-1}, x_t] + b_c).
4. Cell State Update: Ct=ft⊙Ct−1+it⊙C~tC_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t.
5. Output Gate & Hidden State: ot=σ(Wo[ht−1,xt]+bo)o_t = \sigma(W_o [h_{t-1}, x_t] + b_o), ht=ot⊙tanh⁡(Ct)\quad h_t = o_t \odot \tanh(C_t).
Why Gradients Do Not Vanish:
The Jacobian of the cell state is: ∂Ct∂Ct−1=diag(ft)\frac{\partial C_t}{\partial C_{t-1}} = \text{diag}(f_t).
Across TT time steps, the backward gradient is: ∂L∂C1=∂L∂CT∏t=2Tft\frac{\partial \mathcal{L}}{\partial C_1} = \frac{\partial \mathcal{L}}{\partial C_T} \prod_{t=2}^T f_t.
If the forget gate is near 1 (ft≈1f_t \approx 1), the gradient propagates across hundreds of time steps with zero exponential decay!

🏭 Production Trade-offs

Initialization trick: Always initialize forget gate bias bfb_f to a positive constant (e.g., 1.01.0 or 2.02.0). This forces ft≈σ(1.0)≈0.73−0.88f_t \approx \sigma(1.0) \approx 0.73-0.88 at step 0, ensuring the cell starts by remembering historical information.
⚠️ Common Interview Pitfalls
  • ✕
    Initializing forget gate biases to 0, which sets f0≈0.5f_0 \approx 0.5 and attenuates memory retention by 50%50\% per step at initialization
  • ✕
    Assuming LSTM cell state CtC_t can grow unboundedly; without forget gates (ft=1f_t=1), CtC_t variance drifts to infinity
🎯 Interviewer Follow-up Inquiries
  • ?
    Why must the forget gate bias bfb_f be initialized to a positive value (e.g., 1.0)?
  • ?
    How does the additive cell state in LSTM mathematically mirror the identity skip connection in ResNet?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-001: RNN Forward Dynamics, Backpropagation Through Time (BPTT), and Core Limitations📋 Back to BankNext →M4-003: Comparing LSTM and GRU: Structural Simplifications and Trade-offs