M4-001M4: Sequences & TransformersRecurrent Models (RNN/LSTM/GRU)Easy
⚡Flashcard Drill➔Mastery:

RNN Forward Dynamics, Backpropagation Through Time (BPTT), and Core Limitations

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-002 →
📐 Mathematical Definition
ht=tanh⁡(Whht−1+Wxxt+b);∂L∂h1=∏t=2T∂ht∂ht−1⋅∂L∂hTh_t=\tanh(W_h h_{t-1}+W_x x_t+b);\qquad \frac{\partial\mathcal{L}}{\partial h_1}=\prod_{t=2}^{T}\frac{\partial h_t}{\partial h_{t-1}}\cdot\frac{\partial\mathcal{L}}{\partial h_T}
⚡ One-Sentence Core Answer
RNNs unroll recurrent transitions sequentially across time; BPTT multiplies repeated transition Jacobians, causing exponential gradient vanishing/explosion and preventing parallelization.

📌 Key Takeaways

  • •
    Forward recurrence: ht=tanh⁡(Whhht−1+Wxhxt+bh)h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)
  • •
    BPTT gradient chain: ∂hT∂h1=∏t=2Tdiag(1−ht2)WhhT\frac{\partial h_T}{\partial h_1} = \prod_{t=2}^T \text{diag}(1 - h_t^2) W_{hh}^T; exponential scaling (Whh)T(W_{hh})^T
  • •
    Sequential bottleneck: step tt strictly depends on step t−1t-1, preventing parallel GPU training along sequence length

📐 Mathematical Derivations

Mathematical Formulations:
① Forward Recurrence:
ht=tanh⁡(Whhht−1+Wxhxt+bh)h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h), yt=softmax(Whyht+by)\quad y_t = \text{softmax}(W_{hy} h_t + b_y).
② Backpropagation Through Time (BPTT):
Let total loss be L=∑t=1Tℓt\mathcal{L} = \sum_{t=1}^T \ell_t. The gradient with respect to hidden state hkh_k (k<Tk < T) accumulates across all intervening time steps:
∂ℓt∂hk=∂ℓt∂ht∏j=k+1t∂hj∂hj−1=∂ℓt∂ht∏j=k+1t[diag(1−hj2)WhhT]\frac{\partial \ell_t}{\partial h_k} = \frac{\partial \ell_t}{\partial h_t} \prod_{j=k+1}^t \frac{\partial h_j}{\partial h_{j-1}} = \frac{\partial \ell_t}{\partial h_t} \prod_{j=k+1}^t \left[ \text{diag}(1 - h_j^2) W_{hh}^T \right].
- If the spectral radius ρ(Whh)<1\rho(W_{hh}) < 1, the product ∥∏WhhT∥→0\|\prod W_{hh}^T\| \to 0 exponentially as temporal distance t−k>20t - k > 20, causing gradients to vanish completely (inability to model long-term dependencies).
- If ρ(Whh)>1\rho(W_{hh}) > 1, the gradient explodes exponentially, triggering NaN/Inf divergence.

🏭 Production Trade-offs

Architectural death: Beyond gradient instability, standard RNNs require sequential step-by-step unrolling, achieving near-zero GPU Tensor Core utilization compared to parallel matrix multiplications in Transformers.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to train plain RNNs on sequences longer than 50 tokens without gradient clipping or gating mechanisms
  • ✕
    Assuming truncated BPTT resolves vanishing gradients; it only truncates backward horizon, throwing away long-distance information
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does the eigenvalue spectrum of WhhW_{hh} govern gradient stability in BPTT?
  • ?
    How does Truncated BPTT (TBPTT) bound the memory footprint during long-sequence training?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-101: Architectural Design Philosophies: ViT vs Swin Transformer vs ConvNeXt📋 Back to BankNext →M4-002: LSTM: Three Gating Mechanisms and the Additive Cell State Highway