M4-001M4: Sequences & TransformersRecurrent Models (RNN/LSTM/GRU)Easy
⚡Flashcard Drill➔Mastery:
RNN Forward Dynamics, Backpropagation Through Time (BPTT), and Core Limitations
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
RNNs unroll recurrent transitions sequentially across time; BPTT multiplies repeated transition Jacobians, causing exponential gradient vanishing/explosion and preventing parallelization.
📌 Key Takeaways
- •Forward recurrence:
- •BPTT gradient chain: ; exponential scaling
- •Sequential bottleneck: step strictly depends on step , preventing parallel GPU training along sequence length
📐 Mathematical Derivations
Mathematical Formulations:
① Forward Recurrence:
, .
② Backpropagation Through Time (BPTT):
Let total loss be . The gradient with respect to hidden state () accumulates across all intervening time steps:
.
- If the spectral radius , the product exponentially as temporal distance , causing gradients to vanish completely (inability to model long-term dependencies).
- If , the gradient explodes exponentially, triggering NaN/Inf divergence.
① Forward Recurrence:
, .
② Backpropagation Through Time (BPTT):
Let total loss be . The gradient with respect to hidden state () accumulates across all intervening time steps:
.
- If the spectral radius , the product exponentially as temporal distance , causing gradients to vanish completely (inability to model long-term dependencies).
- If , the gradient explodes exponentially, triggering NaN/Inf divergence.
🏭 Production Trade-offs
Architectural death: Beyond gradient instability, standard RNNs require sequential step-by-step unrolling, achieving near-zero GPU Tensor Core utilization compared to parallel matrix multiplications in Transformers.
⚠️ Common Interview Pitfalls
- ✕Attempting to train plain RNNs on sequences longer than 50 tokens without gradient clipping or gating mechanisms
- ✕Assuming truncated BPTT resolves vanishing gradients; it only truncates backward horizon, throwing away long-distance information
🎯 Interviewer Follow-up Inquiries
- ?Why does the eigenvalue spectrum of govern gradient stability in BPTT?
- ?How does Truncated BPTT (TBPTT) bound the memory footprint during long-sequence training?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚