M4-006M4: Sequences & TransformersRecurrent Models (RNN/LSTM/GRU)Hard
⚡Flashcard Drill➔Mastery:
Regularization Techniques in the Recurrent Era: Variational Dropout and Zoneout
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Standard dropout corrupts memory across time steps; recurrent networks require Variational Dropout (freezing the mask across time), Zoneout (stochastic state identity), and weight drop.
📌 Key Takeaways
- •Standard Dropout defect: sampling independent dropout masks at each time step destroys the cell state's memory persistence
- •Variational Dropout (Gal & Ghahramani): samples a single dropout mask per sequence and applies the exact same mask across all time steps
- •Zoneout (Krueger et al.): stochastically preserves previous cell state values:
📐 Mathematical Derivations
Mathematical Formulations:
① The Recurrent Dropout Problem (Zaremba et al., 2014):
Applying standard dropout to recurrent transitions () zeroes out different hidden dimensions at every step. Signal memory decays as , destroying long-term modeling.
② Variational Dropout (Gal & Ghahramani, NeurIPS 2016):
Interpreted as variational inference in Bayesian RNNs. Sample mask vectors and once at step 0, and freeze them across all time steps :
. Memory integrity is preserved throughout the sequence.
③ Zoneout (Krueger et al., 2016):
Instead of dropping activations to zero, randomly forces units to retain their previous value: , where . Acts as stochastic identity skip connection.
④ WeightDrop (Merity et al., AWD-LSTM):
Applies DropConnect directly to recurrent weight matrix , zeroing out connections rather than activations.
① The Recurrent Dropout Problem (Zaremba et al., 2014):
Applying standard dropout to recurrent transitions () zeroes out different hidden dimensions at every step. Signal memory decays as , destroying long-term modeling.
② Variational Dropout (Gal & Ghahramani, NeurIPS 2016):
Interpreted as variational inference in Bayesian RNNs. Sample mask vectors and once at step 0, and freeze them across all time steps :
. Memory integrity is preserved throughout the sequence.
③ Zoneout (Krueger et al., 2016):
Instead of dropping activations to zero, randomly forces units to retain their previous value: , where . Acts as stochastic identity skip connection.
④ WeightDrop (Merity et al., AWD-LSTM):
Applies DropConnect directly to recurrent weight matrix , zeroing out connections rather than activations.
🏭 Production Trade-offs
Historical significance: AWD-LSTM combined Variational Dropout, WeightDrop, and embedding dropout to achieve state-of-the-art language modeling before Transformers eliminated recurrent architectures entirely.
⚠️ Common Interview Pitfalls
- ✕Applying naive PyTorch
nn.Dropout()directly inside the recurrent loop between and , crippling sequence memory - ✕Failing to scale surviving weights by when implementing custom variational dropout masks
🎯 Interviewer Follow-up Inquiries
- ?Why does freezing the dropout mask across all time steps in Variational Dropout preserve recurrent memory?
- ?How does Zoneout function as an implicit stochastic residual connection in recurrent networks?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚