M4-006M4: Sequences & TransformersRecurrent Models (RNN/LSTM/GRU)Hard
⚡Flashcard Drill➔Mastery:

Regularization Techniques in the Recurrent Era: Variational Dropout and Zoneout

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-007 →
📐 Mathematical Definition
naive dropout on ht breaks memory;variational: ht=f(W⋅D(ht−1))\text{naive dropout on }h_t\ \text{breaks memory};\qquad \text{variational}:\ h_t=f(W\cdot D(h_{t-1}))
⚡ One-Sentence Core Answer
Standard dropout corrupts memory across time steps; recurrent networks require Variational Dropout (freezing the mask across time), Zoneout (stochastic state identity), and weight drop.

📌 Key Takeaways

  • •
    Standard Dropout defect: sampling independent dropout masks at each time step destroys the cell state's memory persistence
  • •
    Variational Dropout (Gal & Ghahramani): samples a single dropout mask per sequence and applies the exact same mask across all time steps
  • •
    Zoneout (Krueger et al.): stochastically preserves previous cell state values: ct=dt⊙ct−1+(1−dt)⊙ctnewc_t = d_t \odot c_{t-1} + (1 - d_t) \odot c_t^{\text{new}}

📐 Mathematical Derivations

Mathematical Formulations:
① The Recurrent Dropout Problem (Zaremba et al., 2014):
Applying standard dropout to recurrent transitions (ht=σ(W(mt⊙ht−1)+Uxt)h_t = \sigma(W (m_t \odot h_{t-1}) + U x_t)) zeroes out different hidden dimensions at every step. Signal memory decays as (1−p)T→0(1-p)^T \to 0, destroying long-term modeling.
② Variational Dropout (Gal & Ghahramani, NeurIPS 2016):
Interpreted as variational inference in Bayesian RNNs. Sample mask vectors zx∼Bernoulli(1−px)z_x \sim \text{Bernoulli}(1-p_x) and zh∼Bernoulli(1−ph)z_h \sim \text{Bernoulli}(1-p_h) once at step 0, and freeze them across all time steps t=1,…,Tt=1, \dots, T:
ht=tanh⁡(W(zh⊙ht−1)+U(zx⊙xt))h_t = \tanh(W (z_h \odot h_{t-1}) + U (z_x \odot x_t)). Memory integrity is preserved throughout the sequence.
③ Zoneout (Krueger et al., 2016):
Instead of dropping activations to zero, randomly forces units to retain their previous value: ht=dt⊙ht−1+(1−dt)⊙h~th_t = d_t \odot h_{t-1} + (1 - d_t) \odot \tilde{h}_t, where dt∼Bernoulli(p)d_t \sim \text{Bernoulli}(p). Acts as stochastic identity skip connection.
④ WeightDrop (Merity et al., AWD-LSTM):
Applies DropConnect directly to recurrent weight matrix WhhW_{hh}, zeroing out connections rather than activations.

🏭 Production Trade-offs

Historical significance: AWD-LSTM combined Variational Dropout, WeightDrop, and embedding dropout to achieve state-of-the-art language modeling before Transformers eliminated recurrent architectures entirely.
⚠️ Common Interview Pitfalls
  • ✕
    Applying naive PyTorch nn.Dropout() directly inside the recurrent loop between ht−1h_{t-1} and hth_t, crippling sequence memory
  • ✕
    Failing to scale surviving weights by 1/(1−p)1/(1-p) when implementing custom variational dropout masks
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does freezing the dropout mask across all time steps in Variational Dropout preserve recurrent memory?
  • ?
    How does Zoneout function as an implicit stochastic residual connection in recurrent networks?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-005: The Absolute Necessity of Gradient Clipping in Recurrent Neural Networks📋 Back to BankNext →M4-007: Seq2Seq Architecture and the Fixed-Vector Information Bottleneck