RNN & BPTT

An RNN (recurrent neural network) is the fundamental model for sequential data, defined by weight sharing across time steps. The hidden-state recurrence $h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h)$ with output $y_t = W_{hy}h_t$ reuses the same $W_{hh}, W_{xh}$ at every step, unlike the layer-wise weights of DNNs/CNNs. Training uses BPTT (backpropagation through time): unroll the temporal dimension into a $T$-layer virtual network and backpropagate, so the gradient flows through the chain product $\frac{\partial h_T}{\partial h_1} = \prod_{t=2}^{T} \frac{\partial h_t}{\partial h_{t-1}} = \prod_{t=2}^{T} W_{hh}^T \mathrm{diag}(\tanh'(h_{t-1}))$ — exactly why gradients vanish or explode: if the spectral radius $\rho(W_{hh}) < 1$ the product decays exponentially ($0.9^{50} \approx 0.005$) and dependencies 50 steps apart are essentially unlearnable; if $> 1$ gradients explode. This is the long-term dependency problem.

the entry-point model for sequential data (text, speech, time series) and the first link of the interview-favorite evolution chain “RNN → LSTM/GRU → Transformer/Mamba”; frequent follow-ups include deriving BPTT, contrasting it with standard backprop, and explaining vanishing/exploding gradients with mitigations (gradient clipping, orthogonal initialization, gating, truncated BPTT).

compared with fixed-window models (N-grams, 1D-CNNs), weight sharing keeps the parameter count independent of sequence length ($O(d^2)$ per step), handles variable-length inputs, and can in principle encode arbitrary long-range dependencies. The cost is that chain-multiplied gradients destroy long-term memory, so LSTM/GRU replace the purely multiplicative path with additive gating, and Transformer/Mamba further improve on parallelism and complexity.

Write the vanilla RNN forward pass $h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h)$; why are weights shared across time steps and why is the parameter count independent of $T$?

How does BPTT differ from standard backprop? How does the error travel through $\frac{\partial h_t}{\partial h_{t-1}}$ chain products, and why is the gradient w.r.t. $W_{hh}$ a sum of contributions from all time steps?

Derive vanishing/exploding gradients from $\frac{\partial h_T}{\partial h_1} = \prod_{t=2}^{T} W_{hh}^T \mathrm{diag}(\tanh')$; how do the spectral radius $\rho(W_{hh})$ and the activation derivative determine whether long-range dependencies are learnable ($0.9^{50} \approx 0.005$)?

List the mitigations: why does gradient clipping (e.g. clip 1.0) fix exploding but not vanishing gradients? What roles do orthogonal initialization and gating play?

Why can't RNNs be trained in parallel ($h_t$ depends on $h_{t-1}$)? What does per-step complexity $O(d^2)$ mean? How does this contrast with Transformer ($O(L^2)$, parallel) and Mamba ($O(L)$, parallel)?