🧠 Deep LearningID: rnn-basics

RNN & BPTT

Q: Write the vanilla RNN forward pass $h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h)$; why are weights shared across time steps and why is the parameter count independent of $T$?

【RNN & BPTT 核心考点 1】Write the vanilla RNN forward pass $h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h)$; why are weights shared across time steps and why is the parameter count independent of $T$?。技术解析：Definition: An RNN (recurrent neural network) is the fundamental model for sequential data, defined by weight sharing across time steps. The hidden-state recurrence h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h) with output y_t = W_{hy}h_t reuses the same W_{hh}, W_{xh} at every st... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/rnn-basics

Q: How does BPTT differ from standard backprop? How does the error travel through $\frac{\partial h_t}{\partial h_{t-1}}$ chain products, and why is the gradient w.r.t. $W_{hh}$ a sum of contributions from all time steps?

【RNN & BPTT 核心考点 2】How does BPTT differ from standard backprop? How does the error travel through $\frac{\partial h_t}{\partial h_{t-1}}$ chain products, and why is the gradient w.r.t. $W_{hh}$ a sum of contributions from all time steps?。技术解析：Definition: An RNN (recurrent neural network) is the fundamental model for sequential data, defined by weight sharing across time steps. The hidden-state recurrence h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h) with output y_t = W_{hy}h_t reuses the same W_{hh}, W_{xh} at every st... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/rnn-basics

Q: Derive vanishing/exploding gradients from $\frac{\partial h_T}{\partial h_1} = \prod_{t=2}^{T} W_{hh}^T \mathrm{diag}(\tanh')$; how do the spectral radius $\rho(W_{hh})$ and the activation derivative determine whether long-range dependencies are learnable ($0.9^{50} \approx 0.005$)?

【RNN & BPTT 核心考点 3】Derive vanishing/exploding gradients from $\frac{\partial h_T}{\partial h_1} = \prod_{t=2}^{T} W_{hh}^T \mathrm{diag}(\tanh')$; how do the spectral radius $\rho(W_{hh})$ and the activation derivative determine whether long-range dependencies are learnable ($0.9^{50} \approx 0.005$)?。技术解析：Definition: An RNN (recurrent neural network) is the fundamental model for sequential data, defined by weight sharing across time steps. The hidden-state recurrence h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h) with output y_t = W_{hy}h_t reuses the same W_{hh}, W_{xh} at every st... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/rnn-basics

Q: List the mitigations: why does gradient clipping (e.g. clip 1.0) fix exploding but not vanishing gradients? What roles do orthogonal initialization and gating play?

【RNN & BPTT 核心考点 4】List the mitigations: why does gradient clipping (e.g. clip 1.0) fix exploding but not vanishing gradients? What roles do orthogonal initialization and gating play?。技术解析：Definition: An RNN (recurrent neural network) is the fundamental model for sequential data, defined by weight sharing across time steps. The hidden-state recurrence h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h) with output y_t = W_{hy}h_t reuses the same W_{hh}, W_{xh} at every st... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/rnn-basics

Q: Why can't RNNs be trained in parallel ($h_t$ depends on $h_{t-1}$)? What does per-step complexity $O(d^2)$ mean? How does this contrast with Transformer ($O(L^2)$, parallel) and Mamba ($O(L)$, parallel)?

【RNN & BPTT 核心考点 5】Why can't RNNs be trained in parallel ($h_t$ depends on $h_{t-1}$)? What does per-step complexity $O(d^2)$ mean? How does this contrast with Transformer ($O(L^2)$, parallel) and Mamba ($O(L)$, parallel)?。技术解析：Definition: An RNN (recurrent neural network) is the fundamental model for sequential data, defined by weight sharing across time steps. The hidden-state recurrence h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h) with output y_t = W_{hy}h_t reuses the same W_{hh}, W_{xh} at every st... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/rnn-basics

RNN 与 BPTT

🎯Core Definition

An RNN (recurrent neural network) is the fundamental model for sequential data, defined by weight sharing across time steps. The hidden-state recurrence

h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h)

with output

y_t = W_{hy}h_t

reuses the same

W_{hh}, W_{xh}

at every step, unlike the layer-wise weights of DNNs/CNNs. Training uses BPTT (backpropagation through time): unroll the temporal dimension into a

T

-layer virtual network and backpropagate, so the gradient flows through the chain product

\frac{\partial h_T}{\partial h_1} = \prod_{t=2}^{T} \frac{\partial h_t}{\partial h_{t-1}} = \prod_{t=2}^{T} W_{hh}^T \mathrm{diag}(\tanh'(h_{t-1}))

— exactly why gradients vanish or explode: if the spectral radius

\rho(W_{hh}) < 1

the product decays exponentially (

0.9^{50} \approx 0.005

) and dependencies 50 steps apart are essentially unlearnable; if

> 1

gradients explode. This is the long-term dependency problem.

💡Use Cases

the entry-point model for sequential data (text, speech, time series) and the first link of the interview-favorite evolution chain “RNN → LSTM/GRU → Transformer/Mamba”; frequent follow-ups include deriving BPTT, contrasting it with standard backprop, and explaining vanishing/exploding gradients with mitigations (gradient clipping, orthogonal initialization, gating, truncated BPTT).

⚡Key Problems Solved

compared with fixed-window models (N-grams, 1D-CNNs), weight sharing keeps the parameter count independent of sequence length (

O(d^2)

per step), handles variable-length inputs, and can in principle encode arbitrary long-range dependencies. The cost is that chain-multiplied gradients destroy long-term memory, so LSTM/GRU replace the purely multiplicative path with additive gating, and Transformer/Mamba further improve on parallelism and complexity.

🎯5 High-Frequency Exam Points

Write the vanilla RNN forward pass

h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h)

; why are weights shared across time steps and why is the parameter count independent of

T

How does BPTT differ from standard backprop? How does the error travel through

\frac{\partial h_t}{\partial h_{t-1}}

chain products, and why is the gradient w.r.t.

W_{hh}

a sum of contributions from all time steps?

Derive vanishing/exploding gradients from

\frac{\partial h_T}{\partial h_1} = \prod_{t=2}^{T} W_{hh}^T \mathrm{diag}(\tanh')

; how do the spectral radius

\rho(W_{hh})

and the activation derivative determine whether long-range dependencies are learnable (

0.9^{50} \approx 0.005

List the mitigations: why does gradient clipping (e.g. clip 1.0) fix exploding but not vanishing gradients? What roles do orthogonal initialization and gating play?

Why can't RNNs be trained in parallel (