GRU

GRU (Gated Recurrent Unit) is a simplified LSTM variant with only two gates. The update gate $z_t = \sigma(W_z[h_{t-1}, x_t] + b_z)$ decides how much history to keep vs how much new information to write; the reset gate $r_t = \sigma(W_r[h_{t-1}, x_t] + b_r)$ decides how much of the history to discard; the candidate hidden state is $\tilde{h}_t = \tanh(W_h[r_t \odot h_{t-1}, x_t] + b_h)$; and the output is $h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$ ($\odot$ elementwise). Compared with LSTM: there is no separate cell state, the forget and input gates merge into the update gate, and weights drop from 4 sets to 3 (roughly 25% fewer parameters). When $z_t \approx 0$, $h_t \approx h_{t-1}$ — the same additive shortcut as LSTM's $f_t \approx 1$ that keeps long-range memory; when $r_t \approx 0$, the candidate ignores history and depends only on the current input, capturing short-term dependencies.

the recurrent unit of choice for parameter-sensitive, small-data, or real-time inference settings (speech, forecasting, lightweight sequence models); in interviews it is a staple comparison question — GRU vs LSTM formulas, parameter count, capacity and training efficiency.

LSTM's four gated components are redundant on most tasks; GRU achieves nearly the same long-term memory with 3 weight sets — about 25% fewer parameters and cheaper per-step computation, so it trains faster and overfits less on small data and edge devices. The cost: no dedicated cell state, so stored information is coupled with the output in one $h_t$, a coarser memory granularity.

Write out GRU's update gate $z_t$, reset gate $r_t$, candidate $\tilde{h}_t$ and output $h_t = (1 - z_t)h_{t-1} + z_t\tilde{h}_t$; explain the role of each gate.

Structural differences: why does GRU have ~25% fewer parameters (3 vs 4 weight sets)? How does the absence of a cell state affect memory granularity?

What happens when $z_t \approx 0$ ($h_t \approx h_{t-1}$ additive shortcut)? How does this correspond to LSTM's forget gate $f_t \approx 1$?

Role of the reset gate: when $r_t \approx 0$ the candidate $\tilde{h}_t$ depends only on the current input — why does this help capture short-term dependencies and discard irrelevant history?

When to pick GRU over LSTM (data size, parameter budget, inference latency)? Why is the accuracy gap between them usually small on typical tasks?