LSTM Gating

LSTM (Long Short-Term Memory) solves the RNN long-term dependency problem with “three gates and one state”. Forget gate $f_t = \sigma(W_f[h_{t-1}, x_t] + b_f)$, input gate $i_t = \sigma(W_i[h_{t-1}, x_t] + b_i)$, candidate memory $\tilde{c}_t = \tanh(W_c[h_{t-1}, x_t] + b_c)$; cell state $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$, output gate $o_t = \sigma(W_o[h_{t-1}, x_t] + b_o)$, hidden state $h_t = o_t \odot \tanh(c_t)$ ($\odot$ is elementwise product, $[h_{t-1}, x_t]$ concatenation). The key is that the cell state is an additive path: $\frac{\partial c_t}{\partial c_{t-1}} = f_t$ is a scalar gate instead of a matrix chain product, so when the forget gate is near 1 the gradient passes through almost unchanged over long distances — the core defense against vanishing gradients. Gating costs 4 weight sets on $[h_{t-1}, x_t]$, i.e. roughly 4× the parameters of a vanilla RNN. xLSTM (2024) continues this line: sLSTM adds exponential gating with memory mixing, and mLSTM uses a matrix memory with a covariance update rule for higher capacity.

the standard answer for sequence modeling in the 2010s (MT, speech, forecasting); in interviews it appears mainly in comparison questions — LSTM vs GRU, LSTM vs Transformer, LSTM vs Mamba — with follow-ups on writing out the gate equations, analyzing the additive-path gradient, and forget-gate initialization.

RNN gradients multiply through $\prod W^T \mathrm{diag}(\tanh')$, so long-term memory decays exponentially; LSTM replaces the multiplicative update with $c_t = f_t c_{t-1} + i_t \tilde{c}_t$, turning the chain product into “keep by the gate + write by the gate”. With $f_t \approx 1$ gradients propagate undiminished, making 100+-step dependencies learnable — at the cost of ~4× parameters and two activation computations, in exchange for trainable long-range memory.

Write out the LSTM gate equations (forget/input/output gates, candidate $\tilde{c}_t$, cell state $c_t$, hidden state $h_t$) and explain the role of each gate.

Why does LSTM mitigate vanishing gradients? Derive the additive path $\frac{\partial c_t}{\partial c_{t-1}} = f_t$ and explain why gradients flow through long distances when $f_t \approx 1$.

Why is the forget-gate bias initialized to a large positive value ($b_f \approx 1$)? What happens if it saturates to 0 early (cell cleared, long memory lost)?

Compare LSTM vs GRU: GRU merges forget+input into an update gate and drops the separate cell state, cutting weights from 4 sets to 3; what is the capacity-vs-efficiency trade-off?

What does xLSTM (2024) improve: what problems do sLSTM's exponential gating & memory mixing and mLSTM's matrix memory & covariance update solve?