LSTM Gating

LSTM (Long Short-Term Memory) solves the RNN long-term dependency problem with “three gates and one state”. Forget gate $f_t = \sigma(W_f[h_{t-1}, x_t] + b_f)$, input gate $i_t = \sigma(W_i[h_{t-1}, x_t] + b_i)$, candidate memory $\tilde{c}_t = \tanh(W_c[h_{t-1}, x_t] + b_c)$; cell state $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$, output gate $o_t = \sigma(W_o[h_{t-1}, x_t] + b_o)$, hidden state $h_t = o_t \odot \tanh(c_t)$ ($\odot$ is elementwise product, $[h_{t-1}, x_t]$ concatenation). The key is that the cell state is an additive path: $\frac{\partial c_t}{\partial c_{t-1}} = f_t$ is a scalar gate instead of a matrix chain product, so when the forget gate is near 1 the gradient passes through almost unchanged over long distances — the core defense against vanishing gradients. Gating costs 4 weight sets on $[h_{t-1}, x_t]$, i.e. roughly 4× the parameters of a vanilla RNN. xLSTM (2024) continues this line: sLSTM adds exponential gating with memory mixing, and mLSTM uses a matrix memory with a covariance update rule for higher capacity.

the standard answer for sequence modeling in the 2010s (MT, speech, forecasting); in interviews it appears mainly in comparison questions — LSTM vs GRU, LSTM vs Transformer, LSTM vs Mamba — with follow-ups on writing out the gate equations, analyzing the additive-path gradient, and forget-gate initialization.

RNN gradients multiply through $\prod W^T \mathrm{diag}(\tanh')$, so long-term memory decays exponentially; LSTM replaces the multiplicative update with $c_t = f_t c_{t-1} + i_t \tilde{c}_t$, turning the chain product into “keep by the gate + write by the gate”. With $f_t \approx 1$ gradients propagate undiminished, making 100+-step dependencies learnable — at the cost of ~4× parameters and two activation computations, in exchange for trainable long-range memory.

Test Your Knowledge: Practice Questions for "LSTM Gating"

Single choice pitfall questions with instant feedback and mistake tracking.