SSM & Hybrid Architectures

SSM (State Space Model) models sequences with a linear recurrence: $h_t = A h_{t-1} + B x_t$, $y_t = C h_t$, with fixed state size independent of sequence length; Mamba adds input-dependent selective scanning (using $\Delta_t$ to gate what enters and leaves the state), reaching $O(n)$; linear attention (RWKV) replaces $Q K^T$ with a kernelized form, equivalently collapsing into constant-size state recurrence; hybrid architectures (e.g. Jamba) stack full attention layers and SSM layers alternately.

million-token long context and high-throughput inference; Jamba-1.5 398B (MoE, 52B active) serves 256K context — by 2026 hybrids are the mainstream route for long context, and comparing SSM vs attention complexity and memory is a frequent interview topic.

attention's $O(n^2)$ time/memory grows quadratically with sequence length; SSM keeps per-token memory constant (state $d \times m$) and trains via parallel scan; but a fixed-size state cannot precisely recall history, so pure SSMs underperform on memory/retrieval tasks — hybrids add a few attention layers to restore exact retrieval and approach full-attention quality at lower cost.

Test Your Knowledge: Practice Questions for "SSM & Hybrid Architectures"

Single choice pitfall questions with instant feedback and mistake tracking.