SSM & Hybrid Architectures

SSM (State Space Model) models sequences with a linear recurrence: $h_t = A h_{t-1} + B x_t$, $y_t = C h_t$, with fixed state size independent of sequence length; Mamba adds input-dependent selective scanning (using $\Delta_t$ to gate what enters and leaves the state), reaching $O(n)$; linear attention (RWKV) replaces $Q K^T$ with a kernelized form, equivalently collapsing into constant-size state recurrence; hybrid architectures (e.g. Jamba) stack full attention layers and SSM layers alternately.

million-token long context and high-throughput inference; Jamba-1.5 398B (MoE, 52B active) serves 256K context — by 2026 hybrids are the mainstream route for long context, and comparing SSM vs attention complexity and memory is a frequent interview topic.

attention's $O(n^2)$ time/memory grows quadratically with sequence length; SSM keeps per-token memory constant (state $d \times m$) and trains via parallel scan; but a fixed-size state cannot precisely recall history, so pure SSMs underperform on memory/retrieval tasks — hybrids add a few attention layers to restore exact retrieval and approach full-attention quality at lower cost.

SSM recurrence? How is the state updated, and what does fixed state size imply?

What is Mamba's selective mechanism, and why must B, C, $\Delta_t$ depend on the input?

Why do hybrids like Jamba keep attention layers? What role does each component play?

How does RWKV/linear attention's recurrence correspond to standard attention?

Why are SSMs fast to train and cheap at inference (parallel scan + constant state)?