Inductive Bias

Inductive bias is the set of built-in prior assumptions in a model/algorithm that makes it prefer certain hypotheses over others given finite data, enabling generalization to unseen data. Typical architectural biases: CNN convolutions + weight sharing → translation equivariance (the same filters scan the whole image, outputs translate with the input) and locality (receptive fields) → orders of magnitude fewer parameters than same-size MLPs and better sample efficiency; pooling → near-invariance to small translations; RNN/LSTM → natural temporal dependency modeling; the Transformer's attention is permutation-equivariant by default (permuting input tokens only permutes output positions, not values), so positional encoding (e.g. RoPE's relative positions) must be injected explicitly to exploit order information.

the unified answer framework for interview questions like why CNNs are more sample-efficient than MLPs, why Transformers need positional encoding, and why regularization improves generalization; model selection is essentially selecting a set of biases.

the no-free-lunch theorem says an algorithm without any bias cannot beat random guessing on average over all problems — bias is the source of sample efficiency, but a wrong bias is a systematic error (the high-bias side of the bias-variance tradeoff). Regularization is also bias injection: weight decay → small-norm (smoothness) prior; early stopping → low-norm / flat-minimum prior; data augmentation → invariance prior (flips, crops); dropout → redundancy/ensemble prior. Modern LLMs instead pursue minimal architectural bias plus massive data and scaling laws — a striking contrast with classical strong-bias + small-data models.

Test Your Knowledge: Practice Questions for "Inductive Bias"

Single choice pitfall questions with instant feedback and mistake tracking.