Inductive Bias

Inductive bias is the set of built-in prior assumptions in a model/algorithm that makes it prefer certain hypotheses over others given finite data, enabling generalization to unseen data. Typical architectural biases: CNN convolutions + weight sharing → translation equivariance (the same filters scan the whole image, outputs translate with the input) and locality (receptive fields) → orders of magnitude fewer parameters than same-size MLPs and better sample efficiency; pooling → near-invariance to small translations; RNN/LSTM → natural temporal dependency modeling; the Transformer's attention is permutation-equivariant by default (permuting input tokens only permutes output positions, not values), so positional encoding (e.g. RoPE's relative positions) must be injected explicitly to exploit order information.

the unified answer framework for interview questions like why CNNs are more sample-efficient than MLPs, why Transformers need positional encoding, and why regularization improves generalization; model selection is essentially selecting a set of biases.

the no-free-lunch theorem says an algorithm without any bias cannot beat random guessing on average over all problems — bias is the source of sample efficiency, but a wrong bias is a systematic error (the high-bias side of the bias-variance tradeoff). Regularization is also bias injection: weight decay → small-norm (smoothness) prior; early stopping → low-norm / flat-minimum prior; data augmentation → invariance prior (flips, crops); dropout → redundancy/ensemble prior. Modern LLMs instead pursue minimal architectural bias plus massive data and scaling laws — a striking contrast with classical strong-bias + small-data models.

What is inductive bias? How do CNN translation equivariance and weight sharing buy sample efficiency (vs a same-size MLP)?

Why must the Transformer add positional encoding? Formalize the permutation equivariance of attention (permuting inputs only permutes output positions).

Why are weight decay / early stopping / data augmentation / dropout also inductive biases? What prior does each inject?

Pros and risks of strong bias: when is a bias wrong? How does this relate to the bias-variance tradeoff?

How do modern LLMs trade strong architectural bias for weak bias plus massive data? What are the costs and benefits?