ViT & Inductive Bias

ViT cuts an image into $p \times p$ patches, flattens and linearly projects them into tokens (prepending a [CLS] token for classification plus position embeddings), then feeds them into a standard Transformer encoder; token count $T = (H/p)(W/p)$ — for 224×224 input and patch 16, $T = 14 \times 14 = 196$ (197 with [CLS]). vs CNNs: CNNs have built-in translation-equivariance and locality inductive biases (weight sharing, local connectivity) that are efficient on small data; ViT only has patch-level positional priors, its attention is global, and it lacks pixel-level locality — hence data hunger: ViT-B/16 trained from scratch on ImageNet-1K (1.28M images) reaches ~74% top-1, clearly trailing same-scale CNNs (~76%), while after JFT-300M (300M images) pretraining, ViT-L/16 reaches ~87.8% top-1, surpassing CNNs by ~1–2 points.

backbone of large vision models and the default vision encoder for multimodality (CLIP, DINOv2, LLaVA all use ViT); interviews ask why ViT needs huge data, the effect of patch size, and how to choose between ViT and CNN.

removing vision-specific inductive biases and processing vision with one unified Transformer architecture — paving the way for unified text-image modeling and multimodal concatenation; with enough data it overtakes CNNs (~74% on small data → ~87.8% on big data), and its token-based design is naturally compatible with serialized multimodal inputs.

Why does ViT need far more training data than CNNs? Answer from the inductive-bias perspective.

Hand-compute patchify: 224×224 input, patch 16 — how many tokens (including [CLS])?

Pros and cons of ViT vs CNN; at what data scale does ViT overtake?

What do position embeddings and the [CLS] token do in ViT? What happens without position encoding?

How do you make ViT work with limited data (distillation, augmentation, smaller patches, local attention/Swin)?