Padding & Packing

Padding pads sequences of unequal length in a batch to the same length for GPU matrix alignment — usually dynamically to the batch's longest sequence (or a multiple of 8/16); Packing concatenates multiple short sequences without separators into one long sequence, eliminating pad-token compute waste entirely, with attention/loss masks blocking cross-sample boundaries during training.

batching in training and inference — dynamic padding saves significant compute vs fixed max length; on datasets with skewed lengths (e.g., instruction data), pack samples up to $L_{max}$; also a prerequisite for efficient FlashAttention training.

with fixed-length padding, pad tokens can account for 50%+ of compute when short texts dominate, and pads still produce meaningless attention; packing raises useful-token utilization to near 100%, but sample seams must be masked or the model learns to predict across boundaries — a wrong pattern.

Test Your Knowledge: Practice Questions for "Padding & Packing"

Single choice pitfall questions with instant feedback and mistake tracking.