Padding & Packing

Padding pads sequences of unequal length in a batch to the same length for GPU matrix alignment — usually dynamically to the batch's longest sequence (or a multiple of 8/16); Packing concatenates multiple short sequences without separators into one long sequence, eliminating pad-token compute waste entirely, with attention/loss masks blocking cross-sample boundaries during training.

batching in training and inference — dynamic padding saves significant compute vs fixed max length; on datasets with skewed lengths (e.g., instruction data), pack samples up to $L_{max}$; also a prerequisite for efficient FlashAttention training.

with fixed-length padding, pad tokens can account for 50%+ of compute when short texts dominate, and pads still produce meaningless attention; packing raises useful-token utilization to near 100%, but sample seams must be masked or the model learns to predict across boundaries — a wrong pattern.

Dynamic vs fixed-length padding? Why pad to multiples of 8/16?

How does sequence packing remove pad waste? Masking sample seams?

Problems when pads enter attention? Role of the attention mask?

Loss computation under packing? Why a loss mask is needed?

How padding/packing relate to FlashAttention and PagedAttention?