SFT Data Packing & Loss Masking

SFT Data Packing & Loss Masking are foundational data engineering mechanisms in Supervised Fine-Tuning maximizing training throughput while strictly focusing parameter updates on target completions; it encompasses two vital techniques: 1) Loss Masking (Prompt Label Masking): setting all input prompt and user query tokens to `-100` (PyTorch's `ignore_index`), restricting cross-entropy gradient backpropagation exclusively to Assistant completion tokens to prevent capacity waste on prompt memorization; 2) Data Packing (Sequence Packing): concatenating multiple short conversations separated by `<|endoftext|>` into fixed 4096/8192 sequence blocks, eliminating wasteful `<pad>` tokens, combined with FlashAttention variable-length cu_seqlens masks to prevent cross-sample attention contamination, boosting throughput by 3-5x.

Enterprise domain-specific SFT training, multi-turn conversational fine-tuning, and large-scale synthetic instruction scaling.

Naive padding squanders 70% of GPU compute on useless pad tokens, while computing loss on prompts causes severe overfitting to prompt phrasing; packing and masking achieve near 100% compute efficiency.

Why must prompt tokens be masked with label `-100` during SFT, and what severe degradation occurs if loss is computed on prompts?

How does FlashAttention's variable-length API (`cu_seqlens`) prevent cross-document attention contamination in packed sequences?

How to construct token masks in multi-turn dialogues so that loss is computed solely on the final (or all) Assistant turns while conditioning on previous history?

Quantify the 3-5x speedup and memory efficiency gains of Data Packing over conventional batched padding?

How do Jinja2 Chat Templates standardize special token delimiters across Llama-3, Qwen-2.5, and Mistral in production SFT data pipelines?