🚀 AI Systems EngineerID:
aie-sft-data-packing-loss-maskingSFT Data Packing & Loss Masking
SFT Data Packing 与 Loss Masking
🎯Core Definition
SFT Data Packing & Loss Masking are foundational data engineering mechanisms in Supervised Fine-Tuning maximizing training throughput while strictly focusing parameter updates on target completions; it encompasses two vital techniques: 1) Loss Masking (Prompt Label Masking): setting all input prompt and user query tokens to `-100` (PyTorch's `ignore_index`), restricting cross-entropy gradient backpropagation exclusively to Assistant completion tokens to prevent capacity waste on prompt memorization; 2) Data Packing (Sequence Packing): concatenating multiple short conversations separated by `<|endoftext|>` into fixed 4096/8192 sequence blocks, eliminating wasteful `<pad>` tokens, combined with FlashAttention variable-length cu_seqlens masks to prevent cross-sample attention contamination, boosting throughput by 3-5x.
💡Use Cases
Enterprise domain-specific SFT training, multi-turn conversational fine-tuning, and large-scale synthetic instruction scaling.
⚡Key Problems Solved
Naive padding squanders 70% of GPU compute on useless pad tokens, while computing loss on prompts causes severe overfitting to prompt phrasing; packing and masking achieve near 100% compute efficiency.
🎯5 High-Frequency Exam Points
1
Why must prompt tokens be masked with label `-100` during SFT, and what severe degradation occurs if loss is computed on prompts?
2
How does FlashAttention's variable-length API (`cu_seqlens`) prevent cross-document attention contamination in packed sequences?
3
How to construct token masks in multi-turn dialogues so that loss is computed solely on the final (or all) Assistant turns while conditioning on previous history?
4
Quantify the 3-5x speedup and memory efficiency gains of Data Packing over conventional batched padding?
5
How do Jinja2 Chat Templates standardize special token delimiters across Llama-3, Qwen-2.5, and Mistral in production SFT data pipelines?
🔗Foundational Prerequisite Cards (Click to Review)
📖 In-depth Guide:📄 aie-fine-tuning-and-alignment →
Updated 2026-08-14🎯
🚀 Start Card Practice ➔Test Your Knowledge: Practice Questions for "SFT Data Packing & Loss Masking"
Single choice pitfall questions with instant feedback and mistake tracking.