M5-024M5: NLP & Large Language ModelsInstruction Tuning & Supervised Fine-TuningHard
⚡Flashcard Drill➔Mastery:

SFT Data Mixture and Multi-Task Balancing

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-025 →
📐 Mathematical Definition
p(task i)∝niα (α<1 rebalances);or explicit weights wip(\text{task }i)\propto n_i^{\alpha}\ (\alpha<1\ \text{rebalances});\qquad \text{or explicit weights }w_i
⚡ One-Sentence Core Answer
SFT multi-task balancing allocates data proportions across dialogue, coding, mathematics, reasoning, and safety to prevent over-fitting to single modalities while maintaining generalist conversational capabilities.

📌 Key Takeaways

  • •
    Core objective: prevent task competition and cross-domain interference while maximizing general capabilities across diverse user intents
  • •
    Typical industrial mixture: general chat (30-40%), coding and technical tasks (20-30%), mathematical reasoning (15-20%), structured extraction/tools (10-15%), safety & refusal (5%)
  • •
    Temperature-based rebalancing: prevents dominant high-volume datasets from drowning out small, critical datasets (like safety refusals or multi-turn tool calling)

📐 Mathematical Derivations

Mathematical Mechanism: 1. Multi-Task SFT Loss: Given MM distinct task datasets D1,…,DM\mathcal{D}_1, \dots, \mathcal{D}_M with dataset sizes N1,…,NMN_1, \dots, N_M: L(θ)=∑m=1MwmE(x,y)∼Dm[−∑t=1∣y∣log⁡Pθ(yt∣x,y<t)]\mathcal{L}(\theta) = \sum_{m=1}^M w_m \mathbb{E}_{(x, y) \sim \mathcal{D}_m} \left[ -\sum_{t=1}^{|y|} \log P_\theta(y_t \mid x, y_{<t}) \right] 2. Temperature Sampling Rebalancing: If data is sampled proportional to raw size NmN_m, a massive chat dataset (100k100\text{k} examples) completely starves a small safety dataset (2k2\text{k} examples). Temperature rebalancing scales sampling probability by temperature τ≥1\tau \ge 1: pm=Nm1/τ∑j=1MNj1/τp_m = \frac{N_m^{1 / \tau}}{\sum_{j=1}^M N_j^{1 / \tau}} - When τ=1\tau = 1: sampling is proportional to raw dataset size. - When τ→∞\tau \to \infty: sampling is completely uniform across all tasks (pm=1/Mp_m = 1/M). - Standard practice uses τ∈[2,5]\tau \in [2, 5] to boost the presence of high-value, low-volume datasets without extreme duplication.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① Task Conflict Dynamics: Certain tasks exhibit negative gradient transfer: aggressive safety training degrades coding and math performance (causing false refusal of benign code exploits); aggressive short-answer training degrades detailed multi-step reasoning. Domain weights must be tuned to keep refusal rates bounded. ② Maximum Sample Capping: A simple, robust alternative to temperature sampling is hard capping: set a maximum threshold Nmax⁡=20,000N_{\max} = 20,000 samples per task category, ensuring no single dataset dominates the training run. ③ Role of System Prompts: Prepend task-specific system prompts (e.g., 'You are an expert mathematician', 'You are a helpful coding assistant') to help the Transformer's attention heads condition on distinct behavioral modes, minimizing cross-task interference. ④ Multi-Turn vs Single-Turn Balancing: Maintain at least 30–40%30\text{--}40\% multi-turn conversational data; models trained exclusively on single-turn Q&A fail to track context across extended dialogues. ⑤ Interview Strategy: Present a realistic industrial data mixture breakdown, derive the temperature sampling formula Nm1/τN_m^{1/\tau}, and discuss how system prompts and sample caps mitigate negative task transfer.
⚠️ Common Interview Pitfalls
  • ✕
    Training on raw dataset proportions where a large chat dataset starves critical small datasets like tool-calling or safety
  • ✕
    Over-indexing on safety data, leading to severe false-refusal rates on programming and security queries
  • ✕
    Omitting multi-turn conversation data during SFT (leads to context amnesia in chat applications)
🎯 Interviewer Follow-up Inquiries
  • ?
    How does temperature parameter τ\tau balance sample diversity against overfitting on small datasets in multi-task SFT?
  • ?
    What causes negative transfer between safety refusal training and code vulnerability analysis?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-023: Chain-of-Thought (CoT) Distillation: Benefits and Hallucination Risks📋 Back to BankNext →M5-025: Division of Labor Between SFT and RLHF/DPO