M2-037M2: Classical Machine LearningSupport Vector Machines & KernelsEasy
⚡Flashcard Drill➔Mastery:

Formulate Soft-Margin SVM, Slack Variables, and the Regularization Role of Parameter C

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-038 →
📐 Mathematical Definition
min⁡12∥w∥2+C∑iξi s.t. yi(w⊤xi+b)≥1−ξi\min\tfrac12\|w\|^2+C\sum_i\xi_i\ \text{s.t.}\ y_i(w^\top x_i+b)\ge1-\xi_i
⚡ One-Sentence Core Answer
Soft-margin SVM introduces slack variables ξi≥0\xi_i \ge 0 to tolerate non-separable margin violations, with hyperparameter CC governing the tradeoff between margin width and classification error penalties.

📌 Key Takeaways

  • •
    Primal Objective: min⁡w,b,ξ12∥w∥22+C∑i=1Nξi\min_{w, b, \xi} \frac{1}{2}\|w\|_2^2 + C \sum_{i=1}^N \xi_i subject to yi(wTxi+b)≥1−ξiy_i(w^T x_i + b) \ge 1 - \xi_i and ξi≥0\xi_i \ge 0.
  • •
    Slack Variable ξi\xi_i: ξi=0\xi_i = 0 (correctly classified beyond margin); 0<ξi≤10 < \xi_i \le 1 (correctly classified inside margin); ξi>1\xi_i > 1 (misclassified).
  • •
    Parameter CC: Large CC heavily penalizes errors (narrow margin, high variance / overfitting); small CC tolerates errors (wide margin, high bias / underfitting).

📐 Mathematical Derivations

Equivalence to Hinge Loss: Rearranging the constraint yi(wTxi+b)≥1−ξiy_i(w^T x_i + b) \ge 1 - \xi_i gives ξi≥1−yi(wTxi+b)\xi_i \ge 1 - y_i(w^T x_i + b). Since ξi≥0\xi_i \ge 0, the minimal feasible slack variable is ξi=max⁡(0,1−yi(wTxi+b))\xi_i = \max(0, 1 - y_i(w^T x_i + b)). Substituting ξi\xi_i into the primal objective yields the unconstrained formulation: min⁡w,b∑i=1Nmax⁡(0,1−yi(wTxi+b))+12C∥w∥22\min_{w, b} \sum_{i=1}^N \max(0, 1 - y_i(w^T x_i + b)) + \frac{1}{2C} \|w\|_2^2. This reveals that Soft-Margin SVM is mathematically equivalent to L2-regularized Empirical Risk Minimization using the Hinge Loss function Lhinge(z)=max⁡(0,1−z)L_{\text{hinge}}(z) = \max(0, 1 - z) with regularization parameter λ=1C\lambda = \frac{1}{C}.

🏭 Production Trade-offs

Dual problem formulation: max⁡α∑i=1Nαi−12∑i=1N∑j=1NαiαjyiyjxiTxj\max_\alpha \sum_{i=1}^N \alpha_i - \frac{1}{2}\sum_{i=1}^N \sum_{j=1}^N \alpha_i \alpha_j y_i y_j x_i^T x_j subject to 0≤αi≤C0 \le \alpha_i \le C and ∑αiyi=0\sum \alpha_i y_i = 0. Notice that CC appears strictly as an upper bound (box constraint) on the dual multipliers αi\alpha_i, preventing any single outlier from exerting unbounded influence on the decision hyperplane.
⚠️ Common Interview Pitfalls
  • ✕
    Setting C=∞C = \infty on noisy or non-linearly separable data (causes solver failure or extreme overfitting).
  • ✕
    Confusing the role of CC in SVM (CC is inversely proportional to regularization λ\lambda: large CC means weak regularization, small CC means strong regularization).
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is the Hinge loss piecewise linear and non-differentiable at z=1z=1, and how does SMO handle its subgradients?
  • ?
    What happens to the support vectors when C→0C \to 0 vs C→∞C \to \infty?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-036: Explain the Maximum Margin Principle in Support Vector Machines and the Role of Support Vectors📋 Back to BankNext →M2-038: Explain the Kernel Trick, Mercer's Condition, and Implicit High-Dimensional Inner Products