M2-039M2: Classical Machine LearningSupport Vector Machines & KernelsMedium
⚡Flashcard Drill➔Mastery:

Compare Linear, Polynomial, and RBF Kernels and Their Decision Boundaries

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-040 →
📐 Mathematical Definition
KRBF=e−γ∥x−x′∥2K_{RBF}=e^{-\gamma\|x-x'\|^2}
⚡ One-Sentence Core Answer
Linear kernel is optimal for high-dimensional sparse data (d≫Nd \gg N); RBF is the universal default for complex non-linear boundaries in moderate dimensions; Polynomial kernel is specialized for structured interactions (e.g. Computer vision/NLP degree-2 crossings).

📌 Key Takeaways

  • •
    Linear Kernel: K(x,z)=xTz+cK(x, z) = x^T z + c; O(d)O(d) training and inference, fastest execution, ideal when feature count is massive (e.g. Text classification with d>50,000d > 50,000).
  • •
    RBF (Gaussian) Kernel: K(x,z)=exp⁡(−γ∥x−z∥2)K(x, z) = \exp(-\gamma \|x - z\|^2); universal approximator; hyperparameter γ\gamma dictates locality (large γ\gamma causes tight islands/overfitting).
  • •
    Polynomial Kernel: K(x,z)=(xTz+c)dK(x, z) = (x^T z + c)^d; models explicit feature combinations up to degree dd; computationally sensitive to numerical scaling.

📐 Mathematical Derivations

Analysis of RBF kernel parameter γ\gamma: In K(x,z)=exp⁡(−γ∥x−z∥2)K(x, z) = \exp(-\gamma \|x - z\|^2), parameter γ=12σ2\gamma = \frac{1}{2\sigma^2} governs the radius of influence of individual support vectors. (1) As γ→∞\gamma \to \infty, K(xi,xj)→0K(x_i, x_j) \to 0 for all xi≠xjx_i \ne x_j, meaning each support vector forms an isolated Gaussian spike around itself (extreme memorization / high variance). (2) As γ→0\gamma \to 0, K(x,z)→1K(x, z) \to 1, the kernel becomes a flat constant, flattening the decision boundary into a linear plane (high bias). In the polynomial kernel K(x,z)=(γxTz+c)dK(x, z) = (\gamma x^T z + c)^d, if inputs are unnormalized with ∥x∥>1\|x\| > 1, raising to power dd triggers floating-point overflow.

🏭 Production Trade-offs

Selection rules of thumb (Hsu et al., 2003): (1) If d≫Nd \gg N (features far exceed samples, e.g. Genomics, text TF-IDF), use Linear Kernel (data is already separable in high-dimensional space without kernel mapping; RBF offers zero gain and risks overfitting). (2) If NN is moderate (N<50,000N < 50,000) and dd is small-to-medium, use RBF Kernel with grid search over (C,γ)(C, \gamma). (3) If N>106N > 10^6, switch to tree ensembles (LightGBM) or neural networks.
⚠️ Common Interview Pitfalls
  • ✕
    Using RBF kernel when d>100,000d > 100,000 and N<10,000N < 10,000 (adds massive compute overhead with zero accuracy benefit over linear SVM).
  • ✕
    Setting γ\gamma too large in RBF SVM without realizing it produces 100% training accuracy with 50% random test accuracy.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is linear SVM equivalent to a special case of RBF SVM under specific parameter limits?
  • ?
    How does the Sigmoid (Hyperbolic Tangent) kernel connect SVMs to two-layer neural networks?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-038: Explain the Kernel Trick, Mercer's Condition, and Implicit High-Dimensional Inner Products📋 Back to BankNext →M2-040: Assess the Sensitivity of SVM to Class Imbalance and Label Noise and Industrial Mitigations