M2-039M2: Classical Machine LearningSupport Vector Machines & KernelsMedium
⚡Flashcard Drill➔Mastery:
Compare Linear, Polynomial, and RBF Kernels and Their Decision Boundaries
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Linear kernel is optimal for high-dimensional sparse data (); RBF is the universal default for complex non-linear boundaries in moderate dimensions; Polynomial kernel is specialized for structured interactions (e.g. Computer vision/NLP degree-2 crossings).
📌 Key Takeaways
- •Linear Kernel: ; training and inference, fastest execution, ideal when feature count is massive (e.g. Text classification with ).
- •RBF (Gaussian) Kernel: ; universal approximator; hyperparameter dictates locality (large causes tight islands/overfitting).
- •Polynomial Kernel: ; models explicit feature combinations up to degree ; computationally sensitive to numerical scaling.
📐 Mathematical Derivations
Analysis of RBF kernel parameter : In , parameter governs the radius of influence of individual support vectors. (1) As , for all , meaning each support vector forms an isolated Gaussian spike around itself (extreme memorization / high variance). (2) As , , the kernel becomes a flat constant, flattening the decision boundary into a linear plane (high bias). In the polynomial kernel , if inputs are unnormalized with , raising to power triggers floating-point overflow.
🏭 Production Trade-offs
Selection rules of thumb (Hsu et al., 2003): (1) If (features far exceed samples, e.g. Genomics, text TF-IDF), use Linear Kernel (data is already separable in high-dimensional space without kernel mapping; RBF offers zero gain and risks overfitting). (2) If is moderate () and is small-to-medium, use RBF Kernel with grid search over . (3) If , switch to tree ensembles (LightGBM) or neural networks.
⚠️ Common Interview Pitfalls
- ✕Using RBF kernel when and (adds massive compute overhead with zero accuracy benefit over linear SVM).
- ✕Setting too large in RBF SVM without realizing it produces 100% training accuracy with 50% random test accuracy.
🎯 Interviewer Follow-up Inquiries
- ?Why is linear SVM equivalent to a special case of RBF SVM under specific parameter limits?
- ?How does the Sigmoid (Hyperbolic Tangent) kernel connect SVMs to two-layer neural networks?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚