SVM Margin & Duality

SVM finds the hyperplane with maximum geometric margin. In the separable case the constraint $y_i(w^Tx_i + b) \ge 1$ (normalized functional margin) yields geometric margin $\frac{2}{\Vert w\Vert}$, so maximizing it is equivalent to minimizing $\frac{1}{2}\Vert w\Vert^2$. With Lagrange multipliers $\alpha_i \ge 0$, $L(w, b, \alpha) = \frac{1}{2}\Vert w\Vert^2 - \sum_{i=1}^{n} \alpha_i [y_i(w^Tx_i + b) - 1]$; setting derivatives w.r.t. $w, b$ to zero gives $w = \sum_i \alpha_i y_i x_i$ and $\sum_i \alpha_i y_i = 0$, and substituting back yields the dual $\max_{\alpha} \sum_i \alpha_i - \frac{1}{2}\sum_{i,j} \alpha_i \alpha_j y_i y_j x_i^Tx_j$ s.t. $\alpha_i \ge 0,\ \sum_i \alpha_i y_i = 0$. The KKT complementary-slackness condition $\alpha_i[y_i(w^Tx_i + b) - 1] = 0$ means only samples on the margin boundary ($y_i(w^Tx_i+b) = 1$) have $\alpha_i > 0$; these are the support vectors, and $f(x) = \sum_{i \in SV} \alpha_i y_i x_i^Tx + b$ depends only on them.

the classic whiteboard 'derive the SVM dual' and 'why do only support vectors matter'; prerequisite for the kernel trick (dual inner products $x_i^Tx_j$ get replaced by a kernel).

the primal is convex QP with inequality constraints where KKT is both necessary and sufficient; Lagrangian duality rewrites it in terms of inner products only, which kernels can replace for implicit high-dimensional mapping — and complementary slackness compresses the model to $n_{SV}$ support vectors: $O(n_{SV} \cdot d)$ storage, and each prediction is just inner products with support vectors, decoupled from the full training set.

Test Your Knowledge: Practice Questions for "SVM Margin & Duality"

Single choice pitfall questions with instant feedback and mistake tracking.