SVM Margin & Duality

SVM finds the hyperplane with maximum geometric margin. In the separable case the constraint $y_i(w^Tx_i + b) \ge 1$ (normalized functional margin) yields geometric margin $\frac{2}{\Vert w\Vert}$, so maximizing it is equivalent to minimizing $\frac{1}{2}\Vert w\Vert^2$. With Lagrange multipliers $\alpha_i \ge 0$, $L(w, b, \alpha) = \frac{1}{2}\Vert w\Vert^2 - \sum_{i=1}^{n} \alpha_i [y_i(w^Tx_i + b) - 1]$; setting derivatives w.r.t. $w, b$ to zero gives $w = \sum_i \alpha_i y_i x_i$ and $\sum_i \alpha_i y_i = 0$, and substituting back yields the dual $\max_{\alpha} \sum_i \alpha_i - \frac{1}{2}\sum_{i,j} \alpha_i \alpha_j y_i y_j x_i^Tx_j$ s.t. $\alpha_i \ge 0,\ \sum_i \alpha_i y_i = 0$. The KKT complementary-slackness condition $\alpha_i[y_i(w^Tx_i + b) - 1] = 0$ means only samples on the margin boundary ($y_i(w^Tx_i+b) = 1$) have $\alpha_i > 0$; these are the support vectors, and $f(x) = \sum_{i \in SV} \alpha_i y_i x_i^Tx + b$ depends only on them.

the classic whiteboard 'derive the SVM dual' and 'why do only support vectors matter'; prerequisite for the kernel trick (dual inner products $x_i^Tx_j$ get replaced by a kernel).

the primal is convex QP with inequality constraints where KKT is both necessary and sufficient; Lagrangian duality rewrites it in terms of inner products only, which kernels can replace for implicit high-dimensional mapping — and complementary slackness compresses the model to $n_{SV}$ support vectors: $O(n_{SV} \cdot d)$ storage, and each prediction is just inner products with support vectors, decoupled from the full training set.

Whiteboard the SVM dual: write $L = \frac{1}{2}\Vert w\Vert^2 - \sum_i \alpha_i[y_i(w^Tx_i + b) - 1]$, take $w, b$ derivatives, substitute back, and state the full dual objective with constraints.

Geometric meaning of KKT complementary slackness $\alpha_i[y_i(w^Tx_i+b) - 1] = 0$: why $\alpha_i = 0$ for non-support vectors and how support vectors are identified?

Geometric vs functional margin: derive why the margin is $\frac{2}{\Vert w\Vert}$ and why we normalize the functional margin.

Derive $f(x) = \sum_{i \in SV} \alpha_i y_i x_i^Tx + b$: why is $w$ a linear combination of support vectors, and what are the train/infer complexities?

Why can the dual be kernelized? Which terms in the dual contain only inner products, and where does the primal lack inner products?