Soft Margin & Hinge

when data are not linearly separable, the hard margin (requiring $y_i(w^Tx_i + b) \ge 1$) is infeasible. The soft margin introduces slack variables $\xi_i \ge 0$ allowing violations at a linear penalty: $\min_{w,b,\xi} \frac{1}{2}\Vert w\Vert^2 + C\sum_{i=1}^{n}\xi_i$ s.t. $y_i(w^Tx_i + b) \ge 1 - \xi_i$. Eliminating $\xi_i$ gives the equivalent unconstrained form $\min_w \frac{1}{2}\Vert w\Vert^2 + C\sum_i \max(0, 1 - y_i f(x_i))$, where the hinge loss $\max(0, 1 - y f(x))$ is a convex upper bound on the $0\text{-}1$ loss and has zero gradient once $y f(x) \ge 1$. The hyperparameter $C$ trades margin width against misclassification penalty: $C \to \infty$ recovers the hard margin (zero tolerance, overfits), $C \to 0$ ignores mistakes (maximum margin, underfits).

interview comparisons of hinge vs log loss, the bias-variance reading of $C$, and how the dual constraint changes from $\alpha_i \ge 0$ to $0 \le \alpha_i \le C$.

a single outlier can completely redraw a hard-margin boundary; the hinge's linear penalty makes the boundary robust — it only penalizes samples inside the margin ($y f(x) < 1$) and gives zero gradient to correct high-confidence ones ($y f(x) \ge 1$), a sparsity the log loss (which penalizes everything continuously) lacks. The dual stays convex QP with only the box constraint $0 \le \alpha_i \le C$, so the kernel trick applies unchanged — support vectors split into on-margin ($0 < \alpha_i < C$) and inside-margin ($\alpha_i = C$) types.

Test Your Knowledge: Practice Questions for "Soft Margin & Hinge"

Single choice pitfall questions with instant feedback and mistake tracking.