Soft Margin & Hinge

when data are not linearly separable, the hard margin (requiring $y_i(w^Tx_i + b) \ge 1$) is infeasible. The soft margin introduces slack variables $\xi_i \ge 0$ allowing violations at a linear penalty: $\min_{w,b,\xi} \frac{1}{2}\Vert w\Vert^2 + C\sum_{i=1}^{n}\xi_i$ s.t. $y_i(w^Tx_i + b) \ge 1 - \xi_i$. Eliminating $\xi_i$ gives the equivalent unconstrained form $\min_w \frac{1}{2}\Vert w\Vert^2 + C\sum_i \max(0, 1 - y_i f(x_i))$, where the hinge loss $\max(0, 1 - y f(x))$ is a convex upper bound on the $0\text{-}1$ loss and has zero gradient once $y f(x) \ge 1$. The hyperparameter $C$ trades margin width against misclassification penalty: $C \to \infty$ recovers the hard margin (zero tolerance, overfits), $C \to 0$ ignores mistakes (maximum margin, underfits).

interview comparisons of hinge vs log loss, the bias-variance reading of $C$, and how the dual constraint changes from $\alpha_i \ge 0$ to $0 \le \alpha_i \le C$.

a single outlier can completely redraw a hard-margin boundary; the hinge's linear penalty makes the boundary robust — it only penalizes samples inside the margin ($y f(x) < 1$) and gives zero gradient to correct high-confidence ones ($y f(x) \ge 1$), a sparsity the log loss (which penalizes everything continuously) lacks. The dual stays convex QP with only the box constraint $0 \le \alpha_i \le C$, so the kernel trick applies unchanged — support vectors split into on-margin ($0 < \alpha_i < C$) and inside-margin ($\alpha_i = C$) types.

Whiteboard the soft-margin objective: $\min \frac{1}{2}\Vert w\Vert^2 + C\sum_i \xi_i$ s.t. $y_i(w^Tx_i+b) \ge 1 - \xi_i, \xi_i \ge 0$, then eliminate $\xi$ to reach the hinge form.

Properties of hinge $\max(0, 1-yf(x))$: why it is a convex upper bound on $0\text{-}1$ loss, why gradient vanishes for $yf(x) \ge 1$, and how the non-differentiable point is handled.

Bias-variance tradeoff of $C$: behavior as $C \to \infty$ vs $C \to 0$, and how to choose $C$ via cross-validation.

Derive the box constraint $0 \le \alpha_i \le C$ in the soft-margin dual (eliminating the slack multipliers) and the meaning of the two support-vector types.

Hinge vs log loss: why is the SVM solution sparse but not LR's? How do they differ on probability output, outliers and overlapping distributions?