Logistic Regression

logistic regression models the log-odds as a linear function $\ln\frac{p}{1-p} = w^Tx + b$, which inverts to the Sigmoid $p = \frac{1}{1 + e^{-(w^Tx + b)}} \in [0,1]$. Parameters come from MLE: the per-sample log-likelihood $y\ln p + (1-y)\ln(1-p)$ negated is cross-entropy $L = -\sum_i [y_i \ln p_i + (1-y_i)\ln(1-p_i)]$. Differentiating w.r.t. $z = w^Tx + b$ using $p' = p(1-p)$ gives $\frac{\partial L}{\partial z} = p - y$, hence $\frac{\partial L}{\partial w} = (p - y)x$; there is no closed form, so optimize by gradient descent / Newton (convex, unique global optimum).

the binary-classification baseline (CTR, credit risk, diagnosis); whiteboard staples include deriving Sigmoid from log-odds, deriving $\frac{\partial L}{\partial z} = p - y$, and contrasting LR with SVM / naive Bayes.

fitting raw linear regression to probabilities yields illegal values outside $[0,1]$ and is outlier-sensitive; LR guarantees valid probabilities, and the CE gradient $p - y$ decays only linearly as $z \to \pm\infty$ (never saturates), whereas squared loss $\frac{1}{2}(p-y)^2$ gives vanishing gradients near $p \to y$ — slow to train; adding $\frac{\lambda}{2}\Vert w\Vert^2$ yields regularized LR for high-dimensional collinearity.

Whiteboard: invert $\ln\frac{p}{1-p} = w^Tx + b$ into $p = \frac{1}{1+e^{-w^Tx}}$, then derive $\frac{\partial L}{\partial z} = p - y$ via $p' = p(1-p)$ and the chain rule.

Why does LR use cross-entropy instead of squared loss? Explain the gradient saturation of squared loss as $p \to y$.

Show MLE ≡ cross-entropy: derive the negative log-likelihood $L = -\sum_i [y_i \ln p_i + (1-y_i)\ln(1-p_i)]$.

Compare LR vs SVM: hinge vs log loss, probabilistic output, support-vector dependence, and outlier robustness.

Why can't LR do multiclass directly? Contrast OvR vs Softmax extensions, and how class imbalance shifts the decision threshold.