M2-006M2: Classical Machine LearningLogistic Regression & GLMEasy
⚡Flashcard Drill➔Mastery:

Formulate Logistic Regression, Sigmoid Activation, and Binary Cross-Entropy Loss

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-007 →
📐 Mathematical Definition
p=σ(w⊤x),L=−∑[ylog⁡p+(1−y)log⁡(1−p)]p=\sigma(w^\top x),\qquad \mathcal L=-\sum\big[y\log p+(1-y)\log(1-p)\big]
⚡ One-Sentence Core Answer
Logistic regression models class probability via the Sigmoid function: P(y=1∣x)=σ(wTx)=11+e−wTxP(y=1\mid x) = \sigma(w^T x) = \frac{1}{1 + e^{-w^T x}}, and is optimized using negative log-likelihood (Binary Cross-Entropy Loss).

📌 Key Takeaways

  • •
    Hypothesis: hw(x)=σ(wTx)=11+e−wTx∈(0,1)h_w(x) = \sigma(w^T x) = \frac{1}{1 + e^{-w^T x}} \in (0, 1).
  • •
    Sigmoid derivative: σ′(z)=σ(z)(1−σ(z))\sigma'(z) = \sigma(z)(1 - \sigma(z)), enabling elegant gradient cancellation.
  • •
    Loss function: L(w)=−1N∑i=1N[yilog⁡hw(xi)+(1−yi)log⁡(1−hw(xi))]\mathcal{L}(w) = -\frac{1}{N}\sum_{i=1}^N [y_i \log h_w(x_i) + (1-y_i)\log(1 - h_w(x_i))].
  • •
    Gradient: ∇wL=1N∑i=1N(hw(xi)−yi)xi\nabla_w \mathcal{L} = \frac{1}{N}\sum_{i=1}^N (h_w(x_i) - y_i) x_i, mathematically identical in form to linear regression.

📐 Mathematical Derivations

Likelihood derivation under Bernoulli assumption: Let P(yi=1∣xi)=piP(y_i=1\mid x_i) = p_i and P(yi=0∣xi)=1−piP(y_i=0\mid x_i) = 1 - p_i. The likelihood for NN independent observations is L(w)=∏i=1Npiyi(1−pi)1−yiL(w) = \prod_{i=1}^N p_i^{y_i} (1 - p_i)^{1 - y_i}. Taking the negative natural log and dividing by NN: L(w)=−1N∑i=1N[yilog⁡pi+(1−yi)log⁡(1−pi)]\mathcal{L}(w) = -\frac{1}{N}\sum_{i=1}^N [y_i \log p_i + (1-y_i)\log(1 - p_i)]. To compute the gradient with respect to wjw_j: By the chain rule, ∂L∂wj=∂L∂pi∂pi∂zi∂zi∂wj\frac{\partial \mathcal{L}}{\partial w_j} = \frac{\partial \mathcal{L}}{\partial p_i} \frac{\partial p_i}{\partial z_i} \frac{\partial z_i}{\partial w_j}. Note that ∂Li∂pi=−yipi+1−yi1−pi=pi−yipi(1−pi)\frac{\partial \mathcal{L}_i}{\partial p_i} = -\frac{y_i}{p_i} + \frac{1-y_i}{1-p_i} = \frac{p_i - y_i}{p_i(1-p_i)}, and ∂pi∂zi=σ′(zi)=pi(1−pi)\frac{\partial p_i}{\partial z_i} = \sigma'(z_i) = p_i(1-p_i), and ∂zi∂wj=xij\frac{\partial z_i}{\partial w_j} = x_{ij}. Multiplying together: ∂Li∂wj=pi−yipi(1−pi)⋅pi(1−pi)⋅xij=(pi−yi)xij\frac{\partial \mathcal{L}_i}{\partial w_j} = \frac{p_i - y_i}{p_i(1-p_i)} \cdot p_i(1-p_i) \cdot x_{ij} = (p_i - y_i) x_{ij}. The non-linear terms cancel out completely, yielding the clean gradient ∇wL=1NXT(y^−y)\nabla_w \mathcal{L} = \frac{1}{N} X^T (\hat{y} - y).

🏭 Production Trade-offs

No closed-form solution exists because y^=σ(Xw)\hat{y} = \sigma(Xw) is non-linear in ww. Optimized using: (1) Newton-Raphson / IRLS (Iteratively Reweighted Least Squares): w(t+1)=w(t)−(XTWX)−1XT(p−y)w^{(t+1)} = w^{(t)} - (X^T W X)^{-1} X^T (p - y) with weight matrix Wii=pi(1−pi)W_{ii} = p_i(1-p_i), achieving quadratic convergence for small-to-medium dd. (2) L-BFGS or SGD: Used in industrial production (e.g. Ad click prediction in FTRL-Proximal) for high-dimensional sparse features.
⚠️ Common Interview Pitfalls
  • ✕
    Using Mean Squared Error (MSE) loss with Sigmoid activation (produces a non-convex loss surface with flat plateaus where gradients vanish when predictions are confidently wrong).
  • ✕
    Confusing probabilities p∈[0,1]p \in [0, 1] with binary classifications y^∈{0,1}\hat{y} \in \{0, 1\} (the 0.5 decision threshold can be tuned for precision/recall tradeoffs).
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does combining Sigmoid with Mean Squared Error (MSE) cause vanishing gradients, whereas Cross-Entropy does not?
  • ?
    How does Iteratively Reweighted Least Squares (IRLS) derive from the Newton-Raphson update in Logistic Regression?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-005: Derive the Least Squares Solution for Linear Regression and Explain Its Orthogonal Projection Geometry📋 Back to BankNext →M2-007: Explain Odds, Log-Odds (Logit), and the Exact Multiplicative Interpretation of Logistic Regression Coefficients