M2-006M2: Classical Machine LearningLogistic Regression & GLMEasy
⚡Flashcard Drill➔Mastery:
Formulate Logistic Regression, Sigmoid Activation, and Binary Cross-Entropy Loss
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Logistic regression models class probability via the Sigmoid function: , and is optimized using negative log-likelihood (Binary Cross-Entropy Loss).
📌 Key Takeaways
- •Hypothesis: .
- •Sigmoid derivative: , enabling elegant gradient cancellation.
- •Loss function: .
- •Gradient: , mathematically identical in form to linear regression.
📐 Mathematical Derivations
Likelihood derivation under Bernoulli assumption: Let and . The likelihood for independent observations is . Taking the negative natural log and dividing by : . To compute the gradient with respect to : By the chain rule, . Note that , and , and . Multiplying together: . The non-linear terms cancel out completely, yielding the clean gradient .
🏭 Production Trade-offs
No closed-form solution exists because is non-linear in . Optimized using: (1) Newton-Raphson / IRLS (Iteratively Reweighted Least Squares): with weight matrix , achieving quadratic convergence for small-to-medium . (2) L-BFGS or SGD: Used in industrial production (e.g. Ad click prediction in FTRL-Proximal) for high-dimensional sparse features.
⚠️ Common Interview Pitfalls
- ✕Using Mean Squared Error (MSE) loss with Sigmoid activation (produces a non-convex loss surface with flat plateaus where gradients vanish when predictions are confidently wrong).
- ✕Confusing probabilities with binary classifications (the 0.5 decision threshold can be tuned for precision/recall tradeoffs).
🎯 Interviewer Follow-up Inquiries
- ?Why does combining Sigmoid with Mean Squared Error (MSE) cause vanishing gradients, whereas Cross-Entropy does not?
- ?How does Iteratively Reweighted Least Squares (IRLS) derive from the Newton-Raphson update in Logistic Regression?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚