M2-010M2: Classical Machine LearningLogistic Regression & GLMHard
⚡Flashcard Drill➔Mastery:
Explain Complete Separation (Hauck-Donner Effect) in Logistic Regression and Standard Industrial Solutions
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Complete separation occurs when a linear hyperplane perfectly separates the two classes; unregularized MLE estimates diverge to infinity () with infinite variance, resolved via L2 regularization, Firth's penalized likelihood, or Bayesian priors.
📌 Key Takeaways
- •Mechanism: If there exists such that for all and for all , the likelihood can be pushed arbitrarily close to 1 by scaling .
- •Symptom (Hauck-Donner Effect): Wald -statistics paradoxically collapse toward 0 () because estimated standard errors explode faster than coefficients.
- •Primary Mitigations: (1) L2 Regularization (Ridge / weight decay), (2) Firth's Penalized Likelihood ( penalty), and (3) Bayesian weakly informative priors (Student-t / Cauchy).
📐 Mathematical Derivations
Mathematical divergence: Let dataset be linearly separable. The likelihood is . For any separating vector , for all . Consider multiplying by scalar . As , for all positive instances, and for all negative instances. Thus, , and log-likelihood reaches its supremum of 0 only at . Unregularized gradient descent drives weights toward infinity, triggering floating-point overflow. Firth's penalized likelihood adds a Jeffreys prior penalty: . Differentiating gives modified score equations , guaranteeing finite, unique estimates with zero first-order bias.
🏭 Production Trade-offs
Industrial default: In machine learning libraries (scikit-learn
LogisticRegression), L2 regularization is enabled by default (penalty='l2', C=1.0), preventing weights from diverging to infinity. In statistical modeling (R logistf), Firth's correction is preferred because it maintains unbiased inference without arbitrary hyperparameter tuning.⚠️ Common Interview Pitfalls
- ✕Assuming a model with complete separation has 'failed' (in fact, it fits data perfectly; the failure is in the unbounded nature of unregularized MLE).
- ✕Trusting Wald p-values in unregularized logistic regression when separation occurs (the Hauck-Donner effect makes significant variables look insignificant).
🎯 Interviewer Follow-up Inquiries
- ?What is Firth's logistic regression and how does Jeffreys prior eliminate small-sample separation bias?
- ?Why does the Hauck-Donner effect cause Wald test statistics to approach zero when the true effect size is infinitely large?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚