M2-007M2: Classical Machine LearningLogistic Regression & GLMEasy
⚡Flashcard Drill➔Mastery:

Explain Odds, Log-Odds (Logit), and the Exact Multiplicative Interpretation of Logistic Regression Coefficients

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-008 →
📐 Mathematical Definition
log⁡p1−p=w⊤x,OR=eβj\log\frac{p}{1-p}=w^\top x,\qquad \text{OR}=e^{\beta_j}
⚡ One-Sentence Core Answer
Odds is the ratio of success to failure probability p1−p\frac{p}{1-p}; the log-odds (logit) is linear in features: log⁡(p1−p)=wTx\log\left(\frac{p}{1-p}\right) = w^T x; exponentiating a coefficient ewje^{w_j} represents the Odds Ratio (multiplicative factor shift in odds per unit increase in xjx_j).

📌 Key Takeaways

  • •
    Odds: Odds=p1−p∈[0,∞)\text{Odds} = \frac{p}{1 - p} \in [0, \infty); if probability is 80%80\%, odds are 0.80.2=4\frac{0.8}{0.2} = 4 ('4 to 1').
  • •
    Logit Function: logit(p)=log⁡(p1−p)=w0+w1x1+⋯+wdxd∈(−∞,+∞)\text{logit}(p) = \log\left(\frac{p}{1-p}\right) = w_0 + w_1 x_1 + \dots + w_d x_d \in (-\infty, +\infty).
  • •
    Odds Ratio (OR): For a 1-unit increase in xjx_j, odds are multiplied by ewje^{w_j}: Odds(xj+1)=Odds(xj)⋅ewj\text{Odds}(x_j + 1) = \text{Odds}(x_j) \cdot e^{w_j}.
  • •
    Sign interpretation: wj>0  ⟹  ewj>1w_j > 0 \implies e^{w_j} > 1 (increases odds); wj<0  ⟹  ewj<1w_j < 0 \implies e^{w_j} < 1 (decreases odds).

📐 Mathematical Derivations

Inverting the Sigmoid function: p=11+e−wTx  ⟹  1+e−wTx=1p  ⟹  e−wTx=1−pp  ⟹  ewTx=p1−pp = \frac{1}{1 + e^{-w^T x}} \implies 1 + e^{-w^T x} = \frac{1}{p} \implies e^{-w^T x} = \frac{1 - p}{p} \implies e^{w^T x} = \frac{p}{1 - p}. Taking the natural log on both sides: log⁡(p1−p)=w0+∑i=1dwixi\log\left(\frac{p}{1 - p}\right) = w_0 + \sum_{i=1}^d w_i x_i. Consider increasing feature x1x_1 by 1 unit while holding all other features constant: logit(x1+1)−logit(x1)=w1\text{logit}(x_1 + 1) - \text{logit}(x_1) = w_1. Exponentiating both sides: Odds(x1+1)Odds(x1)=ew0+w1(x1+1)+…ew0+w1x1+…=ew1\frac{\text{Odds}(x_1 + 1)}{\text{Odds}(x_1)} = \frac{e^{w_0 + w_1(x_1+1) + \dots}}{e^{w_0 + w_1 x_1 + \dots}} = e^{w_1}. For example, if w1=0.693w_1 = 0.693, then ew1=e0.693=2.0e^{w_1} = e^{0.693} = 2.0, meaning each unit increase in x1x_1 doubles the odds of the positive outcome.

🏭 Production Trade-offs

Interpretability in regulated industries (Finance, Credit Scoring, Healthcare): Logistic regression is often legally mandated over deep neural networks because credit scoring models must issue Adverse Action notices explaining exactly why a loan was rejected. A coefficient wjw_j directly translates to a transparent, auditable score delta via scorecard point scaling: Score=Offset−Factor⋅logit(p)\text{Score} = \text{Offset} - \text{Factor} \cdot \text{logit}(p).
⚠️ Common Interview Pitfalls
  • ✕
    Confusing Odds Ratio (OR) with Relative Risk (RR = p1p2\frac{p_1}{p_2}); OR approximates RR only when the baseline event probability is very small (p<0.05p < 0.05).
  • ✕
    Stating that 'a 1-unit increase in xx increases probability by wjw_j' (probability changes non-linearly depending on the current baseline probability).
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Odds Ratio diverge sharply from Relative Risk when baseline outcome probability is high?
  • ?
    How is marginal effect at the mean (MEM) computed to express logistic regression impact in percentage probability terms?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-006: Formulate Logistic Regression, Sigmoid Activation, and Binary Cross-Entropy Loss📋 Back to BankNext →M2-008: Define Generalized Linear Models (GLMs) and Detail Their Three Foundational Components