M2-045M2: Classical Machine LearningNaive BayesHard
⚡Flashcard Drill➔Mastery:

Contrast Naive Bayes and Logistic Regression as a Generative-Discriminative Model Pair (Ng & Jordan, 2001)

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-046 →
📐 Mathematical Definition
NB: P(x,y);LR: P(y∣x)\text{NB}:\ P(x,y);\quad \text{LR}:\ P(y\mid x)
⚡ One-Sentence Core Answer
Naive Bayes (generative) models joint distribution P(X,Y)P(X, Y) and converges faster (O(log⁡d)O(\log d) data); Logistic Regression (discriminative) directly models conditional P(Y∣X)P(Y\mid X) and reaches a strictly lower asymptotic error ceiling (O(d)O(d) data).

📌 Key Takeaways

  • •
    Generative vs Discriminative: Naive Bayes models P(X,Y)=P(Y)P(X∣Y)P(X, Y) = P(Y)P(X\mid Y); Logistic Regression directly models P(Y∣X)=σ(wTx)P(Y\mid X) = \sigma(w^T x).
  • •
    Identical Functional Form: Under binary features, Naive Bayes posterior P(Y=1∣X)P(Y=1\mid X) has the EXACT mathematical functional form of Logistic Regression: 11+e−wTx\frac{1}{1 + e^{-w^T x}}.
  • •
    Ng & Jordan (2001) Findings: Naive Bayes reaches its asymptotic error rate with only O(log⁡d)O(\log d) samples, beating Logistic Regression on tiny datasets; Logistic Regression converges with O(d)O(d) samples to a lower asymptotic error floor.

📐 Mathematical Derivations

Proof of identical parametric form: Consider binary features xj∈{0,1}x_j \in \{0, 1\}. In Naive Bayes: P(Y=1∣X)=P(Y=1)∏pj1xj(1−pj1)1−xjP(Y=1)∏pj1xj(1−pj1)1−xj+P(Y=0)∏pj0xj(1−pj0)1−xj=11+exp⁡(−z)P(Y=1\mid X) = \frac{P(Y=1) \prod p_{j1}^{x_j} (1-p_{j1})^{1-x_j}}{P(Y=1)\prod p_{j1}^{x_j} (1-p_{j1})^{1-x_j} + P(Y=0)\prod p_{j0}^{x_j} (1-p_{j0})^{1-x_j}} = \frac{1}{1 + \exp(-z)}, where log-odds z=log⁡P(Y=1∣X)P(Y=0∣X)=log⁡P(Y=1)P(Y=0)+∑j=1d[xjlog⁡pj1pj0+(1−xj)log⁡1−pj11−pj0]=w0+∑j=1dwjxjz = \log\frac{P(Y=1\mid X)}{P(Y=0\mid X)} = \log\frac{P(Y=1)}{P(Y=0)} + \sum_{j=1}^d \left[ x_j \log\frac{p_{j1}}{p_{j0}} + (1-x_j)\log\frac{1-p_{j1}}{1-p_{j0}} \right] = w_0 + \sum_{j=1}^d w_j x_j. This is identical to the linear predictor in Logistic Regression! The only difference is parameter estimation: Naive Bayes estimates ww by maximizing joint likelihood P(X,Y)P(X, Y) (generative), while Logistic Regression optimizes conditional likelihood P(Y∣X)P(Y\mid X) directly (discriminative).

🏭 Production Trade-offs

When each dominates: (1) Naive Bayes wins: Extreme small-sample regimes (N<500N < 500), streaming online updates with microsecond latency requirements, missing feature resilience. (2) Logistic Regression wins: Correlated features (e.g. In ad auctions with overlapping user tags), large training datasets, and scenarios requiring calibrated probability outputs.
⚠️ Common Interview Pitfalls
  • ✕
    Believing Naive Bayes and Logistic Regression learn different decision boundary geometries (both produce linear hyperplanes).
  • ✕
    Using Naive Bayes when features have strong multicollinearity without feature selection.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Logistic Regression achieve lower asymptotic error than Naive Bayes when the conditional independence assumption is false?
  • ?
    How does generative-discriminative pair duality generalize to HMMs vs Linear-Chain CRFs?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-044: Explain Why Naive Bayes Operates in the Log-Probability Domain to Prevent Underflow📋 Back to BankNext →M2-046: Compare XGBoost and LightGBM: Histogram Binning, GOSS, EFB, and Tree Growth Strategies