Naive Bayes

Naive Bayes is a generative classifier built on Bayes' theorem; its core assumption is that features are conditionally independent given the class $C$. By Bayes' theorem

the prior $P(C)$ times the feature-likelihood product yields the posterior, and the class with maximum posterior is predicted — the normalizer $P(x)$ is the same for all classes and cancels under argmax. The parameterization defines three variants: Multinomial NB (count features, multinomial distribution, for word counts in text), Gaussian NB (continuous features fit as $P(x_i \mid C) = \mathcal{N}(x_i; \mu_{C,i}, \sigma_{C,i}^2)$), and Bernoulli NB (binary 0/1 features, Bernoulli distribution). If a feature-class pair never appears in training, the posterior product collapses to zero, hence Laplace smoothing

with $N_y$ the number of class-$y$ samples, $N_{y,x_i}$ the count of feature $x_i$ in class $y$, $|V|$ the vocabulary size, and $\alpha = 1$ giving add-one smoothing.

text classification (spam, sentiment), fast baselines on high-dimensional sparse features, online incremental learning; interviews often ask to derive the conditional-independence posterior and compare NB with logistic regression.

a generative view over high-dimensional discrete features with parameters growing linearly, no iterative training, and streaming updates; the cost is the strong conditional-independence assumption — correlated features bias probability estimates, though classification boundaries usually remain usable and NB stays robust to noisy features.

Write the NB decision rule $P(C \mid x) \propto P(C)\prod_i P(x_i \mid C)$; what is the “naive” assumption and why can the denominator $P(x)$ be dropped?

Why is Laplace smoothing needed? Write $\hat P = \frac{N_{y,x_i}+\alpha}{N_y+\alpha|V|}$ and explain how zero probabilities are avoided.

What feature types are Multinomial, Gaussian, and Bernoulli NB each suited for?

Is NB generative or discriminative? How does it compare with logistic regression under limited data and strongly correlated features?

What happens when the conditional-independence assumption is violated, and why does NB still often deliver usable decisions in practice?