M2-008M2: Classical Machine LearningLogistic Regression & GLMMedium
⚡Flashcard Drill➔Mastery:

Define Generalized Linear Models (GLMs) and Detail Their Three Foundational Components

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-009 →
📐 Mathematical Definition
E[Y]=μ=g−1(Xw)\mathbb E[Y]=\mu=g^{-1}(Xw)
⚡ One-Sentence Core Answer
A Generalized Linear Model (GLM) extends linear regression to non-Gaussian distributions within the exponential family; composed of (1) Random Component (exponential family distribution), (2) Systematic Component (linear predictor η=Xβ\eta = X\beta), and (3) Link Function (g(μ)=ηg(\mu) = \eta).

📌 Key Takeaways

  • •
    1. Random Component: The target variable yy follows an exponential family distribution (Gaussian, Bernoulli, Poisson, Gamma) with mean μ=E[y∣x]\mu = E[y\mid x].
  • •
    2. Systematic Component: Linear predictor η=wTx=∑j=1dwjxj\eta = w^T x = \sum_{j=1}^d w_j x_j.
  • •
    3. Link Function: A monotonic, differentiable function g(⋅)g(\cdot) that connects the conditional mean to the linear predictor: η=g(μ)  ⟺  μ=g−1(η)\eta = g(\mu) \iff \mu = g^{-1}(\eta).

📐 Mathematical Derivations

The canonical link function equates the linear predictor η\eta directly to the natural parameter θ\theta of the exponential family p(y∣θ)=h(y)exp⁡(θy−A(θ))p(y\mid \theta) = h(y)\exp(\theta y - A(\theta)). Since μ=E[y]=A′(θ)\mu = E[y] = A'(\theta), the canonical link is g(μ)=(A′)−1(μ)g(\mu) = (A')^{-1}(\mu). Derivations of classical models: (1) Linear Regression: Normal distribution with A(θ)=θ22  ⟹  A′(θ)=θ=μA(\theta) = \frac{\theta^2}{2} \implies A'(\theta) = \theta = \mu. The canonical link is identity: g(μ)=μ=wTxg(\mu) = \mu = w^T x. (2) Logistic Regression: Bernoulli distribution with A(θ)=log⁡(1+eθ)  ⟹  A′(θ)=eθ1+eθ=μA(\theta) = \log(1 + e^\theta) \implies A'(\theta) = \frac{e^\theta}{1+e^\theta} = \mu. The canonical link is logit: g(μ)=log⁡(μ1−μ)=wTxg(\mu) = \log\left(\frac{\mu}{1-\mu}\right) = w^T x. (3) Poisson Regression: Poisson distribution with A(θ)=eθ  ⟹  A′(θ)=eθ=μA(\theta) = e^\theta \implies A'(\theta) = e^\theta = \mu. The canonical link is log: g(μ)=log⁡(μ)=wTx  ⟹  μ=ewTxg(\mu) = \log(\mu) = w^T x \implies \mu = e^{w^T x}.

🏭 Production Trade-offs

GLMs eliminate ad-hoc non-linear transformations: (1) If modeling positive count data (e.g. Taxi trips or customer purchases), fitting OLS to log⁡(y)\log(y) fails when y=0y=0. Poisson or Negative Binomial regression directly models counts with exact discrete likelihoods. (2) Canonical links guarantee that the negative log-likelihood is strictly convex and that the Hessian equals the Fisher Information Matrix, ensuring fast, stable IRLS convergence.
⚠️ Common Interview Pitfalls
  • ✕
    Using Poisson regression when the variance vastly exceeds the mean (overdispersion; must use Negative Binomial or Quasi-Poisson GLM).
  • ✕
    Confusing the link function g(μ)g(\mu) with transforming the raw response variable g(y)g(y) (GLM models g(E[y])g(E[y]), not E[g(y)]E[g(y)]; by Jensen's inequality, g(E[y])≠E[g(y)]g(E[y]) \ne E[g(y)]).
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Jensen's inequality explain why log⁡(E[y])\log(E[y]) in Poisson regression is fundamentally different from fitting E[log⁡(y)]E[\log(y)] in OLS?
  • ?
    What is Tweedie regression and why is it the gold standard in insurance claims and ad-spend LTV modeling with mass at zero?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-007: Explain Odds, Log-Odds (Logit), and the Exact Multiplicative Interpretation of Logistic Regression Coefficients📋 Back to BankNext →M2-009: Explain the Structural Relationship Between Logistic Regression and a Single-Layer Neural Network