M2-092M2: Classical Machine LearningLinear RegressionMedium
⚡Flashcard Drill➔Mastery:

Weighted Least Squares (WLS) and Handling Heteroscedasticity

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-093 →
📐 Mathematical Definition
min⁡w∑iωi (yi−xi⊤w)2,w^WLS=(X⊤WX)−1X⊤Wy\min_w\sum_i \omega_i\,(y_i-x_i^\top w)^2,\qquad \hat w_{WLS}=(X^\top WX)^{-1}X^\top Wy
⚡ One-Sentence Core Answer
WLS weights each residual inversely by the error variance (wi=1/σi2w_i = 1/\sigma_i^2), transforming heteroscedastic data into homoscedastic space to restore BLUE optimality.

📌 Key Takeaways

  • •
    Heteroscedasticity problem: OLS estimates remain unbiased, but standard errors are biased and efficiency is lost
  • •
    WLS solution: solves min⁡β(y−Xβ)TW(y−Xβ)\min_\beta (y - X\beta)^T W (y - X\beta) with weight matrix W=diag(1/σi2)W = \text{diag}(1/\sigma_i^2)
  • •
    Feasible GLS (FGLS): estimates variance function σ^i2=exp⁡(ziTγ)\hat{\sigma}_i^2 = \exp(z_i^T \gamma) from OLS residuals when variance is unknown

📐 Mathematical Derivations

Mathematical formulation: In ordinary regression y=Xβ+ϵy = X\beta + \epsilon, homoscedasticity assumes Cov(ϵ)=σ2I\text{Cov}(\epsilon) = \sigma^2 I. When heteroscedasticity is present, Cov(ϵ)=Ω=diag(σ12,…,σn2)\text{Cov}(\epsilon) = \Omega = \text{diag}(\sigma_1^2, \dots, \sigma_n^2).
Under Gauss-Markov, OLS is no longer the Best Linear Unbiased Estimator (BLUE). Multiply the system by Ω−1/2\Omega^{-1/2}: Ω−1/2y=Ω−1/2Xβ+Ω−1/2ϵ\Omega^{-1/2} y = \Omega^{-1/2} X\beta + \Omega^{-1/2}\epsilon. The transformed disturbance has covariance Cov(Ω−1/2ϵ)=Ω−1/2ΩΩ−1/2=I\text{Cov}(\Omega^{-1/2}\epsilon) = \Omega^{-1/2}\Omega\Omega^{-1/2} = I.
Applying OLS to the transformed system yields the WLS Estimator: β^WLS=(XTWX)−1XTWy\hat{\beta}_{\text{WLS}} = (X^T W X)^{-1} X^T W y, where W=Ω−1=diag(1/σ12,…,1/σn2)W = \Omega^{-1} = \text{diag}(1/\sigma_1^2, \dots, 1/\sigma_n^2).

🏭 Production Trade-offs

Practical application: If variance σi2\sigma_i^2 is unknown, estimate it via auxiliary regression of squared residuals log⁡(ei2)\log(e_i^2) on covariates (Feasible Generalized Least Squares, FGLS). Alternatively, keep OLS point estimates and use Huber-White robust standard errors (sandwich estimator) to obtain valid confidence intervals without modifying coefficients.
⚠️ Common Interview Pitfalls
  • ✕
    Believing heteroscedasticity biases OLS point estimates β^\hat{\beta}; it only destroys statistical efficiency and invalidates standard errors
  • ✕
    Mis-specifying the variance weighting model in FGLS, which can yield worse estimators than unweighted OLS
🎯 Interviewer Follow-up Inquiries
  • ?
    How does the Huber-White sandwich estimator correct standard errors without requiring WLS?
  • ?
    What statistical tests can detect heteroscedasticity (e.g., Breusch-Pagan, White test)?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-091: Evaluating Feature Importance in Neural Networks vs Tree Models📋 Back to BankNext →M2-093: Interaction and Polynomial Terms in Regression: Formulations and Regularization