M2-005M2: Classical Machine LearningLinear RegressionHard
⚡Flashcard Drill➔Mastery:

Derive the Least Squares Solution for Linear Regression and Explain Its Orthogonal Projection Geometry

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-006 →
📐 Mathematical Definition
∇w∥y−Xw∥2=0⇒X⊤Xw=X⊤y\nabla_w\|y-Xw\|^2=0\Rightarrow X^\top Xw=X^\top y
⚡ One-Sentence Core Answer
The OLS prediction y^=Xβ^\hat{y} = X\hat{\beta} is the orthogonal projection of target vector yy onto the column space of XX; the residual vector e=y−y^e = y - \hat{y} is strictly orthogonal to every feature column: XTe=0X^T e = 0.

📌 Key Takeaways

  • •
    Geometric Projection: The column space Col(X)\text{Col}(X) is a dd-dimensional subspace inside RN\mathbb{R}^N; y^\hat{y} is the point in Col(X)\text{Col}(X) closest to yy under Euclidean distance.
  • •
    Orthogonality Principle: Residual vector e=y−y^e = y - \hat{y} must be perpendicular to Col(X)\text{Col}(X), which algebraically states XT(y−Xβ^)=0X^T (y - X\hat{\beta}) = 0.
  • •
    Projection (Hat) Matrix: H=X(XTX)−1XTH = X(X^T X)^{-1} X^T; idempotent (H2=HH^2 = H) and symmetric (HT=HH^T = H), projecting y↦y^=Hyy \mapsto \hat{y} = Hy.

📐 Mathematical Derivations

Geometric derivation without calculus: We seek vector y^∈Col(X)\hat{y} \in \text{Col}(X) that minimizes Euclidean distance ∥y−y^∥2\|y - \hat{y}\|_2. By the Hilbert Projection Theorem in inner product spaces, the minimum distance point is uniquely characterized by the condition that the error vector e=y−y^e = y - \hat{y} is orthogonal to every vector in the subspace: ⟨v,y−y^⟩=0\langle v, y - \hat{y} \rangle = 0 for all v∈Col(X)v \in \text{Col}(X). Since the columns of X=[x1,…,xd]X = [x_1, \dots, x_d] form a basis for Col(X)\text{Col}(X), this requires xjT(y−y^)=0x_j^T (y - \hat{y}) = 0 for all j=1,…,dj=1,\dots, d, which in matrix form is: XT(y−Xβ^)=0  ⟹  XTXβ^=XTy  ⟹  β^=(XTX)−1XTyX^T (y - X\hat{\beta}) = 0 \implies X^T X \hat{\beta} = X^T y \implies \hat{\beta} = (X^T X)^{-1} X^T y. The fitted values are y^=Xβ^=X(XTX)−1XTy=Hy\hat{y} = X\hat{\beta} = X(X^T X)^{-1} X^T y = H y.

🏭 Production Trade-offs

Properties of the Hat Matrix HH: (1) Tr(H)=Tr(X(XTX)−1XT)=Tr((XTX)−1XTX)=Tr(Id)=d\text{Tr}(H) = \text{Tr}(X(X^T X)^{-1} X^T) = \text{Tr}((X^T X)^{-1} X^T X) = \text{Tr}(I_d) = d, proving that the effective degrees of freedom of linear regression equals parameter count dd. (2) Leverage scores hii=Hii∈[0,1]h_{ii} = H_{ii} \in [0, 1] measure how far observation xix_i is from the feature center; observations with hii>2d/Nh_{ii} > 2d/N are high-leverage points capable of disproportionately rotating the fitted regression plane.
⚠️ Common Interview Pitfalls
  • ✕
    Confusing high leverage (extreme in feature space XX) with high influence / Cook's distance (high leverage combined with large residual error eie_i).
  • ✕
    Assuming the projection matrix HH changes with target yy (the hat matrix depends solely on feature matrix XX).
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is the projection matrix HH idempotent (H2=HH^2 = H) and what is its geometric interpretation?
  • ?
    How does Cook's Distance combine leverage hiih_{ii} and studentized residuals to identify influential outliers?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-004: Explain Why Ridge Regression Mitigates Multicollinearity and Derive Its Closed-Form Solution📋 Back to BankNext →M2-006: Formulate Logistic Regression, Sigmoid Activation, and Binary Cross-Entropy Loss