M2-005M2: Classical Machine LearningLinear RegressionHard
⚡Flashcard Drill➔Mastery:
Derive the Least Squares Solution for Linear Regression and Explain Its Orthogonal Projection Geometry
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
The OLS prediction is the orthogonal projection of target vector onto the column space of ; the residual vector is strictly orthogonal to every feature column: .
📌 Key Takeaways
- •Geometric Projection: The column space is a -dimensional subspace inside ; is the point in closest to under Euclidean distance.
- •Orthogonality Principle: Residual vector must be perpendicular to , which algebraically states .
- •Projection (Hat) Matrix: ; idempotent () and symmetric (), projecting .
📐 Mathematical Derivations
Geometric derivation without calculus: We seek vector that minimizes Euclidean distance . By the Hilbert Projection Theorem in inner product spaces, the minimum distance point is uniquely characterized by the condition that the error vector is orthogonal to every vector in the subspace: for all . Since the columns of form a basis for , this requires for all , which in matrix form is: . The fitted values are .
🏭 Production Trade-offs
Properties of the Hat Matrix : (1) , proving that the effective degrees of freedom of linear regression equals parameter count . (2) Leverage scores measure how far observation is from the feature center; observations with are high-leverage points capable of disproportionately rotating the fitted regression plane.
⚠️ Common Interview Pitfalls
- ✕Confusing high leverage (extreme in feature space ) with high influence / Cook's distance (high leverage combined with large residual error ).
- ✕Assuming the projection matrix changes with target (the hat matrix depends solely on feature matrix ).
🎯 Interviewer Follow-up Inquiries
- ?Why is the projection matrix idempotent () and what is its geometric interpretation?
- ?How does Cook's Distance combine leverage and studentized residuals to identify influential outliers?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚