M2-057M2: Classical Machine LearningDimensionality ReductionEasy
⚡Flashcard Drill➔Mastery:
Formulate Principal Component Analysis (PCA): Maximum Variance and Minimum Reconstruction Error Perspectives
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
PCA finds an orthogonal linear transformation that projects data onto directions of maximum variance, which is mathematically identical to finding the linear subspace that minimizes mean squared reconstruction error.
📌 Key Takeaways
- •Dual Optimality: Maximizing projected variance is mathematically equivalent to minimizing reconstruction error .
- •Spectral Solution: The principal component loading vectors are the top- eigenvectors of the sample covariance matrix (assuming centered ).
- •Explained Variance Ratio: ; quantifies the fraction of total dataset variance captured by the -th principal component.
📐 Mathematical Derivations
Maximum Variance derivation via Lagrange multipliers: Let centered data matrix be () with sample covariance . We seek unit projection vector () that maximizes the variance of projected points : . Formulate the Lagrangian: . Taking gradient with respect to and setting to 0: . This is the exact definition of an eigenvector equation! Multiplying by : . To maximize variance, must be the largest eigenvalue , and its corresponding eigenvector. Subsequent orthogonal components correspond to subsequent descending eigenvalues.
🏭 Production Trade-offs
Step-by-step implementation protocol: (1) Center data: . (2) (Optional) Standardize to unit variance if features have different units. (3) Compute SVD of centered matrix: . (4) The right singular vectors are the principal component loading directions. (5) Projected low-dimensional coordinates: . Computing via SVD avoids explicitly forming the covariance matrix .
⚠️ Common Interview Pitfalls
- ✕Forgetting to zero-center the data before PCA (PCA without centering projects along the mean vector rather than the axis of maximum variance).
- ✕Applying PCA to categorical or binary variables (PCA assumes continuous Euclidean geometry; Multiple Correspondence Analysis (MCA) must be used).
🎯 Interviewer Follow-up Inquiries
- ?Why is computing PCA via SVD on centered numerically superior to eigendecomposition of ?
- ?How does Kernel PCA enable non-linear dimensionality reduction via the kernel trick?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚