M2-057M2: Classical Machine LearningDimensionality ReductionEasy
⚡Flashcard Drill➔Mastery:

Formulate Principal Component Analysis (PCA): Maximum Variance and Minimum Reconstruction Error Perspectives

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-058 →
📐 Mathematical Definition
max⁡w:∥w∥=1Var(Xw),Σ=1nX⊤X\max_{w:\|w\|=1}\mathrm{Var}(Xw),\qquad \Sigma=\frac1n X^\top X
⚡ One-Sentence Core Answer
PCA finds an orthogonal linear transformation that projects data onto directions of maximum variance, which is mathematically identical to finding the linear subspace that minimizes mean squared reconstruction error.

📌 Key Takeaways

  • •
    Dual Optimality: Maximizing projected variance max⁡uuTΣu\max_u u^T \Sigma u is mathematically equivalent to minimizing reconstruction error min⁡u∥X−uuTX∥F2\min_u \|X - u u^T X\|_F^2.
  • •
    Spectral Solution: The principal component loading vectors u1,…,uku_1, \dots, u_k are the top-kk eigenvectors of the sample covariance matrix Σ=1NXTX\Sigma = \frac{1}{N} X^T X (assuming centered XX).
  • •
    Explained Variance Ratio: λi∑j=1dλj\frac{\lambda_i}{\sum_{j=1}^d \lambda_j}; quantifies the fraction of total dataset variance captured by the ii-th principal component.

📐 Mathematical Derivations

Maximum Variance derivation via Lagrange multipliers: Let centered data matrix be X∈RN×dX \in \mathbb{R}^{N \times d} (E[X]=0E[X]=0) with sample covariance Σ=1NXTX\Sigma = \frac{1}{N}X^T X. We seek unit projection vector u1u_1 (∥u1∥2=u1Tu1=1\|u_1\|^2 = u_1^T u_1 = 1) that maximizes the variance of projected points z1=Xu1z_1 = X u_1: Var(z1)=1Nz1Tz1=1Nu1TXTXu1=u1TΣu1\text{Var}(z_1) = \frac{1}{N} z_1^T z_1 = \frac{1}{N} u_1^T X^T X u_1 = u_1^T \Sigma u_1. Formulate the Lagrangian: L(u1,λ1)=u1TΣu1−λ1(u1Tu1−1)\mathcal{L}(u_1, \lambda_1) = u_1^T \Sigma u_1 - \lambda_1(u_1^T u_1 - 1). Taking gradient with respect to u1u_1 and setting to 0: ∇u1L=2Σu1−2λ1u1=0  ⟹  Σu1=λ1u1\nabla_{u_1} \mathcal{L} = 2\Sigma u_1 - 2\lambda_1 u_1 = 0 \implies \Sigma u_1 = \lambda_1 u_1. This is the exact definition of an eigenvector equation! Multiplying by u1Tu_1^T: u1TΣu1=λ1u_1^T \Sigma u_1 = \lambda_1. To maximize variance, λ1\lambda_1 must be the largest eigenvalue λmax⁡(Σ)\lambda_{\max}(\Sigma), and u1u_1 its corresponding eigenvector. Subsequent orthogonal components u2,…,uku_2, \dots, u_k correspond to subsequent descending eigenvalues.

🏭 Production Trade-offs

Step-by-step implementation protocol: (1) Center data: Xcentered=X−μX_{\text{centered}} = X - \mu. (2) (Optional) Standardize to unit variance if features have different units. (3) Compute SVD of centered matrix: Xcentered=UΣVTX_{\text{centered}} = U \Sigma V^T. (4) The right singular vectors V=[v1,…,vk]V = [v_1, \dots, v_k] are the principal component loading directions. (5) Projected low-dimensional coordinates: Z=XcenteredVk=UkΣkZ = X_{\text{centered}} V_k = U_k \Sigma_k. Computing via SVD avoids explicitly forming the d×dd\times d covariance matrix XTXX^T X.
⚠️ Common Interview Pitfalls
  • ✕
    Forgetting to zero-center the data before PCA (PCA without centering projects along the mean vector rather than the axis of maximum variance).
  • ✕
    Applying PCA to categorical or binary variables (PCA assumes continuous Euclidean geometry; Multiple Correspondence Analysis (MCA) must be used).
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is computing PCA via SVD on centered XX numerically superior to eigendecomposition of XTXX^T X?
  • ?
    How does Kernel PCA enable non-linear dimensionality reduction via the kernel trick?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-056: Detail Unsupervised Clustering Evaluation: Internal Metrics vs. External Ground-Truth Metrics📋 Back to BankNext →M2-058: Explain the Exact Mathematical Equivalence Between PCA and Singular Value Decomposition (SVD)