PCA & SVD

PCA (principal component analysis) is unsupervised dimensionality reduction: it finds the orthogonal directions of maximum variance and projects high-dimensional samples onto the top $k$ principal components. The first component $v$ solves the constrained problem $\max_{v} \; v^T \Sigma v \quad \text{s.t.} \quad \Vert v \Vert = 1$, where $\Sigma$ is the covariance matrix.The Lagrangian $L = v^T \Sigma v - \lambda (v^T v - 1)$ differentiated w.r.t. $v$ gives $\Sigma v = \lambda v$ — the solution is the top eigenvector of $\Sigma$ with variance $= \lambda$ (the eigenvalue); the $k$-th component is the $k$-th largest eigenvector.Hence PCA = SVD of the centered data matrix: $X = U \Sigma V^T$, the right singular vectors $V$ are the principal directions, and the projection is $T = X V$ taking the first $k$ columns; by the Eckart-Young theorem the top $k$ components also minimize reconstruction error (max-variance and min-reconstruction are equivalent).Choosing $k$: cumulative explained variance ratio $\frac{\sum_{i=1}^{k} \lambda_i}{\sum_{i=1}^{d} \lambda_i} \ge 0.95$ or the scree-plot elbow; features should be z-scored first.

high-dimensional visualization (2D/3D), feature compression to speed up training, decorrelation/whitening, multicollinearity handling, image and genomics analysis; interviews always include the Lagrange derivation and PCA=SVD.

raw features are redundant, collinear, and noisy, and high dimensions bring the curse of dimensionality; PCA captures maximal variance with $k \ll d$ orthogonal directions — in practice 5% of the dimensions often retain 90%+ of the variance, while decorrelating and denoising. The cost: principal components are linear combinations of original features, hurting interpretability, and PCA cannot capture nonlinear structure (kernel PCA or autoencoders needed).

Test Your Knowledge: Practice Questions for "PCA & SVD"

Single choice pitfall questions with instant feedback and mistake tracking.