PCA & SVD

PCA (principal component analysis) is unsupervised dimensionality reduction: it finds the orthogonal directions of maximum variance and projects high-dimensional samples onto the top $k$ principal components. The first component $v$ solves the constrained problem $\max_{v} \; v^T \Sigma v \quad \text{s.t.} \quad \Vert v \Vert = 1$, where $\Sigma$ is the covariance matrix.The Lagrangian $L = v^T \Sigma v - \lambda (v^T v - 1)$ differentiated w.r.t. $v$ gives $\Sigma v = \lambda v$ — the solution is the top eigenvector of $\Sigma$ with variance $= \lambda$ (the eigenvalue); the $k$-th component is the $k$-th largest eigenvector.Hence PCA = SVD of the centered data matrix: $X = U \Sigma V^T$, the right singular vectors $V$ are the principal directions, and the projection is $T = X V$ taking the first $k$ columns; by the Eckart-Young theorem the top $k$ components also minimize reconstruction error (max-variance and min-reconstruction are equivalent).Choosing $k$: cumulative explained variance ratio $\frac{\sum_{i=1}^{k} \lambda_i}{\sum_{i=1}^{d} \lambda_i} \ge 0.95$ or the scree-plot elbow; features should be z-scored first.

high-dimensional visualization (2D/3D), feature compression to speed up training, decorrelation/whitening, multicollinearity handling, image and genomics analysis; interviews always include the Lagrange derivation and PCA=SVD.

raw features are redundant, collinear, and noisy, and high dimensions bring the curse of dimensionality; PCA captures maximal variance with $k \ll d$ orthogonal directions — in practice 5% of the dimensions often retain 90%+ of the variance, while decorrelating and denoising. The cost: principal components are linear combinations of original features, hurting interpretability, and PCA cannot capture nonlinear structure (kernel PCA or autoencoders needed).

Write PCA's max-variance problem $\max_v v^T \Sigma v \; \text{s.t.} \; \Vert v \Vert = 1$ and use Lagrange multipliers to prove the solution is the top eigenvector.

Derive PCA = SVD: why are the right singular vectors of the centered data the principal directions? Relationship between singular values and variance?

Why is maximizing variance equivalent to minimizing reconstruction error? What is the reconstruction error expression?

How to choose the number of components? Common thresholds for the cumulative explained variance ratio $\frac{\sum_{i=1}^{k} \lambda_i}{\sum_{i=1}^{d} \lambda_i}$?

Why must features be standardized before PCA? How does PCA differ from linear/ridge regression and from kernel PCA / autoencoders?