Matrix Decomposition Applications

Three flagship applications. (1) Low-rank approximation (Eckart-Young theorem): truncating $A = U\Sigma V^T$ to the top $k$ singular values gives $A_k = U_k\Sigma_k V_k^T$, the best rank-$\le k$ approximation of $A$, with Frobenius error $\min_{\text{rank}(B) \le k} \Vert A - B \Vert_F = \sqrt{\sum_{i > k} \sigma_i^2}$ and spectral error $\sigma_{k+1}$. (2) PCA = SVD: for a centered data matrix $X$ (columns mean-subtracted), the principal directions are the eigenvectors of $X^TX$, i.e. the right singular vectors $V$ of $X$; scores are $U\Sigma$ and the variance along principal component $i$ is $\sigma_i^2 / (n-1)$; SVD avoids squaring the condition number by never forming $X^TX$. (3) Pseudoinverse: $A^+ = V\Sigma^+U^T$ ($\Sigma^+$ inverts nonzero singular values) unifies the least-squares solution of overdetermined systems and the minimum-norm solution of underdetermined ones.

compression and dimensionality reduction (PCA, t-SNE initialization); matrix factorization in recommenders; image compression/denoising; low-rank adaptation (LoRA approximates weight updates by $\Delta W = BA$); numerically stable linear-system solving.

high-dimensional storage and compute are expensive — a rank-$k$ approximation stores $k(m+n)$ parameters instead of $mn$, an order-of-magnitude or more reduction when $k \ll \min(m,n)$; Eckart-Young guarantees the discarded tail $\sqrt{\sum_{i>k} \sigma_i^2}$ is both minimal and exactly computable; PCA via SVD is numerically more stable than eigen-decomposing $X^TX$ (avoids $\kappa(X^TX) = \kappa(X)^2$); low-rank methods such as LoRA shrink $d \times d$ updates to $r(d_1 + d_2)$, cutting trainable parameters by orders of magnitude.

State the Eckart-Young theorem: why is $A_k = U_k\Sigma_k V_k^T$ the optimal rank-$k$ approximation? How is $\sqrt{\sum_{i>k}\sigma_i^2}$ derived?

Derive PCA = SVD: why are principal directions right singular vectors of $X$ rather than eigenvectors? Why is the variance of component $i$ equal to $\sigma_i^2/(n-1)$?

By hand: for a 2×2 matrix, compute the best rank-1 approximation $A_1 = \sigma_1 u_1 v_1^T$ and its Frobenius-norm error.

Why does $A^+ = V\Sigma^+U^T$ deliver both least-squares and minimum-norm solutions? Numerical advantages over the normal equations $(X^TX)^{-1}X^T$?

Storage accounting: how many parameters does a rank-$k$ approximation need (vs $mn$)? Why is LoRA a low-rank approximation and by how much does it cut trainable parameters?