M2-058M2: Classical Machine LearningDimensionality ReductionEasy
⚡Flashcard Drill➔Mastery:
Explain the Exact Mathematical Equivalence Between PCA and Singular Value Decomposition (SVD)
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
For a zero-centered data matrix , the right singular vectors are the principal component loading directions, the singular values relate to eigenvalues via , and provides the projected principal coordinates.
📌 Key Takeaways
- •Covariance Connection: .
- •Eigenvalue Equivalence: The eigenvalues of the covariance matrix are directly proportional to squared singular values: .
- •Principal Components: Projected coordinates are .
📐 Mathematical Derivations
Let be mean-centered (). The compact SVD of is , where has orthonormal columns (), with , and has orthonormal columns (). The sample covariance matrix is: . Because is orthogonal and is diagonal with positive entries, this equation is the unique spectral eigenvalue decomposition of ! The right singular vectors are precisely the principal direction eigenvectors of , and the eigenvalues are . The low-dimensional projected data is .
🏭 Production Trade-offs
Why production libraries (scikit-learn
PCA) strictly use SVD rather than : (1) Numerical Precision: Inverting or decomposing squares the condition number: . If , has condition number , causing severe loss of numerical precision. SVD operates directly on , preserving dynamic range. (2) Dimensionality efficiency: When (e.g. samples with features), computing takes memory and time. SVD computes the Gram matrix in time.⚠️ Common Interview Pitfalls
- ✕Computing SVD without centering features (SVD on raw uncentered does NOT equal PCA).
- ✕Dividing singular values by instead of when calculating unbiased sample variance.
🎯 Interviewer Follow-up Inquiries
- ?How does Randomized SVD (Halko et al., 2011) compute truncated PCA in time for massive matrices?
- ?Why does centering data change the right singular vectors of a matrix?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚