M2-059M2: Classical Machine LearningDimensionality ReductionMedium
⚡Flashcard Drill➔Mastery:
Enumerate the Limitations of PCA and Scenarios Where Linear PCA Fails
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
PCA assumes linear relationships, is sensitive to unscaled magnitudes and extreme outliers, maximizes total variance without considering class labels (unsupervised), and fails to capture non-linear manifold structures.
📌 Key Takeaways
- •1. Linear Constraint: Captures only orthogonal linear subspaces; fails on non-linear manifolds (Swiss roll, concentric spheres; requires Kernel PCA, t-SNE, or UMAP).
- •2. Unsupervised / Sub-optimal for Classification: Maximizes variance, not class separability; directions of maximum variance may contain pure noise while class discrimination resides in low-variance directions (Linear Discriminant Analysis (LDA) is preferred).
- •3. Loss of Interpretability: Principal components are linear combinations of all original features (dense loading vectors), making them difficult to explain to domain experts (Sparse PCA mitigates this).
- •4. Outlier Sensitivity: Squared error loss makes PCA fragile to extreme outliers (Robust PCA mitigates this).
📐 Mathematical Derivations
Counterexample where PCA destroys class separability: Consider two Gaussian classes in 2D: Class 1 centered at and Class 2 at , both with horizontal variance and vertical variance . The total horizontal variance is , while total vertical variance is . If the data is slightly stretched horizontally such that , PCA will select the horizontal -axis as the first principal component because it has maximum variance (). Projecting onto this first principal component collapses the vertical separation completely, perfectly blending Class 1 and Class 2 together! Linear Discriminant Analysis (LDA) maximizes , correctly identifying the vertical axis as the optimal discriminative projection.
🏭 Production Trade-offs
When NOT to use PCA: (1) When feature interpretability is mandatory in credit scoring or clinical trials (use L1 Lasso feature selection instead). (2) Supervised classification tasks where LDA or tree-based feature importance achieves higher accuracy. (3) Highly non-linear manifold data (use UMAP or Autoencoders).
⚠️ Common Interview Pitfalls
- ✕Assuming the top principal component is always the most useful feature for downstream supervised classification.
- ✕Applying PCA to categorical or sparse one-hot encoded variables.
🎯 Interviewer Follow-up Inquiries
- ?How does Linear Discriminant Analysis (LDA) solve the supervised dimensionality reduction problem via Fisher's criterion?
- ?How does Sparse PCA enforce L1 penalties on loading vectors to recover interpretable features?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚