M2-059M2: Classical Machine LearningDimensionality ReductionMedium
⚡Flashcard Drill➔Mastery:

Enumerate the Limitations of PCA and Scenarios Where Linear PCA Fails

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-060 →
📐 Mathematical Definition
PCA 无监督; LDA 有监督\text{PCA}\ \text{无监督};\ \text{LDA}\ \text{有监督}
⚡ One-Sentence Core Answer
PCA assumes linear relationships, is sensitive to unscaled magnitudes and extreme outliers, maximizes total variance without considering class labels (unsupervised), and fails to capture non-linear manifold structures.

📌 Key Takeaways

  • •
    1. Linear Constraint: Captures only orthogonal linear subspaces; fails on non-linear manifolds (Swiss roll, concentric spheres; requires Kernel PCA, t-SNE, or UMAP).
  • •
    2. Unsupervised / Sub-optimal for Classification: Maximizes variance, not class separability; directions of maximum variance may contain pure noise while class discrimination resides in low-variance directions (Linear Discriminant Analysis (LDA) is preferred).
  • •
    3. Loss of Interpretability: Principal components are linear combinations of all original features (dense loading vectors), making them difficult to explain to domain experts (Sparse PCA mitigates this).
  • •
    4. Outlier Sensitivity: Squared error loss ∥X−UVT∥F2\|X - U V^T\|_F^2 makes PCA fragile to extreme outliers (Robust PCA mitigates this).

📐 Mathematical Derivations

Counterexample where PCA destroys class separability: Consider two Gaussian classes in 2D: Class 1 centered at (0,10)(0, 10) and Class 2 at (0,−10)(0, -10), both with horizontal variance σx2=100\sigma_x^2 = 100 and vertical variance σy2=1\sigma_y^2 = 1. The total horizontal variance is 100100, while total vertical variance is 1+102=1011 + 10^2 = 101. If the data is slightly stretched horizontally such that σx2=150\sigma_x^2 = 150, PCA will select the horizontal xx-axis as the first principal component because it has maximum variance (150>101150 > 101). Projecting onto this first principal component collapses the vertical separation completely, perfectly blending Class 1 and Class 2 together! Linear Discriminant Analysis (LDA) maximizes wTSBwwTSWw\frac{w^T S_B w}{w^T S_W w}, correctly identifying the vertical axis as the optimal discriminative projection.

🏭 Production Trade-offs

When NOT to use PCA: (1) When feature interpretability is mandatory in credit scoring or clinical trials (use L1 Lasso feature selection instead). (2) Supervised classification tasks where LDA or tree-based feature importance achieves higher accuracy. (3) Highly non-linear manifold data (use UMAP or Autoencoders).
⚠️ Common Interview Pitfalls
  • ✕
    Assuming the top principal component is always the most useful feature for downstream supervised classification.
  • ✕
    Applying PCA to categorical or sparse one-hot encoded variables.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Linear Discriminant Analysis (LDA) solve the supervised dimensionality reduction problem via Fisher's criterion?
  • ?
    How does Sparse PCA enforce L1 penalties on loading vectors to recover interpretable features?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-058: Explain the Exact Mathematical Equivalence Between PCA and Singular Value Decomposition (SVD)📋 Back to BankNext →M2-060: Compare t-SNE and UMAP: Manifold Assumptions, Global vs. Local Structure, and Computational Scaling