M1-030M1: Mathematics & Statistics FundamentalsConvex Optimization & KKTHard
⚡Flashcard Drill➔Mastery:
Explain Subgradients and Proximal Operators, and How Proximal Gradient Descent Optimizes Non-Smooth L1 Penalties
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
A subgradient generalizes derivatives for non-smooth convex functions via supporting hyperplanes, while proximal operators solve composite optimization via exact closed-form soft-thresholding.
📌 Key Takeaways
- •Subgradient: Vector satisfying for all ; for at 0, subdifferential is .
- •Proximal operator: .
- •Proximal Gradient Method (ISTA): Splits into smooth gradient step on followed by exact prox step on non-smooth .
📐 Mathematical Derivations
For composite objective where is smooth and is non-smooth: Standard subgradient descent converges slowly at rate because subgradients do not decrease near the minimum. Proximal Gradient Descent applies the update: . The proximal operator for L1 norm decomposes elementwise: . Setting subgradient to zero yields the soft-thresholding operator: . This achieves the fast convergence rate of smooth gradient descent while enforcing exact sparsity.
🏭 Production Trade-offs
Proximal algorithms (ISTA and accelerated FISTA with Nesterov momentum achieving ) are the industrial gold standard for sparse linear models, compressed sensing, and low-rank matrix recovery. In modern deep learning frameworks, PyTorch native optimizers handle non-smooth penalties via Proximal steps or subgradient sign heuristics (e.g. In AdamW, weight decay is separated from gradient updates).
⚠️ Common Interview Pitfalls
- ✕Confusing soft-thresholding (L1 prox) with hard-thresholding (L0 prox, which discards small values without shifting large ones).
- ✕Applying naive SGD with momentum to L1 loss, which destroys weight sparsity due to momentum accumulation.
🎯 Interviewer Follow-up Inquiries
- ?How does FISTA achieve optimal convergence rate for composite convex optimization?
- ?How does the Alternating Direction Method of Multipliers (ADMM) generalize proximal operators to split constraints?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚