M3-013M3: Deep Learning FoundationsActivation FunctionsEasy
⚡Flashcard Drill➔Mastery:
Comparing Sigmoid, Tanh, and ReLU: Mathematical Properties and Trade-offs
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Sigmoid and Tanh suffer from gradient saturation and computational cost; ReLU solves vanishing gradients with fast piece-wise linearity but risks dying neurons.
📌 Key Takeaways
- •Sigmoid: ; saturates at extremes (max derivative 0.25), non-zero centered
- •Tanh: ; zero-centered, but still saturates when
- •ReLU: ; non-saturating in positive domain, constant unit derivative, vulnerable to Dying ReLU
📐 Mathematical Derivations
Mathematical Comparison:
① Sigmoid: . Derivative .
- Gradient Vanishing: In an -layer network, backpropagation multiplies derivatives: . By layer 5, gradients shrink by factor .
- Non-Zero Centered: Since strictly, for layer , all gradients share the sign of , forcing weights to update in constrained zig-zag trajectories.
② Tanh: . Derivative . Zero-centered, eliminating zig-zag updates, but still vanishes for .
③ ReLU: . Derivative for , and for . Eliminates saturation in positive domain and computes at near-zero CPU/GPU cycle cost.
① Sigmoid: . Derivative .
- Gradient Vanishing: In an -layer network, backpropagation multiplies derivatives: . By layer 5, gradients shrink by factor .
- Non-Zero Centered: Since strictly, for layer , all gradients share the sign of , forcing weights to update in constrained zig-zag trajectories.
② Tanh: . Derivative . Zero-centered, eliminating zig-zag updates, but still vanishes for .
③ ReLU: . Derivative for , and for . Eliminates saturation in positive domain and computes at near-zero CPU/GPU cycle cost.
🏭 Production Trade-offs
Modern usage: Sigmoid is restricted to binary classification output heads and gating mechanisms (LSTM gates, attention gates). Tanh is common in recurrent cell hidden state updates and diffusion score heads. ReLU and its smooth derivatives (GELU, SiLU) dominate feedforward backbones.
⚠️ Common Interview Pitfalls
- ✕Using Sigmoid activations across hidden layers of deep networks ( layers), causing immediate training stagnation
- ✕Using high learning rates with ReLU, driving a large percentage of neurons into permanently inactive dying states
🎯 Interviewer Follow-up Inquiries
- ?Why does the non-zero centered output of Sigmoid cause zig-zag gradient updates in weight space?
- ?How does the biological justification of sparse action potentials relate to the mathematical behavior of ReLU?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚