M3-013M3: Deep Learning FoundationsActivation FunctionsEasy
⚡Flashcard Drill➔Mastery:

Comparing Sigmoid, Tanh, and ReLU: Mathematical Properties and Trade-offs

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-014 →
📐 Mathematical Definition
σ′(x)≤0.25;ReLU′(x)=1[x>0]\sigma'(x)\le0.25;\qquad \mathrm{ReLU}'(x)=\mathbb 1[x>0]
⚡ One-Sentence Core Answer
Sigmoid and Tanh suffer from gradient saturation and computational cost; ReLU solves vanishing gradients with fast piece-wise linearity but risks dying neurons.

📌 Key Takeaways

  • •
    Sigmoid: σ(x)=1/(1+e−x)\sigma(x) = 1/(1+e^{-x}); saturates at extremes (max derivative 0.25), non-zero centered
  • •
    Tanh: tanh⁡(x)=2σ(2x)−1\tanh(x) = 2\sigma(2x) - 1; zero-centered, but still saturates when ∣x∣≫0|x| \gg 0
  • •
    ReLU: max⁡(0,x)\max(0, x); non-saturating in positive domain, constant unit derivative, vulnerable to Dying ReLU

📐 Mathematical Derivations

Mathematical Comparison:
① Sigmoid: σ(x)=11+e−x\sigma(x) = \frac{1}{1 + e^{-x}}. Derivative σ′(x)=σ(x)(1−σ(x))≤0.25\sigma'(x) = \sigma(x)(1 - \sigma(x)) \le 0.25.
- Gradient Vanishing: In an LL-layer network, backpropagation multiplies derivatives: ∏l=1Lσ′(zl)≤(0.25)L\prod_{l=1}^L \sigma'(z_l) \le (0.25)^L. By layer 5, gradients shrink by factor ≤1/1024\le 1/1024.
- Non-Zero Centered: Since σ(x)>0\sigma(x) > 0 strictly, for layer y=Wxy = W x, all gradients ∂L∂Wjk=δjxk\frac{\partial \mathcal{L}}{\partial W_{jk}} = \delta_j x_k share the sign of δj\delta_j, forcing weights to update in constrained zig-zag trajectories.
② Tanh: tanh⁡(x)=ex−e−xex+e−x∈(−1,1)\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} \in (-1, 1). Derivative tanh⁡′(x)=1−tanh⁡2(x)≤1.0\tanh'(x) = 1 - \tanh^2(x) \le 1.0. Zero-centered, eliminating zig-zag updates, but still vanishes for ∣x∣>3|x| > 3.
③ ReLU: f(x)=max⁡(0,x)f(x) = \max(0, x). Derivative f′(x)=1f'(x) = 1 for x>0x > 0, and 00 for x<0x < 0. Eliminates saturation in positive domain and computes at near-zero CPU/GPU cycle cost.

🏭 Production Trade-offs

Modern usage: Sigmoid is restricted to binary classification output heads and gating mechanisms (LSTM gates, attention gates). Tanh is common in recurrent cell hidden state updates and diffusion score heads. ReLU and its smooth derivatives (GELU, SiLU) dominate feedforward backbones.
⚠️ Common Interview Pitfalls
  • ✕
    Using Sigmoid activations across hidden layers of deep networks (>5>5 layers), causing immediate training stagnation
  • ✕
    Using high learning rates with ReLU, driving a large percentage of neurons into permanently inactive dying states
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does the non-zero centered output of Sigmoid cause zig-zag gradient updates in weight space?
  • ?
    How does the biological justification of sparse action potentials relate to the mathematical behavior of ReLU?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-012: Maximal Update Parametrization (μP): Concepts and Hyperparameter Transfer📋 Back to BankNext →M3-014: The Dying ReLU Problem and How LeakyReLU, ELU, and GELU Mitigate It