🧠 Deep LearningID: resnet-skip

ResNet Skip Connections

Q: Derive the residual-block gradient $\frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}$ and explain why the identity shortcut mitigates vanishing gradients.

【ResNet Skip Connections 核心考点 1】Derive the residual-block gradient $\frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}$ and explain why the identity shortcut mitigates vanishing gradients.。技术解析：Definition: A residual block computes y = F(x, \{W_i\}) + x — the layer learns the residual F(x) = \mathcal{H}(x) - x instead of the direct map \mathcal{H}(x). In backprop, \frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}: the identity shortcut gives a gradient h... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/resnet-skip

Q: What is the degradation problem? Why does a 56-layer net have higher training error than a 20-layer one without overfitting?

【ResNet Skip Connections 核心考点 2】What is the degradation problem? Why does a 56-layer net have higher training error than a 20-layer one without overfitting?。技术解析：Definition: A residual block computes y = F(x, \{W_i\}) + x — the layer learns the residual F(x) = \mathcal{H}(x) - x instead of the direct map \mathcal{H}(x). In backprop, \frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}: the identity shortcut gives a gradient h... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/resnet-skip

Q: Why must the shortcut be the identity $x$ rather than a learned transform such as 1×1 conv?

【ResNet Skip Connections 核心考点 3】Why must the shortcut be the identity $x$ rather than a learned transform such as 1×1 conv?。技术解析：Definition: A residual block computes y = F(x, \{W_i\}) + x — the layer learns the residual F(x) = \mathcal{H}(x) - x instead of the direct map \mathcal{H}(x). In backprop, \frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}: the identity shortcut gives a gradient h... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/resnet-skip

Q: When F changes channels/resolution, how do you match the shortcut dimensions, and what does each option cost?

【ResNet Skip Connections 核心考点 4】When F changes channels/resolution, how do you match the shortcut dimensions, and what does each option cost?。技术解析：Definition: A residual block computes y = F(x, \{W_i\}) + x — the layer learns the residual F(x) = \mathcal{H}(x) - x instead of the direct map \mathcal{H}(x). In backprop, \frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}: the identity shortcut gives a gradient h... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/resnet-skip

Q: How does ResNet make 152-layer (even 1000-layer) training possible? Answer from both optimization and gradient perspectives.

【ResNet Skip Connections 核心考点 5】How does ResNet make 152-layer (even 1000-layer) training possible? Answer from both optimization and gradient perspectives.。技术解析：Definition: A residual block computes y = F(x, \{W_i\}) + x — the layer learns the residual F(x) = \mathcal{H}(x) - x instead of the direct map \mathcal{H}(x). In backprop, \frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}: the identity shortcut gives a gradient h... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/resnet-skip

ResNet 残差连接

🎯Core Definition

A residual block computes

y = F(x, \{W_i\}) + x

— the layer learns the residual

F(x) = \mathcal{H}(x) - x

instead of the direct map

\mathcal{H}(x)

. In backprop,

\frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}

: the identity shortcut gives a gradient highway along which the gradient can flow through any number of layers via the constant 1 term, mitigating vanishing gradients (equivalent to ensembling many shallow networks). Degradation: a 56-layer plain network has higher training error than a 20-layer one, and not due to overfitting — the identity map is simply hard to learn; residuals rewrite the goal as 'learn F→0', so deep networks are at least no worse than shallow ones, which is why ResNet-152 trains stably.

💡Use Cases

a standard component of nearly all deep CNNs/Transformers (same idea as Pre-Norm); interview staples — what is the degradation problem, why do residuals make thousand-layer networks trainable, why must the shortcut be identity.

⚡Key Problems Solved

optimization degradation and vanishing gradients beyond a depth threshold — with identity shortcuts, accuracy improves monotonically with depth without a hard ceiling; ImageNet top-5 error drops from ~7.3% (VGG) to ~4.6% (ResNet-152), and 152-layer models became the default backbone for vision.

🎯5 High-Frequency Exam Points

Derive the residual-block gradient