E5Easy★ Core Essential · Industrial BedrockPart E · Loss Functions Handbook
CLIP Symmetric Contrastive Loss
Symmetric image-to-text and text-to-image cross-entropy over an N x N similarity matrix.
⏱️ Time:
O(B^2 * D)💾 Space:
O(B^2)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def clip_symmetric_loss(
image_embeds: np.ndarray, # (B, D) 图像特征
text_embeds: np.ndarray, # (B, D) 文本特征
logit_scale: float = 2.659 # ln(1/tau),CLIP 原文初始值约为 2.659 (tau=0.07)
) -> float:
# 1. L2 范数归一化
img_norm =
txt_norm = text_embeds / np.linalg.norm(text_embeds, axis=-1, keepdims=True)
# 2. 计算缩放点积 Logits 矩阵: (B, D) @ (D, B) -> (B, B)
scale = np.exp(np.clip(logit_scale, 0.0, 4.605)) # 上限 100
logits_per_image = (img_norm @ txt_norm.T) * scale
logits_per_text = logits_per_image.T
# 3. 对称交叉熵计算 (正样本全在对角线上)
B = image_embeds.shape[0]
labels = np.arange(B)
def cross_entropy_2d(logits):
l_max = np.max(logits, axis=-1, keepdims=True)
exp_l = np.exp(logits - l_max)
lse = l_max.squeeze(-1) + np.log(np.sum(exp_l, axis=-1))
pos_logits = np.diag(logits)
return np.mean(lse - pos_logits)
loss_i2t = cross_entropy_2d(logits_per_image)
loss_t2i = cross_entropy_2d(logits_per_text)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does SigLIP redesign CLIP's contrastive objective?
💡
Core Mental Anchor / Mnemonic
Optimize row-wise image->text and column-wise text->image cross-entropy over batch diagonal pairs.
📐 Mathematical Derivation & Core Formula
### Symmetric Bidirectional Supervision
Given normalized image-text pairs and the scaled cosine similarity matrix:
1. Image-to-Text (rows): Match each image to its ground-truth caption among candidates;
2. Text-to-Image (columns): Match each caption to its image among images;
The final loss averages both cross-entropies, with the log-temperature parameter clamped (commonly at ) to prevent gradient instability.
Given normalized image-text pairs and the scaled cosine similarity matrix:
1. Image-to-Text (rows): Match each image to its ground-truth caption among candidates;
2. Text-to-Image (columns): Match each caption to its image among images;
The final loss averages both cross-entropies, with the log-temperature parameter clamped (commonly at ) to prevent gradient instability.
🔄 Tensor Dimensions & Shape Flow
img, txt: (B, D) -> normalize -> logits: (B, B) -> row CE & col CE -> 0.5 * (loss_i2t + loss_t2i)
🛡️ Numerical Stability & Pitfalls Checklist
- Clamp learnable logit_scale upper bound (e.g., ln(100) approx 4.605) to avoid Softmax collapse.
- In distributed setups, gather embeddings cleanly across devices while preserving local gradients.
💻 Reference Code Implementation
import numpy as np
def clip_symmetric_loss(
image_embeds: np.ndarray, # (B, D) 图像特征
text_embeds: np.ndarray, # (B, D) 文本特征
logit_scale: float = 2.659 # ln(1/tau),CLIP 原文初始值约为 2.659 (tau=0.07)
) -> float:
# 1. L2 范数归一化
img_norm = image_embeds / np.linalg.norm(image_embeds, axis=-1, keepdims=True)
txt_norm = text_embeds / np.linalg.norm(text_embeds, axis=-1, keepdims=True)
# 2. 计算缩放点积 Logits 矩阵: (B, D) @ (D, B) -> (B, B)
scale = np.exp(np.clip(logit_scale, 0.0, 4.605)) # 上限 100
logits_per_image = (img_norm @ txt_norm.T) * scale
logits_per_text = logits_per_image.T
# 3. 对称交叉熵计算 (正样本全在对角线上)
B = image_embeds.shape[0]
labels = np.arange(B)
def cross_entropy_2d(logits):
l_max = np.max(logits, axis=-1, keepdims=True)
exp_l = np.exp(logits - l_max)
lse = l_max.squeeze(-1) + np.log(np.sum(exp_l, axis=-1))
pos_logits = np.diag(logits)
return np.mean(lse - pos_logits)
loss_i2t = cross_entropy_2d(logits_per_image)
loss_t2i = cross_entropy_2d(logits_per_text)
return float(0.5 * (loss_i2t + loss_t2i))
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
B, D = 4, 32
img = np.random.randn(B, D)
txt = img.copy() # 理想完美匹配
loss = clip_symmetric_loss(img, txt)
assert loss < 0.05, f"完美匹配下损失应接近 0: {loss}"
print("✓ CLIP 双向对称损失自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How does SigLIP redesign CLIP's contrastive objective?
CLIP uses Softmax normalized across all batch entries, requiring global cross-device synchronization of similarity rows/columns. SigLIP replaces Softmax with pairwise binary sigmoid (BCE) loss over each image-text pair, decoupling batch-wide normalization and easing distributed scaling.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.