E4Medium★ Core Essential · Industrial BedrockPart E · Loss Functions Handbook
InfoNCE Contrastive Loss
Contrastive mutual-information lower-bound loss maximizing cosine similarity along positive batch diagonals.
⏱️ Time:
O(B^2 * D)💾 Space:
O(B^2) 相似度矩阵显存🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def info_nce_loss(
q: np.ndarray, # (B, D) Query 特征向量
k: np.ndarray, # (B, D) Key 特征向量 (第 i 个 q 与第 i 个 k 互为正样本对)
tau: float = 0.07
) -> float:
# 1. L2 范数归一化 (余弦相似度前提)
q_norm =
k_norm = k / np.linalg.norm(k, axis=-1, keepdims=True)
# 2. 计算相似度矩阵: (B, D) @ (D, B) -> (B, B)
similarity = (q_norm @ k_norm.T) / tau
# 3. 构造对角线正样本目标: targets = [0, 1, ..., B-1]
B = q.shape[0]
targets = np.arange(B)
# 4. 数值稳定交叉熵计算
sim_max = np.max(similarity, axis=-1, keepdims=True)
exp_sim = np.exp(similarity - sim_max)
log_sum = sim_max.squeeze(-1) + np.log(np.sum(exp_sim, axis=-1))
# 对角线正样本点积分数
pos_sim = np.diag(similarity)
loss = log_sum - pos_sim
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How do contrastive learning pipelines overcome batch-size limits when GPU memory cannot hold large negative batches?
💡
Core Mental Anchor / Mnemonic
L2-normalize, divide similarities by temperature tau, and run cross-entropy against diagonal indices.
📐 Mathematical Derivation & Core Formula
### Mutual Information Lower Bound & Temperature Scaling
InfoNCE optimizes a lower bound on mutual information between positive views contrasted against negatives by treating pair matching as -way classification over cosine similarities divided by temperature .
Small (e.g., ) sharpens the Softmax distribution and acts as implicit hard-negative mining, heavily penalizing high-similarity negatives.
InfoNCE optimizes a lower bound on mutual information between positive views contrasted against negatives by treating pair matching as -way classification over cosine similarities divided by temperature .
Small (e.g., ) sharpens the Softmax distribution and acts as implicit hard-negative mining, heavily penalizing high-similarity negatives.
🔄 Tensor Dimensions & Shape Flow
q, k: (B, D) -> L2 normalize -> (B, B) similarity / tau -> diagonal targets -> CrossEntropy -> scalar
🛡️ Numerical Stability & Pitfalls Checklist
- L2-normalize embeddings before computing similarities so dot products remain bounded in [-1, 1].
- Positive pairs sit on matrix diagonal [i, i]; off-diagonal entries act as in-batch negatives.
- Larger negative sets / batch sizes tighten the InfoNCE mutual information lower bound.
💻 Reference Code Implementation
import numpy as np
def info_nce_loss(
q: np.ndarray, # (B, D) Query 特征向量
k: np.ndarray, # (B, D) Key 特征向量 (第 i 个 q 与第 i 个 k 互为正样本对)
tau: float = 0.07
) -> float:
# 1. L2 范数归一化 (余弦相似度前提)
q_norm = q / np.linalg.norm(q, axis=-1, keepdims=True)
k_norm = k / np.linalg.norm(k, axis=-1, keepdims=True)
# 2. 计算相似度矩阵: (B, D) @ (D, B) -> (B, B)
similarity = (q_norm @ k_norm.T) / tau
# 3. 构造对角线正样本目标: targets = [0, 1, ..., B-1]
B = q.shape[0]
targets = np.arange(B)
# 4. 数值稳定交叉熵计算
sim_max = np.max(similarity, axis=-1, keepdims=True)
exp_sim = np.exp(similarity - sim_max)
log_sum = sim_max.squeeze(-1) + np.log(np.sum(exp_sim, axis=-1))
# 对角线正样本点积分数
pos_sim = np.diag(similarity)
loss = log_sum - pos_sim
return float(np.mean(loss))
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
B, D = 4, 16
q = np.random.randn(B, D)
k = q + np.random.randn(B, D) * 0.01 # 强正相关
loss = info_nce_loss(q, k, tau=0.07)
# 正样本几乎一致时,损失应极低
assert loss < 0.1, f"强正样本损失应很小: {loss}"
print("✓ InfoNCE 对比损失自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How do contrastive learning pipelines overcome batch-size limits when GPU memory cannot hold large negative batches?
Two classic patterns: (1) MoCo maintains a cross-batch FIFO queue of momentum-encoded negative keys decoupled from batch size; (2) distributed DDP setups use
all_gather across GPUs to assemble a global similarity matrix.⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.