E4Medium★ Core Essential · Industrial BedrockPart E · Loss Functions Handbook

InfoNCE Contrastive Loss

⚡Flashcard Drill➔

Contrastive mutual-information lower-bound loss maximizing cosine similarity along positive batch diagonals.

⏱️ Time: O(B^2 * D)
💾 Space: O(B^2) 相似度矩阵显存
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def info_nce_loss(
q: np.ndarray, # (B, D) Query 特征向量
k: np.ndarray, # (B, D) Key 特征向量 (第 i 个 q 与第 i 个 k 互为正样本对)
tau: float = 0.07
) -> float:
# 1. L2 范数归一化 (余弦相似度前提)
q_norm =
k_norm = k / np.linalg.norm(k, axis=-1, keepdims=True)
# 2. 计算相似度矩阵: (B, D) @ (D, B) -> (B, B)
similarity = (q_norm @ k_norm.T) / tau
# 3. 构造对角线正样本目标: targets = [0, 1, ..., B-1]
B = q.shape[0]
targets = np.arange(B)
# 4. 数值稳定交叉熵计算
sim_max = np.max(similarity, axis=-1, keepdims=True)
exp_sim = np.exp(similarity - sim_max)
log_sum = sim_max.squeeze(-1) + np.log(np.sum(exp_sim, axis=-1))
# 对角线正样本点积分数
pos_sim = np.diag(similarity)
loss = log_sum - pos_sim
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How do contrastive learning pipelines overcome batch-size limits when GPU memory cannot hold large negative batches?
Next: CLIP Symmetric Contrastive Loss →
💡

Core Mental Anchor / Mnemonic

L2-normalize, divide similarities by temperature tau, and run cross-entropy against diagonal indices.

📐 Mathematical Derivation & Core Formula

LInfoNCE=−log⁡esim(q,k+)/τesim(q,k+)/τ+∑iesim(q,ki−)/τ\mathcal{L}_{\text{InfoNCE}} = -\log \frac{e^{\mathrm{sim}(q, k_+) / \tau}}{e^{\mathrm{sim}(q, k_+) / \tau} + \sum_i e^{\mathrm{sim}(q, k_i^-) / \tau}}
### Mutual Information Lower Bound & Temperature Scaling
InfoNCE optimizes a lower bound on mutual information I(X;Y)I(X; Y) between positive views (q,k+)(q, k_+) contrasted against negatives {ki−}\{k_i^-\} by treating pair matching as (K+1)(K+1)-way classification over cosine similarities divided by temperature τ\tau.
Small τ\tau (e.g., 0.070.07) sharpens the Softmax distribution and acts as implicit hard-negative mining, heavily penalizing high-similarity negatives.

🔄 Tensor Dimensions & Shape Flow

q, k: (B, D) -> L2 normalize -> (B, B) similarity / tau -> diagonal targets -> CrossEntropy -> scalar

🛡️ Numerical Stability & Pitfalls Checklist

  • L2-normalize embeddings before computing similarities so dot products remain bounded in [-1, 1].
  • Positive pairs sit on matrix diagonal [i, i]; off-diagonal entries act as in-batch negatives.
  • Larger negative sets / batch sizes tighten the InfoNCE mutual information lower bound.

💻 Reference Code Implementation

import numpy as np

def info_nce_loss(
    q: np.ndarray,      # (B, D) Query 特征向量
    k: np.ndarray,      # (B, D) Key 特征向量 (第 i 个 q 与第 i 个 k 互为正样本对)
    tau: float = 0.07
) -> float:
    # 1. L2 范数归一化 (余弦相似度前提)
    q_norm = q / np.linalg.norm(q, axis=-1, keepdims=True)
    k_norm = k / np.linalg.norm(k, axis=-1, keepdims=True)
    
    # 2. 计算相似度矩阵: (B, D) @ (D, B) -> (B, B)
    similarity = (q_norm @ k_norm.T) / tau
    
    # 3. 构造对角线正样本目标: targets = [0, 1, ..., B-1]
    B = q.shape[0]
    targets = np.arange(B)
    
    # 4. 数值稳定交叉熵计算
    sim_max = np.max(similarity, axis=-1, keepdims=True)
    exp_sim = np.exp(similarity - sim_max)
    log_sum = sim_max.squeeze(-1) + np.log(np.sum(exp_sim, axis=-1))
    
    # 对角线正样本点积分数
    pos_sim = np.diag(similarity)
    
    loss = log_sum - pos_sim
    return float(np.mean(loss))

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
B, D = 4, 16
q = np.random.randn(B, D)
k = q + np.random.randn(B, D) * 0.01 # 强正相关
loss = info_nce_loss(q, k, tau=0.07)
# 正样本几乎一致时,损失应极低
assert loss < 0.1, f"强正样本损失应很小: {loss}"
print("✓ InfoNCE 对比损失自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How do contrastive learning pipelines overcome batch-size limits when GPU memory cannot hold large negative batches?
Two classic patterns: (1) MoCo maintains a cross-batch FIFO queue of momentum-encoded negative keys decoupled from batch size; (2) distributed DDP setups use all_gather across GPUs to assemble a global similarity matrix.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Focal Loss for Dense Object DetectionBack to Part E & All 105Next: CLIP Symmetric Contrastive Loss →