InfoNCE Contrastive Loss

InfoNCE (Info Noise-Contrastive Estimation) is the core loss of contrastive learning (CLIP/SimCLR/MoCo) — pull positives together and push all in-batch negatives apart: $\mathcal{L} = -\log \frac{\exp(\text{sim}(x_i, y_i) / \tau)}{\sum_{j} \exp(\text{sim}(x_i, y_j) / \tau)}$, where $\text{sim}(\cdot, \cdot)$ is cosine similarity, $y_i$ the matching positive of $x_i$, the denominator sums over every candidate in the batch (1 positive + $N-1$ negatives), and $\tau$ is the temperature hyperparameter that scales similarities before softmax.

image-text alignment (CLIP), self-supervised representation (SimCLR/MoCo), cross-modal retrieval; interviews always probe the role of temperature $\tau$ and why larger batches help.

triplet loss compares against a single negative, giving sparse gradients and dependence on hard-negative mining; InfoNCE folds all in-batch negatives into the softmax denominator, far better negative utilization and gradient signal, and is structurally identical to multi-class cross-entropy (positive logit = $\text{sim}(x_i,y_i)/\tau$). Temperature controls sharpness: small $\tau$ amplifies logits, forcing the positive to win decisively; large $\tau$ flattens the distribution, diluting positive-negative separation. Example: positive $\text{sim}=0.9$, negative $\text{sim}=0.2$; with $\tau=0.07$ (CLIP's default) logits become $0.9/0.07 \approx 12.86$ vs $0.2/0.07 \approx 2.86$, the positive softmax probability is $\approx 99.99\%$ and the loss is near zero; with $\tau=1$ the logits are only 0.9 vs 0.2, giving about $66.7\%$ positive probability — the model almost loses discriminative power.

Write the InfoNCE loss and explain every symbol? How is it related to cross-entropy?

How to pick temperature τ? What happens when τ is too small or too large (numeric example)?

Why does a larger batch size help? Where do negatives come from?

InfoNCE vs triplet loss and vanilla NCE: key differences?

How does temperature interact with gradients in CLIP's symmetric loss? Handling hard negatives?