Long-Context Eval Needle/Lost-in-Middle

two evaluation methods for long-context ability. (1) Needle-in-a-Haystack: insert one memorable fact (the needle) at a random position inside a long irrelevant text (the haystack), ask for it, and measure precise recall $\text{Acc} = \tfrac{\text{needles retrieved}}{\text{total tests}}$, plotted as a recall heatmap over needle position and context length; (2) Lost-in-the-Middle (Liu et al., 2023): position experiments show models use content from the beginning and end far better than the middle — a U-shaped dip in the middle region, measured with multi-document KV retrieval tasks.

verifying real long-context capability claims (128K/1M) before release; RAG systems checking whether evidence is found wherever it sits in a document; context-window design (key info at the head/tail, re-ranked retrieval); a classic interview topic on why middle content gets lost.

advertised context length ≠ effective context use. Needle tests expose fake long-context ("can fit but can't read"), and Lost-in-the-Middle reveals positional bias in long-range attention — guiding mitigations on both the training side (positional interpolation, randomized document order) and inference side (key info first, section markers).

How do you design a needle-in-a-haystack test? What are the heatmap axes and what does an ideal model look like?

What is the U-shaped Lost-in-the-Middle dip, and why is middle content most often ignored?

Training-side mitigations for Lost-in-the-Middle: positional encoding, document order, attention bias — list them.

Inference-side organization: how to order retrieved chunks, where to put key info, and what section markers do.

Does a perfect needle score mean long-context is fully usable? What are its blind spots?