M3Medium★ RAG 排障必考 · 召回损失诊断Part M · LLM 评测与 Agent/RAG 算子
RAG 切分断层(Chunk Boundary Collapse)检测与重叠滑窗审计
模拟并审计固定切片无重叠导致的关键证据跨界撕裂,以及滑动重叠(Overlap)或父子块索引对完整语义召回的修复效果。
⏱️ 时间复杂度:
O(doc_len / (chunk_size - overlap))💾 空间复杂度:
O(NumChunks)🎯 免登录实战自测闭环(先做再看答案)
按「机制回忆 → 关键行填空 → 边界反例 → 追问与错题记录」顺序自测,进度自动保存在本地。
Step 1 · 机制口述与数值稳定性自检
Step 2 · 核心代码关键行补全
def chunk_spans_with_overlap(doc_len: int, chunk_size: int, overlap: int) -> list:
"""
Generate [start, end) character/token index spans for sliding-window chunking.
"""
if chunk_size <= 0 or overlap < 0 or overlap >= chunk_size:
raise ValueError("Require chunk_size > 0 and 0 <= overlap < chunk_size")
stride =
spans = []
pos = 0
while pos < doc_len:
end =
spans.append((pos, end))
if end == doc_len:
break
pos += stride
return spans
def span_recall_contained(doc_len: int, evidence_span: tuple, chunk_size: int, overlap: int) -> float:
"""
Return 1.0 if the full golden evidence span [ev_start, ev_end) is intact inside at least one chunk, else 0.0.
"""
ev_start, ev_end = evidence_span
spans = chunk_spans_with_overlap(doc_len, chunk_size, overlap)
intact = any(c_start <= ev_start and ev_end <= c_end for c_start, c_end in spans)
return 1.0 if intact else 0.0
Step 3 · 反例与边界断言核对
Step 4 · 面试追问盲答与错题记录
Q1: Parent-Child Retrieval(父子分块索引)如何同时解决“小块噪音少但上下文残缺”与“大块上下文全但嵌入向量稀释”的矛盾?
💡
实战一句话记忆口诀
无重叠切碎边界证据,滑窗重叠护短跨句,小块检索大块生成
📐 核心数学公式与机理推导
### 切分边界撕裂与步长约束
企业 RAG 线上常见的假性失效并非向量模型能力差,而是文档分块(Chunking)在固定长度 处恰好把“指标数值与限制条件(如
若设置重叠区间 (常取 块长),相邻窗口步长 :只要目标关键证据跨度 ,就一定存在至少一个分块能完整覆盖它而不被切碎。
工程进阶:除简单滑动重叠外,还常用小块检索、大块送入 LLM(Small-to-Big / Parent Document Retriever)兼顾向量表征纯度与上下文完整度。
企业 RAG 线上常见的假性失效并非向量模型能力差,而是文档分块(Chunking)在固定长度 处恰好把“指标数值与限制条件(如
Q4除外)”一刀两断。若设置重叠区间 (常取 块长),相邻窗口步长 :只要目标关键证据跨度 ,就一定存在至少一个分块能完整覆盖它而不被切碎。
工程进阶:除简单滑动重叠外,还常用小块检索、大块送入 LLM(Small-to-Big / Parent Document Retriever)兼顾向量表征纯度与上下文完整度。
🔄 张量维度与形状流转说明
doc_len, chunk_size, overlap -> stride = chunk_size - overlap -> 生成窗口切片 -> 检测 golden_span 是否完整落在单块内
🛡️ 数值稳定性自检与边界要点
- 严格约束
0 <= overlap < chunk_size,否则步长stride <= 0会引发无限死循环 - 对于 Markdown/PDF 表格或代码段,切忌按纯字符硬切,优先按结构分隔符(标题/整张表)+ Parent-Child 回溯完整父块
💻 完整参考代码实现
def chunk_spans_with_overlap(doc_len: int, chunk_size: int, overlap: int) -> list:
"""
Generate [start, end) character/token index spans for sliding-window chunking.
"""
if chunk_size <= 0 or overlap < 0 or overlap >= chunk_size:
raise ValueError("Require chunk_size > 0 and 0 <= overlap < chunk_size")
stride = chunk_size - overlap
spans = []
pos = 0
while pos < doc_len:
end = min(doc_len, pos + chunk_size)
spans.append((pos, end))
if end == doc_len:
break
pos += stride
return spans
def span_recall_contained(doc_len: int, evidence_span: tuple, chunk_size: int, overlap: int) -> float:
"""
Return 1.0 if the full golden evidence span [ev_start, ev_end) is intact inside at least one chunk, else 0.0.
"""
ev_start, ev_end = evidence_span
spans = chunk_spans_with_overlap(doc_len, chunk_size, overlap)
intact = any(c_start <= ev_start and ev_end <= c_end for c_start, c_end in spans)
return 1.0 if intact else 0.0
🧪 本地测试与正确性断言
可直接复制到本地 Python / Jupyter 验证:
# Evidence spans index [9, 12), crossing a strict 10-unit boundary
doc_len = 25
evidence = (9, 12)
recall_no_overlap = span_recall_contained(doc_len, evidence, chunk_size=10, overlap=0)
recall_with_overlap = span_recall_contained(doc_len, evidence, chunk_size=10, overlap=4)
assert recall_no_overlap == 0.0, "Zero-overlap chunking tears boundary evidence apart!"
assert recall_with_overlap == 1.0, "Sliding overlap must keep cross-boundary evidence intact!"
print("✓ RAG chunk-boundary collapse audit assertion passed")🎯 核心工程追问与标准应答
Q1:Parent-Child Retrieval(父子分块索引)如何同时解决“小块噪音少但上下文残缺”与“大块上下文全但嵌入向量稀释”的矛盾?
先将文档切分为 128–256 tokens 的精准子块(Child Chunks)做嵌入检索保证匹配信噪比;命中子块后通过存储的
parent_id 返回其所属的 1024–2048 tokens 父段落或整节 Markdown(Parent Chunk)交给大模型生成,兼得检索精度与推理完整性。⚡
想检验本题的肌肉记忆?进入 3 阶闪卡主动回忆盲测
隐藏答案遮罩,自测:审题锚点 ➔ 数理公式机理 ➔ 工业标杆代码与连环追问。3 分钟攻破认知盲区!