J7Medium★ Enterprise RAG Essential · Retrieval FusionPart J · RecSys & Search Metrics

Pre-Filtered Hybrid Search with RRF Fusion

⚡Flashcard Drill➔

Apply metadata/ACL pre-filtering before computing Reciprocal Rank Fusion across dense and sparse retrievers to prevent recall collapse.

⏱️ Time: O(M_valid log M_valid) 候选集排序融合
💾 Space: O(M_valid)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def hybrid_search_rrf_prefilter(
dense_scores: np.ndarray,
sparse_scores: np.ndarray,
allowed_mask: np.ndarray,
top_k: int = 3,
rrf_k: int = 60,
) -> list:
"""
Pre-filtered Hybrid Search combining Dense cosine and Sparse lexical scores via RRF.
Pre-filtering before Top-K/fusion guarantees recall never collapses under strict ACL/metadata filters.
"""
valid_indices =
if len(valid_indices) == 0:
return []
# Rank strictly within pre-filtered candidate pool (1-based rank, descending score)
dense_order = valid_indices[np.argsort(-dense_scores[valid_indices], kind="mergesort")]
sparse_order = valid_indices[np.argsort(-sparse_scores[valid_indices], kind="mergesort")]
rrf = {}
for rank_0, doc_id in enumerate(dense_order):
rrf[int(doc_id)] = rrf.get(int(doc_id), 0.0) + 1.0 / (rrf_k + rank_0 + 1)
for rank_0, doc_id in enumerate(sparse_order):
rrf[int(doc_id)] = rrf.get(int(doc_id), 0.0) + 1.0 / (rrf_k + rank_0 + 1)
ranked =
return [doc_id for doc_id, _ in ranked[:top_k]]
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why can strict pre-filtering disconnect an HNSW graph under high selectivity, and how do vector DBs solve it?
Next: DDPM Forward Diffusion Process →
💡

Core Mental Anchor / Mnemonic

Pre-filter first to prevent empty Top-K; fuse incompatible dense and sparse scales via 1/(k + rank).

📐 Mathematical Derivation & Core Formula

RRF(d)=∑m∈{dense,sparse}wmk+rankm(d)(d∈DACL)\mathrm{RRF}(d) = \sum_{m \in \{\text{dense}, \text{sparse}\}} \frac{w_m}{k + \mathrm{rank}_m(d)} \quad (d \in \mathcal{D}_{\text{ACL}})
### Avoiding Post-Filtering Collapse & Score-Scale Mismatch
Post-filtering global Top-KK hits under selective metadata or tenant ACL filters (e.g., 1% selectivity) frequently yields empty outputs. Filtering before ranking ensures all Top-KK slots serve valid candidates.
Because cosine similarities and BM25 scores operate on incompatible scales, RRF sums reciprocal ranks 1/(k+rankm(d))1 / (k + \mathrm{rank}_m(d)) cleanly without brittle score calibration.

🔄 Tensor Dimensions & Shape Flow

scores + allowed_mask -> valid_indices -> independent descending ranks -> accumulate 1/(rrf_k + rank) -> Top-K

🛡️ Numerical Stability & Pitfalls Checklist

  • Always filter via allowed_mask before Top-K selection under tight tenant ACL or freshness filters.
  • Return empty results cleanly when zero records satisfy the pre-filter mask.

💻 Reference Code Implementation

import numpy as np

def hybrid_search_rrf_prefilter(
    dense_scores: np.ndarray,
    sparse_scores: np.ndarray,
    allowed_mask: np.ndarray,
    top_k: int = 3,
    rrf_k: int = 60,
) -> list:
    """
    Pre-filtered Hybrid Search combining Dense cosine and Sparse lexical scores via RRF.
    Pre-filtering before Top-K/fusion guarantees recall never collapses under strict ACL/metadata filters.
    """
    valid_indices = np.where(allowed_mask)[0]
    if len(valid_indices) == 0:
        return []

    # Rank strictly within pre-filtered candidate pool (1-based rank, descending score)
    dense_order = valid_indices[np.argsort(-dense_scores[valid_indices], kind="mergesort")]
    sparse_order = valid_indices[np.argsort(-sparse_scores[valid_indices], kind="mergesort")]

    rrf = {}
    for rank_0, doc_id in enumerate(dense_order):
        rrf[int(doc_id)] = rrf.get(int(doc_id), 0.0) + 1.0 / (rrf_k + rank_0 + 1)
    for rank_0, doc_id in enumerate(sparse_order):
        rrf[int(doc_id)] = rrf.get(int(doc_id), 0.0) + 1.0 / (rrf_k + rank_0 + 1)

    ranked = sorted(rrf.items(), key=lambda kv: (-kv[1], kv[0]))
    return [doc_id for doc_id, _ in ranked[:top_k]]

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
# Counterexample: docs 0..3 rank highest globally but fail tenant ACL; only docs 4, 5 match ACL!
dense = np.array([0.99, 0.98, 0.97, 0.96, 0.60, 0.55])
sparse = np.array([9.0, 8.5, 8.0, 7.5, 4.0, 5.2])
acl_mask = np.array([False, False, False, False, True, True])
res = hybrid_search_rrf_prefilter(dense, sparse, acl_mask, top_k=2, rrf_k=60)
assert res == [4, 5] or res == [5, 4], f"Pre-filter must retrieve valid docs [4, 5], got {res}"
print("✓ Hybrid Search pre-filter RRF assertion passed")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why can strict pre-filtering disconnect an HNSW graph under high selectivity, and how do vector DBs solve it?
When 99% of nodes are masked out, graph traversal may lose navigable edges. Engines either traverse masked nodes as routing bridges while only emitting allowed results, or fall back to exact flat scans when the filtered subset is small.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: BM25 (Okapi BM25) Ranking AlgorithmBack to Part J & All 105Next: DDPM Forward Diffusion Process →