M5MediumPart M · LLM Eval & RAG/Agent Ops

Semantic Cache & Confidence-Based Cascading Router

⚡Flashcard Drill➔

Intercept repeat queries via cosine semantic cache and route remaining traffic through small models before escalating uncertain queries.

⏱️ Time: O(M_cache * D + |C|)
💾 Space: O(M_cache * D)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def route_query_with_cache_and_confidence(
query_emb: np.ndarray,
cache_embs: np.ndarray,
small_logits: np.ndarray,
cache_threshold: float = 0.92,
conf_threshold: float = 0.80,
) -> tuple:
"""
Two-stage LLM FinOps router:
1) Semantic cache lookup via normalized cosine similarity;
2) Fast small-model calibrated confidence check; escalate to frontier model only on low confidence.
"""
# Stage 1: L2-normalized semantic cache similarity
if len(cache_embs) > 0:
q_norm =
c_norms = np.maximum(np.linalg.norm(cache_embs, axis=1, keepdims=True), 1e-12)
sims = (cache_embs / c_norms) @ q_norm
best_idx = int(np.argmax(sims))
if float(sims[best_idx]) >= cache_threshold:
return "CACHE_HIT", float(sims[best_idx])
# Stage 2: Stable softmax confidence check on small model
shifted = small_logits - np.max(small_logits)
probs = np.exp(shifted) / np.sum(np.exp(shifted))
top_conf =
if top_conf >= conf_threshold:
return "SMALL_MODEL", top_conf
return "FRONTIER_MODEL", top_conf
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What is the biggest failure mode of semantic caching, and how do you guard against false hits?
Next: Cohort Retention Matrix (SQL-to-Python Bridge) →
💡

Core Mental Anchor / Mnemonic

Check tenant-scoped semantic cache first, pass high-confidence small model responses, escalate uncertain ones.

📐 Mathematical Derivation & Core Formula

Route(x)={Cachemax⁡ccos⁡(ex,ec)≥τcacheSmallmax⁡yPsmall(y∣x)≥τconfFrontierotherwise\text{Route}(x) = \begin{cases} \text{Cache} & \max_c \cos(e_x, e_c) \ge \tau_{\text{cache}} \\ \text{Small} & \max_y P_{\text{small}}(y \mid x) \ge \tau_{\text{conf}} \\ \text{Frontier} & \text{otherwise} \end{cases}
### Funneling Traffic Across Cost Tiers
Routing every request directly to a frontier model wastes cost and latency on routine queries. A two-stage cascade first checks L2-normalized embedding similarity against a cached answer store (sim >= cache_threshold), then evaluates small-model confidence (max_prob >= conf_threshold), escalating only low-confidence queries to the frontier tier.

🔄 Tensor Dimensions & Shape Flow

query_emb vs cache -> sim >= tau_cache -> CACHE_HIT else softmax(small_logits) -> >= tau_conf ? SMALL : FRONTIER

🛡️ Numerical Stability & Pitfalls Checklist

  • Scope semantic cache keys strictly by tenant_id and authorization context to prevent cross-tenant data leakage.
  • Calibrate small-model probabilities (e.g., Temperature Scaling) and verify entities/numbers on near-match cache hits.

💻 Reference Code Implementation

import numpy as np

def route_query_with_cache_and_confidence(
    query_emb: np.ndarray,
    cache_embs: np.ndarray,
    small_logits: np.ndarray,
    cache_threshold: float = 0.92,
    conf_threshold: float = 0.80,
) -> tuple:
    """
    Two-stage LLM FinOps router:
    1) Semantic cache lookup via normalized cosine similarity;
    2) Fast small-model calibrated confidence check; escalate to frontier model only on low confidence.
    """
    # Stage 1: L2-normalized semantic cache similarity
    if len(cache_embs) > 0:
        q_norm = query_emb / max(float(np.linalg.norm(query_emb)), 1e-12)
        c_norms = np.maximum(np.linalg.norm(cache_embs, axis=1, keepdims=True), 1e-12)
        sims = (cache_embs / c_norms) @ q_norm
        best_idx = int(np.argmax(sims))
        if float(sims[best_idx]) >= cache_threshold:
            return "CACHE_HIT", float(sims[best_idx])

    # Stage 2: Stable softmax confidence check on small model
    shifted = small_logits - np.max(small_logits)
    probs = np.exp(shifted) / np.sum(np.exp(shifted))
    top_conf = float(np.max(probs))
    if top_conf >= conf_threshold:
        return "SMALL_MODEL", top_conf
    return "FRONTIER_MODEL", top_conf

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
cache = np.array([[1.0, 0.0], [0.0, 1.0]])
# 1. Near-duplicate query hits semantic cache
route1, _ = route_query_with_cache_and_confidence(np.array([0.99, 0.05]), cache, np.array([1.0, 1.0]))
assert route1 == "CACHE_HIT"
# 2. Cache miss + confident small model stays on small model
route2, _ = route_query_with_cache_and_confidence(np.array([0.6, 0.8]), cache, np.array([5.0, 0.0]))
assert route2 == "SMALL_MODEL"
# 3. Cache miss + uncertain small model escalates to frontier
route3, _ = route_query_with_cache_and_confidence(np.array([0.6, 0.8]), cache, np.array([0.2, 0.1]))
assert route3 == "FRONTIER_MODEL"
print("✓ Cascading router & semantic cache assertion passed")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:What is the biggest failure mode of semantic caching, and how do you guard against false hits?
Queries differing only in a crucial entity, date, or negation often score > 0.95 cosine similarity and return the wrong answer. Guard with strict thresholds, numeric/entity exact-match verification, and disabling semantic cache on parameter-sensitive or personalized routes.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Agent Tool Loop Breaker & Idempotency GuardBack to Part M & All 105Next: Cohort Retention Matrix (SQL-to-Python Bridge) →