M5MediumPart M · LLM Eval & RAG/Agent Ops
Semantic Cache & Confidence-Based Cascading Router
Intercept repeat queries via cosine semantic cache and route remaining traffic through small models before escalating uncertain queries.
⏱️ Time:
O(M_cache * D + |C|)💾 Space:
O(M_cache * D)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def route_query_with_cache_and_confidence(
query_emb: np.ndarray,
cache_embs: np.ndarray,
small_logits: np.ndarray,
cache_threshold: float = 0.92,
conf_threshold: float = 0.80,
) -> tuple:
"""
Two-stage LLM FinOps router:
1) Semantic cache lookup via normalized cosine similarity;
2) Fast small-model calibrated confidence check; escalate to frontier model only on low confidence.
"""
# Stage 1: L2-normalized semantic cache similarity
if len(cache_embs) > 0:
q_norm =
c_norms = np.maximum(np.linalg.norm(cache_embs, axis=1, keepdims=True), 1e-12)
sims = (cache_embs / c_norms) @ q_norm
best_idx = int(np.argmax(sims))
if float(sims[best_idx]) >= cache_threshold:
return "CACHE_HIT", float(sims[best_idx])
# Stage 2: Stable softmax confidence check on small model
shifted = small_logits - np.max(small_logits)
probs = np.exp(shifted) / np.sum(np.exp(shifted))
top_conf =
if top_conf >= conf_threshold:
return "SMALL_MODEL", top_conf
return "FRONTIER_MODEL", top_conf
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What is the biggest failure mode of semantic caching, and how do you guard against false hits?
💡
Core Mental Anchor / Mnemonic
Check tenant-scoped semantic cache first, pass high-confidence small model responses, escalate uncertain ones.
📐 Mathematical Derivation & Core Formula
### Funneling Traffic Across Cost Tiers
Routing every request directly to a frontier model wastes cost and latency on routine queries. A two-stage cascade first checks L2-normalized embedding similarity against a cached answer store (
Routing every request directly to a frontier model wastes cost and latency on routine queries. A two-stage cascade first checks L2-normalized embedding similarity against a cached answer store (
sim >= cache_threshold), then evaluates small-model confidence (max_prob >= conf_threshold), escalating only low-confidence queries to the frontier tier.🔄 Tensor Dimensions & Shape Flow
query_emb vs cache -> sim >= tau_cache -> CACHE_HIT else softmax(small_logits) -> >= tau_conf ? SMALL : FRONTIER
🛡️ Numerical Stability & Pitfalls Checklist
- Scope semantic cache keys strictly by
tenant_idand authorization context to prevent cross-tenant data leakage. - Calibrate small-model probabilities (e.g., Temperature Scaling) and verify entities/numbers on near-match cache hits.
💻 Reference Code Implementation
import numpy as np
def route_query_with_cache_and_confidence(
query_emb: np.ndarray,
cache_embs: np.ndarray,
small_logits: np.ndarray,
cache_threshold: float = 0.92,
conf_threshold: float = 0.80,
) -> tuple:
"""
Two-stage LLM FinOps router:
1) Semantic cache lookup via normalized cosine similarity;
2) Fast small-model calibrated confidence check; escalate to frontier model only on low confidence.
"""
# Stage 1: L2-normalized semantic cache similarity
if len(cache_embs) > 0:
q_norm = query_emb / max(float(np.linalg.norm(query_emb)), 1e-12)
c_norms = np.maximum(np.linalg.norm(cache_embs, axis=1, keepdims=True), 1e-12)
sims = (cache_embs / c_norms) @ q_norm
best_idx = int(np.argmax(sims))
if float(sims[best_idx]) >= cache_threshold:
return "CACHE_HIT", float(sims[best_idx])
# Stage 2: Stable softmax confidence check on small model
shifted = small_logits - np.max(small_logits)
probs = np.exp(shifted) / np.sum(np.exp(shifted))
top_conf = float(np.max(probs))
if top_conf >= conf_threshold:
return "SMALL_MODEL", top_conf
return "FRONTIER_MODEL", top_conf
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
cache = np.array([[1.0, 0.0], [0.0, 1.0]])
# 1. Near-duplicate query hits semantic cache
route1, _ = route_query_with_cache_and_confidence(np.array([0.99, 0.05]), cache, np.array([1.0, 1.0]))
assert route1 == "CACHE_HIT"
# 2. Cache miss + confident small model stays on small model
route2, _ = route_query_with_cache_and_confidence(np.array([0.6, 0.8]), cache, np.array([5.0, 0.0]))
assert route2 == "SMALL_MODEL"
# 3. Cache miss + uncertain small model escalates to frontier
route3, _ = route_query_with_cache_and_confidence(np.array([0.6, 0.8]), cache, np.array([0.2, 0.1]))
assert route3 == "FRONTIER_MODEL"
print("✓ Cascading router & semantic cache assertion passed")🎯 Core Follow-up Inquiries & Reference Answers
Q1:What is the biggest failure mode of semantic caching, and how do you guard against false hits?
Queries differing only in a crucial entity, date, or negation often score > 0.95 cosine similarity and return the wrong answer. Guard with strict thresholds, numeric/entity exact-match verification, and disabling semantic cache on parameter-sensitive or personalized routes.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.