M7-088M7: Retrieval, Ranking & RecSysOffline Evaluation & OPEHard
⚡Flashcard Drill➔Mastery:
Explain Core Assumptions, Failure Modes, and Violation Diagnostics in Off-Policy Evaluation
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Off-Policy Evaluation relies on three foundational causal assumptions—Unconfoundedness, Common Support (Positivity), and SUTVA; violations caused by unobserved features, zero-probability actions, interference, or non-stationarity trigger catastrophic evaluation failures.
📌 Key Takeaways
- •Unconfoundedness (No Hidden Confounders): Action selection depends strictly on observed features logged in data: r indep a | x.
- •Common Support / Positivity: Every action the candidate policy can propose must have had a strictly positive probability of being chosen by the logging policy: pi_0(a | x) > 0.
- •SUTVA (No Interference): One user's recommendation treatment does not alter or interfere with the reward observed by another user.
- •Non-Stationarity: Temporal drift in user preferences or catalog inventory breaks the assumption that historical response distributions mirror future reality.
📐 Mathematical Derivations
Mathematical & Causal Formulation: Foundational OPE Assumptions.
(1) Assumption 1: Unconfoundedness (Conditional Ignorability):
Failure Mode: An unobserved variable (e.g., user seasonal mood, external marketing campaign, sudden celebrity tweet) influenced which item the old policy recommended AND influenced user click probability. Because is missing from feature log , the backdoor path remains open, inducing severe causal bias.
(2) Assumption 2: Common Support / Overlap (Positivity):
Failure Mode: Candidate policy proposes recommending newly ingested items or novel category crosses that the historical policy strictly never explored (). The importance ratio is mathematically undefined. OPE cannot evaluate actions never taken in history.
(3) Assumption 3: SUTVA (Stable Unit Treatment Value Assumption):
Failure Mode (Marketplace Interference): If policy recommends a limited-inventory luxury hotel, User 1 booking it prevents User 2 from booking it. In two-sided ridesharing, dispatching driver to Rider 1 removes the driver from Rider 2. One user's assignment directly corrupts another user's potential outcome.
(4) Assumption 4: Stationarity (Distribution Stability):
Failure Mode: User preferences drift post-holiday or after competitor product launches.
(1) Assumption 1: Unconfoundedness (Conditional Ignorability):
Failure Mode: An unobserved variable (e.g., user seasonal mood, external marketing campaign, sudden celebrity tweet) influenced which item the old policy recommended AND influenced user click probability. Because is missing from feature log , the backdoor path remains open, inducing severe causal bias.
(2) Assumption 2: Common Support / Overlap (Positivity):
Failure Mode: Candidate policy proposes recommending newly ingested items or novel category crosses that the historical policy strictly never explored (). The importance ratio is mathematically undefined. OPE cannot evaluate actions never taken in history.
(3) Assumption 3: SUTVA (Stable Unit Treatment Value Assumption):
Failure Mode (Marketplace Interference): If policy recommends a limited-inventory luxury hotel, User 1 booking it prevents User 2 from booking it. In two-sided ridesharing, dispatching driver to Rider 1 removes the driver from Rider 2. One user's assignment directly corrupts another user's potential outcome.
(4) Assumption 4: Stationarity (Distribution Stability):
Failure Mode: User preferences drift post-holiday or after competitor product launches.
🏭 Production Trade-offs
In-Depth Analysis & Engineering Trade-offs: ① Deterministic logging policies break Positivity—if production runs a deterministic ranker (), 99.9% of catalog items have for a given context; Positivity is violated for virtually all candidate policies; reserving a continuous 2% traffic slice for randomized or epsilon-greedy logging is mandatory for OPE. ② Hidden confounders in multi-device sessions—a user browses shoes on desktop at work, then opens the mobile app at home and buys; logging mobile interactions without linking desktop session features introduces severe unobserved confounding; cross-device identity stitching closes the backdoor path. ③ Detecting Positivity violations—evaluating the fraction of candidate policy mass placed on zero-propensity actions: ; if uncovered mass exceeds 5%, OPE must be rejected. ④ Synthetic actions and content generation—in generative recommendation or LLM prompt variations, the new policy outputs novel tokens never generated by the old policy; Positivity is 100% violated; OPE must rely on learned reward models (Direct Method) rather than importance sampling. ⑤ Marketplace cannibalization adjustments—when evaluating inventory-constrained systems, counterfactual simulators must model global inventory state transitions. ⑥ Interview takeaway—state the three core assumptions (Unconfoundedness, Positivity, SUTVA), detail how each assumption breaks in real-world recommender systems, explain why deterministic logging destroys Positivity, and outline diagnostic checks.
⚠️ Common Interview Pitfalls
- ✕Attempting to perform importance sampling on candidate policies that propose new items, ignoring severe positivity violations (division by zero).
- ✕Applying standard OPE to ride-sharing or marketplace inventory search without accounting for SUTVA violations caused by shared physical supply.
- ✕Evaluating OPE across seasonal holiday boundaries (e.g., using Black Friday logs to evaluate a January shopping policy), violating stationarity.
🎯 Interviewer Follow-up Inquiries
- ?How does randomized exploration traffic guarantee the Positivity (Common Support) condition for downstream offline policy evaluation?
- ?What statistical tests detect the presence of unobserved confounders in historical recommendation click logs?
- ?How does cluster-based network randomization resolve SUTVA interference violations in off-policy marketplace evaluation?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚