M7-088M7: Retrieval, Ranking & RecSysOffline Evaluation & OPEHard
⚡Flashcard Drill➔Mastery:

Explain Core Assumptions, Failure Modes, and Violation Diagnostics in Off-Policy Evaluation

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M7-089 →
📐 Mathematical Definition
assumptions: unconfoundedness, positivity, SUTVA;violation⇒fail\text{assumptions}:\ \text{unconfoundedness},\ \text{positivity},\ \text{SUTVA};\qquad \text{violation}\Rightarrow\text{fail}
⚡ One-Sentence Core Answer
Off-Policy Evaluation relies on three foundational causal assumptions—Unconfoundedness, Common Support (Positivity), and SUTVA; violations caused by unobserved features, zero-probability actions, interference, or non-stationarity trigger catastrophic evaluation failures.

📌 Key Takeaways

  • •
    Unconfoundedness (No Hidden Confounders): Action selection depends strictly on observed features logged in data: r indep a | x.
  • •
    Common Support / Positivity: Every action the candidate policy can propose must have had a strictly positive probability of being chosen by the logging policy: pi_0(a | x) > 0.
  • •
    SUTVA (No Interference): One user's recommendation treatment does not alter or interfere with the reward observed by another user.
  • •
    Non-Stationarity: Temporal drift in user preferences or catalog inventory breaks the assumption that historical response distributions mirror future reality.

📐 Mathematical Derivations

Mathematical & Causal Formulation: Foundational OPE Assumptions.

(1) Assumption 1: Unconfoundedness (Conditional Ignorability):
{r(x,a)}a∈A⊥A∣X\{r(x, a)\}_{a \in \mathcal{A}} \perp A \mid X
Failure Mode: An unobserved variable UU (e.g., user seasonal mood, external marketing campaign, sudden celebrity tweet) influenced which item the old policy recommended AND influenced user click probability. Because UU is missing from feature log XX, the backdoor path A←U→RA \leftarrow U \to R remains open, inducing severe causal bias.

(2) Assumption 2: Common Support / Overlap (Positivity):
∀x∈X, a∈A:π1(a∣x)>0  ⟹  π0(a∣x)≥ϵ>0\forall x \in \mathcal{X}, \, a \in \mathcal{A}: \quad \pi_1(a \mid x) > 0 \implies \pi_0(a \mid x) \ge \epsilon > 0
Failure Mode: Candidate policy π1\pi_1 proposes recommending newly ingested items or novel category crosses that the historical policy π0\pi_0 strictly never explored (π0(a∣x)=0\pi_0(a \mid x) = 0). The importance ratio π1(a)π0(a)=>00\frac{\pi_1(a)}{\pi_0(a)} = \frac{>0}{0} is mathematically undefined. OPE cannot evaluate actions never taken in history.

(3) Assumption 3: SUTVA (Stable Unit Treatment Value Assumption):
Ri(A1,A2,…,AN)=Ri(Ai)R_i(A_1, A_2, \dots, A_N) = R_i(A_i)
Failure Mode (Marketplace Interference): If policy π1\pi_1 recommends a limited-inventory luxury hotel, User 1 booking it prevents User 2 from booking it. In two-sided ridesharing, dispatching driver to Rider 1 removes the driver from Rider 2. One user's assignment directly corrupts another user's potential outcome.

(4) Assumption 4: Stationarity (Distribution Stability):
Pfuture(R∣X,A)=Ppast(R∣X,A)P_{\text{future}}(R \mid X, A) = P_{\text{past}}(R \mid X, A)
Failure Mode: User preferences drift post-holiday or after competitor product launches.

🏭 Production Trade-offs

In-Depth Analysis & Engineering Trade-offs: ① Deterministic logging policies break Positivity—if production runs a deterministic ranker (at=arg⁡max⁡f(x)a_t = \arg\max f(x)), 99.9% of catalog items have π0(a)=0\pi_0(a) = 0 for a given context; Positivity is violated for virtually all candidate policies; reserving a continuous 2% traffic slice for randomized or epsilon-greedy logging is mandatory for OPE. ② Hidden confounders in multi-device sessions—a user browses shoes on desktop at work, then opens the mobile app at home and buys; logging mobile interactions without linking desktop session features introduces severe unobserved confounding; cross-device identity stitching closes the backdoor path. ③ Detecting Positivity violations—evaluating the fraction of candidate policy mass placed on zero-propensity actions: αuncovered=Ex[∑a:π0(a∣x)=0π1(a∣x)]\alpha_{\text{uncovered}} = \mathbb{E}_{x}[\sum_{a: \pi_0(a \mid x)=0} \pi_1(a \mid x)]; if uncovered mass exceeds 5%, OPE must be rejected. ④ Synthetic actions and content generation—in generative recommendation or LLM prompt variations, the new policy outputs novel tokens never generated by the old policy; Positivity is 100% violated; OPE must rely on learned reward models (Direct Method) rather than importance sampling. ⑤ Marketplace cannibalization adjustments—when evaluating inventory-constrained systems, counterfactual simulators must model global inventory state transitions. ⑥ Interview takeaway—state the three core assumptions (Unconfoundedness, Positivity, SUTVA), detail how each assumption breaks in real-world recommender systems, explain why deterministic logging destroys Positivity, and outline diagnostic checks.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to perform importance sampling on candidate policies that propose new items, ignoring severe positivity violations (division by zero).
  • ✕
    Applying standard OPE to ride-sharing or marketplace inventory search without accounting for SUTVA violations caused by shared physical supply.
  • ✕
    Evaluating OPE across seasonal holiday boundaries (e.g., using Black Friday logs to evaluate a January shopping policy), violating stationarity.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does randomized exploration traffic guarantee the Positivity (Common Support) condition for downstream offline policy evaluation?
  • ?
    What statistical tests detect the presence of unobserved confounders in historical recommendation click logs?
  • ?
    How does cluster-based network randomization resolve SUTVA interference violations in off-policy marketplace evaluation?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM7-087: Explain the Methodological Verification of Correlation Between Offline OPE Metrics and Online A/B Results📋 Back to BankNext →M7-089: Explain Causal Directed Acyclic Graphs (DAGs), Do-Calculus, and Backdoor Paths in Counterfactual Evaluation