M7-083M7: Retrieval, Ranking & RecSysOffline Evaluation & OPEEasy
⚡Flashcard Drill➔Mastery:
Explain Why Offline Evaluation Cannot Naively Use Historical Logged Data to Evaluate New Policies
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Historical interaction logs reflect biased observations generated by a past logging policy; naively averaging historical rewards evaluates the old policy rather than the candidate policy, failing to observe how users would respond to novel recommendations.
📌 Key Takeaways
- •Counterfactual unobservability: We only observe user feedback for items the old policy chose to display; user response to items the new policy would choose remains unobserved.
- •Selection bias & logged distribution shift: P(Logged | x) reflects the historical algorithm's biases, not the true global relevance distribution.
- •The offline evaluation imperative: Demands Off-Policy Evaluation (OPE) frameworks—such as Inverse Propensity Scoring (IPS) and Doubly Robust estimators—to compute unbiased policy values.
📐 Mathematical Derivations
Mathematical & Counterfactual Formulation: The Offline Policy Evaluation Gap.
(1) The Logged Bandit Feedback Problem:
Let user/context be , item action be , and reward be .
Historical logged dataset was collected under logging policy $\pi_0$:
We want to evaluate the expected performance of a novel candidate policy $\pi_1$:
(2) Why Direct Empirical Averaging Fails (Naive Evaluation):
Suppose we filter historical logs where the old action matches the new action () and compute the average reward:
Mathematical Failure: This estimator is heavily biased because the condition is conditioned on the old policy choosing . Actions that chose frequently are over-represented; actions rarely explored are under-represented or have zero samples.
(3) The Common Support Condition (Positivity):
Unbiased counterfactual estimation is mathematically possible if and only if the logging policy has non-zero probability for every action the new policy might take:
If strictly never displayed action , historical data contains zero information regarding how users respond to .
(1) The Logged Bandit Feedback Problem:
Let user/context be , item action be , and reward be .
Historical logged dataset was collected under logging policy $\pi_0$:
We want to evaluate the expected performance of a novel candidate policy $\pi_1$:
(2) Why Direct Empirical Averaging Fails (Naive Evaluation):
Suppose we filter historical logs where the old action matches the new action () and compute the average reward:
Mathematical Failure: This estimator is heavily biased because the condition is conditioned on the old policy choosing . Actions that chose frequently are over-represented; actions rarely explored are under-represented or have zero samples.
(3) The Common Support Condition (Positivity):
Unbiased counterfactual estimation is mathematically possible if and only if the logging policy has non-zero probability for every action the new policy might take:
If strictly never displayed action , historical data contains zero information regarding how users respond to .
🏭 Production Trade-offs
In-Depth Analysis & Engineering Trade-offs: ① The necessity of exploration traffic for offline evaluation—if production systems run deterministic greedy policies (), the positivity condition is violated for 99% of actions, making offline policy evaluation mathematically impossible; reserving 2%–5% of production traffic for randomized exploration logging enables offline counterfactual evaluation. ② The Direct Method (Model-based) vs. Inverse Propensity Scoring (Importance Sampling)—Direct Method trains a regression model and predicts rewards for ; it has low variance but suffers high bias if the reward model is misspecified. IPS is strictly unbiased but suffers high variance. ③ Logging propensity scores ($p_i$) at inference time—to evaluate policies offline using IPS, production systems must log the exact probability into Kafka alongside the interaction event; reconstructing propensity scores retrospectively is impossible. ④ Position bias confounding—in search ranking, actions are slates of items; a relevant item may have received zero clicks because placed it at rank 20; counterfactual models must account for position propensities alongside item selection propensities. ⑤ Offline replay testing vs. Full A/B testing—unbiased offline policy evaluation acts as a rigorous gatekeeper, eliminating 70% of candidate models before spending cloud budget on live A/B tests. ⑥ Interview takeaway—formalize counterfactual unobservability ( observed only for ), prove why naive averaging is biased, state the Common Support condition , and motivate Off-Policy Evaluation (OPE).
⚠️ Common Interview Pitfalls
- ✕Evaluating a new ranking policy by filtering historical click logs for matching items and calculating empirical click rates, committing severe selection bias.
- ✕Deploying deterministic greedy logging policies that assign zero probability to un-ranked items, permanently preventing counterfactual offline evaluation.
- ✕Failing to log the execution propensity p_i = pi_0(a_i | x_i) at online inference time, rendering historical logs useless for importance sampling.
🎯 Interviewer Follow-up Inquiries
- ?Why is the Common Support (Positivity) assumption mandatory for unbiased Off-Policy Evaluation?
- ?What architectural logging patterns capture exact action propensities p_i without impacting online query latency?
- ?How does the Direct Method (DM) compare against Inverse Propensity Scoring (IPS) in terms of the bias-variance trade-off?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚