M7-083M7: Retrieval, Ranking & RecSysOffline Evaluation & OPEEasy
⚡Flashcard Drill➔Mastery:

Explain Why Offline Evaluation Cannot Naively Use Historical Logged Data to Evaluate New Policies

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M7-084 →
📐 Mathematical Definition
history∼πold;Eπnew≠1n∑ri (biased)\text{history}\sim\pi_{\text{old}};\qquad \mathbb{E}_{\pi_{\text{new}}}\ne\frac1n\sum r_i\ (\text{biased})
⚡ One-Sentence Core Answer
Historical interaction logs reflect biased observations generated by a past logging policy; naively averaging historical rewards evaluates the old policy rather than the candidate policy, failing to observe how users would respond to novel recommendations.

📌 Key Takeaways

  • •
    Counterfactual unobservability: We only observe user feedback for items the old policy chose to display; user response to items the new policy would choose remains unobserved.
  • •
    Selection bias & logged distribution shift: P(Logged | x) reflects the historical algorithm's biases, not the true global relevance distribution.
  • •
    The offline evaluation imperative: Demands Off-Policy Evaluation (OPE) frameworks—such as Inverse Propensity Scoring (IPS) and Doubly Robust estimators—to compute unbiased policy values.

📐 Mathematical Derivations

Mathematical & Counterfactual Formulation: The Offline Policy Evaluation Gap.

(1) The Logged Bandit Feedback Problem:
Let user/context be x∼P(x)x \sim P(x), item action be a∈Aa \in \mathcal{A}, and reward be r(x,a)r(x, a).
Historical logged dataset D={(xi,ai,ri,pi)}i=1N\mathcal{D} = \{(x_i, a_i, r_i, p_i)\}_{i=1}^N was collected under logging policy $\pi_0$:
ai∼π0(a∣xi),pi=π0(ai∣xi)a_i \sim \pi_0(a \mid x_i), \quad p_i = \pi_0(a_i \mid x_i)ri is observed ONLY for chosen action air_i \text{ is observed ONLY for chosen action } a_i
We want to evaluate the expected performance of a novel candidate policy $\pi_1$:
V(π1)=Ex∼P(x),a∼π1(a∣x)[r(x,a)]V(\pi_1) = \mathbb{E}_{x \sim P(x), a \sim \pi_1(a \mid x)} [r(x, a)]

(2) Why Direct Empirical Averaging Fails (Naive Evaluation):
Suppose we filter historical logs where the old action matches the new action (ai=π1(xi)a_i = \pi_1(x_i)) and compute the average reward:
V^naive(π1)=∑i=1Nri⋅I(ai=π1(xi))∑i=1NI(ai=π1(xi))\hat{V}_{\text{naive}}(\pi_1) = \frac{\sum_{i=1}^N r_i \cdot \mathbb{I}(a_i = \pi_1(x_i))}{\sum_{i=1}^N \mathbb{I}(a_i = \pi_1(x_i))}
Mathematical Failure: This estimator is heavily biased because the condition ai=π1(xi)a_i = \pi_1(x_i) is conditioned on the old policy choosing aia_i. Actions that π0\pi_0 chose frequently are over-represented; actions π0\pi_0 rarely explored are under-represented or have zero samples.

(3) The Common Support Condition (Positivity):
Unbiased counterfactual estimation is mathematically possible if and only if the logging policy has non-zero probability for every action the new policy might take:
∀x,a:π1(a∣x)>0  ⟹  π0(a∣x)>0\forall x, a: \quad \pi_1(a \mid x) > 0 \implies \pi_0(a \mid x) > 0
If π0\pi_0 strictly never displayed action a∗a^*, historical data contains zero information regarding how users respond to a∗a^*.

🏭 Production Trade-offs

In-Depth Analysis & Engineering Trade-offs: ① The necessity of exploration traffic for offline evaluation—if production systems run deterministic greedy policies (π0(a)∈{0,1}\pi_0(a) \in \{0, 1\}), the positivity condition is violated for 99% of actions, making offline policy evaluation mathematically impossible; reserving 2%–5% of production traffic for randomized exploration logging enables offline counterfactual evaluation. ② The Direct Method (Model-based) vs. Inverse Propensity Scoring (Importance Sampling)—Direct Method trains a regression model r^(x,a)\hat{r}(x, a) and predicts rewards for π1\pi_1; it has low variance but suffers high bias if the reward model is misspecified. IPS is strictly unbiased but suffers high variance. ③ Logging propensity scores ($p_i$) at inference time—to evaluate policies offline using IPS, production systems must log the exact probability pi=π0(ai∣xi)p_i = \pi_0(a_i \mid x_i) into Kafka alongside the interaction event; reconstructing propensity scores retrospectively is impossible. ④ Position bias confounding—in search ranking, actions are slates of items; a relevant item may have received zero clicks because π0\pi_0 placed it at rank 20; counterfactual models must account for position propensities alongside item selection propensities. ⑤ Offline replay testing vs. Full A/B testing—unbiased offline policy evaluation acts as a rigorous gatekeeper, eliminating 70% of candidate models before spending cloud budget on live A/B tests. ⑥ Interview takeaway—formalize counterfactual unobservability (rr observed only for π0(a)\pi_0(a)), prove why naive averaging is biased, state the Common Support condition π1>0  ⟹  π0>0\pi_1 > 0 \implies \pi_0 > 0, and motivate Off-Policy Evaluation (OPE).
⚠️ Common Interview Pitfalls
  • ✕
    Evaluating a new ranking policy by filtering historical click logs for matching items and calculating empirical click rates, committing severe selection bias.
  • ✕
    Deploying deterministic greedy logging policies that assign zero probability to un-ranked items, permanently preventing counterfactual offline evaluation.
  • ✕
    Failing to log the execution propensity p_i = pi_0(a_i | x_i) at online inference time, rendering historical logs useless for importance sampling.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is the Common Support (Positivity) assumption mandatory for unbiased Off-Policy Evaluation?
  • ?
    What architectural logging patterns capture exact action propensities p_i without impacting online query latency?
  • ?
    How does the Direct Method (DM) compare against Inverse Propensity Scoring (IPS) in terms of the bias-variance trade-off?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM7-082: Explain Customer Lifetime Value (LTV) Modeling versus Short-Term Metric Trade-Offs📋 Back to BankNext →M7-084: Write the Inverse Propensity Scoring (IPS) Estimator and Explain Its High Variance and Mitigation Strategies