M7-089M7: Retrieval, Ranking & RecSysOffline Evaluation & OPEHard
⚡Flashcard Drill➔Mastery:
Explain Causal Directed Acyclic Graphs (DAGs), Do-Calculus, and Backdoor Paths in Counterfactual Evaluation
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Counterfactual evaluation utilizes causal Directed Acyclic Graphs (DAGs) to isolate causal policy effects from observational correlations; closing backdoor confounding paths via Pearl's do-calculus transforms observational logs into interventional distributions.
📌 Key Takeaways
- •Observational vs. Interventional distribution: P(R | A=a) reflects correlation confounded by context; P(R | do(A=a)) reflects true causal outcome under algorithmic intervention.
- •The Backdoor Criterion: Confounding occurs when context X simultaneously influences action selection A and reward R; conditioning on X blocks the backdoor path.
- •Causal DAG topology: Nodes represent User Context (X), Logging Policy Action (A), Candidate Action (A*), and Observable Reward (R).
📐 Mathematical Derivations
Causal Graph Theory & Do-Calculus: Pearl's Causal Framework for RecSys.
(1) The Observational Causal DAG:
In a production recommendation logging system, the causal relationships are represented by Directed Acyclic Graph :
- Edge : The logging policy chooses action conditioning on context : .
- Edge : User context directly influences intrinsic click/purchase affinity (e.g., wealthy users buy more).
- Edge : The causal impact of presenting item on user reward .
(2) The Backdoor Confounding Path:
The path is a backdoor path from to .
Because of this path, the observational conditional distribution mixes true causation with spurious context correlation:
Example: If policy only recommends luxury watches () to wealthy users (), then luxury watches exhibit high purchase rates not because the watch is appealing, but because wealthy users buy everything.
(3) The Backdoor Adjustment Formula (Pearl's Do-Calculus):
To evaluate the causal effect of an intervention , we sever all incoming edges into (), forcing to be chosen by candidate policy :
The expected policy value under intervention is:
This bridges the gap between observational logging and counterfactual interventional reasoning.
(1) The Observational Causal DAG:
In a production recommendation logging system, the causal relationships are represented by Directed Acyclic Graph :
- Edge : The logging policy chooses action conditioning on context : .
- Edge : User context directly influences intrinsic click/purchase affinity (e.g., wealthy users buy more).
- Edge : The causal impact of presenting item on user reward .
(2) The Backdoor Confounding Path:
The path is a backdoor path from to .
Because of this path, the observational conditional distribution mixes true causation with spurious context correlation:
Example: If policy only recommends luxury watches () to wealthy users (), then luxury watches exhibit high purchase rates not because the watch is appealing, but because wealthy users buy everything.
(3) The Backdoor Adjustment Formula (Pearl's Do-Calculus):
To evaluate the causal effect of an intervention , we sever all incoming edges into (), forcing to be chosen by candidate policy :
The expected policy value under intervention is:
This bridges the gap between observational logging and counterfactual interventional reasoning.
🏭 Production Trade-offs
In-Depth Analysis & Engineering Trade-offs: ① The Backdoor Criterion requirement: Logging all confounding variables—conditioning on blocks the backdoor path if and only if all variables that simultaneously influence action choice and user reward are observed and logged; if the online service used user location to choose items but failed to record location in Kafka, unobserved confounding reopens the backdoor, making unbiased counterfactual inference impossible. ② Front-door adjustment when context is unobserved—if context is partially hidden, but an intermediate mediator exists (e.g., , such as user reading the product title), Pearl's front-door criterion can theoretically recover causal effects without observing . ③ Selection bias as collider conditioning—conditioning on clicks () when evaluating post-click conversion (CVR) treats a collider as observed: ; conditioning on collider opens a spurious correlation between and unobserved user intent , mathematically explaining Sample Selection Bias (SSB) and justifying ESMM. ④ Instrumental Variables (IV) for recommendation—when unobserved confounding is pervasive, using random algorithmic assignment or network jitter as an Instrumental Variable ( with ) isolates true causal treatment effects. ⑤ Causal DAG validation in system design—drawing explicit causal DAGs during architecture reviews forces engineering teams to identify which features must be captured at inference time to enable future counterfactual modeling. ⑥ Interview takeaway—draw the causal DAG (), explain why due to the backdoor path , derive the Backdoor Adjustment formula, and explain collider bias in CVR modeling.
⚠️ Common Interview Pitfalls
- ✕Confusing observational correlation P(R | A) with interventional causation P(R | do(A)), deploying policies that replicate spurious historical correlations.
- ✕Failing to log critical contextual features that the online ranking algorithm used, permanently opening unblocked backdoor confounding paths.
- ✕Conditioning on post-treatment colliders (e.g., conditioning only on clicked samples in CVR), inducing spurious correlations between items and user intent.
🎯 Interviewer Follow-up Inquiries
- ?How does conditioning on a collider node (e.g., user click C in A -> C <- U) induce selection bias in downstream conversion modeling?
- ?What properties must an Instrumental Variable (IV) satisfy to recover unbiased causal treatment effects when hidden confounders exist?
- ?How does Pearl's do-calculus convert interventional counterfactual distributions into estimable observational probabilities?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚