M7-089M7: Retrieval, Ranking & RecSysOffline Evaluation & OPEHard
⚡Flashcard Drill➔Mastery:

Explain Causal Directed Acyclic Graphs (DAGs), Do-Calculus, and Backdoor Paths in Counterfactual Evaluation

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M7-090 →
📐 Mathematical Definition
causal graph: X→A, X→R, A→R;goal: P(R∣do(A))\text{causal graph}:\ X\to A,\ X\to R,\ A\to R;\qquad \text{goal}:\ P(R|do(A))
⚡ One-Sentence Core Answer
Counterfactual evaluation utilizes causal Directed Acyclic Graphs (DAGs) to isolate causal policy effects from observational correlations; closing backdoor confounding paths via Pearl's do-calculus transforms observational logs into interventional distributions.

📌 Key Takeaways

  • •
    Observational vs. Interventional distribution: P(R | A=a) reflects correlation confounded by context; P(R | do(A=a)) reflects true causal outcome under algorithmic intervention.
  • •
    The Backdoor Criterion: Confounding occurs when context X simultaneously influences action selection A and reward R; conditioning on X blocks the backdoor path.
  • •
    Causal DAG topology: Nodes represent User Context (X), Logging Policy Action (A), Candidate Action (A*), and Observable Reward (R).

📐 Mathematical Derivations

Causal Graph Theory & Do-Calculus: Pearl's Causal Framework for RecSys.

(1) The Observational Causal DAG:
In a production recommendation logging system, the causal relationships are represented by Directed Acyclic Graph G\mathcal{G}:
X (User Context: Demographics, History, Time)↙↘A (Logged Action)→R (Observed Reward: Click, CVR)\begin{matrix} & X \text{ (User Context: Demographics, History, Time)} & \\ & \swarrow \qquad\qquad\qquad\qquad \searrow & \\ A \text{ (Logged Action)} & \xrightarrow{\qquad\qquad\qquad\qquad} & R \text{ (Observed Reward: Click, CVR)} \end{matrix}
- Edge X→AX \to A: The logging policy chooses action AA conditioning on context XX: A∼π0(a∣X)A \sim \pi_0(a \mid X).
- Edge X→RX \to R: User context directly influences intrinsic click/purchase affinity (e.g., wealthy users buy more).
- Edge A→RA \to R: The causal impact of presenting item AA on user reward RR.

(2) The Backdoor Confounding Path:
The path A←X→RA \leftarrow X \to R is a backdoor path from AA to RR.
Because of this path, the observational conditional distribution mixes true causation with spurious context correlation:
P(R∣A=a)=∑xP(R∣A=a,X=x)P(X=x∣A=a)≠P(R∣do(A=a))P(R \mid A = a) = \sum_x P(R \mid A=a, X=x) P(X=x \mid A=a) \neq P(R \mid do(A = a))
Example: If policy π0\pi_0 only recommends luxury watches (AA) to wealthy users (XX), then luxury watches exhibit high purchase rates P(R∣A)P(R \mid A) not because the watch is appealing, but because wealthy users buy everything.

(3) The Backdoor Adjustment Formula (Pearl's Do-Calculus):
To evaluate the causal effect of an intervention do(A=π1(X))do(A = \pi_1(X)), we sever all incoming edges into AA (X↛AX \not\to A), forcing AA to be chosen by candidate policy π1\pi_1:
P(R∣do(A=a))=∑x∈XP(R∣A=a,X=x)P(X=x)P(R \mid do(A = a)) = \sum_{x \in \mathcal{X}} P(R \mid A = a, X = x) P(X = x)
The expected policy value under intervention π1\pi_1 is:
V(π1)=EX∼P(X)[∑a∈Aπ1(a∣X)E[R∣X,A=a]]V(\pi_1) = \mathbb{E}_{X \sim P(X)} \left[ \sum_{a \in \mathcal{A}} \pi_1(a \mid X) \mathbb{E}[R \mid X, A = a] \right]
This bridges the gap between observational logging and counterfactual interventional reasoning.

🏭 Production Trade-offs

In-Depth Analysis & Engineering Trade-offs: ① The Backdoor Criterion requirement: Logging all confounding variables—conditioning on XX blocks the backdoor path A←X→RA \leftarrow X \to R if and only if all variables that simultaneously influence action choice and user reward are observed and logged; if the online service used user location to choose items but failed to record location in Kafka, unobserved confounding reopens the backdoor, making unbiased counterfactual inference impossible. ② Front-door adjustment when context is unobserved—if context XX is partially hidden, but an intermediate mediator MM exists (e.g., A→M→RA \to M \to R, such as user reading the product title), Pearl's front-door criterion can theoretically recover causal effects without observing XX. ③ Selection bias as collider conditioning—conditioning on clicks (C=1C=1) when evaluating post-click conversion (CVR) treats a collider as observed: A→C←UA \to C \leftarrow U; conditioning on collider CC opens a spurious correlation between AA and unobserved user intent UU, mathematically explaining Sample Selection Bias (SSB) and justifying ESMM. ④ Instrumental Variables (IV) for recommendation—when unobserved confounding is pervasive, using random algorithmic assignment or network jitter as an Instrumental Variable ZZ (Z→A→RZ \to A \to R with Z⊥UZ \perp U) isolates true causal treatment effects. ⑤ Causal DAG validation in system design—drawing explicit causal DAGs during architecture reviews forces engineering teams to identify which features must be captured at inference time to enable future counterfactual modeling. ⑥ Interview takeaway—draw the causal DAG (X→A,X→R,A→RX \to A, X \to R, A \to R), explain why P(R∣A)≠P(R∣do(A))P(R \mid A) \neq P(R \mid do(A)) due to the backdoor path A←X→RA \leftarrow X \to R, derive the Backdoor Adjustment formula, and explain collider bias in CVR modeling.
⚠️ Common Interview Pitfalls
  • ✕
    Confusing observational correlation P(R | A) with interventional causation P(R | do(A)), deploying policies that replicate spurious historical correlations.
  • ✕
    Failing to log critical contextual features that the online ranking algorithm used, permanently opening unblocked backdoor confounding paths.
  • ✕
    Conditioning on post-treatment colliders (e.g., conditioning only on clicked samples in CVR), inducing spurious correlations between items and user intent.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does conditioning on a collider node (e.g., user click C in A -> C <- U) induce selection bias in downstream conversion modeling?
  • ?
    What properties must an Instrumental Variable (IV) satisfy to recover unbiased causal treatment effects when hidden confounders exist?
  • ?
    How does Pearl's do-calculus convert interventional counterfactual distributions into estimable observational probabilities?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM7-088: Explain Core Assumptions, Failure Modes, and Violation Diagnostics in Off-Policy Evaluation📋 Back to BankNext →M7-090: Explain Off-Policy Evaluation for Sequential Decision-Making: Multi-Step Trajectory IPS and Doubly Robust