M7-096M7: Retrieval, Ranking & RecSysOnline Metrics & GuardrailsMedium
⚡Flashcard Drill➔Mastery:

Explain Network Interference, Marketplace Cannibalization, and SUTVA Violations in Online Experimentation

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M7-097 →
📐 Mathematical Definition
SUTVA violated: units interact;fix: cluster randomization, switchback\text{SUTVA violated}:\ \text{units interact};\qquad \text{fix}:\ \text{cluster randomization},\ \text{switchback}
⚡ One-Sentence Core Answer
Network effects and shared marketplace resources violate the Stable Unit Treatment Value Assumption (SUTVA); treatment users interfere with control users through social contagion or inventory cannibalization, requiring cluster randomization, switchback tests, or synthetic controls.

📌 Key Takeaways

  • •
    SUTVA definition: Assumes the potential outcome of any experimental unit is independent of the treatment assignments of all other units.
  • •
    Two-sided marketplace interference: In Uber, Airbnb, or Amazon, treatment users booking rides or buying limited stock cannibalizes availability for control users.
  • •
    Social network spillover: In social feeds, treatment users sharing more content directly alters the feed experience of their control group friends.
  • •
    Mitigation architectures: Cluster-based randomization (graph community partitioning), Switchback testing (time-slice randomization), and Synthetic Controls.

📐 Mathematical Derivations

Causal & Experimental Methodology: Interference & SUTVA Violations.

(1) The Stable Unit Treatment Value Assumption (SUTVA):
Let W=(W1,W2,…,WN)∈{0,1}N\mathbf{W} = (W_1, W_2, \dots, W_N) \in \{0, 1\}^N be the global vector of treatment assignments. SUTVA requires:
Yi(W)=Yi(Wi)∀i∈[1,N]Y_i(\mathbf{W}) = Y_i(W_i) \quad \forall i \in [1, N]
The potential outcome YiY_i depends only on user ii's own treatment WiW_i, completely invariant to how any other user jj is treated.

(2) Mechanisms of SUTVA Violations:
- Marketplace Supply Cannibalization (Indirect Interference):
Suppose Treatment algorithm recommends discounted hotel rooms. Treatment users book 80% of rooms. Control users search for rooms, find them sold out, and bounce (Ycontrol↓Y_{\text{control}} \downarrow). The estimated treatment effect τ^=Yˉtreatment−Yˉcontrol\hat{\tau} = \bar{Y}_{\text{treatment}} - \bar{Y}_{\text{control}} artificially inflates, falsely attributing control group starvation to treatment brilliance.
- Social Contagion (Direct Graph Spillover):
Treatment user ii is given a viral sharing feature; they share 10 posts. Friend jj (assigned to Control) sees these posts in their feed, increasing their engagement (Ycontrol↑Y_{\text{control}} \uparrow). The estimated lift YˉT−YˉC\bar{Y}_T - \bar{Y}_C collapses toward zero.

(3) Methodological Solutions:
- Cluster-Based Graph Randomization:
Partition the user social graph into disjoint communities C1,…,CK\mathcal{C}_1, \dots, \mathcal{C}_K using graph clustering algorithms (e.g., Infomap / Louvain). Randomize entire clusters as atomic units (Wc∈{0,1}W_c \in \{0, 1\}), bounding 95%+ of interference within cluster borders.
- Switchback Testing (Time-Slice Randomization):
For physical supply-demand platforms (Uber, DoorDash). Partition time into discrete windows (e.g., 30-minute intervals). The entire city alternates between Treatment and Control: [T1=Control,T2=Treatment,T3=Control,… ][T_1 = \text{Control}, T_2 = \text{Treatment}, T_3 = \text{Control}, \dots]. Completely eliminates user-to-user supply cannibalization.

🏭 Production Trade-offs

In-Depth Analysis & Engineering Trade-offs: ① Switchback testing trade-offs—switchbacks eliminate marketplace cannibalization, but suffer from carryover effects: a surge in ride requests generated during Treatment period T2T_2 depletes driver supply at the start of Control period T3T_3; introducing 5-minute washout / buffer windows between switchback intervals mitigates carryover. ② Cluster randomization variance penalty—randomizing by clusters reduces sample size from NN users to KK clusters (e.g., 50 metropolitan regions); statistical variance increases dramatically, demanding larger effect sizes or multi-month experiment durations. ③ Synthetic Control Methods (SCM)—when testing platform-wide algorithmic changes across entire cities or countries (where only 1 or 2 markets receive treatment), synthetic controls construct a weighted combination of un-treated cities to form an artificial counterfactual baseline. ④ Detecting interference in standard user-split tests—tracking whether control group behavior changes as the treatment traffic allocation ramps from 5% to 50%; if control metrics drift as treatment volume expands, interference is actively contaminating the experiment. ⑤ Budget-split experiments in ad auctions—in ad bidding, treatment and control advertisers compete for the same query auctions; partitioning ad campaign budgets rather than user impressions isolates auction dynamics. ⑥ Interview takeaway—define SUTVA formally, explain supply cannibalization and social spillover, detail Cluster Randomization (for social graphs) and Switchback Testing (for two-sided physical marketplaces), and describe washout windows.
⚠️ Common Interview Pitfalls
  • ✕
    Running standard user-split A/B tests in two-sided marketplaces (ride-sharing, food delivery, hotel booking), severely inflating treatment effects via supply cannibalization.
  • ✕
    Deploying switchback testing without washout buffer intervals, allowing carryover supply depletion to poison adjacent control periods.
  • ✕
    Ignoring social graph network spillover, concluding a viral feature failed because control group friends also increased their activity.
🎯 Interviewer Follow-up Inquiries
  • ?
    How do washout periods (buffer intervals) in switchback experiments prevent temporal carryover contamination between treatments?
  • ?
    What graph partitioning objectives (such as minimizing normalized edge cuts) optimize cluster randomization in social networks?
  • ?
    How does Synthetic Control Method (SCM) construct counterfactual baselines when experimenting on city-scale units?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM7-095: Explain the Impact of Position Bias on Online Experimentation and Metric Confounding📋 Back to BankNext →M7-097: Explain Methodologies for Measuring Long-Term Algorithmic Effects in Digital Platforms