M7-096M7: Retrieval, Ranking & RecSysOnline Metrics & GuardrailsMedium
⚡Flashcard Drill➔Mastery:
Explain Network Interference, Marketplace Cannibalization, and SUTVA Violations in Online Experimentation
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Network effects and shared marketplace resources violate the Stable Unit Treatment Value Assumption (SUTVA); treatment users interfere with control users through social contagion or inventory cannibalization, requiring cluster randomization, switchback tests, or synthetic controls.
📌 Key Takeaways
- •SUTVA definition: Assumes the potential outcome of any experimental unit is independent of the treatment assignments of all other units.
- •Two-sided marketplace interference: In Uber, Airbnb, or Amazon, treatment users booking rides or buying limited stock cannibalizes availability for control users.
- •Social network spillover: In social feeds, treatment users sharing more content directly alters the feed experience of their control group friends.
- •Mitigation architectures: Cluster-based randomization (graph community partitioning), Switchback testing (time-slice randomization), and Synthetic Controls.
📐 Mathematical Derivations
Causal & Experimental Methodology: Interference & SUTVA Violations.
(1) The Stable Unit Treatment Value Assumption (SUTVA):
Let be the global vector of treatment assignments. SUTVA requires:
The potential outcome depends only on user 's own treatment , completely invariant to how any other user is treated.
(2) Mechanisms of SUTVA Violations:
- Marketplace Supply Cannibalization (Indirect Interference):
Suppose Treatment algorithm recommends discounted hotel rooms. Treatment users book 80% of rooms. Control users search for rooms, find them sold out, and bounce (). The estimated treatment effect artificially inflates, falsely attributing control group starvation to treatment brilliance.
- Social Contagion (Direct Graph Spillover):
Treatment user is given a viral sharing feature; they share 10 posts. Friend (assigned to Control) sees these posts in their feed, increasing their engagement (). The estimated lift collapses toward zero.
(3) Methodological Solutions:
- Cluster-Based Graph Randomization:
Partition the user social graph into disjoint communities using graph clustering algorithms (e.g., Infomap / Louvain). Randomize entire clusters as atomic units (), bounding 95%+ of interference within cluster borders.
- Switchback Testing (Time-Slice Randomization):
For physical supply-demand platforms (Uber, DoorDash). Partition time into discrete windows (e.g., 30-minute intervals). The entire city alternates between Treatment and Control: . Completely eliminates user-to-user supply cannibalization.
(1) The Stable Unit Treatment Value Assumption (SUTVA):
Let be the global vector of treatment assignments. SUTVA requires:
The potential outcome depends only on user 's own treatment , completely invariant to how any other user is treated.
(2) Mechanisms of SUTVA Violations:
- Marketplace Supply Cannibalization (Indirect Interference):
Suppose Treatment algorithm recommends discounted hotel rooms. Treatment users book 80% of rooms. Control users search for rooms, find them sold out, and bounce (). The estimated treatment effect artificially inflates, falsely attributing control group starvation to treatment brilliance.
- Social Contagion (Direct Graph Spillover):
Treatment user is given a viral sharing feature; they share 10 posts. Friend (assigned to Control) sees these posts in their feed, increasing their engagement (). The estimated lift collapses toward zero.
(3) Methodological Solutions:
- Cluster-Based Graph Randomization:
Partition the user social graph into disjoint communities using graph clustering algorithms (e.g., Infomap / Louvain). Randomize entire clusters as atomic units (), bounding 95%+ of interference within cluster borders.
- Switchback Testing (Time-Slice Randomization):
For physical supply-demand platforms (Uber, DoorDash). Partition time into discrete windows (e.g., 30-minute intervals). The entire city alternates between Treatment and Control: . Completely eliminates user-to-user supply cannibalization.
🏭 Production Trade-offs
In-Depth Analysis & Engineering Trade-offs: ① Switchback testing trade-offs—switchbacks eliminate marketplace cannibalization, but suffer from carryover effects: a surge in ride requests generated during Treatment period depletes driver supply at the start of Control period ; introducing 5-minute washout / buffer windows between switchback intervals mitigates carryover. ② Cluster randomization variance penalty—randomizing by clusters reduces sample size from users to clusters (e.g., 50 metropolitan regions); statistical variance increases dramatically, demanding larger effect sizes or multi-month experiment durations. ③ Synthetic Control Methods (SCM)—when testing platform-wide algorithmic changes across entire cities or countries (where only 1 or 2 markets receive treatment), synthetic controls construct a weighted combination of un-treated cities to form an artificial counterfactual baseline. ④ Detecting interference in standard user-split tests—tracking whether control group behavior changes as the treatment traffic allocation ramps from 5% to 50%; if control metrics drift as treatment volume expands, interference is actively contaminating the experiment. ⑤ Budget-split experiments in ad auctions—in ad bidding, treatment and control advertisers compete for the same query auctions; partitioning ad campaign budgets rather than user impressions isolates auction dynamics. ⑥ Interview takeaway—define SUTVA formally, explain supply cannibalization and social spillover, detail Cluster Randomization (for social graphs) and Switchback Testing (for two-sided physical marketplaces), and describe washout windows.
⚠️ Common Interview Pitfalls
- ✕Running standard user-split A/B tests in two-sided marketplaces (ride-sharing, food delivery, hotel booking), severely inflating treatment effects via supply cannibalization.
- ✕Deploying switchback testing without washout buffer intervals, allowing carryover supply depletion to poison adjacent control periods.
- ✕Ignoring social graph network spillover, concluding a viral feature failed because control group friends also increased their activity.
🎯 Interviewer Follow-up Inquiries
- ?How do washout periods (buffer intervals) in switchback experiments prevent temporal carryover contamination between treatments?
- ?What graph partitioning objectives (such as minimizing normalized edge cuts) optimize cluster randomization in social networks?
- ?How does Synthetic Control Method (SCM) construct counterfactual baselines when experimenting on city-scale units?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚