M1-086M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Hard
⚡Flashcard Drill➔Mastery:
Explain Trigger Analysis in Online Experiments and How It Enhances Statistical Power
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Trigger analysis evaluates treatment effects exclusively among the sub-population of users who actually encountered the experimental trigger condition in both groups, filtering out un-exposed noise and dramatically boosting statistical power.
📌 Key Takeaways
- •Core Intuition: If a new feature is located deep inside the checkout settings page, 95% of users never visit that page. Pooling all users dilutes the true effect by factor 20.
- •Dual Triggering: Must identify users in BOTH Treatment and Control who satisfied the trigger condition (e.g. Navigated to settings page), preserving randomized balance.
- •Mathematical Consistency: Under the assumption of zero effect on untriggered users, Triggered Effect = , with standard error reduced by factor .
📐 Mathematical Derivations
Let indicate whether user triggered the feature surface. By the law of total expectation, overall population treatment effect is: . Under the exclusion restriction that untriggered users experience zero effect (), we have . The variance of the naive overall estimator is . If we evaluate only triggered users (), the variance of is . Transforming back to population scale: , reducing variance by exactly factor .
🏭 Production Trade-offs
Triggering implementation protocol: Logging the trigger event must be executed counterfactually in the control group. For example, if Treatment adds a modal popup upon clicking a button, Control must log a hidden event 'would_have_triggered_modal' at that exact button click without showing the popup. If triggering is only logged when the Treatment UI renders, Control has no trigger events, making direct comparison impossible without catastrophic selection bias.
⚠️ Common Interview Pitfalls
- ✕Conditioning on a post-treatment trigger that is itself influenced by treatment (e.g. A trigger that Treatment makes easier to reach), which introduces severe selection bias and invalidates randomization.
- ✕Failing to implement virtual / counterfactual trigger logging in the Control group.
🎯 Interviewer Follow-up Inquiries
- ?Why does counterfactual trigger logging in Control preserve exact internal validity?
- ?How does Trigger Analysis relate to Complier Average Causal Effect (CACE) in Instrumental Variable analysis?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚