M1-086M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Hard
⚡Flashcard Drill➔Mastery:

Explain Trigger Analysis in Online Experiments and How It Enhances Statistical Power

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-087 →
📐 Mathematical Definition
trigger:subset actually exposed to the treatment\text{trigger}: \text{subset actually exposed to the treatment}
⚡ One-Sentence Core Answer
Trigger analysis evaluates treatment effects exclusively among the sub-population of users who actually encountered the experimental trigger condition in both groups, filtering out un-exposed noise and dramatically boosting statistical power.

📌 Key Takeaways

  • •
    Core Intuition: If a new feature is located deep inside the checkout settings page, 95% of users never visit that page. Pooling all users dilutes the true effect by factor 20.
  • •
    Dual Triggering: Must identify users in BOTH Treatment and Control who satisfied the trigger condition (e.g. Navigated to settings page), preserving randomized balance.
  • •
    Mathematical Consistency: Under the assumption of zero effect on untriggered users, Triggered Effect = Overall EffectP(Triggered)\frac{\text{Overall Effect}}{P(\text{Triggered})}, with standard error reduced by factor ≈P(Triggered)\approx \sqrt{P(\text{Triggered})}.

📐 Mathematical Derivations

Let Ei∈{0,1}E_i \in \{0, 1\} indicate whether user ii triggered the feature surface. By the law of total expectation, overall population treatment effect is: τall=E[Y(1)−Y(0)]=P(E=1)E[Y(1)−Y(0)∣E=1]+P(E=0)E[Y(1)−Y(0)∣E=0]\tau_{\text{all}} = E[Y(1) - Y(0)] = P(E=1) E[Y(1) - Y(0) \mid E=1] + P(E=0) E[Y(1) - Y(0) \mid E=0]. Under the exclusion restriction that untriggered users experience zero effect (E[Y(1)−Y(0)∣E=0]=0E[Y(1) - Y(0) \mid E=0] = 0), we have τall=ptrig⋅τtrig\tau_{\text{all}} = p_{\text{trig}} \cdot \tau_{\text{trig}}. The variance of the naive overall estimator is Var(τ^all)≈2σ2N\text{Var}(\hat{\tau}_{\text{all}}) \approx \frac{2\sigma^2}{N}. If we evaluate only triggered users (Ntrig=NptrigN_{\text{trig}} = N p_{\text{trig}}), the variance of τ^trig\hat{\tau}_{\text{trig}} is 2σ2Nptrig\frac{2\sigma^2}{N p_{\text{trig}}}. Transforming back to population scale: Var(ptrigτ^trig)=ptrig2(2σ2Nptrig)=ptrig(2σ2N)=ptrigVar(τ^all)\text{Var}(p_{\text{trig}} \hat{\tau}_{\text{trig}}) = p_{\text{trig}}^2 \left(\frac{2\sigma^2}{N p_{\text{trig}}}\right) = p_{\text{trig}} \left(\frac{2\sigma^2}{N}\right) = p_{\text{trig}} \text{Var}(\hat{\tau}_{\text{all}}), reducing variance by exactly factor ptrigp_{\text{trig}}.

🏭 Production Trade-offs

Triggering implementation protocol: Logging the trigger event must be executed counterfactually in the control group. For example, if Treatment adds a modal popup upon clicking a button, Control must log a hidden event 'would_have_triggered_modal' at that exact button click without showing the popup. If triggering is only logged when the Treatment UI renders, Control has no trigger events, making direct comparison impossible without catastrophic selection bias.
⚠️ Common Interview Pitfalls
  • ✕
    Conditioning on a post-treatment trigger that is itself influenced by treatment (e.g. A trigger that Treatment makes easier to reach), which introduces severe selection bias and invalidates randomization.
  • ✕
    Failing to implement virtual / counterfactual trigger logging in the Control group.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does counterfactual trigger logging in Control preserve exact internal validity?
  • ?
    How does Trigger Analysis relate to Complier Average Causal Effect (CACE) in Instrumental Variable analysis?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-085: Explain Why Individual Randomization Fails Under Network Spillover/Interference and How Cluster Randomization Solves It📋 Back to BankNext →M1-087: Explain Methodologies for Measuring Long-Term Treatment Effects Beyond Short-Term Metrics