M1-055M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Hard
⚡Flashcard Drill➔Mastery:

Analyze the Statistical Diagnostics When CTR Increases but CVR Remains Flat in an Online Experiment

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-056 →
📐 Mathematical Definition
中介效应: CTR→detail view→CVR\text{中介效应}:\ \text{CTR}\to\text{detail view}\to\text{CVR}
⚡ One-Sentence Core Answer
A CTR increase alongside flat CVR typically indicates user intent dilution (traffic expansion pulling in lower-intent marginal users) or funnel cannibalization; diagnosed via funnel decomposition and quantile treatment analysis.

📌 Key Takeaways

  • •
    Funnel Decomposition: Overall Conversions = Impressions×CTR×CVR\text{Impressions} \times \text{CTR} \times \text{CVR}. If CTR rises and CVR is constant, Total Conversions must strictly increase!
  • •
    Traffic Dilution / Selection Bias: Treatment attracts marginal users with lower baseline intent, dragging down the average CVR of the post-click cohort.
  • •
    Diagnostic Tools: Segmented CVR by user tenure/intent, Simpson's Paradox decomposition, and Intent-to-Treat (ITT) evaluation on Total Revenue per Impression.

📐 Mathematical Derivations

Let total impressions be II. Expected clicks C=I⋅CTRC = I \cdot \text{CTR}, and expected conversions K=C⋅CVR=I⋅CTR⋅CVRK = C \cdot \text{CVR} = I \cdot \text{CTR} \cdot \text{CVR}. Taking relative lift derivatives via Taylor expansion: ΔKK≈ΔCTRCTR+ΔCVRCVR\frac{\Delta K}{K} \approx \frac{\Delta \text{CTR}}{\text{CTR}} + \frac{\Delta \text{CVR}}{\text{CVR}}. If ΔCTRCTR=+10%\frac{\Delta \text{CTR}}{\text{CTR}} = +10\% and ΔCVRCVR=0%\frac{\Delta \text{CVR}}{\text{CVR}} = 0\%, then net conversions ΔK/K=+10%\Delta K / K = +10\%! The apparent 'concern' that CVR didn't move is often a cognitive fallacy: maintaining identical conversion rate over a 10% larger, more diluted user pool represents a massive business victory.

🏭 Production Trade-offs

When CVR actually drops: Let users comprise high-intent (HH) and marginal (MM) cohorts. If Treatment UI makes click bait enticing, it adds 10001000 marginal clicks with 1%1\% CVR while baseline high-intent clicks have 10%10\% CVR. Overall CVR drops due to composition shift (Simpson's Paradox). The definitive industrial guardrail is evaluating End-to-End Conversion Rate per Impression (K/IK/I), bypassing intermediate funnel conditioning bias.
⚠️ Common Interview Pitfalls
  • ✕
    Conditioning on post-treatment clicks (evaluating CVR only among clickers), which introduces collider stratification bias.
  • ✕
    Misinterpreting a flat CVR as a feature failure when top-of-funnel conversion volume expanded significantly.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does evaluating post-click CVR introduce collider bias in causal DAGs?
  • ?
    How can Survival Analysis (Hazard Rates) model multi-step user purchase conversion delays?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-054: Explain CUPED (Controlled-experiment Using Pre-Experiment Data) and Why It Achieves Massive Variance Reduction📋 Back to BankNext →M1-056: Define Causal Effects and the Potential Outcomes Framework (Rubin Causal Model)