M1-055M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Hard
⚡Flashcard Drill➔Mastery:
Analyze the Statistical Diagnostics When CTR Increases but CVR Remains Flat in an Online Experiment
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
A CTR increase alongside flat CVR typically indicates user intent dilution (traffic expansion pulling in lower-intent marginal users) or funnel cannibalization; diagnosed via funnel decomposition and quantile treatment analysis.
📌 Key Takeaways
- •Funnel Decomposition: Overall Conversions = . If CTR rises and CVR is constant, Total Conversions must strictly increase!
- •Traffic Dilution / Selection Bias: Treatment attracts marginal users with lower baseline intent, dragging down the average CVR of the post-click cohort.
- •Diagnostic Tools: Segmented CVR by user tenure/intent, Simpson's Paradox decomposition, and Intent-to-Treat (ITT) evaluation on Total Revenue per Impression.
📐 Mathematical Derivations
Let total impressions be . Expected clicks , and expected conversions . Taking relative lift derivatives via Taylor expansion: . If and , then net conversions ! The apparent 'concern' that CVR didn't move is often a cognitive fallacy: maintaining identical conversion rate over a 10% larger, more diluted user pool represents a massive business victory.
🏭 Production Trade-offs
When CVR actually drops: Let users comprise high-intent () and marginal () cohorts. If Treatment UI makes click bait enticing, it adds marginal clicks with CVR while baseline high-intent clicks have CVR. Overall CVR drops due to composition shift (Simpson's Paradox). The definitive industrial guardrail is evaluating End-to-End Conversion Rate per Impression (), bypassing intermediate funnel conditioning bias.
⚠️ Common Interview Pitfalls
- ✕Conditioning on post-treatment clicks (evaluating CVR only among clickers), which introduces collider stratification bias.
- ✕Misinterpreting a flat CVR as a feature failure when top-of-funnel conversion volume expanded significantly.
🎯 Interviewer Follow-up Inquiries
- ?Why does evaluating post-click CVR introduce collider bias in causal DAGs?
- ?How can Survival Analysis (Hazard Rates) model multi-step user purchase conversion delays?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚