A/B Test Design & Analysis

Help me design and analyze this A/B test.

What I'm testing:

• Hypothesis: [specific belief I'm testing - be precise]
• Variant A (control): [current version]
• Variant B (treatment): [new version - what's different]

Context:

• Where: [email/landing page/ad/feature]
• Target audience: [who sees this test]
• Expected traffic: [visitors/users per day or week]

Primary metric:

• [Click rate / Conversion rate / Revenue / Engagement - pick ONE primary]
• Current baseline: [existing metric value if known]
• Minimum detectable effect: [smallest change that matters to you - e.g., 5% lift]

Secondary metrics:

• [Other things to watch that might be affected]

Test design questions:

1. Sample size needed (how many observations for statistical significance)
2. Test duration (how long to run based on traffic)
3. Split ratio (50/50 or different - with reasoning)
4. Exclusion criteria (who should NOT be in test)

Analysis needs:

• Statistical significance threshold: [typically 95%]
• How to interpret results (what conclusion for different outcomes)
• Confounding factors to watch (seasonality, external events)

What I need:

1. Complete test plan (setup, duration, sample size)
2. Statistical validity check (is this properly designed)
3. How to analyze results (what calculations to make)
4. Decision framework (when to ship B, keep A, or keep testing)

Success criteria: I run a statistically valid test that produces actionable, trustworthy insights.