A/B Test Conversion Analysis

A reproducible analysis of a two-group experiment measuring binary conversion. The analysis validates the supplied CSV before calculating conversion rates, confidence intervals, and a two-sided two-proportion z-test.

Business question

Does the variant (group B) convert users at a different rate from the control (group A)?

The supplied CSV does not describe the product change, eligibility rules, randomization process, or the business definition of a conversion. This project therefore treats group A as the control, group B as the variant, and conversion as the observed binary outcome; it does not infer a product-level causal narrative beyond the measured association.

Dataset audit

ab_test_data.csv contains one record per user with four fields: user_id, timestamp, test_group, and conversion.

• 19,998 records and 19,998 unique user IDs
• No missing values or duplicate records/user IDs
• Valid groups: a and b; valid outcomes: 0 and 1
• Observed timestamps: 2023-07-03 01:42:34 to 2023-07-25 01:41:19 (21 days, 23:58:45 elapsed; 22 calendar dates inclusive)

Verified results

Metric	A (control)	B (variant)
Users	10,013	9,985
Conversions	611	889
Conversion rate	6.10%	8.90%
95% Wald CI for conversion rate	5.63% to 6.57%	8.34% to 9.46%

The variant's conversion rate is 2.80 percentage points higher than the control's, equivalent to a 45.91% relative lift. The 95% Wald confidence interval for the absolute lift (B minus A) is 2.07 to 3.53 percentage points.

Methodology

• Unit of analysis: one unique user_id in the supplied file.
• Primary metric: conversion rate = converted users / users assigned to a group.
• Null hypothesis (H0): groups A and B have equal conversion rates.
• Alternative hypothesis (H1): the conversion rates differ.
• Test: two-sided two-proportion z-test using a pooled standard error under H0, evaluated at alpha = 0.05.
• Intervals: two-sided 95% Wald (normal-approximation) intervals. This matches the original project’s interval approach and is stated explicitly for reproducibility.

The test returns z = -7.5197 for A minus B and p = 5.4912e-14. Because the p-value is below 0.05, reject H0: the observed conversion-rate difference is statistically significant.

Business interpretation

Within this dataset, B materially outperforms A on the recorded conversion outcome. If the experiment was correctly randomized, independently assigned, and run without unmeasured implementation issues, the evidence supports considering B for a controlled rollout.

The data cannot verify randomization, exposure, revenue, refunds/cancellations, novelty effects, or long-term retention. Those should be checked before a full rollout; statistical significance alone does not establish a complete business case.

Project structure

ab-test-analysis/
├── ab_test_data.csv                                  # source dataset
├── a_b_testing_statistical_analysis_&_conversion.py # validated, reproducible analysis
├── A_B_Testing_Statistical_Analysis_&_Conversion.ipynb # notebook entry point
├── requirements.txt
└── README.md

Run the analysis

Requires Python 3.10+ for the type-hint syntax used by the script.

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python 'a_b_testing_statistical_analysis_&_conversion.py'

The command prints all verified metrics and saves outputs/conversion_rate_comparison.png. For a dependency-free statistical check without the chart:

python 'a_b_testing_statistical_analysis_&_conversion.py' --no-plot

To use the notebook, install Jupyter in the same environment and open A_B_Testing_Statistical_Analysis_&_Conversion.ipynb. The notebook invokes the same script, so it does not maintain a second copy of the calculations.

Technologies

• Python standard library (csv, datetime, statistics, and math) for validation and inferential calculations
• Matplotlib for the conversion-rate chart
• Jupyter Notebook as an optional presentation layer