A/B Testing
In one line
A/B testing splits users between two versions and measures which performs better — data-driven decision making for features.
In simple words
A/B testing (split testing) is an experiment where you compare two versions of something to see which performs better. Version A is the control (current), version B is the variant (new).
How it works:
- Define a metric — click rate, conversion rate, revenue, etc.
- Create a variant — change a button color, headline, layout, algorithm
- Split traffic — randomly assign users to A (50%) or B (50%)
- Collect data — run until you have statistical significance
- Decide — if B is significantly better, ship it; if not, keep A
Key concepts:
- Statistical significance — the difference is real, not due to chance (usually p < 0.05)
- Sample size — you need enough users to detect a real difference
- Confounding variables — make sure only the change differs between A and B
- Effect size — how big is the difference? (1% improvement might not be worth the change)
A/B testing is widely used in product development, marketing, and UX design. Tools: Optimizely, Google Optimize, LaunchDarkly, GrowthBook.
Example
// A/B test: compare two checkout button colors
const variant = Math.random() < 0.5 ? 'A' : 'B';
// Version A: blue button (control)
// Version B: green button (variant)
if (variant === 'A') {
renderButton({ text: 'Buy Now', color: 'blue' });
} else {
renderButton({ text: 'Buy Now', color: 'green' });
}
// Track conversion
button.onclick = () => {
analytics.track('checkout_click', {
variant: variant, // 'A' or 'B'
timestamp: Date.now(),
});
proceedToCheckout();
};
// After 10,000 visitors:
// Version A: 320/5000 = 6.4% conversion
// Version B: 410/5000 = 8.2% conversion
// p-value = 0.002 → statistically significant!
// Ship version B (green button)Randomly split users between A (blue) and B (green). Track conversions. After enough data, check if the difference is statistically significant. If B is better, ship it.
Common confusions
Confused with: multivariate testing
The difference: A/B testing compares two versions with one change — simple and clear. Multivariate testing changes multiple elements simultaneously and tests all combinations (e.g., 2 headlines × 2 buttons × 2 images = 8 variants). A/B is simpler and needs less traffic; multivariate finds interactions between changes but needs much more traffic.
Related terms
Related Resources
Related Lessons
- Product Testing
A/B testing.
Decode error messages
Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.