Skip to content

A/B Testing

In one line

A/B testing splits users between two versions and measures which performs better — data-driven decision making for features.

In simple words

A/B testing (split testing) is an experiment where you compare two versions of something to see which performs better. Version A is the control (current), version B is the variant (new).

How it works:

  1. Define a metric — click rate, conversion rate, revenue, etc.
  2. Create a variant — change a button color, headline, layout, algorithm
  3. Split traffic — randomly assign users to A (50%) or B (50%)
  4. Collect data — run until you have statistical significance
  5. Decide — if B is significantly better, ship it; if not, keep A

Key concepts:

  • Statistical significance — the difference is real, not due to chance (usually p < 0.05)
  • Sample size — you need enough users to detect a real difference
  • Confounding variables — make sure only the change differs between A and B
  • Effect size — how big is the difference? (1% improvement might not be worth the change)

A/B testing is widely used in product development, marketing, and UX design. Tools: Optimizely, Google Optimize, LaunchDarkly, GrowthBook.

Example

javascript
// A/B test: compare two checkout button colors
const variant = Math.random() < 0.5 ? 'A' : 'B';

// Version A: blue button (control)
// Version B: green button (variant)

if (variant === 'A') {
  renderButton({ text: 'Buy Now', color: 'blue' });
} else {
  renderButton({ text: 'Buy Now', color: 'green' });
}

// Track conversion
button.onclick = () => {
  analytics.track('checkout_click', {
    variant: variant,        // 'A' or 'B'
    timestamp: Date.now(),
  });
  proceedToCheckout();
};

// After 10,000 visitors:
// Version A: 320/5000 = 6.4% conversion
// Version B: 410/5000 = 8.2% conversion
// p-value = 0.002 → statistically significant!
// Ship version B (green button)

Randomly split users between A (blue) and B (green). Track conversions. After enough data, check if the difference is statistically significant. If B is better, ship it.

Common confusions

  • Confused with: multivariate testing

    The difference: A/B testing compares two versions with one change — simple and clear. Multivariate testing changes multiple elements simultaneously and tests all combinations (e.g., 2 headlines × 2 buttons × 2 images = 8 variants). A/B is simpler and needs less traffic; multivariate finds interactions between changes but needs much more traffic.

Related terms

Related Resources

Related Lessons

Decode error messages

Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.

View errors

← Back to glossary