Glossary · Glossary

A/B testing: decide with data, not taste

Two versions, an audience split at random, and the data choosing for you.

EnglishItaliano
The short answer

An A/B test compares two versions of a page, an email or an ad by showing them to randomly split groups of people, to see which gets more conversions. You change one thing at a time, such as the headline or the button. The test runs until the difference is statistically reliable, not until one version seems to be ahead.

How to set one up

Start with a written hypothesis: “if the headline states the result instead of the service, more people will fill in the form”. Choose the metric that decides, usually conversion rate. Split traffic in half at random and let it run.

One variable per test. If you change headline, image and button together, even when you win you will not know why.

When a result can be trusted

A difference can appear by chance, especially with little data. Statistical significance tells you how unlikely a gap this size would be if both versions actually performed the same. The usual threshold is 95%, which means a p-value below 0.05.

Is your test result reliable?

Everything is calculated in your browser: nothing is saved or sent.

The calculator uses a two-proportion z-test, the standard method for comparing two conversion rates.

Mistakes that skew the test

Stopping as soon as one version pulls ahead: if you check the data every day, sooner or later a “significant” difference shows up by chance. Decide in advance how many visitors you need.

Running for less than a full week. Monday and Saturday bring different audiences, and a three-day test mixes them up.

Testing details on low traffic. With few visits a month, test the big things: the offer, the headline, the page structure. Button colour can wait. More in our guide to landing page anatomy.

Frequently asked questions

How long should an A/B test run?

At least one or two full weeks, and in any case until it reaches the number of visitors decided at the start. Duration depends on traffic and on the size of difference you want to detect.

What is the difference between A/B and multivariate testing?

An A/B test compares two versions that differ in one thing. A multivariate test tries several elements and their combinations at once, but needs much more traffic.

Can I run an A/B test on Meta campaigns?

Yes. Meta has a dedicated tool that splits the audience between versions, so the same people do not see both ads.

Related terms

Full glossary →

Want to know which numbers matter in your business?

Seven short questions. The founders reply within 48 hours.

Apply