Skip to main content
An A/B test splits traffic between the current version and a variant, then measures which performs better. It is the only way to know whether a change helped rather than assuming it did.

​
Before you test

Do you have enough traffic? This is the question most stores skip. A test needs enough conversions to distinguish a real effect from noise. Detecting a small improvement takes thousands of conversions per variant. A store with 50 orders a month cannot power a test to detect a 5% lift — it would take years. If you cannot power a test, do not run one. Fix obvious friction directly instead. A false result is worse than no result, because you will act on it.

​
Running a valid test

Change one thing. If you change layout, copy, and imagery together, you learn that the bundle worked, not what in it did. Decide the metric first, and prefer revenue per visitor over conversion rate. A variant that raises conversion while lowering average order value can reduce revenue. Calculate sample size before starting. Decide how long the test runs before you look at it. Run full weeks. Weekday and weekend behaviour differ. A test running Tuesday to Friday is measuring the days, not the variant. Do not stop early because it looks good. Peeking at results and stopping when significance appears is the most common way to generate false positives. Early results swing wildly.

​
Reading the result

No difference is a real result. Most tests show no meaningful effect. That is information: the thing you changed does not matter, so stop spending time on it. Significance is not certainty. At 95% confidence, one in twenty results is a false positive. Run enough tests and some winners are noise. Small lifts need large samples. A claimed 2% improvement from 200 conversions is not a finding.

​
Common mistakes

  • Testing during a sale, holiday, or campaign spike
  • Running several overlapping tests on the same pages
  • Changing the test mid-flight
  • Excluding data that does not suit the conclusion
  • Testing trivia — button colours — while checkout has a broken step
  • Declaring a winner from a few dozen conversions

​
What to test

Prioritise by expected impact:
  1. Shipping cost presentation and thresholds
  2. Product page information and layout
  3. Imagery — count, type, and prominence
  4. Add-to-cart placement and mobile layout
  5. Review prominence
  6. Copy and headlines

​
What Ryze does

Ryze runs layout experiments and A/B tests as part of CRO on Ecom Autopilot, alongside the AI landing page builder. Tests are reported with the metric and sample they were judged on, not just a winner.