> ## Documentation Index
> Fetch the complete documentation index at: https://help.get-ryze.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# A/B testing

> How to run valid A/B tests on an ecommerce store: sample size, significance, what to measure, and the mistakes that produce false conclusions.

An A/B test splits traffic between the current version and a variant, then measures which performs better. It is the only way to know whether a change helped rather than assuming it did.

## Before you test

**Do you have enough traffic?** This is the question most stores skip.

A test needs enough conversions to distinguish a real effect from noise. Detecting a small improvement takes thousands of conversions per variant. A store with 50 orders a month cannot power a test to detect a 5% lift — it would take years.

**If you cannot power a test, do not run one.** Fix obvious friction directly instead. A false result is worse than no result, because you will act on it.

## Running a valid test

**Change one thing.** If you change layout, copy, and imagery together, you learn that the bundle worked, not what in it did.

**Decide the metric first**, and prefer **revenue per visitor** over conversion rate. A variant that raises conversion while lowering average order value can reduce revenue.

**Calculate sample size before starting.** Decide how long the test runs before you look at it.

**Run full weeks.** Weekday and weekend behaviour differ. A test running Tuesday to Friday is measuring the days, not the variant.

**Do not stop early because it looks good.** Peeking at results and stopping when significance appears is the most common way to generate false positives. Early results swing wildly.

## Reading the result

**No difference is a real result.** Most tests show no meaningful effect. That is information: the thing you changed does not matter, so stop spending time on it.

**Significance is not certainty.** At 95% confidence, one in twenty results is a false positive. Run enough tests and some winners are noise.

**Small lifts need large samples.** A claimed 2% improvement from 200 conversions is not a finding.

## Common mistakes

* Testing during a sale, holiday, or campaign spike
* Running several overlapping tests on the same pages
* Changing the test mid-flight
* Excluding data that does not suit the conclusion
* Testing trivia — button colours — while checkout has a broken step
* Declaring a winner from a few dozen conversions

## What to test

Prioritise by expected impact:

1. Shipping cost presentation and thresholds
2. Product page information and layout
3. Imagery — count, type, and prominence
4. Add-to-cart placement and mobile layout
5. Review prominence
6. Copy and headlines

## What Ryze does

Ryze runs layout experiments and A/B tests as part of CRO on **Ecom Autopilot**, alongside the AI landing page builder. Tests are reported with the metric and sample they were judged on, not just a winner.

## Related

* [CRO overview](/cro/overview)
* [Benchmarks](/cro/benchmarks)