Before you test
Do you have enough traffic? This is the question most stores skip. A test needs enough conversions to distinguish a real effect from noise. Detecting a small improvement takes thousands of conversions per variant. A store with 50 orders a month cannot power a test to detect a 5% lift — it would take years. If you cannot power a test, do not run one. Fix obvious friction directly instead. A false result is worse than no result, because you will act on it.Running a valid test
Change one thing. If you change layout, copy, and imagery together, you learn that the bundle worked, not what in it did. Decide the metric first, and prefer revenue per visitor over conversion rate. A variant that raises conversion while lowering average order value can reduce revenue. Calculate sample size before starting. Decide how long the test runs before you look at it. Run full weeks. Weekday and weekend behaviour differ. A test running Tuesday to Friday is measuring the days, not the variant. Do not stop early because it looks good. Peeking at results and stopping when significance appears is the most common way to generate false positives. Early results swing wildly.Reading the result
No difference is a real result. Most tests show no meaningful effect. That is information: the thing you changed does not matter, so stop spending time on it. Significance is not certainty. At 95% confidence, one in twenty results is a false positive. Run enough tests and some winners are noise. Small lifts need large samples. A claimed 2% improvement from 200 conversions is not a finding.Common mistakes
- Testing during a sale, holiday, or campaign spike
- Running several overlapping tests on the same pages
- Changing the test mid-flight
- Excluding data that does not suit the conclusion
- Testing trivia — button colours — while checkout has a broken step
- Declaring a winner from a few dozen conversions
What to test
Prioritise by expected impact:- Shipping cost presentation and thresholds
- Product page information and layout
- Imagery — count, type, and prominence
- Add-to-cart placement and mobile layout
- Review prominence
- Copy and headlines
What Ryze does
A/B Testing in the sidebar is part of the Shopify CRO plan ($1,499/month). On other plans the item shows Unlock, and clicking it opens a short page about the plan with a Talk to us button to book a call with the team. See Plans & pricing. Tests are designed, put on your store and measured by the Ryze team. Click Request an experiment and describe what you want to test; the experiment then appears on the A/B Testing page with live results. Each experiment shows:- Page, Metric (visitors who started checkout, added to cart, or purchased), Split, and how long it runs
- Hypothesis — what you wanted to learn, in your words
- Variants — the control and the variant, each with a preview you can open on your store
- Results — visitors and conversions per side, Confidence, and a verdict: Winner, Loser, Inconclusive, or Split is off (the traffic split is uneven, so the numbers can’t be trusted). Until each side has enough visitors, the result reads “too early to call”.