A/B test an offer
Compare up to three versions of an offer, set each version's share of buyers and pick a winner.
An A/B test shows different versions of one offer to different buyers and compares what they do. The offer itself is variant A. You can add variant B and variant C.
What you can test
You can test Message, Trust badges, Product upsell, Add-on checkbox, Progress bar, Testimonials, Gift note, Quick question, Upgrade and Bundle offers. Free gift, Edit items and Item protection can't be A/B tested.
Variants can differ in their content and their discount: the text, the products, the discount, the button text. Where the offer shows and who sees it are the same for every variant, and are taken from variant A.
Set up a test
Open the offer. Above the preview, click Setup A/B test. Variant B is created as a copy of the offer and opens for editing. The shares start at an even split: A and B get 50% each.
Click A, B or C above the preview to switch between the variants, and change anything you like: the heading, the products, the discount, the button text. Nothing is saved until you click Save.
Optional: click Add variant for a variant C. The shares then start as A 34%, B 33% and C 33%.
Click Settings, next to the variant buttons, to open A/B test settings. For each variant you can switch it on or off, set its share of buyers and delete it. Variant A (this offer) shows what is left, and Total shows 100%. Click Done.
Click Save. When the offer is Live, the test runs. Each buyer keeps the same variant for the whole checkout, reloads included.
How the shares work:
- Each variant needs 10% to 90% of buyers. Variant A takes whatever the others don't, so the total is always 100% and A keeps at least 10%.
- A variant you switch off keeps its share on paper but gets no buyers; variant A takes them.
- If you delete both variants, the window says "Without variants B and C there is no test: all buyers see variant A." Click Done, then Save.
- On variant B or C, Where it shows and Who sees it are greyed out with the note "Where it shows and who sees it are the same for every variant. Switch to variant A to change them."
- If you change shares, switches or variants in A/B test settings while the test is running, the window warns you: "Changing this mid-test makes results harder to read."
On the Checkout Offers (Beta) page, a tested offer carries a badge such as A/B test · 2 variants. Variants follow variant A: they go Live when A goes Live and stop when A is paused. Deleting the offer deletes its variants and test.
If a variant that is switched on can't show, for example because its discount isn't ready, the whole test is dropped from checkout and every buyer sees variant A. Open each variant, check its Discount status and save again.
Read the results and pick a winner
Once you have saved a test, the A/B test card appears at the top of the offer's page. It shows Running when the offer is Live, or Starts when Live when it isn't, and a line such as "Running · day 3 · we suggest at least 14 days". It then lists each variant's:
- Share of buyers
- Visits: checkouts that were shown that variant
- Orders, with the share of visits
- Extra revenue and Extra revenue per visit: for offers that add products
- Order value per visit
Which number decides the winner depends on the offer. Product upsell, Add-on checkbox, Progress bar, Upgrade and Bundle offers compare the offer's own revenue per visit. The other offers compare the value of whole orders per visit.
Below the table, the card says in plain words how the test stands:
- "Too early to tell: N of 100 visits needed for each version." A verdict needs at least 100 visits for every variant.
- "No revenue yet in either version." (or "any version" with three).
- "No clear winner yet: …, which could still be chance."
- "Version B earns 32% more per visit … B is likely better (97% confident)." A winner needs at least 95% confidence.
Click Refresh to read the results again.
When one variant is clearly ahead, a button appears: Make B the winner (or C), or Keep variant A when A is ahead. End test… is always there.
Click the winner button, or End test…. The End the test window asks Keep which variant?.
Choose Variant A (the other variants are deleted) or another variant (this offer takes that variant's text, products and discount; results so far stay on the offer).
Click End test. The variants are deleted and the offer carries on as one version.
If you have unsaved changes, the buttons are off and the card says "Save or discard your changes first." After a test ends, the card keeps one line about the last test: when it ended, which variant you kept and the verdict at that time.