A/B testing
Show different versions of one bundle to different shoppers, compare the results, and keep the winner.
An A/B test splits your visitors between up to three versions of the same bundle — A, B and C — and tracks how each one performs. When you have enough data, you end the test and the version you pick becomes your bundle.
What a version can change
Each version has its own:
- Block title
- Offers — tiers, discounts, badges, labels, and the free gifts, upsells and highlights on each tier
- Design — colors, typography, spacing and sizing
Everything else is shared by all versions, so changing it changes it for the whole test: the bundle name, Visibility and Markets, Layout, Advanced Settings, swatches, Custom CSS, and the Timer Bar, Live Social Proof, Scratch Deal, Sticky add to cart, Progressive Gifts and Subscriptions cards.
Start a test
Save your bundle. The version buttons only appear once the bundle has been saved.
In the Preview card on the right of the editor, click B variant. Your current setup becomes A variant, and B variant is created as a copy of it. This also saves the bundle.
Change what you want to test in B variant — its Block title, offers or Design.
Click Save.
Click A variant or B variant at any time to choose which version you are editing. Save always saves the version you are on.
To add a third version, click Add variant. Three is the maximum; the button disappears once you have them.
Save every version after editing it
Saving also sends each version's offers to checkout, so its discounts apply to the shoppers who see it.
Set the traffic split
Traffic is split evenly by default. To change it:
Click Settings in the Preview card. It appears once the test has started.
In AB Test Settings, choose Even allocation or Custom allocation.
For a custom split, drag the slider to set the share for variant A. The remaining traffic is divided evenly between the other versions, and Calculated distribution shows the result.
Click Save.
If you add a version after setting a custom split, open Settings and save the split again. Until you do, traffic is split evenly.
Each visitor is assigned to one version and keeps seeing it on later visits from the same browser. A visitor on a different browser or device may be assigned again.
Read the results
On the app Home page, a bundle running a test shows an A/B test badge with its number of variants. Click the arrow beside it to expand one row per version, with that version's visitors, conversion rate, average order value, revenue and more.
While a test is running, it takes priority over other bundles targeting the same product.
End the test and pick a winner
On Home, click End A/B test in the bundle's row.
Compare the versions in the table: Visitors, Extra Rev., Total rev., Orders, AOV, CR and Rev. per visitor. The version with the highest revenue per visitor is marked Winner and is selected for you.
Read the Statistical Significance banner. It compares A with the best of the other versions on conversion rate. Need more data means one of the two versions being compared has fewer than 100 visitors; Not significant means the difference could be chance; a result such as 95% confident B variant wins means it is very unlikely to be chance.
Click Choose beside the version you want to keep (the buttons read Choose A, Choose B and so on), then click End test.
All traffic then goes to the version you chose, and it becomes your main bundle. The other versions are archived — they stop showing, and you can't switch back to them. If you choose a version that isn't the top performer, the app shows a warning with a shortcut back to the best one.
Give the test time
A few days of traffic on a quiet store can point either way. Wait until each version has plenty of visitors, and change one thing at a time so you know what made the difference.
The block title and offer texts of each version have their own entries on the Translations page, labelled Variant A (Control), Variant B and so on. Texts shared by every version, such as the Timer Bar, are listed once, under Variant A (Control).