Tab Commerce
Contact Explore apps

A/B test a Shopify discount against a no-discount holdout

Tab Promo runs A/B tests on the Growth plan that split shoppers between two variants: two different campaigns, or one campaign against "No promotion (holdout)", where shoppers do not get that offer. While the test runs, Shopify checkout applies each campaign's discount only to shoppers assigned to its variant, and Tab Promo shows each variant's orders, net sales, refunds, AOV and estimated margin per order side by side. It reports what it observed; it does not run significance tests or pick a winner for you.

Install Tab Promo About Tab Promo

Plan: Growth — $199/month (lower plans can draft a test but not activate it)

See it work

What the test observed.

Observed results — USD

Ended

−$24.63−9.4%

Observed margin delta · Variant A against the baseline

Variant B (baseline) · 50% traffic$261.63

No promotion (holdout)

Variant A · 50% traffic$237.00

Tailoring 10% off

Observed totals for each variant. No significance test, and no winner is picked for you.

A/B test · Session traffic

Tailoring: 10% off or not

ATailoring 10% off50% traffic
BNo promotion (holdout)Baseline · 50% traffic

Run for14 days · Sep 2 – 15

Checkout eligibilityB2B excluded

Example store and numbers. Each side is what its shoppers ordered; deciding what to run next stays with you.

Examples

Offers you can set up today.

Discount against no discount

"15% off orders over $100" vs No promotion (holdout), 50% / 50%, Run for 14 days

Half of the assigned shoppers can get 15% off orders over $100 at checkout; the other half pay full price. You compare orders, net sales and margin per order for each half.

Two offers head to head

"$10 off orders over $75" vs "Free gift over $75", Compare by Margin per order

Each order counts toward the variant its shopper was assigned to, and the results show an observed margin delta against the baseline.

A small holdout

"Buy 2, get 1 free" at 90% vs No promotion (holdout) at 10%

About 1 in 10 assigned shoppers is held back from the offer, so most shoppers keep it while you measure the difference.

Logged-in customers only

Audience: Known customers (segment). "VIP 20% off" vs No promotion (holdout)

Each customer stays in the same variant on every visit. Guests are not part of the test and do not get the tested offer while it runs.

Setup

How to set it up.

  1. Publish the campaigns you want to test. A campaign must be active and live in Shopify, and republished after its latest edit, before it can go into a test.
  2. Create the test. Open A/B Tests, click Create A/B test and enter a Test name.
  3. Choose the Audience. Pick Known customers (segment), All traffic (rollout) or Session traffic (anonymous included). Session traffic assigns visitors wherever the Tab Promo theme app embed is on. All traffic uses Shopify Rollouts and needs the A/B test rollout marker block in each theme.
  4. Set variants, metric and length. For each variant choose a Campaign or No promotion (holdout), set Traffic % so the two add up to 100, and mark the Baseline. Pick Compare by (Margin per order, Net sales, AOV or Orders) and Run for 1 to 90 days, then click Create.
  5. Activate and read the results. Click Activate (for known customers, Start customer sync and then Publish experiment). Observed results update as orders arrive. The test ends on schedule, or earlier with End comparison.

Good to know

Where it stops.

  • Two variants per test and one test at a time, drafts included. Campaigns in a running test are read-only until it ends and normal delivery is restored.
  • Direct-to-consumer only: B2B company buyers are excluded from tested discounts. Campaigns with purchase requirements, such as minimum or maximum quantity rules, cannot be tested.
  • While a test runs, the tested discount reaches only assigned shoppers. Shoppers without an assignment, such as guests in a Known customers test, do not get it. Run Session traffic tests with the Tab Promo theme app embed; the current SDK release should not be used on a storefront running one.
  • No winner and no statistical significance. Results are observed totals per variant, there is no conversion rate because visitors are not counted, and ending a test returns campaigns to normal delivery rather than rolling out a variant.
  • Estimated margin per order needs a Cost per item in Shopify for the products in every order of a variant.

Plan: Growth — $199/month (lower plans can draft a test but not activate it). Compare plans

Before you launch Storefront SDK Publish, version, and roll back

FAQ

Common questions.

Can I test a discount against no discount at all?

Yes. Choose No promotion (holdout) as one variant. Shoppers assigned to it do not get the tested campaign, so you can compare what the promotion's orders looked like against orders without it.

How are shoppers split between variants?

By the Audience you choose. Known customers are placed by a fixed split of their customer ID, so they stay in one variant. Session traffic assigns each visitor by your Traffic % when the Tab Promo theme app embed loads. All traffic follows the theme Shopify Rollouts shows each visitor.

Does Tab Promo tell me which variant won?

No. It shows observed results side by side, with a delta against the baseline, and leaves the decision to you. It does not calculate significance or switch campaigns on or off when a test ends.

Can I run an A/B test on the Pro plan?

You can build a draft test on any plan, but activating it needs Growth ($199/month).

Can I A/B test quantity rules or B2B pricing?

No. Campaigns with purchase requirements cannot be added to a test, and B2B company buyers are excluded from tested discounts.

What happens when the test ends?

Tested campaigns go back to normal delivery and keep their status. Refunds and late order updates keep counting for 72 hours, and the next test can start once that window closes.

More with Tab Promo

Related guides.