Subscription Pricing Test: Why We Did Not Ship a +11% Conversion Winner
[ SUMMARIZE WITH AI ]
[ FREE CRO TEARDOWN ]
Find the 3 biggest revenue leaks on your store.
Every day a conversion leak goes unfixed, you're paying for traffic that doesn't buy. Get a 5-minute Loom through your PDP, cart, and checkout, with mockups of the fixes. No pitch.
Get My TeardownThis subscription pricing test had a clear winner: conversion up 11.2% at 97% probability of beating control, subscription starts up 13.3%. We told the brand not to ship it. The winner gave every subscriber 20% more product for the same price, and the extra sign-ups did not pay for that.
Most test reports stop at "variation won". On subscriptions, that is where the real question starts. A subscription pricing test changes what every future order is worth, so a higher conversion rate can still mean less money.
What We Tested
The brand sells a daily supplement on subscription. Its main pack held 24 servings and shipped every 24 days. We tested three changes against it:
| Version | What changed |
|---|---|
| Control | 24 servings at the regular price, ships every 24 days |
| Variation 1 | 30 servings for the same price, ships every 30 days |
| Variation 2 | 24 servings with a price increase |
| Variation 3 | 24 servings with a second, different price increase |
The test ran 27 days.
Variation 1 Won the Funnel
| Variation 1 vs control | Result |
|---|---|
| Conversion rate | +11.2% (97% probability of beating control) |
| Subscription starts | +13.3% |
| Revenue per visitor | +10.3% |
| Average order value | Flat |
The revenue per visitor gain came from more orders, not bigger ones. On a normal product test, this is a ship.
The price increases did not work. Variation 2 came out flat, trending negative on subscription starts (-4.6%). Variation 3 lost 20.5%.
The Math That Killed the Winner
Revenue per visitor in a test only counts the first order. A subscription keeps billing, and Variation 1 changed every bill after the first.
- Price per serving dropped 20%. 30 servings for the price of 24 is the same product at a 20% discount
- Subscribers get charged less often. A 30-day cycle means fewer charges per year than a 24-day cycle
- To break even, it needed about 25% more subscribers. If each subscriber is worth 20% less, you need 1 / 0.8 = 1.25 times as many to hold revenue flat
- It delivered 13.3% more. A real lift, but about half of what the discount costs
The only way Variation 1 pays is if subscribers on the 30-day cycle stay much longer than those on the 24-day cycle. That takes 5 to 6 months of retention data to know. This was also the third test of 24 versus 30 days for this brand, and the earlier two pointed the same way. So we recommended keeping the 24-day pack.
How to Read a Subscription Pricing Test
- Work out the price per unit of each variation before launch, so you know the break-even lift in advance
- Judge the winner on revenue per subscriber over time, not first-order revenue per visitor
- If a variation gives more product for the same price, treat it as a discount, because it is one
- Track retention by variation for months after the test ends, before you call it
- Do not read a failed price increase as "our customers will not pay more". Read it as "this increase, on this page, did not work"
For subscription tests that did ship, see how to increase subscription rate on Shopify and our subscription frequency price anchoring case study.
Subscription Pricing Test Questions
Can an A/B test winner lose money?
Yes. In this subscription pricing test, a bigger pack at the same price lifted conversion 11.2% at 97% probability of beating control. But it cut the price per serving by 20%, so it needed about 25% more subscribers to break even and delivered 13.3%.
Should I offer more product per subscription for the same price?
Only if the lift in subscribers is bigger than the discount. More servings for the same price is a discount. Work out the break-even lift before the test, not after.
Do subscription price increases work?
Not in this test. One price increase came out flat, trending negative on subscription starts (-4.6%), and the other lost 20.5%.
What metric should I use for a subscription pricing test?
Revenue per subscriber over time. Revenue per visitor in the test window only counts the first order, so it misses what the change does to every later bill.
How long should I track a subscription pricing test?
Run the test until the funnel read is clear, then track retention by variation for 5 to 6 months before deciding on any change that affects billing.
Want this kind of testing on your store? See how our Shopify CRO agency plans, builds and reads tests like this one.