Subscription Buy Box: +18% Conversion From an A/B/n Test

An 18% Conversion Lift on a Subscription Buy Box (From the Variable We Didn't Bet On)

Two variations of a subscription buy box. Both lifted conversion rate +18% over control. The difference between them, the variable we designed the test around, didn't drive the lift. The contents preview did.

What follows is the test, the data, and what it took to avoid attributing the win to the wrong variable.

The Variable We Designed the Test Around Didn't Win

We ran an A/B/n on the homepage subscription buy box of an 8-figure DTC gin subscription brand. Control plus 2 variations:

• Control: a subscription buy box with price and frequency dropdown. No preview of what's in the box. No upgrade option.
• Variation 1: added an "Inside November's Box" contents preview (item names + thumbnails) and a Deluxe upgrade toggle. No per-item RRP values.
• Variation 2 (winner deployed): same contents preview, same Deluxe upgrade toggle, plus per-item RRP (recommended retail price) values shown next to each item.

The variable we designed the test around was the RRP column. Our hypothesis: showing customers that the box contained A$223+ of items for A$99 would unlock conversions and Deluxe upgrades.

Both Variation 1 and Variation 2 lifted conversion rate ~18% over Control with 95%+ probability of beating it. They finished statistically tied. Variation 1 nudged slightly ahead on revenue per visitor (52.0% probability of being best vs Variation 2's 46.7%).

The RRP column added no measurable lift over showing contents alone. The lever was the contents preview itself, added in both variations.

What We Were Actually Testing on the Homepage Buy Box

Page: Homepage (Subscription Buy Box module)
Location: Buy Box
Platform: Intelligems on a Convertibles A/B testing program
Test Type: A/B/n with control + 2 variations
Test Duration: 2 weeks
Sample: ~7,000 per variation (Control 7,080 / V1 7,313 / V2 7,198), 1,147 orders, A$119,679 in test revenue
Traffic Split: even three-way (~33/33/33)

Control

A subscription buy box with the brand's "Ready to Join the Club?" header, price (A$99 per box, total value A$230+), a frequency dropdown ("Deliver every month"), an Add to Cart button, fast-delivery copy, and a refund guarantee. No preview of box contents. No upgrade option.

Variation 1: Contents Preview (No RRP)

Same header and price as Control. Added:

• "Every 1 Month / Every 2 Months" frequency tabs replacing the dropdown.
• An "Inside November's Box" section showing item names with thumbnails:
  • 700ml Collector's Series Triple Juniper Dry Gin
  • Two bottles of Fever-Tree Mediterranean Tonic Water
  • Perfectly paired dehydrated apple and juniper garnish
  • Red Rock Deli Sea Salt & Balsamic Vinegar Crisps
  • Aussie Nut Co. Crème Brûlée snack
• A Deluxe upgrade toggle (+A$30) with benefit bullets ("Extra mixers and garnish sized for the full 700ml bottle / Enough for every serve, from first glass to last / Delivered together, no top-ups needed").

No per-item RRP values shown.

Variation 2: Contents Preview + RRP (Winner Deployed)

Same as Variation 1, with one structural addition: per-item RRP values shown next to each item in the contents preview.

• 700ml Collector's Series Triple Juniper Dry Gin: A$85.00
• Two Fever-Tree Mediterranean Tonic Waters: A$5.00
• Garnish: A$5.00
• Red Rock Deli Crisps: A$2.50

Total declared value: A$223+. Same Deluxe upgrade toggle and benefit bullets as Variation 1.

Why Both Variations Tied: Reading the Result

Metric	Control	V1 (No RRP)	V2 (RRP, deployed)
Visitors	7,080	7,313	7,198
Orders	335	409	403
Conversion Rate	4.73%	5.59% (+18%)	5.60% (+18%)
Revenue per Visitor	A$4.94	A$5.86 (+19%)	A$5.82 (+18%)
AOV	A$104.43	A$104.74	A$103.86
Subscription % of orders	87.8%	87.5%	88.1%
Probability to Beat Control	n/a	95.4%	94.5%
Probability to Be Best	1.31%	52.0%	46.7%

Variation 1 and Variation 2 are statistically tied on every primary metric. Conversion rate differs by 0.006 percentage points (5.599% vs 5.593%). Revenue per visitor differs by A$0.04. Both clear the 95% confidence threshold against Control. The probability-to-be-best split (52% vs 47%) is functionally a coin flip.

The estimated monthly revenue lift at deploy-time was +A$41,233. As more data accumulated, the dashboard's monthly projection settled at +A$40,517 (V1) with V2 a touch behind. Both variations project well above the brand's monthly revenue baseline on this surface.

If we'd run a single-variation test (Control vs Variation 2) without Variation 1 in the mix, we'd have rolled V2 out and credited the RRP column for the +18% lift. Adding the second variation caught the false attribution.

Why the Contents Preview Was the Real Lever

The shift from Control to either variation wasn't "pay A$30 more." It was "subscribe to a mystery box for A$99" becoming "subscribe to a box containing a 700ml gin, two tonic waters, garnishes, and a snack for A$99." That single change reframed the purchase decision.

Three readings from the result:

1. Specificity beat valuation

Customers didn't need to do the math to value the box. They needed to know what was in it. Once "mystery box" became "box of named, recognizable items," the purchase decision got concrete. Adding RRP values on top, in Variation 2, was incremental information rather than a new piece of context.

2. The Deluxe upgrade rate was identical across both variations

The Deluxe upgrade text was identical in V1 and V2. AOV across control and both variations was statistically flat (Control A$104.43, V1 A$104.74, V2 A$103.86). The conversion lift came from net new subscribers, not from upsell capture.

3. Subscription mix held flat

Subscription % of orders: Control 87.8%, V1 87.5%, V2 88.1%. Adding contents and RRP context didn't shift the order mix toward subscription. It increased the absolute number of customers who completed a subscription order.

What This Means If You Run a Subscription Buy Box

Show what's in the box before you justify what it costs

The default DTC subscription playbook leans on value math (RRP totals, total-value-vs-price callouts, savings percentages). This test suggests value math doesn't move the needle if customers can't picture what they're buying. Show contents first. The math is supporting evidence, not the lever.

Add a second variation when you have a specific variable hypothesis

If Variation 1 hadn't existed, we'd have rolled Variation 2 out and built the next test on a false belief about what worked. The "extra" variation was the most informative one. It cost the test some statistical power per variation, and it was worth it.

Statistical ties matter for deployment, not just for analysis

V1 and V2 split the "best" probability roughly 52/47. We deployed V2 because the RRP framing was operationally useful for the brand on adjacent surfaces (paid ads, retention emails). The conversion lift wasn't dependent on the RRP being there.

Subscription Buy Box Test Questions

Why didn't the RRP column add lift?

Because customers in Variation 1 already knew what they were buying. The contents preview did the heavy lifting. RRP was incremental information, not a new piece of context that changed the purchase decision.

Should we still show item RRP values on a buy box?

If your contents preview is doing its job (specific items, recognizable brands, real photos), RRP is a nice-to-have, not a lever. If your value-to-price ratio is genuinely strong, surfacing it can help on retention emails or paid ads where users haven't seen the contents yet. On the buy box itself, lead with what's in the box.

When should you add a second variation to isolate a variable?

Add a second variation when you have a specific variable hypothesis you want to isolate. Here, we wanted to know if the RRP column carried the lift. Variation 1 (without RRP) was the cost of being able to attribute. Without it, we would have shipped a winner with the wrong story attached.

Should the Deluxe upgrade be opt-in or opt-out?

Opt-in (toggle off by default). Customers who wanted the upgrade selected it. AOV held flat across control and both variations, which means upsell rate wasn't the lever. Acquisition was.

What does a "contents preview" need to include?

Item names plus thumbnails (or real product photos), at minimum. Specific enough that customers can picture what they will receive. In this test, item names with thumbnails were enough. The RRP column on top didn't compound the lift.

This test was run using Intelligems as part of a CONVERTIBLES CRO program. See more wins like this in our aggregate case study archive, or book a call for three tailored recommendations on your store's subscription buy box.