Homepage A/B Testing: What 7 of Our Own Tests Actually Moved

Homepage A/B testing rarely moves revenue, and when it does, it is because the change helped people who already wanted to buy. Across seven of our own homepage tests, one lifted revenue per visitor: a sticky "Join Now" bar that showed up after the hero. New hero images, a full redesign, city-specific copy and a new subscription section all came in flat or lost.

That is not a reason to skip homepage A/B testing. It is a reason to change what you test there. Below are all seven, losses included, with the orders behind each one.

The seven tests at a glance

What we changed	Revenue per visitor	Orders per variation	Verdict
Sticky CTA bar after the hero	+10.5% (76% probability to beat)	199 vs 177	Won, shipped
Three new hero visuals	-8.7% to -27.3%	103 to 236	Lost
Full homepage redesign	-20.1% (29%)	22 vs 28	Lost, small sample
City-specific homepage copy	-1.1%, +8.3%, -0.2%	136 to 175	Flat
Redesigned subscription section	-5.6% to +0.8%	721 to 782	Not a needle mover
Brand-story hero section	-17.0% (34%)	20 vs 19	Inconclusive
Product quiz promoted on the homepage	+0.3% (52%)	about 2,100 each	Flat

The one that won: a sticky CTA after the hero

A subscription brand had a long homepage with one main button, up in the hero. Our hypothesis: people get convinced somewhere mid-scroll, then have to travel all the way back up to act. Some of them never do.

So we added a slim bar fixed to the bottom of the screen. It only appeared once the visitor scrolled past the hero, with a "Join Now" button and a line of reassurance next to it: skip or cancel anytime.

Over 26 days, revenue per visitor rose 10.5% and conversion rate rose 11.8%, on 199 orders against 177. Order value stayed flat. Mobile and desktop both came out positive.

Now the honest part. That is a 76% probability to beat control, well under the 95% line. The brand shipped it on direction, and we agreed: it is cheap to keep, it cannot break anything, and both revenue and conversion pointed the same way.

The detail that explains it: only 1.4% of visitors clicked the bar, but 36% of those who clicked placed an order. It did not persuade anyone. It gave the already-convinced a shortcut.

What lost: changing how the homepage looks

Three new hero visuals, all below control

A home decor brand had a plain before/after slider in the hero. We tested three prettier options against it: a lifestyle photo, an animated collage of customer photos, and a before/after carousel with color swatches. Control plus three variations, about two weeks.

All three lost. The lifestyle photo was worst, -27.3% revenue per visitor, and we stopped it after about 8 days. The collage came in at -14.3%. The carousel, which kept the before/after idea, lost the least at -8.7%, on 217 orders against 236.

The twist: the prettier heroes got more add-to-carts, then lost those people at the cart. Abandonment went from 43% to 53% on the lifestyle photo. The images pulled clicks from people who did not yet understand the product. The more a variation kept the before/after, the less it lost.

A full homepage redesign

A subscription box brand designed a cleaner homepage in-house and asked us to test it against the current one. Cleaner hero, cleaner messaging, less on the page.

It lost: revenue per visitor -20.1%, at 29% probability to beat. But the test ended after 9 days on 22 orders against 28, so treat that as directional at best.

The bigger problem is that even a clear loss would have taught us nothing. A whole-page redesign changes ten things at once. When it loses, you cannot tell which section hurt. The relaunch is planned around one stated hypothesis, the hero and one section only, with a run time sized before launch.

What stayed flat: personalization and new sections

City-specific homepage copy

The sticky CTA brand also had three big city audiences. We showed each city its own homepage copy, speaking to local identity, against the generic version.

Over 40 days and 136 to 175 orders per variation, nothing moved: -1.1%, +8.3% and -0.2% revenue per visitor. The best one sat at 71% probability to beat, which is noise.

The reason was in the order data, not the test. Customers were spread evenly across regions, so no city message ever reached a big enough share of buyers to matter. Check the segment size against your full order base before you build for it.

A redesigned subscription section

A hobby products brand with a monthly subscription box tested three new versions of its homepage subscription section: exclusivity messaging, exclusivity plus past box themes, and this month's theme. Control plus three variations, 721 to 782 orders each in the clean window.

Sitewide it looked flat: +0.8%, -5.6% and -3.7% revenue per visitor, none close to significant. But only 27% to 30% of homepage visitors ever scrolled down to that section. Among the people who did see it, all three variations were about 14% behind control on revenue per visitor, and subscription clicks fell 61% to 78%.

Two lessons. A second-fold section reaches about a quarter of your traffic, so its effect gets diluted to almost nothing sitewide. And a flat sitewide number can hide a clearly negative result on the people who actually saw the change.

A brand-story hero section

A high-ticket store tested a brand-story section near the top of the homepage. It ran about 15 active days and collected 20 orders against 19. Revenue per visitor read -17.0% at 34%, and the custom event tracking views turned out to be broken.

We call this inconclusive, not a loss. On this store, most orders start on product pages, not the homepage, so 3,000 visitors per variation produced almost no orders to read. Brand story did win elsewhere: see our homepage storytelling module case study.

A product quiz promoted on the homepage

An equipment brand added a product-finder quiz, promoted through a homepage banner and a menu link. Over 20 days and about 2,100 orders per variation, revenue per visitor came out +0.3%, at 52%. A coin flip.

The quiz itself held people: in an early read, about 70% of those who started it reached the email step. The problem was getting in: in the first 12 days, 0.32% of visitors started it. The homepage banner drove about three times more quiz opens than the menu link, which says entry placement was the bottleneck, not the quiz.

What we test on homepages now

• Help people act, not admire. The only revenue win was a shortcut to the button. Visual refreshes lost or stayed flat.
• Clarity beats pretty. If the product needs explaining, the hero's job is to explain it.
• One hypothesis per test. Test a redesign section by section, or at least name the one thing it should fix.
• Check reach before you test. If only a quarter of visitors scroll to a section, read the result on the people who saw it, and expect a tiny sitewide effect.
• Size the segment first. Personalize for a group only after checking how much of your revenue it actually is.
• Test where orders start. If most buyers land on product pages, a homepage revenue test will not read. Move the idea to the product page or measure engagement instead.

How sure are we?

Not very, on the exact numbers. Only two of these tests went past 700 orders per variation, and both were flat. The one win sits at 76%, and the redesign loss rests on 50 orders total. None reached the 1,000 orders per variation we now aim for.

The pattern is what we trust. Seven tests, six brands, and the direction held: changes that made the page look different did not move revenue, and the change that made buying easier did. For more of our results, see our Shopify A/B test case studies, a separate homepage sticky CTA case study, and the press logos vs USP bar test. Looking for layout ideas instead of test results? Start with our ecommerce homepage examples.

Homepage A/B testing questions

What should I A/B test first on an ecommerce homepage?

Start with anything that shortens the path to buying, like a sticky CTA that appears after the hero. In our tests that was the only homepage change that lifted revenue per visitor (+10.5%). Visual refreshes of the hero lost.

Does a homepage redesign increase conversion rate?

Not in our test. A cleaner, client-built redesign lost 20.1% revenue per visitor, on a small sample. Even with a clear result, a full redesign cannot tell you which part helped or hurt. Test it one section at a time.

How many orders does a homepage A/B test need?

We do not read any test under about 150 orders per variation, and we aim for 1,000. On a smaller store that can take 6 weeks or more, so size the run before launch.

Why did my homepage test come out flat?

Usually because too few people saw the change. A section below the hero reached only 27% to 30% of visitors in one of our tests. Filter the result to visitors who saw it before calling it flat.

Want homepage tests that actually read?

Our team runs testing programs for 8 and 9-figure Shopify brands, from the hypothesis to the readout. See how we run Shopify A/B testing, or book a free strategy call to talk through your homepage.