Testing that ships a winner, not a slide.
Most landing pages are decided by opinion and defended by taste. We settle them with traffic. Prager designs, instruments and runs A/B and multivariate tests on the pages that actually move revenue, and we call a result only when the math does, from one office outside Philadelphia, since 2003.
What is A/B testing, and what is multivariate testing?
A/B testing compares one version of a page against another and sends live traffic to both until one wins. Multivariate testing changes several elements at once and measures every combination to find which pieces, together, perform best. A/B answers which page is better. Multivariate answers which parts are doing the work.
An A/B test is a controlled experiment. You split visitors between a control and one or more variants, hold everything else constant, and let conversion data decide. Because only one thing changes, a win is easy to attribute. The new headline did that, not luck and not the weather.
A multivariate test changes several elements in the same experiment. That could be a headline, a button color, and a hero image. It then serves every combination and measures not just which elements win but how they interact, since a headline that lifts on its own can fall flat next to a particular image. That extra information is the appeal, and the traffic it demands is the catch.
Both are optimization, not decoration. Neither invents demand. They take the visitors you already have and convert more of them, which is why testing usually returns more per dollar than buying additional traffic to a page that leaks.
Control versus variant, one difference between them. A win points at a specific cause, which makes the result easy to trust and easy to roll out.
Several elements at once, every combination measured. It reveals how parts of a page work together, at the cost of far more traffic and time.
How do A/B testing and multivariate testing differ?
A/B testing compares whole versions of a page and needs relatively little traffic. Multivariate testing compares combinations of elements and needs a great deal. The difference is not sophistication, it is arithmetic. Every element and every variation you add multiplies the number of combinations that each need enough visitors to reach significance.
In practice, almost all serious testing is A/B. In Convert.com’s 2026 experiment dataset, straight A/B tests were the overwhelming majority of everything run, personalization a small slice, and multivariate under one percent. The reality is most pages do not have the visitors to finish a multivariate test before the finding goes stale.
| Experiment type | Share |
|---|---|
| A/B tests | 67.6% |
| Personalization | 4.6% |
| Multivariate | 0.9% |
| Other | 26.9% |
Source: Convert.com 2026 experiment dataset. Chart drawn from the table at left.
How combinations explode
This is the whole reason multivariate testing is rare. Set the number of elements and the variations per element, and watch the number of combinations, each of which needs its own traffic to reach significance.
A/B/n testing of a single element is the far-left case. Everything to the right is multivariate, and the traffic bill grows with it.
When is A/B testing preferred, and when is multivariate?
Prefer A/B testing for one change, a redesign, a new offer, or any page under roughly 100,000 monthly unique visitors. Reserve multivariate testing for high traffic pages that already work, where you want to tune the interaction between several elements. The traffic threshold decides more often than the question does.
Choose A/B testing when
- You are testing one idea, a redesign, or a brand new offer.
- The page draws fewer than about 100,000 unique visitors a month.
- You need a clean, attributable answer you can act on quickly.
- You want a result in weeks rather than a quarter.
Choose multivariate testing when
- The page already converts and you are tuning, not fixing.
- Traffic is high enough to finish before the finding goes stale.
- You specifically want to learn how elements interact.
- You can leave the test running long enough to be believed.
Peep Laja, CXL’s founder, makes the cost vivid. A site with 3,000 visits a month and a 7% conversion rate can need years to finish a meaningful multivariate test. If traffic is under about 100,000 monthly uniques, default to A/B testing. Paras Chopra of VWO frames it the same way. Use A/B for large changes, and save multivariate for smaller, multi-element optimization after a page already works.
How much traffic does a test actually need?
Enough that a real difference clears the noise. A page converting at 4% chasing a 20% relative lift needs on the order of 9,600 visitors per variant. Lower your current conversion rate, or hunt a smaller lift, and the requirement climbs fast. Most disappointing tests were underpowered, not wrong.
The uncomfortable truth of testing is that most of it fails on arithmetic, not ideas. A test that has not gathered enough visitors cannot tell a real 10% lift from random daily variation, so it either declares a false winner or, more often, declares nothing and gets switched off as a waste of time. Both outcomes are traffic problems wearing a strategy costume. So the first question is never what to test. It is whether the page has the traffic to finish. Set your numbers below and find out before anyone designs a variant.
Sample size planner
Uses the two-proportion rule of thumb for 80% power at 95% confidence. A planning estimate, not a substitute for a full power calculation on your real variance.
- Total visitors needed 19,200
- At your traffic, about 2.8 weeks
This page has the traffic to run this test in a sensible timeframe.
This is also why hosting and analytics are not separate conversations. If the page is slow, the conversion rate you are trying to improve is depressed before you start, and if the conversion does not fire cleanly into analytics we can trust, the test has nothing to measure. We test on infrastructure we manage for exactly that reason.
How does Prager run an A/B or multivariate test?
Four steps. Research the leak, write one hypothesis, instrument the variants, then run to the sample and ship the winner. The discipline is in the order. We do not design a variant until the data says which page is bleeding and the math says the traffic can prove a fix.
- 1Research the leak
Analytics, session data and funnels locate where visitors actually drop, so we test the page that is costing money rather than the one someone dislikes.
- 2Write one hypothesis
A single, falsifiable statement. Change this, because of that, expecting this effect. If we cannot write it in one sentence, it is not ready to test.
- 3Instrument the variants
Build the variants, verify the conversion fires, confirm the split is clean and the sample target is set before a single visitor is sent.
- 4Run, then ship
Run to the pre-set sample, not to the first day the number looks good. Then ship the winner, document the loss, and feed both into the next hypothesis.
The step everyone skips is the last discipline in step four. Stopping a test early because the variant is ahead on day two. Early leads reverse constantly, and calling a winner before the sample is reached is the single most common way testing programs fool themselves. We run to the number.
What should you test first on a landing page?
The elements closest to the conversion, in this order. The headline and offer, the call to action, the form, then social proof and layout. Of these, the form is usually the fastest win, because most forms ask for more than they need and every extra field costs completions.
VWO’s 2026 report lists the most-tested elements as headlines, CTA buttons, images, forms and page layout, which matches where the money is. But if you want the single highest probability first test, it is almost always the form. Baymard’s checkout research found the average checkout carries around 11 form fields against roughly 8 that are actually needed, and cart abandonment sits near 70%. Fields you do not need are conversions you do not get.
Form conversion by field count
| Fields | Median conversion |
|---|---|
| 3 fields | 23.1% |
| 5 fields | 17.0% |
| 7 fields | 11.4% |
| 10 or more | 6.9% |
Source: Digital Applied, Form Conversion Rate Benchmarks 2026, compiling HubSpot, Unbounce, WordStream, Baymard, Hotjar, FullStory and Contentsquare. Chart drawn from the table at left.
Why hire Prager for A/B and multivariate testing?
Because we build, host and instrument the sites we test, so a test is a change we can actually ship and measure, not a recommendation we hand off. The people reading the data are the people who can change the page, from one office outside Philadelphia, since 2003.
Most testing vendors deliver a slide recommending a change. We make the change, because the developers who build the page are in the same building as the analysts reading the test.
A test is only as trustworthy as the conversion behind it. We implement and verify tracking through our own managed infrastructure, so a win is a real win and not a tagging artifact.
We set the sample before launch and hold to it. No calling a winner on day two, no quietly stopping a test that turned against the idea we liked.
If the page has 400 visits a month, testing is theater. We will say so and point the budget at traffic or at a straight redesign instead.
What wins on the page informs the ad copy, the SEO and the paid search. A tested message is a better message everywhere.
We manage more than $20 million a year in paid search, so for many clients we can direct qualified traffic at a test rather than wait months for it to accumulate.
Where testing pays off
- Pages with enough traffic to finish a test before the finding ages.
- A clear, valuable conversion. A booked job, a qualified lead, a sale.
- A team that will actually ship the winner rather than admire it.
- A real leak to fix, found in the data rather than in a meeting.
Where we will tell you to wait
- If the page has 400 visits a month, no test will reach significance. Fix traffic or the page first.
- If the conversion does not fire cleanly into analytics, there is nothing to measure yet.
- If the site cannot be crawled or the page is fundamentally broken, that is a redesign, not a test.
- No agency can promise a specific lift. Anyone guaranteeing a conversion increase before seeing your data is guessing.
What do you receive?
A prioritized test roadmap, each test written as one hypothesis, the built and instrumented variants, an honest read at the pre-set sample, and the winner shipped to your live site. Losses are documented too, because a well run loss tells you where not to spend next.
Prioritized by expected value and by whether the traffic can actually prove the idea, so the first test is the one most likely to pay.
A single falsifiable statement with a predicted effect, so the result is unambiguous whichever way it goes.
We build the variants and verify the conversion fires before launch, on hosting and analytics we control.
Called at the pre-set sample, with significance stated plainly. No early calls, no moving the goalposts to rescue a favorite.
We deploy the winning variant to your live site rather than hand you a recommendation to implement yourself.
When the variant loses, you get why it lost and what it rules out, which is the input to a better next hypothesis.
A/B and multivariate testing FAQ
Direct answers, written so a person or an answer engine can quote them without hunting through marketing copy.
What is the difference between A/B testing and multivariate testing?
A/B testing compares whole versions of a page, sending live traffic to a control and one or more variants until one wins. Multivariate testing changes several elements at once and measures every combination to learn which elements, and which interactions between them, perform best. A/B answers which page is better and needs relatively little traffic. Multivariate answers which parts are doing the work and needs a great deal more.
How much traffic do I need to run an A/B test?
Enough that a real difference clears normal daily noise. As a rough guide, a page converting at 4% and chasing a 20% relative lift needs on the order of 9,600 visitors per variant for 80% power at 95% confidence. Lower your starting conversion rate or aim for a smaller lift and the requirement climbs quickly. The sample-size planner on this page estimates it for your own numbers.
How much traffic do I need for multivariate testing?
Considerably more than for A/B testing, because every element and variation multiplies the number of combinations, each of which needs its own traffic to reach significance. A useful rule from CXL is that pages under roughly 100,000 monthly unique visitors should default to A/B testing. Multivariate is best reserved for high-traffic pages that already convert, where you are tuning interactions rather than fixing a leak.
Which should I run, A/B or multivariate?
Use A/B testing when you are testing one idea, a redesign, a new offer, or any page under about 100,000 monthly unique visitors, which covers most business pages. Reserve multivariate testing for high-traffic pages that already work and where you specifically want to understand how several elements interact. In practice the traffic threshold decides more often than the question does.
What should I test first on a landing page?
Start with the elements closest to the conversion: the headline and offer, then the call to action, then the form, then social proof and layout. The form is usually the fastest win, because most forms ask for more than they need and every unnecessary field costs completions. Baymard’s research found the average checkout carries around 11 fields against roughly 8 that are needed.
How long should an A/B test run?
Until it reaches the sample size you set before launch, and ideally across at least one or two full business cycles so weekends and weekdays are both represented. The mistake to avoid is stopping early because a variant is ahead on day two. Early leads reverse constantly, and calling a winner before the sample is reached is the most common way testing programs fool themselves.
What percentage of A/B tests actually win?
Across a meta-analysis of 1,001 A/B tests by Analytics-Toolkit, about 33.5% produced a statistically significant winner, with a mean lift near 15.9% among those winners. In other words, roughly two in three tests do not produce a clear win. That is normal and healthy: a program where every test wins is usually not testing anything ambitious, or is calling winners too early.
Can you guarantee a conversion increase?
No, and we would treat any such guarantee as a warning sign. Testing is a method for finding what works, not a promise that a particular idea will. What we commit to is a properly powered test, a clean conversion we have verified, an honest read of the result, and a documented loss when the variant does not win, so the next hypothesis is better informed.
Do I need a lot of traffic before testing is worth it?
You need enough for a test to finish before its finding goes stale. A page with a few hundred visits a month cannot reach significance in any reasonable timeframe, so the honest move there is to grow traffic or improve the page directly rather than test. A page with several thousand visits a month and a clear conversion can usually support a straightforward A/B test.
What tools do you use now that Google Optimize is gone?
Google sunset Optimize in September 2023, so testing moved to dedicated platforms and server-side or edge implementations. We select the tool to fit the site and the test rather than force every client onto one platform, and because we build and host the sites we test, we can implement variants cleanly rather than bolting a third-party script onto a page we do not control.
Does testing replace good design and copywriting?
No. Testing decides between good options; it does not invent them. A test can only choose among the variants you give it, so the quality of the hypotheses sets the ceiling on what testing can find. That is why we research the leak and write a specific hypothesis before designing anything, rather than testing arbitrary changes and hoping one lands.
Sources cited on this page
- Analytics-Toolkit, meta-analysis of 1,001 A/B tests: 33.5% of tests reached statistical significance, with a mean lift near 15.9% and a median near 7.5% among winners, and a mean near 2% across all tests.
- Convert.com, 2026 experiment dataset: 67.6% of completed experiments were A/B tests, 4.6% personalization, and under 1% multivariate.
- VWO, State of A/B Testing Report 2026: most-tested elements are headlines, CTA buttons, images, forms and page layout.
- CXL (Peep Laja) on traffic requirements, and Paras Chopra of VWO, both quoted in CXL’s multivariate testing guidance: default to A/B testing under roughly 100,000 monthly unique visitors; reserve multivariate for high-traffic pages that already convert.
- Baymard Institute, checkout usability research: the average checkout carries around 11 form fields against roughly 8 needed, with documented cart abandonment near 70%.
- Digital Applied, Form Conversion Rate Benchmarks 2026, compiling HubSpot, Unbounce, WordStream, Baymard, Hotjar, FullStory and Contentsquare: median form conversion falls as field count rises.
- Google: Optimize was sunset in September 2023.
Let’s find out whether your pages can be tested
Give us the page and its traffic. We will tell you whether a test can reach significance, what to test first, and what a win would be worth, before anyone designs a variant.
Prager Microsystems, Inc. · 3819 Germantown Pike, Suite A, Collegeville, PA 19426 · Monday to Friday, 8:30 a.m. to 4:30 p.m. Eastern
Related services: Web Design · Paid Search · Search Engine Optimization · Managed Web Hosting · About Prager
