Beginner’s Guide to A/B Split Testing

“Help! I’ve been doing online marketing for some time now, but I still don’t fully understand this whole ‘split testing’ thing I keep hearing about. What is it, why is it so important, and how do I do it?”

Split Testing, also known as A/B testing, is an experiment in marketing where you ‘split’ your audience into two or more groups to test variations of a campaign or website page. Your goal is to find out which variation works better and thus will earn you more money.

You can also split test things unrelated to selling online, such as which blogpost headline pulls in the most views, which subject line gets the most opens, and so forth.

Essentially, split Testing helps you to solve problems and improve conversions.

Here are just a few reasons why you might split test:

To Remove Guesswork

Anyone can follow a gut feeling on what will work best. But if you use hard data, then you know for sure which variables are working better than others.

To Improve Visitor Experience

When people come to your website, it’s to do something: Maybe it’s to find information, solve a problem, or simply to browse. But if they encounter confusing copy, hard to find buttons, and so forth, it’s hard for them to achieve their goals, and they have a terrible user experience.

This negatively impacts your conversion rates - always a bad thing for you and your visitors.

But by using data gathered through visitor behavior analysis tools such as Google Analytics, heatmaps, and surveys, you can solve these problems.

For example, if people are having a hard time finding the “buy now” button, you can test placing the button in different places, making it bigger and even changing the color and the copy before the button and on the button itself.

To Reduce Bounce Rates

Bounce rate represents the percentage of visitors who enter your site and then leave, rather than continuing to visit more pages on your website.

“Sticky” websites that hold visitors for a more extended time are better for SEO and traffic conversion. A high bounce rate may indicate that your landing page (the site’s entrance page) isn’t relevant to your visitors.

For example, if people come to your site expecting to find a cure for acne, and instead they find an article describing why acne occurs, they’re likely to go to another site to find the answer they’re looking for.

With split testing, you can test multiple variations of elements on your site until you find the optimal version that improves user experience and reduces bounce rates.

To Get Better ROI from Existing Traffic

The cost of acquiring quality traffic can be your highest expense. Split Testing allows you to make the most of the traffic you receive, increasing your conversions without buying more traffic.

For example, if right now you are making one sale from 100 visitors, and with A/B testing, you can increase that to 2 sales or even 3 sales per 100 visitors, then you have just doubled or tripled your profits without having to increase your traffic.

And sometimes, even a minor change can create a significant shift in your conversion rates. The problem is, you’ll never know this unless you test.

To Make Minor Modifications to Proven Pages

Let’s say you want to make some incremental changes to a webpage that is already performing well. If you don’t test the changes, you are jeopardizing your current conversion rate.

For example, if you change your product description, it could improve sales, or it could have the opposite effect. By testing the new copy against the old, or by testing two versions of the original text, you can learn which version your prospects prefer.

To Lower Cart Abandonment

Half or more of customers who place items in their shopping cart leave the website without checking out.

By testing things like urgency, scarcity, product photos, and descriptions, the copy on your checkout page, and so forth, you can lower your abandonment rate.

How Can You Split Test?

Running an A/B test is merely creating two different versions of one piece of content, with changes made to a single variable. You’ll show these two pieces of content to very similar audiences and analyze which one performed better.

In A/B testing, always test one thing at a time. If you change your headline, your call to action, and your button color all in the same test, you won’t know which of these performed better. It could be that headline A, call to action B, and button color A are the winners, but you’ll have no way of knowing those are the three items you should use.

Let’s say you want to test your “Buy Now” button. I know this sounds like a trivial test, but even something as simple as button size, color, location, and the copy can sometimes make a significant difference in your conversion rate.

Here’s how to do it:

• Create another web page identical to the first page, except for this one change.
• Test these two versions by showing each of them to half of your visitors, with the visitors originating from the same place.
• The one that converts higher is your winner.

A/B testing can be a continuous process.

Let’s say you’re testing the location of testimonials on your sales letter. Currently, you have the testimonials in the middle of the sales copy. You check to place the testimonials at the bottom of the page, just before the buy now button. This new location tests higher.

Now you test placing the testimonials at the top of the page against having them at the bottom. Having them at the top tests higher, so this is where you place them.

But you’re not done yet.

Next, you test splitting the testimonials, with half at the top of the page and the other half before the buy button. This option works even better than having them all at the top.

And so it goes. You could test placing the testimonials in a sidebar on the right side of the page or the left side of the page.

You can test what kind of testimonials to use: Long ones, short ones, video or written, photos or no photos, blue background or grey background, and so forth.

It sounds like a lot of work, but thankfully it’s not that difficult.

There are A/B testing tools you can use, such as Google Analytics’ Experiments, which will let you A/B test up to 10 different versions of a single web page and compare their performance, using a random sample of users.

In our testimonial example above, if you don’t mind making several different versions of your sales page and if you have enough traffic, you can simultaneously test all the locations for the testimonials at once.

Once you have the winner, you can then simultaneously test all the different versions of the testimonials, and so forth.

What Can You A/B Test?

Short answer – just about anything on your pages.

Long answer – I’ve compiled a list of possibilities below to get you started.

Calls to Action

You want your visitors to take some sort of action, right? It might be purchasing, signing up, downloading, sharing, or simply reading. Here are a few things you can test on your calls to action:

• The call-to-action (CTA) text on your buttons
• The locations of your CTA buttons
• The color, size, and shape of your CTA buttons
• Testing multiple CTAs per page against one CTA per page
• Hyperlinks versus buttons or in addition to buttons
• CTA hover states to show the click-ability of buttons

Content

As they say, content is king. Content writing entertains, informs, and entices online audiences to stay longer on your website. It fuels your online business, and testing your content may lead to better engagement and more significant conversions. Here are some content tests that might surprise you:

• Test the tone of your content. Professional or laid back? Speaking to the group or the individual? Comic versus more serious? Reporter versus guru?
• Test non-gated (free to read) content against gated content. The gate could be as simple as sharing their email address or more exclusive such as joining a monthly membership.
• Your “About” info can be crucial to building trust—test placing your “about” content on the home page.
• Content font type and font size
• Scroll down or click to next page for more content
• Short or long content

Copy

Copy pushes people to the desired action, such a reading an article or buying a product. It can set the tone and helps users understand what you’re all about and the options they have on your site. These tests can help you find a copy that resonates with your audience.

• Test headlines – both the content of the headline and the headline length
• Test taglines
• Test product descriptions
• Test using paragraphs versus bulleted lists
• Test using many sub-headlines throughout your text, versus just one headline at the top followed by all content
• Test using positive versus negative headlines and framing
• Test usage and placement of endorsements and testimonials

Visual Media

This is one of the most critical areas to test, yet most marketers never do it. If you have just the right photo, video, or illustration, it can improve engagement and conversions in a dramatic way, often on a subconscious level. Conversely, you can also lose a visitor or even a sale with one wrong visual choice. Here’s what to test:

• Test the image(s) on your landing page extensively. Begin by testing products versus people.
• If users prefer people in the landing page image, test age, gender, close-up versus full body, and so forth.
• If the user prefers a product image, test several different photos, and even a product video or 360-degree product image.
• On your site, test stock images versus your images.
• Test people images versus animal versus object or scenery and so forth.
• For videos, test auto-play versus click-to-play.
• If click-to-play wins, test the static image that appears before play.
• Test different voice overs for videos.
• Test humorous images versus serious images.
• Test adding different image captions.
• Test using one image in each blog post versus several images.
• Test using images on your sales copy.
• Test using larger fonts, different colors for your website, various headers, and different menu bars.
• Test rotating pictures versus a static image on your homepage.

Funnels

If your goal is to get more people from one page to the next – like in a checkout funnel, sign-up flow, or lead nurture – then A/B testing is your best bet. Funnels are rife with low-hanging fruit to test:

• Test removing extraneous distractions – like other product offers, promotions, or shipping information – from each page in the purchase flow. Often a simplified experience can drive more conversions.
• Test the number of pages in your funnel. How does a single page sales letter compare to spreading information across multiple pages?
• Test removing navigation to any pages outside the checkout funnel.
• Or try replacing specific steps within your funnel with modal boxes. For example, try making shipping options a modal box instead of a page.
• Test letting your customers know upfront about an upsell versus not letting them know. For example, if you offer a done-for-you service for one of the steps in your program, this can overcome an objection that they can’t do this and save the sale.
• Test different price points for the main product, the upsells, and the down sells.
• Test packaging two upsells together versus selling them separately.
• Test the copy you place at the top of each upsell page, letting them know their product will be in their inbox soon, and in the meantime, you’ve got another great offer for them. The wording on this can make or break your upsell sales.

Site Navigation

Imagine suddenly finding yourself in a strange land, not knowing exactly where you are or where you should go. That’s what happens every time someone lands on your website for the first time. The navigation menu can help people find what they want or annoy them into leaving posthaste. Here are some ideas on how to make it better:

• Test the order of menu items in your site navigation.
• Test the titles on each button, as well as drop-down menus—a simple change, like Why Use Us to How it Works, may have a significant impact.
• Test the display of your navigation bar. Do site visitors prefer a horizontal or vertical orientation?
• Test a fixed navigation bar that travels down the page as your site visitors scroll.
• Test button colors and sizes, as well as the color and size of the navigation bar.
• Test having a detailed navigation bar versus a simplified version.

A word of caution: Returning visitors may be accustomed to seeing your site in a certain way. If they can’t immediately find what they are looking for, they may or may not do the work to locate it. If possible, run your A/B tests on new traffic only, and once you have a winning design, then roll it out to everyone.

Forms

Forms can make or break a user’s experience, and you could be losing new subscribers or customers for the silliest of reasons. For example, if the data field boxes are the same color or nearly the same color as the background (something I see all the time), it can be challenging to figure out where to type information.

Try these tests on the forms on your site:

• Test the length and even the formatting of sign-up forms. Try removing non-essential sign-up boxes or relocating them to a page further down the funnel.
• Everybody loves free stuff, and they like MORE free stuff even more. Try a special offer, discount, or promotion to increase sign-ups, or also combining an exclusive discount along with several free reports for an entire package of goodies.
• Test reassuring visitors that under no circumstances will you share their information for any purposes, including spam.
• Test adding humorous elements to the sign-up process.
• Try making individual form fields larger as well as a different color from the background. Larger areas feel much more friendly, and they’re more accessible to those with low vision.
• Try asking for different information in your form fields. For example, business email versus email, or work phone versus cellphone.
• Test asking a surprising question, like what their favorite color is. You never know – this could increase response. (Yes, I’ve seen it happen.)

Email Marketing

How do you get your marketing emails opened and clicked? Here are some testable elements that can increase open rates and click-throughs:

• Test subject lines extensively. Ideally (if you have an extensive list), test headlines with a small portion of your list and then send the winner out to the rest of the list.
• Test the length of your email subject lines, as well as testing punctuation and other symbols versus no symbols.
• Test personalized versus un-personalized emails.
• Test different days and times to send emails to find the optimal window.
• Test sending an email daily versus two or three times per week. I can almost guarantee sales will be better with more emails.
• Test your “from” field.
• Test changing the look and design of your emails.
• Test adding images versus no images.
• Test using teasers at the end of each email that lets them know what’s coming in the next email.
• Test hiding Easter eggs inside your emails, such as secret codes or words that unlock some exclusive freebie.
• Test sending out the same emails twice in one day. The first time is to your entire list, and the second time is those who didn’t open the first email.

Pricing and Shipping

Finding just the right price point can increase revenue, sometimes with fewer sales. Use these tests to maximize revenue from your site:

• Test offering a free trial versus the full payment upfront.
• For large ticket items, test a free trial versus several payments versus paying upfront.
• Test raising your price to include shipping, thus being able to say the shipping is free. Place ‘free shipping!’ notice at the top of each page for your test.
• Test having checkboxes auto-selected as default. For example, a customer’s billing address could default to the same as their shipping address.
• For a membership or subscription site, test whether annual billing or monthly billing generates more revenue. Be sure to factor in attrition on the monthly rate to know for a fact which is more profitable.
• Try anchoring customers at a high price before revealing a lower price. This can take the form of, “Normally $200, now just $47!” or, “Competitors charge $200, but you can own it just $47!”

Remember, these are just starting points and not meant to be a comprehensive guide. You’ll find things to test on your site that aren’t on this list.

A Few A/B Testing Tips

Know Your Goal – Your first step in A/B testing is to think about which metrics are important to you and what your goals are. This will help you to set up your tests in the most effective way.

Create a ‘Control” and a ‘Challenger’

Your control is the page as it is now, completely unaltered. Your challenger is an identical page except for the one element you are testing.

Determine Your Sample Size – Decide ahead of time how many visitors you need to complete your test. In the case of testing an email, if your list is big enough, you might send the two variations to smaller samples of your list. Determine the winner and then send the winning email to the rest of the list.

Multiple Versions of One Variable - While you only want to test one variable at a time, it’s fine to check multiple versions of that one variable simultaneously, as long as you don’t mind making all the different versions at once.

Simultaneous Testing - Always test both variations at the same time. If you check one variation at the beginning of the week and the second variation at the end of the week, how do you know that being closer to or further away from Friday payday isn’t making a difference?

The Exception - Conversely, the exception to the previous rule is if you are testing for optimal timing. For example, you want to see if sending emails at 6 am or 6 pm is better.

Use Adequate Sample Size - Let your test run long enough to produce useful data. If your sample size isn’t large enough, then you don’t know for a fact that there is a statistically significant difference.

For example, if your conversion rate is currently 1%, and you only test 100 visitors, the fact that one variable produced one sale and the other variable generated no sales is statistically irrelevant.

However, if you test 10,000 visitors and one variable produces 130 sales, and the other one generates 100 sales, now you’ve got statistically significant results to make a decision.

Take action Based on Your Results – If one variation is statistically superior to the other, you have a winner. Disable the losing version in your A/B testing tool.

If there is no significant statistical difference between the two, then the variable you tested didn’t impact results. Stick with your original version for now and run your next test.

What is Radical A/B Testing?

You’ll recall that we just said to test only one variable at a time so that you know for a fact which one improves your conversions. This is known as Gradual Testing, and it’s hugely effective at optimizing your site.

That said, there is a method called Radical A/B Testing that you should be aware of

Radical Testing is when you test many elements, all at the same time.

You might be wondering how the person running a radical A/B test knows which elements of the old site to keep and which to change.

They don’t.

This is an all or nothing technique, but it can be useful in certain situations, such as these:

You’re Launching a New Page

It could be a sales page or something else. The point is, you haven’t tested anything yet, and you’re looking to find something that starts out converting well. You create two entirely different designs or approaches, and you test to see which one does better.

Once you have your winner, you continue using that page as you begin A/B testing each element on the page to make it even better.

Your Testing Program Has Flatlined

You’ve exhausted your A/B testing variables. You’ve tested your headlines, your colors, your calls to action, your images, and everything else. Your results have reached a plateau.

You could stop testing at this point, or you could take an entirely different approach and shake things up by trying a total re-design.

You Want to See Major Results Fast

Maybe demand is low, or you need a sudden influx of money. If you choose, you can take a gamble on a radical A/B test and see if it pays off.

You’ve Run Out of Ideas

The page you’ve been testing hasn’t been performing as well as you like, despite plenty of A/B testing. Maybe it’s time to develop an entirely new approach and test it against your current page.

You and Your Partner Disagree

If you’re working with someone on your page, you might have vastly different ideas of what should be done. Rather than argue, go to your respective corners and produce your best work. Then do a radical test to see who has the best page and work together to improve that page with A/B testing.

Your Visitors Are Baffled

You’re getting traffic, but it’s not converting. Data analytics confirms that visitors can’t find what they’re looking for. In a case like this, fixing little things might not be enough. Design an entirely new page and test it against the current page.

Your website Doesn’t Get Much Traffic (Yet)

To do A/B tests, you need a high volume of traffic to get accurate test results. If your traffic is still low, Testing will take a lot longer while you wait for appropriate sample sizes. Consider doing a radical test, making several changes at once to begin the optimization process.

Beware of Radical A/B Testing Hazards

Radical Testing is a bit like tossing dice in Vegas. Yes, you might win big, but you can also lose big, too. Just as there is a chance your conversions can improve dramatically in a short time, there is also a chance your re-design won’t work at all, and conversion rates will fall like a rock.

Next Steps

If your test results proved that your new design works better than the old one, it’s time to implement the latest version. But you’re not done yet.

Go back to your A/B testing to adjust and fine-tune your new page for optimal conversions.

Remember, Radical Testing is not a substitute for testing one element at a time. Incremental A/B testing is always going to be your most explicit, cleanest approach to maximizing your website’s conversions.

Now you know more about A/B testing than probably 95% of online marketers. The next time you hear of someone launching a website or funnel only to be sadly disappointed by the results, let them know about split testing.

And if you’re not already performing split tests yourself, I highly recommend it. As you find ways to optimize your websites and sales funnels, you’ll discover it’s a lot like giving yourself raises. You’re doing the same work to bring in customers, but you’ll be earning more with each successful test.