eCommerce marketers know that they’re supposed to test their pages to find the optimal content for their target audience. However, many of us waste enormous amounts of time running tests that never seem to produce noticeable improvements. What are we doing wrong?
Tell me if the following conversation sounds somewhat familiar to you:
Marketer: “Boss, I’ve got the latest wireframes of our new shopping cart page for you to review. We went with the green colour scheme this time because it matches our logo.”
You: “Green? Hmmm I don’t like the green, the buttons don’t stand out on the page enough.”
Marketer: “Well we can try orange but it might break the continuity. I know! Why don’t we try a split test and prove which one is better?”
You: “Sure, if that’s what you think is best.”
7 days later…
Marketer (smugly): “Look! The green version showed a 10% improvement! I was right! We should use green for all our pages.”
You (begrudgingly): “Ok fine, I guess we’ll go with the green.”
These kind of conversations happen all the time in marketing offices all around the world. It’s an example of the single most common testing strategy – I call it the “I was right!” approach. We use it to settle debates about content and prove to everyone else that we’re the best marketer around. It’s scientific evidence that you guys should listen to me more, right? Wrong.
Every marketer knows that they’re supposed to rely on data over gut feeling. The ability to analyse data and compare actionable ideas is one of the efficiencies that draws marketers to the digital sphere.
Unfortunately, most of us aren’t sure what a real testing process actually looks like, or really where to start with creating a rigorous testing regime that generates noticeable results for a page.
The ‘I was right’ approach is pseudo-scientific. In truth, the test above tells us little more about theme colours than ‘the green page got more clicks that one time’. It doesn’t tell us what the best colour will be going forward, and it definitely doesn’t tell us whether page colouration has any actual effect on our bottom line.
So how do we go about constructing a split testing strategy that helps us get more sales? It all comes down to the scientific method. I hope you remember your Year 5 science class!
Phase one: Observation
Perhaps the most important factor in a successful split test is the question of what the test is predicated upon. The test above was founded on a subjective preference for green over orange, or vice versa. The problem here is that we’ve absolutely no indication that either green or orange, or page colour in general, affects the final result. Without an observable link between the target variable and the result, our tests offer us no more insight than random variance.
The solution is to found our tests based on the observable behaviour of our existing pages.
Try gathering together some screenshots of your landing pages and organise them in order of performance. What do the top performers have in common? Are there any noticeable differences between the best and worst performers?
Everything from the design, colour and layout to headline style, copy length, content themes, call to action text, pricing, testimonials and videos/images could potentially affect your results.
Make a note of everything that you think could be a differentiating factor for your pages in order of importance.
Phase two: Hypothesis
The next step is to build a concrete hypothesis around our observations. We already know the question we’re trying to solve; why do some of our pages perform better than others? Based on our observations, let’s form a series of statements that we believe may be the answer to that question. For example:
– Adding urgent sounding phrases to the CTA encourages people to click more on that button
– Bigger headlines mean more impact and leads to more sales
– Orange colour schemes are more inviting than red ones and encourage more engagement
We’ll be testing one of these hypotheses at a time in order to assign true or false statements to them.
Phase three: Predictions
This is a relatively simple, but crucial part of the process. The predictions we make about the outcome of a test help us to confirm the cause and effect relationship between our variable and the outcome.
The goal with our predictions is to account for the differences in goal conversions between the high performing and low performing pages we analysed in phase one. Keep your predictions unambiguous with specific number values attached. If your best performing pages are getting 15% more clicks than your lower performing pages, you can base your predictions on this data.
For example:
– The call to action “Buy now before we’re sold out” will encourage 15% more clicks than just “Buy now”
Phase four: Testing
Set up your split test using a reliable split testing tool. If you don’t have one already, check out this list of great split testing tools and their features.
If you’re using a tool for the first time, it’s worth running an A/A test to make sure that your tests are functioning correctly. You should see the same number of goal completions for each segment of an A/A test.
If you’re confident that you’ve set up your test correctly, it’s time to run your A/B test. Remember that in order to perform a meaningful test we need to isolate the variable we want to analyse. This means that all other variables on the page must remain identical throughout the test. If more than one variable changes from page A to page B, then it’s impossible for us to know for certain which variable is affecting the final outcome. We’ll then be unable to use the results to prove or disprove our hypothesis.
Phase five: Data analysis
After two to four weeks of testing you should have enough traffic to build a workable analysis of your results. Taking a look at your data, you should find that one variation has more goal completions than the other.
Great! That means that the one with more completions is the winner right?
Not so fast. Even though we have an apparent winner, we have yet to prove that the probability of achieving the result with a null hypothesis is significantly less than the probability of achieving it with our valid hypothesis. In other words, we need to show that the result is statistically significant and not just random variation.
Your split testing tool may help you to do this, but if not you can use a free online tool here. We’re looking for a statistical significance rate of above 95% to show that our test is valid. Anything below this means that our results are not more significant than random variation.
Phase six: Evaluation and further testing
Now we’ve proven that our result is significant we have evidence to support our hypothesis. At this stage we can justify rolling out our winning variation to the page. However, there’s still more questions to ask about our results before we can start to form a theory around our hypothesis.
Do the results match the predictions you made in phase three? If not then we still have to account for the difference in page success. Go back to your list of hypotheses and choose one for further testing.
Do you see the increase you expect when rolling out the page to your full segment? If not, don’t panic! It doesn’t mean that your hypothesis is false. Remember that the audience segment for each page is selected randomly by your testing tool. It could be that your first test had badly weighted segmentation. Try retesting to see if you get the similar results. If you’re seeing a broad range of results when retesting, it could that you need to segment your audience.
Not all visitors to your page are equal, and some demographics might be closer to a goal completion than others from the outset. If your randomized segment is badly weighted across these various demographics then it could affect your results. Try thinking of ways to segment your audience and test specifically to each of these demographics. For example, you might try testing users that have downloaded your brochure in a separate test to those who have not. This can often yield some useful (and surprising!) results.
Remember, the scientific process is an on-going process; you’re never really done testing! Keep making new hypotheses and testing your ideas, and you’ll never stop improving!
[bctt tweet=”Keep making new hypotheses and testing your ideas, and you’ll never stop improving.”]
Thanks for reading! Do you struggle with getting results from split-testing? If you have any questions feel free to post them in the comments section and we’ll do our best to answer them. Good luck and happy testing!
