A bulletproof guide to running marketing experiments
Experimentation and data drive growth.
“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.” - Jeff Bezos, Amazon
But the question is, how can you set up a self-driving experimentation machine that helps you run marketing tests.
And is that the best use of your marketing team’s time?
You’re most probably not running Amazon (if you are, hi Jeff 👋). So why should you apply the same practices as the biggest retail giant?
Because it impacts the bottom line:
“e-commerce companies that consistently conduct marketing experiments instead of taking a “one and done” approach produce 30% higher ad performance in the same year and a 45% increase in performance the year after that.” - Think With Google
But it goes beyond getting to the highest conversion rate or improving profitability.
In a land of ever-evolving digital marketing channels and brands adapting to demographics as fast as the season change, marketing experiments help you to keep up with your competition and test new strategies to win over potential customers:
“There’s no marketing strategy that you can use forever because you are always exposed to new challenges. If you do not evolve or adapt yourself, you will be left behind by your competitors.” - Palson Yi, Marketing Director at Realme Indonesia
So what does a well-designed marketing experiment look like?
The methodology behind marketing experiments is rather straightforward:
There are many possible caveats to how your experiment turns wrong for all the wrong reasons.
But the three most common issues are:
Test too many things at once. It is called an A/B test because it compares side by side two possible realities: A, the world as is, and B, the world as it could be. If you introduce too many additional variables (C/D/E/…) it becomes hard to tell what really drove the KPI metric up or down. Was it the loading time optimization or the hero visual change on the homepage? Maybe it was a new value proposition? Did you try and test new paid marketing campaigns on TikTok? Introducing too many independent variables makes the interpretation murkier.
Case Study: A subject line worth 2 million dollars
Obama’s presidential campaign happened many presidential mandates ago, but the methods used will go down in history for their incredible success.
The majority of the campaign funds were raised via email marketing, and to make the outreach successful, the campaign managers conducted experiments with the email’s copy to optimize it.
Concretely, multiple subject lines were shortlisted as the best contenders, and emails were sent out to smaller batches of the email list for an A/B test. The winning email of those experiments was then sent to the remainder of the email list.
How impactful were those tests?
According to Toby Fallsgraff, Email Director, and Amelia Showalter, Director of Digital Analytics, the difference between the best and the worst email equated to 2 million dollars in donations.
This is why testing pays off.
Many analytical tools can help you with the heavy-statistical-lifting needed to run and interpret marketing experiments.
These tools are great and we highly recommend them.
But their main challenge is that they are hard to scale.
The more you experiment, the more data you collect, the more you become interested in the big picture. When you direct your marketing efforts towards multiple growth levers, you want to make sure the omnichannel approach is working.
So tools like Unbounce can only inform you about the landing page experience, and Facebook ads can only offer insights about PPC advertising.
Understanding and running experiments on your end-to-end marketing funnel necessitate a tool that can collect data from all the different tools you use for every marketing touchpoint.
Keboola can help with that.
Keboola brings together data from advertising platforms, CRM systems, and other data sources to help you establish and showcase the true value of your marketing efforts.
And you don’t need a single data scientist or engineer on your team. With its plug-and-play design, you can easily export data from all of the marketing tools you are using to analyze the big picture behind your marketing experimentation.
Check why Keboola is trusted by over 95 marketing agencies globally.