- Things to consider before employing experiments
- The science behind business experimentation
- Lessons to learn from corporate behemoths: how the big ones do it
- Experimentation changes the decision making process
In the world of science, experiments are the norm. It’s a reliable way to see if your ideas work and to check if an innovation is a true breakthrough or something that works well only on paper. Nevertheless, it is always a vision that comes first, then a string of trials and failures until we get the result we want. The same is true for business. Experimentation in business has exploded in recent years, promising concrete results and eliminating guesswork. Sticking to only one method without keeping a watchful eye on the market’s pulse or considering whether your customers actually want a new or improved product or service can cost a company its life. And while most of us like our Coca-Cola in the same container, with a recognisable logo, we also appreciate new and exciting ads. That’s the key to success – experimentation. However, knowing what to change isn’t always clear, but there are several key elements future-focused entrepreneurs may want to consider before starting their experiments.
Things to consider before employing experiments
Great ideas don’t always yield great results. For one, most businesses rely on data from previous experiences. They rely on knowing how their customers react based on already tried and tested products or services. But figuring out how they might respond to a new service you plan to implement is largely based on guesswork, and great ideas don’t always translate into success. Launching new products or services based on not much more than a hunch could cause irreparable damage to your brand. Running experiments, however, can provide valuable insights into your target market and ideas or projects worth investing in.
Keep in mind, though, that experimentation in business isn’t simple. Having a great idea is a great starting point, but the idea alone isn’t enough. To make running business experiments a bit easier, you can outline a checklist with as many steps and details as possible to help you gather the maximum amount of data. It’s always smart, though, to take into consideration what could hypothetically go wrong – and add potential pitfalls to your checklist.
Firstly, it’s important to set the right parameters – meaning, decide exactly what can be measured and at what cost. For example, when you’re running an ad it’s important that you have a clear vision of what you’d like to achieve and then set corresponding parameters. These can include choosing your target market using geographic, demographic, and behaviouristic characteristics, the media you’d use, the time frame during which the ad would be active, to name a few. In short, leave out as little as possible. The more detailed the preparation, the better results. Once the results have been generated, you’ll need a great data analyst to translate all the data into actionable information your product development and marketing team can use. Before running ‘real’ experiments that can be costly, consider using a scientific A/B test model.
The science behind business experimentation
A/B testing means that you work with two groups, each treated differently. If you want to see, for instance, how your cold email campaign would work, you can run a test within your organisation. The first step could include choosing two sets of emails and sending them out to your colleagues. One group would receive emails written in your recognisable tone and style, while the other would receive a completely different set of emails sent out over a certain period. You could also include your business partners, as their opinion is valuable, too. Then, run the campaign, collect data, and employ an in-house analyst to help you get actionable information. Bear in mind, however, that the results most likely won’t tell you much as you’re dealing with small groups, but you’d most definitely gain valuable experience from this exercise. To find out which innovation is most effective, it’s essential to run a number of carefully planned experiments. Take Microsoft for example. It discovered that “only one-third of its experiments prove effective, one-third have neutral results, and one-third have negative results.”
The online accommodation platform Booking.com also conducts experiments within their organisation, which can serve as pilot for conducting experiments in a larger ecosystem. As much as 80 per cent of the product development teams is conducting experiments, ranging from customer facing platform experiments to customer service and marketing to partner oriented ones. Given that about 1,500 employees are involved in experimentation, the gathered data is qualitatively as well as quantitatively valuable. Once finished, results are saved in a centralised repository allowing transparency and granting anyone on a team access to see outcomes of previous experiments. Naturally, engaging more employees in the experiments results in getting valid experimental evidence that can later be used to make product-related decisions.
Lessons to learn from corporate behemoths: how the big ones do it
Online accommodation rental giant Airbnb also runs experiments to solve serious challenges like racial discrimination. Airbnb establishment owners/managers as well as customers have been found to have been subjected to racial discrimination. Customer complaints of being rejected because of their skin colour and African-American accommodation hosts earning less money were the reason to run an experiment and determine the extent of the racial discrimination on the platform. Michael Luca and Max Bazerman, authors of the book The Power of Experiments: Decision-Making in a Data Driven World, described running an audit experiment in which 16 per cent of inquiries with distinctively African American sounding names were less likely to rent accommodation than those signed with “white-sounding names”, despite the content of the inquiry being identical. Reportedly, the company gathered a team together to study the case and find a solution. “Research is scheduled to begin in September 2020, and all hosts and guests will have an opportunity to opt out. Starting June 30, we shared details about how the process works and how you can opt out – and everyone will receive at least 30 days’ notice to opt out should they choose not to participate,” stated the company. “We’ll use our partner’s perceptions, for example, to figure out whether the reservations of those seen as a certain race are declined more often than others, which will help us create new features and policies to address any difference. We’ve partnered with civil rights and privacy organizations to make sure we do this work in a way that’s both thoughtful and respectful of everyone’s privacy.”
Knowing what to measure and the right timeframe are also valuable parameters in business experimentation. Take StubHub, the American ticket exchange and resale company, for example. They run experiments to help them decide if they should mention transaction fees upfront or shroud them until the final checkout screen. Their study showed that not mentioning them upfront increased revenue as customers were more likely to purchase when they only saw the transaction costs right at the end of the proces, at the checkout. However, it’s worth noting that this revenue increase might only reflect short term profits. The company tracked customers for a year to find that “these customers were less likely to come back in the next few months, but this is dominated by the revenue effect of increased short term sales.” Still, even a one-year measurement isn’t sufficient to give a clear picture of long term reputation effects of their decision to move away from their ‘no surprise fees’ policy.
Introducing a new product is a challenge for small and large companies alike. Customers are often hesitant to try new products, which can deter companies from rolling out new products or services. This is where running an experiment could help, too. In 2018, Uber tested out a new service called Express Pool, a cheaper alternative to their UberPool. “Uber says these Express Pool trips will be up to 50 percent cheaper than Uber Pool and 70 percent cheaper than Uber X,” reported Engadget. The service would save passengers a few coins, but they are asked to walk a short distance to meet their ride. The service was introduced in six larger markets, and the gathered data not only showed how their customers reacted to Express Pool, they also managed to compare its impact on existing services. As the results were positive, the company decided to introduce the new service to its major markets.
Experimentation changes the decision making process
Experimentation in business makes it possible to test every assumption or parameter in order to make the best possible data-backed decision. Single experiments run on a small scale and within an organisation won’t necessarily generate actionable information, but it will arm you with experience by pointing out potential pain points. This makes it easier to outline detailed experiments, decide which parameters should be measured, and determine the right timeframe. Supporting your teams to experiment with new ideas is important, and a scientific approach to making data-driven decisions is the key to success.
