The Sevensix way

In this write up, the Sevensix way will introduce the simple principles to get started with data driven experimentation. Most product or ecommerce organizations do not utilize AB testing, due to difficulties of test design, complex technical experimentation setup or analysis-paralysis when retrieving results. The Sevensix way propose to simplify AB testing, as the objective is about testing as frequent as possible.

Define your hypothesis, define your metric.

Your hypothesis should be closely tied with one metric.

We believe that if we implement X, it will have a positive impact on Y

In reality, your ideas might have an impact on multiple metrics, but distilling into a simple metric upfront is important to avoid the temptation of later on finding the metric that suits your narrative. It’s hard to make it that simple, but it’s a good exercise to get into the habit for doing simple AB tests. Do not get lost in the endless possibilities and ways to measure, but stay aware that this is just a tool in your CRO method stack. The more you practice, the better you become at knowing what metrics you should be looking at.

What metric matters

Often you will see AB testing guides recommending you to select a business metric like revenue, as it will directly tell you if your idea has a negative or positive impact on business revenue. This is a good rule of thumb to at least consider the impact on revenue, but you can go ahead and test other metrics to easier break down impact of an experiment, as long as you consider the potentially other important metrics being affected that you should include in your overall evaluation of the test.

AB testing are not a silver bullet

Using Sevensix is just a method to test whether one idea has a higher statistically significant conversion rate over another idea. This is one way to test and improve your digital product, but you should always consider other circumstances important for your business to make an informed decision. Sevensix do not claim to know everything about your business, by hooking up to your warehouse or testing for multiple metrics at once, we want to avoid that complexity. Sevensix makes is simple and easy to start data-driven experimentation. We are not a swiss-army knife, we are just one tool.

Focus on what you want to test, we will handle the statistics

Setting up an A/B test with us is like figuring out what to eat and wait for the chef to bring the dish. You set it up, and then... relax. We handle the heat, the stirring, and even the taste testing. Most A/B test never happen because people stale at the work needed to analyze and report findings. We will run the statistics with simple frequentist methods.

Why you generally only should implement winners

If your idea seems a bit better or on par with the existing conversion rate, it might be tempting to roll out the new idea fully. There is a few reasons why that might not be a good idea.

  1. Your new idea could disrupt the experience users currently have of the product without improving the overall experience. In the long term these changes can have a negative impact, if you don’t see they improve something on your website.
  2. If you keep the existing idea, you can easily experiment with a new and potentially better idea on the same terms without confusing users further.
  3. It could be expensive to roll out and maintain a new service that does not significantly improve your business.

Most Ideas Stink, and That’s Okay!

Remember, most ideas don’t work out, and that’s perfectly fine. Each surprising moment is a step away from potential disaster. We celebrate the flops because they save time, resources, and sometimes, even your brand’s reputation. It hurts when your idea didn’t “win”, but you can turn it around to a positive experience, when decide to not pursue a mediocre idea, it will be the best business or UX decision. Most people will not do that, they will let their ego win.

Test ideas, not buttons…

AB tests can be expensive, time-consuming and complex, but it’s potentially way more expensive and time-consuming on the long run if left out of your development process. Most people think AB testing is about choosing a color for a button. Unless you are Amazon, leave these kind of tests alone, and instead test real innovative ideas close to your strategy, to check and improve your real product strategy direction.