A/B Testing

Summary

Fig 1 - Source: The Complete Guide to A/B Testing, VWO

Does your manager decide based only on his intuition and experience, not data? Do you like to experiment and test results? Would you like data to validate your assumptions? Do you prefer to make data-driven decisions?

If your answers to the above questions are "YES", then you are in the right place - learn more about A/B Testing.

A/B Testing, also known as Split Testing, is a method to compare two or more variations of an app or a webpage or an email and determine which one works better. In this method, the users are randomly assigned to the different variants and statistical analysis is performed to determine which variation performs better for a defined business goal.[1]

Milestones

Discussion

• Why A/B Testing?

  With A/B Testing, you can make more out of your existing traffic. Since even small changes can lead to better conversion or lead generation, the return on investment (RoI) can be massive sometimes.[1] The results help us to understand user behavior better and determine what impacts the user experience. Since the testing validates assumptions, it removes guesswork and enables data-informed decisions from "we-think" to "we-know".[2]

• How should you do A/B Testing?

  The correct way to run a A/B Testing is to follow a scientific process:[2]

  • Identify Goal: Based on the business goal, a conversion metric is decided. Example: Number of visitors who sign up for free trial.
  • Observe User Behavior: Observe the user behavior and understand his/her motivations.
  • Construct Hypothesis: Time to brainstorm hypotheses, prioritize based on expected impact and difficulty of implementation.
  • Create Variations: Decide on which feature you want to improve such as changing the color or size of the CTA button.
  • Run Experiment: Randomly assign the users to test and control group and collect relevant data for analysis.
  • Analyze Results: Once the experiment is complete, you can analyze the results of which variation performed better and whether there was a statistically significant difference.
• When should you not do A/B Testing?

  

  Fig 2 - Source: A/B Testing Literature Review, Information Architecture and Web Usability

  A/B Testing cannot make a good design choice on its own. It can compare two or more design choices and help us decide which is better.

  Incremental changes can improve design only upto a certain extent. In order to improve it radically, the designer should think more creatively and understand the user needs better.[5]

• How does A/B testing affect SEO? Can you provide the best practices?

  A/B Testing or multivariate testing does not pose any inherent risk to the website's search rank if used properly. However, it is important to know the best practices.[2]

  1. No Cloaking

  2. Use rel="canonical"

  3. Use 302 Redirects Instead Of 301s

  4. Run Experiments Only As Long As Necessary