5 Questions To Ask After Each A/B Test

Why do we test?

Very often the discussion about testing is framed in the measurement side of things.  "To improve ROI" is a fairly typical the answer to the question above.  Yet, from an analytic or data science perspective that actually misses the point.

The reason we test is to gain knowledge.

To be sure testing green buttons vs. red buttons should be framed in terms of conversion metrics.  But more importantly we need to be asking ourselves the following questions:

1. Why did it work?
2. What are the segment characteristics of the winning option?
3. Among which segment did it NOT work?
4. What do the results tell us about the decision making process?
5. Where else in the pipeline or funnel can we apply this learning?

Understanding the answers to these questions will likely have a bigger impact on the business then testing purple in the next round.