Combinatorial Complexity

Are there too many good ideas to test?

We often have lots of ideas that we know we should test, but sequentially going through the list would take forever and a day. The good news is that this problem is readily solvable using something called experimental design.

On a replay of a Direct webinar on multi-variate email testing Katie Cole of MerkleQuiris addressed this issue and stated something like “our account managers love to say fractional factorial in a sales meeting.”

Obviously alliteration; but what does it mean?

When designing a campaign we can use any number of factors – creative, list, segment, offer, etc. each with any number of variations. If we want to test this whole lot we need to consider ‘multivariate testing.’ Note that the old A/B test is the simplest case: One factor with two variations. When lots of factors are involved the number of possible combinations is simply the product of the number of variations for each factor.

For example: If an email test has 3 subject lines, 5 lists, and 4 segments and we evaluate all 60 possible (3 * 5 * 4) versions we’d be doing a ‘full factorial’ test.

At some point there are too many combinations to test. Imagine adding text vs. html, three wire-frame layouts, ads on left or right side, three offers located in four locations. Now we have over 14,000 combinations! This is where ‘fractional factorial’ comes in. It is a means of systematically reducing the number of combinations to a manageable number but still retaining the ability to answer some, but not all possible questions.

The smallest magic subset of combinations allows us to test the fundamental questions about each of the major factors:
- Does subject line matter?
- Does list impact response?
- Do segments behave differently?
- etc.

As more combinations are added to the mix we can ask questions with nuance.
- Does the choice of list impact how the subject line works?
- Does format affect the click-through rate of ad location?

The only way to answer all possible questions is to test all possible combinations – full factorial. However, in the majority of cases we’re not looking to understand the subtleties of how each factor interacts with each other. We want to know the major levers to customer behavior.

Like triage, ‘fractional factorial’ focuses on the important cases and allows us to assess the big questions of campaign design. Once we’ve identified the factors that drive behavior we’ll work on fine tuning the variations.