As I learn more about A/B testing I’m increasingly excited about its potential impact on the value design and the web can deliver. During the design process we do our best to design the IA, interaction, and look and feel to be as effective as possible. However, even after all our experience, patterns, and principles have been applied there is still a significant amount of best-guess involved. Even usability tests, while definitely helpful, don’t give complete visibility into how a page or flow will actually perform. For example, we recently ran a usability study on several pages describing Sun’s programs for students and educators. One of the conclusions was that few people would click on one of links in one section of the page. But when we look at the metrics, that link is one of the most clicked on the page. Not exactly crystal-clear insight. So are A/B tests valuable?
Recently there has been a lot of debate about Google’s approach to design. Douglas Bowman may have kicked off the debate with his blog post earlier this year about why he left Google. He describes his frustration with Google’s push to test everything and requiring every decision be driven by data. He gives that now legendary example of Google testing 41 shades of blue to see which one best accomplished their goals. Since then there have been many posts, articles, and conversations debating whether making design decisions based on metrics is good or if it takes the creative heart out of the design process.
But when I see the results of A/B tests I’m amazed at the insights I come away with. When changing a couple of words results in a 20% improvement in conversion or Google sees differences in performance just by changing a color or the amount of white space, I realize the complexity of a human being interacting with a web site – and how little we really know about how various design elements impact our desired outcome(s).
Data analysis is not going to result in the definition of new strategic direction or a break-through concept. Our creativity and the design process can give us the breakthrough idea and even get us 80% of the way toward the perfect solution. We’ll never arrive at the 100% perfect solution but it seems that A/B testing can help us move from an 80% solution to an 85% solution. An improvement that can have a measurable impact on the bottom-line.
I think Irene Au, Google’s Director of User Experience summarizes it well in a recent article,
“Data informs decision-making but it’s less useful for conceiving and building conceptually new directions. It’s most useful for optimising and refining an established concept.”
I’m finding that not only can A/B testing give us insight into a specific project, but reviewing the results of other tests can be hugely informative for future design projects.
A great new resource is ABtests.com which I believe was recently launched by Joshua Porter, Performable, and KISSmetrics. The short and simple case studies provide a wealth of information (and may result in a few “wows” on your part). Take a look at some of the projects. I think you’ll learn a lot.
Other reference and resources:
