After posting about Split Testing a few weeks ago, I found some very nice material about the implementation of Split Testing from the folks at Microsoft’s Experiment Platform. Be sure to check out the slides for a quick and excellent overview with good examples (a lot of interesting stuff about user feedback in there) and if you fancy some academic nitty-gritty, check out the paper as well. Both are excellent resources if you want some inspiration for your Split Testing.
Split testing is a cheap and reliable way to test two or more versions of a design against each other and see how they perform under live conditions. When split testing you focus one or a couple of quantitative metrics (such as like revenue, number of completed sales or sign-ups) and use them to judge how each design performs. It is a great method if you want to try out an advertising campaign, a set of rewritten purchase instructions or a new sign-up process. Sure, the method has its flaws (which I will get to later) but it is still a great method for finetuning a web page that every web developer should have in her toolbox.
In this post I will be outlining the basics concepts and list the pros and cons of the two main methods for split testing – A/B Testing and Multivariate Testing.
