Online controlled experiments at large scale
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
A month in the life of a production news recommender system
Proceedings of the 2013 workshop on Living labs for information retrieval evaluation
Hi-index | 0.00 |
The web provides an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using controlled experiments (e.g., A/B tests and their generalizations). Whether for front-end user-interface changes, or backend recommendation systems and relevance algorithms, online controlled experiments are now utilized to make data-driven decisions at Amazon, Microsoft, eBay, Facebook, Google, Yahoo, Zynga, and at many other companies. While the theory of a controlled experiment is simple, and dates back to Sir Ronald A. Fisher's experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s, the deployment and mining of online controlled experiments at scale--thousands of experiments now--has taught us many lessons. We provide an introduction, share real examples, key learnings, cultural challenges, and humbling statistics.