Discovery of Web Robot Sessions Based on their Navigational Patterns
Data Mining and Knowledge Discovery
IEEE Internet Computing
Lessons and Challenges from Mining Retail E-Commerce Data
Machine Learning
Evaluating implicit measures to improve web search
ACM Transactions on Information Systems (TOIS)
Design and Analysis of Experiments
Design and Analysis of Experiments
Practical guide to controlled experiments on the web: listen to your customers not to the hippo
Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining
Controlled experiments on the web: survey and practical guide
Data Mining and Knowledge Discovery
Integrating OLAP and recommender systems: an evaluation perspective
DOLAP '10 Proceedings of the ACM 13th international workshop on Data warehousing and OLAP
Unexpected results in online controlled experiments
ACM SIGKDD Explorations Newsletter
Hierarchical composable optimization of web pages
Proceedings of the 21st international conference companion on World Wide Web
Trustworthy online controlled experiments: five puzzling outcomes explained
Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
Case study on the business value impact of personalized recommendations on a large online retailer
Proceedings of the sixth ACM conference on Recommender systems
Online controlled experiments at large scale
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Uncertainty in online experiments with dependent data: an evaluation of bootstrap methods
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Designing and deploying online field experiments
Proceedings of the 23rd international conference on World wide web
Hi-index | 0.00 |
Controlled experiments, also called randomized experiments and A/B tests, have had a profound influence on multiple fields, including medicine, agriculture, manufacturing, and advertising. While the theoretical aspects of offline controlled experiments have been well studied and documented, the practical aspects of running them in online settings, such as web sites and services, are still being developed. As the usage of controlled experiments grows in these online settings, it is becoming more important to understand the opportunities and pitfalls one might face when using them in practice. A survey of online controlled experiments and lessons learned were previously documented in Controlled Experiments on the Web: Survey and Practical Guide (Kohavi, et al., 2009). In this follow-on paper, we focus on pitfalls we have seen after running numerous experiments at Microsoft. The pitfalls include a wide range of topics, such as assuming that common statistical formulas used to calculate standard deviation and statistical power can be applied and ignoring robots in analysis (a problem unique to online settings). Online experiments allow for techniques like gradual ramp-up of treatments to avoid the possibility of exposing many customers to a bad (e.g., buggy) Treatment. With that ability, we discovered that it's easy to incorrectly identify the winning Treatment because of Simpson's paradox.