Web Metrics: Proven Methods for Measuring Web Site Success
Web Metrics: Proven Methods for Measuring Web Site Success
Lessons and Challenges from Mining Retail E-Commerce Data
Machine Learning
Google Analytics
Call to Action: Secret Formulas to Improve Online Results
Call to Action: Secret Formulas to Improve Online Results
Web site measurement hacks
Data acquisition and cost-effective predictive modeling: targeting offers for electronic commerce
Proceedings of the ninth international conference on Electronic commerce
Productivity as a metric for visual analytics: reflections on e-discovery
Proceedings of the 2008 Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization
Design as exploration: creating interface alternatives through parallel authoring and runtime tuning
Proceedings of the 21st annual ACM symposium on User interface software and technology
The cost of a cloud: research problems in data center networks
ACM SIGCOMM Computer Communication Review
Call browser: a system to improve the caller experience by analyzing live calls end-to-end
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Seven pitfalls to avoid when running controlled experiments on the web
Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
Catching the drift: learning broad matches from clickthrough data
Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
Facilitating Controlled Tests of Website Design Changes: A Systematic Approach
ICWE '9 Proceedings of the 9th International Conference on Web Engineering
On the impact of sequence and time in rich media advertising
MM '09 Proceedings of the 17th ACM international conference on Multimedia
Practical lessons of data mining at Yahoo!
Proceedings of the 18th ACM conference on Information and knowledge management
Technical Section: CGV-An interactive graph visualization system
Computers and Graphics
Improving ad relevance in sponsored search
Proceedings of the third ACM international conference on Web search and data mining
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Measuring the user experience on a large scale: user-centered metrics for web applications
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Automatically generating personalized user interfaces with Supple
Artificial Intelligence
Evaluating whole-page relevance
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Proceedings of the ACM SIGCOMM 2010 conference
Optimizing cost and performance in online service provider networks
NSDI'10 Proceedings of the 7th USENIX conference on Networked systems design and implementation
WebProphet: automating performance prediction for web services
NSDI'10 Proceedings of the 7th USENIX conference on Networked systems design and implementation
A DNS reflection method for global traffic management
USENIXATC'10 Proceedings of the 2010 USENIX conference on USENIX annual technical conference
CloudCmp: shopping for a cloud made easy
HotCloud'10 Proceedings of the 2nd USENIX conference on Hot topics in cloud computing
Experiments in the wild: public evaluation of off-screen visualizations in the Android market
Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries
CloudCmp: comparing public cloud providers
IMC '10 Proceedings of the 10th ACM SIGCOMM conference on Internet measurement
Placing a value on aesthetics in online casual games
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Utility of human-computer interactions: toward a science of preference measurement
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Inter-datacenter bulk transfers with netstitcher
Proceedings of the ACM SIGCOMM 2011 conference
Effects of search success on search engine re-use
Proceedings of the 20th ACM international conference on Information and knowledge management
Recency ranking by diversification of result set
Proceedings of the 20th ACM international conference on Information and knowledge management
Evolving a learning analytics platform
Proceedings of the 1st International Conference on Learning Analytics and Knowledge
On the harmfulness of secondary game objectives
Proceedings of the 6th International Conference on Foundations of Digital Games
A practical solution to the client-LDNS mismatch problem
ACM SIGCOMM Computer Communication Review
Collaborative Filtering Recommender Systems
Foundations and Trends in Human-Computer Interaction
How far can client-only solutions go for mobile browser speed?
Proceedings of the 21st international conference on World Wide Web
Potential good abandonment prediction
Proceedings of the 21st international conference companion on World Wide Web
The impact of tutorials on games of varying complexity
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Backtracking Events as Indicators of Usability Problems in Creation-Oriented Applications
ACM Transactions on Computer-Human Interaction (TOCHI)
Trustworthy online controlled experiments: five puzzling outcomes explained
Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
High performance vehicular connectivity with opportunistic erasure coding
USENIX ATC'12 Proceedings of the 2012 USENIX conference on Annual Technical Conference
The unified logging infrastructure for data analytics at Twitter
Proceedings of the VLDB Endowment
Webzeitgeist: design mining the web
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A pilot study of using crowds in the classroom
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Scaling big data mining infrastructure: the twitter experience
ACM SIGKDD Explorations Newsletter
A provider-side view of web search response time
Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM
WTF: the who to follow service at Twitter
Proceedings of the 22nd international conference on World Wide Web
Evaluating and predicting user engagement change with degraded search relevance
Proceedings of the 22nd international conference on World Wide Web
Pushing CDN-ISP collaboration to the limit
ACM SIGCOMM Computer Communication Review
Session modeling to predict online buyer behavior
Proceedings of the 2013 workshop on Data-driven user behavioral modelling and mining from social media
Robust assessment of changes in cellular networks
Proceedings of the ninth ACM conference on Emerging networking experiments and technologies
Hi-index | 0.00 |
The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments (single factor or factorial designs), A/B tests (and their generalizations), split tests, Control/Treatment tests, and parallel flights. Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior. We provide a practical guide to conducting online experiments, where end-users can help guide the development of features. Our experience indicates that significant learning and return-on-investment (ROI) are seen when development teams listen to their customers, not to the Highest Paid Person's Opinion (HiPPO). We provide several examples of controlled experiments with surprising results. We review the important ingredients of running controlled experiments, and discuss their limitations (both technical and organizational). We focus on several areas that are critical to experimentation, including statistical power, sample size, and techniques for variance reduction. We describe common architectures for experimentation systems and analyze their advantages and disadvantages. We evaluate randomization and hashing techniques, which we show are not as simple in practice as is often assumed. Controlled experiments typically generate large amounts of data, which can be analyzed using data mining techniques to gain deeper understanding of the factors influencing the outcome of interest, leading to new hypotheses and creating a virtuous cycle of improvements. Organizations that embrace controlled experiments with clear evaluation criteria can evolve their systems with automated optimizations and real-time analyses. Based on our extensive practical experience with multiple systems and organizations, we share key lessons that will help practitioners in running trustworthy controlled experiments.