Cumulated gain-based evaluation of IR techniques
ACM Transactions on Information Systems (TOIS)
ACM SIGIR Forum
Investigating behavioral variability in web search
Proceedings of the 16th international conference on World Wide Web
Determining the user intent of web search engine queries
Proceedings of the 16th international conference on World Wide Web
Practical guide to controlled experiments on the web: listen to your customers not to the hippo
Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining
Introduction to Information Retrieval
Introduction to Information Retrieval
LIBLINEAR: A Library for Large Linear Classification
The Journal of Machine Learning Research
How does clickthrough data reflect retrieval quality?
Proceedings of the 17th ACM conference on Information and knowledge management
PSkip: estimating relevance ranking quality from web search clickthrough data
Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
What's in a session: tracking individual behavior on the web
Proceedings of the 20th ACM conference on Hypertext and hypermedia
Optimal rare query suggestion with implicit user feedback
Proceedings of the 19th international conference on World wide web
Predicting searcher frustration
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Overlapping experiment infrastructure: more, better, faster experimentation
Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
Simulating simple user behavior for system effectiveness evaluation
Proceedings of the 20th ACM international conference on Information and knowledge management
Effects of search success on search engine re-use
Proceedings of the 20th ACM international conference on Information and knowledge management
Proceedings of the fifth ACM international conference on Web search and data mining
Evaluating the effectiveness of search task trails
Proceedings of the 21st international conference on World Wide Web
Modeling and predicting behavioral dynamics on the web
Proceedings of the 21st international conference on World Wide Web
Hi-index | 0.00 |
User engagement in search refers to the frequency for users (re-)using the search engine to accomplish their tasks. Among factors that affected users' visit frequency, relevance of search results is believed to play a pivotal role. While multiple work in the past has demonstrated the correlation between search success and user engagement based on longitudinal analysis, we examine this problem from a different perspective in this work. Specifically, we carefully designed a large-scale controlled experiment on users of a large commercial Web search engine, in which users were separated into control and treatment groups, where users in treatment group were presented with search results which are deliberate degraded in relevance. We studied users' responses to the relevance degradation through tracking several behavioral metrics (such as query per user, click per session) over an extended period of time both during and following the experiment. By quantifying the relationship between user engagement and search relevance, we observe significant differences between user's short-term search behavior and long-term engagement change. By leveraging some of the key findings from the experiment, we developed a machine learning model to predict the long term impact of relevance degradation on user engagement. Overall, our model achieves over 67% of accuracy in predicting user engagement drop. Besides, our model is also capable of predicting engagement change for low-frequency users with very few user signals. We believe that insights from this study can be leveraged by search engine companies to detect and intervene search relevance degradation and to prevent long term user engagement drop.