Evaluation of web robot discovery techniques: a benchmarking study

Authors:
Nick Geens;Johan Huysmans;Jan Vanthienen
Affiliations:
Department of Decision Sciences and Information Management, Katholieke Universiteit Leuven, Leuven, Belgium;Department of Decision Sciences and Information Management, Katholieke Universiteit Leuven, Leuven, Belgium;Department of Decision Sciences and Information Management, Katholieke Universiteit Leuven, Leuven, Belgium
Venue:
ICDM'06 Proceedings of the 6th Industrial Conference on Data Mining conference on Advances in Data Mining: applications in Medicine, Web Mining, Marketing, Image and Signal Mining
Year:
2006

Citing 7
Cited 3

Ethical Web agents

Computer Networks and ISDN Systems
Discovery of Web Robot Sessions Based on their Navigational Patterns

Data Mining and Knowledge Discovery
Sequence Rules for Web Clickstream Analysis

Industrial Conference on Data Mining: Advances in Data Mining, Applications in E-Commerce, Medicine, and Knowledge Management
Intelligent E-marketing with Web Mining, Personalization, and User-Adpated Interfaces

Industrial Conference on Data Mining: Advances in Data Mining, Applications in E-Commerce, Medicine, and Knowledge Management
Web usage mining: discovery and applications of usage patterns from Web data

ACM SIGKDD Explorations Newsletter
Web usage mining: discovery and application of interesting patterns from web data

Web usage mining: discovery and application of interesting patterns from web data
Shopbots and pricebots

IJCAI'99 Proceedings of the 16th international joint conference on Artifical intelligence - Volume 1

Distinguishing humans from robots in web search logs: preliminary results using query rates and intervals

Proceedings of the 2009 workshop on Web Search Click Data
Web robot detection techniques: overview and limitations

Data Mining and Knowledge Discovery
Web robot detection based on pattern-matching technique

Journal of Information Science

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper describes part of a web usage mining study executed on log files obtained from a Belgian e-commerce company. From these log files, it can be observed that numerous web robots are active on the site. Most of these robots show a crawling behavior that is radically different from the browsing behavior of human visitors. Because the owners of the e-shop desire information about the paths that human visitors follow through the site, it is of crucial importance to remove these robotic visits from the log files. Several existing methods for web robot discovery are evaluated and compared, none of them leading to satisfying results. Therefore, a new technique is developed that results in a successful and reliable identification of web robots.