Discovery of Web Robot Sessions Based on their Navigational Patterns
Data Mining and Knowledge Discovery
Catching web crawlers in the act
ICWE '06 Proceedings of the 6th international conference on Web engineering
New Developments in Classification and Data Analysis: Proceedings of the Meeting of the Classification and Data Analysis Group (CLADAG) of the Italian ... Data Analysis, and Knowledge Organization)
Web robot detection in the scholarly information environment
Journal of Information Science
Web robot detection: A probabilistic reasoning approach
Computer Networks: The International Journal of Computer and Telecommunications Networking
Proceedings of the 2009 workshop on Web Search Click Data
Web robot detection techniques: overview and limitations
Data Mining and Knowledge Discovery
Evaluation of web robot discovery techniques: a benchmarking study
ICDM'06 Proceedings of the 6th Industrial Conference on Data Mining conference on Advances in Data Mining: applications in Medicine, Web Mining, Marketing, Image and Signal Mining
Slash-based relevance propagation model for topic distillation
Journal of Web Engineering
| Hi-index | 0.00 |
In web robot detection it is important is to find features that are common characteristics of diverse robots, in order to differentiate between them and humans. Existing approaches employ fairly simple features (e.g. empty referrer field, interval between successive requests), which often fail to reflect web robots' behaviour accurately. False alarms may therefore occur unacceptably often. In this paper we propose a fresh approach that expresses the behaviour of interactive users and various web robots in terms of a sequence of request types, called request patterns. Previous proposals have primarily targeted the detection of text crawlers, but our approach works well on many other web robots, such as image crawlers, email collectors and link checkers. In empirical evaluation of more than 1 billion requests collected at www.microsoft.com, our approach achieved 94% accuracy in web robot detection, estimated by F-measure. A decision tree algorithm proposed by Tan and Kumar was also applied to the same data. A comparison shows that the proposed approach is more accurate, and that real-time detection of web robots is feasible.