A classification framework for web robots

Authors:
Derek Doran;Swapna S. Gokhale
Affiliations:
Department of Computer Science & Engineering, University of Connecticut, 371 Fairfield Road, Unit 2155, Storrs, CT06269-2155;Department of Computer Science & Engineering, University of Connecticut, 371 Fairfield Road, Unit 2155, Storrs, CT06269-2155
Venue:
Journal of the American Society for Information Science and Technology
Year:
2012

Citing 5
Cited 1

Evaluating topic-driven web crawlers

Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Agents and the Semantic Web

IEEE Intelligent Systems
The portrait of a common HTML web page

Proceedings of the 2006 ACM symposium on Document engineering
Web robot detection in the scholarly information environment

Journal of Information Science
Searching for Heavy Tails in Web Robot Traffic

QEST '10 Proceedings of the 2010 Seventh International Conference on the Quantitative Evaluation of Systems

A comparison of web robot and human requests

Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining

Quantified Score

Hi-index	0.00

Visualization

Abstract

The behavior of modern web robots varies widely when they crawl for different purposes. In this article, we present a framework to classify these web robots from two orthogonal perspectives, namely, their functionality and the types of resources they consume. Applying the classification framework to a year-long access log from the UConn SoE web server, we present trends that point to significant differences in their crawling behavior. © 2012 Wiley Periodicals, Inc.