Web robot detection techniques: overview and limitations
Data Mining and Knowledge Discovery
Analysis of web logs: challenges and findings
PERFORM'10 Proceedings of the 2010 IFIP WG 6.3/7.3 international conference on Performance Evaluation of Computer and Communication Systems: milestones and future challenges
PUBCRAWL: protecting users and businesses from CRAWLers
Security'12 Proceedings of the 21st USENIX conference on Security symposium
A comparison of web robot and human requests
Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining
| Hi-index | 0.00 |
This paper proposes a novel functional classification scheme to understand and analyze web robot traffic. The scheme is rooted in the recognition that the crawling behavior of a robot on a site is primarily governed byits intended purpose or functionality. We apply the classification rules to analyze web server access logs from the University of Connecticut School of Engineering domain. The analysis results indicate how the classification scheme can provide insights into the robot traffic based on their functionality.