A Web Crawler Detection Algorithm Based on Web Page Member List

Authors:
Weigang Guo;Yong Zhong;Jianqin Xie
Affiliations:
-;-;-
Venue:
IHMSC '12 Proceedings of the 2012 4th International Conference on Intelligent Human-Machine Systems and Cybernetics - Volume 01
Year:
2012

Citing 0
Cited 1

Access patterns for robots and humans in web archives

Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries

Quantified Score

Hi-index	0.00

Visualization

Abstract

Following the widely use of search engines, the impact Web crawlers have on the Web sites should not be ignored. After analyzing the navigational patterns of Web crawlers from Web logs, a new algorithm based on Web page member list is proposed. The algorithm constructs one member list for every Web page and one show table for every visitor. The experiment shows that the new algorithm can detect the unknown crawlers and unfriendly crawlers who do not obey the Standard for Robot Exclusion.