A classification framework for web robots
Journal of the American Society for Information Science and Technology
A comparison of web robot and human requests
Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining
| Hi-index | 0.00 |
This paper presents a study on whether the heavy-tailed trends reported in Web traffic are present in the traffic generated by Web robots. The study is motivated by three factors: (i)~a significant volume of Web server traffic can now be attributed to Web robots, (ii)~the Web is continuing to evolve into a semantic and service-oriented environment where Web robots will play a central role, and (iii)~there are fundamental differences in the way robots and humans visit a site and search for information and these differences may lead to contrasts in the statistical patterns of the robots' requests compared to humans. We analyze Web robot traffic from a two-year access log from a Web server in the academic domain and study whether the response sizes, request inter-arrival times, and inter-session times exhibit heavy-tailed properties. In a multi-faceted analysis of the data we find that the response sizes and request inter-arrival times of robot requests do not exhibit heavy-tailed characteristics, contrasting the trends in these metrics in human traffic. However, we find that inter-session times of robots follow heavy-tailed characteristics similar to that of humans.