Searching for Heavy Tails in Web Robot Traffic

This paper presents a study on whether the heavy-tailed trends reported in Web traffic are present in the traffic generated by Web robots. The study is motivated by three factors: (i) a significant volume of Web server traffic can now be attributed to Web robots, (ii) the Web is continuing to evolve into a semantic and service-oriented environment where Web robots will play a central role, and (iii) there are fundamental differences in the way robots and humans visit a site and search for information and these differences may lead to contrasts in the statistical patterns of the robots' requests compared to humans. We analyze Web robot traffic from a two-year access log from a Web server in the academic domain and study whether the response sizes, request inter-arrival times, and inter-session times exhibit heavy-tailed properties. In a multi-faceted analysis of the data we find that the response sizes and request inter-arrival times of robot requests do not exhibit heavy-tailed characteristics, contrasting the trends in these metrics in human traffic. However, we find that inter-session times of robots follow heavy-tailed characteristics similar to that of humans.