TY - JOUR
T1 - Some (Non-)universal features of Web robot traffic
AU - Zabihimayvan, Mahdieh
AU - Doran, Derek
N1 - Understanding the qualities of Web robot traffic is essential to build mechanisms for mitigating the impact of their traffic on Web systems. This paper pre
PY - 2018/5/24
Y1 - 2018/5/24
N2 - Understanding the qualities of Web robot traffic is essential to build mechanisms for mitigating the impact of their traffic on Web systems. This paper presents an updated characterization of the navigational and session patterns of Web robot traffic across three Web servers in the United States, Europe, and the Middle East under 30 different features. The results indicate that some features may be fitted to the same heavy-tailed model across the Web servers, but the best fitting models for other features depend on the Web server. Due to some different tasks of Web robots and security policies set by website administrators, there are thus some features of Web robot traffic that cannot be universally modeled.
AB - Understanding the qualities of Web robot traffic is essential to build mechanisms for mitigating the impact of their traffic on Web systems. This paper presents an updated characterization of the navigational and session patterns of Web robot traffic across three Web servers in the United States, Europe, and the Middle East under 30 different features. The results indicate that some features may be fitted to the same heavy-tailed model across the Web servers, but the best fitting models for other features depend on the Web server. Due to some different tasks of Web robots and security policies set by website administrators, there are thus some features of Web robot traffic that cannot be universally modeled.
KW - Web robot characterization
KW - Web robot
KW - maximum likelihood estimation
KW - Vuong test
KW - heavy-tailed distribution
UR - https://ieeexplore.ieee.org/document/8362266/
U2 - 10.1109/CISS.2018.8362266
DO - 10.1109/CISS.2018.8362266
M3 - Article
JO - 2018 52nd Annual Conference on Information Sciences and Systems (CISS)
JF - 2018 52nd Annual Conference on Information Sciences and Systems (CISS)
ER -