Detecting Web Robots Using Resource Request Patterns

Derek Doran; Swapna S. Gokhale

doi:10.1109/ICMLA.2012.11

Detecting Web Robots Using Resource Request Patterns

Derek Doran, Swapna S. Gokhale

Computer Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

A significant proportion of Web traffic is now attributed to Web robots, and this proportion is likely to grow over time. These robots may threaten the security, privacy, functionality, and performance of a Web server due to their unregulated crawling behavior. Therefore, to assess their impact, it must be possible to accurately detect Web robot requests. Contemporary detection approaches, however, may cease to be effective as the behavior of both robots and humans evolves. In this paper, we present a novel detection approach that is based on the contrasts in the resource request patterns of robots and humans. The proposed scheme, which relies on an invariant behavioral difference between humans and robots, builds on the lessons from contemporary approaches. We demonstrate that the proposed approach can accurately detect Web robots and argue that it is expected to remain effective even as they continue their rapid evolution.

Original language	American English
Title of host publication	2012 11th International Conference on Machine Learning and Applications
Publisher	IEEE
Pages	7-12
Number of pages	6
ISBN (Electronic)	978-0-7695-4913-2
ISBN (Print)	978-1-4673-4651-1
DOIs	https://doi.org/10.1109/ICMLA.2012.11
State	Published - Jan 10 2013
Event	11th IEEE International Conference on Machine Learning and Applications, ICMLA 2012 - Boca Raton, FL, United States Duration: Dec 12 2012 → Dec 15 2012

Conference

Conference	11th IEEE International Conference on Machine Learning and Applications, ICMLA 2012
Country/Territory	United States
City	Boca Raton, FL
Period	12/12/12 → 12/15/12

ASJC Scopus Subject Areas

Human-Computer Interaction
Education

Keywords

Detection
User Classification
Web crawler
Web Log Analysis
Web Mining
Web robot

Disciplines

Computer Sciences
Engineering

Access to Document

10.1109/ICMLA.2012.11License: Unspecified

Cite this

@inproceedings{0e8181490d914f8a9ab42f00d49af558,

title = "Detecting Web Robots Using Resource Request Patterns",

abstract = " A significant proportion of Web traffic is now attributed to Web robots, and this proportion is likely to grow over time. These robots may threaten the security, privacy, functionality, and performance of a Web server due to their unregulated crawling behavior. Therefore, to assess their impact, it must be possible to accurately detect Web robot requests. Contemporary detection approaches, however, may cease to be effective as the behavior of both robots and humans evolves. In this paper, we present a novel detection approach that is based on the contrasts in the resource request patterns of robots and humans. The proposed scheme, which relies on an invariant behavioral difference between humans and robots, builds on the lessons from contemporary approaches. We demonstrate that the proposed approach can accurately detect Web robots and argue that it is expected to remain effective even as they continue their rapid evolution.",

keywords = "Detection, User Classification, Web crawler, Web Log Analysis, Web Mining, Web robot",

author = "Derek Doran and Gokhale, {Swapna S.}",

year = "2013",

month = jan,

day = "10",

doi = "10.1109/ICMLA.2012.11",

language = "American English",

isbn = "978-1-4673-4651-1",

pages = "7--12",

booktitle = "2012 11th International Conference on Machine Learning and Applications",

publisher = "IEEE",

note = "11th IEEE International Conference on Machine Learning and Applications, ICMLA 2012 ; Conference date: 12-12-2012 Through 15-12-2012",

}

TY - GEN

T1 - Detecting Web Robots Using Resource Request Patterns

AU - Doran, Derek

AU - Gokhale, Swapna S.

PY - 2013/1/10

Y1 - 2013/1/10

N2 - A significant proportion of Web traffic is now attributed to Web robots, and this proportion is likely to grow over time. These robots may threaten the security, privacy, functionality, and performance of a Web server due to their unregulated crawling behavior. Therefore, to assess their impact, it must be possible to accurately detect Web robot requests. Contemporary detection approaches, however, may cease to be effective as the behavior of both robots and humans evolves. In this paper, we present a novel detection approach that is based on the contrasts in the resource request patterns of robots and humans. The proposed scheme, which relies on an invariant behavioral difference between humans and robots, builds on the lessons from contemporary approaches. We demonstrate that the proposed approach can accurately detect Web robots and argue that it is expected to remain effective even as they continue their rapid evolution.

AB - A significant proportion of Web traffic is now attributed to Web robots, and this proportion is likely to grow over time. These robots may threaten the security, privacy, functionality, and performance of a Web server due to their unregulated crawling behavior. Therefore, to assess their impact, it must be possible to accurately detect Web robot requests. Contemporary detection approaches, however, may cease to be effective as the behavior of both robots and humans evolves. In this paper, we present a novel detection approach that is based on the contrasts in the resource request patterns of robots and humans. The proposed scheme, which relies on an invariant behavioral difference between humans and robots, builds on the lessons from contemporary approaches. We demonstrate that the proposed approach can accurately detect Web robots and argue that it is expected to remain effective even as they continue their rapid evolution.

KW - Detection

KW - User Classification

KW - Web crawler

KW - Web Log Analysis

KW - Web Mining

KW - Web robot

UR - http://www.scopus.com/inward/record.url?scp=84873576467&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84873576467&partnerID=8YFLogxK

UR - https://corescholar.libraries.wright.edu/cse/267

U2 - 10.1109/ICMLA.2012.11

DO - 10.1109/ICMLA.2012.11

M3 - Conference contribution

SN - 978-1-4673-4651-1

SP - 7

EP - 12

BT - 2012 11th International Conference on Machine Learning and Applications

PB - IEEE

T2 - 11th IEEE International Conference on Machine Learning and Applications, ICMLA 2012

Y2 - 12 December 2012 through 15 December 2012

ER -

Detecting Web Robots Using Resource Request Patterns

Abstract

Conference

ASJC Scopus Subject Areas

Keywords

Disciplines

Access to Document

Other files and links

Cite this