Adaptive crawler for external hyperlinks search and acquisition

Authors:
A. A. Pechnikov;D. I. Chernobrovkin
Affiliations:
Institute of Applied Mathematical Research Karelian Research Center, Russian Academy of Sciences, Petrozavodsk, Russia;St. Petersburg State University, St. Petersburg, Russia
Venue:
Automation and Remote Control
Year:
2014

Citing 2
Cited 0

Finite-time Analysis of the Multiarmed Bandit Problem

Machine Learning
On URL normalization

ICCSA'05 Proceedings of the 2005 international conference on Computational Science and Its Applications - Volume Part II

Quantified Score

Hi-index	0.00

Visualization

Abstract

We describe a search robot (crawler) intended to collect information regarding outgoing hyperlinks from a given set of web sites related to a certain topic. The crawler's adaptive behavior is formulated in terms of a multi-armed bandit problem. Our experiments show that the choice of an adaptive algorithm for the crawler's rational behavior depends on the actual topic of the underlying set of web sites.