Improving the performance of focused web crawlers

  • Authors:

  • Sotiris Batsakis;Euripides G. M. Petrakis;Evangelos Milios

  • Affiliations:

  • Department of Electronic and Computer Engineering, Technical University of Crete (TUC), Chania, Crete GR-73100, Greece;Department of Electronic and Computer Engineering, Technical University of Crete (TUC), Chania, Crete GR-73100, Greece;Faculty of Computer Science, Dalhousie University, Halifax, Nova Scotia, Canada B3H 1W5

  • Venue:
  • Data & Knowledge Engineering
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

This work addresses issues related to the design and implementation of focused crawlers. Several variants of state-of-the-art crawlers relying on web page content and link information for estimating the relevance of web pages to a given topic are proposed. Particular emphasis is given to crawlers capable of learning not only the content of relevant pages (as classic crawlers do) but also paths leading to relevant pages. A novel learning crawler inspired by a previously proposed Hidden Markov Model (HMM) crawler is described as well. The crawlers have been implemented using the same baseline implementation (only the priority assignment function differs in each crawler) providing an unbiased evaluation framework for a comparative analysis of their performance. All crawlers achieve their maximum performance when a combination of web page content and (link) anchor text is used for assigning download priorities to web pages. Furthermore, the new HMM crawler improved the performance of the original HMM crawler and also outperforms classic focused crawlers in searching for specialized topics.