An extended method for finding related web pages with focused crawling techniques

  • Authors:

  • Kazutaka Furuse;Hiroaki Ohmura;Hanxiong Chen;Hiroyuki Kitagawa

  • Affiliations:

  • Department of Computer Science, Graduate School of Systems and Information Engineering, University of Tsukuba;Department of Computer Science, Graduate School of Systems and Information Engineering, University of Tsukuba;Department of Computer Science, Graduate School of Systems and Information Engineering, University of Tsukuba;Department of Computer Science, Graduate School of Systems and Information Engineering, University of Tsukuba

  • Venue:
  • KES'11 Proceedings of the 15th international conference on Knowledge-based and intelligent information and engineering systems - Volume Part II
  • Year:
  • 2011

Quantified Score

Hi-index 0.01

Visualization

Abstract

This paper proposes an extended mechanism for efficiently finding related web pages, which is constructed by introducing some focused crawling techniques. One of the successful methods for finding related web pages is Kleinberg's HITS algorithm, and this method determines web pages which are related to a set of given web pages by calculating the hub and authority scores. Although this method is effective for extracting fine related web pages, it has a limitation that it only concerns the web pages which are directly connected to the given web pages for the score calculation. The proposed method of this paper extends the HITS algorithm by enlarging neighborhood graph used for the score calculation. By navigating links forward and backward, pages which are not directly connected to the given web pages are included in the neighborhood graph. Since the navigation is done by using the focused crawling techniques, the proposed method effectively collects promising pages which contribute to improve accuracy of the scores. Moreover, unrelated pages are filtered out for avoiding topic drift in the course of the navigation. Consequently, the proposed method successfully finds related pages, since scores are calculated with adequately extended neighborhood graphs. The effectiveness and the efficiency of the proposed method is confirmed by the results of experiments performed with real data sets.