Using HMM to learn user browsing patterns for focused web crawling

  • Authors:

  • Hongyu Liu;Jeannette Janssen;Evangelos Milios

  • Affiliations:

  • Faculty of Computer Science, Dalhousie University, Halifax, NS, Canada;Department of Mathematics and Statistics, Dalhousie University, Halifax, NS, Canada;Faculty of Computer Science, Dalhousie University, Halifax, NS, Canada

  • Venue:
  • Data & Knowledge Engineering - Special issue: WIDM 2004
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

A focused crawler is designed to traverse the Web to gather documents on a specific topic. It can be used to build domain-specific Web search portals and online personalized search tools. To estimate the relevance of a newly seen URL, it must use information gleaned from previously crawled page sequences.In this paper, we present a new approach for prediction of the links leading to relevant pages based on a Hidden Markov Model (HMM). The system consists of three stages: user data collection, user modelling via sequential pattern learning, and focused crawling. In particular, we first collect the Web pages visited during a user browsing session. These pages are clustered, and the link structure among pages from different clusters is then used to learn page sequences that are likely to lead to target pages. The learning is performed using HMM. During crawling, the priority of links to follow is based on a learned estimate of how likely the page is to lead to a target page. We compare the performance with Context-Graph crawling and Best-First crawling. Our experiments demonstrate that this approach performs better than Context-Graph crawling and Best-First crawling.