Efficient, automatic web resource harvesting

  • Authors:

  • Michael L. Nelson;Joan A. Smith;Ignacio Garcia del Campo

  • Affiliations:

  • Old Dominion University, Norfolk VA;Old Dominion University, Norfolk VA;Old Dominion University, Norfolk VA

  • Venue:
  • WIDM '06 Proceedings of the 8th annual ACM international workshop on Web information and data management
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

There are two problems associated with conventional web crawling techniques: a crawler cannot know if all resources at a non-trivial web site have been discovered and crawled ("the counting problem") and the human-readable format of the resources are not always suitable for machine processing ("the representation problem"). We introduce an approach that solves these two problems by implementing support for both the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) and MPEG-21 Digital Item Declaration Language (DIDL) into the web server itself. We present the Apache module "mod_oai", which can be used to address the counting problem by listing all valid URIs at a web server and efficiently discovering updates and additions on subsequent crawls. Our experiments indicated comparable performance for initial crawls, and dramatic increases in update speed mod_oaican also be used to address the representation problem by providing "preservation ready" versions of web resources aggregated with their respective forensic metadata in MPEG-21 DIDL format.