Stanford WebBase components and applications

  • Authors:

  • Junghoo Cho;Hector Garcia-Molina;Taher Haveliwala;Wang Lam;Andreas Paepcke;Sriram Raghavan;Gary Wesley

  • Affiliations:

  • Stanford University, Los Angeles, CA;Stanford University, Stanford, CA;Stanford University, Mountain View, CA;Stanford University, Mountain View, CA;Stanford University, Stanford, CA;Stanford University, San Jose, CA;Stanford University, Stanford, CA

  • Venue:
  • ACM Transactions on Internet Technology (TOIT)
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

We describe the design and performance of WebBase, a tool for Web research. The system includes a highly customizable crawler, a repository for collected Web pages, an indexer for both text and link-related page features, and a high-speed content distribution facility. The distribution module enables researchers world-wide to retrieve pages from WebBase, and stream them across the Internet at high speed. The advantage for the researchers is that they need not all crawl the Web before beginning their research. WebBase has been used by scores of research and teaching organizations world-wide, mostly for investigations into Web topology and linguistic content analysis. After describing the system's architecture, we explain our engineering decisions for each of the WebBase components, and present respective performance measurements.