Collaborative Web Crawling: Information Gathering/Processing over Internet

Authors:
Shang-Hua Teng;Qi Lu;Matthias Eichstaedt;Daniel Ford;Tobin Lehman
Affiliations:
-;-;-;-;-
Venue:
HICSS '99 Proceedings of the Thirty-second Annual Hawaii International Conference on System Sciences-Volume 5 - Volume 5
Year:
1999

Citing 0
Cited 7

e-Colabra: An Enterprise Collaboration & Reuse Environment

NGIT '99 Proceedings of the 4th International Workshop on Next Generation Information Technologies and Systems
Researchexplorer: gaining insights through exploration in multimedia scientific data

Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval
The design and evaluation of accessibility on web navigation

Decision Support Systems
Architecture of a grid-enabled Web search engine

Information Processing and Management: an International Journal
On the feasibility of geographically distributed web crawling

Proceedings of the 3rd international conference on Scalable information systems
Efficient Partitioning Strategies for Distributed Web Crawling

Information Networking. Towards Ubiquitous Networking and Services
Design and evaluation of improvement method on the web information navigation - A stochastic search approach

Decision Support Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

IBM Almaden Research Center, 650 Harry Road, San Jose, California 95120-6099. In this paper, we present a scalable method for collaborative web crawling and information processing. The method includes an automatic cyberspace partitioner which is designed to dynamically balance and re-balance the load among processors. It can be can be used when all web crawlers are located on a tightly coupled high-performance system as well as when they are scattered in a distributed environment. We have implemented our algorithms in Java as a part of the IBM Grand Central Station (GCS) system.