A Dynamic Reconfiguration Model for a Distributed Web Crawling System

Authors:
Hongfie Yan;Jianyong Wang;Xiaoming Li
Affiliations:
-;-;-
Venue:
ICCNMC '01 Proceedings of the 2001 International Conference on Computer Networks and Mobile Computing (ICCNMC'01)
Year:
2001

Citing 3
Cited 0

The anatomy of a large-scale hypertextual Web search engine

WWW7 Proceedings of the seventh international conference on World Wide Web 7
Architectural Design and Evaluation of an Efficient Web-crawling System

IPDPS '01 Proceedings of the 15th International Parallel & Distributed Processing Symposium
Digging for Gold on the Web: Experience with the WebGather

HPC '00 Proceedings of the The Fourth International Conference on High-Performance Computing in the Asia-Pacific Region-Volume 2 - Volume 2

Quantified Score

Hi-index	0.00

Visualization

Abstract

A web crawling system employing a parallel and distributed architecture needs to have a mechanism to bring the whole system in a coordinated state when the nodes are added to or removed from the system. This paper presents an efficient dynamic reconfiguration model that can be used in such a system. The study shows that this model leads to some nice properties, such as load balance and low traffic in the system, which contribute to high performance. Currently this model is being implemented in WebGather, a well-known Chinese and English web search engine.