Parallel web spiders for cooperative information gathering

Authors:
Jiewen Luo;Zhongzhi Shi;Maoguang Wang;Wei Wang
Affiliations:
The Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China;The Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China;The Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China;Graduate School of Chinese Academy of Sciences
Venue:
GCC'05 Proceedings of the 4th international conference on Grid and Cooperative Computing
Year:
2005

Citing 4
Cited 1

Parallel crawlers

Proceedings of the 11th international conference on World Wide Web
Design and evaluation of a multi-agent collaborative Web mining system

Decision Support Systems - Web retrieval and mining
MAGE: An Agent-Oriented Programming Environment

ICCI '04 Proceedings of the Third IEEE International Conference on Cognitive Informatics
China's E-Science Knowledge Grid Environment

IEEE Intelligent Systems

Multi-modal services for web information collection based on multi-agent techniques

PRIMA'06 Proceedings of the 9th Pacific Rim international conference on Agent Computing and Multi-Agent Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

Web spider is a widely used approach to obtain information for search engines. As the size of the Web grows, it becomes a natural choice to parallelize the spider’s crawling process. This paper presents a parallel web spider model based on multi-agent system for cooperative information gathering. It uses the dynamic assignment mechanism to wipe off redundant web pages caused by parallelization. Experiments show that the parallel spider is effective to improve the information gathering performance within an acceptable interaction efficiency cost for controlling. This approach provides a novel perspective for the next generation advanced search engine.