Proceedings of the 11th international conference on World Wide Web
A New Algorithm for Multi-objective Graph Partitioning
Euro-Par '99 Proceedings of the 5th International Euro-Par Conference on Parallel Processing
A linear-time heuristic for improving network partitions
DAC '82 Proceedings of the 19th Design Automation Conference
Collaborative Web Crawling: Information Gathering/Processing over Internet
HICSS '99 Proceedings of the Thirty-second Annual Hawaii International Conference on System Sciences-Volume 5 - Volume 5
UbiCrawler: a scalable fully distributed web crawler
Software—Practice & Experience
Geographical partition for distributed web crawling
Proceedings of the 2005 workshop on Geographic information retrieval
Discovering URLs through user feedback
Proceedings of the 20th ACM international conference on Information and knowledge management
Crawling rich internet applications: the state of the art
CASCON '12 Proceedings of the 2012 Conference of the Center for Advanced Studies on Collaborative Research
| Hi-index | 0.00 |
This paper presents a multi-objective approach to Web space partitioning, aimed to improve distributed crawling efficiency. The investigation is supported by the construction of two different weighted graphs. The first is used to model the topological communication infrastructure between crawlers and Web servers and the second is used to represent the amount of link connections between servers' pages. The values of the graph edges represent, respectively, computed RTTs and pages links between nodes. The two graphs are further combined, using a multi-objective partitioning algorithm, to support Web space partitioning and load allocation for an adaptable number of geographical distributed crawlers. Partitioning strategies were evaluated by varying the number of partitions (crawlers) to obtain merit figures for: i) download time, ii) exchange time and iii) relocation time. Evaluation has showed that our partitioning schemes outperform traditional hostname hash based counterparts in all evaluated metric, achieving on average 18% reduction for download time, 78% reduction for exchange time and 46% reduction for relocation time.