Towards very large scale digital library building in greenstone using parallel processing

Authors:
John Thompson;David Bainbridge;Hussein Suleman
Affiliations:
Department of Computer Science, University of Waikato, Hamilton, New Zealand;Department of Computer Science, University of Waikato, Hamilton, New Zealand;Department of Computer Science, University of Cape Town, Cape Town, South Africa
Venue:
ICADL'11 Proceedings of the 13th international conference on Asia-pacific digital libraries: for cultural heritage, knowledge dissemination, and future creation
Year:
2011

Citing 6
Cited 0

Partitioned posting files: a parallel inverted file structure for information retrieval

SIGIR '90 Proceedings of the 13th annual international ACM SIGIR conference on Research and development in information retrieval
Introduction: parallel processing and information retrieval

Information Processing and Management: an International Journal - Special issue on parallel processing and information retrieval
Web Search for a Planet: The Google Cluster Architecture

IEEE Micro
Stress-testing general purpose digital library software

ECDL'09 Proceedings of the 13th European conference on Research and advanced technology for digital libraries
Second workshop on very large digital libraries: in conjunction with the european conference on digital libraries Corfu, Greece, 2 october 2009

ACM SIGMOD Record
How to Build a Digital Library, Second Edition

How to Build a Digital Library, Second Edition

Quantified Score

Hi-index	0.00

Visualization

Abstract

As very large digital library collections become more commonplace, software tools must adapt appropriately. This paper reports on an evolution of the Greenstone Digital Library software to support parallel processing during the collection building phase. A series of experiments were conducted to first establish a basic speed-up factor, and then deconstruct the parallelisation process to understand the execution profile of the application. Several bottlenecks were identified and resolved to further improve the performance. The adaptation of Greenstone confirms that the build phase is indeed a suitable candidate for parallelisation; and suggests that parallelisation of processing is a new avenue for exploration in emerging digital library architectures.