PyMW - A Python module for desktop grid and volunteer computing

  • Authors:

  • Eric M. Heien;Yusuke Takata;Kenichi Hagihara;Adam Kornafeld

  • Affiliations:

  • Graduate School of Information Science and Technology, Osaka University, Toyonaka, 560-8531, Japan;Graduate School of Information Science and Technology, Osaka University, Toyonaka, 560-8531, Japan;Graduate School of Information Science and Technology, Osaka University, Toyonaka, 560-8531, Japan;Laboratory of Parallel and Distributed Systems, Computer and Automation Research Institute, Hungarian Academy of Sciences, H-1132 Victor Hugo u. 18-22, Budapest, Hungary

  • Venue:
  • IPDPS '09 Proceedings of the 2009 IEEE International Symposium on Parallel&Distributed Processing
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

We describe a general purpose master-worker parallel computation Python module called PyMW. PyMW is intended to support rapid development, testing and deployment of large scale master-worker style computations on a desktop grid or volunteer computing environment. This module targets non-expert computer users by hiding complicated task submission and result retrieval procedures behind a simple interface. PyMW also provides a unified interface to multiple computing environments with easy extension to support additional environments. In this paper, we describe the internal structure and external interface to the PyMW module and its support for the Condor computing environment and the Berkeley Open Infrastructure for Network Computing (BOINC) platform. We demonstrate the effectiveness and scalability of PyMW by performing master-worker style computations on a desktop grid using Condor and a BOINC volunteer computing project.