Lightweight causal and atomic group multicast
ACM Transactions on Computer Systems (TOCS)
SmartPointers: personalized scientific data portals in your hand
Proceedings of the 2002 ACM/IEEE conference on Supercomputing
Fast Heterogeneous Binary Data Interchange
HCW '00 Proceedings of the 9th Heterogeneous Computing Workshop
Lightweight Morphing Support for Evolving Middleware Data Exchanges in Distributed Applications
ICDCS '05 Proceedings of the 25th IEEE International Conference on Distributed Computing Systems
Resource-Aware Distributed Stream Management Using Dynamic Overlays
ICDCS '05 Proceedings of the 25th IEEE International Conference on Distributed Computing Systems
Leading Computational Methods on Scalar and Vector HEC Platforms
SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
Publish-Subscribe for High-Performance Computing
IEEE Internet Computing
WS-Messenger: A Web Services-Based Messaging System for Service-Oriented Grid Computing
CCGRID '06 Proceedings of the Sixth IEEE International Symposium on Cluster Computing and the Grid
Type-based publish/subscribe: Concepts and experiences
ACM Transactions on Programming Languages and Systems (TOPLAS)
Proceedings of the 2006 ACM/IEEE conference on Supercomputing
Cayuga: a high-performance event processing engine
Proceedings of the 2007 ACM SIGMOD international conference on Management of data
Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS)
CLADE '08 Proceedings of the 6th international workshop on Challenges of large applications in distributed environments
Concurrency and Computation: Practice & Experience
DataStager: scalable data staging services for petascale applications
Proceedings of the 18th ACM international symposium on High performance distributed computing
Towards expressive publish/subscribe systems
EDBT'06 Proceedings of the 10th international conference on Advances in Database Technology
A Maya use case: adaptable scientific workflows with ADIOS for general relativistic astrophysics
Proceedings of the Conference on Extreme Science and Engineering Discovery Environment: Gateway to Discovery
Active workflow system for near real-time extreme-scale science
Proceedings of the first workshop on Parallel programming for analytics applications
| Hi-index | 0.00 |
Streaming data models have been shown to be useful in many applications requiring high-performance data exchange. Application-level overlay networks are a natural way to realize these applications' data flows and their internal computations, but existing middleware is not designed to scale to the data rates and low overhead computations necessary for the high performance domain. This paper describes EVPath, a middleware infrastructure that supports the construction and management of overlay networks that can be customized both in topology and in the data manipulations being performed. Extending from a previous high-performance publish-subscribe system, EVPath not only provides for the low overhead movement and in-line processing of large data volumes, but also offers the flexibility needed to support the varied data flow and control needs of alternative higher-level streaming models. We explore some of the challenges of high performance event systems, including those experienced when operating an event infrastructure used to transport IO events at the scale of hundred+ thousand nodes. Specifically, when transporting output data from a large-scale simulation running on the ORNL Cray Jaguar petascale machine, a surprising new issue seen in experimentation at scale is the potential for strong perturbation of running applications from inappropriate speeds at which IO is performed. This requires the IO system's event transport to be explicitly scheduled to constrain resource competition, in addition to dynamically setting and changing the topologies of event delivery.