MagPIe: MPI's collective communication operations for clustered wide area systems
Proceedings of the seventh ACM SIGPLAN symposium on Principles and practice of parallel programming
Future Generation Computer Systems - Special issue on metacomputing
Topology discovery for large ethernet networks
Proceedings of the 2001 conference on Applications, technologies, architectures, and protocols for computer communications
Maximum likelihood network topology identification from edge-based unicast measurements
SIGMETRICS '02 Proceedings of the 2002 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
TOPOMON: A Monitoring Tool for Grid Network Topology
ICCS '02 Proceedings of the International Conference on Computational Science-Part II
Distributed Computing in a Heterogeneous Computing Environment
Proceedings of the 5th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface
MPICH/MADIII: a Cluster of Clusters Enabled MPI Implementation
CCGRID '03 Proceedings of the 3st International Symposium on Cluster Computing and the Grid
MPICH-G2: a Grid-enabled implementation of the Message Passing Interface
Journal of Parallel and Distributed Computing - Special issue on computational grids
Merging logical topologies using end-to-end measurements
Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement
Enabling Network Measurement Portability Through a Hierarchy of Characteristics
GRID '03 Proceedings of the 4th International Workshop on Grid Computing
Topology discovery in heterogeneous IP networks: the NetInventory system
IEEE/ACM Transactions on Networking (TON)
Ethernet Topology Discovery without Network Assistance
ICNP '04 Proceedings of the 12th IEEE International Conference on Network Protocols
Efficient algorithms for large-scale topology discovery
SIGMETRICS '05 Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Inference and Labeling of Metric-Induced Network Topologies
IEEE Transactions on Parallel and Distributed Systems
Balanced Multicasting: High-throughput Communication for Grid Applications
SC '05 Proceedings of the 2005 ACM/IEEE conference on Supercomputing
Multicast topology inference from measured end-to-end loss
IEEE Transactions on Information Theory
High performance wide-area overlay using deadlock-free routing
Proceedings of the 18th ACM international symposium on High performance distributed computing
GMount: An Ad Hoc and Locality-Aware Distributed File System by Using SSH and FUSE
CCGRID '09 Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid
Access-pattern and bandwidth aware file replication algorithm in a grid environment
GRID '08 Proceedings of the 2008 9th IEEE/ACM International Conference on Grid Computing
GMount: Build your grid file system on the fly
GRID '08 Proceedings of the 2008 9th IEEE/ACM International Conference on Grid Computing
Maximum-bandwidth ALM tree on tree network
CCNC'10 Proceedings of the 7th IEEE conference on Consumer communications and networking conference
Dynamic Load-Balanced Multicast for Data-Intensive Applications on Clouds
CCGRID '10 Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing
The topology aware file distribution problem
COCOON'11 Proceedings of the 17th annual international conference on Computing and combinatorics
The topology aware file distribution problem
Journal of Combinatorial Optimization
| Hi-index | 0.00 |
Adapting to the network is the key to achieving high performance for communication-intensive applications, including scientific computing,data intensive computing, and multicast, especially in Grid environments. This paper investigates an approach of representing network as a tree of participating hosts and switches matching or approximating their physical topology, and describes a fast, non-intrusive, and portable algorithm for inferring such a topology. This representation and the proposed inference algorithm serves as a key to building network-aware applications in a portable manner. The algorithm is based solely on RTTs of small packets between end hosts; it does not rely on popular but not universally available protocols such as trace route and SNMP. Another benefit is that it can handle all layers of network uniformly without any a priori knowledge of cluster configurations. The required number of measurements is O(Nd) in certain idealizing assumptions made for the purpose of analysis, where N is the number of participating processes and d the diameter of the network, which is usually small in real networks. In our experimental environment, the inference algorithm built a topology of 64 hosts in a single cluster in 4 seconds and and that of 256 hosts across 4 clusters in 15 seconds. It is able to not only identify clusters within a Grid, but also to partially identify the Layer 2 topology within a cluster. This is important for optimizing bandwidth-limited operations such as broadcast. We built several network-aware applications upon the inference system, including efficient bandwidth measurements and long message broadcasts. The topology is used to schedule as many measurements as possible in parallel without competing on shared links. We were able to build a bandwidth map of 256 hosts across 4 clusters in 27 seconds.