(Re)design considerations for scalable large-file content distribution

Authors:
Brian Biskeborn;Michael Golightly;KyoungSoo Park;Vivek S. Pai
Affiliations:
Department of Computer Science, Princeton University;UC Irvine Computer Science Department and Department of Computer Science, Princeton University;Department of Computer Science, Princeton University;Department of Computer Science, Princeton University
Venue:
WORLDS'05 Proceedings of the 2nd conference on Real, Large Distributed Systems - Volume 2
Year:
2005

Citing 8
Cited 0

Consistent hashing and random trees: distributed caching protocols for relieving hot spots on the World Wide Web

STOC '97 Proceedings of the twenty-ninth annual ACM symposium on Theory of computing
Using name-based mappings to increase hit rates

IEEE/ACM Transactions on Networking (TON)
Logistical Computing and Internetworking: Middleware for the Use of Storage in Communication

AMS '01 Proceedings of the Third Annual International Workshop on Active Middleware Services
Maintaining high bandwidth under dynamic network conditions

ATEC '05 Proceedings of the annual conference on USENIX Annual Technical Conference
Handling churn in a DHT

ATEC '04 Proceedings of the annual conference on USENIX Annual Technical Conference
Reliability and security in the CoDeeN content distribution network

ATEC '04 Proceedings of the annual conference on USENIX Annual Technical Conference
Democratizing content publication with coral

NSDI'04 Proceedings of the 1st conference on Symposium on Networked Systems Design and Implementation - Volume 1
Shark: scaling file servers via cooperative caching

NSDI'05 Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation - Volume 2

Quantified Score

Hi-index	0.00

Visualization

Abstract

The CoBlitz system was designed to provide efficient large file transfer in a managed infrastructure environment. It uses a content distribution network (CDN) coupled with a swarm-style chunk distribution system to reduce the bandwidth required at origin servers. With 6 months of operation, we have been able to observe its behavior in typical usage, and glean information on how it could be redesigned to better suit its target audience. At its heart, this paper describes what happens when a plausible conceptual design meets the harsh realities of life on the Internet. We describe our experiences improving CoBlitz's performance via a range of techniques, including measurement-based feedback, heuristic changes, and new algorithms. In the process, we triple CoBlitz's performance, and we reduce the load it places on origin servers by a factor of five. In addition to improving performance for CoBlitz's users, we believe that our experiences will also be beneficial to other researchers working on large-file transfer and content distribution networks.