The nature of data center traffic: measurements & analysis

Authors:
Srikanth Kandula;Sudipta Sengupta;Albert Greenberg;Parveen Patel;Ronnie Chaiken
Affiliations:
Microsoft Research, Redmond, WA, USA;Microsoft Research, Redmond, WA, USA;Microsoft Research, Redmond, WA, USA;Microsoft Research, Redmond, WA, USA;Microsoft, Redmond, WA, USA
Venue:
Proceedings of the 9th ACM SIGCOMM conference on Internet measurement conference
Year:
2009

Citing 22
Cited 64

Fast accurate computation of large-scale IP traffic matrices from link loads

SIGMETRICS '03 Proceedings of the 2003 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
Estimating flow distributions from sampled flow statistics

Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications
Gigascope: a stream database for network applications

Proceedings of the 2003 ACM SIGMOD international conference on Management of data
The Google file system

SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles
Building a better NetFlow

Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications
Traffic matrix estimation on a large IP backbone: a comparison on real data

Proceedings of the 4th ACM SIGCOMM conference on Internet measurement
Traffic matrices: balancing measurements, inference and modeling

SIGMETRICS '05 Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systems
A first look at modern enterprise traffic

IMC '05 Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement
The power of slicing in internet flow measurement

IMC '05 Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement
Network anomography

IMC '05 Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement
MapReduce: simplified data processing on large clusters

OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6
Bigtable: a distributed storage system for structured data

OSDI '06 Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation - Volume 7
Dryad: distributed data-parallel programs from sequential building blocks

Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007
Pig latin: a not-so-foreign language for data processing

Proceedings of the 2008 ACM SIGMOD international conference on Management of data
Network exception handlers: host-network control in enterprise networks

Proceedings of the ACM SIGCOMM 2008 conference on Data communication
How healthy are today's enterprise networks?

Proceedings of the 8th ACM SIGCOMM conference on Internet measurement
SCOPE: easy and efficient parallel processing of massive data sets

Proceedings of the VLDB Endowment
VL2: a scalable and flexible data center network

Proceedings of the ACM SIGCOMM 2009 conference on Data communication
Safe and effective fine-grained TCP retransmissions for datacenter communication

Proceedings of the ACM SIGCOMM 2009 conference on Data communication
Understanding data center traffic characteristics

Proceedings of the 1st ACM workshop on Research on enterprise networking
Understanding TCP incast throughput collapse in datacenter networks

Proceedings of the 1st ACM workshop on Research on enterprise networking
The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines

The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines

Differential virtual time (DVT): rethinking I/O service differentiation for virtual machines

Proceedings of the 1st ACM symposium on Cloud computing
Data center TCP (DCTCP)

Proceedings of the ACM SIGCOMM 2010 conference
Hedera: dynamic flow scheduling for data center networks

NSDI'10 Proceedings of the 7th USENIX conference on Networked systems design and implementation
The case for fine-grained traffic engineering in data centers

INM/WREN'10 Proceedings of the 2010 internet network management conference on Research on enterprise networking
DevoFlow: cost-effective flow management for high performance enterprise networks

Hotnets-IX Proceedings of the 9th ACM SIGCOMM Workshop on Hot Topics in Networks
CloudPolice: taking access control out of the network

Hotnets-IX Proceedings of the 9th ACM SIGCOMM Workshop on Hot Topics in Networks
Network traffic characteristics of data centers in the wild

IMC '10 Proceedings of the 10th ACM SIGCOMM conference on Internet measurement
vSnoop: Improving TCP Throughput in Virtualized Environments via Acknowledgement Offload

Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
VL2: a scalable and flexible data center network

Communications of the ACM
ICTCP: Incast Congestion Control for TCP in data center networks

Proceedings of the 6th International COnference
LEGUP: using heterogeneity to reduce the cost of data center network upgrades

Proceedings of the 6th International COnference
A cost comparison of datacenter network architectures

Proceedings of the 6th International COnference
Reining in the outliers in map-reduce clusters using Mantri

OSDI'10 Proceedings of the 9th USENIX conference on Operating systems design and implementation
Scarlett: coping with skewed content popularity in mapreduce clusters

Proceedings of the sixth conference on Computer systems
Profiling network performance for multi-tier data center applications

Proceedings of the 8th USENIX conference on Networked systems design and implementation
Sharing the data center network

Proceedings of the 8th USENIX conference on Networked systems design and implementation
VMFlow: leveraging VM mobility to reduce network power costs in data centers

NETWORKING'11 Proceedings of the 10th international IFIP TC 6 conference on Networking - Volume Part I
Augmenting data center networks with multi-gigabit wireless links

Proceedings of the ACM SIGCOMM 2011 conference
Towards predictable datacenter networks

Proceedings of the ACM SIGCOMM 2011 conference
DevoFlow: scaling flow management for high-performance networks

Proceedings of the ACM SIGCOMM 2011 conference
Small-world datacenters

Proceedings of the 2nd ACM Symposium on Cloud Computing
Opportunistic flooding to improve TCP transmit performance in virtualized clouds

Proceedings of the 2nd ACM Symposium on Cloud Computing
Traffic engineering for multiple spanning tree protocol in large data centers

Proceedings of the 23rd International Teletraffic Congress
Cross-layer flow and congestion control for datacenter networks

Proceedings of the 3rd Workshop on Data Center - Converged and Virtual Ethernet Switching
3D beamforming for wireless data centers

Proceedings of the 10th ACM Workshop on Hot Topics in Networks
MicroTE: fine grained traffic engineering for data centers

Proceedings of the Seventh COnference on emerging Networking EXperiments and Technologies
Exploring online virtual networks mapping with stochastic bandwidth demand in multi-datacenter

Photonic Network Communications
On controller performance in software-defined networks

Hot-ICE'12 Proceedings of the 2nd USENIX conference on Hot Topics in Management of Internet, Cloud, and Enterprise Networks and Services
Lowering Inter-datacenter Bandwidth Costs via Bulk Data Scheduling

CCGRID '12 Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012)
NetPilot: automating datacenter network failure mitigation

Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication
Surviving failures in bandwidth-constrained datacenters

Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication
Mirror mirror on the ceiling: flexible wireless links for data centers

Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication
GRIN: utilizing the empty half of full bisection networks

HotCloud'12 Proceedings of the 4th USENIX conference on Hot Topics in Cloud Ccomputing
vCRIB: virtualized rule management in the cloud

HotCloud'12 Proceedings of the 4th USENIX conference on Hot Topics in Cloud Ccomputing
Spatio-temporal compressive sensing and internet traffic matrices

IEEE/ACM Transactions on Networking (TON)
ASIC: an architecture for scalable intra-domain control in OpenFlow

Proceedings of the 7th International Conference on Future Internet Technologies
NetPilot: automating datacenter network failure mitigation

ACM SIGCOMM Computer Communication Review - Special october issue SIGCOMM '12
Surviving failures in bandwidth-constrained datacenters

ACM SIGCOMM Computer Communication Review - Special october issue SIGCOMM '12
Mirror mirror on the ceiling: flexible wireless links for data centers

ACM SIGCOMM Computer Communication Review - Special october issue SIGCOMM '12
Improving large graph processing on partitioned graphs in the cloud

Proceedings of the Third ACM Symposium on Cloud Computing
On the feasibility of completely wireless datacenters

Proceedings of the eighth ACM/IEEE symposium on Architectures for networking and communications systems
Tuning ECN for data center networks

Proceedings of the 8th international conference on Emerging networking experiments and technologies
PAST: scalable ethernet for data centers

Proceedings of the 8th international conference on Emerging networking experiments and technologies
Demand-aware flow allocation in data center networks

Proceedings of the 2012 ACM workshop on Capacity sharing
Revisiting flow-based load balancing: Stateless path selection in data center networks

Computer Networks: The International Journal of Computer and Telecommunications Networking
Queueing system topologies with limited flexibility

Proceedings of the ACM SIGMETRICS/international conference on Measurement and modeling of computer systems
Challenges in cloud scale data centers

Proceedings of the ACM SIGMETRICS/international conference on Measurement and modeling of computer systems
Scalable rule management for data centers

nsdi'13 Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation
EyeQ: practical network performance isolation at the edge

nsdi'13 Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation
Leveraging endpoint flexibility in data-intensive clusters

Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM
ElasticSwitch: practical work-conserving bandwidth guarantees for cloud computing

Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM
Fast, accurate simulation for SDN prototyping

Proceedings of the second ACM SIGCOMM workshop on Hot topics in software defined networking
Protocol Responsibility Offloading to Improve TCP Throughput in Virtualized Environments

ACM Transactions on Computer Systems (TOCS)
Scissors: dealing with header redundancies in data centers through SDN

Proceedings of the 8th International Conference on Network and Service Management
Rethinking the physical layer of data center networks of the next decade: using optics to enable efficient *-cast connectivity

ACM SIGCOMM Computer Communication Review
ICTCP: incast congestion control for TCP in data-center networks

IEEE/ACM Transactions on Networking (TON)
Next stop, the cloud: understanding modern web service deployment in EC2 and azure

Proceedings of the 2013 conference on Internet measurement conference
When the network crumbles: an empirical study of cloud network failures and their impact on services

Proceedings of the 4th annual Symposium on Cloud Computing
Scalable, optimal flow routing in datacenters via local link balancing

Proceedings of the ninth ACM conference on Emerging networking experiments and technologies
DBridges: Flexible floodless frame forwarding

Computer Networks: The International Journal of Computer and Telecommunications Networking
Power consumption evaluation of all-optical data center networks

Cluster Computing
On the feasibility of completely wirelesss datacenters

IEEE/ACM Transactions on Networking (TON)
Circuit switching under the radar with REACToR

NSDI'14 Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation
Catch the whole lot in an action: rapid precise packet loss notification in data centers

NSDI'14 Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation

Quantified Score

Hi-index	0.02

Visualization

Abstract

We explore the nature of traffic in data centers, designed to support the mining of massive data sets. We instrument the servers to collect socket-level logs, with negligible performance impact. In a 1500 server operational cluster, we thus amass roughly a petabyte of measurements over two months, from which we obtain and report detailed views of traffic and congestion conditions and patterns. We further consider whether traffic matrices in the cluster might be obtained instead via tomographic inference from coarser-grained counter data.