Scuba: diving into data at facebook

Authors:
Lior Abraham;John Allen;Oleksandr Barykin;Vinayak Borkar;Bhuwan Chopra;Ciprian Gerea;Daniel Merl;Josh Metzler;David Reiss;Subbu Subramanian;Janet L. Wiener;Okay Zed
Affiliations:
Facebook, Inc. Menlo Park, CA;Facebook, Inc. Menlo Park, CA;Facebook, Inc. Menlo Park, CA;Facebook, Inc. Menlo Park, CA;Facebook, Inc. Menlo Park, CA;Facebook, Inc. Menlo Park, CA;Facebook, Inc. Menlo Park, CA;Facebook, Inc. Menlo Park, CA;Facebook, Inc. Menlo Park, CA;Facebook, Inc. Menlo Park, CA;Facebook, Inc. Menlo Park, CA;Facebook, Inc. Menlo Park, CA
Venue:
Proceedings of the VLDB Endowment
Year:
2013

Citing 12
Cited 0

The temporal query language TQuel

ACM Transactions on Database Systems (TODS)
A relational approach to monitoring complex systems

ACM Transactions on Computer Systems (TOCS)
The implementation and performance of compressed databases

ACM SIGMOD Record
C-store: a column-oriented DBMS

VLDB '05 Proceedings of the 31st international conference on Very large data bases
Breaking the memory wall in MonetDB

Communications of the ACM - Surviving the data deluge
Hive: a warehousing solution over a map-reduce framework

Proceedings of the VLDB Endowment
Dremel: interactive analysis of web-scale datasets

Proceedings of the VLDB Endowment
HyPer: A hybrid OLTP&OLAP main memory database system based on virtual memory snapshots

ICDE '11 Proceedings of the 2011 IEEE 27th International Conference on Data Engineering
Shark: fast data analysis using coarse-grained distributed memory

SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
Efficient transaction processing in SAP HANA database: the end of a column store myth

SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
Peregrine: Low-latency queries on Hive warehouse data

XRDS: Crossroads, The ACM Magazine for Students - Big Data
Processing a trillion cells per mouse click

Proceedings of the VLDB Endowment

Quantified Score

Hi-index	0.00

Visualization

Abstract

Facebook takes performance monitoring seriously. Performance issues can impact over one billion users so we track thousands of servers, hundreds of PB of daily network traffic, hundreds of daily code changes, and many other metrics. We require latencies of under a minute from events occuring (a client request on a phone, a bug report filed, a code change checked in) to graphs showing those events on developers' monitors. Scuba is the data management system Facebook uses for most real-time analysis. Scuba is a fast, scalable, distributed, in-memory database built at Facebook. It currently ingests millions of rows (events) per second and expires data at the same rate. Scuba stores data completely in memory on hundreds of servers each with 144 GB RAM. To process each query, Scuba aggregates data from all servers. Scuba processes almost a million queries per day. Scuba is used extensively for interactive, ad hoc, analysis queries that run in under a second over live data. In addition, Scuba is the workhorse behind Facebook's code regression analysis, bug report monitoring, ads revenue monitoring, and performance debugging.