Multilevel k-way partitioning scheme for irregular graphs
Journal of Parallel and Distributed Computing
A polylogarithmic approximation of the minimum bisection
FOCS '00 Proceedings of the 41st Annual Symposium on Foundations of Computer Science
Convex Optimization
Group formation in large social networks: membership, growth, and evolution
Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining
Find me if you can: improving geographical prediction with social and spatial proximity
Proceedings of the 19th international conference on World wide web
Streaming graph partitioning for large distributed graphs
Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
Proceedings of the sixth ACM international conference on Web search and data mining
Graph cluster randomization: network exposure to multiple universes
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Restreaming graph partitioning: simple versatile algorithms for advanced balancing
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Unicorn: a system for searching the social graph
Proceedings of the VLDB Endowment
FENNEL: streaming graph partitioning for massive scale graphs
Proceedings of the 7th ACM international conference on Web search and data mining
Hi-index | 0.00 |
Partitioning graphs at scale is a key challenge for any application that involves distributing a graph across disks, machines, or data centers. Graph partitioning is a very well studied problem with a rich literature, but existing algorithms typically can not scale to billions of edges, or can not provide guarantees about partition sizes. In this work we introduce an efficient algorithm, balanced label propagation, for precisely partitioning massive graphs while greedily maximizing edge locality, the number of edges that are assigned to the same shard of a partition. By combining the computational efficiency of label propagation --- where nodes are iteratively relabeled to the same 'label' as the plurality of their graph neighbors --- with the guarantees of constrained optimization --- guiding the propagation by a linear program constraining the partition sizes --- our algorithm makes it practically possible to partition graphs with billions of edges. Our algorithm is motivated by the challenge of performing graph predictions in a distributed system. Because this requires assigning each node in a graph to a physical machine with memory limitations, it is critically necessary to ensure the resulting partition shards do not overload any single machine. We evaluate our algorithm for its partitioning performance on the Facebook social graph, and also study its performance when partitioning Facebook's 'People You May Know' service (PYMK), the distributed system responsible for the feature extraction and ranking of the friends-of-friends of all active Facebook users. In a live deployment, we observed average query times and average network traffic levels that were 50.5% and 37.1% (respectively) when compared to the previous naive random sharding.