The input/output complexity of sorting and related problems
Communications of the ACM
Introduction to algorithms
Efficient external memory algorithms by simulating coarse-grained parallel algorithms
Proceedings of the ninth annual ACM symposium on Parallel algorithms and architectures
Random permutations on distributed, external and hierarchical memory
Information Processing Letters
External-memory graph algorithms
Proceedings of the sixth annual ACM-SIAM symposium on Discrete algorithms
Cache performance analysis of traversals and random accesses
Proceedings of the tenth annual ACM-SIAM symposium on Discrete algorithms
I/O-complexity of graph algorithms
Proceedings of the tenth annual ACM-SIAM symposium on Discrete algorithms
On external memory graph traversal
SODA '00 Proceedings of the eleventh annual ACM-SIAM symposium on Discrete algorithms
I/O-efficient algorithms for graphs of bounded treewidth
SODA '01 Proceedings of the twelfth annual ACM-SIAM symposium on Discrete algorithms
Breadth-first crawling yields high-quality pages
Proceedings of the 10th international conference on World Wide Web
External memory algorithms and data structures: dealing with massive data
ACM Computing Surveys (CSUR)
I/O-optimal algorithms for planar graphs using separators
SODA '02 Proceedings of the thirteenth annual ACM-SIAM symposium on Discrete algorithms
Implementing I/O-efficient Data Structures Using TPIE
ESA '02 Proceedings of the 10th Annual European Symposium on Algorithms
External-Memory Breadth-First Search with Sublinear I/O
ESA '02 Proceedings of the 10th Annual European Symposium on Algorithms
FOCS '99 Proceedings of the 40th Annual Symposium on Foundations of Computer Science
Design and Implementation of a High-Performance Distributed Web Crawler
ICDE '02 Proceedings of the 18th International Conference on Data Engineering
Algorithms for memory hierarchies: advanced lectures
Algorithms for memory hierarchies: advanced lectures
Towards realistic implementations of external memory algorithms using a coarse grained paradigm
ICCSA'03 Proceedings of the 2003 international conference on Computational science and its applications: PartII
STXXL: standard template library for XXL data sets
ESA'05 Proceedings of the 13th annual European conference on Algorithms
Terracost: Computing least-cost-path surfaces for massive grid terrains
Journal of Experimental Algorithmics (JEA)
ESA '08 Proceedings of the 16th annual European symposium on Algorithms
Linear-time disk-based implicit graph search
Journal of the ACM (JACM)
External memory layout vs. schematic
ACM Transactions on Design Automation of Electronic Systems (TODAES)
A Heuristic Strong Connectivity Algorithm for Large Graphs
SEA '09 Proceedings of the 8th International Symposium on Experimental Algorithms
Design and Engineering of External Memory Traversal Algorithms for General Graphs
Algorithmics of Large and Complex Networks
Multithreaded Asynchronous Graph Traversal for In-Memory and Semi-External Memory
Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
Algorithm engineering: bridging the gap between algorithm theory and practice
Algorithm engineering: bridging the gap between algorithm theory and practice
Accelerating CUDA graph algorithms at maximum warp
Proceedings of the 16th ACM symposium on Principles and practice of parallel programming
External memory breadth-first search with delayed duplicate detection on the GPU
MoChArt'10 Proceedings of the 6th international conference on Model checking and artificial intelligence
Parallel breadth-first search on distributed memory systems
Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis
Efficient processing of distance queries in large graphs: a vertex cover approach
SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
I/O-efficient shortest path algorithms for undirected graphs with random or bounded edge lengths
ACM Transactions on Algorithms (TALG)
I/O efficient: computing SCCs in massive graphs
Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data
Efficient single-source shortest path and distance queries on large graphs
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
IS-Label: an independent-set based labeling scheme for point-to-point distance querying
Proceedings of the VLDB Endowment
| Hi-index | 0.00 |
Breadth First Search (BFS) traversal is an archetype for many important graph problems. However, computing a BFS level decomposition for massive graphs was considered nonviable so far, because of the large number of I/Os it incurs. This paper presents the first experimental evaluation of recent external-memory BFS algorithms for general graphs. With our STXXL based implementations exploiting pipelining and disk-parallelism, we were able to compute the BFS level decomposition of a web-crawl based graph of around 130 million nodes and 1.4 billion edges in less than 4 hours using single disk and 2.3 hours using 4 disks. We demonstrate that some rather simple external-memory algorithms perform significantly better (minutes as compared to hours) than internal-memory BFS, even if more than half of the input resides internally.