Self-Join Size Estimation in Large-scale Distributed Data Systems

Authors:
Theoni Pitoura;Peter Triantafillou
Affiliations:
Research Academic Computer Technology Institute, and University of Patras, 26500 Rio, Patras, Greece. pitoura@ceid.upatras.gr;Research Academic Computer Technology Institute, and University of Patras, 26500 Rio, Patras, Greece. peter@ceid.upatras.gr
Venue:
ICDE '08 Proceedings of the 2008 IEEE 24th International Conference on Data Engineering
Year:
2008

Citing 0
Cited 4

Power-law based estimation of set similarity join size

Proceedings of the VLDB Endowment
Characterizing web syndication behavior and content

WISE'11 Proceedings of the 12th international conference on Web information system engineering
On optimizing relational self-joins

Proceedings of the 15th International Conference on Extending Database Technology
A gossip-based approach for Internet-scale cardinality estimation of XPath queries over distributed semistructured data

The VLDB Journal — The International Journal on Very Large Data Bases

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this work we tackle the open problem of self-join size (SJS) estimation in a large-scale Distributed Data System, where tuples of a relation are distributed over data nodes which comprise an overlay network. Our contributions include adaptations of five well-known SJS estimation centralized techniques (coined sequential, cross-sampling, adaptive, bifocal, and sample-count) to the network environment and a novel technique which is based on the use of the Gini coefficient. We develop analyses showing how Gini estimations can lead to estimations of the underlying Zipfian or power-law value distributions. We further contribute distributed sampling algorithms that can estimate accurately and efficiently the Gini coefficient. Finally, we provide detailed experimental evidence testifying for the claimed increased accuracy, precision, and efficiency of the proposed SJS estimation method, compared to the other methods. The proposed approach is the only one to ensure high efficiency, precision, and accuracy regardless of the skew of the underlying data.