Mining association rules between sets of items in large databases
SIGMOD '93 Proceedings of the 1993 ACM SIGMOD international conference on Management of data
Mining quantitative association rules in large relational tables
SIGMOD '96 Proceedings of the 1996 ACM SIGMOD international conference on Management of data
A new framework for itemset generation
PODS '98 Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systems
Data mining: concepts and techniques
Data mining: concepts and techniques
Empirical bayes screening for multi-item associations
Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining
Beyond Market Baskets: Generalizing Association Rules to Dependence Rules
Data Mining and Knowledge Discovery
What Makes Patterns Interesting in Knowledge Discovery Systems
IEEE Transactions on Knowledge and Data Engineering
Determining Hit Rate in Pattern Search
Proceedings of the ESF Exploratory Workshop on Pattern Detection and Discovery
On the discovery of significant statistical quantitative rules
Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining
Probability and Computing: Randomized Algorithms and Probabilistic Analysis
Probability and Computing: Randomized Algorithms and Probabilistic Analysis
Fast discovery of unexpected patterns in data, relative to a Bayesian network
Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining
Mining compressed frequent-pattern sets
VLDB '05 Proceedings of the 31st international conference on Very large data bases
Introduction to Data Mining, (First Edition)
Introduction to Data Mining, (First Edition)
Proceedings of the 1st international workshop on open source data mining: frequent pattern mining implementations
Assessing data mining results via swap randomization
Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining
Frequent pattern mining: current status and future directions
Data Mining and Knowledge Discovery
Efficient Discovery of Statistically Significant Association Rules
ICDM '08 Proceedings of the 2008 Eighth IEEE International Conference on Data Mining
Fun at a department store: data mining meets switching theory
FUN'10 Proceedings of the 5th international conference on Fun with algorithms
Interestingness measures for association rules based on statistical validity
Knowledge-Based Systems
Assessing and ranking structural correlations in graphs
Proceedings of the 2011 ACM SIGMOD International Conference on Management of data
Discovering highly reliable subgraphs in uncertain graphs
Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining
Controlling false positives in association rule mining
Proceedings of the VLDB Endowment
Algorithms for detecting significantly mutated pathways in cancer
RECOMB'10 Proceedings of the 14th Annual international conference on Research in Computational Molecular Biology
| Hi-index | 0.00 |
As advances in technology allow for the collection, storage, and analysis of vast amounts of data, the task of screening and assessing the significance of discovered patterns is becoming a major challenge in data mining applications. In this work, we address significance in the context of frequent itemset mining. Specifically, we develop a novel methodology to identify a meaningful support threshold s* for a dataset, such that the number of itemsets with support at least s* represents a substantial deviation from what would be expected in a random dataset with the same number of transactions and the same individual item frequencies. These itemsets can then be flagged as statistically significant with a small false discovery rate. Our methodology hinges on a Poisson approximation to the distribution of the number of itemsets in a random dataset with support at least s, for any s greater than or equal to a minimum threshold smin. We obtain this result through a novel application of the Chen-Stein approximation method, which is of independent interest. Based on this approximation, we develop an efficient parametric multi-hypothesis test for identifying the desired threshold s*. A crucial feature of our approach is that, unlike most previous work, it takes into account the entire dataset rather than individual discoveries. It is therefore better able to distinguish between significant observations and random fluctuations. We present extensive experimental results to substantiate the effectiveness of our methodology.