Towards exploratory hypothesis testing and analysis

Authors:
Guimei Liu;Mengling Feng;Yue Wang;Limsoon Wong;See-Kiong Ng;Tzia Liang Mah;Edmund Jon Deoon Lee
Affiliations:
Department of Computer Science, National University of Singapore, Singapore;Data Mining Department, Institute for Infocomm Research, Singapore;Graduate School for Integrative Science and Engineering, National University of Singapore, Singapore;Department of Computer Science, National University of Singapore, Singapore;Data Mining Department, Institute for Infocomm Research, Singapore;Data Mining Department, Institute for Infocomm Research, Singapore;Pharmacology Department, National University of Singapore, Singapore
Venue:
ICDE '11 Proceedings of the 2011 IEEE 27th International Conference on Data Engineering
Year:
2011

Citing 0
Cited 2

SPHINX: rich insights into evidence-hypotheses relationships via parameter space-based exploration

Proceedings of the 22nd ACM international conference on Conference on information & knowledge management
A performance study of three disk-based structures for indexing and querying frequent itemsets

Proceedings of the VLDB Endowment

Quantified Score

Hi-index	0.01

Visualization

Abstract

Hypothesis testing is a well-established tool for scientific discovery. Conventional hypothesis testing is carried out in a hypothesis-driven manner. A scientist must first formulate a hypothesis based on his/her knowledge and experience, and then devise a variety of experiments to test it. Given the rapid growth of data, it has become virtually impossible for a person to manually inspect all the data to find all the interesting hypotheses for testing. In this paper, we propose and develop a data-driven system for automatic hypothesis testing and analysis. We define a hypothesis as a comparison between two or more sub-populations. We find sub-populations for comparison using frequent pattern mining techniques and then pair them up for statistical testing. We also generate additional information for further analysis of the hypotheses that are deemed significant. We conducted a set of experiments to show the efficiency of the proposed algorithms, and the usefulness of the generated hypotheses. The results show that our system can help users (1) identify significant hypotheses; (2) isolate the reasons behind significant hypotheses; and (3) find confounding factors that form Simpson's Paradoxes with discovered significant hypotheses.