Statistical analysis with missing data
Statistical analysis with missing data
Mining Imperfect Data: Dealing with Contamination and Incomplete Records
Mining Imperfect Data: Dealing with Contamination and Incomplete Records
The problem of disguised missing data
ACM SIGKDD Explorations Newsletter
Further experimental evidence against the utility of Occam's razor
Journal of Artificial Intelligence Research
DiMaC: a system for cleaning disguised missing data
Proceedings of the 2008 ACM SIGMOD international conference on Management of data
DiMaC: a disguised missing data cleaning tool
Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Exploiting contexts to deal with uncertainty in classification
Proceedings of the 1st ACM SIGKDD Workshop on Knowledge Discovery from Uncertain Data
Information enhancement for data mining
Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery
| Hi-index | 0.00 |
In some applications such as filling in a customer information form on the web, some missing values may not be explicitly represented as such, but instead appear as potentially valid data values. Such missing values are known as disguised missing data, which may impair the quality of data analysis severely, such as causing significant biases and misleading results in hypothesis tests, correlation analysis and regressions. The very limited previous studies on cleaning disguised missing data use outlier mining and distribution anomaly detection. They highly rely on domain background knowledge in specific applications and may not work well for the cases where the disguise values are inliers. To tackle the problem of cleaning disguised missing data, in this paper, we first model the distribution of disguised missing data, and propose the embedded unbiased sample heuristic. Then, we develop an effective and efficient method to identify the frequently used disguise values which capture the major body of the disguised missing data. Our method does not require any domain background knowledge to find the suspicious disguise values. We report an empirical evaluation using real data sets, which shows that our method is effective - the frequently used disguise values found by our method match the values identified by the domain experts nicely. Our method is also efficient and scalable for processing large data sets.