Usher: Improving Data Quality with Dynamic Forms

Authors:
Kuang Chen;Harr Chen;Neil Conway;Joseph M. Hellerstein;Tapan S. Parikh
Affiliations:
University of California, Berkeley, Berkeley;Massachusetts Institute of Technology, Cambridge;University of California, Berkeley, Berkeley;University of California, Berkeley, Berkeley;University of California, Berkeley, Berkeley
Venue:
IEEE Transactions on Knowledge and Data Engineering
Year:
2011

Citing 0
Cited 5

Automated quality control for mobile data collection

Proceedings of the 2nd ACM Symposium on Computing for Development
Shreddr: pipelined paper digitization for low-resource organizations

Proceedings of the 2nd ACM Symposium on Computing for Development
Analysis of the impact of errors made during health data collection using mobile phones: exploring error modeling and automatic diagnosis

Proceedings of the 3rd ACM Symposium on Computing for Development
Using behavioral data to identify interviewer fabrication in surveys

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Towards operationalizing outlier detection in community health programs

Proceedings of the Sixth International Conference on Information and Communications Technologies and Development: Notes - Volume 2

Quantified Score

Hi-index	0.00

Visualization

Abstract

Data quality is a critical problem in modern databases. data-entry forms present the first and arguably best opportunity for detecting and mitigating errors, but there has been little research into automatic methods for improving data quality at entry time. In this paper, we propose Usher, an end-to-end system for form design, entry, and data quality assurance. Using previous form submissions, Usher learns a probabilistic model over the questions of the form. Usher then applies this model at every step of the data-entry process to improve data quality. Before entry, it induces a form layout that captures the most important data values of a form instance as quickly as possible and reduces the complexity of error-prone questions. During entry, it dynamically adapts the form to the values being entered by providing real-time interface feedback, reasking questions with dubious responses, and simplifying questions by reformulating them. After entry, it revisits question responses that it deems likely to have been entered incorrectly by reasking the question or a reformulation thereof. We evaluate these components of Usher using two real-world data sets. Our results demonstrate that Usher can improve data quality considerably at a reduced cost when compared to current practice.