Complementing data in the ETL process

  • Authors:
  • Lívia de S. Ribeiro;Ronaldo R. Goldschmidt;Maria Cláudia Cavalcanti

  • Affiliations:
  • Instituto Militar de Engenharia, Praça General Tiburcio, Praia Vermelha, Rio de Janeiro, RJ;Universidade Federal Rural do Rio de Janeiro, Nova Iguaçu, RJ;Instituto Militar de Engenharia, Praça General Tiburcio, Praia Vermelha, Rio de Janeiro, RJ

  • Venue:
  • DaWaK'11 Proceedings of the 13th international conference on Data warehousing and knowledge discovery
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

Data quality in a typical Data Warehouse (DW) environment is critical. The process of transferring data from different sources into the DW environment, known as ETL (Extraction, Transformation, and Load), usually takes care of improving the data quality. However, it is not unusual to identify null values in a DW fact table during the ETL process, and this may impact negatively on the accuracy of data analyses results. Data imputation1 techniques are commonly used for dealing with the missing value problem. Some of them observe table values to generate a new value for the missing one. This paper proposes a new strategy to address the missing data problem on the ETL process. The idea is to enrich the DW fact table with dimension attributes, in order to reach better imputation results. The strategy uses the k-NN algorithm as the imputation approach. Tests performed on an implemented prototype showed promising results with respect to imputation quality.