Unsupervised data pruning for clustering of noisy data

  • Authors:
  • Yi Hong;Sam Kwong;Yuchou Chang;Qingsheng Ren

  • Affiliations:
  • Department of Computer Science, City University of Hong Kong, Tat Chee Avenue, Kowloon, Hong Kong;Department of Computer Science, City University of Hong Kong, Tat Chee Avenue, Kowloon, Hong Kong;Department of Electrical and Computer Engineering, Brigham Young University, USA;Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, PR China

  • Venue:
  • Knowledge-Based Systems
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

Data pruning works with identifying noisy instances of a data set and removing them from the data set in order to improve the generalization of a learning algorithm. It has been well studied in supervised classification where the identification and removal of noisy instances are guided by available labels of instances. However, to the best knowledge of the authors', very few work has been done on data pruning for unsupervised clustering. This paper deals with the problem of data pruning for unsupervised clustering under the condition that labels of instances are unknown beforehand. We claim that unsupervised data pruning can benefit for the clustering of the data with noise. We propose a feasible approach, termed as unsupervised Data Pruning using Ensembles of multiple Clusterers (DPEC), to identify noisy instances of a data set. DPEC checks all instances of a data set and identifies noisy instances by using ensembles of multiple clustering results provided by different clusterers on the same data set. We test the performance of DPEC on several real data sets with artificial noise. Experimental results demonstrate that DPEC is often able to improve the accuracy and robustness of the clustering algorithm.