On challenges in evaluating malware clustering

  • Authors:
  • Peng Li;Limin Liu;Debin Gao;Michael K. Reiter

  • Affiliations:
  • Department of Computer Science, University of North Carolina, Chapel Hill, NC;State Key Lab of Information Security, Graduate School of Chinese Academy of Sciences;School of Information Systems, Singapore Management University, Singapore;Department of Computer Science, University of North Carolina, Chapel Hill, NC

  • Venue:
  • RAID'10 Proceedings of the 13th international conference on Recent advances in intrusion detection
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

Malware clustering and classification are important tools that enable analysts to prioritize their malware analysis efforts. The recent emergence of fully automated methods for malware clustering and classification that report high accuracy suggests that this problem may largely be solved. In this paper, we report the results of our attempt to confirm our conjecture that the method of selecting ground-truth data in prior evaluations biases their results toward high accuracy. To examine this conjecture, we apply clustering algorithms from a different domain (plagiarism detection), first to the dataset used in a prior work's evaluation and then to a wholly new malware dataset, to see if clustering algorithms developed without attention to subtleties of malware obfuscation are nevertheless successful. While these studies provide conflicting signals as to the correctness of our conjecture, our investigation of possible reasons uncovers, we believe, a cautionary note regarding the significance of highly accurate clustering results, as can be impacted by testing on a dataset with a biased cluster-size distribution.