An improved clustering validity index for determining the number of malware clusters

• Authors: Youyu Wang;Yanfang Ye;Haishan Chen;Qingshan Jiang
• Venue: ASID'09 Proceedings of the 3rd international conference on Anti-Counterfeiting, security, and identification in communication
• Year: 2009

Nowadays, along with the development of the malware writing techniques, the diversity and amount of malware variants are constantly increasing and proliferation of these malware has posed major threats to the network computer users. However, variants of malware families always share typical characteristics which reflect their origin and purpose. Therefore, categorizing malware to different families is one of the computer security topics that are of great interest. A fundamental and difficult problem for malware clustering is the determination of the "true" number of malware families on real data collection. To address this issue, in this paper, resting on the analysis of the extracted instruction of malware samples, we propose an improved clustering validity index VNFS for determining the number of malware clusters based on hierarchical clustering algorithm for malware categorization. VNFS is defined based on the concepts of the compactness within each cluster and distances of the clusters representatives. By using a novel function proposed in this paper for total separation between clusters, VNFS can deal with different density and irregular data sets more robustly. A comprehensive experimental study on real daily collections of PE malware samples is performed to compare various clustering validation measure methods, such as RS-RMSSTD, PST2, VFS and VXB. Promising experimental results demonstrate that our proposed clustering validity index VNFS always leads to better cluster number than other clustering validity indices for hierarchical clustering and the number of malware clusters it recommends is almost the same as the virus analysts suggest.