An investigation of clustering algorithms in the identification of similar web pages
Journal of Web Engineering
| Hi-index | 0.00 |
Automatic knowledge discovery from texts (KDT) is proving to be a promising method for businesses today to deal with the overload of textual information. In this paper, we first explore the possibilities for KDT to enhance communication in virtual communities, and then we present a practical case study with real-life Internet data. The problem in the case study is to manage the very successful virtual communities known as 'clubs' of the largest Dutch Internet Service Provider. It is possible for anyone to start a club about any subject, resulting in over 10,000 active clubs today. At the beginning, the founder assigns the club to a predefined category. This often results in illogical or inconsistent placements, which means that interesting clubs may be hard to locate for potential new members. The ISP therefore is looking for an automated way to categorize clubs in a logical and consistent manner. The method used is the so-called bag-of-words approach, previously applied mostly to scientific texts and structured documents. Each club is described by a vector of word occurrences of all communications within that club. Latent Semantic Indexing (LSI) is applied to reduce the dimensionality problem prior to clustering. Clustering is done by the Within Groups Clustering method using a cosine distance measure appropriate for texts. The results show that KDT and the LSI method can successfully be applied for clustering the very volatile and unstructured textual communication on the Internet.