Building a dynamic classifier for large text data collections

  • Authors:

  • Pavel Kalinov;Bela Stantic;Abdul Sattar

  • Affiliations:

  • Griffith University, Brisbane, Australia;Griffith University, Brisbane, Australia;Griffith University, Brisbane, Australia

  • Venue:
  • ADC '10 Proceedings of the Twenty-First Australasian Conference on Database Technologies - Volume 104
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

Due to the lack of in-built tools to navigate the web, people have to use external solutions to find information. The most popular of these are search engines and web directories. Search engines allow users to locate specific information about a particular topic, whereas web directories facilitate exploration over a wider topic. In the recent past, statistical machine learning methods have been successfully exploited in search engines. Web directories remained in their primitive state, which resulted in their decline. Exploration however is a task which answers a different information need of the user and should not be neglected. Web directories should provide a user experience of the same quality as search engines. Their development by machine learning methods however is hindered by the noisy nature of the web, which makes text classifiers unreliable when applied to web data. In this paper we propose Stochastic Prior Distribution Adjustment (SPDA) - a variation of the Multinomial Naïve Bayes (MNB) classifier which makes it more suitable to classify real-world data. By stochastically adjusting class prior distributions we achieve a better overall success rate, but more importantly we also significantly improve error distribution across classes, making the classifier equally reliable for all classes and therefore more usable.