Machine learning in automated text categorization
ACM Computing Surveys (CSUR)
Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms
Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms
A Comparative Study on Feature Selection in Text Categorization
ICML '97 Proceedings of the Fourteenth International Conference on Machine Learning
Feature Selection for Unbalanced Class Distribution and Naive Bayes
ICML '99 Proceedings of the Sixteenth International Conference on Machine Learning
Editorial: special issue on learning from imbalanced data sets
ACM SIGKDD Explorations Newsletter - Special issue on learning from imbalanced datasets
Feature selection for text categorization on imbalanced data
ACM SIGKDD Explorations Newsletter - Special issue on learning from imbalanced datasets
Bias Analysis in Text Classification for Highly Skewed Data
ICDM '05 Proceedings of the Fifth IEEE International Conference on Data Mining
A novel feature selection algorithm for text categorization
Expert Systems with Applications: An International Journal
An improved centroid classifier for text categorization
Expert Systems with Applications: An International Journal
Feature selection with a measure of deviations from Poisson in text categorization
Expert Systems with Applications: An International Journal
Comparison of metrics for feature selection in imbalanced text classification
Expert Systems with Applications: An International Journal
Expert Systems with Applications: An International Journal
| Hi-index | 12.05 |
In the previous paper (Ogura, H., Amano, H., & Kondo, M. (2009). Feature selection with a measure of deviations from Poisson in text categorization. Expert Systems with Applications, 36, 6826-6832.), we proposed a new metric, @g"P^2, for selecting features in text classification which estimates term importance based on how largely the probability distribution of a considered term deviates from the standard Poisson distribution. In this study, to establish the validity and advantage of using @g"P^2, we conducted experiments of automatic text classification on 20 NewsGroups data collection with binary setting. In the experiments, other three metrics for feature selection, i.e., Gini index, @g^2 statistic and information gain, were also used for comparison. From the results, it was confirmed that @g"P^2 and Gini index are much better than @g^2 statistic and information gain in terms of F"1 performance when they handle imbalanced data set. Furthermore, through another experiment in which the degree of imbalance in class distribution was explicitly controlled, we clarified that the origin of the superiority of @g"P^2 and Gini index is their ability to pick up suitable negative features in imbalanced data set. The ability of these two metrics to select suitable negative features is explained based on the analysis of their limiting behaviors at some extreme cases.