Measuring relevance between discrete and continuous features based on neighborhood mutual information

Authors:
Qinghua Hu;Lei Zhang;David Zhang;Wei Pan;Shuang An;Witold Pedrycz
Affiliations:
Harbin Institute of Technology, PO 458, Harbin 150001, PR China and Department of Computing, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, China;Department of Computing, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, China;Department of Computing, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, China;Department of Computing, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, China;Harbin Institute of Technology, PO 458, Harbin 150001, PR China;Department of Electrical and Computer Engineering, University of Alberta, Canada
Venue:
Expert Systems with Applications: An International Journal
Year:
2011

Citing 30
Cited 4

Small Sample Size Effects in Statistical Pattern Recognition: Recommendations for Practitioners

IEEE Transactions on Pattern Analysis and Machine Intelligence
On the Handling of Continuous-Valued Attributes in Decision Tree Generation

Machine Learning
Statistical evaluation of rough set dependency analysis

International Journal of Human-Computer Studies
A Review and Empirical Evaluation of Feature Weighting Methods for aClass of Lazy Learning Algorithms

Artificial Intelligence Review - Special issue on lazy learning
Axiomatic Approach to Feature Subset Selection Based on Relevance

IEEE Transactions on Pattern Analysis and Machine Intelligence
Rough Sets: Theoretical Aspects of Reasoning about Data

Rough Sets: Theoretical Aspects of Reasoning about Data
Discretization: An Enabling Technique

Data Mining and Knowledge Discovery
A Formalism for Relevance and Its Application in Feature Subset Selection

Machine Learning
Feature Selection via Discretization

IEEE Transactions on Knowledge and Data Engineering
Input Feature Selection by Mutual Information Based on Parzen Window

IEEE Transactions on Pattern Analysis and Machine Intelligence
Induction of Decision Trees

Machine Learning
Correlation-based Feature Selection for Discrete and Numeric Class Machine Learning

ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning
Theoretical and Empirical Analysis of ReliefF and RReliefF

Machine Learning
An introduction to variable and feature selection

The Journal of Machine Learning Research
Consistency-based search in feature selection

Artificial Intelligence
Efficient Feature Selection via Analysis of Relevance and Redundancy

The Journal of Machine Learning Research
Fast Binary Feature Selection with Conditional Mutual Information

The Journal of Machine Learning Research
Toward Integrating Feature Selection Algorithms for Classification and Clustering

IEEE Transactions on Knowledge and Data Engineering
Feature Selection Based on Mutual Information: Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy

IEEE Transactions on Pattern Analysis and Machine Intelligence
Nearest Neighbors by Neighborhood Counting

IEEE Transactions on Pattern Analysis and Machine Intelligence
Information-preserving hybrid data reduction based on fuzzy-rough techniques

Pattern Recognition Letters
Short communication: Uncertainty measures for fuzzy relations and their applications

Applied Soft Computing
Hybrid attribute reduction based on a novel fuzzy-rough model and information granulation

Pattern Recognition
EROS: Ensemble rough subspaces

Pattern Recognition
A parameterless feature ranking algorithm based on MI

Neurocomputing
Attribute reduction in decision-theoretic rough set models

Information Sciences: an International Journal
Neighborhood rough set based heterogeneous feature subset selection

Information Sciences: an International Journal
Knowledge structure, knowledge granulation and knowledge distance in a knowledge base

International Journal of Approximate Reasoning
Input feature selection for classification problems

IEEE Transactions on Neural Networks
Using mutual information for selecting features in supervised neural net learning

IEEE Transactions on Neural Networks

Nearest neighbor estimate of conditional mutual information in feature selection

Expert Systems with Applications: An International Journal
Quantitative analysis for covering-based rough sets through the upper approximation number

Information Sciences: an International Journal
Characteristic matrix of covering and its application to Boolean matrix decomposition

Information Sciences: an International Journal
Quality of information-based source assessment and selection

Neurocomputing

Quantified Score

Hi-index	12.05

Visualization

Abstract

Measures of relevance between features play an important role in classification and regression analysis. Mutual information has been proved an effective measure for decision tree construction and feature selection. However, there is a limitation in computing relevance between numerical features with mutual information due to problems of estimating probability density functions in high-dimensional spaces. In this work, we generalize Shannon's information entropy to neighborhood information entropy and propose a measure of neighborhood mutual information. It is shown that the new measure is a natural extension of classical mutual information which reduces to the classical one if features are discrete; thus the new measure can also be used to compute the relevance between discrete variables. In addition, the new measure introduces a parameter delta to control the granularity in analyzing data. With numeric experiments, we show that neighborhood mutual information produces the nearly same outputs as mutual information. However, unlike mutual information, no discretization is required in computing relevance when used the proposed algorithm. We combine the proposed measure with four classes of evaluating strategies used for feature selection. Finally, the proposed algorithms are tested on several benchmark data sets. The results show that neighborhood mutual information based algorithms yield better performance than some classical ones.