Theoretical and Empirical Analysis of ReliefF and RReliefF

Authors:
Marko Robnik-Šikonja;Igor Kononenko
Affiliations:
University of Ljubljana, Faculty of Computer and Information Science, Tržaška 25, 1001 Ljubljana, Slovenia. Marko.Robnik@fri.uni-lj.si;University of Ljubljana, Faculty of Computer and Information Science, Tržaška 25, 1001 Ljubljana, Slovenia. Igor.Kononenko@fri.uni-lj.si
Venue:
Machine Learning
Year:
2003

Citing 18
Cited 161

Algorithms in C

Algorithms in C
Learning hard concepts through constructive induction: framework and rationale

Computational Intelligence
C4.5: programs for machine learning

C4.5: programs for machine learning
Estimating attributes: analysis and extensions of RELIEF

ECML-94 Proceedings of the European conference on machine learning on Machine Learning
Control-Sensitive Feature Selection for Lazy Learners

Artificial Intelligence Review - Special issue on lazy learning
A Review and Empirical Evaluation of Feature Weighting Methods for aClass of Lazy Learning Algorithms

Artificial Intelligence Review - Special issue on lazy learning
Multidimensional binary search trees used for associative searching

Communications of the ACM
Overcoming the Myopia of Inductive Learning Algorithms with RELIEFF

Applied Intelligence
Use of Contextual Information for Feature Ranking and Discretization

IEEE Transactions on Knowledge and Data Engineering
Induction of Decision Trees

Machine Learning
A Practical Approach to Feature Selection

ML '92 Proceedings of the Ninth International Workshop on Machine Learning
An adaptation of Relief for attribute estimation in regression

ICML '97 Proceedings of the Fourteenth International Conference on Machine Learning
Efficient Locally Weighted Polynomial Regression Predictions

ICML '97 Proceedings of the Fourteenth International Conference on Machine Learning
Attribute Dependencies, Understandability and Split Selection in Tree Based Models

ICML '99 Proceedings of the Sixteenth International Conference on Machine Learning
Learning a Local Similarity Metric for Case-Based Reasoning

ICCBR '95 Proceedings of the First International Conference on Case-Based Reasoning Research and Development
An Algroithm for Finding Best Matches in Logarithmic Expected Time

An Algroithm for Finding Best Matches in Logarithmic Expected Time
On biases in estimating multi-valued attributes

IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2
Multiresolution instance-based learning

IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2

Redundancy based feature selection for microarray data

Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining
Application of Relief-F Feature Filtering Algorithm to Selecting Informative Genes for Cancer Classification Using Microarray Data

CSB '04 Proceedings of the 2004 IEEE Computational Systems Bioinformatics Conference
A selective sampling approach to active feature selection

Artificial Intelligence
Efficient Feature Selection via Analysis of Relevance and Redundancy

The Journal of Machine Learning Research
Feature influence for evolutionary learning

GECCO '05 Proceedings of the 7th annual conference on Genetic and evolutionary computation
Stability of Feature Selection Algorithms

ICDM '05 Proceedings of the Fifth IEEE International Conference on Data Mining
Stability of Feature Selection Algorithms

ICDM '05 Proceedings of the Fifth IEEE International Conference on Data Mining
Feature Selection for Building Cost-Effective Data Stream Classifiers

ICDM '05 Proceedings of the Fifth IEEE International Conference on Data Mining
Quality assessment of individual classifications in machine learning and data mining

Knowledge and Information Systems
A case-based reasoning system for PCB principal process parameter identification

Expert Systems with Applications: An International Journal
A decision tree-based attribute weighting filter for naive Bayes

Knowledge-Based Systems
Evaluation of ordinal attributes at value level

Data Mining and Knowledge Discovery
Automatic web pages categorization with ReliefF and Hidden Naive Bayes

Proceedings of the 2007 ACM symposium on Applied computing
A combined MRI and MRSI based multiclass system for brain tumour recognition using LS-SVMs with class probabilities and feature selection

Artificial Intelligence in Medicine
Stability of feature selection algorithms: a study on high-dimensional spaces

Knowledge and Information Systems
Spectral feature selection for supervised and unsupervised learning

Proceedings of the 24th international conference on Machine learning
Direct integration of microarrays for selecting informative genes and phenotype classification

Information Sciences: an International Journal
Feature selection and classification model construction on type 2 diabetic patients' data

Artificial Intelligence in Medicine
Using genetic feature selection for improving cyber attack detection rate

ACST'07 Proceedings of the third conference on IASTED International Conference: Advances in Computer Science and Technology
Invariant optimal feature selection: A distance discriminant and feature ranking based solution

Pattern Recognition
Using expert knowledge in initialization for genome-wide analysis of epistasis using genetic programming

Proceedings of the 10th annual conference on Genetic and evolutionary computation
Stable feature selection via dense feature groups

Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Identifying biologically relevant genes via multiple heterogeneous data sources

Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Memetic Algorithms for Feature Selection on Microarray Data

ISNN '07 Proceedings of the 4th international symposium on Neural Networks: Advances in Neural Networks
Multi-resolution Image Parametrization in Stepwise Diagnostics of Coronary Artery Disease

AIME '07 Proceedings of the 11th conference on Artificial Intelligence in Medicine
An Integrated Approach for Identifying Relevant Factors Influencing Software Development Productivity

Balancing Agility and Formalism in Software Engineering
Locally Adaptive Neighborhood Selection for Collaborative Filtering Recommendations

AH '08 Proceedings of the 5th international conference on Adaptive Hypermedia and Adaptive Web-Based Systems
Ant Colony Optimization for Genome-Wide Genetic Analysis

ANTS '08 Proceedings of the 6th international conference on Ant Colony Optimization and Swarm Intelligence
LEARNING RANDOM NUMBERS: A MATLAB ANOMALY

Applied Artificial Intelligence
Biological pathways as features for microarray data classification

Proceedings of the 2nd international workshop on Data and text mining in bioinformatics
Automatic Web Page Classification Using Various Features

PCM '08 Proceedings of the 9th Pacific Rim Conference on Multimedia: Advances in Multimedia Information Processing
Feature selection with dynamic mutual information

Pattern Recognition
Forecasting market prices in a supply chain game

Electronic Commerce Research and Applications
Solving complex problems in human genetics using GP: challenges and opportunities

ACM SIGEVOlution
Optimal Use of Expert Knowledge in Ant Colony Optimization for the Analysis of Epistasis in Human Disease

EvoBIO '09 Proceedings of the 7th European Conference on Evolutionary Computation, Machine Learning and Data Mining in Bioinformatics
Detecting Abnormal Events via Hierarchical Dirichlet Processes

PAKDD '09 Proceedings of the 13th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining
The Effect of Varying Parameters and Focusing on Bus Travel Time Prediction

PAKDD '09 Proceedings of the 13th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining
The Impact of Gene Selection on Imbalanced Microarray Expression Data

BICoB '09 Proceedings of the 1st International Conference on Bioinformatics and Computational Biology
Parallel Selection of Informative Genes for Classification

BICoB '09 Proceedings of the 1st International Conference on Bioinformatics and Computational Biology
Searching for interacting features in subset selection

Intelligent Data Analysis
Supporting Diagnostics of Coronary Artery Disease with Multi-resolution Image Parameterization and Data Mining

MIRAGE '09 Proceedings of the 4th International Conference on Computer Vision/Computer Graphics CollaborationTechniques
Learning to recognize webpage genres

Information Processing and Management: an International Journal
Feature subset selection in large dimensionality domains

Pattern Recognition
Design of input vector for day-ahead price forecasting of electricity markets

Expert Systems with Applications: An International Journal
Ordinal Evaluation: A New Perspective on Country Images

ICDM '09 Proceedings of the 9th Industrial Conference on Advances in Data Mining. Applications and Theoretical Aspects
Improving Probabilistic Interpretation of Medical Diagnoses with Multi-resolution Image Parameterization: A Case Study

AIME '09 Proceedings of the 12th Conference on Artificial Intelligence in Medicine: Artificial Intelligence in Medicine
Ensemble Learning: A Study on Different Variants of the Dynamic Selection Approach

MLDM '09 Proceedings of the 6th International Conference on Machine Learning and Data Mining in Pattern Recognition
A General Framework of Feature Selection for Text Categorization

MLDM '09 Proceedings of the 6th International Conference on Machine Learning and Data Mining in Pattern Recognition
Trace ratio criterion for feature selection

AAAI'08 Proceedings of the 23rd national conference on Artificial intelligence - Volume 2
Searching for interacting features

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
OWA rough set model for forecasting the revenues growth rate of the electronic industry

Expert Systems with Applications: An International Journal
An optimization of ReliefF for classification in large datasets

Data & Knowledge Engineering
Classifying Web Pages by Genre: An n-Gram Approach

WI-IAT '09 Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology - Volume 01
ReliefMSS: a variation on a feature ranking ReliefF algorithm

International Journal of Business Intelligence and Data Mining
Efficient feature weighting methods for ranking

Proceedings of the 18th ACM conference on Information and knowledge management
Feature selection with biased sample distributions

IRI'09 Proceedings of the 10th IEEE international conference on Information Reuse & Integration
Sensible initialization using expert knowledge for genome-wide analysis of epistasis using genetic programming

CEC'09 Proceedings of the Eleventh conference on Congress on Evolutionary Computation
Pigmented Skin Lesions Classification Using Dermatoscopic Images

CIARP '09 Proceedings of the 14th Iberoamerican Conference on Pattern Recognition: Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications
Feature Selection for Gene Expression Using Model-Based Entropy

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Selecting discrete and continuous features based on neighborhood decision error minimization

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics
Reducing the number of DNA primers for classifying pejibaye palm races using SVM

E-ACTIVITIES'09/ISP'09 Proceedings of the 8th WSEAS International Conference on E-Activities and information security and privacy
Gaussian kernel based fuzzy rough sets: Model, uncertainty measures and applications

International Journal of Approximate Reasoning
Feature Selection with Ensembles, Artificial Variables, and Redundancy Elimination

The Journal of Machine Learning Research
Tuning ReliefF for genome-wide genetic analysis

EvoBIO'07 Proceedings of the 5th European conference on Evolutionary computation, machine learning and data mining in bioinformatics
An effective gene selection method based on relevance analysis and discernibility matrix

PAKDD'07 Proceedings of the 11th Pacific-Asia conference on Advances in knowledge discovery and data mining
Filter methods for feature selection: a comparative study

IDEAL'07 Proceedings of the 8th international conference on Intelligent data engineering and automated learning
An expert knowledge-guided mutation operator for genome-wide genetic analysis using genetic programming

PRIB'07 Proceedings of the 2nd IAPR international conference on Pattern recognition in bioinformatics
extraRelief: improving relief by efficient selection of instances

AI'07 Proceedings of the 20th Australian joint conference on Advances in artificial intelligence
Efficient feature selection in the presence of outliers and noises

AIRS'08 Proceedings of the 4th Asia information retrieval conference on Information retrieval technology
Extraction of informative genes from integrated microarray data

ISMIS'08 Proceedings of the 17th international conference on Foundations of intelligent systems
Feature selection by nonparametric Bayes error minimization

PAKDD'08 Proceedings of the 12th Pacific-Asia conference on Advances in knowledge discovery and data mining
Generalized iterative RELIEF for supervised distance metric learning

Pattern Recognition
Feature Selection via Maximizing Fuzzy Dependency

Fundamenta Informaticae
LIBGS: A MATLAB software package for gene selection

International Journal of Data Mining and Bioinformatics
Towards a memetic feature selection paradigm

IEEE Computational Intelligence Magazine
Empirical evaluation of feature selection methods in classification

Intelligent Data Analysis
Feature selection of RAPD haplotypes for identifying peach palm (Bactris gasipaes) landraces using SVM

WSEAS Transactions on Computers
Robust relief-feature weighting, margin maximization, and fuzzy optimization

IEEE Transactions on Fuzzy Systems
Quadratic Programming Feature Selection

The Journal of Machine Learning Research
Selecting useful features for personal credit risk analysis

International Journal of Business Information Systems
Support feature machine for DNA microarray data

RSCTC'10 Proceedings of the 7th international conference on Rough sets and current trends in computing
Margin based sample weighting for stable feature selection

WAIM'10 Proceedings of the 11th international conference on Web-age information management
On the relevance of automatically selected single-voxel MRS and multimodal MRI and MRSI features for brain tumour differentiation

Computers in Biology and Medicine
Empirical study of feature selection methods based on individual feature evaluation for classification problems

Expert Systems with Applications: An International Journal
Implementing ReliefF filters to extract meaningful features from genetic lifetime datasets

Journal of Biomedical Informatics
Measuring relevance between discrete and continuous features based on neighborhood mutual information

Expert Systems with Applications: An International Journal
Robust Feature Selection for Microarray Data Based on Multicriterion Fusion

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Correntropy based feature selection using binary projection

Pattern Recognition
Constrained laplacian score for semi-supervised feature selection

ECML PKDD'11 Proceedings of the 2011 European conference on Machine learning and knowledge discovery in databases - Volume Part I
Minimizing neighborhood evidential decision error for feature evaluation and selection based on evidence theory

Expert Systems with Applications: An International Journal
Research on CBR system based on data mining

Applied Soft Computing
Benefits and barriers of user evaluation in software engineering research

Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications
Toward a combined tool to assist dermatologists in melanoma detection from dermoscopic images of pigmented skin lesions

Pattern Recognition Letters
Image processing and machine learning for fully automated probabilistic evaluation of medical images

Computer Methods and Programs in Biomedicine
Input variable selection for feature extraction in classification problems

Signal Processing
Software fault localization using feature selection

Proceedings of the International Workshop on Machine Learning Technologies in Software Engineering
Stable Gene Selection from Microarray Data via Sample Weighting

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Exploiting expert knowledge in genetic programming for genome-wide genetic analysis

PPSN'06 Proceedings of the 9th international conference on Parallel Problem Solving from Nature
Using kNN model for automatic feature selection

ICAPR'05 Proceedings of the Third international conference on Advances in Pattern Recognition - Volume Part I
DClusterE: A Framework for Evaluating and Understanding Document Clustering Using Visualization

ACM Transactions on Intelligent Systems and Technology (TIST)
Selection of effective features for ECG beat recognition based on nonlinear correlations

Artificial Intelligence in Medicine
Fuzzy criteria for feature selection

Fuzzy Sets and Systems
Feature subset selection with cumulate conditional mutual information minimization

Expert Systems with Applications: An International Journal
The informative extremes: using both nearest and farthest individuals can improve relief algorithms in the domain of human genetics

EvoBIO'10 Proceedings of the 8th European conference on Evolutionary Computation, Machine Learning and Data Mining in Bioinformatics
Machine learning for query formulation in question answering

Natural Language Engineering
Feature selection algorithm for data with both nominal and continuous features

PAKDD'05 Proceedings of the 9th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining
Derivation of an artificial gene to improve classification accuracy upon gene selection

Computational Biology and Chemistry
Conquering the curse of dimensionality in gene expression cancer diagnosis: tough problem, simple models

AIME'05 Proceedings of the 10th conference on Artificial Intelligence in Medicine
Improving example selection for agents teaching ontology concepts

CIA'06 Proceedings of the 10th international conference on Cooperative Information Agents
Feature selection based on run covering

PSIVT'06 Proceedings of the First Pacific Rim conference on Advances in Image and Video Technology
Energy-based feature selection and its ensemble version

ICONIP'11 Proceedings of the 18th international conference on Neural Information Processing - Volume Part II
A minority class feature selection method

CIARP'11 Proceedings of the 16th Iberoamerican Congress conference on Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications
Feature evaluation and selection with cooperative game theory

Pattern Recognition
Feature selection for dimensionality reduction

SLSFS'05 Proceedings of the 2005 international conference on Subspace, Latent Structure and Feature Selection
Computer---Aided diagnosis of pigmented skin dermoscopic images

MCBR-CDS'11 Proceedings of the Second MICCAI international conference on Medical Content-Based Retrieval for Clinical Decision Support
Feature selection using structural similarity

Information Sciences: an International Journal
Use of machine learning techniques for educational proposes: a decision support system for forecasting students' grades

Artificial Intelligence Review
Large-margin feature selection for monotonic classification

Knowledge-Based Systems
Feature selection for MAUC-oriented classification systems

Neurocomputing
Analyzing Online Review Helpfulness Using a Regressional ReliefF-Enhanced Text Mining Method

ACM Transactions on Management Information Systems (TMIS)
Improving the ranking quality of medical image retrieval using a genetic feature selection method

Decision Support Systems
A RELIEF-based modality weighting approach for multimodal information retrieval

Proceedings of the 2nd ACM International Conference on Multimedia Retrieval
Evolutionary feature selection for classification: a plug-in hybrid vehicle adoption application

Proceedings of the 14th annual conference on Genetic and evolutionary computation
A human study of patch maintainability

Proceedings of the 2012 International Symposium on Software Testing and Analysis
Model mining for robust feature selection

Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
Integrating a differential evolution feature weighting scheme into prototype generation

Neurocomputing
An adaption of relief for redundant feature elimination

ISNN'12 Proceedings of the 9th international conference on Advances in Neural Networks - Volume Part II
Memetic three-dimensional gabor feature extraction for hyperspectral imagery classification

ICSI'12 Proceedings of the Third international conference on Advances in Swarm Intelligence - Volume Part I
Ensemble approaches for regression: A survey

ACM Computing Surveys (CSUR)
Evolutionary generation of neural network classifiers-An empirical comparison

Neurocomputing
A variance reduction framework for stable feature selection

Statistical Analysis and Data Mining
Using data mining to enable integration of wind resources on the power grid

Statistical Analysis and Data Mining
Divergence-based feature selection for separate classes

Neurocomputing
Machine learning methods to forecast temperature in buildings

Expert Systems with Applications: An International Journal
Feature selection using dynamic weights for classification

Knowledge-Based Systems
Explaining subgroups through ontologies

PRICAI'12 Proceedings of the 12th Pacific Rim international conference on Trends in Artificial Intelligence
Hypergraph spectra for semi-supervised feature selection

ECML PKDD'12 Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases - Volume Part I
Massively parallel feature selection: an approach based on variance preservation

ECML PKDD'12 Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases - Volume Part I
Improving robustness of gene ranking by multi-criterion combination with novel gene importance transformation

International Journal of Data Mining and Bioinformatics
A novel divide-and-merge classification for high dimensional datasets

Computational Biology and Chemistry
Filter approach feature selection methods to support multi-label learning based on relieff and information gain

SBIA'12 Proceedings of the 21st Brazilian conference on Advances in Artificial Intelligence
GARF: towards self-optimised random forests

ICONIP'12 Proceedings of the 19th international conference on Neural Information Processing - Volume Part II
Feature selection for electricity load prediction

ICONIP'12 Proceedings of the 19th international conference on Neural Information Processing - Volume Part II
RFS: Efficient feature selection method based on R-value

Computers in Biology and Medicine
A scalable approach to simultaneous evolutionary instance and feature selection

Information Sciences: an International Journal
Minimum-maximum local structure information for feature selection

Pattern Recognition Letters
A Comparison of Multi-label Feature Selection Methods using the Problem Transformation Approach

Electronic Notes in Theoretical Computer Science (ENTCS)
Power and Performance Management of GPUs Based Cluster

International Journal of Cloud Applications and Computing
Self-adaptive differential evolution for feature selection in hyperspectral image data

Applied Soft Computing
Unsupervised Feature Selection with Feature Clustering

WI-IAT '12 Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 01
Selection of interdependent genes via dynamic relevance analysis for cancer diagnosis

Journal of Biomedical Informatics
Effective search for genetic-based machine learning systems via estimation of distribution algorithms and embedded feature reduction techniques

Neurocomputing
Large Margin Subspace Learning for feature selection

Pattern Recognition
G-Optimal Feature Selection with Laplacian regularization

Neurocomputing
Machine learning based typology development in archaeology

Journal on Computing and Cultural Heritage (JOCCH)
Spatial distance join based feature selection

Engineering Applications of Artificial Intelligence
A new matching strategy for content based image retrieval system

Applied Soft Computing
A feature subset selection algorithm automatic recommendation method

Journal of Artificial Intelligence Research
Learning attribute weighted AODE for ROC area ranking

International Journal of Information and Communication Technology
Learning dependence from samples

International Journal of Bioinformatics Research and Applications
A random forest classifier for lymph diseases

Computer Methods and Programs in Biomedicine

Quantified Score

Hi-index	0.01

Visualization

Abstract

Relief algorithms are general and successful attribute estimators. They are able to detect conditional dependencies between attributes and provide a unified view on the attribute estimation in regression and classification. In addition, their quality estimates have a natural interpretation. While they have commonly been viewed as feature subset selection methods that are applied in prepossessing step before a model is learned, they have actually been used successfully in a variety of settings, e.g., to select splits or to guide constructive induction in the building phase of decision or regression tree learning, as the attribute weighting method and also in the inductive logic programming.A broad spectrum of successful uses calls for especially careful investigation of various features Relief algorithms have. In this paper we theoretically and empirically investigate and discuss how and why they work, their theoretical and practical properties, their parameters, what kind of dependencies they detect, how do they scale up to large number of examples and features, how to sample data for them, how robust are they regarding the noise, how irrelevant and redundant attributes influence their output and how different metrics influences them.