Evaluation methods and strategies for the interactive use of classifiers

Authors:
Silvia Acid;Luis M. de Campos;Moisés Fernández
Affiliations:
Departamento de Ciencias de la Computación e Inteligencia Artificial, E.T.S.I. Informática y de Telecomunicación, CITIC-UGR Universidad de Granada, 18071 Granada, Spain;Departamento de Ciencias de la Computación e Inteligencia Artificial, E.T.S.I. Informática y de Telecomunicación, CITIC-UGR Universidad de Granada, 18071 Granada, Spain;Departamento de Ciencias de la Computación e Inteligencia Artificial, E.T.S.I. Informática y de Telecomunicación, CITIC-UGR Universidad de Granada, 18071 Granada, Spain
Venue:
International Journal of Human-Computer Studies
Year:
2012

Citing 17
Cited 1

Instance-Based Learning Algorithms

Machine Learning
The weighted majority algorithm

Information and Computation
A sequential algorithm for training text classifiers

SIGIR '94 Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval
Improving Generalization with Active Learning

Machine Learning - Special issue on structured connectionist systems
Learning Bayesian Networks: The Combination of Knowledge and Statistical Data

Machine Learning
Some label efficient learning results

COLT '97 Proceedings of the tenth annual conference on Computational learning theory
Hierarchical classification of Web content

SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Machine learning in automated text categorization

ACM Computing Surveys (CSUR)
Interactive machine learning: letting users build classifiers

International Journal of Human-Computer Studies
Neural Networks: A Comprehensive Foundation

Neural Networks: A Comprehensive Foundation
Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm

Machine Learning
Statistical Comparisons of Classifiers over Multiple Data Sets

The Journal of Machine Learning Research
Argument based machine learning

Artificial Intelligence
Interacting meaningfully with machine learning systems: Three experiments

International Journal of Human-Computer Studies
Bayesian network models for hierarchical text classification from a thesaurus

International Journal of Approximate Reasoning
An analysis of active learning strategies for sequence labeling tasks

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
The WEKA data mining software: an update

ACM SIGKDD Explorations Newsletter

Simulating user intervention for interactive semantic place recognition with mobile devices

Proceedings of the 2012 RecSys workshop on Personalizing the local mobile experience

Quantified Score

Hi-index	0.00

Visualization

Abstract

We consider the scenario in which an automatic classifier (previously built) is available. It is used to classify new instances but, in some cases, the classifier may request the intervention of a human (the oracle), who gives it the correct class. In this scenario, first it is necessary to study how the performance of the system should be evaluated, as it cannot be based solely on the predictive accuracy obtained by the classifier but it should also take into account the cost of the human intervention; second, studying the concrete circumstances under which the classifier decides to query the oracle is also important. In this paper we study these two questions and include also an experimental evaluation of the different proposed alternatives.