Multi-label classification with a reject option

Authors:
Ignazio Pillai;Giorgio Fumera;Fabio Roli
Affiliations:
Department of Electrical and Electronic Engineering, University of Cagliari, Piazza d'Armi, 09123 Cagliari, Italy;Department of Electrical and Electronic Engineering, University of Cagliari, Piazza d'Armi, 09123 Cagliari, Italy;Department of Electrical and Electronic Engineering, University of Cagliari, Piazza d'Armi, 09123 Cagliari, Italy
Venue:
Pattern Recognition
Year:
2013

Citing 16
Cited 0

A re-examination of text categorization methods

Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval
A study of thresholding strategies for text categorization

Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Machine learning in automated text categorization

ACM Computing Surveys (CSUR)
Classification with Reject Option in Text Categorisation Systems

ICIAP '03 Proceedings of the 12th International Conference on Image Analysis and Processing
RCV1: A New Benchmark Collection for Text Categorization Research

The Journal of Machine Learning Research
Survey of semantic annotation platforms

Proceedings of the 2005 ACM symposium on Applied computing
A ROC-based reject rule for dichotomizers

Pattern Recognition Letters
On the use of ROC analysis for the optimization of abstaining classifiers

Machine Learning
An efficient manual image annotation approach based on tagging and browsing

Workshop on multimedia information retrieval on The many faces of multimedia semantics
Multi-label Classification Using Ensembles of Pruned Sets

ICDM '08 Proceedings of the 2008 Eighth IEEE International Conference on Data Mining
MULAN: A Java Library for Multi-Label Learning

The Journal of Machine Learning Research
A classification approach with a reject option for multi-label problems

ICIAP'11 Proceedings of the 16th international conference on Image analysis and processing: Part I
On optimum recognition error and reject tradeoff

IEEE Transactions on Information Theory
Multilabel classifiers with a probabilistic thresholding strategy

Pattern Recognition
On label dependence and loss minimization in multi-label classification

Machine Learning
Threshold optimisation for multi-label classifiers

Pattern Recognition

Quantified Score

Hi-index	0.01

Visualization

Abstract

We consider multi-label classification problems in application scenarios where classifier accuracy is not satisfactory, but manual annotation is too costly. In single-label problems, a well known solution consists of using a reject option, i.e., allowing a classifier to withhold unreliable decisions, leaving them (and only them) to human operators. We argue that this solution can be exploited also in multi-label problems. However, the current theoretical framework for classification with a reject option applies only to single-label problems. We thus develop a specific framework for multi-label ones. In particular, we extend multi-label accuracy measures to take into account rejections, and define manual annotation cost as a cost function. We then formalise the goal of attaining a desired trade-off between classifier accuracy on non-rejected decisions, and the cost of manually handling rejected decisions, as a constrained optimisation problem. We finally develop two possible implementations of our framework, tailored to the widely used F accuracy measure, and to the only cost models proposed so far for multi-label annotation tasks, and experimentally evaluate them on five application domains.