Small Sample Size Effects in Statistical Pattern Recognition: Recommendations for Practitioners

Authors:
Sarunas J. Raudys;Anil K. Jain
Affiliations:
-;-
Venue:
IEEE Transactions on Pattern Analysis and Machine Intelligence
Year:
1991

Citing 2
Cited 149

Recent advances in error rate estimation

Pattern Recognition Letters
Bootstrap Techniques for Error Estimation

IEEE Transactions on Pattern Analysis and Machine Intelligence

On the Behavior of Artificial Neural Network Classifiers in High-Dimensional Spaces

IEEE Transactions on Pattern Analysis and Machine Intelligence
Feature Selection: Evaluation, Application, and Small Sample Performance

IEEE Transactions on Pattern Analysis and Machine Intelligence
Expected classification error of the Fisher linear classifier with pseudo-inverse covariance matrix

Pattern Recognition Letters
A Handwritten Character Recognition System Using Directional Element Feature and Asymmetric Mahalanobis Distance

IEEE Transactions on Pattern Analysis and Machine Intelligence
Statistical Pattern Recognition: A Review

IEEE Transactions on Pattern Analysis and Machine Intelligence
A Noise-Adaptive Discriminant Function and Its Application to Blurred Machine-Printed Kanji Recognition

IEEE Transactions on Pattern Analysis and Machine Intelligence
First-Order Tree-Type Dependence between Variables and Classification Performance

IEEE Transactions on Pattern Analysis and Machine Intelligence
Complexity Measures of Supervised Classification Problems

IEEE Transactions on Pattern Analysis and Machine Intelligence
Unsupervised Learning of Finite Mixture Models

IEEE Transactions on Pattern Analysis and Machine Intelligence
Problems in pattern classification in high domensional spaces: behavior of a class of combined neuro-fuzzy classifiers

Fuzzy Sets and Systems - Featured Issue: Selected papers from ACIDCA 2000
Off-Line Handwritten Word Recognition Using a Hidden Markov Model Type Stochastic Network

IEEE Transactions on Pattern Analysis and Machine Intelligence
An approach to the evaluation of the performance of a discrete classifier

Pattern Recognition Letters
The Optimum Classifier and the Performance Evaluation by Bayesian Approach

Proceedings of the Joint IAPR International Workshops on Advances in Pattern Recognition
Classification of SPECT Images of Normal Subjects versus Images of Alzheimer's Disease Patients

MICCAI '01 Proceedings of the 4th International Conference on Medical Image Computing and Computer-Assisted Intervention
Data Complexity Analysis for Classifier Combination

MCS '01 Proceedings of the Second International Workshop on Multiple Classifier Systems
Decision-Level Fusion in Fingerprint Verification

MCS '01 Proceedings of the Second International Workshop on Multiple Classifier Systems
Adaptive Feature Spaces for Land Cover Classification with Limited Ground Truth Data

MCS '02 Proceedings of the Third International Workshop on Multiple Classifier Systems
Complexity of Classification Problems and Comparative Advantages of Combined Classifiers

MCS '00 Proceedings of the First International Workshop on Multiple Classifier Systems
Feature Selection for Classification Using Genetic Algorithms with a Novel Encoding

CAIP '01 Proceedings of the 9th International Conference on Computer Analysis of Images and Patterns
The influence of prior knowledge on the expected performance of a classifier

Pattern Recognition Letters
A General Model for Finite-Sample Effects in Training and Testing of Competing Classifiers

IEEE Transactions on Pattern Analysis and Machine Intelligence
Exploitation of Unlabeled Sequences in Hidden Markov Models

IEEE Transactions on Pattern Analysis and Machine Intelligence
Regularized discriminant analysis for the small sample size problem in face recognition

Pattern Recognition Letters
Results in statistical discriminant analysis: a review of the former Soviet union literature

Journal of Multivariate Analysis
Simultaneous Feature Selection and Clustering Using Mixture Models

IEEE Transactions on Pattern Analysis and Machine Intelligence
Classifier geometrical characteristic comparison and its application in classifier selection

Pattern Recognition Letters
Estimates of Error Probability for Complex Gaussian Channels with Generalized Likelihood Ratio Detection

IEEE Transactions on Pattern Analysis and Machine Intelligence
Robust and Accurate Cancer Classification with Gene Expression Profiling

CSB '05 Proceedings of the 2005 IEEE Computational Systems Bioinformatics Conference
Regularization studies of linear discriminant analysis in small sample size scenarios with application to face recognition

Pattern Recognition Letters
Privacy leakage in multi-relational databases via pattern based semi-supervised learning

Proceedings of the 14th ACM international conference on Information and knowledge management
Learning Weighted Metrics to Minimize Nearest-Neighbor Classification Error

IEEE Transactions on Pattern Analysis and Machine Intelligence
Bounds for validation

Fundamenta Informaticae
Random subspace method for multivariate feature selection

Pattern Recognition Letters
A local mean-based nonparametric classifier

Pattern Recognition Letters
Privacy leakage in multi-relational databases: a semi-supervised learning perspective

The VLDB Journal — The International Journal on Very Large Data Bases
Automatic texture feature selection for image pixel classification

Pattern Recognition
Face recognition from a single image per person: A survey

Pattern Recognition
On solving the face recognition problem with one training sample per subject

Pattern Recognition
Optimal convex error estimators for classification

Pattern Recognition
Learning prototypes and distances: A prototype reduction technique based on nearest neighbor error minimization

Pattern Recognition
Experimental study for the comparison of classifier combination methods

Pattern Recognition
Cell-nuclear data reduction and prognostic model selection in bladder tumor recurrence

Artificial Intelligence in Medicine
Trainable fusion rules. I. Large sample size case

Neural Networks
Trainable fusion rules. II. Small sample-size effects

Neural Networks
Improved discriminate analysis for high-dimensional data and its application to face recognition

Pattern Recognition
A tree-based decision rule for identifying profile groups of cases without predefined classes: application in diffuse large B-cell lymphomas

Computers in Biology and Medicine
On the Classification of a Small Imbalanced Cytogenetic Image Database

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
On the Bayes fusion of visual features

Image and Vision Computing
Local decomposition for rare class analysis

Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining
Visual inspection of machined metallic high-precision surfaces

EURASIP Journal on Applied Signal Processing
Data-driven decomposition for multi-class classification

Pattern Recognition
A new classification model with simple decision rule for discovering optimal feature gene pairs

Computers in Biology and Medicine
Improvement of the Parzen classifier in small training sample size situations

Intelligent Data Analysis
Gene selection for multiclass prediction by weighted fisher criterion

EURASIP Journal on Bioinformatics and Systems Biology
Kernel quadratic discriminant analysis for small sample size problem

Pattern Recognition
Kernel quadratic discriminant analysis for small sample size problem

Pattern Recognition
2008 Special Issue: Training neural network classifiers for medical decision making: The effects of imbalanced datasets on classification performance

Neural Networks
The peaking phenomenon in the presence of feature-selection

Pattern Recognition Letters
General support vector representation machine for one-class classification of non-stationary classes

Pattern Recognition
A discriminant analysis for undersampled data

AIDM '07 Proceedings of the 2nd international workshop on Integrating artificial intelligence and data mining - Volume 84
An efficient spatial semi-supervised learning algorithm

International Journal of Parallel, Emergent and Distributed Systems
Incremental Bayesian classification for multivariate normal distribution data

Pattern Recognition Letters
Data Complexity Analysis: Linkage between Context and Solution in Classification

SSPR & SPR '08 Proceedings of the 2008 Joint IAPR International Workshop on Structural, Syntactic, and Statistical Pattern Recognition
Identity Management in Face Recognition Systems

Biometrics and Identity Management
An image content description technique for the inspection of specular objects

EURASIP Journal on Advances in Signal Processing
An Approximation Method of the Quadratic Discriminant Function and Its Application to Estimation of High-Dimensional Distribution

IEICE - Transactions on Information and Systems
Mining extremely small data sets with application to software reuse

Software—Practice & Experience
A Statistical Approach for Binary Vectors Modeling and Clustering

PAKDD '09 Proceedings of the 13th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining
Normal maps vs. visible images: Comparing classifiers and combining modalities

Journal of Visual Languages and Computing
Feature extraction based on Laplacian bidirectional maximum margin criterion

Pattern Recognition
A Random Network Ensemble for Face Recognition

ICB '09 Proceedings of the Third International Conference on Advances in Biometrics
Letters: Feature extraction using fuzzy inverse FDA

Neurocomputing
Network Protocol Verification by a Classifier Selection Ensemble

MCS '09 Proceedings of the 8th International Workshop on Multiple Classifier Systems
Classification of SPECT Images Using Clustering Techniques Revisited

IWINAC '09 Proceedings of the 3rd International Work-Conference on The Interplay Between Natural and Artificial Computation: Part II: Bioinspired Applications in Artificial and Natural Computation
An experimental decision of samples for RBF neural networks

MUSP'09 Proceedings of the 9th WSEAS international conference on Multimedia systems & signal processing
A pilot sampling method for multi-layer perceptrons

ICCOMP'09 Proceedings of the WSEAES 13th international conference on Computers
Performance comparison of RBF networks and MLPs for classification

AIC'09 Proceedings of the 9th WSEAS international conference on Applied informatics and communications
The relationship of sample size and accuracy in radial basis function networks

WSEAS Transactions on Computers
Sampling scheme for better RBF network

Proceedings of the 2009 International Conference on Hybrid Information Technology
Discriminative wavelet packet filter bank selection for pattern recognition

IEEE Transactions on Signal Processing
An empirical improvement of the accuracy of RBF networks

Proceedings of the 2nd International Conference on Interaction Sciences: Information Technology, Culture and Human
Two-dimensional maximum margin feature extraction for face recognition

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics - Special issue on cybernetics and cognitive informatics
Sport training using body sensor networks: a statistical approach to measure wrist rotation for golf swing

BodyNets '09 Proceedings of the Fourth International Conference on Body Area Networks
On the window size for classification in changing environments

Intelligent Data Analysis
Evaluating classifiers: relation between area under the receiver operator characteristic curve and overall accuracy

IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
Empirical determination of sample sizes for multi-layer perceptrons by simple RBF networks

WSEAS Transactions on Computers
A method for extracting temporal parameters based on hidden Markov models in body sensor networks with inertial sensors

IEEE Transactions on Information Technology in Biomedicine - Special section on body sensor networks
Selecting discrete and continuous features based on neighborhood decision error minimization

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics
Distance approximating dimension reduction of Riemannian manifolds

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics
Wearable coach for sport training: A quantitative model to evaluate wrist-rotation in golf

Journal of Ambient Intelligence and Smart Environments
COG: local decomposition for rare class analysis

Data Mining and Knowledge Discovery
Determination of the optimal number of features for quadratic discriminant analysis via the normal approximation to the discriminant distribution

Pattern Recognition
Editorial: The fundamental role of pattern recognition for gene-expression/microarray data in bioinformatics

Pattern Recognition
Successive pattern classification based on test feature classifier and its application to defect image classification

Pattern Recognition
A novel feature selection based semi-supervised method for image classification

ICVS'08 Proceedings of the 6th international conference on Computer vision systems
Feature extraction based on fuzzy 2DLDA

Neurocomputing
Reduction of processing time for optimal and quadratic discriminant analyses

Pattern Recognition
Transfer estimation of evolving class priors in data stream classification

Pattern Recognition
Creating diverse nearest-neighbour ensembles using simultaneous metaheuristic feature selection

Pattern Recognition Letters
On Guo and Nixon's criterion for feature subset selection: assumptions, implications, and alternative options

IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans - Special issue on recent advances in biometrics
Face recognition using discriminant locality preserving projections based on maximum margin criterion

Pattern Recognition
Binary spectral minutiae representation with multi-sample fusion for fingerprint recognition

Proceedings of the 12th ACM workshop on Multimedia and security
Laser Doppler vibrometry measures of physiological function: evaluation of biometric capabilities

IEEE Transactions on Information Forensics and Security
A multi-resolution hidden Markov model using class-specific features

IEEE Transactions on Signal Processing
Dual unsupervised discriminant projection for face recognition

ICIC'10 Proceedings of the 6th international conference on Advanced intelligent computing theories and applications: intelligent computing
GMM based SPECT image classification for the diagnosis of Alzheimer's disease

Applied Soft Computing
Feature Extraction Using Laplacian Maximum Margin Criterion

Neural Processing Letters
A hybrid classification scheme for mining multisource geospatial data

Geoinformatica
Pattern recognition using boundary data of component distributions

Computers and Industrial Engineering
A multi-manifold discriminant analysis method for image feature extraction

Pattern Recognition
Virtual sample generation using concurrent-self-organizing maps and its application for facial expression recognition

MMES'10 Proceedings of the 2010 international conference on Mathematical models for engineering science
The effect of training set size for the performance of neural networks of classification

WSEAS Transactions on Computers
Measuring relevance between discrete and continuous features based on neighborhood mutual information

Expert Systems with Applications: An International Journal
Property of artificial neural networks of classification with respect to training set size

ICS'10 Proceedings of the 14th WSEAS international conference on Systems: part of the 14th WSEAS CSCC multiconference - Volume II
Simultaneous model selection and feature selection via BYY harmony learning

ISNN'11 Proceedings of the 8th international conference on Advances in neural networks - Volume Part II
On the use of diagonal and class-dependent weighted distances for the probabilistic k-nearest neighbor

IbPRIA'11 Proceedings of the 5th Iberian conference on Pattern recognition and image analysis
Addressing the classification with imbalanced data: open problems and new challenges on class distribution

HAIS'11 Proceedings of the 6th international conference on Hybrid artificial intelligent systems - Volume Part I
Thinned-ECOC ensemble based on sequential code shrinking

Expert Systems with Applications: An International Journal
A study on the influence of shape in classifying small spectral data sets

SIMBAD'11 Proceedings of the First international conference on Similarity-based pattern recognition
The effect of class imbalance on case selection for case-based classifiers: An empirical study in the context of medical decision support

Neural Networks
Feature over-selection

SSPR'06/SPR'06 Proceedings of the 2006 joint IAPR international conference on Structural, Syntactic, and Statistical Pattern Recognition
Face recognition by inverse fisher discriminant features

ICB'06 Proceedings of the 2006 international conference on Advances in Biometrics
Feature extraction for time series classification using discriminating wavelet coefficients

ISNN'06 Proceedings of the Third international conference on Advances in Neural Networks - Volume Part I
A new monte carlo-based error rate estimator

ANNPR'10 Proceedings of the 4th IAPR TC3 conference on Artificial Neural Networks in Pattern Recognition
Between-Source modelling for likelihood ratio computation in forensic biometric recognition

AVBPA'05 Proceedings of the 5th international conference on Audio- and Video-Based Biometric Person Authentication
Gene selection by cooperative competition clustering

ICIC'06 Proceedings of the 2006 international conference on Computational Intelligence and Bioinformatics - Volume Part III
Modeling context as statistical dependence

CONTEXT'05 Proceedings of the 5th international conference on Modeling and Using Context
Similarity measurement for off-line signature verification

ICIC'05 Proceedings of the 2005 international conference on Advances in Intelligent Computing - Volume Part I
On sample size and classification accuracy: a performance comparison

ISBMDA'05 Proceedings of the 6th International conference on Biological and Medical Data Analysis
On understanding and assessing feature selection bias

AIME'05 Proceedings of the 10th conference on Artificial Intelligence in Medicine
Feature extraction using a fast null space based linear discriminant analysis algorithm

Information Sciences: an International Journal
Feature Extraction Using a Complete Kernel Extension of Supervised Graph Embedding

Neural Processing Letters
Face recognition using discriminant sparsity neighborhood preserving embedding

Knowledge-Based Systems
Gait verification using knee acceleration signals

Expert Systems with Applications: An International Journal
Exact representation of the second-order moments for resubstitution and leave-one-out error estimation for linear discriminant analysis in the univariate heteroskedastic Gaussian model

Pattern Recognition
Multi-modal region selection approach for training object detectors

Proceedings of the 2nd ACM International Conference on Multimedia Retrieval
Automatic Segmentation and Recognition in Body Sensor Networks Using a Hidden Markov Model

ACM Transactions on Embedded Computing Systems (TECS) - Special Section on CAPA'09, Special Section on WHS'09, and Special Section VCPSS' 09
A synthesised word approach to word retrieval in handwritten documents

Pattern Recognition
Wearable coach for sport training: A quantitative model to evaluate wrist-rotation in golf

Journal of Ambient Intelligence and Smart Environments
Bounds for Validation

Fundamenta Informaticae
Ensemble-based regression analysis of multimodal medical data for osteopenia diagnosis

Expert Systems with Applications: An International Journal
DBFS: An effective Density Based Feature Selection scheme for small sample size and high dimensional imbalanced data sets

Data & Knowledge Engineering
Efficient sampling and handling of variance in tuning data mining models

PPSN'12 Proceedings of the 12th international conference on Parallel Problem Solving from Nature - Volume Part I
Optimized projections for sparse representation based classification

Neurocomputing
Double linear regressions for single labeled image per person face recognition

Pattern Recognition
Sparse regularization discriminant analysis for face recognition

Neurocomputing
Local maximal margin discriminant embedding for face recognition

Journal of Visual Communication and Image Representation
Learning algorithms may perform worse with increasing training set size: Algorithm-data incompatibility

Computational Statistics & Data Analysis
Kernel inverse Fisher discriminant analysis for face recognition

Neurocomputing

Quantified Score

Hi-index	0.16

Visualization

Abstract

The effects of sample size on feature selection and error estimation for several types of classifiers are discussed. The focus is on the two-class problem. Classifier design in the context of small design sample size is explored. The estimation of error rates under small test sample size is given. Sample size effects in feature selection are discussed. Recommendations for the choice of learning and test sample sizes are given. In addition to surveying prior work in this area, an emphasis is placed on giving practical advice to designers and users of statistical pattern recognition systems.