Supervised and unsupervised co-training of adaptive activation functions in neural nets

Authors:
Ilaria Castelli;Edmondo Trentin
Affiliations:
Dipartimento di Ingegneria dell'Informazione, Università di Siena, Siena, Italy;Dipartimento di Ingegneria dell'Informazione, Università di Siena, Siena, Italy
Venue:
PSL'11 Proceedings of the First IAPR TC3 conference on Partially Supervised Learning
Year:
2011

Citing 3
Cited 3

Pattern Classification (2nd Edition)

Pattern Classification (2nd Edition)
Semi-unsupervised weighted maximum-likelihood estimation of joint densities for the co-training of adaptive activation functions

PSL'11 Proceedings of the First IAPR TC3 conference on Partially Supervised Learning
Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems

IEEE Transactions on Neural Networks

Semi-unsupervised weighted maximum-likelihood estimation of joint densities for the co-training of adaptive activation functions

PSL'11 Proceedings of the First IAPR TC3 conference on Partially Supervised Learning
Solving graph data issues using a layered architecture approach with applications to web spam detection

Neural Networks
Combination of supervised and unsupervised learning for training the activation functions of neural networks

Pattern Recognition Letters

Quantified Score

Hi-index	0.00

Visualization

Abstract

In spite of the nice theoretical properties of mixtures of logistic activation functions, standard feedforward neural network with limited resources and gradient-descent optimization of the connection weights may practically fail in several, difficult learning tasks. Such tasks would be better faced by relying on a more appropriate, problem-specific basis of activation functions. The paper introduces a connectionist model which features adaptive activation functions. Each hidden unit in the network is associated with a specific pair (f(·), p(·)), where f(·) (the very activation) is modeled via a specialized neural network, and p(·) is a probabilistic measure of the likelihood of the unit itself being relevant to the computation of the output over the current input. While f(·) is optimized in a supervised manner (through a novel backpropagation scheme of the target outputs which do not suffer from the traditional phenomenon of "vanishing gradient" that occurs in standard backpropagation), p(·) is realized via a statistical parametric model learned through unsupervised estimation. The overall machine is implicitly a co-trained coupled model, where the topology chosen for learning each f(·) may vary on a unit-by-unit basis, resulting in a highly non-standard neural architecture.