Accelerated training of conditional random fields with stochastic gradient methods

Authors:
S. V. N. Vishwanathan;Nicol N. Schraudolph;Mark W. Schmidt;Kevin P. Murphy
Affiliations:
National ICT Australia, Australia and Australian National University, Australia;National ICT Australia, Australia and Australian National University, Australia;University of British Columbia, Canada;University of British Columbia, Canada
Venue:
ICML '06 Proceedings of the 23rd international conference on Machine learning
Year:
2006

Citing 13
Cited 61

Fast exact multiplication by the Hessian

Neural Computation
Evaluating derivatives: principles and techniques of algorithmic differentiation

Evaluating derivatives: principles and techniques of algorithmic differentiation
Fast Approximate Energy Minimization via Graph Cuts

IEEE Transactions on Pattern Analysis and Machine Intelligence
Fast curvature matrix-vector products for second-order gradient descent

Neural Computation
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

ICML '01 Proceedings of the Eighteenth International Conference on Machine Learning
Understanding belief propagation and its generalizations

Exploring artificial intelligence in the new millennium
Shallow parsing with conditional random fields

NAACL '03 Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1
Introduction to the CoNLL-2000 shared task: chunking

ConLL '00 Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning - Volume 7
Discriminative training methods for hidden Markov models: theory and experiments with perceptron algorithms

EMNLP '02 Proceedings of the ACL-02 conference on Empirical methods in natural language processing - Volume 10
Introduction to the bio-entity recognition task at JNLPBA

JNLPBA '04 Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications
Biomedical named entity recognition using conditional random fields and rich feature sets

JNLPBA '04 Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications
Man-made structure detection in natural images using a causal multiscale random field

CVPR'03 Proceedings of the 2003 IEEE computer society conference on Computer vision and pattern recognition
Image Analysis, Random Fields and Dynamic Monte Carlo Methods: A Mathematical Introduction

Image Analysis, Random Fields and Dynamic Monte Carlo Methods: A Mathematical Introduction

Semi-supervised conditional random fields for improved sequence segmentation and labeling

ACL-44 Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics
Step Size Adaptation in Reproducing Kernel Hilbert Space

The Journal of Machine Learning Research
Dynamic Conditional Random Fields: Factorized Probabilistic Models for Labeling and Segmenting Sequence Data

The Journal of Machine Learning Research
Combining statistical and knowledge-based spoken language understanding in conditional models

COLING-ACL '06 Proceedings of the COLING/ACL on Main conference poster sessions
Exponentiated gradient algorithms for log-linear structured prediction

Proceedings of the 24th international conference on Machine learning
Piecewise pseudolikelihood for efficient training of conditional random fields

Proceedings of the 24th international conference on Machine learning
Classification of biomedical high-resolution micro-CT images for direct volume rendering

AIAP'07 Proceedings of the 25th conference on Proceedings of the 25th IASTED International Multi-Conference: artificial intelligence and applications
Training structural SVMs when exact inference is intractable

Proceedings of the 25th international conference on Machine learning
Training structural svms with kernels using sampled cuts

Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Efficient Weight Learning for Markov Logic Networks

PKDD 2007 Proceedings of the 11th European conference on Principles and Practice of Knowledge Discovery in Databases
Approximate Parameter Learning in Conditional Random Fields: An Empirical Investigation

Proceedings of the 30th DAGM symposium on Pattern Recognition
Exponentiated Gradient Algorithms for Conditional Random Fields and Max-Margin Markov Networks

The Journal of Machine Learning Research
Learning a two-stage SVM/CRF sequence classifier

Proceedings of the 17th ACM conference on Information and knowledge management
Minimum tag error for discriminative training of conditional random fields

Information Sciences: an International Journal
Information Extraction

Foundations and Trends in Databases
An algorithm for analyzing personalized online commercial intention

Proceedings of the 2nd International Workshop on Data Mining and Audience Intelligence for Advertising
Limited stochastic meta-descent for kernel-based online learning

Neural Computation
Modeling latent-dynamic in shallow parsing: a latent conditional model with improved inference

COLING '08 Proceedings of the 22nd International Conference on Computational Linguistics - Volume 1
Cutting-plane training of structural SVMs

Machine Learning
A Generalization of Forward-Backward Algorithm

ECML PKDD '09 Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases: Part I
Learning large scale common sense models of everyday life

AAAI'07 Proceedings of the 22nd national conference on Artificial intelligence - Volume 1
A discriminative latent variable chinese segmenter with hybrid word/character information

NAACL '09 Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics
A new gaussian mixture conditional random field model for indoor image labeling

IMCE '09 Proceedings of the 1st international workshop on Interactive multimedia for consumer electronics
Recognizing events with temporal random forests

Proceedings of the 2009 international conference on Multimodal interfaces
Speech recognition using augmented conditional random fields

IEEE Transactions on Audio, Speech, and Language Processing
Probabilistic models for concurrent chatting activity recognition

IJCAI'09 Proceedings of the 21st international jont conference on Artifical intelligence
Selecting features of linear-chain conditional random fields via greedy stage-wise algorithms

Pattern Recognition Letters
Periodic step-size adaptation in second-order gradient descent for single-pass on-line structured learning

Machine Learning
Piecewise training for structured prediction

Machine Learning
Stochastic gradient descent training for L1-regularized log-linear models with cumulative penalty

ACL '09 Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1
ECOC Random Fields for Lumen Segmentation in Radial Artery IVUS Sequences

MICCAI '09 Proceedings of the 12th International Conference on Medical Image Computing and Computer-Assisted Intervention: Part II
Sinuhe: statistical machine translation using a globally trained conditional exponential family translation model

EMNLP '09 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2 - Volume 2
Iterative Scaling and Coordinate Descent Methods for Maximum Entropy Models

The Journal of Machine Learning Research
Minimizing and learning energy functions for side-chain prediction

RECOMB'07 Proceedings of the 11th annual international conference on Research in computational molecular biology
Large margin cost-sensitive learning of conditional random fields

Pattern Recognition
Grafting-light: fast, incremental feature selection and structure learning of Markov random fields

Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
Conditional random field for text segmentation from images with complex background

Pattern Recognition Letters
Practical very large scale CRFs

ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
Document structure meets page layout: loopy random fields for web news content extraction

Proceedings of the 10th ACM symposium on Document engineering
Improving Hierarchical Classification with Partial Labels

Proceedings of the 2010 conference on ECAI 2010: 19th European Conference on Artificial Intelligence
Prostate cancer localization with multispectral MRI using cost-sensitive support vector machines and conditional random fields

IEEE Transactions on Image Processing
Hidden conditional ordinal random fields for sequence classification

ECML PKDD'10 Proceedings of the 2010 European conference on Machine learning and knowledge discovery in databases: Part II
Probabilistic models for concurrent chatting activity recognition

ACM Transactions on Intelligent Systems and Technology (TIST)
Structured output ordinal regression for dynamic facial emotion intensity prediction

ECCV'10 Proceedings of the 11th European conference on computer vision conference on Computer vision: Part III
Accelerated training of maximum margin Markov models for sequence labeling: a case study of NP chunking

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
Efficient structured support vector regression

ACCV'10 Proceedings of the 10th Asian conference on Computer vision - Volume Part III
Multi-scale stacked sequential learning

Pattern Recognition
Towards adjusting mobile devices to user's behaviour

MSM'10/MUSE'10 Proceedings of the 2010 international conference on Analysis of social media and ubiquitous data
Conditional graphical models for protein structure prediction

Conditional graphical models for protein structure prediction
Conditional random fields for urban scene classification with full waveform LiDAR data

PIA'11 Proceedings of the 2011 ISPRS conference on Photogrammetric image analysis
Structured Learning and Prediction in Computer Vision

Foundations and Trends® in Computer Graphics and Vision
Rich set of features for proper name recognition in polish texts

SIIS'11 Proceedings of the 2011 international conference on Security and Intelligent Information Systems
On Learning Conditional Random Fields for Stereo

International Journal of Computer Vision
Gradient computation in linear-chain conditional random fields using the entropy message passing algorithm

Pattern Recognition Letters
Minimum-risk training of approximate CRF-based NLP systems

NAACL HLT '12 Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Fast online training with frequency-adaptive learning rates for Chinese word segmentation and new word detection

ACL '12 Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1
Online semantic mapping of urban environments

SC'12 Proceedings of the 2012 international conference on Spatial Cognition VIII
Lifted online training of relational models with stochastic gradient methods

ECML PKDD'12 Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases - Volume Part I
Class-Specified segmentation with multi-scale superpixels

ACCV'12 Proceedings of the 11th international conference on Computer Vision - Volume Part I
3D segmentation of abdominal CT imagery with graphical models, conditional random fields and learning

Machine Vision and Applications
Conditional ordinal random fields for structured ordinal-valued label prediction

Data Mining and Knowledge Discovery

Quantified Score

Hi-index	0.00

Visualization

Abstract

We apply Stochastic Meta-Descent (SMD), a stochastic gradient optimization method with gain vector adaptation, to the training of Conditional Random Fields (CRFs). On several large data sets, the resulting optimizer converges to the same quality of solution over an order of magnitude faster than limited-memory BFGS, the leading method reported to date. We report results for both exact and inexact inference techniques.