On the limited memory BFGS method for large scale optimization
Mathematical Programming: Series A and B
Selective Sampling Using the Query by Committee Algorithm
Machine Learning
Model-based and empirical evaluation of multimodal interactive error correction
Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Toward Optimal Active Learning through Sampling Estimation of Error Reduction
ICML '01 Proceedings of the Eighteenth International Conference on Machine Learning
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
ICML '01 Proceedings of the Eighteenth International Conference on Machine Learning
Active Learning for Natural Language Parsing and Information Extraction
ICML '99 Proceedings of the Sixteenth International Conference on Machine Learning
Active Hidden Markov Models for Information Extraction
IDA '01 Proceedings of the 4th International Conference on Advances in Intelligent Data Analysis
Employing EM and Pool-Based Active Learning for Text Classification
ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Table extraction using conditional random fields
Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval
Proposal for an Interactive Environment for Information Extraction
Proposal for an Interactive Environment for Information Extraction
Shallow parsing with conditional random fields
NAACL '03 Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1
Active learning for Hidden Markov Models: objective functions and algorithms
ICML '05 Proceedings of the 22nd international conference on Machine learning
Confidence estimation for translation prediction
CONLL '03 Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - Volume 4
CONLL '03 Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - Volume 4
Active learning for logistic regression
Active learning for logistic regression
Interactive information extraction with constrained conditional random fields
AAAI'04 Proceedings of the 19th national conference on Artifical intelligence
Active learning with strong and weak views: a case study on wrapper induction
IJCAI'03 Proceedings of the 18th international joint conference on Artificial intelligence
Efficiently inducing features of conditional random fields
UAI'03 Proceedings of the Nineteenth conference on Uncertainty in Artificial Intelligence
Dynamic hierarchical Markov random fields and their application to web data extraction
Proceedings of the 24th international conference on Machine learning
Integrating rich user feedback into intelligent user interfaces
Proceedings of the 13th international conference on Intelligent user interfaces
Dynamic Hierarchical Markov Random Fields for Integrated Web Data Extraction
The Journal of Machine Learning Research
Foundations and Trends in Databases
Amplifying community content creation with mixed initiative information extraction
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Review: A review of machine learning approaches to Spam filtering
Expert Systems with Applications: An International Journal
Interacting meaningfully with machine learning systems: Three experiments
International Journal of Human-Computer Studies
Training conditional random fields using incomplete annotations
COLING '08 Proceedings of the 22nd International Conference on Computational Linguistics - Volume 1
Integrating high precision rules with statistical sequence classifiers for accuracy and speed
SETQA-NLP '09 Proceedings of the Workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing
Semi-automated named entity annotation
LAW '07 Proceedings of the Linguistic Annotation Workshop
Hi-index | 0.00 |
To successfully embed statistical machine learning models in real world applications, two post-deployment capabilities must be provided: (1) the ability to solicit user corrections and (2) the ability to update the model from these corrections. We refer to the former capability as corrective feedback and the latter as persistent learning. While these capabilities have a natural implementation for simple classification tasks such as spam filtering, we argue that a more careful design is required for structured classification tasks.One example of a structured classification task is information extraction, in which raw text is analyzed to automatically populate a database. In this work, we augment a probabilistic information extraction system with corrective feedback and persistent learning components to assist the user in building, correcting, and updating the extraction model. We describe methods of guiding the user to incorrect predictions, suggesting the most informative fields to correct, and incorporating corrections into the inference algorithm. We also present an active learning framework that minimizes not only how many examples a user must label, but also how difficult each example is to label. We empirically validate each of the technical components in simulation and quantify the user effort saved. We conclude that more efficient corrective feedback mechanisms lead to more effective persistent learning.