Interacting meaningfully with machine learning systems: Three experiments

Authors:
Simone Stumpf;Vidya Rajaram;Lida Li;Weng-Keen Wong;Margaret Burnett;Thomas Dietterich;Erin Sullivan;Jonathan Herlocker
Affiliations:
Oregon State University, School of Electrical Engineering and Computer Science, Corvallis, OR 97331, USA;Oregon State University, School of Electrical Engineering and Computer Science, Corvallis, OR 97331, USA;Oregon State University, School of Electrical Engineering and Computer Science, Corvallis, OR 97331, USA;Oregon State University, School of Electrical Engineering and Computer Science, Corvallis, OR 97331, USA;Oregon State University, School of Electrical Engineering and Computer Science, Corvallis, OR 97331, USA;Oregon State University, School of Electrical Engineering and Computer Science, Corvallis, OR 97331, USA;Oregon State University, School of Electrical Engineering and Computer Science, Corvallis, OR 97331, USA;Oregon State University, School of Electrical Engineering and Computer Science, Corvallis, OR 97331, USA
Venue:
International Journal of Human-Computer Studies
Year:
2009

Citing 56
Cited 14

Reconstructive expert system explanation

Artificial Intelligence
Explanation in Bayesian belief networks

Explanation in Bayesian belief networks
Information-based objective functions for active data selection

Neural Computation
Watch what I do: programming by demonstration

Watch what I do: programming by demonstration
Prototyping for tiny fingers

Communications of the ACM
Trading Accuracy for Simplicity in Decision Trees

Machine Learning
Knowledge-based artificial neural networks

Artificial Intelligence
Agents that learn to explain themselves

AAAI'94 Proceedings of the twelfth national conference on Artificial intelligence (vol. 2)
WordNet: a lexical database for English

Communications of the ACM
Creating advice-taking reinforcement learners

Machine Learning - Special issue on reinforcement learning
Combining labeled and unlabeled data with co-training

COLT' 98 Proceedings of the eleventh annual conference on Computational learning theory
MailCat: an intelligent assistant for organizing e-mail

Proceedings of the third annual conference on Autonomous Agents
Principles of mixed-initiative user interfaces

Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Getting more out of programming-by-demonstration

Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Representation of electronic mail filtering profiles: a user study

Proceedings of the 5th international conference on Intelligent user interfaces
Explaining collaborative filtering recommendations

CSCW '00 Proceedings of the 2000 ACM conference on Computer supported cooperative work
Visualizing the simple Baysian classifier

Information visualization in data mining and knowledge discovery
An intelligent interface for sorting electronic mail

Proceedings of the 7th international conference on Intelligent user interfaces
Interactive machine learning: letting users build classifiers

International Journal of Human-Computer Studies
Interactive machine learning

Proceedings of the 8th international conference on Intelligent user interfaces
Multiple-Knowledge Representation in Concept Learning

ECML '95 Proceedings of the 8th European Conference on Machine Learning
IEMS - The Intelligent Email Sorter

ICML '02 Proceedings of the Nineteenth International Conference on Machine Learning
Challenges of the Email Domain for Text Classification

ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning
Incremental Learning in SwiftFile

ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning
Email classification with co-training

CASCON '01 Proceedings of the 2001 conference of the Centre for Advanced Studies on Collaborative research
First Steps in Programming: A Rationale for Attention Investment Models

HCC '02 Proceedings of the IEEE 2002 Symposia on Human Centric Computing Languages and Environments (HCC'02)
Extracting comprehensible models from trained neural networks

Extracting comprehensible models from trained neural networks
Support vector machine active learning with applications to text classification

The Journal of Machine Learning Research
Preference elicitation via theory refinement

The Journal of Machine Learning Research
Fusion of domain knowledge with data for structural learning in object oriented domains

The Journal of Machine Learning Research
A review of explanation methods for Bayesian networks

The Knowledge Engineering Review
Adversarial classification

Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining
User interfaces with semi-formal representations: a study of designing argumentation structures

Proceedings of the 10th international conference on Intelligent user interfaces
Improving proactive information systems

Proceedings of the 10th international conference on Intelligent user interfaces
Experiments in dynamic critiquing

Proceedings of the 10th international conference on Intelligent user interfaces
Task learning by instruction in tailor

Proceedings of the 10th international conference on Intelligent user interfaces
Nomograms for visualization of naive Bayesian classifier

PKDD '04 Proceedings of the 8th European Conference on Principles and Practice of Knowledge Discovery in Databases
Named entity recognition using an HMM-based chunk tagger

ACL '02 Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Providing Expert Advice by Analogy for On-Line Help

IAT '05 Proceedings of the IEEE/WIC/ACM International Conference on Intelligent Agent Technology
Garbage in, Garbage out? An Empirical Look at Oracle Mistakes by End-User Programmers

VLHCC '05 Proceedings of the 2005 IEEE Symposium on Visual Languages and Human-Centric Computing
Automatically classifying emails into activities

Proceedings of the 11th international conference on Intelligent user interfaces
A hybrid learning system for recognizing user tasks from desktop activities and email messages

Proceedings of the 11th international conference on Intelligent user interfaces
Trust building with explanation interfaces

Proceedings of the 11th international conference on Intelligent user interfaces
Augmentation-based learning: combining observations and user edits for programming-by-demonstration

Proceedings of the 11th international conference on Intelligent user interfaces
Answering why and why not questions in user interfaces

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Text clustering with extended user feedback

SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
CueTIP: a mixed-initiative interface for correcting handwriting errors

UIST '06 Proceedings of the 19th annual ACM symposium on User interface software and technology
Toward harnessing user feedback for machine learning

Proceedings of the 12th international conference on Intelligent user interfaces
Corrective feedback and persistent learning for information extraction

Artificial Intelligence
CueFlik: interactive concept learning in image search

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Toward establishing trust in adaptive agents

Proceedings of the 13th international conference on Intelligent user interfaces
Visual explanation of evidence in additive classifiers

IAAI'06 Proceedings of the 18th conference on Innovative applications of artificial intelligence - Volume 2
Text classification by labeling words

AAAI'04 Proceedings of the 19th national conference on Artifical intelligence
Activity-centric email: a machine learning approach

AAAI'06 proceedings of the 21st national conference on Artificial intelligence - Volume 2
Active learning with statistical models

Journal of Artificial Intelligence Research
XPLAIN: a system for creating and explaining expert consulting programs

Artificial Intelligence

Toolkit to support intelligibility in context-aware applications

Proceedings of the 12th ACM international conference on Ubiquitous computing
End-user feature labeling: a locally-weighted regression approach

Proceedings of the 16th international conference on Intelligent user interfaces
Integrating knowledge capture and supervised learning through a human-computer interface

Proceedings of the sixth international conference on Knowledge capture
Towards understanding how humans teach robots

UMAP'11 Proceedings of the 19th international conference on User modeling, adaption, and personalization
Why-oriented end-user debugging of naive Bayes text classification

ACM Transactions on Interactive Intelligent Systems (TiiS)
Are explanations always important?: a study of deployed, low-cost intelligent interactive systems

Proceedings of the 2012 ACM international conference on Intelligent User Interfaces
An explanation-centric approach for personalizing intelligent agents

Proceedings of the 2012 ACM international conference on Intelligent User Interfaces
Evaluation methods and strategies for the interactive use of classifiers

International Journal of Human-Computer Studies
Tell me more?: the effects of mental model soundness on personalizing an intelligent agent

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
End-user interactions with intelligent and autonomous systems

CHI '12 Extended Abstracts on Human Factors in Computing Systems
Continuous user feedback learning for data capture from business documents

HAIS'12 Proceedings of the 7th international conference on Hybrid Artificial Intelligent Systems - Volume Part II
Simulating user intervention for interactive semantic place recognition with mobile devices

Proceedings of the 2012 RecSys workshop on Personalizing the local mobile experience
Learning from a learning thermostat: lessons for intelligent systems for the home

Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing
End-user feature labeling: Supervised and semi-supervised approaches based on locally-weighted logistic regression

Artificial Intelligence

Quantified Score

Hi-index	0.00

Visualization

Abstract

Although machine learning is becoming commonly used in today's software, there has been little research into how end users might interact with machine learning systems, beyond communicating simple ''right/wrong'' judgments. If the users themselves could work hand-in-hand with machine learning systems, the users' understanding and trust of the system could improve and the accuracy of learning systems could be improved as well. We conducted three experiments to understand the potential for rich interactions between users and machine learning systems. The first experiment was a think-aloud study that investigated users' willingness to interact with machine learning reasoning, and what kinds of feedback users might give to machine learning systems. We then investigated the viability of introducing such feedback into machine learning systems, specifically, how to incorporate some of these types of user feedback into machine learning systems, and what their impact was on the accuracy of the system. Taken together, the results of our experiments show that supporting rich interactions between users and machine learning systems is feasible for both user and machine. This shows the potential of rich human-computer collaboration via on-the-spot interactions as a promising direction for machine learning systems and users to collaboratively share intelligence.