Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction

Authors:
James Fogarty;Ryan S. Baker;Scott E. Hudson
Affiliations:
Carnegie Mellon University;Carnegie Mellon University;Carnegie Mellon University
Venue:
GI '05 Proceedings of Graphics Interface 2005
Year:
2005

Citing 8
Cited 20

Wrappers for feature subset selection

Artificial Intelligence - Special issue on relevance
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

Machine Learning
Predicting human interruptibility with sensors: a Wizard of Oz feasibility study

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Examining the robustness of sensor-based statistical models of human interruptibility

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Examining task engagement in sensor-based statistical models of human interruptibility

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Predicting human interruptibility with sensors

ACM Transactions on Computer-Human Interaction (TOCHI)
Comparison of four primary methods for coordinating the interruption of people in human-computer interaction

Human-Computer Interaction
The use of the area under the ROC curve in the evaluation of machine learning algorithms

Pattern Recognition

Eye-tracking to model and adapt to user meta-cognition in intelligent learning environments

Proceedings of the 11th international conference on Intelligent user interfaces
Toolkit support for developing and deploying sensor-based statistical models of human situations

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Eye-tracking for user modeling in exploratory learning environments: An empirical evaluation

Knowledge-Based Systems
A multifactor approach to student model evaluation

User Modeling and User-Adapted Interaction
On using existing time-use study data for ubiquitous computing applications

UbiComp '08 Proceedings of the 10th international conference on Ubiquitous computing
Performance Factors Analysis --A New Alternative to Knowledge Tracing

Proceedings of the 2009 conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling
Group-wise similarity and classification of aggregate scanpaths

Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications
Web spam classification: a few features worth more

Proceedings of the 2011 Joint WICOW/AIRWeb Workshop on Web Quality
Learning patterns of pick-ups and drop-offs to support busy family coordination

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Detecting carelessness through contextual estimation of slip probabilities among students using an intelligent tutor for mathematics

AIED'11 Proceedings of the 15th international conference on Artificial intelligence in education
Contextual slip and prediction of student performance after use of an intelligent tutor

UMAP'10 Proceedings of the 18th international conference on User Modeling, Adaptation, and Personalization
Content-based trust and bias classification via biclustering

Proceedings of the 2nd Joint WICOW/AIRWeb Workshop on Web Quality
A review of recent advances in learner and skill modeling in intelligent learning environments

User Modeling and User-Adapted Interaction
Using machine learning techniques and genomic/proteomic information from known databases for defining relevant features for PPI classification

Computers in Biology and Medicine
Improving construct validity yields better models of systematic inquiry, even with less information

UMAP'12 Proceedings of the 20th international conference on User Modeling, Adaptation, and Personalization
A recognition safety net: bi-level threshold recognition for mobile motion gestures

MobileHCI '12 Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services
Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill

User Modeling and User-Adapted Interaction
Automatic Detection of Arrow Annotation Overlays in Biomedical Images

International Journal of Healthcare Information Systems and Informatics
Cross-lingual web spam classification

Proceedings of the 22nd international conference on World Wide Web companion
Iolaus: securing online content rating systems

Proceedings of the 22nd international conference on World Wide Web

Quantified Score

Hi-index	0.00

Visualization

Abstract

Applications that use sensor-based estimates face a fundamental tradeoff between true positives and false positives when examining the reliability of these estimates, one that is inadequately described by the straightforward notion of accuracy. To address this tradeoff, this paper examines the use of Receiver Operating Characteristic (ROC) curve analysis, a method that has a long history but is under-appreciated in the human computer interaction research community. We present the fundamentals of ROC analysis, the use of the A' statistic to compute the area under an ROC curve, and the equivalence of A' to the Wilcoxon statistic. We then present several case studies, framed in the context of our work on human interruptibility, demonstrating how ROC analysis can yield better results than analyses based on accuracy. These case studies compare sensor-based estimates with human performance, optimize a feature selection process for the area under the ROC curve, and examine end-user selection of a desirable tradeoff.