Automatically Labeling Video Data Using Multi-class Active Learning
ICCV '03 Proceedings of the Ninth IEEE International Conference on Computer Vision - Volume 2
Labeling images with a computer game
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Affective multimodal human-computer interaction
Proceedings of the 13th annual ACM international conference on Multimedia
Get another label? improving data quality and data mining using multiple, noisy labelers
Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Data quality from crowdsourcing: a study of annotation selection criteria
HLT '09 Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Leveraging crowdsourcing heuristics to improve search in Wikipedia
Proceedings of the 5th International Symposium on Wikis and Open Collaboration
Proceedings of the international conference on Multimedia information retrieval
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
IEEE Transactions on Affective Computing
A Blueprint for Affective Computing: A sourcebook and manual
A Blueprint for Affective Computing: A sourcebook and manual
Crowdsourcing and human computation: systems, studies and platforms
CHI '11 Extended Abstracts on Human Factors in Computing Systems
Machine analysis and recognition of social contexts
Proceedings of the 14th ACM international conference on Multimodal interaction
Crowdsourcing micro-level multimedia annotations: the challenges of evaluation and interface
Proceedings of the ACM multimedia 2012 workshop on Crowdsourcing for multimedia
Sentiment analysis using a novel human computation game
Proceedings of the 3rd Workshop on the People's Web Meets NLP: Collaboratively Constructed Semantic Resources and their Applications to NLP
Proceedings of the 15th ACM on International conference on multimodal interaction
Proceedings of the 19th international conference on Intelligent User Interfaces
| Hi-index | 0.00 |
One of the most time consuming and laborious problems facing researchers in Affective Computing is annotation of data, particularly with the recent adoption of multimodal data. Other fields, such as Computer Vision, Language Processing and Information Retrieval have successfully used crowd sourcing (or human computation) games to label their data sets. Inspired by their work, we have developed a Facebook game called Guess What? for labeling multimodal, affective video data. This paper describes the game and an initial evaluation of it for social context labeling. In our experiment, 33 participants used the game to label 154 video/question pairs over the course of a few days, and their overall inter-rater reliability was good (Krippendorff's α = .70). We believe this game will be a useful resource for other researchers and ultimately plan to make Guess What? open source and available to anyone who is interested.