Crowdsourcing syntactic relatedness judgements for opinion mining in the study of information technology adoption

Authors:
Asad B. Sayeed;Bryan Rusk;Martin Petrov;Hieu C. Nguyen;Timothy J. Meyer;Amy Weinberg
Affiliations:
University of Maryland, College Park, MD;University of Maryland, College Park, MD;University of Maryland, College Park, MD;University of Maryland, College Park, MD;University of Maryland, College Park, MD;University of Maryland, College Park, MD
Venue:
LaTeCH '11 Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities
Year:
2011

Citing 9
Cited 1

Recognizing contextual polarity in phrase-level sentiment analysis

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
A novel scheme for domain-transfer problem in the context of sentiment analysis

Proceedings of the sixteenth ACM conference on Conference on information and knowledge management
Opinion Mining and Sentiment Analysis

Foundations and Trends in Information Retrieval
Data quality from crowdsourcing: a study of annotation selection criteria

HLT '09 Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing
Annotating attributions and private states

CorpusAnno '05 Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Extracting opinions, opinion holders, and topics expressed in online news media text

SST '06 Proceedings of the Workshop on Sentiment and Subjectivity in Text
Crowdsourcing the evaluation of a domain-adapted named entity recognition system

HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Expresses-an-opinion-about: using corpus statistics in an information extraction approach to opinion mining

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters

Grammatical structures for word-level sentiment detection

NAACL HLT '12 Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present an end-to-end pipeline including a user interface for the production of word-level annotations for an opinion-mining task in the information technology (IT) domain. Our pre-annotation pipeline selects candidate sentences for annotation using results from a small amount of trained annotation to bias the random selection over a large corpus. Our user interface reduces the need for the user to understand the "meaning" of opinion in our domain context, which is related to community reaction. It acts as a preliminary buffer against low-quality annotators. Finally, our post-annotation pipeline aggregates responses and applies a more aggressive quality filter. We present positive results using two different evaluation philosophies and discuss how our design decisions enabled the collection of high-quality annotations under subjective and fine-grained conditions.