Contextual correlates of synonymy
Communications of the ACM
Building a large annotated corpus of English: the penn treebank
Computational Linguistics - Special issue on using large corpora: II
COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 1
Labeling images with a computer game
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Development and use of a gold-standard data set for subjectivity classifications
ACL '99 Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics
Scaling to very very large corpora for natural language disambiguation
ACL '01 Proceedings of the 39th Annual Meeting on Association for Computational Linguistics
HLT '93 Proceedings of the workshop on Human Language Technology
Building a sense tagged corpus with open mind word expert
WSD '02 Proceedings of the ACL-02 workshop on Word sense disambiguation: recent successes and future directions - Volume 8
The Proposition Bank: An Annotated Corpus of Semantic Roles
Computational Linguistics
Verbosity: a game for collecting common-sense facts
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Internet-scale collection of human-reviewed data
Proceedings of the 16th international conference on World Wide Web
Crowdsourcing user studies with Mechanical Turk
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Get another label? improving data quality and data mining using multiple, noisy labelers
Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Automatic Extraction of Useful Facet Hierarchies from Text Databases
ICDE '08 Proceedings of the 2008 IEEE 24th International Conference on Data Engineering
SemEval-2007 task 14: affective text
SemEval '07 Proceedings of the 4th International Workshop on Semantic Evaluations
SemEval-2007 task 17: English lexical sample, SRL and all words
SemEval '07 Proceedings of the 4th International Workshop on Semantic Evaluations
SWAT-MP: the SemEval-2007 systems for task 5 and task 14
SemEval '07 Proceedings of the 4th International Workshop on Semantic Evaluations
The PASCAL recognising textual entailment challenge
MLCW'05 Proceedings of the First international conference on Machine Learning Challenges: evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment
Supervised learning from multiple experts: whom to trust when everyone lies a bit
ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning
Efficiently learning the accuracy of labeling sources for selective sampling
Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
Data quality from crowdsourcing: a study of annotation selection criteria
HLT '09 Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing
Proactive learning for building machine translation systems for minority languages
HLT '09 Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing
txteagle: Mobile Crowdsourcing
IDGD '09 Proceedings of the 3rd International Conference on Internationalization, Design and Global Development: Held as Part of HCI International 2009
Jointly combining implicit constraints improves temporal ordering
EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
For a few dollars less: identifying review pages sans human labels
NAACL '09 Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics
SemEval-2010 task 9: the interpretation of noun compounds using paraphrasing verbs and prepositions
DEW '09 Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions
Wikipedia-based semantic interpretation for natural language processing
Journal of Artificial Intelligence Research
Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility
Completing wikipedia's hyperlink structure through dimensionality reduction
Proceedings of the 18th ACM conference on Information and knowledge management
Answer typing for information retrieval
Proceedings of the 18th ACM conference on Information and knowledge management
Feature selection for ranking using boosted trees
Proceedings of the 18th ACM conference on Information and knowledge management
Wikispeedia: an online game for inferring semantic distances between concepts
IJCAI'09 Proceedings of the 21st international jont conference on Artifical intelligence
The lie detector: explorations in the automatic recognition of deceptive language
ACLShort '09 Proceedings of the ACL-IJCNLP 2009 Conference Short Papers
Learning with annotation noise
ACL '09 Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1
Distant supervision for relation extraction without labeled data
ACL '09 Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2
Complex linguistic annotation --- no easy way out!: a case from Bangla and Hindi POS labeling tasks
ACL-IJCNLP '09 Proceedings of the Third Linguistic Annotation Workshop
Feasibility of human-in-the-loop minimum error rate training
EMNLP '09 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1
Fast, cheap, and creative: evaluating translation quality using Amazon's Mechanical Turk
EMNLP '09 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1
EMNLP '09 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1
Acquiring high quality non-expert knowledge from on-demand workforce
People's Web '09 Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources
What do we know about conversation participants: experiments on conversation entailment
SIGDIAL '09 Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Crowd translator: on building localized speech recognizers through micropayments
ACM SIGOPS Operating Systems Review
Coupled semi-supervised learning for information extraction
Proceedings of the third ACM international conference on Web search and data mining
Proceedings of the international conference on Multimedia information retrieval
Why pay?: exploring how financial incentives are used for question & answer
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Characterizing debate performance via aggregated twitter sentiment
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Who are the crowdworkers?: shifting demographics in mechanical turk
CHI '10 Extended Abstracts on Human Factors in Computing Systems
Modulating video credibility via visualization of quality evaluations
Proceedings of the 4th workshop on Information credibility
The labor economics of paid crowdsourcing
Proceedings of the 11th ACM conference on Electronic commerce
Crowdsourcing the assembly of concept hierarchies
Proceedings of the 10th annual joint conference on Digital libraries
Collecting high quality overlapping labels at low cost
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Learning more powerful test statistics for click-based retrieval evaluation
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
The anatomy of a large-scale human computation engine
Proceedings of the ACM SIGKDD Workshop on Human Computation
Exploring iterative and parallel human computation processes
Proceedings of the ACM SIGKDD Workshop on Human Computation
Toward automatic task design: a progress report
Proceedings of the ACM SIGKDD Workshop on Human Computation
Exploring the use of crowdsourcing to support empirical studies in software engineering
Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement
Multi-prototype vector-space models of word meaning
HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Cheap, fast and good enough: automatic speech recognition with non-expert transcription
HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Time-efficient creation of an accurate sentence fusion corpus
HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Crowdsourcing the evaluation of a domain-adapted named entity recognition system
HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Predicting human-targeted translation edit rate via untrained human annotators
HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Some empirical evidence for annotation noise in a benchmarked dataset
HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
The best lexical metric for phrase-based statistical MT system optimization
HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Cross-lingual induction of selectional preferences with bilingual vector spaces
HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Distinguishing use and mention in natural language
HLT-SRWS '10 Proceedings of the NAACL HLT 2010 Student Research Workshop
"Was it good? It was provocative." Learning the meaning of scalar adjectives
ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
Bucking the trend: large-scale cost-focused active learning for statistical machine translation
ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
Learning script knowledge with web experiments
ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
SemEval-2010 task 9: The interpretation of noun compounds using paraphrasing verbs and prepositions
SemEval '10 Proceedings of the 5th International Workshop on Semantic Evaluation
The Journal of Machine Learning Research
Unsupervised Supervised Learning I: Estimating Classification and Regression Errors without Labels
The Journal of Machine Learning Research
Emotions evoked by common words and phrases: using mechanical turk to create an emotion lexicon
CAAGET '10 Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text
Identifying emotions, intentions, and attitudes in text using a game with a purpose
CAAGET '10 Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text
Using Crowdsourcing and Active Learning to Track Sentiment in Online Media
Proceedings of the 2010 conference on ECAI 2010: 19th European Conference on Artificial Intelligence
An elaborated model of social search
Information Processing and Management: an International Journal
TurKit: human computation algorithms on mechanical turk
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Soylent: a word processor with a crowd inside
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Creating speech and language data with Amazon's Mechanical Turk
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Corpus creation for new genres: A crowdsourced approach to PP attachment
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Clustering dictionary definitions using Amazon Mechanical Turk
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Rating computer-generated questions with Mechanical Turk
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Shared task: crowdsourced accessibility elicitation of Wikipedia articles
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Using Amazon Mechanical Turk for transcription of non-native speech
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Can crowds build parallel corpora for machine translation systems?
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Annotating large email datasets for named entity recognition with Mechanical Turk
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Annotating named entities in Twitter data with crowdsourcing
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
MTurk crowdsourcing: a viable method for rapid discovery of Arabic nicknames?
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Using Mechanical Turk to annotate lexicons for less commonly used languages
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Opinion mining of Spanish customer comments with non-expert annotations on Mechanical Turk
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Crowdsourcing and language studies: the new generation of linguistic data
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Not-so-latent dirichlet allocation: collapsed Gibbs sampling using human judgments
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Collecting image annotations using Amazon's Mechanical Turk
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Non-expert evaluation of summarization systems is risky
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Crowdsourcing document relevance assessment with Mechanical Turk
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Preliminary experience with Amazon's Mechanical Turk for annotating medical named entities
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Amazon Mechanical Turk for subjectivity word sense disambiguation
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Non-expert correction of automatically generated relation annotations
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Using Mechanical Turk to build machine translation evaluation sets
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Rethinking grammatical error annotation and evaluation with the Amazon Mechanical Turk
IUNLPBEA '10 Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications
Anveshan: a framework for analysis of multiple annotators' labeling behavior
LAW IV '10 Proceedings of the Fourth Linguistic Annotation Workshop
PackPlay: mining semantic data in collaborative games
LAW IV '10 Proceedings of the Fourth Linguistic Annotation Workshop
No sentence is too confusing to ignore
NLPLING '10 Proceedings of the 2010 Workshop on NLP and Linguistics: Finding the Common Ground
Incorporating content structure into text analysis applications
EMNLP '10 Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing
NLP on spoken documents without ASR
EMNLP '10 Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing
Computer Speech and Language
Scalable crisis relief: Crowdsourced SMS translation and categorization with Mission 4636
Proceedings of the First ACM Symposium on Computing for Development
ICWE'10 Proceedings of the 10th international conference on Current trends in web engineering
Everyone's an influencer: quantifying influence on twitter
Proceedings of the fourth ACM international conference on Web search and data mining
Towards a research agenda for enterprise crowdsourcing
ISoLA'10 Proceedings of the 4th international conference on Leveraging applications of formal methods, verification, and validation - Volume Part I
Text mining for automatic image tagging
COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
MT error detection for cross-lingual question answering
COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
Human-assisted graph search: it's okay to ask questions
Proceedings of the VLDB Endowment
Designing incentives for inexpert human raters
Proceedings of the ACM 2011 conference on Computer supported cooperative work
Filtering microblogging messages for social tv
Proceedings of the 20th international conference companion on World wide web
A word at a time: computing word relatedness using temporal semantic analysis
Proceedings of the 20th international conference on World wide web
Turkalytics: analytics for human computation
Proceedings of the 20th international conference on World wide web
Taxonomy induction based on a collaboratively built knowledge repository
Artificial Intelligence
SDDB: a self-dependent and data-based method for constructing bilingual dictionary from the web
APWeb'11 Proceedings of the 13th Asia-Pacific web conference on Web technologies and applications
Design and implementation of relevance assessments using crowdsourcing
ECIR'11 Proceedings of the 33rd European conference on Advances in information retrieval
HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1
HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1
Crowdsourcing translation: professional quality from non-professionals
HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1
Unsupervised discovery of domain-specific knowledge from text
HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1
Automatic labelling of topic models
HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1
Using query patterns to learn the duration of events
IWCS '11 Proceedings of the Ninth International Conference on Computational Semantics
Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking
Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
Identifying aspects for web-search queries
Journal of Artificial Intelligence Research
How good is the crowd at "real" WSD?
LAW V '11 Proceedings of the 5th Linguistic Annotation Workshop
Reducing the need for double annotation
LAW V '11 Proceedings of the 5th Linguistic Annotation Workshop
Crowdsourcing word sense definition
LAW V '11 Proceedings of the 5th Linguistic Annotation Workshop
Deriving the Pricing Power of Product Features by Mining Consumer Reviews
Management Science
Detecting adversarial advertisements in the wild
Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining
Colourful language: measuring word-colour associations
CMCL '11 Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics
How can you say such things?!?: recognizing disagreement in informal political argument
LSM '11 Proceedings of the Workshop on Languages in Social Media
Paraphrase fragment extraction from monolingual comparable corpora
BUCC '11 Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web
Learning from multiple annotators with Gaussian processes
ICANN'11 Proceedings of the 21st international conference on Artificial neural networks - Volume Part II
Generalized agreement statistics over fixed group of experts
ECML PKDD'11 Proceedings of the 2011 European conference on Machine learning and knowledge discovery in databases - Volume Part III
ECML PKDD'11 Proceedings of the 2011 European conference on Machine learning and knowledge discovery in databases - Volume Part III
ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP) - Special section on ACM multimedia 2010 best paper candidates, and issue on social media
Moving towards adaptive search in digital libraries
NLP4DL'09/AT4DL'09 Proceedings of the 2009 international conference on Advanced language technologies for digital libraries
Readability annotation: replacing the expert by the crowd
IUNLPBEA '11 Proceedings of the 6th Workshop on Innovative Use of NLP for Building Educational Applications
Building a cross-language entity linking collection in twenty-one languages
CLEF'11 Proceedings of the Second international conference on Multilingual and multimodal information access evaluation
The importance of visual context clues in multimedia translation
CLEF'11 Proceedings of the Second international conference on Multilingual and multimodal information access evaluation
Instrumenting the crowd: using implicit behavioral measures to predict task performance
Proceedings of the 24th annual ACM symposium on User interface software and technology
CrowdForge: crowdsourcing complex work
Proceedings of the 24th annual ACM symposium on User interface software and technology
High-throughput crowdsourcing mechanisms for complex tasks
SocInfo'11 Proceedings of the Third international conference on Social informatics
3D corpus of spontaneous complex mental states
ACII'11 Proceedings of the 4th international conference on Affective computing and intelligent interaction - Volume Part I
Guess what? a game for affective annotation of video using crowd sourcing
ACII'11 Proceedings of the 4th international conference on Affective computing and intelligent interaction - Volume Part I
Facilitating pattern discovery for relation extraction with semantic-signature-based clustering
Proceedings of the 20th ACM international conference on Information and knowledge management
Worker types and personality traits in crowdsourcing relevance labels
Proceedings of the 20th ACM international conference on Information and knowledge management
Crowdsourcing visual detectors for video search
MM '11 Proceedings of the 19th ACM international conference on Multimedia
What determines inter-coder agreement in manual annotations? a meta-analytic investigation
Computational Linguistics
LaTeCH '11 Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities
Towards strict sentence intersection: decoding and evaluation strategies
MTTG '11 Proceedings of the Workshop on Monolingual Text-To-Text Generation
WebSets: extracting sets of entities from the web using unsupervised information extraction
Proceedings of the fifth ACM international conference on Web search and data mining
Crowdsourcing assessments for XML ranked retrieval
ECIR'2010 Proceedings of the 32nd European conference on Advances in Information Retrieval
Evaluating unsupervised learning for natural language processing tasks
EMNLP '11 Proceedings of the First Workshop on Unsupervised Learning in NLP
Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work
Collaborative workflow for crowdsourcing translation
Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work
Data-driven response generation in social media
EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Divide and conquer: crowdsourcing the creation of cross-lingual textual entailment corpora
EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Active learning with Amazon Mechanical Turk
EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Sentence-Level attachment prediction
IRFC'10 Proceedings of the First international Information Retrieval Facility conference on Adbances in Multidisciplinary Retrieval
Eliminating spammers and ranking annotators for crowdsourced labeling tasks
The Journal of Machine Learning Research
A multimedia retrieval framework based on automatic graded relevance judgments
MMM'12 Proceedings of the 18th international conference on Advances in Multimedia Modeling
Active cleaning for video corpus annotation
MMM'12 Proceedings of the 18th international conference on Advances in Multimedia Modeling
LemonAid: selection-based crowdsourced contextual help for web applications
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CrowdScreen: algorithms for filtering data with humans
SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
Large-scale machine learning at twitter
SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
Impacts of machine translation and speech synthesis on speech-to-speech translation
Speech Communication
Generating ground truth for music mood classification using mechanical turk
Proceedings of the 12th ACM/IEEE-CS joint conference on Digital Libraries
An iterative reliability measure for semi-anonymous annotators
Proceedings of the 12th ACM/IEEE-CS joint conference on Digital Libraries
Building subjectivity lexicon(s) from scratch for essay data
CICLing'12 Proceedings of the 13th international conference on Computational Linguistics and Intelligent Text Processing - Volume Part I
On aggregating labels from multiple crowd workers to infer relevance of documents
ECIR'12 Proceedings of the 34th European conference on Advances in Information Retrieval
Did it happen? the pragmatic complexity of veridicality assessment
Computational Linguistics
A human study of patch maintainability
Proceedings of the 2012 International Symposium on Software Testing and Analysis
Learning from crowds in the presence of schools of thought
Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
Large-scale learning of word relatedness with constraints
Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
Active learning with c-certainty
PAKDD'12 Proceedings of the 16th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining - Volume Part I
Combining human and machine intelligence in large-scale crowdsourcing
Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1
Quality through flow and immersion: gamifying crowdsourced relevance assessments
SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Crowdsourcing research opportunities: lessons from natural language processing
Proceedings of the 12th International Conference on Knowledge Management and Knowledge Technologies
Using crowdsourcing for TREC relevance assessment
Information Processing and Management: an International Journal
CrowdScape: interactively visualizing user behavior and output
Proceedings of the 25th annual ACM symposium on User interface software and technology
CLex: a lexicon for exploring color, concept and emotion associations in language
EACL '12 Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics
Expectations of word sense in parallel corpora
NAACL HLT '12 Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Mind the gap: learning to choose gaps for question generation
NAACL HLT '12 Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations
Language Resources and Evaluation
FrameNet, current collaborations and future goals
Language Resources and Evaluation
SemEval-2012 task 2: measuring degrees of relational similarity
SemEval '12 Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation
Towards multimodal deception detection -- step 1: building a collection of deceptive videos
Proceedings of the 14th ACM international conference on Multimodal interaction
Workflow transparency in a microtask marketplace
Proceedings of the 17th ACM international conference on Supporting group work
Language identification for creating language-specific Twitter collections
LSM '12 Proceedings of the Second Workshop on Language in Social Media
Fast online lexicon learning for grounded language acquisition
ACL '12 Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1
Improving word representations via global context and multiple word prototypes
ACL '12 Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1
Ecological evaluation of persuasive messages using Google AdWords
ACL '12 Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1
Crowdsourcing inference-rule evaluation
ACL '12 Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2
ACL '12 Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2
Crowdsouring in emotion studies across time and culture
Proceedings of the ACM multimedia 2012 workshop on Crowdsourcing for multimedia
Crowdsourcing micro-level multimedia annotations: the challenges of evaluation and interface
Proceedings of the ACM multimedia 2012 workshop on Crowdsourcing for multimedia
EMNLP-CoNLL '12 Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning
Concept-based indexing of annotated images using semantic DNA
Engineering Applications of Artificial Intelligence
Supporting factual statements with evidence from the web
Proceedings of the 21st ACM international conference on Information and knowledge management
Proceedings of the 21st ACM international conference on Information and knowledge management
VisualizIR: a game for identifying and categorizing relevant text in documents
Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design
PKAW'12 Proceedings of the 12th Pacific Rim conference on Knowledge Management and Acquisition for Intelligent Systems
Situated cognition and knowledge acquisition research
International Journal of Human-Computer Studies
GeoCrowd: enabling query answering with spatial crowdsourcing
Proceedings of the 20th International Conference on Advances in Geographic Information Systems
Locational relativity and domain constraints in spatial questions
Proceedings of the 20th International Conference on Advances in Geographic Information Systems
Enhancing reliability using peer consistency evaluation in human computation
Proceedings of the 2013 conference on Computer supported cooperative work
Co-worker transparency in a microtask marketplace
Proceedings of the 2013 conference on Computer supported cooperative work
Proceedings of the 2013 conference on Computer supported cooperative work
Supervised collaboration for syntactic annotation of Quranic Arabic
Language Resources and Evaluation
Perspectives on crowdsourcing annotations for natural language processing
Language Resources and Evaluation
Creating a system for lexical substitutions from scratch using crowdsourcing
Language Resources and Evaluation
Language Resources and Evaluation
Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation
ACM Transactions on Interactive Intelligent Systems (TiiS) - Special section on internet-scale human problem solving and regular papers
Leveraging the crowd to improve feature-sentiment analysis of user reviews
Proceedings of the 2013 international conference on Intelligent user interfaces
An introduction to crowdsourcing for language and multimedia technology research
PROMISE'12 Proceedings of the 2012 international conference on Information Retrieval Meets Information Visualization
Crowdsourcing for information retrieval: introduction to the special issue
Information Retrieval
Increasing cheat robustness of crowdsourcing tasks
Information Retrieval
An analysis of human factors and label accuracy in crowdsourcing relevance judgments
Information Retrieval
Identifying top news using crowdsourcing
Information Retrieval
Implementing crowdsourcing-based relevance experimentation: an industrial perspective
Information Retrieval
Tagging human activities in video by crowdsourcing
Proceedings of the 3rd ACM conference on International conference on multimedia retrieval
Turkopticon: interrupting worker invisibility in amazon mechanical turk
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Paraphrase acquisition via crowdsourcing and machine learning
ACM Transactions on Intelligent Systems and Technology (TIST) - Special Sections on Paraphrasing; Intelligent Systems for Socially Aware Computing; Social Computing, Behavioral-Cultural Modeling, and Prediction
Beliefs and biases in web search
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
News vertical search: when and what to display to users
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Evaluating the crowd with confidence
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Statistical quality estimation for general crowdsourcing tasks
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Aggregating crowdsourced binary ratings
Proceedings of the 22nd international conference on World Wide Web
A threshold method for imbalanced multiple noisy labeling
Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining
Learning from multiple annotators: Distinguishing good from random labelers
Pattern Recognition Letters
Games with a Purpose or Mechanised Labour?: A Comparative Study
Proceedings of the 13th International Conference on Knowledge Management and Knowledge Technologies
A Named Entity Recognition Method Based on Decomposition and Concatenation of Word Chunks
ACM Transactions on Asian Language Information Processing (TALIP)
MedLDA: maximum margin supervised topic models
The Journal of Machine Learning Research
Selective sampling and active learning from single and multiple teachers
The Journal of Machine Learning Research
SRbench--a benchmark for soundtrack recommendation systems
Proceedings of the 22nd ACM international conference on Conference on information & knowledge management
Reporting bias and knowledge acquisition
Proceedings of the 2013 workshop on Automated knowledge base construction
Improving Text Classification Accuracy by Training Label Cleaning
ACM Transactions on Information Systems (TOIS)
GeoTruCrowd: trustworthy query answering with spatial crowdsourcing
Proceedings of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems
Maximizing the number of worker's self-selected tasks in spatial crowdsourcing
Proceedings of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems
A lightweight combinatorial approach for inferring the ground truth from multiple annotators
MLDM'13 Proceedings of the 9th international conference on Machine Learning and Data Mining in Pattern Recognition
Crowd synthesis: extracting categories and clusters from complex data
Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing
Accurate integration of crowdsourced labels using workers' self-reported confidence scores
IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Inferring mood in ubiquitous conversational video
Proceedings of the 12th International Conference on Mobile and Ubiquitous Multimedia
Trust, but verify: predicting contribution quality for knowledge base construction and curation
Proceedings of the 7th ACM international conference on Web search and data mining
Proceedings of the 19th international conference on Intelligent User Interfaces
Learning an accurate entity resolution model from crowdsourced labels
Proceedings of the 8th International Conference on Ubiquitous Information Management and Communication
Visual tracking via weakly supervised learning from multiple imperfect oracles
Pattern Recognition
Learning classification models from multiple experts
Journal of Biomedical Informatics
Expert Systems with Applications: An International Journal
STFU NOOB!: predicting crowdsourced decisions on toxic behavior in online games
Proceedings of the 23rd international conference on World wide web
The wisdom of minority: discovering and targeting the right group of workers for crowdsourcing
Proceedings of the 23rd international conference on World wide web
MediaQ: mobile multimedia management system
Proceedings of the 5th ACM Multimedia Systems Conference
Contextual keyword extraction by building sentences with crowdsourcing
Multimedia Tools and Applications
Crowdsourced Knowledge Acquisition: Towards Hybrid-Genre Workflows
International Journal on Semantic Web & Information Systems
Repeated labeling using multiple noisy labelers
Data Mining and Knowledge Discovery
Evaluation in Music Information Retrieval
Journal of Intelligent Information Systems
Multimedia Tools and Applications
Bucking the trend: improved evaluation and annotation practices for ESL error detection systems
Language Resources and Evaluation
Hi-index | 0.00 |
Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We explore the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. For all five, we show high agreement between Mechanical Turk non-expert annotations and existing gold standard labels provided by expert labelers. For the task of affect recognition, we also show that using non-expert labels for training machine learning algorithms can be as effective as using gold standard annotations from experts. We propose a technique for bias correction that significantly improves annotation quality on two tasks. We conclude that many large labeling tasks can be effectively designed and carried out in this method at a fraction of the usual expense.