Automating the design of graphical presentations of relational information
ACM Transactions on Graphics (TOG)
Tree visualization with tree-maps: 2-d space-filling approach
ACM Transactions on Graphics (TOG)
Visualizing Data
CHI '99 Extended Abstracts on Human Factors in Computing Systems
Information Visualization: Perception for Design
Information Visualization: Perception for Design
White rooms and morphing don't mix: setting and the evaluation of visualization techniques
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Perception of elementary graphical elements in tabletop and multi-surface environments
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Touchstone: exploratory design of experiments
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Practical guide to controlled experiments on the web: listen to your customers not to the hippo
Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining
ManyEyes: a Site for Visualization at Internet Scale
IEEE Transactions on Visualization and Computer Graphics
Crowdsourcing user studies with Mechanical Turk
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
The Shaping of Information by Visual Metaphors
IEEE Transactions on Visualization and Computer Graphics
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
How well do line drawings depict shape?
ACM SIGGRAPH 2009 papers
Financial incentives and the "performance of crowds"
Proceedings of the ACM SIGKDD Workshop on Human Computation
Sellers' problems in human computation markets
Proceedings of the ACM SIGKDD Workshop on Human Computation
Exploring iterative and parallel human computation processes
Proceedings of the ACM SIGKDD Workshop on Human Computation
Tracing genealogical data with TimeNets
Proceedings of the International Conference on Advanced Visual Interfaces
Exploring the use of crowdsourcing to support empirical studies in software engineering
Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement
TurKit: human computation algorithms on mechanical turk
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Soylent: a word processor with a crowd inside
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Color compatibility from large datasets
ACM SIGGRAPH 2011 papers
Affective computational priming and creativity
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
The impact of social information on visual judgments
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Utility of human-computer interactions: toward a science of preference measurement
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Data cracker: developing a visual game analytic tool for analyzing online gameplay
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A crowdsourcing model for receiving design critique
CHI '11 Extended Abstracts on Human Factors in Computing Systems
ViewSer: enabling large-scale remote user studies of web search examination and interaction
Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval
Perceptual models of viewpoint preference
ACM Transactions on Graphics (TOG)
Exploring micro-incentive strategies for participant compensation in high-burden studies
Proceedings of the 13th international conference on Ubiquitous computing
CrowdForge: crowdsourcing complex work
Proceedings of the 24th annual ACM symposium on User interface software and technology
ReVision: automated classification, analysis and redesign of chart images
Proceedings of the 24th annual ACM symposium on User interface software and technology
Taking advantage of collective knowledge in emergency response systems
Journal of Network and Computer Applications
Proceedings of the Seventh Symposium on Usable Privacy and Security
Do Mechanical Turks dream of square pie charts?
Proceedings of the 3rd BELIV'10 Workshop: BEyond time and errors: novel evaLuation methods for Information Visualization
Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work Companion
Shepherding the crowd yields better work
Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work
Pushing the boundaries of crowd-enabled databases with query-driven schema expansion
Proceedings of the VLDB Endowment
Towards an integrated crowdsourcing definition
Journal of Information Science
Schelling points on 3D surface meshes
ACM Transactions on Graphics (TOG) - SIGGRAPH 2012 Conference Proceedings
Strategies for crowdsourcing social data analysis
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Shape-changing interfaces: a review of the design space and open research questions
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Socially computed scripts to support social problem solving skills
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Comparing averages in time series data
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CommunitySourcing: engaging local crowds to perform expert work via physical kiosks
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Micro perceptual human computation for visual tasks
ACM Transactions on Graphics (TOG)
Putting humans in the loop: Social computing for Water Resources Management
Environmental Modelling & Software
Effective temporal graph layout: a comparative study of animation versus static display methods
Information Visualization
“Without the clutter of unimportant words”: Descriptive keyphrases for text visualization
ACM Transactions on Computer-Human Interaction (TOCHI)
Rectangle orientation in area judgment task for treemap design
Proceedings of the 24th Australian Computer-Human Interaction Conference
Swayed by friends or by the crowd?
SocInfo'12 Proceedings of the 4th international conference on Social Informatics
The cost of collaboration for code and art: evidence from a remixing community
Proceedings of the 2013 conference on Computer supported cooperative work
The four-level nested model revisited: blocks and guidelines
Proceedings of the 2012 BELIV Workshop: Beyond Time and Errors - Novel Evaluation Methods for Visualization
Patterns for visualization evaluation
Proceedings of the 2012 BELIV Workshop: Beyond Time and Errors - Novel Evaluation Methods for Visualization
How to filter out random clickers in a crowdsourcing-based study?
Proceedings of the 2012 BELIV Workshop: Beyond Time and Errors - Novel Evaluation Methods for Visualization
Lessons Learned from Large-Scale User Studies: Using Android Market as a Source of Data
International Journal of Mobile Human Computer Interaction
Gloss perception in painterly and cartoon rendering
ACM Transactions on Graphics (TOG)
Modeling how people extract color themes from images
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Crowdsourcing performance evaluations of user interfaces
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A pilot study of using crowds in the classroom
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Using fNIRS brain sensing to evaluate information visualization interfaces
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Combining crowdsourcing and google street view to identify street-level accessibility problems
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
TypeRighting: combining the benefits of handwriting and typeface in online educational videos
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CrowdStudy: general toolkit for crowdsourced evaluation of web interfaces
Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems
Influencing visual judgment through affective priming
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Keep it simple: reward and task design in crowdsourcing
Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI
CrowdLearner: rapidly creating mobile recognizers using crowdsourcing
Proceedings of the 26th annual ACM symposium on User interface software and technology
Understand users’ comprehension and preferences for composing information visualizations
ACM Transactions on Computer-Human Interaction (TOCHI)
Recommender narrative visualization
CASCON '13 Proceedings of the 2013 Conference of the Center for Advanced Studies on Collaborative Research
Complementing text entry evaluations with a composition task
ACM Transactions on Computer-Human Interaction (TOCHI)
STFU NOOB!: predicting crowdsourced decisions on toxic behavior in online games
Proceedings of the 23rd international conference on World wide web
EvalBench: a software library for visualization evaluation
EuroVis '13 Proceedings of the 15th Eurographics Conference on Visualization
EuroVis '13 Proceedings of the 15th Eurographics Conference on Visualization
Rule-based visual mappings - with a case study on poetry visualization
EuroVis '13 Proceedings of the 15th Eurographics Conference on Visualization
Selecting semantically-resonant colors for data visualization
EuroVis '13 Proceedings of the 15th Eurographics Conference on Visualization
imMens: real-time visual querying of big data
EuroVis '13 Proceedings of the 15th Eurographics Conference on Visualization
Crowdsourcing traffic simulation to improve signal timing
Journal of Computing Sciences in Colleges
| Hi-index | 0.01 |
Understanding perception is critical to effective visualization design. With its low cost and scalability, crowdsourcing presents an attractive option for evaluating the large design space of visualizations; however, it first requires validation. In this paper, we assess the viability of Amazon's Mechanical Turk as a platform for graphical perception experiments. We replicate previous studies of spatial encoding and luminance contrast and compare our results. We also conduct new experiments on rectangular area perception (as in treemaps or cartograms) and on chart size and gridline spacing. Our results demonstrate that crowdsourced perception experiments are viable and contribute new insights for visualization design. Lastly, we report cost and performance data from our experiments and distill recommendations for the design of crowdsourced studies.