The role of iterative evaluation in designing systems for usability
Proceedings of the Second Conference of the British Computer Society, human computer interaction specialist group on People and computers: designing for usability
Plans and situated actions: the problem of human-machine communication
Plans and situated actions: the problem of human-machine communication
Ethnographically-informed systems design for air traffic control
CSCW '92 Proceedings of the 1992 ACM conference on Computer-supported cooperative work
Evaluating hypermedia and learning: methods and results from the Perseus Project
ACM Transactions on Information Systems (TOIS)
An applied ethnographic method for redesigning user interfaces
Proceedings of the 1st conference on Designing interactive systems: processes, practices, methods, & techniques
The design and long-term use of a personal electronic notebook: a reflective analysis
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Designing with ethnography: a presentation framework for design
DIS '97 Proceedings of the 2nd conference on Designing interactive systems: processes, practices, methods, and techniques
Contextual design: defining customer-centered systems
Contextual design: defining customer-centered systems
Creating creativity: user interfaces for supporting innovation
ACM Transactions on Computer-Human Interaction (TOCHI) - Special issue on human-computer interaction in the new millennium, Part 1
Rapid ethnography: time deepening strategies for HCI field research
DIS '00 Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques
Turning pictures into numbers: extracting and generating information from complex visualizations
International Journal of Human-Computer Studies - Empirical evaluation of information visualizations
A Practical Guide to Usability Testing
A Practical Guide to Usability Testing
Groupware and computer-supported cooperative work
The human-computer interaction handbook
Explorations in art and technology
Explorations in art and technology
The challenge of information visualization evaluation
Proceedings of the working conference on Advanced visual interfaces
An Evaluation of Microarray Visualization Tools for Biological Insight
INFOVIS '04 Proceedings of the IEEE Symposium on Information Visualization
IEEE Transactions on Visualization and Computer Graphics
Informing the design of computer-based environments to support creativity
International Journal of Human-Computer Studies - Special issue: Computer support for creativity
The studio as laboratory: combining creative practice and digital technology research
International Journal of Human-Computer Studies - Special issue: Computer support for creativity
Group Support Systems: A Descriptive Evaluation of Case and Field Studies
Journal of Management Information Systems
Balancing Systematic and Flexible Exploration of Social Networks
IEEE Transactions on Visualization and Computer Graphics
Establishing the utility of non-text search for news video retrieval with real world users
Proceedings of the 15th international conference on Multimedia
Promoting Insight-Based Evaluation of Visualizations: From Contest to Benchmark Repository
IEEE Transactions on Visualization and Computer Graphics
Creativity support tools: accelerating discovery and innovation
Communications of the ACM
Users can change their web search tactics: Design guidelines for categorized overviews
Information Processing and Management: an International Journal
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
LiveRAC: interactive visual exploration of system management time-series data
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Access to recorded interviews: A research agenda
Journal on Computing and Cultural Heritage (JOCCH)
Designing semantic substrates for visual network exploration
Information Visualization
Increasing the utility of quantitative empirical studies for meta-analysis
Proceedings of the 2008 Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization
Beyond time and error: a cognitive approach to the evaluation of graph drawings
Proceedings of the 2008 Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization
Proceedings of the 2008 Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization
Internalization, qualitative methods, and evaluation
Proceedings of the 2008 Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization
Grounded evaluation of information visualizations
Proceedings of the 2008 Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization
Qualitative analysis of visualization: a building design field study
Proceedings of the 2008 Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization
Using multi-dimensional in-depth long-term case studies for information visualization evaluation
Proceedings of the 2008 Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization
Systematic yet flexible discovery: guiding domain experts through exploratory data analysis
Proceedings of the 13th international conference on Intelligent user interfaces
Evaluating the contributions of video representation for a life oral history collection
Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries
Supporting video library exploratory search: when storyboards are not enough
CIVR '08 Proceedings of the 2008 international conference on Content-based image and video retrieval
Plastic: a metaphor for integrated technologies
UbiComp '08 Proceedings of the 10th international conference on Ubiquitous computing
Evaluating Information Visualizations
Information Visualization
Process and Pitfalls in Writing Information Visualization Research Papers
Information Visualization
Visualizations at First Sight: Do Insights Require Training?
USAB '08 Proceedings of the 4th Symposium of the Workgroup Human-Computer Interaction and Usability Engineering of the Austrian Computer Society on HCI and Usability for Education and Work
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Visual overviews for discovering key papers and influences across research fronts
Journal of the American Society for Information Science and Technology
IEEE Computer Graphics and Applications - Special issue on sketching tangible interfaces augmented reality on mobile phones
Motivation and procrastination: methods for evaluating pragmatic casual information visualizations
IEEE Computer Graphics and Applications - Special issue on non-photorealistic rendering a virtual environment for teaching social skills
Making sense of archived e-mail: Exploring the Enron collection with NetLens
Journal of the American Society for Information Science and Technology
ManyNets: an interface for multiple network analysis and visualization
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Using concept maps to evaluate the usability of APIs
CHI '10 Extended Abstracts on Human Factors in Computing Systems
Measuring effective data visualization
ISVC'07 Proceedings of the 3rd international conference on Advances in visual computing - Volume Part II
No Code Required: Giving Users Tools to Transform the Web
No Code Required: Giving Users Tools to Transform the Web
Revealing uncertainty for information visualization
Information Visualization
Exploring temporal patterns with information visualization: keynote
Proceedings of Graphics Interface 2010
An heuristic set for evaluation in information visualization
Proceedings of the International Conference on Advanced Visual Interfaces
AutoLearn's authoring tool: a piece of cake for teachers
IUNLPBEA '10 Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications
Zinsight: a visual and analytic environment for exploring large event traces
Proceedings of the 5th international symposium on Software visualization
Proceedings of the 1st ACM International Health Informatics Symposium
Information and Software Technology
Evaluating video visualizations of human behavior
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Cardiogram: visual analytics for automotive engineers
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
LifeFlow: visualizing an overview of event sequences
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CHI '11 Extended Abstracts on Human Factors in Computing Systems
Using gaze data in evaluating interactive visualizations
HCIV'09 Proceedings of the Second IFIP WG 13.7 conference on Human-computer interaction and visualization
Technology-mediated social participation: the next 25 years of HCI challenges
HCII'11 Proceedings of the 14th international conference on Human-computer interaction: design and development approaches - Volume Part I
Insitu: sketching architectural designs in context
Proceedings of the 2011 SIGGRAPH Asia Conference
Learning-based evaluation of visual analytic systems
Proceedings of the 3rd BELIV'10 Workshop: BEyond time and errors: novel evaLuation methods for Information Visualization
Comparing benchmark task and insight evaluation methods on timeseries graph visualizations
Proceedings of the 3rd BELIV'10 Workshop: BEyond time and errors: novel evaLuation methods for Information Visualization
Evaluating information visualization in large companies: challenges, experiences and recommendations
Proceedings of the 3rd BELIV'10 Workshop: BEyond time and errors: novel evaLuation methods for Information Visualization
Activities & artifacts: the dual nature of image-making in communicative practice
Proceedings of the 2012 iConference
StaTube: facilitating state management in instant messaging systems
Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction
Technical Section: Semantics by analogy for illustrative volume visualization
Computers and Graphics
JigsawMap: connecting the past to the future by mapping historical textual cadasters
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Querying event sequences by exact match or similarity search: Design and empirical evaluation
Interacting with Computers
Supporting user of information retrieval systems by visualizing the information dialog context
Proceedings of the International Working Conference on Advanced Visual Interfaces
A comparison of benchmark task and insight evaluation methods for information visualization
Information Visualization - Special issue on Evaluation for Information Visualization
Developing guidelines for assessing visual analytics environments
Information Visualization - Special issue on Evaluation for Information Visualization
Information visualization evaluation in large companies: challenges, experiences and recommendations
Information Visualization - Special issue on Evaluation for Information Visualization
Bisociative Knowledge Discovery
Life on the line: interacting with temporal event sequence representations
Diagrams'12 Proceedings of the 7th international conference on Diagrammatic Representation and Inference
EACL 2012 Proceedings of the EACL 2012 Joint Workshop of LINGVIS & UNCLH
Evaluation of visual aid suite for desktop searching
ECDL'07 Proceedings of the 11th European conference on Research and Advanced Technology for Digital Libraries
Journal of the American Society for Information Science and Technology
Experiences in involving analysts in visualisation design
Proceedings of the 2012 BELIV Workshop: Beyond Time and Errors - Novel Evaluation Methods for Visualization
Evaluating visualization using cognitive measures
Proceedings of the 2012 BELIV Workshop: Beyond Time and Errors - Novel Evaluation Methods for Visualization
How Do Users Search the Mobile Web with a Clustering Interface?: A Longitudinal Study
International Journal of Mobile Human Computer Interaction
GD'12 Proceedings of the 20th international conference on Graph Drawing
Evaluation methods for creativity support environments
CHI '13 Extended Abstracts on Human Factors in Computing Systems
Adaptive visualization for exploratory information retrieval
Information Processing and Management: an International Journal
Hi-index | 0.03 |
After an historical review of evaluation methods, we describe an emerging research method called Multi-dimensional In-depth Long-term Case studies (MILCs) which seems well adapted to study the creative activities that users of information visualization systems engage in. We propose that the efficacy of tools can be assessed by documenting 1) usage (observations, interviews, surveys, logging etc.) and 2) expert users' success in achieving their professional goals. We summarize lessons from related ethnography methods used in HCI and provide guidelines for conducting MILCs for information visualization. We suggest ways to refine the methods for MILCs in modest sized projects and then envision ambitious projects with 3-10 researchers working over 1-3 years to understand individual and organizational use of information visualization by domain experts working at the frontiers of knowledge in their fields.