Why CSCW applications fail: problems in the design and evaluationof organizational interfaces
CSCW '88 Proceedings of the 1988 ACM conference on Computer-supported cooperative work
Cognitive walkthroughs: a method for theory-based evaluation of user interfaces
International Journal of Man-Machine Studies
Getting around the task-artifact cycle: how to make claims and design by scenario
ACM Transactions on Information Systems (TOIS)
Groupware and social dynamics: eight challenges for developers
Communications of the ACM
Usability inspection methods
Moving out from the control room: ethnography in system design
CSCW '94 Proceedings of the 1994 ACM conference on Computer supported cooperative work
Situated evaluation for cooperative systems
CSCW '94 Proceedings of the 1994 ACM conference on Computer supported cooperative work
Technological frames: making sense of information technology in organizations
ACM Transactions on Information Systems (TOIS) - Special issue on social science perspectives on IS
Scenario-based design: envisioning work and technology in system development
Scenario-based design: envisioning work and technology in system development
Scenarios in discount usability engineering
Scenario-based design
Design space analysis and use representations
Scenario-based design
Evaluating usability evaluation techniques
ACM Computing Surveys (CSUR) - Special issue: position statements on strategic directions in computing research
Recognising “success” and “failure”: evaluating groupware in a commercial context
GROUP '97 Proceedings of the international ACM SIGGROUP conference on Supporting group work: the integration challenge
Software architecture in practice
Software architecture in practice
Contextual design: defining customer-centered systems
Contextual design: defining customer-centered systems
The Use of Cooperation Scenarios in the Design and Evaluation of a CSCW System
IEEE Transactions on Software Engineering
Evaluation for collaborative systems
ACM Computing Surveys (CSUR)
Methodologies for evaluation of collaborative systems
ACM SIGGROUP Bulletin
Methodologies for evaluation of collaborative systems workshop
ACM SIGGROUP Bulletin
An Integrated Approach to Designing and Evaluating CollaborativeApplications and Infrastructures
Computer Supported Cooperative Work
A comparison of usage evaluation and inspection methods for assessing groupware usability
GROUP '01 Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work
Groupware walkthrough: adding context to groupware usability evaluation
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Domain Theory: Patterns for Knowledge and Software Reuse
Domain Theory: Patterns for Knowledge and Software Reuse
Making Use: Scenario-Based Design of Human-Computer Interactions
Making Use: Scenario-Based Design of Human-Computer Interactions
Usability Engineering
Workshop on methods and techniques for software architecture review and assessment (SARA)
Proceedings of the 24th International Conference on Software Engineering
Scenario-based evaluation of loosely-integrated collaborative systems
CHI '00 Extended Abstracts on Human Factors in Computing Systems
"Bloat": the objective and subject dimensions
CHI '00 Extended Abstracts on Human Factors in Computing Systems
Return on investment and organizational adoption
CSCW '04 Proceedings of the 2004 ACM conference on Computer supported cooperative work
Tasks and scenario-based evaluation of information visualization techniques
TAMODIA '04 Proceedings of the 3rd annual conference on Task models and diagrams
ICITA '05 Proceedings of the Third International Conference on Information Technology and Applications (ICITA'05) Volume 2 - Volume 02
Damaged merchandise? a review of experiments that compare usability evaluation methods
Human-Computer Interaction
The intellectual challenge of CSCW: the gap between social requirements and technical feasibility
Human-Computer Interaction
Training systems design: bridging the gap between users and developers using storyboards
Proceedings of the 29th Annual European Conference on Cognitive Ergonomics
Event-driven adaptive collaboration using semantically-enriched patterns
Expert Systems with Applications: An International Journal
Caracterização das adaptações em métodos de avaliação para aplicações colaborativas
Proceedings of the 11th Brazilian Symposium on Human Factors in Computing Systems
Hi-index | 0.00 |
Evaluating collaborative systems remains a significant challenge. Most evaluation methods approach the problem from one of two extremes: focused evaluation of specific system features, or broad ethnographic investigations of system use in context. In this paper, we develop and demonstrate a middle ground for evaluation: explicit reflections on scenarios of system use coupled with analysis of the consequences of these use scenarios, represented as claims. Extending prior work in scenario-based design and claims analysis, we develop a framework for a multi-perspective, multi-level evaluation of collaborative systems called SWIMs: scenario walkthrough and inspection methods. This approach is centered on the capture, aggregation, and analysis of users' reflections on system support for specific scenarios. We argue that this approach not only provides the contextual sensitivity and use centricity of ethnographic techniques, but also sufficient structure for method replication, which is common to more feature-based evaluation techniques. We demonstrate with an extensive field study how SWIMs can be used for summative assessment of system performance and organizational contributions, and formative assessment to guide system and feature re-design. Results from the field study provide preliminary indications of the method's effectiveness and suggest directions for future research.