Promoting Insight-Based Evaluation of Visualizations: From Contest to Benchmark Repository
IEEE Transactions on Visualization and Computer Graphics
PerMIS '08 Proceedings of the 8th Workshop on Performance Metrics for Intelligent Systems
A scenario based analysis of e-collaboration environments
ITS'12 Proceedings of the 11th international conference on Intelligent Tutoring Systems
| Hi-index | 0.00 |
Collaboration technologies are seeing widespread deployment even though it is difficult to assess the effectiveness of these systems. This paper presents an evaluation method that addresses this issue and the use of the method in the field. The method uses a framework to structure evaluations by mapping system goals to evaluation objectives, metrics, and measures. The upper-most levels of the framework are conceptual in nature, while the bottom level is implementation-specific, i.e., evaluation-specific. Capitalizing on this top-down approach, an evaluation template specifying the conceptual elements can be constructed for a series of evaluations. Then implementation-specific measures are specified for individual experiments. This structure makes use of the framework ideal for comparison of the effectiveness of collaboration tools in a particular environment. We present our findings from use of the method in the field to assess the performance of a particular collaboration technology deployment and its impact on the work process.