Evaluating a scientific collaboratory: Results of a controlled experiment

  • Authors:

  • Diane H. Sonnenwald;Mary C. Whitton;Kelly L. Maglaughlin

  • Affiliations:

  • University of North Carolina at Chapel Hill, Chapel Hill, NC;University of North Carolina at Chapel Hill, Chapel Hill, NC;University of North Carolina at Chapel Hill, Chapel Hill, NC

  • Venue:
  • ACM Transactions on Computer-Human Interaction (TOCHI)
  • Year:
  • 2003

Quantified Score

Hi-index 0.00

Visualization

Abstract

The evaluation of scientific collaboratories has lagged behind their development. Do the capabilities afforded by collaboratories outweigh their disadvantages? To evaluate a scientific collaboratory system, we conducted a repeated-measures controlled experiment that compared the outcomes and process of scientific work completed by 20 pairs of participants (upper level undergraduate science students) working face-to-face and remotely. We collected scientific outcomes (graded lab reports) to investigate the quality of scientific work, post-questionnaire data to measure the adoptability of the system, and post-interviews to understand the participants' views of doing science under both conditions. We hypothesized that study participants would be less effective, report more difficulty, and be less favorably inclined to adopt the system when collaborating remotely. Contrary to expectations, the quantitative data showed no statistically significant differences with respect to effectiveness and adoption.The qualitative data helped explain this null result: participants reported advantages and disadvantages working under both conditions and developed work-arounds to cope with the perceived disadvantages of collaborating remotely. While the data analysis produced null results, considered as a whole, the analysis leads us to conclude there is positive potential for the development and adoption of scientific collaboratory systems.