Studying interaction with documentation: methodological lessons learned in two semi-naturalistic designs

• Authors: Brigit van Loggem
• Venue: Proceedings of the 30th European Conference on Cognitive Ergonomics
• Year: 2012

Semi-naturalistic research designs allow for studying behaviour in a realistic setting, achieving a fair degree of ecological validity without the disadvantages of purely naturalistic designs. A semi-naturalistic study sets boundaries for the behaviour under investigation, within which respondents still act freely. In order to allow for between-subjects comparisons, the raw data obtained must be structured, either through pre-structuring or through post-structuring. Motivation -- There exists little methodological guidance in the field of documentation design. Qualitative and quantitative studies alike are carried out using an amalgam of methods that were developed for other disciplines. This paper contributes to awareness of the pitfalls (but also the benefits) of doing so. Research approach -- This paper considers two semi-naturalistic studies into interaction with software and documentation from a methodological point of view. In the first, the data was collected in the respondents' workplace and then post-structured. In the second, the data was collected in a laboratory setting and pre-structured through the use of an observation tool. Findings/Design -- Both methods are described in some detail, followed by a discussion of methodological issues discovered after the design had been executed. Finally, the relative advantages and disadvantages of the two approaches are highlighted. Take-away message -- Documentation design can fruitfully combine methodological approaches originally developed for other disciplines, provided these are adapted for the purpose with care and discretion.