An approach to benchmarking configuration complexity
Proceedings of the 11th workshop on ACM SIGOPS European workshop
Reducing the cost of IT operations: is automation always the answer?
HOTOS'05 Proceedings of the 10th conference on Hot Topics in Operating Systems - Volume 10
On the roles of policies in computer systems management
International Journal of Human-Computer Studies
| Hi-index | 0.00 |
In previous work we laid out an approach to quantifying configuration complexity [3]. In that earlier work, we explicitly focused on complexity as experienced by expert systems managers, and thus looked at straight-line configuration procedures, ignoring the complexity faced by non-experts as they have to decide what configuration steps to follow. Decision complexity is the complexity faced by a non-expert system administrator---the person providing IT support in a small-business environment, who is confronted by decisions during the configuration process, and is a measure of how easy or hard it is to identify the appropriate sequence of configuration actions to perform in order to achieve a specified configuration goal. To identify spots of high decisionmaking complexity, we need a model of decision complexity for configuring and operating computing systems. This paper extends previous work on models and metrics for IT configuration complexity by adding the concept of decision complexity. As the first step towards a complete model of decision complexity, we describe an extensive user study of decision making in a carefully-mapped analogous domain (route planning), and illustrate how the results of that study suggest an initial model of decision complexity applicable to IT configuration. The model identifies the key factors affecting decision complexity and highlights several interesting results, including the fact that decision complexity has significantly different impacts on user-perceived difficulty than on objective measures like time and error rate. We also describe some of the implications of our decision complexity model for system designers seeking to automate the decision-making and reduce the configuration complexity of their systems.