The case for repeated research in operating systems
ACM SIGOPS Operating Systems Review
Dynamic instrumentation of production systems
ATEC '04 Proceedings of the annual conference on USENIX Annual Technical Conference
Comparing the performance of web server architectures
Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007
Our troubles with Linux and why you should care
Proceedings of the Second Asia-Pacific Workshop on Systems
To chunk or not to chunk: implications for HTTP streaming video server performance
Proceedings of the 22nd international workshop on Network and Operating System Support for Digital Audio and Video
Comparing high-performance multi-core web-server architectures
Proceedings of the 5th Annual International Systems and Storage Conference
Methodologies for generating HTTP streaming video workloads to evaluate web server performance
Proceedings of the 5th Annual International Systems and Storage Conference
| Hi-index | 0.00 |
Linux and other open-source Unix variants (and their distributors) provide researchers with full-fledged operating systems that are widely used. However, due to their complexity and rapid development, care should be exercised when using these operating systems for performance experiments, especially in systems research. In particular, the size and continual evolution of the Linux code-base makes it difficult to understand, and as a result, decipher and explain the reasons for performance improvements. In addition, the rapid kernel development cycle means that experimental results can be viewed as out of date, or meaningless, very quickly. We demonstrate that this viewpoint is incorrect because kernel changes can and have introduced both bugs and performance degradations. This paper describes some of our experiences using Linux and FreeBSD as platforms for conducting performance evaluations and some performance regressions we have found. Our results show, these performance regressions can be serious (e.g., repeating identical experiments results in large variability in results) and long lived despite having a large negative effect on performance (one problem was present for more than 3 years). Based on these experiences, we argue: it is sometimes reasonable to use an older kernel version, experimental results need careful analysis to explain why a performance effect occurs, and publishing papers validating prior research is essential.