Resource usage monitoring for web systems using real-time statistical analysis of log data

  • Authors:
  • Matsuki Yoshino;Atsuro Handa;Norihisa Komoda;Michiko Oba

  • Affiliations:
  • Software Division, Hitachi Ltd., Yokohama, Japan;Software Division, Hitachi Ltd., Yokohama, Japan;Graduate School of Information Science and Technology, Osaka University, Suita, Osaka, Japan;The School of Systems Information Science, Future University Hakodate, Hakodate, Hokkaido, Japan

  • Venue:
  • MAMECTIS/NOLASC/CONTROL/WAMUS'11 Proceedings of the 13th WSEAS international conference on mathematical methods, computational techniques and intelligent systems, and 10th WSEAS international conference on non-linear analysis, non-linear systems and chaos, and 7th WSEAS international conference on dynamical systems and control, and 11th WSEAS international conference on Wavelet analysis and multirate systems: recent researches in computational techniques, non-linear systems and control
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

For Web-based systems accessible from the Internet, it is difficult to estimate workloads precisely. Precise estimation of resources necessary for the system is important for effective utilization of resources in a datacenter. Therefore, capacity planning to forecast the amount of IT resources necessary for a system is important. In capacity planning, the amount of resources necessary for a system is calculated based upon numbers determined by the architecture and business requirements of the system. An example of a number determined by the architecture is the amount of memory required by a single user. An example of a number determined by business requirements is the estimated maximum number of simultaneous users. By multiplying these two numbers, a maximum memory requirement can be calculated. Usually, system memory consumption and the number of simultaneous users are monitored during operation, and if either value exceeds a threshold, an alarm is sent to operators. The authors propose a method to monitor memory consumption per user from memory consumption data and the number of users, and perform statistical significance testing in real time by applying a stream database. The window size used in a CQL statement for the test affects the precision of the test and memory consumption of the stream database. Through experimentation, the authors propose an optimal window size.