An agent-based approach for building complex software systems
Communications of the ACM
Agent Intelligence Through Data Mining (Multiagent Systems, Artificial Societies, and Simulated Organizations)
Reinforcement learning: a survey
Journal of Artificial Intelligence Research
Evaluating knowledge intensive multi-agent systems
AIS-ADM'07 Proceedings of the 2nd international conference on Autonomous intelligent systems: agents and data mining
| Hi-index | 0.00 |
Although the need for well-established engineering approaches in Intelligent Systems (IS) performance evaluation is urging, currently no widely accepted methodology exists, mainly due to lack of consensus on relevant definitions and scope of applicability, multi-disciplinary issues and immaturity of the field of IS. Even existing well-tested evaluation methodologies applied in other domains, such as (traditional) software engineering, prove inadequate to address the unpredictable emerging factors of the behavior of intelligent components. In this paper, we present a generic methodology and associated tools for evaluating the performance of IS, by exploiting the software agent paradigm as a representative modeling concept for intelligent systems. Based on the assessment of observable behavior of agents or multi-agent systems, the proposed methodology provides a concise set of guidelines and representation tools for evaluators to use. The methodology comprises three main tasks, namely metrics selection, monitoring agent activities for appropriate measurements, and aggregation of the conducted measurements. Coupled to this methodology is the Evaluator Agent Framework, which aims at the automation of most of the provided steps of the methodology, by providing Graphical User Interfaces for metrics organization and results presentation, as well as a code generating module that produces a skeleton of a monitoring agent. Once this agent is completed with domain-specific code, it is appended to the runtime of a multi-agent system and collects information from observable events and messages. Both the evaluation methodology and the automation framework are tested and demonstrated in Symbiosis, a MAS simulation environment for competing groups of autonomous entities.