A process for evaluating legal knowledge-based systems based upon the context criteria contingency-guidelines framework

  • Authors:

  • Maria Jean J. Hall;Richard Hall;John Zeleznikow

  • Affiliations:

  • La Trobe University Bundoora, Vic., Australia;La Trobe University Bundoora, Vic., Australia;University of Edinburgh, Edinburgh, Scotland, UK

  • Venue:
  • ICAIL '03 Proceedings of the 9th international conference on Artificial intelligence and law
  • Year:
  • 2003

Quantified Score

Hi-index 0.00

Visualization

Abstract

In an ideal world, a legal knowledge-based system would be evaluated by an evaluator with expertise in both the legal domain and software engineering evaluation processes. However in the real world, this task is typically undertaken by legal professionals, who may lack software engineering expertise. Where software engineers do have this responsibility, they may lack legal domain knowledge. We extend the ISO 14598 evaluation process with a novel evaluation framework which satisfies three important requirements: elements of existing software engineering evaluation methodologies are integrated and subsumed, making them more readily accessible to the evaluator; requirements specific to the legal domain are included; and the intended users do not necessarily require extensive software engineering expertise. The framework emphasises the importance of the evaluation context and goals and integrates these with system properties and contingency-guidelines to suggest appropriate evaluation criteria. The evaluation process supports the selection of criteria by manual, semi-automated or automated methods and a design of an architecture to support this choice of appropriate criteria, is presented. Two evaluations are discussed that were conducted using the process and its associated framework. With ongoing research and development in the field of Artificial intelligence and Law, the need for an easily accessible and specialized evaluation methodology is apparent. Such a method would assist legal professionals frame an evaluation of legal knowledge-based systems and help software engineers understand the evaluation requirements of legal professionals.