Evaluating CBR systems using different data sources: a case study

  • Authors:

  • Mingyang Gu;Agnar Aamodt

  • Affiliations:

  • Department of Computer and Information Science, Norwegian University of Science and Technology, Trondheim, Norway;Department of Computer and Information Science, Norwegian University of Science and Technology, Trondheim, Norway

  • Venue:
  • ECCBR'06 Proceedings of the 8th European conference on Advances in Case-Based Reasoning
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

The complexity and high construction cost of case bases make it very difficult, if not impossible, to evaluate a CBR system, especially a knowledge-intensive CBR system, using statistical evaluation methods on many case bases. In this paper, we propose an evaluation strategy, which uses both many simple case bases and a few complex case bases to evaluate a CBR system, and show how this strategy may satisfy different evaluation goals. The identified evaluation goals are classified into two categories: domain-independent and domain-dependent. For the evaluation goals in the first category, we apply the statistical evaluation method using many simple case bases (for example, UCI data sets); for evaluation goals in the second category, we apply different, relatively weak, evaluation methods on a few complex domain-specific case bases. We apply this combined evaluation strategy to evaluate our knowledge-intensive conversational CBR method as a case study.