EWCBR '98 Proceedings of the 4th European Workshop on Advances in Case-Based Reasoning
Refining Conversational Case Libraries
ICCBR '97 Proceedings of the Second International Conference on Case-Based Reasoning Research and Development
Similarity of personal preferences: theoretical foundations and empirical analysis
Artificial Intelligence
Rank quality for evaluating ccbr system performance
Rank quality for evaluating ccbr system performance
What evaluation criteria are right for CCBR? considering rank quality
ECCBR'06 Proceedings of the 8th European conference on Advances in Case-Based Reasoning
| Hi-index | 0.00 |
In previous work, Bogaerts and Leake [1,2] introduced the rank qualitymeasure for the evaluation of conversational case-based reasoning (CCBR) systems. Rank quality assesses how well a system copes with the limited problem information available in an ongoing dialog, giving useful evaluation information not readily available from standard precision and efficiency measures. However, that work also revealed surprising challenges for developing rank quality measures, restricting the proposed measures' applicability. This paper explores two open questions from that work: 1) how to define a rank quality measure immune to the previous pitfalls, and 2) how to assess the meaningfulness of any proposed rank quality measure. The paper establishes formal requirements for a rank quality measure, presents a new formulation of the measure, and provides a formal proof and empirical evidence to support that the new measure avoids previous pitfalls and meets the formal requirements.