Evaluation Measures for TCBR Systems

Authors:
M. A. Raghunandan;Nirmalie Wiratunga;Sutanu Chakraborti;Stewart Massie;Deepak Khemani
Affiliations:
Department of Computer Science and Engineering, Indian Institute of Technology, Madras, India;School of Computing, The Robert Gordon University, Scotland, UK;Tata Research Development and Design Centre, Pune, India;School of Computing, The Robert Gordon University, Scotland, UK;Department of Computer Science and Engineering, Indian Institute of Technology, Madras, India
Venue:
ECCBR '08 Proceedings of the 9th European conference on Advances in Case-Based Reasoning
Year:
2008

Citing 7
Cited 6

Text Categorization with Suport Vector Machines: Learning with Many Relevant Features

ECML '98 Proceedings of the 10th European Conference on Machine Learning
Learning to Adapt for Case-Based Design

ECCBR '02 Proceedings of the 6th European Conference on Advances in Case-Based Reasoning
Introduction

Case-Based Reasoning Technology, From Foundations to Applications
Random Graphs for Statistical Pattern Recognition

Random Graphs for Statistical Pattern Recognition
From Anomaly Reports to Cases

ICCBR '07 Proceedings of the 7th international conference on Case-Based Reasoning: Case-Based Reasoning Research and Development
Complexity profiling for informed case-base editing

ECCBR'06 Proceedings of the 8th European conference on Advances in Case-Based Reasoning
Measuring the complexity of a collection of documents

ECIR'06 Proceedings of the 28th European conference on Advances in Information Retrieval

Case Retrieval Reuse Net (CR2N): An Architecture for Reuse of Textual Solutions

ICCBR '09 Proceedings of the 8th International Conference on Case-Based Reasoning: Case-Based Reasoning Research and Development
Robust Measures of Complexity in TCBR

ICCBR '09 Proceedings of the 8th International Conference on Case-Based Reasoning: Case-Based Reasoning Research and Development
Distributed representations to detect higher order term correlations in textual content

RSCTC'10 Proceedings of the 7th international conference on Rough sets and current trends in computing
Recognition of higher-order relations among features in textual cases using random indexing

ICCBR'10 Proceedings of the 18th international conference on Case-Based Reasoning Research and Development
Reexamination of CBR hypothesis

ICCBR'10 Proceedings of the 18th international conference on Case-Based Reasoning Research and Development
Query suggestions for textual problem solution repositories

ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval

Quantified Score

Hi-index	0.00

Visualization

Abstract

Textual-case based reasoning (TCBR) systems where the problem and solution are in free text form are hard to evaluate. In the absence of class information, domain experts are needed to evaluate solution quality, and provide relevance information. This approach is costly and time consuming. We propose three measures that can be used to compare alternate TCBR system configurations, in the absence of class information. The main idea is to quantify alignment as the degree to which similar problems have similar solutions. Two local measures capture this information by analysing similarity between problem and solution neighbourhoods at different levels of granularity, whilst a global measure achieves the same by analyzing similarity between problem and solution clusters. We determine the suitability of the proposed measures by studying their correlation with classifier accuracy on a health and safety incident reporting task. Strong correlation is observed with all three approaches with local measures being slightly superior over the global one.