A framework for the competitive evaluation of model inference techniques

Authors:
Neil Walkinshaw;Kirill Bogdanov;Christophe Damas;Bernard Lambeau;Pierre Dupont
Affiliations:
The University of Sheffield, Sheffield, UK;The University of Sheffield, Sheffield, UK;Université Catholique de Louvain (UCL), Louvain-la-Neuve, Belgium;Université Catholique de Louvain (UCL), Louvain-la-Neuve, Belgium;Université Catholique de Louvain (UCL), Louvain-la-Neuve, Belgium
Venue:
Proceedings of the First International Workshop on Model Inference In Testing
Year:
2010

Citing 23
Cited 4

Learning regular sets from queries and counterexamples

Information and Computation
Discovering models of software processes from event-based data

ACM Transactions on Software Engineering and Methodology (TOSEM)
Authoritative sources in a hyperlinked environment

Journal of the ACM (JACM)
Encoding program executions

ICSE '01 Proceedings of the 23rd International Conference on Software Engineering
Mining specifications

POPL '02 Proceedings of the 29th ACM SIGPLAN-SIGACT symposium on Principles of programming languages
Results of the Abbadingo One DFA Learning Competition and a New Evidence-Driven State Merging Algorithm

ICGI '98 Proceedings of the 4th International Colloquium on Grammatical Inference
What Is the Search Space of the Regular Inference?

ICGI '94 Proceedings of the Second International Colloquium on Grammatical Inference and Applications
Generating Annotated Behavior Models from End-User Scenarios

IEEE Transactions on Software Engineering
The state of CASC

AI Communications
SMArTIC: towards building an accurate, robust and scalable specification miner

Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering
Graph evolution: Densification and shrinking diameters

ACM Transactions on Knowledge Discovery from Data (TKDD)
Progressing the state-of-the-art in grammatical inference by competition: The Omphalos Context-Free Language Learning Competition

AI Communications
Constructing Programs from Example Computations

IEEE Transactions on Software Engineering
Reverse Engineering State Machines by Interactive Grammar Inference

WCRE '07 Proceedings of the 14th Working Conference on Reverse Engineering
Automatic generation of software behavioral models

Proceedings of the 30th international conference on Software engineering
Evaluation and Comparison of Inferred Regular Grammars

ICGI '08 Proceedings of the 9th international colloquium on Grammatical Inference: Algorithms and Applications
Problems with Evaluation of Unsupervised Empirical Grammatical Inference Systems

ICGI '08 Proceedings of the 9th international colloquium on Grammatical Inference: Algorithms and Applications
THE QSM ALGORITHM AND ITS APPLICATION TO SOFTWARE BEHAVIOR MODEL INDUCTION

Applied Artificial Intelligence
Introduction to Automata Theory, Languages, and Computation

Introduction to Automata Theory, Languages, and Computation
A systematic analysis of performance measures for classification tasks

Information Processing and Management: an International Journal
On the Synthesis of Finite-State Machines from Samples of Their Behavior

IEEE Transactions on Computers
Computing the Structural Difference between State-Based Models

WCRE '09 Proceedings of the 2009 16th Working Conference on Reverse Engineering
The tenjinno machine translation competition

ICGI'06 Proceedings of the 8th international conference on Grammatical Inference: algorithms and applications

Extracting QuickCheck specifications from EUnit test cases

Proceedings of the 10th ACM SIGPLAN workshop on Erlang
Assessing test adequacy for black-box systems without specifications

ICTSS'11 Proceedings of the 23rd IFIP WG 6.1 international conference on Testing software and systems
Automated Comparison of State-Based Software Models in Terms of Their Language and Structure

ACM Transactions on Software Engineering and Methodology (TOSEM)
Software model synthesis using satisfiability solvers

Empirical Software Engineering

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper describes the STAMINA competition1, which is designed to drive the evaluation and improvement of software model-inference approaches. To this end, the target models have certain characteristics that tend to appear in software-models; they have large alphabets, and states are not evenly connected by transitions (as has been the case in previous similar competitions). The paper describes the set-up of the competition that extends previous similar competitions in the field of regular grammar inference. However, this competition focusses on target models that are characteristic of software systems, and features a suitably adapted protocol for the generation of training and testing samples. Besides providing details of the competition itself, it also discusses how outcomes from the competition will be used to gain broader insights into the relative accuracy and efficiency of competing techniques.