Cognitive walkthroughs: a method for theory-based evaluation of user interfaces
International Journal of Man-Machine Studies
The GOMS family of user interface analysis techniques: comparison and contrast
ACM Transactions on Computer-Human Interaction (TOCHI)
Inductive Inference: Theory and Methods
ACM Computing Surveys (CSUR)
POPL '02 Proceedings of the 29th ACM SIGPLAN-SIGACT symposium on Principles of programming languages
Automatic generation of program specifications
ISSTA '02 Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis
Automated assumption generation for compositional verification
Formal Methods in System Design
Sikuli: using GUI screenshots for search and automation
Proceedings of the 22nd annual ACM symposium on User interface software and technology
Model-Based Testing of GUI-Driven Applications
SEUS '09 Proceedings of the 7th IFIP WG 10.2 International Workshop on Software Technologies for Embedded and Ubiquitous Systems
GUI testing using computer vision
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
User interface model discovery: towards a generic approach
Proceedings of the 2nd ACM SIGCHI symposium on Engineering interactive computing systems
Grammatical Inference: Learning Automata and Grammars
Grammatical Inference: Learning Automata and Grammars
Exact DFA identification using SAT solvers
ICGI'10 Proceedings of the 10th international colloquium conference on Grammatical inference: theoretical results and applications
Spreadsheet data manipulation using examples
Communications of the ACM
A Systematic Approach to Model Checking Human–Automation Interaction Using Task Analytic Models
IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans
| Hi-index | 0.00 |
In this paper, we present a promising approach to systematically testing graphical user interfaces (GUI) in a platform independent manner. Our framework uses standard computer vision techniques through a python-based scripting language (Sikuli script) to identify key graphical elements in the screen and automatically interact with these elements by simulating keypresses and pointer clicks. The sequence of inputs and outputs resulting from the interaction is analyzed using grammatical inference techniques that can infer the likely internal states and transitions of the GUI based on the observations. Our framework handles a wide variety of user interfaces ranging from traditional pull down menus to interfaces built for mobile platforms such as Android and iOS. Furthermore, the automaton inferred by our approach can be used to check for potentially harmful patterns in the interface's internal state machine such as design inconsistencies (eg,. a keypress does not have the intended effect) and mode confusion that can make the interface hard to use. We describe an implementation of the framework and demonstrate its working on a variety of interfaces including the user-interface of a safety critical insulin infusion pump that is commonly used by type-1 diabetic patients.