Sikuli: using GUI screenshots for search and automation

Authors:
Tom Yeh;Tsung-Hsiang Chang;Robert C. Miller
Affiliations:
MIT CSAIL, Cambridge, MA, USA;MIT CSAIL, Cambridge, MA, USA;MIT CSAIL, Cambridge, MA, USA
Venue:
Proceedings of the 22nd annual ACM symposium on User interface software and technology
Year:
2009

Citing 19
Cited 39

Coupling a UI framework with automatic generation of context-sensitive animated help

UIST '90 Proceedings of the 3rd annual ACM SIGGRAPH symposium on User interface software and technology
Automatic generation of help from interface design models

CHI '94 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Automatic generation of task-oriented help

Proceedings of the 8th annual ACM symposium on User interface and software technology
Apple guide: a case study in user-aided design of online help

CHI '95 Conference Companion on Human Factors in Computing Systems
A comparison of still, animated, or nonillustrated on-line help with written or spoken instructions in a graphical user interface

CHI '95 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A visual medium for programmatic control of interactive applications

Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Programming by example: visual generalization in programming by example

Communications of the ACM
Computer Vision: A Modern Approach

Computer Vision: A Modern Approach
Object Recognition from Local Scale-Invariant Features

ICCV '99 Proceedings of the International Conference on Computer Vision-Volume 2 - Volume 2
Pixel data access: interprocess communication in the user interface for end-user programming and graphical macros

Pixel data access: interprocess communication in the user interface for end-user programming and graphical macros
Video Google: A Text Retrieval Approach to Object Matching in Videos

ICCV '03 Proceedings of the Ninth IEEE International Conference on Computer Vision - Volume 2
Stencils-based tutorials: design and evaluation

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Automation and customization of rendered web pages

Proceedings of the 18th annual ACM symposium on User interface software and technology
DocWizards: a system for authoring follow-me documentation wizards

Proceedings of the 18th annual ACM symposium on User interface software and technology
An evaluation of using programming by demonstration and guided walkthrough techniques for authoring and utilizing documentation

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Multiple Object Class Detection with a Generative Model

CVPR '06 Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1
Assieme: finding and leveraging implicit references in a web search interface for programmers

Proceedings of the 20th annual ACM symposium on User interface software and technology
Graphstract: minimal graphical help for computers

Proceedings of the 20th annual ACM symposium on User interface software and technology
CoScripter: automating & sharing how-to knowledge in the enterprise

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Automatically identifying targets users interact with during real world tasks

Proceedings of the 15th international conference on Intelligent user interfaces
Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
GUI testing using computer vision

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
TapShot: screenshot snippets as GUI shortcuts

ACM SIGGRAPH 2010 Posters
Chronicle: capture, exploration, and playback of document workflow histories

UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Generating client workloads and high-fidelity network traffic for controllable, repeatable experiments in computer security

RAID'10 Proceedings of the 13th international conference on Recent advances in intrusion detection
A case for query by image and text content: searching computer help using screenshots and keywords

Proceedings of the 20th international conference on World wide web
Content and hierarchy in pixel-based methods for reverse engineering interface structure

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Ambient help

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Using graphical representation of user interfaces as visual references

Proceedings of the 24th annual ACM symposium adjunct on User interface software and technology
Real-time crowd control of existing interfaces

Proceedings of the 24th annual ACM symposium on User interface software and technology
Pause-and-play: automatically linking screencast video tutorials with applications

Proceedings of the 24th annual ACM symposium on User interface software and technology
Creating contextual help for GUIs using screenshots

Proceedings of the 24th annual ACM symposium on User interface software and technology
d.tour: style-based exploration of design example galleries

Proceedings of the 24th annual ACM symposium on User interface software and technology
IP-QAT: in-product questions, answers, & tips

Proceedings of the 24th annual ACM symposium on User interface software and technology
Cracking the cocoa nut: user interface programming at runtime

Proceedings of the 24th annual ACM symposium on User interface software and technology
Associating the visual representation of user interfaces with their internal structures and metadata

Proceedings of the 24th annual ACM symposium on User interface software and technology
JPF-AWT: Model checking GUI applications

ASE '11 Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering
A general-purpose target-aware pointing enhancement using pixel-level analysis of graphical interfaces

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
ScreenMatch: providing context to software translators by displaying screenshots

CHI '12 Extended Abstracts on Human Factors in Computing Systems
Waken: reverse engineering usage information and interface structure from software videos

Proceedings of the 25th annual ACM symposium on User interface software and technology
MixT: automatic generation of step-by-step mixed media tutorials

Proceedings of the 25th annual ACM symposium on User interface software and technology
Tutorial-based interfaces for cloud-enabled applications

Proceedings of the 25th annual ACM symposium on User interface software and technology
Midas: fabricating custom capacitive touch sensors to prototype interactive objects

Proceedings of the 25th annual ACM symposium on User interface software and technology
Enjoying text input with image-enabled IME

ACE'12 Proceedings of the 9th international conference on Advances in Computer Entertainment
Automatic testing of real-time graphics systems

TACAS'13 Proceedings of the 19th international conference on Tools and Algorithms for the Construction and Analysis of Systems
Picode: inline photos representing posture data in source code

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Multiverse: crowd algorithms on existing interfaces

CHI '13 Extended Abstracts on Human Factors in Computing Systems
Software test automation using DEVSimPy environment

Proceedings of the 2013 ACM SIGSIM conference on Principles of advanced discrete simulation
Exploring the internal state of user interfaces by combining computer vision techniques with grammatical inference

Proceedings of the 2013 International Conference on Software Engineering
On extracting unit tests from interactive live programming sessions

Proceedings of the 2013 International Conference on Software Engineering
HydraScope: creating multi-surface meta-applications through view synchronization and input multiplexing

Proceedings of the 2nd ACM International Symposium on Pervasive Displays
SeeSS: seeing what i broke -- visualizing change impact of cascading style sheets (css)

Proceedings of the 26th annual ACM symposium on User interface software and technology
Interactive record/replay for web application debugging

Proceedings of the 26th annual ACM symposium on User interface software and technology
Privacy-preserving screen capture: Towards closing the loop for health IT usability

Journal of Biomedical Informatics
Pixel-based reverse engineering of graphical interfaces

Proceedings of the adjunct publication of the 26th annual ACM symposium on User interface software and technology
Integrated visual representations for programming with real-world input and output

Proceedings of the adjunct publication of the 26th annual ACM symposium on User interface software and technology
Undo/redo by trajectory

HCI'13 Proceedings of the 15th international conference on Human-Computer Interaction: interaction modalities and techniques - Volume Part IV
WidgetLens: a system for adaptive content magnification of widgets

BCS-HCI '13 Proceedings of the 27th International BCS Human Computer Interaction Conference

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present Sikuli, a visual approach to search and automation of graphical user interfaces using screenshots. Sikuli allows users to take a screenshot of a GUI element (such as a toolbar button, icon, or dialog box) and query a help system using the screenshot instead of the element's name. Sikuli also provides a visual scripting API for automating GUI interactions, using screenshot patterns to direct mouse and keyboard events. We report a web-based user study showing that searching by screenshot is easy to learn and faster to specify than keywords. We also demonstrate several automation tasks suitable for visual scripting, such as map navigation and bus tracking, and show how visual scripting can improve interactive help systems previously proposed in the literature.