A visual medium for programmatic control of interactive applications
Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Programming by example: visual generalization in programming by example
Communications of the ACM
Pixel data access: interprocess communication in the user interface for end-user programming and graphical macros
Sikuli: using GUI screenshots for search and automation
Proceedings of the 22nd annual ACM symposium on User interface software and technology
Automatically identifying targets users interact with during real world tasks
Proceedings of the 15th international conference on Intelligent user interfaces
Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
GUI testing using computer vision
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Content and hierarchy in pixel-based methods for reverse engineering interface structure
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Deep shot: a framework for migrating tasks across devices using mobile phone cameras
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Associating the visual representation of user interfaces with their internal structures and metadata
Proceedings of the 24th annual ACM symposium on User interface software and technology
Software test automation using DEVSimPy environment
Proceedings of the 2013 ACM SIGSIM conference on Principles of advanced discrete simulation
| Hi-index | 0.00 |
Many user interfaces use indirect references to identify specific objects and devices. My thesis investigates using graphical representations of user interfaces (i.e. screenshots) as direct visual references to support various kinds of applications. Sikuli Script enables users to programmatically control GUIs without the support from the underlying applications. Sikuli Test lets GUI developers and testers create test scripts without coding. Deep Shot introduces a framework and interaction techniques to migrate work states across heterogeneous devices in one action, taking a picture. In addition to these pure pixel-based systems, PAX associates the pixel representation with the internal structures and metadata of the user interface. Based on these building blocks, we propose to develop a visual history system that enables users to search and browse what they have seen on their computer screens. We outline some interesting use cases and discuss the challenges in this ongoing work.