Incremental attribute evaluation: a flexible algorithm for lazy update
ACM Transactions on Programming Languages and Systems (TOPLAS)
Supporting dynamic downloadable appearances in an extensible user interface toolkit
Proceedings of the 10th annual ACM symposium on User interface software and technology
IBOTS: agent control through the user interface
IUI '99 Proceedings of the 4th international conference on Intelligent user interfaces
A visual medium for programmatic control of interactive applications
Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Programming by example: visual generalization in programming by example
Communications of the ACM
Past, present, and future of user interface software tools
ACM Transactions on Computer-Human Interaction (TOCHI) - Special issue on human-computer interaction in the new millennium, Part 1
Providing visually rich resizable images for user interface components
UIST '00 Proceedings of the 13th annual ACM symposium on User interface software and technology
MORE for less: model recovery from visual interfaces for multi-device application design
Proceedings of the 8th international conference on Intelligent user interfaces
WinCuts: manipulating arbitrary window regions for more effective use of screen space
CHI '04 Extended Abstracts on Human Factors in Computing Systems
ScreenCrayons: annotating anything
Proceedings of the 17th annual ACM symposium on User interface software and technology
Proceedings of the 17th annual ACM symposium on User interface software and technology
The bubble cursor: enhancing target acquisition by dynamic resizing of the cursor's activation area
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Extensible input handling in the subArctic toolkit
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Stencils-based tutorials: design and evaluation
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Automation and customization of rendered web pages
Proceedings of the 18th annual ACM symposium on User interface software and technology
Phosphor: explaining transitions in the user interface using afterglow effects
UIST '06 Proceedings of the 19th annual ACM symposium on User interface software and technology
User interface façades: towards fully adaptable user interfaces
UIST '06 Proceedings of the 19th annual ACM symposium on User interface software and technology
Recognition of Screen-Rendered Text
ICPR '06 Proceedings of the 18th International Conference on Pattern Recognition - Volume 02
Koala: capture, share, automate, personalize business processes on the web
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Programming by a sample: rapidly creating web applications with d.mix
Proceedings of the 20th annual ACM symposium on User interface software and technology
Evaluating user interface systems research
Proceedings of the 20th annual ACM symposium on User interface software and technology
Usability evaluation considered harmful (some of the time)
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Mobilization by demonstration: using traces to re-author existing web sites
Proceedings of the 13th international conference on Intelligent user interfaces
Social accessibility: achieving accessibility through collaborative metadata authoring
Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility
End-user programming of mashups with vegemite
Proceedings of the 14th international conference on Intelligent user interfaces
Ephemeral adaptation: the use of gradual onset to improve menu selection performance
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Sikuli: using GUI screenshots for search and automation
Proceedings of the 22nd annual ACM symposium on User interface software and technology
Automatically identifying targets users interact with during real world tasks
Proceedings of the 15th international conference on Intelligent user interfaces
Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
GUI testing using computer vision
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Paper: Modeling by shortest data description
Automatica (Journal of IFAC)
Using graphical representation of user interfaces as visual references
Proceedings of the 24th annual ACM symposium adjunct on User interface software and technology
Creating contextual help for GUIs using screenshots
Proceedings of the 24th annual ACM symposium on User interface software and technology
Cracking the cocoa nut: user interface programming at runtime
Proceedings of the 24th annual ACM symposium on User interface software and technology
Associating the visual representation of user interfaces with their internal structures and metadata
Proceedings of the 24th annual ACM symposium on User interface software and technology
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
ScreenMatch: providing context to software translators by displaying screenshots
CHI '12 Extended Abstracts on Human Factors in Computing Systems
Waken: reverse engineering usage information and interface structure from software videos
Proceedings of the 25th annual ACM symposium on User interface software and technology
Patina: dynamic heatmaps for visualizing application usage
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Pixel-based reverse engineering of graphical interfaces
Proceedings of the adjunct publication of the 26th annual ACM symposium on User interface software and technology
Model-driven reverse engineering of legacy graphical user interfaces
Automated Software Engineering
| Hi-index | 0.01 |
The rigidity and fragmentation of GUI toolkits are fundamentally limiting the progress and impact of interaction research. Pixel-based methods offer unique potential for addressing these challenges independent of the implementation of any particular interface or toolkit. This work builds upon Prefab, which enables the modification of existing interfaces. We present new methods for hierarchical models of complex widgets, real-time interpretation of interface content, and real-time interpretation of content and hierarchy throughout an entire interface. We validate our new methods through implementations of four applications: stencil-based tutorials, ephemeral adaptation, interface translation, and end-user interface customization. We demonstrate these enhancements in complex existing applications created from different user interface toolkits running on different operating systems.