An evaluation of animated demonstrations of learning computer-based tasks
CHI '91 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Apple guide: a case study in user-aided design of online help
CHI '95 Conference Companion on Human Factors in Computing Systems
CHI '95 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Stencils-based tutorials: design and evaluation
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Multimedia Tools and Applications
DocWizards: a system for authoring follow-me documentation wizards
Proceedings of the 18th annual ACM symposium on User interface software and technology
Graphstract: minimal graphical help for computers
Proceedings of the 20th annual ACM symposium on User interface software and technology
Video browsing by direct manipulation
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
An application-independent system for visualizing user operation history
Proceedings of the 21st annual ACM symposium on User interface software and technology
SmartPlayer: user-centric video fast-forwarding
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Generating photo manipulation tutorials by demonstration
ACM SIGGRAPH 2009 papers
Sikuli: using GUI screenshots for search and automation
Proceedings of the 22nd annual ACM symposium on User interface software and technology
Automatically identifying targets users interact with during real world tasks
Proceedings of the 15th international conference on Intelligent user interfaces
ToolClips: an investigation of contextual video assistance for functionality understanding
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
GUI testing using computer vision
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Content-aware dynamic timeline for video browsing
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Chronicle: capture, exploration, and playback of document workflow histories
UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Content and hierarchy in pixel-based methods for reverse engineering interface structure
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
ShowMeHow: translating user interface instructions between applications
Proceedings of the 24th annual ACM symposium on User interface software and technology
Pause-and-play: automatically linking screencast video tutorials with applications
Proceedings of the 24th annual ACM symposium on User interface software and technology
Creating contextual help for GUIs using screenshots
Proceedings of the 24th annual ACM symposium on User interface software and technology
Associating the visual representation of user interfaces with their internal structures and metadata
Proceedings of the 24th annual ACM symposium on User interface software and technology
Sketch-sketch revolution: an engaging tutorial system for guided sketching and application learning
Proceedings of the 24th annual ACM symposium on User interface software and technology
Discovery-based games for learning software
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Pixel-based reverse engineering of graphical interfaces
Proceedings of the adjunct publication of the 26th annual ACM symposium on User interface software and technology
| Hi-index | 0.00 |
We present Waken, an application-independent system that recognizes UI components and activities from screen captured videos, without any prior knowledge of that application. Waken can identify the cursors, icons, menus, and tooltips that an application contains, and when those items are used. Waken uses frame differencing to identify occurrences of behaviors that are common across graphical user interfaces. Candidate templates are built, and then other occurrences of those templates are identified using a multi-phase algorithm. An evaluation demonstrates that the system can successfully reconstruct many aspects of a UI without any prior application-dependant knowledge. To showcase the design opportunities that are introduced by having this additional meta-data, we present the Waken Video Player, which allows users to directly interact with UI components that are displayed in the video.