A general-purpose target-aware pointing enhancement using pixel-level analysis of graphical interfaces

Authors:
Morgan Dixon;James Fogarty;Jacob Wobbrock
Affiliations:
University of Washington, Seattle, Washington, United States;University of Washington, Seattle, Washington, United States;University of Washington, Seattle, Washington, United States
Venue:
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Year:
2012

Citing 40
Cited 3

The “prince” technique: Fitts' law and selection using area cursors

CHI '95 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Making computers easier for older adults to use: area cursors and sticky icons

Proceedings of the ACM SIGCHI Conference on Human factors in computing systems
Systematic output modification in a 2D user interface toolkit

Proceedings of the 10th annual ACM symposium on User interface software and technology
IBOTS: agent control through the user interface

IUI '99 Proceedings of the 4th international conference on Intelligent user interfaces
Implementing interface attachments based on surface representations

Proceedings of the SIGCHI conference on Human Factors in Computing Systems
A visual medium for programmatic control of interactive applications

Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Manual and gaze input cascaded (MAGIC) pointing

Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Programming by example: visual generalization in programming by example

Communications of the ACM
Acquisition of expanding targets

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Fluid annotations in an open world

Proceedings of the 12th ACM conference on Hypertext and Hypermedia
Multiple haptic targets for motion-impaired computer users

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Semantic pointing: improving target acquisition with control-display ratio adaptation

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Mouse ether: accelerating the acquisition of targets across multi-monitor displays

CHI '04 Extended Abstracts on Human Factors in Computing Systems
WinCuts: manipulating arbitrary window regions for more effective use of screen space

CHI '04 Extended Abstracts on Human Factors in Computing Systems
Object pointing: a complement to bitmap pointing in GUIs

GI '04 Proceedings of the 2004 Graphics Interface Conference
ScreenCrayons: annotating anything

Proceedings of the 17th annual ACM symposium on User interface software and technology
Clip, connect, clone: combining application elements to build custom interfaces for information access

Proceedings of the 17th annual ACM symposium on User interface software and technology
The bubble cursor: enhancing target acquisition by dynamic resizing of the cursor's activation area

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Automation and customization of rendered web pages

Proceedings of the 18th annual ACM symposium on User interface software and technology
User interface façades: towards fully adaptable user interfaces

UIST '06 Proceedings of the 19th annual ACM symposium on User interface software and technology
An evaluation of sticky and force enhanced targets in multi target situations

Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles
Koala: capture, share, automate, personalize business processes on the web

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Dirty desktops: using a patina of magnetic mouse dust to make common interactor targets easier to select

Proceedings of the 20th annual ACM symposium on User interface software and technology
Programming by a sample: rapidly creating web applications with d.mix

Proceedings of the 20th annual ACM symposium on User interface software and technology
Mobilization by demonstration: using traces to re-author existing web sites

Proceedings of the 13th international conference on Intelligent user interfaces
Social accessibility: achieving accessibility through collaborative metadata authoring

Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility
PointAssist: helping four year olds point with ease

IDC '08 Proceedings of the 7th international conference on Interaction design and children
End-user programming of mashups with vegemite

Proceedings of the 14th international conference on Intelligent user interfaces
DynaSpot: speed-dependent area cursor

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
The angle mouse: target-agnostic dynamic gain adjustment based on angular deviation

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Ephemeral adaptation: the use of gradual onset to improve menu selection performance

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Sikuli: using GUI screenshots for search and automation

Proceedings of the 22nd annual ACM symposium on User interface software and technology
Automatically identifying targets users interact with during real world tasks

Proceedings of the 15th international conference on Intelligent user interfaces
Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Enhanced area cursors: reducing fine pointing demands for people with motor impairments

UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
UIMarks: quick graphical interaction with specific targets

UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology
Content and hierarchy in pixel-based methods for reverse engineering interface structure

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
From the lab to the world: lessons from extending a pointing technique for real-world use

CHI '11 Extended Abstracts on Human Factors in Computing Systems
Cracking the cocoa nut: user interface programming at runtime

Proceedings of the 24th annual ACM symposium on User interface software and technology
Associating the visual representation of user interfaces with their internal structures and metadata

Proceedings of the 24th annual ACM symposium on User interface software and technology

Waken: reverse engineering usage information and interface structure from software videos

Proceedings of the 25th annual ACM symposium on User interface software and technology
Pixel-based reverse engineering of graphical interfaces

Proceedings of the adjunct publication of the 26th annual ACM symposium on User interface software and technology
Effects of Target Expansion on Selection Performance in Older Computer Users

ACM Transactions on Accessible Computing (TACCESS)

Quantified Score

Hi-index	0.01

Visualization

Abstract

We present a general-purpose implementation of a target aware pointing technique, functional across an entire desktop and independent of application implementations. Specifically, we implement Grossman and Balakrishnan's Bubble Cursor, the fastest general pointing facilitation technique in the literature. Our implementation obtains the necessary knowledge of interface targets using a combination of pixel-level analysis and social annotation. We discuss the most novel aspects of our implementation, including methods for interactive creation and correction of pixel-level prototypes of interface elements and methods for interactive annotation of how the cursor should select identified elements. We also report on limitations of the Bubble Cursor unearthed by examining our implementation in the complexity of real-world interfaces. We therefore contribute important progress toward real-world deployment of an important family of techniques and shed light on the gap between understanding techniques in controlled settings versus behavior with real-world interfaces.