Crowd-based recognition of web interaction patterns

  • Authors:

  • Walter Lasecki;Tessa Lau;Grant He;Jeffrey Bigham

  • Affiliations:

  • University of Rochester, Rochester, New York, USA;IBM Research - Almaden, San Jose, California, USA;University of Rochester, Rochester, New York, USA;University of Rochester, Rochester, New York, USA

  • Venue:
  • Adjunct proceedings of the 25th annual ACM symposium on User interface software and technology
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Web automation often involves users describing complex tasks to a system, with directives generally limited to low-level constituent actions like "click the search button." This level of description is unnatural and makes it difficult to generalize the task across websites. In this paper, we propose a system for automatically recognizing higher-level interaction patterns from user's completion of tasks, such as "searching for cat videos" or "replying to a post". We present PatFinder, a system that identifies these patterns using the input of crowd workers. We validate the system by generating data for 10 tasks, having 62 crowd workers label them, and automatically extracting 14 interaction patterns. Our results show that the number of patterns grows sublinearly with the number of tasks, suggesting that a small finite set of patterns may suffice to describe the vast majority of tasks on the web.