Using real-time feedback to improve visual question answering

  • Authors:

  • Yu Zhong;Phyo Thiha;Grant He;Walter Lasecki;Jeffrey Bigham

  • Affiliations:

  • University of Rochester, Rochester, New York, USA;University of Rochester, Rochester, New York, USA;University of Rochester, Rochester, New York, USA;University of Rochester, Rochester, New York, USA;University of Rochester, Rochester, New York, USA

  • Venue:
  • CHI '12 Extended Abstracts on Human Factors in Computing Systems
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Technology holds great promise for improving the everyday lives of people with disabilities; however, automated systems are prone to errors and cannot handle many real-world tasks. VizWiz, a system for answering visual questions for blind users, has shown that crowdsourcing can be used for assistive technology in such domains. Our work extends the VizWiz model by enabling users to interact with the crowd via a real-time feedback loop. We introduce Legion:View, a system that enables such a real-time feedback loop for visual questions between users and crowd workers. Legion:View sends audio questions and streaming video to the crowd, and forwards feedback about the position and orientation of the camera and answers to questions back to users.