iSpy: automatic reconstruction of typed input from compromising reflections

  • Authors:

  • Rahul Raguram;Andrew M. White;Dibyendusekhar Goswami;Fabian Monrose;Jan-Michael Frahm

  • Affiliations:

  • University of North Carolina at Chapel Hill, Chapel Hill, NC, USA;University of North Carolina at Chapel Hill, Chapel Hill, NC, USA;University of North Carolina at Chapel Hill, Chapel Hill, NC, USA;University of North Carolina at Chapel Hill, Chapel Hill, NC, USA;University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

  • Venue:
  • Proceedings of the 18th ACM conference on Computer and communications security
  • Year:
  • 2011

Quantified Score

Hi-index 0.02

Visualization

Abstract

We investigate the implications of the ubiquity of personal mobile devices and reveal new techniques for compromising the privacy of users typing on virtual keyboards. Specifi- cally, we show that so-called compromising reflections (in, for example, a victim's sunglasses) of a device's screen are sufficient to enable automated reconstruction, from video, of text typed on a virtual keyboard. Despite our deliberate use of low cost commodity video cameras, we are able to compensate for variables such as arbitrary camera and device positioning and motion through the application of advanced computer vision and machine learning techniques. Using footage captured in realistic environments (e.g., on a bus), we show that we are able to reconstruct fluent translations of recorded data in almost all of the test cases, correcting users' typing mistakes at the same time. We believe these results highlight the importance of adjusting privacy expectations in response to emerging technologies.