Man versus machine: evaluating IVR versus a live operator for phone surveys in India

  • Authors:
  • Dipanjan Chakraborty;Indrani Medhi;Edward Cutrell;William Thies

  • Affiliations:
  • IIT Delhi;Microsoft Research India;Microsoft Research India;Microsoft Research India

  • Venue:
  • Proceedings of the 3rd ACM Symposium on Computing for Development
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

Many organizations in the developing world need to conduct phone surveys to collect data from low-income respondents. Such organizations generally have two options: employ a live operator, or utilize interactive voice response (IVR). Despite the relevance of this question, we are unaware of any work that rigorously compares the accuracy, speed, and cost of an IVR survey relative to a live operator. In this paper, we address these questions by giving two identical interviews -- one using IVR, and one using a live operator -- to 31 low-income job seekers in India. The IVR interview included a brief introduction by a live operator, to provide context for the call. Out of the 20 people who completed both surveys, we found that IVR incurs a 4.0% error rate (95% C. I. 2.5% -- 6.1%) and requires 2.5 times longer for users. We summarize our experience as a set of recommendations for practitioners in this space.