Scalable similarity-based neighborhood methods with MapReduce

  • Authors:
  • Sebastian Schelter;Christoph Boden;Volker Markl

  • Affiliations:
  • Technische Universität Berlin, Berlin, Germany;Technische Universität Berlin, Berlin, Germany;Technische Universität Berlin, Berlin, Germany

  • Venue:
  • Proceedings of the sixth ACM conference on Recommender systems
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Similarity-based neighborhood methods, a simple and popular approach to collaborative filtering, infer their predictions by finding users with similar taste or items that have been similarly rated. If the number of users grows to millions, the standard approach of sequentially examining each item and looking at all interacting users does not scale. To solve this problem, we develop a MapReduce algorithm for the pairwise item comparison and top-N recommendation problem that scales linearly with respect to a growing number of users. This parallel algorithm is able to work on partitioned data and is general in that it supports a wide range of similarity measures. We evaluate our algorithm on a large dataset consisting of 700 million song ratings from Yahoo! Music.