Network-Based inference algorithm on hadoop

Authors:
Zhen Tang;Qingxian Wang;Shimin Cai
Affiliations:
Web Sciences Center, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, P.R. China;Web Sciences Center, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, P.R. China;Web Sciences Center, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, P.R. China
Venue:
ISMIS'12 Proceedings of the 20th international conference on Foundations of Intelligent Systems
Year:
2012

Citing 11
Cited 0

Amazon.com Recommendations: Item-to-Item Collaborative Filtering

IEEE Internet Computing
Optimizing data aggregation for cluster-based internet services

Proceedings of the ninth ACM SIGPLAN symposium on Principles and practice of parallel programming
The Google file system

SOSP '03 Proceedings of the nineteenth ACM symposium on Operating systems principles
Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions

IEEE Transactions on Knowledge and Data Engineering
Dryad: distributed data-parallel programs from sequential building blocks

Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007
Dynamo: amazon's highly available key-value store

Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles
Evaluating MapReduce for Multi-core and Multiprocessor Systems

HPCA '07 Proceedings of the 2007 IEEE 13th International Symposium on High Performance Computer Architecture
MapReduce: simplified data processing on large clusters

Communications of the ACM - 50th anniversary issue: 1958 - 2008
Cloud Computing: Distributed Internet Computing for IT and Scientific Research

IEEE Internet Computing
A survey of collaborative filtering techniques

Advances in Artificial Intelligence
User-Based Collaborative-Filtering Recommendation Algorithms on Hadoop

WKDD '10 Proceedings of the 2010 Third International Conference on Knowledge Discovery and Data Mining

Quantified Score

Hi-index	0.00

Visualization

Abstract

Network-based inference (NBI) algorithm is a new but effective personalized recommendation algorithm based on bipartite networks, and it performs better than global ranking method (GRM) and collaborative filtering (CF).However, the complexity of NBI is high thus hinder NBI's use in large scale system. In this paper, we implement NBI algorithm on a cloud computing platform, namely Hadoop, to solve its scalability problem. We use MapReduce model to distribute the NBI algorithm into serial parallel MapReduce jobs, and implement them in parallel on Hadoop platform. Through performing extensive experiments on the data sets of Netflix, the result shows that the NBI algorithm can scale well and process large datasets on commodity hardware effectively.