Proceedings of the international conference on Multimedia information retrieval
Quadrant of euphoria: a crowdsourcing platform for QoE assessment
IEEE Network: The Magazine of Global Internetworking - Special issue on improving quality of experience for network services
Collecting image annotations using Amazon's Mechanical Turk
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Evaluation of commonsense knowledge with Mechanical Turk
CSLDAMT '10 Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk
Video summarization via crowdsourcing
CHI '11 Extended Abstracts on Human Factors in Computing Systems
Platemate: crowdsourcing nutritional analysis from food photographs
Proceedings of the 24th annual ACM symposium on User interface software and technology
#EpicPlay: crowd-sourcing sports video highlights
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Using Machine Learning to Detect Cyberbullying
ICMLA '11 Proceedings of the 2011 10th International Conference on Machine Learning and Applications and Workshops - Volume 02
Pushing the limits of mechanical turk: qualifying the crowd for video geo-location
Proceedings of the ACM multimedia 2012 workshop on Crowdsourcing for multimedia
Crowdsourcing approach for evaluation of privacy filters in video surveillance
Proceedings of the ACM multimedia 2012 workshop on Crowdsourcing for multimedia
Online crowdsourcing subjective image quality assessment
Proceedings of the 20th ACM international conference on Multimedia
| Hi-index | 0.00 |
In this paper, we present a subjective video quality evaluation system that has been integrated with different crowdsourcing platforms. We try to evaluate the feasibility of replacing the time consuming and expensive traditional tests with a faster and less expensive crowdsourcing alternative. CrowdFlower and Amazon's Mechanical Turk were used as the crowdsourcing platforms to collect data. The data was compared with the formal subjective tests conducted by MPEG as part of the video standardization process, as well as with previous results from a study we ran at the university level. High quality compressed videos with known Mean Opinion Score (MOS) are used as references instead of the original lossless videos in order to overcome intrinsic bandwidth limitations. The bitrates chosen for the experiment were selected targeting Internet use, since this is the environment in which users were going to be evaluating the videos. Evaluations showed that the results are consistent with formal subjective evaluation scores, and can be reproduced across different crowds with low variability, which makes this type of test setting very promising.