The use of psychophysical data and models in the analysis of display system performance
Digital images and human vision
An efficient boosting algorithm for combining preferences
The Journal of Machine Learning Research
STOC '04 Proceedings of the thirty-sixth annual ACM symposium on Theory of computing
Discrete & Computational Geometry
Crowdsourcing user studies with Mechanical Turk
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Crowdsourcing for relevance evaluation
ACM SIGIR Forum
A crowdsourceable QoE evaluation framework for multimedia content
MM '09 Proceedings of the 17th ACM international conference on Multimedia
Proceedings of the international conference on Multimedia information retrieval
Approaching Optimality for Solving SDD Linear Systems
FOCS '10 Proceedings of the 2010 IEEE 51st Annual Symposium on Foundations of Computer Science
Statistical ranking and combinatorial Hodge theory
Mathematical Programming: Series A and B - Special Issue on "Optimization and Machine learning"; Alexandre d’Aspremont • Francis Bach • Inderjit S. Dhillon • Bin Yu
Online crowdsourcing subjective image quality assessment
Proceedings of the 20th ACM international conference on Multimedia
Robust evaluation for quality of experience in crowdsourcing
Proceedings of the 21st ACM international conference on Multimedia
| Hi-index | 0.00 |
Subjective visual quality evaluation provides the groundtruth and source of inspiration in building objective visual quality metrics. Paired comparison is expected to yield more reliable results; however, this is an expensive and timeconsuming process. In this paper, we propose a novel framework of HodgeRank on Random Graphs (HRRG) to achieve efficient and reliable subjective Video Quality Assessment (VQA). To address the challenge of a potentially large number of combinations of videos to be assessed, the proposed methodology does not require the participants to perform the complete comparison of all the paired videos. Instead, participants only need to perform a random sample of all possible paired comparisons, which saves a great amount of time and labor. In contrast to the traditional deterministic incomplete block designs, our random design is not only suitable for traditional laboratory and focus-group studies, but also fit for crowdsourcing experiments on Internet where the raters are distributive over Internet and it is hard to control with precise experimental designs. Our contribution in this work is three-fold: 1) a HRRG framework is proposed to quantify the quality of video; 2) a new random design principle is investigated to conduct paired comparison based on Erdos-Renyi random graph theory; 3) Hodge decomposition is introduced to derive, from incomplete and imbalanced data, quality scores of videos and inconsistency of participants'judgments. We demonstrate the effectiveness of the proposed framework on LIVE Database. Equipped with random graph theory and HodgeRank, our scheme has the following advantages over the traditional ones: 1) data collection is simple and easy to handle, and thus is more suitable for crowdsourcing on Internet; 2) workload on participants is lower and more flexible; 3) the rating procedure is efficient, labor-saving, and more importantly, without jeopardizing the accuracy of the results.