Efficient use of local edge histogram descriptor
MULTIMEDIA '00 Proceedings of the 2000 ACM workshops on Multimedia
A crowdsourceable QoE evaluation framework for multimedia content
MM '09 Proceedings of the 17th ACM international conference on Multimedia
MPEG-7 visual motion descriptors
IEEE Transactions on Circuits and Systems for Video Technology
Online crowdsourcing subjective image quality assessment
Proceedings of the 20th ACM international conference on Multimedia
| Hi-index | 0.00 |
Subjective quality perception studies with human observers are essential for multimedia system design. Such studies are known to be expensive and difficult to administer. They require time, a detailed knowledge of experimental designs and a level of control which can often only be achieved in a laboratory setting. Hence, only very few researchers consider running subjective studies at all. In this paper we present Randomised Pair Comparison (r/pc), an easy-to-use, flexible, economic and robust extension to conventional pair comparison methods. r/pc uses random sampling to select a unique and small subset of pairs for each assessor, thus separating session duration from the experimental design. With r/pc an experimenter can freely define the duration of sessions and balance between costs and accuracy of an experiment. On a realistic example study we show that r/pc is able to create stable results with an accuracy close to full factorial designs, yet much lower costs. We also provide initial evidence that r/pc can avoid unpleasant fatigue and learning effects which are common in long experiment sessions.