Regular Articles

Performance measure of image and video quality assessment algorithms: subjective root-mean-square error

[+] Author Affiliations
Mikko Nuutinen, Toni Virtanen, Jukka Häkkinen

University of Helsinki, Institute of Behavioural Sciences, PO Box 9, 00014, Helsinki, Finland

J. Electron. Imaging. 25(2), 023012 (Mar 28, 2016). doi:10.1117/1.JEI.25.2.023012
History: Received April 14, 2015; Accepted March 3, 2016
Text Size: A A A

Abstract.  Evaluating algorithms used to assess image and video quality requires performance measures. Traditional performance measures (e.g., Pearson’s linear correlation coefficient, Spearman’s rank-order correlation coefficient, and root mean square error) compare quality predictions of algorithms to subjective mean opinion scores (mean opinion score/differential mean opinion score). We propose a subjective root-mean-square error (SRMSE) performance measure for evaluating the accuracy of algorithms used to assess image and video quality. The SRMSE performance measure takes into account dispersion between observers. The other important property of the SRMSE performance measure is its measurement scale, which is calibrated to units of the number of average observers. The results of the SRMSE performance measure indicate the extent to which the algorithm can replace the subjective experiment (as the number of observers). Furthermore, we have presented the concept of target values, which define the performance level of the ideal algorithm. We have calculated the target values for all sample sets of the CID2013, CVD2014, and LIVE multiply distorted image quality databases.The target values and MATLAB implementation of the SRMSE performance measure are available on the project page of this study.

Figures in this Article
© 2016 SPIE and IS&T

Citation

Mikko Nuutinen ; Toni Virtanen and Jukka Häkkinen
"Performance measure of image and video quality assessment algorithms: subjective root-mean-square error", J. Electron. Imaging. 25(2), 023012 (Mar 28, 2016). ; http://dx.doi.org/10.1117/1.JEI.25.2.023012


Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

PubMed Articles
Advertisement
  • Don't have an account?
  • Subscribe to the SPIE Digital Library
  • Create a FREE account to sign up for Digital Library content alerts and gain access to institutional subscriptions remotely.
Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).
Access This Proceeding
Sign in or Create a personal account to Buy this article ($15 for members, $18 for non-members).
Access This Chapter

Access to SPIE eBooks is limited to subscribing institutions and is not available as part of a personal subscription. Print or electronic versions of individual SPIE books may be purchased via SPIE.org.