Online subjective testing for consumer-photo quality evaluation

Michele A. Saad; Patrick McKnight; Jacob Quartuccio; David Nicholas; Ramesh Jaladi; Philip Corriveau

doi:10.1117/1.JEI.25.4.043009

15 July 2016 Online subjective testing for consumer-photo quality evaluation

Michele A. Saad, Patrick McKnight, Jacob Quartuccio, David Nicholas, Ramesh Jaladi, Philip Corriveau

Author Affiliations +

Journal of Electronic Imaging, Vol. 25, Issue 4, 043009 (July 2016). https://doi.org/10.1117/1.JEI.25.4.043009

Abstract

We take a look at crowdsourcing for subjective image quality evaluation using real image stimuli with nonsimulated distortions. Our aim is to scale the task of subjectively rating images while ensuring maximal data validity and accuracy. While previous work has begun to explore crowdsourcing for quality assessment, it has either used images that are not representative of popular consumer scenarios or used crowdsourcing to collect data without comparison to experiments in a controlled environment. Here, we address the challenges imposed by the highly variable online environment, using stimuli that are subtle and more complex than has traditionally been used in quality assessment experiments. In a series of experiments, we vary different design parameters and demonstrate how they impact the subjective responses obtained. Of the parameters examined are stimulus display mode, study length, stimulus habituation, and content homogeneity/heterogeneity. Our method was tested on a database that was rated in a laboratory test previously. Once our design parameters were chosen, we rated a database of consumer photographs and are making this data available to the research community.

Citation Download Citation

Michele A. Saad, Patrick McKnight, Jacob Quartuccio, David Nicholas, Ramesh Jaladi, and Philip Corriveau "Online subjective testing for consumer-photo quality evaluation," Journal of Electronic Imaging 25(4), 043009 (15 July 2016). https://doi.org/10.1117/1.JEI.25.4.043009

Published: 15 July 2016

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available