Regular Articles

Online subjective testing for consumer-photo quality evaluation

[+] Author Affiliations
Michele A. Saad

Intel Corporation, AN4, 1300 South Mopac Expressway, Austin, Texas 78746, United States

Patrick McKnight, Jacob Quartuccio

George Mason University, Department of Psychology, MSN 3F5, 4400 University Drive, Fairfax, Virginia 22030-4444, United States

David Nicholas, Philip Corriveau

Intel Corporation, Jones Farm 2, 2111 Northeast 25th Avenue, Hillsboro, Oregon 97124-5961, United States

Ramesh Jaladi

Intel Corporation, Santa Clara 12, 3600 Juliette Lane, Santa Clara, California 95054, United States

J. Electron. Imaging. 25(4), 043009 (Jul 15, 2016). doi:10.1117/1.JEI.25.4.043009
History: Received March 2, 2016; Accepted June 22, 2016
Text Size: A A A

Abstract.  We take a look at crowdsourcing for subjective image quality evaluation using real image stimuli with nonsimulated distortions. Our aim is to scale the task of subjectively rating images while ensuring maximal data validity and accuracy. While previous work has begun to explore crowdsourcing for quality assessment, it has either used images that are not representative of popular consumer scenarios or used crowdsourcing to collect data without comparison to experiments in a controlled environment. Here, we address the challenges imposed by the highly variable online environment, using stimuli that are subtle and more complex than has traditionally been used in quality assessment experiments. In a series of experiments, we vary different design parameters and demonstrate how they impact the subjective responses obtained. Of the parameters examined are stimulus display mode, study length, stimulus habituation, and content homogeneity/heterogeneity. Our method was tested on a database that was rated in a laboratory test previously. Once our design parameters were chosen, we rated a database of consumer photographs and are making this data available to the research community.

© 2016 SPIE and IS&T

Citation

Michele A. Saad ; Patrick McKnight ; Jacob Quartuccio ; David Nicholas ; Ramesh Jaladi, et al.
"Online subjective testing for consumer-photo quality evaluation", J. Electron. Imaging. 25(4), 043009 (Jul 15, 2016). ; http://dx.doi.org/10.1117/1.JEI.25.4.043009


Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

PubMed Articles
Advertisement
  • Don't have an account?
  • Subscribe to the SPIE Digital Library
  • Create a FREE account to sign up for Digital Library content alerts and gain access to institutional subscriptions remotely.
Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).
Access This Proceeding
Sign in or Create a personal account to Buy this article ($15 for members, $18 for non-members).
Access This Chapter

Access to SPIE eBooks is limited to subscribing institutions and is not available as part of a personal subscription. Print or electronic versions of individual SPIE books may be purchased via SPIE.org.