Computer vision researchers have recently developed automated methods for rating the aesthetic appeal of a photograph. Machine learning techniques, applied to large databases of photos, mimic with reasonably good accuracy the mean ratings of online viewers. However, owing to the many factors underlying aesthetics, it is likely that such techniques for rating photos do not generalize well beyond the data on which they are trained. This paper reviews recent attempts to compare human ratings, obtained in a controlled setting, to ratings provided by machine learning techniques. We review methods to obtain meaningful ratings both from selected groups of judges and also from crowd sourcing. We find that state-of-the-art techniques for automatic aesthetic evaluation are only weakly correlated with human ratings. This shows the importance of obtaining data used for training automated systems under carefully controlled conditions.
In this study, our primary aim is to determine empirically the role that skill plays in determining image aesthetics, and
whether it can be deciphered from the ratings given by a diverse group of judges. To this end, we have collected and
analyzed data from a large number of subjects (total 168) on a set of 221 of images taken by 33 photographers having
different photographic skill and experience. We also experimented with the rating scales used by previous studies in this
domain by introducing a binary rating system for collecting judges’ opinions. The study also demonstrates the use of
Amazon Mechanical Turk as a crowd-sourcing platform in collecting scientific data and evaluating the skill of the judges
participating in the experiment. We use a variety of performance and correlation metrics to evaluate the consistency of
ratings across different rating scales and compare our findings. A novel feature of our study is an attempt to define a
threshold based on the consistency of ratings when judges rate duplicate images. Our conclusion deviates from earlier
findings and our own expectations, with ratings not being able to determine skill levels of photographers to a statistically
significant level.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.