MATLAB toolbox for ROC analysis of multi-reader multi-case diagnostic imaging studies

Brian J. Smith; Stephen L. Hillis

doi:10.1117/12.2610663

4 April 2022 MATLAB toolbox for ROC analysis of multi-reader multi-case diagnostic imaging studies

Brian J. Smith, Stephen L. Hillis

Proceedings Volume 12035, Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment; 120350G (2022) https://doi.org/10.1117/12.2610663
Event: SPIE Medical Imaging, 2022, San Diego, California, United States

Abstract

A common study design for comparing the performances of diagnostic imaging tests is to obtain ratings from multiple readers of multiple cases whose true statuses are known. Typically, there is overlap between the tests, readers, and/or cases for which special analytical methods are needed to perform statistical comparisons. We present our new MATLAB MRMCaov toolbox, which is designed for multi-reader multi-case comparisons of two or more diagnostic tests. The toolbox allows for statistical comparison of reader performance metrics, such as area under the receiver operating characteristic curve (ROC AUC), with analysis of variance methods originally proposed by Obuchowski and Rockette (1995) and later unified and improved by Hillis and colleagues (2005, 2007, 2008, 2018). MRMCaov is open-source software with an integrated command-line interface for performing multi-reader multi-case statistical analysis, plotting, and presenting results. Its features (1) ROC AUC, likelihood ratios of positive or negative ratings, sensitivity, specificity, and expected utility reader performance metrics; (2) reader-specific ROC curves; (3) user-definable performance metrics; (4) test-specific estimates of mean performance along with confidence intervals and p-values for statistical comparisons; (5) support for factorial, nested, or partially paired study designs; (6) inference for random or fixed readers and cases; (7) DeLong, jackknife, or unbiased covariance estimation; and (8) compatibility with Microsoft Windows, Mac OS, and Linux.

Conference Presentation

Citation Download Citation

Brian J. Smith and Stephen L. Hillis "MATLAB toolbox for ROC analysis of multi-reader multi-case diagnostic imaging studies", Proc. SPIE 12035, Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment, 120350G (4 April 2022); https://doi.org/10.1117/12.2610663

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available