ML health monitor: taking the pulse of machine learning algorithms in production

Sindhu Ghanta; Sriram Subramanian; Lior Khermosh; Swaminathan Sundararaman; Harshil Shah; Yakov Goldberg; Drew Roselli; Nisha Talagala

doi:10.1117/12.2529598

12 September 2019 ML health monitor: taking the pulse of machine learning algorithms in production

Sindhu Ghanta, Sriram Subramanian, Lior Khermosh, Swaminathan Sundararaman, Harshil Shah, Yakov Goldberg, Drew Roselli, Nisha Talagala

Author Affiliations +

Proceedings Volume 11139, Applications of Machine Learning; 111390R (2019) https://doi.org/10.1117/12.2529598
Event: SPIE Optical Engineering + Applications, 2019, San Diego, California, United States

Abstract

Bringing the research advances in Machine Learning (ML) to production is necessary for businesses to gain value from ML. A key challenge of production ML is the monitoring and management of real-time prediction quality. This is complicated by the variability of live production data, the absence of real-time labels and the non-determinism posed by ML techniques themselves. We define ML Health as the real time assessment of ML prediction quality and present an approach to monitoring and improving ML Health. Specifically, a complete solution to monitor and manage ML Health within a realistic full production ML lifecycle. We describe a number of ML Health techniques and assess their efficacy via publicly available datasets. Our solution handles production realities such as scale, heterogeneity and distributed runtimes. We present what we believe is the first solution to production ML Health explored at both an empirical and complete system implementation level.

Conference Presentation

Citation Download Citation

Sindhu Ghanta, Sriram Subramanian, Lior Khermosh, Swaminathan Sundararaman, Harshil Shah, Yakov Goldberg, Drew Roselli, and Nisha Talagala "ML health monitor: taking the pulse of machine learning algorithms in production", Proc. SPIE 11139, Applications of Machine Learning, 111390R (12 September 2019); https://doi.org/10.1117/12.2529598

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available