Integrating the ODI-PPA scientific gateway with the QuickReduce pipeline for on-demand processing

Michael D. Young; Ralf Kotulla; Arvind Gopu; Wilson Liu

doi:10.1117/12.2056763

18 July 2014 Integrating the ODI-PPA scientific gateway with the QuickReduce pipeline for on-demand processing

Michael D. Young, Ralf Kotulla, Arvind Gopu, Wilson Liu

Proceedings Volume 9152, Software and Cyberinfrastructure for Astronomy III; 91522U (2014) https://doi.org/10.1117/12.2056763
Event: SPIE Astronomical Telescopes + Instrumentation, 2014, Montréal, Quebec, Canada

Abstract

As imaging systems improve, the size of astronomical data has continued to grow, making the transfer and processing of data a significant burden. To solve this problem for the WIYN Observatory One Degree Imager (ODI), we developed the ODI-Portal, Pipeline, and Archive (ODI-PPA) science gateway, integrating the data archive, data reduction pipelines, and a user portal. In this paper, we discuss the integration of the QuickReduce (QR) pipeline into PPA's Tier 2 processing framework. QR is a set of parallelized, stand-alone Python routines accessible to all users, and operators who can create master calibration products and produce standardized calibrated data, with a short turn-around time. Upon completion, the data are ingested into the archive and portal, and made available to authorized users. Quality metrics and diagnostic plots are generated and presented via the portal for operator approval and user perusal. Additionally, users can tailor the calibration process to their specific science objective(s) by selecting custom datasets, applying preferred master calibrations or generating their own, and selecting pipeline options. Submission of a QuickReduce job initiates data staging, pipeline execution, and ingestion of output data products all while allowing the user to monitor the process status, and to download or further process/analyze the output within the portal. User-generated data products are placed into a private user-space within the portal. ODI-PPA leverages cyberinfrastructure at Indiana University including the Big Red II supercomputer, the Scholarly Data Archive tape system and the Data Capacitor shared file system.

Citation Download Citation

Michael D. Young, Ralf Kotulla, Arvind Gopu, and Wilson Liu "Integrating the ODI-PPA scientific gateway with the QuickReduce pipeline for on-demand processing", Proc. SPIE 9152, Software and Cyberinfrastructure for Astronomy III, 91522U (18 July 2014); https://doi.org/10.1117/12.2056763

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available