Prototyping of log analysis infrastructure for the Subaru telescope based on the ALMA experience

Kyoko Nakamura; George Kosugi; Tatsuhiro Sato; Eisuke Morita; Yohei Hayashi

doi:10.1117/12.2561190

13 December 2020 Prototyping of log analysis infrastructure for the Subaru telescope based on the ALMA experience

Kyoko Nakamura, George Kosugi, Tatsuhiro Sato, Eisuke Morita, Yohei Hayashi

Proceedings Volume 11449, Observatory Operations: Strategies, Processes, and Systems VIII; 114492D (2020) https://doi.org/10.1117/12.2561190
Event: SPIE Astronomical Telescopes + Instrumentation, 2020, Online Only

Conference Poster

Abstract

Elasticsearch is one of solutions to monitor and analyze logs. Even with ALMA∗, observation logs are stored and anyone can look into it according to their purpose. For example, Hastings, which is a tool discovers the root cause of the defect, is utilized for ACA Correlator subsystem†. It queries logs to an ALMA Elasticsearch storing operational logs, analyzes specific messages which infer troubles, then outputs a result. Before the ALMA Elasticsearch was deployed, logs should have been collected manually in advance. Now the ALMA Elasticsearch has become available and we’ve known: 1) Elasticsearch can directly configure and access features by using REST API, 2) Logs taken even years ago can also be retrieved easily, 3) Elasticsearch’s major update didn’t cause much loss of time to change Hastings, 4) Python has several methods to manage Elasticsearch so that we can choose a favorite one. Therefore, we thought to apply Elasticsearch to the Subaru telescope‡. Size of Subaru logs are quite large but they are not stored in any database yet and just archived. We created a cluster system with Elasticsearch for the evaluation purpose and found ways to store data in a short time. We estimated the total ingestion time for 20 years of telescope status data to be at most 5 months. Our goal is to find a feasible cause of any defects in near real time, to predict any errors that may occur in near future, and to analyze communication between the telescope and observational equipment to optimize observations.

Citation Download Citation

Kyoko Nakamura, George Kosugi, Tatsuhiro Sato, Eisuke Morita, and Yohei Hayashi "Prototyping of log analysis infrastructure for the Subaru telescope based on the ALMA experience", Proc. SPIE 11449, Observatory Operations: Strategies, Processes, and Systems VIII, 114492D (13 December 2020); https://doi.org/10.1117/12.2561190

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available