Optimizing log parsing efficiency: a heuristic-based distributed system approach

Dongqing Liu

doi:10.1117/12.3032526

12 July 2024 Optimizing log parsing efficiency: a heuristic-based distributed system approach

Dongqing Liu

Proceedings Volume 13185, International Conference on Communication, Information, and Digital Technologies (CIDT2024) ; 131850G (2024) https://doi.org/10.1117/12.3032526
Event: International Conference on Communication, Information and Digital Technologies, 2024, Wuhan, China

Abstract

System logs are widely used in system management for reliability assurance and serve as crucial data sources for detecting system anomalies. Prior to anomaly detection, it is necessary to parse raw textual logs into structured logs that can be recognized by anomaly detection models. To address the inefficiency of existing log parsing algorithms, this study proposes a distributed system log parsing task allocation model and a heuristic-based log parsing algorithm. By combining the task allocation model with the parsing algorithm, the log parsing task is completed. Experimental results demonstrate the feasibility and effectiveness of the proposed model and algorithm, which can reduce parsing time and enhance parsing efficiency. Future research could explore log parsing methods based on deep learning and log parsing algorithms integrating natural language processing techniques to improve the accuracy and applicability of log parsing.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Dongqing Liu "Optimizing log parsing efficiency: a heuristic-based distributed system approach", Proc. SPIE 13185, International Conference on Communication, Information, and Digital Technologies (CIDT2024) , 131850G (12 July 2024); https://doi.org/10.1117/12.3032526

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
10 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Distributed computing

Computing systems

Detection and tracking algorithms

Design

Mining

Mathematical optimization

Reliability

Show All Keywords

Keywords/Phrases

Search In:

Publication Years