Paper published in a book (Scientific congresses, symposiums and conference proceedings)
Guidelines for Assessing the Accuracy of Log Message Template Identification Techniques
Khan, Zanis Ali; Shin, Donghwan; Bianculli, Domenico et al.
2022In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22)
Peer reviewed
 

Files


Full Text
icse2022.pdf
Author postprint (625.21 kB)
Download

All documents in ORBilu are protected by a user license.

Send to



Details



Keywords :
logs; template identification; metrics
Abstract :
[en] Log message template identification aims to convert raw logs containing free-formed log messages into structured logs to be processed by automated log-based analysis, such as anomaly detection and model inference. While many techniques have been proposed in the literature, only two recent studies provide a comprehensive evaluation and comparison of the techniques using an established benchmark composed of real-world logs. Nevertheless, we argue that both studies have the following issues: (1) they used different accuracy metrics without comparison between them, (2) some ground-truth (oracle) templates are incorrect, and (3) the accuracy evaluation results do not provide any information regarding incorrectly identified templates. In this paper, we address the above issues by providing three guidelines for assessing the accuracy of log template identification techniques: (1) use appropriate accuracy metrics, (2) perform oracle template correction, and (3) perform analysis of incorrect templates. We then assess the application of such guidelines through a comprehensive evaluation of 14 existing template identification techniques on the established benchmark logs. Results show very different insights than existing studies and in particular a much less optimistic outlook on existing techniques.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > Software Verification and Validation Lab (SVV Lab)
Disciplines :
Computer science
Author, co-author :
Khan, Zanis Ali ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Shin, Donghwan ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Bianculli, Domenico  ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Briand, Lionel ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
External co-authors :
no
Language :
English
Title :
Guidelines for Assessing the Accuracy of Log Message Template Identification Techniques
Publication date :
July 2022
Event name :
44th International Conference on Software Engineering (ICSE ’22)
Event date :
from 21-05-2022 to 29-05-2022
Audience :
International
Main work title :
Proceedings of the 44th International Conference on Software Engineering (ICSE ’22)
Publisher :
ACM, New York, NY, United States
Pages :
1095-1106
Peer reviewed :
Peer reviewed
Focus Area :
Security, Reliability and Trust
Commentary :
The paper presentation can be found here: https://youtu.be/R_bEdohzn6M
Available on ORBilu :
since 27 January 2022

Statistics


Number of views
649 (80 by Unilu)
Number of downloads
1005 (27 by Unilu)

Scopus citations®
 
13
Scopus citations®
without self-citations
13
OpenCitations
 
2

Bibliography


Similar publications



Contact ORBilu