Reference : Guidelines for Assessing the Accuracy of Log Message Template Identification Techniques
Scientific congresses, symposiums and conference proceedings : Paper published in a book
Engineering, computing & technology : Computer science
Security, Reliability and Trust
http://hdl.handle.net/10993/50072
Guidelines for Assessing the Accuracy of Log Message Template Identification Techniques
English
Khan, Zanis Ali mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV >]
Shin, Donghwan mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV >]
Bianculli, Domenico mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV >]
Briand, Lionel mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV >]
In press
Proceedings of the 44th International Conference on Software Engineering (ICSE ’22)
ACM
Yes
No
International
New York, NY
USA
44th International Conference on Software Engineering (ICSE ’22)
from 21-05-2022 to 29-05-2022
[en] logs ; template identification ; metrics
[en] Log message template identification aims to convert raw logs
containing free-formed log messages into structured logs to be
processed by automated log-based analysis, such as anomaly detection
and model inference. While many techniques have been proposed in the
literature, only two recent studies provide a comprehensive evaluation
and comparison of the techniques using an established benchmark
composed of real-world logs. Nevertheless, we argue that both studies
have the following issues: (1) they used different accuracy metrics
without comparison between them, (2) some ground-truth (oracle)
templates are incorrect, and (3) the accuracy evaluation
results do not provide any information regarding incorrectly
identified templates.

In this paper, we address the above issues by providing three
guidelines for assessing the accuracy of log template identification
techniques: (1) use appropriate accuracy metrics, (2) perform oracle
template correction, and (3) perform analysis of incorrect templates. We then
assess the application of such guidelines through a comprehensive
evaluation of 14 existing template identification techniques on the
established benchmark logs. Results show very different insights than
existing studies and in particular a much less optimistic outlook on
existing techniques.
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > Software Verification and Validation Lab (SVV Lab)
Researchers ; Professionals
http://hdl.handle.net/10993/50072
10.1145/3510003.3510101

File(s) associated to this reference

Fulltext file(s):

FileCommentaryVersionSizeAccess
Open access
icse2022.pdfAuthor postprint610.56 kBView/Open

Bookmark and Share SFX Query

All documents in ORBilu are protected by a user license.