Reference : Guidelines for Assessing the Accuracy of Log Message Template Identification Techniques

	Document type :	Scientific congresses, symposiums and conference proceedings : Paper published in a book
	Discipline(s) :	Engineering, computing & technology : Computer science
	Focus Areas :	Security, Reliability and Trust
	To cite this reference:	http://hdl.handle.net/10993/50072

Title :	Guidelines for Assessing the Accuracy of Log Message Template Identification Techniques
Language :	English
Author, co-author :	Khan, Zanis Ali [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV >]
	Shin, Donghwan [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV >]
	Bianculli, Domenico [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV >]
	Briand, Lionel [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV >]
Publication date :	Jul-2022
Main document title :	Proceedings of the 44th International Conference on Software Engineering (ICSE ’22)
Publisher :	ACM
Pages :	1095-1106
Peer reviewed :	Yes
On invitation :	No
Audience :	International
City :	New York, NY
Country :	USA
Event name :	44th International Conference on Software Engineering (ICSE ’22)
Event date :	from 21-05-2022 to 29-05-2022
Keywords :	[en] logs ; template identification ; metrics
Abstract :	[en] Log message template identification aims to convert raw logs containing free-formed log messages into structured logs to be processed by automated log-based analysis, such as anomaly detection and model inference. While many techniques have been proposed in the literature, only two recent studies provide a comprehensive evaluation and comparison of the techniques using an established benchmark composed of real-world logs. Nevertheless, we argue that both studies have the following issues: (1) they used different accuracy metrics without comparison between them, (2) some ground-truth (oracle) templates are incorrect, and (3) the accuracy evaluation results do not provide any information regarding incorrectly identified templates. In this paper, we address the above issues by providing three guidelines for assessing the accuracy of log template identification techniques: (1) use appropriate accuracy metrics, (2) perform oracle template correction, and (3) perform analysis of incorrect templates. We then assess the application of such guidelines through a comprehensive evaluation of 14 existing template identification techniques on the established benchmark logs. Results show very different insights than existing studies and in particular a much less optimistic outlook on existing techniques.
Research centres :	Interdisciplinary Centre for Security, Reliability and Trust (SnT) > Software Verification and Validation Lab (SVV Lab)
Target :	Researchers ; Professionals
Permalink :	http://hdl.handle.net/10993/50072
DOI :	10.1145/3510003.3510101
Commentary :	The paper presentation can be found here: https://youtu.be/R_bEdohzn6M

	File(s) associated to this reference
	Fulltext file(s): File Commentary Version Size Access Open access icse2022.pdf Author postprint 610.56 kB View/Open

All documents in ORBi^lu are protected by a user license.

 

University of Luxembourg Library |  Feedback |  Legal notices Site Map