Article (Scientific journals)
Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction
Hadadi, Fatemeh; DAWES, Joshua; Shin, Donghwan et al.
2024In Empirical Software Engineering, 29, p. 105:1-105:53
Peer Reviewed verified by ORBi
 

Files


Full Text
EMSE2024-108.pdf
Author postprint (1.95 MB) Creative Commons License - Attribution
Download

All documents in ORBilu are protected by a user license.

Send to



Details



Keywords :
Computer Science - Software Engineering
Abstract :
[en] With the increasing complexity and scope of software systems, their dependability is crucial. The analysis of log data recorded during system execution can enable engineers to automatically predict failures at run time. Several Machine Learning (ML) techniques, including traditional ML and Deep Learning (DL), have been proposed to automate such tasks. However, current empirical studies are limited in terms of covering all main DL types -- Recurrent Neural Network (RNN), Convolutional Neural network (CNN), and transformer -- as well as examining them on a wide range of diverse datasets. In this paper, we aim to address these issues by systematically investigating the combination of log data embedding strategies and DL types for failure prediction. To that end, we propose a modular architecture to accommodate various configurations of embedding strategies and DL-based encoders. To further investigate how dataset characteristics such as dataset size and failure percentage affect model accuracy, we synthesised 360 datasets, with varying characteristics, for three distinct system behavioral models, based on a systematic and automated generation approach. Using the F1 score metric, our results show that the best overall performing configuration is a CNN-based encoder with Logkey2vec. Additionally, we provide specific dataset conditions, namely a dataset size >350 or a failure percentage >7.5%, under which this configuration demonstrates high accuracy for failure prediction.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation
Disciplines :
Computer science
Author, co-author :
Hadadi, Fatemeh
DAWES, Joshua ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Shin, Donghwan
BIANCULLI, Domenico  ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
BRIAND, Lionel 
External co-authors :
yes
Language :
English
Title :
Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction
Publication date :
20 June 2024
Journal title :
Empirical Software Engineering
ISSN :
1382-3256
eISSN :
1573-7616
Volume :
29
Pages :
105:1-105:53
Peer reviewed :
Peer Reviewed verified by ORBi
Focus Area :
Security, Reliability and Trust
European Projects :
H2020 - 957254 - COSMOS - DevOps for Complex Cyber-physical Systems
Funders :
Union Européenne [BE]
Available on ORBilu :
since 22 November 2023

Statistics


Number of views
32 (13 by Unilu)
Number of downloads
17 (2 by Unilu)

Scopus citations®
 
0
Scopus citations®
without self-citations
0
OpenCitations
 
0

Bibliography


Similar publications



Contact ORBilu