[en] The growth of systems complexity increases the need of automated techniques
dedicated to different log analysis tasks such as Log-based Anomaly Detection
(LAD). The latter has been widely addressed in the literature, mostly by means
of different deep learning techniques. Nevertheless, the focus on deep learning
techniques results in less attention being paid to traditional Machine Learning
(ML) techniques, which may perform well in many cases, depending on the context
and the used datasets. Further, the evaluation of different ML techniques is
mostly based on the assessment of their detection accuracy. However, this is is
not enough to decide whether or not a specific ML technique is suitable to
address the LAD problem. Other aspects to consider include the training and
prediction time as well as the sensitivity to hyperparameter tuning. In this
paper, we present a comprehensive empirical study, in which we evaluate
different supervised and semi-supervised, traditional and deep ML techniques
w.r.t. four evaluation criteria: detection accuracy, time performance,
sensitivity of detection accuracy as well as time performance to hyperparameter
tuning. The experimental results show that supervised traditional and deep ML
techniques perform very closely in terms of their detection accuracy and
prediction time. Moreover, the overall evaluation of the sensitivity of the
detection accuracy of the different ML techniques to hyperparameter tuning
shows that supervised traditional ML techniques are less sensitive to
hyperparameter tuning than deep learning techniques. Further, semi-supervised
techniques yield significantly worse detection accuracy than supervised
techniques.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation
Disciplines :
Computer science
Author, co-author :
Ali, Shan
Boufaied, Chaima
BIANCULLI, Domenico ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Branco, Paula
Briand, Lionel
External co-authors :
yes
Language :
English
Title :
A Comprehensive Study of Machine Learning Techniques for Log-Based Anomaly Detection
LOGODOR - Automated Log Smell Detection and Removal
Funders :
FNR - Luxembourg National Research Fund
Funding number :
C22/IS/17373407/LOGODOR
Funding text :
This work was supported by the Natural Sciences and Research Council of Canada (NSERC) Discov-
ery Grant program, the Canada Research Chairs (CRC) program, the Mitacs Accelerate program, the Ontario
Graduate Scholarship (OGS) program, and the Luxembourg National Research Fund (FNR), grant reference
C22/IS/17373407/LOGODOR; Lionel Briand was partly funded by the Research Ireland grant 13/RC/2094-2.
For the purpose of open access, and in fulfillment of the obligations arising from the grant agreement, the
authors have applied a Creative Commons Attribution 4.0 International (CC BY 4.0) license to any Author
Accepted Manuscript version arising from this submission.
Adeba JL, Kim D-H, Kwak J (2024) Sarlog: semantic-aware robust log anomaly detection via bert-augmented contrastive learning. IEEE Int Things J
Almodovar C, Sabrina F, Karimi S, Azad S (2023) Logfit: log anomaly detection using fine-tuned language models
Bengio Y (2012) Practical recommendations for gradient-based training of deep architectures. In: Neural networks: tricks of the trade: second edition. Springer, pp 437–478
Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. J Mach Learn Res 13(2)
Bernard S, Heutte L, Adam S (2009) Influence of hyperparameters on random forest accuracy. In: Multiple classifier systems: 8th international workshop, MCS 2009, Reykjavik, Iceland, June 10-12, 2009. Proceedings 8. Springer, pp 171–180
L. Breiman Random forests Mach Learn 45 5 32 10.1023/A:1010933404324
M. Catillo A. Pecchia U. Villano Autolog: anomaly detection by deep autoencoding of system logs Expert Syst Appl 191 116263 10.1016/j.eswa.2021.116263
Chen J, Chong W, Yu S, Xu Z, Tan C, Chen N (2022) Tcn-based lightweight log anomaly detection in cloud-edge collaborative environment. In: 2022 Tenth international conference on advanced cloud and big data (CBD). IEEE, pp 13–18
Chen Z, Liu J, Gu W, Su Y, Lyu MR (2021) Experience report: Deep learning-based system log analysis for anomaly detection. arXiv:2107.05908
Chen M, Zheng AX, Lloyd J, Jordan MI, Brewer E (2004) Failure diagnosis using decision trees. In: International conference on autonomic computing, 2004. Proceedings. IEEE, pp 36–43
C. Cortes V. Vapnik Support vector machine Mach Learn 20 3 273 297 10.1007/BF00994018
Cotroneo D, De Simone L, Liguori P, Natella R, Bidokhti N (2019) How bad can a bug get? an empirical analysis of software failures in the openstack cloud computing platform. In: Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 200–211
Creech G, Hu J (2013) Generation of a new ids test dataset: time to retire the kdd collection. In: 2013 IEEE wireless communications and networking conference (WCNC). IEEE, pp 4487–4492
Devlin J, Chang M-W, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805
Digital Research Alliance of Canada. https://alliancecan.ca/en
Du M, Li F, Zheng G, Srikumar V (2017) Deeplog: anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp 1285–1298
O.J. Dunn Multiple comparisons using rank sums Technometrics 6 3 241 252 10.1080/00401706.1964.10490181
Du Q, Zhao L, Xu J, Han Y, Zhang S (2021) Log-based anomaly detection with multi-head scaled dot-product attention mechanism. In: Database and expert systems applications: 32nd international conference, DEXA 2021, Virtual Event, September 27–30, 2021, Proceedings, Part I 32. Springer, pp 335–347
M. Fernández-Delgado E. Cernadas S. Barro D. Amorim Do we need hundreds of classifiers to solve real world classification problems? J Mach Learn Res 15 1 3133 3181 3277155
E. Fix J. Hodges Discriminatory analysis. Nonparametric discrimination: consistency properties Int Stat Rev /Revue Internationale de Statistique 57 3 238 247
Genuer R, Poggi J-M, Tuleau C (2008) Random forests: some methodological insights. arXiv:0811.3619
K. Gong S. Luo L. Pan L. Zhang Y. Zhang H. Yu Logeta: time-aware cross-system log-based anomaly detection with inter-class boundary optimization Future Gener Comput Syst 157 16 28 10.1016/j.future.2024.03.028
Guo H, Yang J, Liu J, Bai J, Wang B, Li Z, Zheng T, Zhang B, Peng J, Tian Q (2024) Logformer: a pre-train and tuning pipeline for log anomaly detection. In: Proceedings of the AAAI conference on artificial intelligence, vol 38, pp 135–143
Guo H, Yuan S, Wu X (2021) Logbert: log anomaly detection via bert. In: 2021 International joint conference on neural networks (IJCNN). IEEE, pp 1–8
Han X, Cheng H, Xu D, Yuan S (2021) Interpretablesad: interpretable anomaly detection in sequential log data. In: 2021 IEEE international conference on big data (Big Data). IEEE, pp 1183–1192
J.T. Hancock T.M. Khoshgoftaar J.M. Johnson Evaluating classifier performance with highly imbalanced big data J Big Data 10 1 42 10.1186/s40537-023-00724-5
Hashemi S, Mäntylä M (2021) Onelog: towards end-to-end training in software log anomaly detection. arXiv:2104.07324
He S, Zhu J, He P, Lyu MR (2020) Loghub: a large collection of system log datasets towards automated log analytics. arXiv:2008.06448
He P, Zhu J, Zheng Z, Lyu MR (2017) Drain: an online log parsing approach with fixed depth tree. In: 2017 IEEE international conference on web services (ICWS). IEEE, pp 33–40
S. Hochreiter J. Schmidhuber Long short-term memory Neural Comput 9 8 1735 1780 10.1162/neco.1997.9.8.1735
S. Huang Y. Liu C. Fung R. He Y. Zhao H. Yang Z. Luan Hitanomaly: hierarchical transformers for anomaly detection in system log IEEE Trans Netw Serv Manag 17 4 2064 2076 10.1109/TNSM.2020.3034647
Huang S, Liu Y, Fung C, Wang H, Yang H, Luan Z (2023) Improving log-based anomaly detection by pre-training hierarchical transformers. IEEE Trans Comput
Huo Y, Su Y, Lee C, Lyu MR (2023) Semparser: a semantic parser for log analytics. In: 2023 IEEE/ACM 45th international conference on software engineering (ICSE). IEEE, pp 881–893
Joulin A, Grave E, Bojanowski P, Douze M, Jégou H, Mikolov T (2016) Fasttext. Zip: compressing text classification models. arXiv:1612.03651
Khan ZA, Shin D, Bianculli D, Briand L (2022) Guidelines for assessing the accuracy of log message template identification techniques. In: Proceedings of the 44th international conference on software engineering, pp 1095–1106
Z.A. Khan D. Shin D. Bianculli L.C. Briand Impact of log parsing on deep learning-based anomaly detection Empir Softw Eng 29 6 139 10.1007/s10664-024-10533-w
Kiefer J, Wolfowitz J (1952) Stochastic estimation of the maximum of a regression function. Ann Math Stat 462–466
Kruskal W (1952) Kruskal and wallis’ test. J Am Stat Assoc 583–618
M. Landauer S. Onder F. Skopik M. Wurzenberger Deep learning for anomaly detection in log data: a survey Mach Learn Appl 12 100470
M. Landauer F. Skopik M. Wurzenberger A critical review of common log data sets used for evaluation of sequence-based anomaly detection techniques Proc ACM Softw Eng 1 FSE 1354 1375 10.1145/3660768
Le V-H, Zhang H (2021) Log-based anomaly detection without log parsing. In: 2021 36th IEEE/ACM international conference on automated software engineering (ASE). IEEE, pp 492–504
Le V-H, Zhang H (2022) Log-based anomaly detection with deep learning: how far are we? https://github.com/LogIntelligence/LogADEmpirical
Le V-H, Zhang H (2022) Log-based anomaly detection with deep learning: How far are we? In: Proceedings of the 44th international conference on software engineering. ICSE ’22, pp. 1356–1367. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3510003.3510155
Lee C, Yang T, Chen Z, Su Y, Yang Y, Lyu MR (2023) Heterogeneous anomaly detection for software systems via semi-supervised cross-modal attention. In: 2023 IEEE/ACM 45th international conference on software engineering (ICSE). IEEE, pp 1724–1736
Li X, Chen P, Jing L, He Z, Yu G (2022) Swisslog: robust anomaly detection and localization for interleaved unstructured logs. IEEE Trans Depend Secure Comput
Lin Y, Deng H, Li X (2024) Fastlogad: log anomaly detection with mask-guided pseudo anomaly generation and discrimination. arXiv:2404.08750
Lin Q, Zhang H, Lou J-G, Zhang Y, Chen X (2016) Log clustering based problem identification for online service systems. In: Proceedings of the 38th international conference on software engineering companion, pp 102–111
Li Z, Shi J, Van Leeuwen M (2024) Graph neural networks based log anomaly detection and explanation. In: Proceedings of the 2024 IEEE/ACM 46th international conference on software engineering: companion proceedings, pp 306–307
X. Liu W. Liu X. Di J. Li B. Cai W. Ren H. Yang Lognads: network anomaly detection scheme based on log semantics representation Future Gener Comput Syst 124 390 405 10.1016/j.future.2021.05.024
LogIntelligence (2021) Log-based Anomaly Detection Without Log Parsing, ASE, Research Track. https://github.com/LogIntelligence/NeuralLog (2021)
LOGPAI:Log Analytics Powered by AI. https://github.com/logpai
Lu S, Wei X, Li Y, Wang L (2018) Detecting anomaly in big data system logs using convolutional neural network. In: 2018 IEEE 16th intl conf on dependable, autonomic and secure computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/DataCom/CyberSciTech). IEEE, pp 151–158
Meng W, Liu Y, Zhu Y, Zhang S, Pei D, Liu Y, Chen Y, Zhang R, Tao S, Sun P et al (2019) Loganomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs. In: IJCAI, vol 19, pp 4739–4745
Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv:1301.3781
Nguyen H-T, Nguyen L-V, Le V-H, Zhang H, Le M-T (2024) Efficient log-based anomaly detection with knowledge distillation. In: 2024 IEEE international conference on web services (ICWS). IEEE, pp 578–589
Okewu E, Adewole P, Sennaike O (2019) Experimental comparison of stochastic optimizers in deep learning. In: Computational science and its applications–ICCSA 2019: 19th international conference, Saint Petersburg, Russia, July 1–4, 2019, Proceedings, Part V 19. Springer, pp 704–715
Oliner A, Stearley J (2007) What supercomputers say: a study of five system logs. In: 37th Annual IEEE/IFIP international conference on dependable systems and networks (DSN’07). IEEE, pp 575–584
Oshiro TM, Perez PS, Baranauskas JA (2012) How many trees in a random forest? In: Machine learning and data mining in pattern recognition: 8th international conference, MLDM 2012, Berlin, Germany, July 13-20, 2012. Proceedings 8. Springer, pp 154–168
Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543
Perin G, Picek S (2021) On the influence of optimizers in deep learning-based side-channel analysis. In: Selected areas in cryptography: 27th international conference, Halifax, NS, Canada (Virtual Event), October 21-23, 2020, Revised Selected Papers 27. Springer, pp 615–636
P. Probst A.-L. Boulesteix To tune or not to tune the number of trees in random forest J Mach Learn Res 18 181 1 18 3827069
P. Probst A.-L. Boulesteix B. Bischl Tunability: importance of hyperparameters of machine learning algorithms J Mach Learn Res 20 1 1934 1965 3948093
P. Probst M.N. Wright A.-L. Boulesteix Hyperparameters and tuning strategies for random forest Wiley Interdisc Rev Data Min Knowl Disc 9 3 1301 10.1002/widm.1301
Python Software Foundation: Time module in Python. https://docs.python.org/3/library/time.html
Qi J, Luan Z, Huang S, Fung C, Yang H, Li H, Zhu D, Qian D (2023) Logencoder: log-based contrastive representation learning for anomaly detection. IEEE Trans Netw Serv Manag
Qi J, Luan Z, Huang S, Wang Y, Fung C, Yang H, Qian D (2022) Adanomaly: adaptive anomaly detection for system logs with adversarial learning. In: NOMS 2022-2022 IEEE/IFIP network operations and management symposium. IEEE, pp 1–5
G. Salton C. Buckley Term-weighting approaches in automatic text retrieval Inf Process Manag 24 5 513 523 10.1016/0306-4573(88)90021-0
B. Schölkopf J.C. Platt J. Shawe-Taylor A.J. Smola R.C. Williamson Estimating the support of a high-dimensional distribution Neural Comput 13 7 1443 1471 10.1162/089976601750264965
Z. Wang J. Tian H. Fang L. Chen J. Qin Lightlog: a lightweight temporal convolutional network for log anomaly detection on the edge Comput Netw 203 108616 10.1016/j.comnet.2021.108616
P. Wang X. Zhang Z. Cao W. Xu W. Li Loggt: cross-system log anomaly detection via heterogeneous graph feature and transfer learning Expert Syst Appl 251 124082 10.1016/j.eswa.2024.124082
X. Wu H. Li F. Khomh On the effectiveness of log representation for log-based anomaly detection Empir Softw Eng 28 6 137 10.1007/s10664-023-10364-1
wuyifan18 (2020) Pytorch Implementation of DeepLog. https://github.com/wuyifan18/DeepLog
B. Xia Y. Bai J. Yin Y. Li J. Xu Loggan: a log-level generative adversarial network for anomaly detection using permutation event modeling Inf Syst Front 23 285 298 10.1007/s10796-020-10026-3
Xiao R, Li W, Lu J, Jin S (2024) Contexlog: non-parsing log anomaly detection with all information preservation and enhanced contextual representation. IEEE Trans Netw Serv Manag
Y. Xie L. Ji X. Cheng An attention-based gru network for anomaly detection from system logs IEICE Trans Inf Syst 103 8 1916 1919 10.1587/transinf.2020EDL8016
Xie Y, Zhang H, Babar MA (2022) Loggd: detecting anomalies from system logs with graph neural networks. In: 2022 IEEE 22nd international conference on software quality, reliability and security (QRS). IEEE, pp 299–310
Xu W, Huang L, Fox A, Patterson D, Jordan MI (2009) Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22nd symposium on operating systems principles, pp 117–132
L. Yang J. Chen S. Gao Z. Gong H. Zhang Y. Kang H. Li Try with simpler-an evaluation of improved principal component analysis in log-based anomaly detection ACM Trans Softw Eng Methodol 33 5 1 27 10.1145/3644386
Yang L, Chen J, Wang Z, Wang W, Jiang J, Dong X, Zhang W (2021) Semi-supervised log-based anomaly detection via probabilistic label estimation. In: 2021 IEEE/ACM 43rd international conference on software engineering (ICSE). IEEE, pp 1448–1460
J. Yao M. Shepperd The impact of using biased performance metrics on software defect prediction research Inf Softw Technol 139 106664 10.1016/j.infsof.2021.106664
Z. Yin X. Kong C. Yin Semi-supervised log anomaly detection based on bidirectional temporal convolution network Comput Secur 140 103808 10.1016/j.cose.2024.103808
J. Yu S. Kang Clustering-based proxy measure for optimizing one-class classifiers Pattern Recognit Lett 117 37 44 10.1016/j.patrec.2018.11.017
Yu B, Yao J, Fu Q, Zhong Z, Xie H, Wu Y, Ma Y, He P (2024) Deep learning or classical machine learning? an empirical study on log-based anomaly detection. In: Proceedings of the 46th IEEE/ACM international conference on software engineering
Zang R, Guo H, Yang J, Liu J, Li Z, Zheng T, Shi X, Zheng L, Zhang B (2024) Mlad: a unified model for multi-system log anomaly detection. arXiv:2401.07655
C. Zhang X. Wang H. Zhang J. Zhang H. Zhang C. Liu P. Han Layerlog: log sequence anomaly detection based on hierarchical semantics Appl Soft Comput 132 109860 10.1016/j.asoc.2022.109860
Zhang M, Chen J, Liu J, Wang J, Shi R, Sheng H (2022) Logst: log semi-supervised anomaly detection based on sentence-bert. In: 2022 7th International conference on signal and image processing (ICSIP). IEEE, pp 356–361
Zhang X, Xu Y, Lin Q, Qiao B, Zhang H, Dang Y, Xie C, Yang X, Cheng Q, Li Z et al (2019) Robust log-based anomaly detection on unstable log data. In: Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 807–817
Zhu B, Li J, Gu R, Wang L (2020) An approach to cloud platform log anomaly detection based on natural language processing and lstm. In: 2020 3rd International conference on algorithms, computing and artificial intelligence, pp 1–7
