An Empirical Study on Log-based Anomaly Detection Using Machine Learning

The growth of systems complexity increases the need of automated techniques
dedicated to different log analysis tasks such as Log-based Anomaly Detection
(LAD). The latter has been widely addressed in the literature, mostly by means
of different deep learning techniques. Nevertheless, the focus on deep learning
techniques results in less attention being paid to traditional Machine Learning
(ML) techniques, which may perform well in many cases, depending on the context
and the used datasets. Further, the evaluation of different ML techniques is
mostly based on the assessment of their detection accuracy. However, this is is
not enough to decide whether or not a specific ML technique is suitable to
address the LAD problem. Other aspects to consider include the training and
prediction time as well as the sensitivity to hyperparameter tuning. In this
paper, we present a comprehensive empirical study, in which we evaluate
different supervised and semi-supervised, traditional and deep ML techniques
w.r.t. four evaluation criteria: detection accuracy, time performance,
sensitivity of detection accuracy as well as time performance to hyperparameter
tuning. The experimental results show that supervised traditional and deep ML
techniques perform very closely in terms of their detection accuracy and
prediction time. Moreover, the overall evaluation of the sensitivity of the
detection accuracy of the different ML techniques to hyperparameter tuning
shows that supervised traditional ML techniques are less sensitive to
hyperparameter tuning than deep learning techniques. Further, semi-supervised
techniques yield significantly worse detection accuracy than supervised
techniques.