A Distance-Based Method to Detect Anomalous Attributes in Log Files

Dealing with large volumes of logs is like the prover-
bial needle in the haystack problem. Finding relevant events that
might be associated with an incident, or real time analysis of
operational logs is extremely difficult when the underlying data
volume is huge and when no explicit misuse model exists. While
domain-specific knowledge and human expertise may be useful in
analysing log data, automated approaches for detecting anomalies
and track incidents are the only viable solutions when confronted
with large volumes of data. In this paper we address the issue of
automated log analysis and consider more specifically the case
of ISP-provided firewall logs. We leverage approaches derived
from statistical process control and information theory in order
to track potential incidents and detect suspicious network activity.