Big Data Architectures For Robust Systems

Monitoring robustness of critical systems/infrastructures is the major use case for anomaly detection. A robust system designates a structure not only safe against intentional attacks, but also capable of stemming internal failures. These systems face two primary risks: cyber attacks fall into the first category, whereas failing hardware components are part of the second category. In both cases, fast decision making is crucial. Hence, streaming data processing is the decisive asset to consider.

With this background, in this thesis, we investigate two scenarios from the fields of mobile network sanity monitoring and cyber-physical security. Our contribution is threefold: We display how the real-time requirements of the two use cases push existing frameworks to their utter limits; We show which anomaly detection methods can be used to facilitate instant assessment rendering; We blueprint the extensions we contributed to big data frameworks, which are powering major silicon valley companies, to make them capable of supporting our use cases.

The data-sets issued by our monitoring systems yield different properties than data from internet companies such as Google, Facebook or LinkedIn. In this work we establish our use cases, illustrate the mathematical models employed for the decision taking, and examine how big data architectures have to be altered to support our scenarios.