AIR: A Light-Weight Yet High-Performance Dataflow Engine based on Asynchronous Iterative Routing

Distributed Stream Processing Engines (DSPEs) are
currently among the most emerging topics in data management,
with applications ranging from real-time event monitoring to
processing complex dataflow programs and big data analytics.
In this paper, we describe the architecture of our AIR engine,
which is designed from scratch in C++ using the Message Passing
Interface (MPI), pthreads for multithreading, and is directly
deployed on top of a common HPC workload manager such as
SLURM. AIR implements a light-weight, dynamic sharding protocol (referred to as “Asynchronous Iterative Routing”), which
facilitates a direct and asynchronous communication among all
worker nodes and thereby completely avoids any additional
communication overhead with a dedicated master node. With its
unique design, AIR fills the gap between the prevalent scale-out
(but Java-based) architectures like Apache Spark and Flink, on
one hand, and recent scale-up (and C++ based) prototypes such
as StreamBox and PiCo, on the other hand. Our experiments
over various benchmark settings confirm that AIR performs as
good as the best scale-up SPEs on a single-node setup, while
it outperforms existing scale-out DSPEs in terms of processing
latency and sustainable throughput by a factor of up to 15 in a
distributed setting.