Reference : AIR: A Light-Weight Yet High-Performance Dataflow Engine based on Asynchronous Iterat... |
Scientific congresses, symposiums and conference proceedings : Paper published in a book | |||
Engineering, computing & technology : Computer science | |||
http://hdl.handle.net/10993/45326 | |||
AIR: A Light-Weight Yet High-Performance Dataflow Engine based on Asynchronous Iterative Routing | |
English | |
Ellampallil Venugopal, Vinu ![]() | |
Theobald, Martin ![]() | |
Chaychi, Samira ![]() | |
Tawakuli, Amal ![]() | |
1-Sep-2020 | |
AIR: A Light-Weight Yet High-Performance Dataflow Engine based on Asynchronous Iterative Routing | |
IEEE | |
51-58 | |
Yes | |
Yes | |
International | |
978-1-7281-9924-5 | |
32nd International Symposium on Computer Architecture and High Performance Computing | |
from 08-09-2020 to 11-09-2020 | |
[en] Stream data processing ; Big Data ; sustainable-throughput | |
[en] Distributed Stream Processing Engines (DSPEs) are
currently among the most emerging topics in data management, with applications ranging from real-time event monitoring to processing complex dataflow programs and big data analytics. In this paper, we describe the architecture of our AIR engine, which is designed from scratch in C++ using the Message Passing Interface (MPI), pthreads for multithreading, and is directly deployed on top of a common HPC workload manager such as SLURM. AIR implements a light-weight, dynamic sharding protocol (referred to as “Asynchronous Iterative Routing”), which facilitates a direct and asynchronous communication among all worker nodes and thereby completely avoids any additional communication overhead with a dedicated master node. With its unique design, AIR fills the gap between the prevalent scale-out (but Java-based) architectures like Apache Spark and Flink, on one hand, and recent scale-up (and C++ based) prototypes such as StreamBox and PiCo, on the other hand. Our experiments over various benchmark settings confirm that AIR performs as good as the best scale-up SPEs on a single-node setup, while it outperforms existing scale-out DSPEs in terms of processing latency and sustainable throughput by a factor of up to 15 in a distributed setting. | |
University of Luxembourg - UL | |
http://hdl.handle.net/10993/45326 | |
https://conferences.computer.org/sbacpad/pdfs/SBAC-PAD2020-2fQ2vSYuhExkpkZ9tActSv/992400a051/992400a051.pdf |
File(s) associated to this reference | ||||||||||||||
Fulltext file(s):
| ||||||||||||||
All documents in ORBilu are protected by a user license.