Reference : AIR: A Light-Weight Yet High-Performance Dataflow Engine based on Asynchronous Iterat...
Scientific congresses, symposiums and conference proceedings : Paper published in a book
Engineering, computing & technology : Computer science
http://hdl.handle.net/10993/45326
AIR: A Light-Weight Yet High-Performance Dataflow Engine based on Asynchronous Iterative Routing
English
Ellampallil Venugopal, Vinu mailto [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS) >]
Theobald, Martin mailto [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS) >]
Chaychi, Samira mailto [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS) >]
Tawakuli, Amal mailto [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS) >]
1-Sep-2020
AIR: A Light-Weight Yet High-Performance Dataflow Engine based on Asynchronous Iterative Routing
IEEE
51-58
Yes
Yes
International
978-1-7281-9924-5
32nd International Symposium on Computer Architecture and High Performance Computing
from 08-09-2020 to 11-09-2020
[en] Stream data processing ; Big Data ; sustainable-throughput
[en] Distributed Stream Processing Engines (DSPEs) are
currently among the most emerging topics in data management,
with applications ranging from real-time event monitoring to
processing complex dataflow programs and big data analytics.
In this paper, we describe the architecture of our AIR engine,
which is designed from scratch in C++ using the Message Passing
Interface (MPI), pthreads for multithreading, and is directly
deployed on top of a common HPC workload manager such as
SLURM. AIR implements a light-weight, dynamic sharding protocol (referred to as “Asynchronous Iterative Routing”), which
facilitates a direct and asynchronous communication among all
worker nodes and thereby completely avoids any additional
communication overhead with a dedicated master node. With its
unique design, AIR fills the gap between the prevalent scale-out
(but Java-based) architectures like Apache Spark and Flink, on
one hand, and recent scale-up (and C++ based) prototypes such
as StreamBox and PiCo, on the other hand. Our experiments
over various benchmark settings confirm that AIR performs as
good as the best scale-up SPEs on a single-node setup, while
it outperforms existing scale-out DSPEs in terms of processing
latency and sustainable throughput by a factor of up to 15 in a
distributed setting.
University of Luxembourg - UL
http://hdl.handle.net/10993/45326
https://conferences.computer.org/sbacpad/pdfs/SBAC-PAD2020-2fQ2vSYuhExkpkZ9tActSv/992400a051/992400a051.pdf

File(s) associated to this reference

Fulltext file(s):

FileCommentaryVersionSizeAccess
Limited access
sbac_cr.pdfAuthor postprint573.75 kBRequest a copy

Bookmark and Share SFX Query

All documents in ORBilu are protected by a user license.