Reference : Performance Analysis of Distributed and Scalable Deep Learning
Scientific congresses, symposiums and conference proceedings : Paper published in a book
Engineering, computing & technology : Computer science
Computational Sciences
http://hdl.handle.net/10993/43342
Performance Analysis of Distributed and Scalable Deep Learning
English
Mahon, S. [> >]
Varrette, Sébastien mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > Computer Science and Communications Research Unit (CSC)]
Plugaru, Valentin mailto [University of Luxembourg > Faculty of Science, Technology and Communication (FSTC) > Computer Science and Communications Research Unit (CSC)]
Pinel, Frederic mailto [University of Luxembourg > Faculty of Science, Technology and Communication (FSTC) > Computer Science and Communications Research Unit (CSC)]
Bouvry, Pascal mailto [University of Luxembourg > Faculty of Science, Technology and Communication (FSTC) > Computer Science and Communications Research Unit (CSC)]
May-2020
20th IEEE/ACM Intl. Symp. on Cluster, Cloud and Internet Computing (CCGrid'20)
IEEE/ACM
760--766
Yes
International
978-1-7281-6095-5
Melbourne
Australia
20th IEEE/ACM Intl. Symp. on Cluster, Cloud and Internet Computing (CCGrid'20)
May 11-14, 2020
Melbourne
Australia
[en] Deep Learning ; Performance Evaluation ; GPU
[en] With renewed global interest for Artificial Intelligence (AI) methods, the past decade has seen a myriad of new programming models and tools that enable better and faster Machine Learning (ML). More recently, a subset of ML known as Deep Learning (DL) raised an increased interest due to its inherent ability to tackle efficiently novel cognitive computing applications. DL allows computational models that are composed of multiple processing layers to learn in an automated way representations of data with multiple levels of abstraction, and can deliver higher predictive accuracy when trained on larger data sets. Based on Artificial Neural Networks (ANN), DL is now at the core of state of the art voice recognition systems (which enable easy control over e.g. Internet-of- Things (IoT) smart home appliances for instance), self-driving car engine, online recommendation systems. The ecosystem of DL frameworks is fast evolving, as well as the DL architectures that are shown to perform well on specialized tasks and to exploit GPU accelerators. For this reason, the frequent performance evaluation of the DL ecosystem is re- quired, especially since the advent of novel distributed training frameworks such as Horovod allowing for scalable training across multiple computing resources.
In this paper, the scalability evaluation of the reference DL frameworks (Tensorflow, Keras, MXNet, and PyTorch) is performed over up-to-date High Performance Comput- ing (HPC) resources to compare the efficiency of differ- ent implementations across several hardware architectures (CPU and GPU). Experimental results demonstrate that the DistributedDataParallel features in the Pytorch library seem to be the most efficient framework for distributing the training process across many devices, allowing to reach a throughput speedup of 10.11 when using 12 NVidia Tesla V100 GPUs when training Resnet44 on the CIFAR10 dataset.
University of Luxembourg: High Performance Computing - ULHPC
http://hdl.handle.net/10993/43342

File(s) associated to this reference

Fulltext file(s):

FileCommentaryVersionSizeAccess
Limited access
Performance-Analysis-of-Distributed-and-Scalable-Deep-Learning_CCGrid20_609500a760.pdfPublisher postprint501.88 kBRequest a copy

Bookmark and Share SFX Query

All documents in ORBilu are protected by a user license.