Performance Analysis of Distributed and Scalable Deep Learning

in 20th IEEE/ACM Intl. Symp. on Cluster, Cloud and Internet Computing (CCGrid'20) (2020, May)

With renewed global interest for Artificial Intelligence (AI) methods, the past decade has seen a myriad of new programming models and tools that enable better and faster Machine Learning (ML). More ... [more ▼]

With renewed global interest for Artificial Intelligence (AI) methods, the past decade has seen a myriad of new programming models and tools that enable better and faster Machine Learning (ML). More recently, a subset of ML known as Deep Learning (DL) raised an increased interest due to its inherent ability to tackle efficiently novel cognitive computing applications. DL allows computational models that are composed of multiple processing layers to learn in an automated way representations of data with multiple levels of abstraction, and can deliver higher predictive accuracy when trained on larger data sets. Based on Artificial Neural Networks (ANN), DL is now at the core of state of the art voice recognition systems (which enable easy control over e.g. Internet-of- Things (IoT) smart home appliances for instance), self-driving car engine, online recommendation systems. The ecosystem of DL frameworks is fast evolving, as well as the DL architectures that are shown to perform well on specialized tasks and to exploit GPU accelerators. For this reason, the frequent performance evaluation of the DL ecosystem is re- quired, especially since the advent of novel distributed training frameworks such as Horovod allowing for scalable training across multiple computing resources. In this paper, the scalability evaluation of the reference DL frameworks (Tensorflow, Keras, MXNet, and PyTorch) is performed over up-to-date High Performance Comput- ing (HPC) resources to compare the efficiency of differ- ent implementations across several hardware architectures (CPU and GPU). Experimental results demonstrate that the DistributedDataParallel features in the Pytorch library seem to be the most efficient framework for distributing the training process across many devices, allowing to reach a throughput speedup of 10.11 when using 12 NVidia Tesla V100 GPUs when training Resnet44 on the CIFAR10 dataset. [less ▲]

Automatic Software Tuning of Parallel Programs for Energy-Aware Executions

in Proc. of 13th Intl. Conf. on Parallel Processing and Applied Mathematics (PPAM 2019) (2019, December)

For large scale systems, such as data centers, energy efficiency has proven to be key for reducing capital, operational expenses and environmental impact. Power drainage of a system is closely related to ... [more ▼]

For large scale systems, such as data centers, energy efficiency has proven to be key for reducing capital, operational expenses and environmental impact. Power drainage of a system is closely related to the type and characteristics of workload that the device is running. For this reason, this paper presents an automatic software tuning method for parallel program generation able to adapt and exploit the hardware features available on a target computing system such as an HPC facility or a cloud system in a better way than traditional compiler infrastructures. We propose a search based approach combining both exact methods and approximated heuristics evolving programs in order to find optimized configurations relying on an ever-increasing number of tunable knobs i.e., code transformation and execution options (such as the num- ber of OpenMP threads and/or the CPU frequency settings). The main objective is to outperform the configurations generated by traditional compiling infrastructures for selected KPIs i.e., performance, energy and power usage (for both for the CPU and DRAM), as well as the runtime. First experimental results tied to the local optimization phase of the proposed framework are encouraging, demonstrating between 8% and 41% improvement for all considered metrics on a reference benchmark- ing application (i.e., Linpack). This brings novel perspectives for the global optimization step currently under investigation within the presented framework, with the ambition to pave the way toward automatic tuning of energy-aware applications beyond the performance of the current state-of-the-art compiler infrastructures. [less ▲]

Energy-Performance Optimization for the Cloud

Doctoral thesis (2014)

It's Not a Bug, It'sa Feature: Wait-free Asynchronous Cellular Genetic Algorithm

in Wyrzykowski, Roman; Dongarra, Jack (Eds.) Parallel Processing and Applied Mathematics 10th International Conference, PPAM 2013 Warsaw, Poland, September 8–11, 2013 (2014)

Comparing the Performance and Power Usage of GPU and ARM Clusters for Map-Reduce

in Proc. of the 3rd Intl. Conf. on Cloud and Green Computing (CGC'13) (2013, October)

Solving very large instances of the scheduling of independent tasks problem on the GPU

in Journal of Parallel & Distributed Computing (2013), 73(1), 101-110

A two-phase heuristic for the energy-efficient scheduling of independent tasks on computational grids

in Cluster Computing (2012)

A Multi-Objective GRASP for Energy-aware Scheduling

in International Congress on Computer Science Research (2011, October 28)

Energy-efficient scheduling on milliclusters with performance constraints

in Green Computing and Communications (GreenCom), 2011 IEEE/ACM International Conference on (2011)

Today’s datacenters and large scale enterprise com- puting are power hungry. A lot of research effort is devoted in industry and academy to address this challenging issue. In this context, a new type of ... [more ▼]

Today’s datacenters and large scale enterprise com- puting are power hungry. A lot of research effort is devoted in industry and academy to address this challenging issue. In this context, a new type of enterprise computing platform is being investigated. This computing platform is composed of hundred of millicomputers, each requiring orders of magnitude less power. However, this approach brings challenges that must be met in order to compete with the current practice. This paper addresses two such critical challenges. First, it suggests how to decompose large applications into smaller tasks, better suited to millicomputers. Then, it casts the performance oriented and energy efficient problem into a soft real-time scheduling problem, for which several algorithms are then proposed and evaluated. Sensitivity analysis is used to provide insights into the model, and plan the evaluation of the scheduling algorithms. The contention found in multi-core millicomputing processors is also accounted for. [less ▲]

A Review on Task Performance Prediction in Multi-core Based Systems

in Computer and Information Technology (CIT), 2011 IEEE 11th International Conference on (2011)