OpenMP optimisation of the eXtended Discrete Element Method (XDEM)

Report (2021)

The eXtended Discrete Element Method (XDEM) is an extension of the regular Discrete Element Method (DEM) which is a software for simulating the dynamics of granular material. XDEM extends the regular DEM ... [more ▼]

The eXtended Discrete Element Method (XDEM) is an extension of the regular Discrete Element Method (DEM) which is a software for simulating the dynamics of granular material. XDEM extends the regular DEM method by adding features where both micro and macroscopic observables can be computed simultaneously by coupling different time and length scales. In this sense XDEM belongs the category of multi-scale/multi-physics applications which can be used in realistic simulations. In this whitepaper, we detail the different optimisations done during the preparatory PRACE project to overcome known bottlenecks in the OpenMP implementation of XDEM. We analysed the Conversion, Dynamic, and the combined Dynamics-Conversion modules with Extrae/Paraver and Intel VTune profiling tools in order to find the most expensive functions. The proposed code modifications improved the performance of XDEM by ~17% for the computational expensive Dynamics-Conversion combined modules (with 48 cores, full node). Our analysis was performed in the Marenostrum 4 (MN4) PRACE infrastructure at Barcelona Supercomputing Center (BSC). [less ▲]

Numerical Analysis of Interaction between a Reacting Fluid and a Moving Bed with Spatially and Temporally Fluctuating Porosity

Scientific Conference (2020, August 31)

The purpose of this study is to propose a numerical approach that combines low computational costs through the use of high computing efficiency, allowing the realistic use of the design with a sufficient ... [more ▼]

The purpose of this study is to propose a numerical approach that combines low computational costs through the use of high computing efficiency, allowing the realistic use of the design with a sufficient result's accuracy for industrial applications to investigate biomass combustion in a large-scale reciprocating grate. In the present contribution, a Biomass combustion chamber of a 16 MW geothermal steam super-heater, which is part of the Enel Green Power "Cornia 2" power plant,is being investigated with high-performance computing methods. For this purpose, the extended discrete element method (XDEM) developed at the University of Luxembourg is used in an HPC environment, which includes both the moving wooden bed and the combustion chamber above it. The XDEM simulation platform is based on a hybrid four-way coupling between the Discrete Element Method (DEM) and Computational Fluid Dynamics (CFD). In this approach, particles are treated as discrete elements that are coupled by heat, mass, and momentum transfer to the surrounding gas as a continuous phase. For individual wood particles, besides the equations of motion, the differential conservation equations for mass, heat, and momentum are solved, which describe the thermodynamic state during thermal conversion. The grate system has three different moving sections to ensure good mixing of the biomass parts and appropriate residence time. The primary air enters from below the grate and is split into four different zones. Furthermore, a secondary air is injected at high velocity straight over the fuel bed through nozzles. A Flue Gas Recirculation is present and partly injected through two jets along the vertical channel and partly from below the grate. The numerical 3D model presented is based on a multi-phase approach. The biomass particles are taken into consideration via the XDEM Method, while the gaseous phase is described by CFD with OpenFOAM. Thus, the combustion of the particles on the moving beds in the furnace is processed by XDEM through conduction, radiation and conversion along with the interaction with the surrounding gas phase accounted for by CFD. The coupling of CFD-XDEM as an Euler-Lagrange model is used. The fluid phase is a continuous phase handled with an Eulerian approach and each particle is tracked with a Lagrangian approach. Energy, mass and momentum conservation is applied for every single particle and the interaction of particles with each other in the bed and with the surrounding gas phase are taken into account. An individual particle can have a solid, liquid, gas or inert material phases (immobile species) at the same time. The different phases can undergo a series of conversion through various reactions that can be homogeneous, heterogeneous or intrinsic (drying, pyrolysis, gasification and oxidation). Our first results are consistent with actual data obtained from the sampling of the residual solid in the industrial plant. Our model is also able to predict gas flux behaviour inside the furnace, particularly the flue gas recirculation on the combustion process injection. [less ▲]

Predicting near-optimal skin distance in Verlet buffer approach for Discrete Element Method

in 10th IEEE Workshop on Parallel / Distributed Combinatorics and Optimization (2020, June)

The Verlet list method is a well-known bookkeeping technique of the interaction list used both in Molecular Dynamic (MD) and Discrete Element Method (DEM). The Verlet buffer technique is an enhancement of ... [more ▼]

The Verlet list method is a well-known bookkeeping technique of the interaction list used both in Molecular Dynamic (MD) and Discrete Element Method (DEM). The Verlet buffer technique is an enhancement of the Verlet list that consists of extending the interaction radius of each particle by an extra margin to take into account more particles in the interaction list. The extra margin is based on the local flow regime of each particle to account for the different flow regimes that can coexist in the domain. However, the choice of the near-optimal extra margin (which ensures the best performance) for each particle and the related parameters remains unexplored in DEM unlike in MD. In this study, we demonstrate that the near-optimal extra margin can fairly be characterized by four parameters that describe each particle local flow regime: the particle velocity, the ratio of the containing cell size to particle size, the containing cell solid fraction, and the total number of particles in the system. For this purpose, we model the near-optimal extra margin as a function of these parameters using a quadratic polynomial function. We use the DAKOTA SOFTWARE to carry out the Design and Analysis of Computer Experiments (DACE) and the sampling of the parameters for the simulations. For a given instance of the set of parameters, a global optimization method is considered to find the near-optimal extra margin. The latter is required for the construction of the quadratic polynomial model. The numerous simulations generated by the sampling of the parameter were performed on a High-Performance Computing (HPC) environment granting parallel and concurrent executions. This work provides a better understanding of the Verlet buffer method in DEM simulations by analyzing its performances and behavior in various configurations. The near-optimal extra margin can reasonably be predicted by two out of the four chosen parameters using the quadratic polynomial model. This model has been integrated into XDEM in order to automatically choose the extra margin without any input from the user. Evaluations on real industrial-level test cases show up to a 26% reduction of the execution time. [less ▲]

Verlet buffer for broad-phase interaction detection in Discrete Element Method

Scientific Conference (2019, July 26)

The Extended Discrete Element Method (XDEM) is a novel and innovative numerical simulation technique that extends the dynamics of granular materials or particles as described through the classical ... [more ▼]

The Extended Discrete Element Method (XDEM) is a novel and innovative numerical simulation technique that extends the dynamics of granular materials or particles as described through the classical discrete element method (DEM) by additional properties such as the thermodynamic state, stress/strain for each particle. Such DEM simulations used by industries to set up their experimental processes are complex and heavy in computation time. Those simulations perform at each time step a collision detection to generate a list of interacting particles that is one of the most expensive computation parts of a DEM simulation. The Verlet buffer method, which was first introduced in Molecular Dynamic (MD) (and is also used in DEM) allows to keep the interaction list for many time step by extending each particle neighborhood by a certain extension range, and thus broadening the interaction list. The method relies mainly on the stability of the DEM, which ensures that no particles move erratically or unpredictably from one time step to the next: this is called temporal coherency. In the classical and current approach, all the particles have their neighborhood extended by the same value which leads to suboptimal performances in simulations where different flow regimes coexist. Additionally, and unlike in MD (which remains very different from DEM on several aspects), there is no comprehensive study analyzing the different parameters that affect the performance of the Verlet buffer method in DEM. In this work, we apply a dynamic neighbor list update method that depends on the particle's individual displacement, and an extension range specific to each particle and based on their local flow regime for the generation of the neighbor list. The update of the interaction list is analyzed throughout the simulation based on the displacement of the particle allowing a flexible update according to the flow regime conditions. We evaluate the influence of the Verlet extension range on the performance of the execution time through different test cases and we empirically analyze and define the extension range value giving the minimum of the global simulation time. [less ▲]

Co-located Partitioning Strategy and Dual-grid Multiscale Approach for Parallel Coupling of CFD-DEM Simulations

Scientific Conference (2019, June 05)

Hybrid MPI+OpenMP Implementation of eXtended Discrete Element Method

in Proc. of the 9th Workshop on Applications for Multi-Core Architectures (WAMCA'18), part of 30th Intl. Symp. on Computer Architecture and High Performance Computing (SBAC-PAD 2018) (2018, September)

The Extended Discrete Element Method (XDEM) is a novel and innovative numerical simulation technique that ex- tends classical Discrete Element Method (DEM) (which simulates the motion of granular material ... [more ▼]

The Extended Discrete Element Method (XDEM) is a novel and innovative numerical simulation technique that ex- tends classical Discrete Element Method (DEM) (which simulates the motion of granular material), by additional properties such as the chemical composition, thermodynamic state, stress/strain for each particle. It has been applied successfully to numerous industries involving the processing of granular materials such as sand, rock, wood or coke [16], [17]. In this context, computational simulation with (X)DEM has become a more and more essential tool for researchers and scientific engineers to set up and explore their experimental processes. However, increasing the size or the accuracy of a model requires the use of High Performance Computing (HPC) platforms over a parallelized implementation to accommodate the growing needs in terms of memory and computation time. In practice, such a parallelization is traditionally obtained using either MPI (distributed memory computing), OpenMP (shared memory computing) or hybrid approaches combining both of them. In this paper, we present the results of our effort to implement an OpenMP version of XDEM allowing hybrid MPI+OpenMP simulations (XDEM being already parallelized with MPI). Far from the basic OpenMP paradigm and recommendations (which simply summarizes by decorating the main computation loops with a set of OpenMP pragma), the OpenMP parallelization of XDEM required a fundamental code re-factoring and careful tuning in order to reach good performance. There are two main reasons for those difficulties. Firstly, XDEM is a legacy code devel- oped for more than 10 years, initially focused on accuracy rather than performance. Secondly, the particles in a DEM simulation are highly dynamic: they can be added, deleted and interaction relations can change at any timestep of the simulation. Thus this article details the multiple layers of optimization applied, such as a deep data structure profiling and reorganization, the usage of fast multithreaded memory allocators and of advanced process/thread-to-core pinning techniques. Experimental results evaluate the benefit of each optimization individually and validate the implementation using a real-world application executed on the HPC platform of the University of Luxembourg. Finally, we present our Hybrid MPI+OpenMP results with a 15%-20% performance gain and how it overcomes scalability limits (by increasing the number of compute cores without dropping of performances) of XDEM-based pure MPI simulations. [less ▲]

On the performance of an overlapping-domain parallelization strategy for Eulerian-Lagrangian Multiphysics software

in AIP Conference Proceedings of 15th International Conference of Numerical Analysis and Applied Mathematics (ICNAAM) (2018, July 10)

In this work, a strategy for the parallelization of a two-way CFD-DEM coupling is investigated. It consists on adopting balanced overlapping partitions for the CFD and the DEM domains, that aims to reduce ... [more ▼]

In this work, a strategy for the parallelization of a two-way CFD-DEM coupling is investigated. It consists on adopting balanced overlapping partitions for the CFD and the DEM domains, that aims to reduce the memory consumption and inter-process communication between CFD and DEM. Two benchmarks are proposed to assess the consistency and scalability of this approach, coupled execution on 252 cores shows that less than 1\% of time is used to perform inter-physics data exchange. [less ▲]