![]() Mainassara Chekaraou, Abdoul Wahid ![]() ![]() ![]() E-print/Working paper (2022) The Extended Discrete Element Method (XDEM) is an innovative numerical simulation technique that extends the dynamics of granular materials known as Discrete Element Method (DEM) by additional properties ... [more ▼] The Extended Discrete Element Method (XDEM) is an innovative numerical simulation technique that extends the dynamics of granular materials known as Discrete Element Method (DEM) by additional properties such as the thermodynamic state, stress/strain for each particle. Such DEM simulations used by industries to set up their experimental processes are complexes and heavy in computation time. At each time step, those simulations generate a list of interacting particles and this phase is one of the most computationally expensive parts of a DEM simulation. The Verlet buffer method, initially introduced in Molecular Dynamic (MD) (and also used in DEM), allows keeping the interaction list for many time steps by extending each particle neighbourhood by a certain extension range, and thus broadening the interaction list. The method relies on the temporal coherency of DEM, which guarantees that no particles move erratically from one time step to the next. In the classical approach, all the particles have their neighbourhood extended by the same value which leads to suboptimal performances in simulations where different flow regimes coexist. Additionally, and unlike in MD, there is no comprehensive study analysing the different parameters that affect the performance of the Verlet buffer method in DEM. In this work, we propose a new method for the dynamic update of the neighbour list that depends on the particles individual displacement and define a particle-specific extension range based on the local flow regime. The interaction list is analysed throughout the simulation based on the particle's displacement allowing a flexible update according to the flow regime conditions. We evaluate the influence of the Verlet extension range on the execution time through different test cases and analyse empirically the extension range value giving the best performance. [less ▲] Detailed reference viewed: 278 (87 UL)![]() ; ; Rousset, Alban ![]() Report (2021) The eXtended Discrete Element Method (XDEM) is an extension of the regular Discrete Element Method (DEM) which is a software for simulating the dynamics of granular material. XDEM extends the regular DEM ... [more ▼] The eXtended Discrete Element Method (XDEM) is an extension of the regular Discrete Element Method (DEM) which is a software for simulating the dynamics of granular material. XDEM extends the regular DEM method by adding features where both micro and macroscopic observables can be computed simultaneously by coupling different time and length scales. In this sense XDEM belongs the category of multi-scale/multi-physics applications which can be used in realistic simulations. In this whitepaper, we detail the different optimisations done during the preparatory PRACE project to overcome known bottlenecks in the OpenMP implementation of XDEM. We analysed the Conversion, Dynamic, and the combined Dynamics-Conversion modules with Extrae/Paraver and Intel VTune profiling tools in order to find the most expensive functions. The proposed code modifications improved the performance of XDEM by ~17% for the computational expensive Dynamics-Conversion combined modules (with 48 cores, full node). Our analysis was performed in the Marenostrum 4 (MN4) PRACE infrastructure at Barcelona Supercomputing Center (BSC). [less ▲] Detailed reference viewed: 175 (17 UL)![]() Peters, Bernhard ![]() ![]() ![]() in Scherer, Viktor; Fricker, Neil; Reis, Albino (Eds.) Proceedings of the 12th European Conference on Industrial Furnaces and Boilers (2020, November) Biomass as a renewable energy source continues to grow in popularity to reduce fossil fuel consumption for environmental and economic benefits. In the present contribution, the combustion chamber of a 16 ... [more ▼] Biomass as a renewable energy source continues to grow in popularity to reduce fossil fuel consumption for environmental and economic benefits. In the present contribution, the combustion chamber of a 16 MW geothermal steam super-heater, which is part of the Enel Green Power "Cornia 2" power plant, is being investigated with high-performance computing methods. For this purpose, the extended discrete element method (XDEM) developed at the University of Luxembourg is used in a high-performance computing environment, which includes both the moving wooden bed and the combustion chamber above it. The XDEM simulation platform is based on a hybrid four-way coupling between the Discrete Element Method (DEM) and Computational Fluid Dynamics (CFD). In this approach, particles are treated as discrete elements that are coupled by heat, mass, and momentum transfer to the surrounding gas as a continuous phase. For individual wood particles, besides the equations of motion, the differential conservation equations for mass, heat, and momentum are solved, which describe the thermodynamic state during thermal conversion. The consistency of the numerical results with the actual system performance is discussed in this paper to determine the potentials and limitations of the approach. [less ▲] Detailed reference viewed: 259 (57 UL)![]() Rousset, Alban ![]() ![]() ![]() Scientific Conference (2020, August 31) The purpose of this study is to propose a numerical approach that combines low computational costs through the use of high computing efficiency, allowing the realistic use of the design with a sufficient ... [more ▼] The purpose of this study is to propose a numerical approach that combines low computational costs through the use of high computing efficiency, allowing the realistic use of the design with a sufficient result's accuracy for industrial applications to investigate biomass combustion in a large-scale reciprocating grate. In the present contribution, a Biomass combustion chamber of a 16 MW geothermal steam super-heater, which is part of the Enel Green Power "Cornia 2" power plant,is being investigated with high-performance computing methods. For this purpose, the extended discrete element method (XDEM) developed at the University of Luxembourg is used in an HPC environment, which includes both the moving wooden bed and the combustion chamber above it. The XDEM simulation platform is based on a hybrid four-way coupling between the Discrete Element Method (DEM) and Computational Fluid Dynamics (CFD). In this approach, particles are treated as discrete elements that are coupled by heat, mass, and momentum transfer to the surrounding gas as a continuous phase. For individual wood particles, besides the equations of motion, the differential conservation equations for mass, heat, and momentum are solved, which describe the thermodynamic state during thermal conversion. The grate system has three different moving sections to ensure good mixing of the biomass parts and appropriate residence time. The primary air enters from below the grate and is split into four different zones. Furthermore, a secondary air is injected at high velocity straight over the fuel bed through nozzles. A Flue Gas Recirculation is present and partly injected through two jets along the vertical channel and partly from below the grate. The numerical 3D model presented is based on a multi-phase approach. The biomass particles are taken into consideration via the XDEM Method, while the gaseous phase is described by CFD with OpenFOAM. Thus, the combustion of the particles on the moving beds in the furnace is processed by XDEM through conduction, radiation and conversion along with the interaction with the surrounding gas phase accounted for by CFD. The coupling of CFD-XDEM as an Euler-Lagrange model is used. The fluid phase is a continuous phase handled with an Eulerian approach and each particle is tracked with a Lagrangian approach. Energy, mass and momentum conservation is applied for every single particle and the interaction of particles with each other in the bed and with the surrounding gas phase are taken into account. An individual particle can have a solid, liquid, gas or inert material phases (immobile species) at the same time. The different phases can undergo a series of conversion through various reactions that can be homogeneous, heterogeneous or intrinsic (drying, pyrolysis, gasification and oxidation). Our first results are consistent with actual data obtained from the sampling of the residual solid in the industrial plant. Our model is also able to predict gas flux behaviour inside the furnace, particularly the flue gas recirculation on the combustion process injection. [less ▲] Detailed reference viewed: 143 (32 UL)![]() Besseron, Xavier ![]() ![]() ![]() Scientific Conference (2020, June 18) Multi-physics problems containing discrete particles interacting with fluid phases are widely used industry for example in biomass combustion on a moving grate, particle sedimentation, iron production ... [more ▼] Multi-physics problems containing discrete particles interacting with fluid phases are widely used industry for example in biomass combustion on a moving grate, particle sedimentation, iron production within a blast furnace, and selective laser melting for additive manufacturing. The eXtended Discrete Element Method (XDEM) uses a coupled Eulerian-Lagrangian approach to simulate these complex phenomena, and relies on the Discrete Element Method (DEM) to model the particle phase and Computational Fluid Dynamics (CFD) for the fluid phases, solved respectively with XDEM and OpenFOAM. However, such simulations are very computationally intensive. Additionally, because the DEM particles move within the CFD phases, a 3D volume coupling is required, hence it represents an important amount of data to be exchanged. This volume of communication can have a considerable impact on the performance of the parallel execution. To address this issue, XDEM has proposed a coupling strategy relying on a co-located partitioning. This approach coordinates the domain decomposition of the two independent solvers, XDEM and OpenFOAM, to impose some co-location constraints and reduce the overhead due to the coupling data exchange. This strategy for the parallel coupling of CFD-DEM has been evaluated to perform large scale simulations of debris within a dam break flow. [less ▲] Detailed reference viewed: 185 (15 UL)![]() Mainassara Chekaraou, Abdoul Wahid ![]() ![]() ![]() in 10th IEEE Workshop on Parallel / Distributed Combinatorics and Optimization (2020, June) The Verlet list method is a well-known bookkeeping technique of the interaction list used both in Molecular Dynamic (MD) and Discrete Element Method (DEM). The Verlet buffer technique is an enhancement of ... [more ▼] The Verlet list method is a well-known bookkeeping technique of the interaction list used both in Molecular Dynamic (MD) and Discrete Element Method (DEM). The Verlet buffer technique is an enhancement of the Verlet list that consists of extending the interaction radius of each particle by an extra margin to take into account more particles in the interaction list. The extra margin is based on the local flow regime of each particle to account for the different flow regimes that can coexist in the domain. However, the choice of the near-optimal extra margin (which ensures the best performance) for each particle and the related parameters remains unexplored in DEM unlike in MD. In this study, we demonstrate that the near-optimal extra margin can fairly be characterized by four parameters that describe each particle local flow regime: the particle velocity, the ratio of the containing cell size to particle size, the containing cell solid fraction, and the total number of particles in the system. For this purpose, we model the near-optimal extra margin as a function of these parameters using a quadratic polynomial function. We use the DAKOTA SOFTWARE to carry out the Design and Analysis of Computer Experiments (DACE) and the sampling of the parameters for the simulations. For a given instance of the set of parameters, a global optimization method is considered to find the near-optimal extra margin. The latter is required for the construction of the quadratic polynomial model. The numerous simulations generated by the sampling of the parameter were performed on a High-Performance Computing (HPC) environment granting parallel and concurrent executions. This work provides a better understanding of the Verlet buffer method in DEM simulations by analyzing its performances and behavior in various configurations. The near-optimal extra margin can reasonably be predicted by two out of the four chosen parameters using the quadratic polynomial model. This model has been integrated into XDEM in order to automatically choose the extra margin without any input from the user. Evaluations on real industrial-level test cases show up to a 26% reduction of the execution time. [less ▲] Detailed reference viewed: 118 (23 UL)![]() Mainassara Chekaraou, Abdoul Wahid ![]() ![]() ![]() Scientific Conference (2020, April 27) Detailed reference viewed: 121 (20 UL)![]() Mainassara Chekaraou, Abdoul Wahid ![]() ![]() ![]() Scientific Conference (2019, July 26) The Extended Discrete Element Method (XDEM) is a novel and innovative numerical simulation technique that extends the dynamics of granular materials or particles as described through the classical ... [more ▼] The Extended Discrete Element Method (XDEM) is a novel and innovative numerical simulation technique that extends the dynamics of granular materials or particles as described through the classical discrete element method (DEM) by additional properties such as the thermodynamic state, stress/strain for each particle. Such DEM simulations used by industries to set up their experimental processes are complex and heavy in computation time. Those simulations perform at each time step a collision detection to generate a list of interacting particles that is one of the most expensive computation parts of a DEM simulation. The Verlet buffer method, which was first introduced in Molecular Dynamic (MD) (and is also used in DEM) allows to keep the interaction list for many time step by extending each particle neighborhood by a certain extension range, and thus broadening the interaction list. The method relies mainly on the stability of the DEM, which ensures that no particles move erratically or unpredictably from one time step to the next: this is called temporal coherency. In the classical and current approach, all the particles have their neighborhood extended by the same value which leads to suboptimal performances in simulations where different flow regimes coexist. Additionally, and unlike in MD (which remains very different from DEM on several aspects), there is no comprehensive study analyzing the different parameters that affect the performance of the Verlet buffer method in DEM. In this work, we apply a dynamic neighbor list update method that depends on the particle's individual displacement, and an extension range specific to each particle and based on their local flow regime for the generation of the neighbor list. The update of the interaction list is analyzed throughout the simulation based on the displacement of the particle allowing a flexible update according to the flow regime conditions. We evaluate the influence of the Verlet extension range on the performance of the execution time through different test cases and we empirically analyze and define the extension range value giving the minimum of the global simulation time. [less ▲] Detailed reference viewed: 87 (9 UL)![]() Besseron, Xavier ![]() ![]() Presentation (2019, June 21) Detailed reference viewed: 295 (31 UL)![]() Besseron, Xavier ![]() ![]() ![]() Scientific Conference (2019, June 05) Detailed reference viewed: 108 (14 UL)![]() Mainassara Chekaraou, Abdoul Wahid ![]() ![]() ![]() Poster (2018, September 24) The Extended Discrete Element Method (XDEM) is a novel and innovative numerical simulation technique that extends the dynamics of granular materials or particles as described through the classical ... [more ▼] The Extended Discrete Element Method (XDEM) is a novel and innovative numerical simulation technique that extends the dynamics of granular materials or particles as described through the classical discrete element method (DEM) by additional properties such as the thermodynamic state, stress/strain for each particle. Such DEM simulations used by industries to set up their experimental processes are complexes and heavy in computation time. Therefore, simulations have to be precise, efficient and fast in order to be able to process hundreds of millions of particles. To tackle this issue, such DEM simulations are usually parallelized with MPI. One of the most expensive computation parts of a DEM simulation is the collision detection of particles. It is classically divided into two steps: the broad phase and the narrow phase. The broad phase uses simplified bounding volumes to perform an approximated but fast collision detection. It returns a list of particle pairs that could interact. The narrow phase is applied to the result of the broad phase and returns the exact list of colliding particles. The goal of this research is to apply a Verlet buffer method to (X)DEM simulations regardless of which broad phase algorithm is used. We rely on the fact that such DEM simulations are temporal coherent: the neighborhood only changes slightly from the last time-step to the current time-step. We use the Verlet buffer method to extend the list of pairs returned by the broad phase by stretching the particles bounding volume with an extension range. This allows re-using the result of the broad phase for several time-steps before an update is required once again and thereby its reduce the number of times the broad phase is executed. We have implemented a condition based on particles displacements to ensure the validity of the broad phase: a new one is executed to update the list of colliding particles only when necessary. This guarantees identical results because approximations introduced in the broad phase by our approach are corrected in the narrow phase which is executed at every time-steps anyway. We perform an extensive study to evaluate the influence of the Verlet extension range on the performance of the execution in terms of computation time and memory consumption. We consider different test-cases, partitioners (ORB, Zoltan, METIS, SCOTCH, ...), broad phase algorithms (Link cell, Sweep and prune, ...) and grid configurations (fine, coarse), sequential and parallel (up to 280 cores). While a larger Verlet buffer increases the cost of the broad phase and narrow phase, it also allows skipping a significant number of broad phase execution (> 99 \%). As a consequence, our first results show that this approach can speeds up the total .execution time up to a factor of 5 for sequential executions, and up to a factor of 3 parallel executions on 280 cores while maintaining a reasonable memory consumption. [less ▲] Detailed reference viewed: 196 (33 UL)![]() Mainassara Chekaraou, Abdoul Wahid ![]() ![]() ![]() in Proc. of the 9th Workshop on Applications for Multi-Core Architectures (WAMCA'18), part of 30th Intl. Symp. on Computer Architecture and High Performance Computing (SBAC-PAD 2018) (2018, September) The Extended Discrete Element Method (XDEM) is a novel and innovative numerical simulation technique that ex- tends classical Discrete Element Method (DEM) (which simulates the motion of granular material ... [more ▼] The Extended Discrete Element Method (XDEM) is a novel and innovative numerical simulation technique that ex- tends classical Discrete Element Method (DEM) (which simulates the motion of granular material), by additional properties such as the chemical composition, thermodynamic state, stress/strain for each particle. It has been applied successfully to numerous industries involving the processing of granular materials such as sand, rock, wood or coke [16], [17]. In this context, computational simulation with (X)DEM has become a more and more essential tool for researchers and scientific engineers to set up and explore their experimental processes. However, increasing the size or the accuracy of a model requires the use of High Performance Computing (HPC) platforms over a parallelized implementation to accommodate the growing needs in terms of memory and computation time. In practice, such a parallelization is traditionally obtained using either MPI (distributed memory computing), OpenMP (shared memory computing) or hybrid approaches combining both of them. In this paper, we present the results of our effort to implement an OpenMP version of XDEM allowing hybrid MPI+OpenMP simulations (XDEM being already parallelized with MPI). Far from the basic OpenMP paradigm and recommendations (which simply summarizes by decorating the main computation loops with a set of OpenMP pragma), the OpenMP parallelization of XDEM required a fundamental code re-factoring and careful tuning in order to reach good performance. There are two main reasons for those difficulties. Firstly, XDEM is a legacy code devel- oped for more than 10 years, initially focused on accuracy rather than performance. Secondly, the particles in a DEM simulation are highly dynamic: they can be added, deleted and interaction relations can change at any timestep of the simulation. Thus this article details the multiple layers of optimization applied, such as a deep data structure profiling and reorganization, the usage of fast multithreaded memory allocators and of advanced process/thread-to-core pinning techniques. Experimental results evaluate the benefit of each optimization individually and validate the implementation using a real-world application executed on the HPC platform of the University of Luxembourg. Finally, we present our Hybrid MPI+OpenMP results with a 15%-20% performance gain and how it overcomes scalability limits (by increasing the number of compute cores without dropping of performances) of XDEM-based pure MPI simulations. [less ▲] Detailed reference viewed: 273 (49 UL)![]() Rousset, Alban ![]() ![]() ![]() in AIP Conference Proceedings of 15th International Conference of Numerical Analysis and Applied Mathematics (ICNAAM) (2018, July 10) Collision detection is an ongoing source of research and optimization in many fields including video-games and numerical simulations [6, 7, 8]. The goal of collision detection is to report a geometric ... [more ▼] Collision detection is an ongoing source of research and optimization in many fields including video-games and numerical simulations [6, 7, 8]. The goal of collision detection is to report a geometric contact when it is about to occur or has actually occurred. Unfortunately, detailed and exact collision detection for large amounts of objects represent an immense amount of computations, naively n 2 operation with n being the number of objects [9]. To avoid and reduce these expensive computations, the collision detection is decomposed in two phases as it shown on Figure 1: the Broad-Phase and the Narrow-Phase. In this paper, we focus on Broad-Phase algorithm in a large dynamic three-dimensional environment. We studied two kinds of Broad-Phase algorithms: spatial partitioning and spatial sorting. Spatial partitioning techniques operate by dividing space into a number of regions that can be quickly tested against each object. Two types of spatial partitioning will be considered: grids and trees. The grid-based algorithms consist of a spatial partitioning processing by dividing space into regions and testing if objects overlap the same region of space. And this reduces the number of pairwise to test. The tree-based algorithms use a tree structure where each node spans a particular space area. This reduces the pairwise checking cost because only tree leaves are checked. The spatial sorting based algorithm consists of a sorted spatial ordering of objects. Axis-Aligned Bounding Boxes (AABBs) are projected onto x, y and z axes and put into sorted lists. By sorting projection onto axes, two objects collide if and only if they collide on the three axes. This axis sorting reduces the number of pairwise to tested by reducing the number of tests to perform to only pairs which collide on at least one axis. For this study, ten different Broad-Phase collision detection algorithms or framework have been considered. The Bullet [6], CGAL [10, 11] frameworks have been used. Concerning the implemented algorithms most of them come from papers or given implementation. [less ▲] Detailed reference viewed: 363 (63 UL)![]() Pozzetti, Gabriele ![]() ![]() ![]() in AIP Conference Proceedings of 15th International Conference of Numerical Analysis and Applied Mathematics (ICNAAM) (2018, July 10) In this work, a strategy for the parallelization of a two-way CFD-DEM coupling is investigated. It consists on adopting balanced overlapping partitions for the CFD and the DEM domains, that aims to reduce ... [more ▼] In this work, a strategy for the parallelization of a two-way CFD-DEM coupling is investigated. It consists on adopting balanced overlapping partitions for the CFD and the DEM domains, that aims to reduce the memory consumption and inter-process communication between CFD and DEM. Two benchmarks are proposed to assess the consistency and scalability of this approach, coupled execution on 252 cores shows that less than 1\% of time is used to perform inter-physics data exchange. [less ▲] Detailed reference viewed: 267 (66 UL) |
||