Reference : Aggregating and Consolidating two High Performant Network Topologies: The ULHPC Experience
Scientific congresses, symposiums and conference proceedings : Paper published in a book
Engineering, computing & technology : Computer science
Computational Sciences
http://hdl.handle.net/10993/51828
Aggregating and Consolidating two High Performant Network Topologies: The ULHPC Experience
English
Varrette, Sébastien mailto [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)]
Cartiaux, Hyacinthe mailto [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)]
Valette, Teddy mailto [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)]
Olloh, Abatcha mailto [University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)]
Jul-2022
ACM Practice and Experience in Advanced Research Computing (PEARC'22)
Association for Computing Machinery (ACM)
Yes
No
International
Boston
USA
Practice and Experience in Advanced Research Computing (PEARC ’22),
July 8-14, 2022
Boston
USA
[en] HPC Management ; Network ; Performance Evaluation
[en] High Performance Computing (HPC) encompasses advanced computation over parallel processing. The execution time of a given simulation depends upon many factors, such as the number of CPU/GPU cores, their utilisation factor and, of course, the inter- connect performance, efficiency, and scalability. In practice, this last component and the associated topology remains the most significant differentiators between HPC systems and lesser perfor- mant systems. The University of Luxembourg operates since 2007 a large academic HPC facility which remains one of the reference implementation within the country and offers a cutting-edge re- search infrastructure to Luxembourg public research. The main high-bandwidth low-latency network of the operated facility relies on the dominant interconnect technology in the HPC market i.e., Infiniband (IB) over a Fat-tree topology. It is complemented by an Ethernet-based network defined for management tasks, external access and interactions with user’s applications that do not support Infiniband natively. The recent acquisition of a new cutting-edge supercomputer Aion which was federated with the previous flag- ship cluster Iris was the occasion to aggregate and consolidate the two types of networks. This article depicts the architecture and the solutions designed to expand and consolidate the existing networks beyond their seminal capacity limits while keeping at best their Bisection bandwidth. At the IB level, and despite moving from a non-blocking configuration, the proposed approach defines a blocking topology maintaining the previous Fat-Tree height. The leaf connection capacity is more than tripled (moving from 216 to 672 end-points) while exhibiting very marginal penalties, i.e. less than 3% (resp. 0.3%) Read (resp. Write) bandwidth degradation against reference parallel I/O benchmarks, or a stable and sustain- able point-to-point bandwidth efficiency among all possible pairs of nodes (measured above 95.45% for bi-directional streams). With regards the Ethernet network, a novel 2-layer topology aiming for improving the availability, maintainability and scalability of the interconnect is described. It was deployed together with consistent network VLANs and subnets enforcing strict security policies via ACLs defined on the layer 3, offering isolated and secure net- work environments. The implemented approaches are applicable to a broad range of HPC infrastructures and thus may help other HPC centres to consolidate their own interconnect stacks when designing or expanding their network infrastructures.
University of Luxembourg: High Performance Computing - ULHPC
http://hdl.handle.net/10993/51828
10.1145/3491418.3535159
https://pearc.acm.org/pearc22/

File(s) associated to this reference

Fulltext file(s):

FileCommentaryVersionSizeAccess
Open access
final_pearc22-64.pdfAuthor preprint1.25 MBView/Open

Additional material(s):

File Commentary Size Access
Open access
slides_pearc22.pdfSlides conference2.55 MBView/Open

Bookmark and Share SFX Query

All documents in ORBilu are protected by a user license.