ATIS: Automated Testing of Installed Software. Or so far, How to validate MPI stacks of an HPC cluster?

Automatic Testing of Installed Software is a testing framework to validate the various flavors of software installed on an HPC site. It is composed of a set of unit tests, a runtime and a result-gathering dashboard. These tests are user-oriented as they assess the basic features that a general user expect to work on an HPC platform.

Currently, it only focuses on generic MPI functionality as it is one complex and critical component of an HPC platform, but it will be extended to compilers, libraries and performance validation and regression in the future.

HPC centers tend to provide a wide choice a software. Different users requires different software, but also different versions of the same software. Combined with the different compilers, MPI stacks, library dependencies, there is an explosion of software flavors installed on an HPC site.

Tools already exist to help managing this large variety of software. Users can choose their software through the software list using the 'module' system. Administrators can perform automatic compilation and installation of software using EasyBuild. Additionally, software also require some customizations on some HPC sites. Thus, software flavors need to be validated after installation to check they're working as expected by the users.

We developed and provide a set of unit tests together with a runtime and result-gathering framework to perform a such Automatic Testing of Installed Software. These tests take the side of the users in order to test any basic feature that a general user expect to work on an HPC platform. So far, the proposed tests only focus on generic MPI functionality as it is one complex and critical component of an HPC platform. The unit tests include, for example, compilation with mpicc and distributed execution with mpiexec. It has been applied successfully on the HPC platforms of the University of Luxembourg to assess builds of OpenMPI, MPICH, MVAPICH2 and IntelMPI generated with EasyBuild.

In the future, we consider extending our unit tests to validate more components like compilers, libraries, toolchains and even applications. Another future direction is to consider performance validation and regression.