ParaDiS with precipitates optimized for AMD Zen2

Discrete dislocation dynamics (DDD) simulations usually treat with “pure” crystals and dislocations in them. An extension of the ParaDIS DDD code (LLNL, that includes dislocation/precipitate interactions has been developed (E-CAM module: ParaDiS with precipitates).

This module provides a guide for optimal porting of the ParaDiS with precipitates to the AMD Rome CPUs, in preparation for the Mahti supercomputer service at CSC, Finland. Mahti is an Atos BullSequana XH2000 system consisting of 1404 nodes each with two 64-core AMD Zen2 CPUs (AMD EPYC 7H42, 2.6GHz). Since Mahti is not ready for general access at this moment, the module was prepared based on a single testing node which has 2 AMD EPYC 7742 @2.25GHz (128 cores in total).

By choosing a suitable compiler and compiler optimization flags, the application works more efficiently on the target platform. On the testing node, Intel compilers with either AVX or AVX2 vector sets gives the best performance for ParaDiS with precipitates. Alternatively, GCC compilers with AVX2 vector support is competitive with the Intel compilers.

Purpose of Module

This module helps to run simulations of the ParaDiS with precipitates more efficiently. By using a suitable set of optimization flags for compilers, especially the one determining vectorization type, the best library routines can be chosen.

Background Information

The module is based on the ParaDiS ( extension ParaDiS with precipitates.

Building and Testing

Build instructions for ParaDiS with precipitates are provided with the extension.

Different compilers and compiler options were tested to find the most optimal ones for the Zen2 architecture. Figure 1 (below) shows a comparison of normalized running times between different vectorization extensions and compilers. On the testing platform, Intel compilers with either AVX or AVX2 helps the application to achieve good performance. Alternatively, GCC compilers with AVX2 can be used to obtain the same performance as the Intel ones.

Table 1 presents a comparison of different optimization flags for the Intel and GCC compilers. For the Intel compilers, the optimal performance is reached with the compiler options: -O3 -mavx2 or -O3 -mavx. For the GCC compilers, -O2 -march=znver2 -pipe -fomit-frame-pointer -ftree-vectorize compiler options help the application to achieve good performance.


Fig. 6 Figure 1: Comparison of normalized times between different compilers and vectorization extensions (smaller is better)

Table 1: Comparison between different optimization flag options

Compilers Flags Time (s)
Intel -O2 -axCORE-AVX2 328
  -O2 -axHASWELL 360
  -O2 -mavx2 309
  -O3 -mavx2 295
  -O3 -mavx 298
  -Ofast -mavx2 301
  -O3 -mavx2 -funroll-all-loops 317
GCC -O2 -march=znver1 -pipe -fomit-frame-pointer -ftree-vectorize 352
  -O2 -march=znver2 -pipe -fomit-frame-pointer -ftree-vectorize 296
  -O3 -march=znver2 382
  -O2 -march=haswell -pipe -fomit-frame-pointer -ftree-vectorize 352

* The input case in these tests are different to the ones at ParaDiS with precipitates optimized for Puhti .

In the ParaDiS with precipitates optimized to HPC environment, it’s written that using multiple threads through a hybrid OpenMP and MPI model speeds up the calculation up to a factor of 1.5, especially for large-scale simulations. However, this combination did not give an advantage of performance on the Zen2 testing machine. Thus, using a single thread for each MPI process is recommended.

Source Code

Source code modifications for the extension ParaDiS with precipitates are available here: