Cubble: Hip implementation

The mechanics of foams exhibit dynamical behavior familiar from other jammed materials, such as granular matter. These are so called yield stress fluids i.e. their flow requires external stress that exceeds a paricular value, called the yield stress. On the other hand, foams differ from other materials by their internal structure development, coarsening, where while the gas concentration of the foam remains constant, the gas diffuses between the bubbles. The larger bubbles grow at the expense of the smaller ones.

This module provides a recepie and makefile for converting the c++/CUDA implementation of a foam coarsening simulator (https://journals.aps.org/pre/abstract/10.1103/PhysRevE.98.012607) to instead use HIP to allow it to be run on both AMD and Nvidia GPUs.

Purpose of Module

The goal was set to be able to run simulations involving up to one million bubbles reaching a scaling state in systems with non-periodic boundary conditions in three dimensions, an undoable task for even the most efficient single core CPU implementation. The Cubble code demonstrates the power of efficiently used GPU code, and provides a model implementation strategy for mesoscale DEM simulators.

Background Information

The converted code runs on single MI50 and Radeon VII GPUs.

Building and Testing

Converting the Cubble code to use HIP instead of Cuda enables it to run on AMD GPUs, in addition to Nvidia GPUs. The conversion process is easy.

First the source code needs to be converted from Cuda to HIP, this is done by converting everything in the src directory using the hipify-perl script. All .cu files are changed to cpp files and .cuh changed to .h, we then need to update any include statements to reflect this.

When converting Util.h the converter will emit the following warnings

warning: old/Util.h:#25 :   cubble::cudaCallAndLog((call), #call, __FILE__, __LINE__)
warning: old/Util.h:#27 :   cubble::cudaCallAndThrow((call), #call, __FILE__, __LINE__)
warning: old/Util.h:#125 : inline bool cudaCallAndLog(hipError_t result, const char *callStr,
warning: old/Util.h:#137 : inline void cudaCallAndThrow(hipError_t result, const char *callStr,

This is due to the cubble program using the same naming scheme as cuda for some functions, i.e the name is cudaSomething and the converter is warning that it was uable to convert them, however in this case these are not actual cuda calls and it should not covert these, so it is safe to ignore the warnings.

The cubble code also uses the NVTX library for better profiling, this library does not work on AMD hardware and while comparable functionality exists this will not be automatically converted. The Cubble program will work without NVTX so in this case we just remove the inclusion of nvToolsExt.h and not enable profiling when compiling the Cubble program.

On some systems we may need to add -DENABLE_HIP_PROFILE=0 to the compilation flags to suppress errors about some profiling headers not being found.

After this we should have a version of the code that can be compiled with the included makefile. Unfortunately this version will not run with the current version of HIP, tested with version 3.1. The program will fail with not finding symbols in hipGetSymbolAddress calls or complaining it does not have device functions for certain calls. The cubble code places the majority of its code in the cubble namespace, unfortunately currently the HIP compiler struggles with device symbols and functions being in a namespace. The easiest solution in this case is to move all the code out of the cubble namespace, this will give you a code that will run on AMD hardware. It is likely this will improve in the future and this step will no longer be needed.

Source Code

The source code used as a base for the conversion is freely available for download in Cubble sources <https://github.com/KJLankinen/cubble>. In addition the modified makefile is included in this repository: makefile