Cubble: Hip implementation¶
The mechanics of foams exhibit dynamical behavior familiar from other jammed materials, such as granular matter. These are so called yield stress fluids i.e. their flow requires external stress that exceeds a paricular value, called the yield stress. On the other hand, foams differ from other materials by their internal structure development, coarsening, where while the gas concentration of the foam remains constant, the gas diffuses between the bubbles. The larger bubbles grow at the expense of the smaller ones.
This module provides a recepie and makefile for converting the c++/CUDA implementation of a foam coarsening simulator (https://journals.aps.org/pre/abstract/10.1103/PhysRevE.98.012607) to instead use HIP to allow it to be run on both AMD and Nvidia GPUs.
Purpose of Module¶
The goal was set to be able to run simulations involving up to one million bubbles reaching a scaling state in systems with non-periodic boundary conditions in three dimensions, an undoable task for even the most efficient single core CPU implementation. The Cubble code demonstrates the power of efficiently used GPU code, and provides a model implementation strategy for mesoscale DEM simulators.
Background Information¶
The converted code runs on single MI50 and Radeon VII GPUs.
Building and Testing¶
Converting the Cubble code to use HIP instead of Cuda enables it to run on AMD GPUs, in addition to Nvidia GPUs. The conversion process is easy.
First the source code needs to be converted from Cuda to HIP, this is done
by converting everything in the src directory using the hipify-perl
script. All .cu
files are changed to cpp files and .cuh
changed
to .h
, we then need to update any include statements to reflect this.
When converting Util.h
the converter will emit the following warnings
warning: old/Util.h:#25 : cubble::cudaCallAndLog((call), #call, __FILE__, __LINE__)
warning: old/Util.h:#27 : cubble::cudaCallAndThrow((call), #call, __FILE__, __LINE__)
warning: old/Util.h:#125 : inline bool cudaCallAndLog(hipError_t result, const char *callStr,
warning: old/Util.h:#137 : inline void cudaCallAndThrow(hipError_t result, const char *callStr,
This is due to the cubble program using the same naming scheme as cuda for some functions, i.e the name is cudaSomething and the converter is warning that it was uable to convert them, however in this case these are not actual cuda calls and it should not covert these, so it is safe to ignore the warnings.
The cubble code also uses the NVTX library for better profiling, this
library does not work on AMD hardware and while comparable functionality
exists this will not be automatically converted. The Cubble program
will work without NVTX so in this case we just remove the inclusion
of nvToolsExt.h
and not enable profiling when compiling the
Cubble program.
On some systems we may need to add -DENABLE_HIP_PROFILE=0
to the
compilation flags to suppress errors about some profiling headers not
being found.
After this we should have a version of the code that can be compiled
with the included makefile
. Unfortunately this version will not run
with the current version of HIP, tested with version 3.1. The program
will fail with not finding symbols in hipGetSymbolAddress
calls or
complaining it does not have device functions for certain calls. The
cubble code places the majority of its code in the cubble namespace,
unfortunately currently the HIP compiler struggles with device symbols
and functions being in a namespace. The easiest solution in this case
is to move all the code out of the cubble namespace, this will give
you a code that will run on AMD hardware. It is likely this will
improve in the future and this step will no longer be needed.
Source Code¶
The source code used as a base for the conversion is freely available
for download in Cubble sources <https://github.com/KJLankinen/cubble>.
In addition the modified makefile
is included in this repository:
makefile