HTC Library Configuration in YAML

This module is the second in a sequence that will form the overall capabilities of the library (see E-CAM High Throughput Computing Library for the previous module). This module deals with creating a more comprehensive configuration format for the Dask-Jobqueue Python library in YAML format.

Purpose of Module

The goal is to allow numerous cluster instances (which is a place where tasks are executed) to be defined more broadly and cover all possibilities that the queueing system might offer as well as in configurations that are required to execute MPI/OpenMP tasks.

The implementation is generic but the specific example provided is for SLURM on the JURECA system.

Background Information

This module builds upon the work described in E-CAM High Throughput Computing Library and the mechanism already provided by the Dask configuration and the Dask-Jobqueue configuration

Building and Testing

The library is a Python module and can be installed with

python install

More details about how to install a Python package can be found at, for example, Install Python packages on the research computing systems at IU

To run the tests for the decorators within the library, you need the pytest Python package. You can run all the relevant tests from the jobqueue_features directory with

pytest tests/

Source Code

The latest version of the library is available on the jobqueue_features GitHub repository

The code that was originally created specifically for this module can be seen in the HTC/Yaml Merge Request which can be found in the original private repository of the code.