HTC Library Configuration in YAML¶
This module is the second in a sequence that will form the overall capabilities of the library (see E-CAM High Throughput Computing Library for the previous module). This module deals with creating a more comprehensive configuration format for the Dask-Jobqueue Python library in YAML format.
Purpose of Module¶
The goal is to allow numerous cluster instances (which is a place where tasks are executed) to be defined more
broadly and cover all possibilities that the queueing system might offer as well as in configurations that are required
to execute MPI/OpenMP tasks.
The implementation is generic but the specific example provided is for SLURM on the JURECA system.
Background Information¶
This module builds upon the work described in E-CAM High Throughput Computing Library and the mechanism already provided by the Dask configuration and the Dask-Jobqueue configuration
Building and Testing¶
The library is a Python module and can be installed with
python setup.py install
More details about how to install a Python package can be found at, for example, Install Python packages on the research computing systems at IU
To run the tests for the decorators within the library, you need the pytest Python package. You can run all the
relevant tests from the jobqueue_features directory with
pytest tests/test_cluster.py
Source Code¶
The latest version of the library is available on the jobqueue_features GitHub repository
The code that was originally created specifically for this module can be seen in the HTC/Yaml Merge Request which can be found in the original private repository of the code.