Adding HTC Library to EasyBuild

This module is the fourth in a sequence that will form the overall capabilities of the HTC library (see HTC Multi-node Tasks for the previous module). This module deals with installing the software on HPC systems in a coherent manner through the tool EasyBuild.

Purpose of Module

The HTC library requires configuration for the target system. Typically, this configuration is applicable system-wide. If the software is provided in the main software stack of the system, this configuration can also be provided centrally. The goal of the integration with EasyBuild is to highlight how this configuration can be made with an explicit example of the configuration for the JURECA system.

Background Information

EasyBuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way. Full details on can be found in the EasyBuild documentation.

EasyBuild already has support for Python packages, what we describe here is the specific configuration required to install a particular version of the library on a specific software stack on JURECA.

Building and Testing

To build the software requires EasyBuild (see installation instructions for EasyBuild here) and the build command:

eb jobqueue_features-0.0.4-intel-para-2018b-Python-3.6.6.eb

However, please note that this will only work “out of the box” for those with software installation rights on the JURECA system. The provided sources (as described below) are intended as templates for those who are familiar with EasyBuild to adapt to their system (the only expected adaption would be to change the toolchain to suit their own system).

Source Code

The latest version of the library itself is available on the jobqueue_features GitHub repository.

There is an open Pull Request for the JURECA software stack that provides all necessary dependencies for the library.

The configuration file required for JURECA is included below (a version for Python 2 can also be created by simply changing the Python dependency version):

easyblock = 'PythonBundle'

name = 'jobqueue_features'
version = '0.0.4'
versionsuffix = '-Python-%(pyver)s'

homepage = ''
description = """
A Python module that adds features to dask-jobqueue to handle MPI workloads and different clusters.
Examples of usage can be found in the examples folder of the installation ($JOBQUEUE_FEATURES_EXAMPLES)

toolchain = {'name': 'intel-para', 'version': '2018b'}

dependencies = [
    ('Python', '3.6.6'),
    ('Dask', 'Nov2018Bundle', versionsuffix),

use_pip = True

exts_list = [
    ('typing', '3.6.6', {
        'source_urls': [''],
        'checksums': ['4027c5f6127a6267a435201981ba156de91ad0d1d98e9ddc2aa173453453492d'],
    ('pytest-cov', '2.6.0', {
        'source_urls': [''],
        'checksums': ['e360f048b7dae3f2f2a9a4d067b2dd6b6a015d384d1577c994a43f3f7cbad762'],
    (name, version, {
        'patches': ['jobqueue_features-%s.patch' % version],
        'source_tmpl': 'v%(version)s.tar.gz',
        'source_urls': [''],
        'checksums': [
            '0152ff89f237225656348865073f73f46bda7a17c97e3bc1de8227eea450fb09',  # v0.0.4.tar.gz
            '698204ef68f5842c82c5f04bfb614335254fae293f00ca65719559582c1fb181',  # jobqueue_features-env.patch

postinstallcmds = [
    'cp -r %(builddir)s/%(name)s/%(name)s-%(version)s/examples %(installdir)s/examples',
    'mkdir %(installdir)s/config && cp %(builddir)s/%(name)s/%(name)s-%(version)s/%(name)s/%(name)s.yaml %(installdir)s/config'

modextravars = {
    'DASK_ROOT_CONFIG': '%(installdir)s/config',
    'JOBQUEUE_FEATURES_EXAMPLES': '%(installdir)s/examples',

sanity_check_paths = {
    'files': ['config/jobqueue_features.yaml'],
    'dirs': ['lib/python%(pyshortver)s/site-packages', 'examples', 'config'],

moduleclass = 'devel'