Why Manage My Own Software?

Overview

Teaching: 15 min
Exercises: 5 min

Questions

• When would I want to manage my own software packages on an HPC cluster?

• What can I expect to learn from this course?

Objectives

• Be able to describe why self-managed software can be important.

• Identify how self-managed software would be useful to you.

Before we start, a few definitions for terms that will come up:

• Software package: A single program or toolkit, e.g. tensorflow, gromacs, python, fftw
• Software environment: A set of packages that work together to create a pipeline or workflow
• Dependency: A package that is required in order to use or build another package, e.g. fftw is a dependency of gromacs

The vast majority of modern research relies on software in some form or another to generate, analyze, and visualize data. As researchers a an HPC center we recognize that large datasets and intensive calculations can cause a computational workflow to outgrow running on your personal computer. Similarly, complex and numerous software environments involved in a project can cause it to outgrow the installation and management approaches you would use on your personal computer:

• A physics student needs to validate their results against an older paper. This involves installing the exact version of the simulation engine the previous authors used, but some of the dependencies for that version of the package conflict with other software that the student needs.

• A researcher has been contacted by a journal about a possible error in the results of a previous paper. They want to go back and re-run the model, but they don’t have the software they used installed anymore and didn’t document their installation procedure.

• An economist wants to expand the scope of their simulations by running them on an HPC cluster. They use a software package with highly restrictive licensing terms that prohibits use by any other users, so they can’t request for it to be installed system-wide.

In each of these cases, the lack of a robust software management workflow could lead to wasted time and potential dead-ends in research progress.

And what do you do?

Talk to your neighbour, office mate or rubber duck about your research. How do you currently manage the software used in your research? How could better software management practices help you or your group?

Principles of a robust software environment

Academic research patterns present specific challenges for maintaining software environments. Members of a lab are constantly in flux as students and post-docs arrive and depart, and research projects may continue through several staff turnovers. Cutting edge research demands installing cutting edge software, and reproducibility demands that the exact software environment used to work on a paper remain available indefinitely. How do you quickly create environments so that a researcher of any level of expertise can use them, and that can be easily re-created years down the line long after anyone involved in their setup has left?

• Packages should be as independent as possible: Avoid your workflows breaking because of changes to the computing environment or installation of new packages

• Keep enough documentation that the environment should be trivial to reproduce: Troubleshooting or recreating an environment is much easier when you keep the instructions used to maintain it

• Reduce dependence on HPC support staff: The combination of software you use to run your workflows is unique to your group, and your group is in the best position to quickly fix issues or deploy updates

Key Points

• Software managed by other parties can be subject to change and difficult to reproduce in the future.

• Research workflows require increasingly sophisticated software workflows to support them.

• A small time investment up front in developing a robust software workflow can be invaluable in a research environment.