Overview

Teaching: 25 min
Exercises: 10 min

Questions

• What is the standard GNU installation workflow?

• How do I customize this workflow to tailor it to my needs?

• What are best-practices for managing source code and install documentation?

Objectives

• Understand the steps in the GNU installation workflow.

• Create a modulefile for an example package that uses the GNU workflow.

• Set up the directory structure for storing install recipes long-term.

Setting up our workspace

We will start by creating two new directories that will be used for our software management workflow:

$ mkdir ~/software.source
$ mkdir ~/software.install

The software.source directory will be used to hold the source code (or binary installers for proprietary software) and installation recipes for software you install, and the software.install directory will be where the actual installations are placed. Separating these directories is important because many build systems require us to start from a clean slate to rebuild a package, and we want the freedom to build and rebuild our software without deleting any previously installed versions.

With these directories in-place, let’s proceed with installing our example package, hmmer. This is a bioinformatics tool that may or may not be relevant to your research, but we have chosen it as our example here because it uses the standard GNU build procedure and because it compiles from source relatively quickly. Using our new software.source directory, let’s download the source code and get started:

$ cd ~/software.source
$ mkdir hmmer
$ cd hmmer
$ wget http://eddylab.org/software/hmmer/hmmer-3.2.1.tar.gz

Each software package should have its own subdirectory in software.source, similar to how we arrange the modulefiles.local directory. With the source code stored locally, let’s unpack the archive and build it:

$ tar xvf hmmer.3.2.1.tar.gz
$ cd hmmer-3.2.1
$ ls

config.guess  configure.ac    INSTALL         LICENSE      profmark/      src/         tutorial/
config.sub    documentation/  install-sh      Makefile.in  README.md      test-speed/  Userguide.pdf
configure     easel/          libdivsufsort/  makeTAGS.sh  RELEASE-3.2.1  testsuite/

The installation procedure recommended by the developer is often available on the software’s webpage, but we can also check the contents of the source directory for hints as to how we should built it. The first place to check would be files named INSTALL or README, as these are standard filenames for documentation for setting up software from source. In this case, we can take a look at the INSTALL file, since it is present:

$ more INSTALL

Brief installation instructions
HMMER 3.2.1 (June 2018)

Starting from a source distribution, hmmer-3.2.1.tar.gz:
   uncompress hmmer-3.2.1.tar.gz
   tar xf hmmer-3.2.1.tar
   cd hmmer-3.2.1
   ./configure
   make
   make check                        # optional: automated tests
   make install                      # optional: install HMMER programs and man pages
   (cd easel; make install)          # optional: install Easel tools too

For more details including customization, supported platforms, and
troubleshooting, see the Installation chapter in the HMMER User's
Guide (Userguide.pdf).

We’ve already started this process by decompressing the tar.gz archive that we downloaded and entering the resulting directory. The next steps make up the standard configure; make; make install workflow popularized by GNU software. This basic procedure is very commonly used in open-source software projects, and you will likely see it again if you install research software.

The steps can be briefly described as:

• configure - Check the current system for software and directory details, and create a Makefile that incorporates this information as well as user-specified customizations.
• make - Build the code from source using the instructions in the Makefile generated in the previous step. This is the step that actually creates the executable files and libraries that you interact with when using the package.
• make install - Copy the executables, libraries, and any built-in documentation to the final installation directory, where we will ultimately store them.

There is an additional step for this package that you will commonly see as well: make check. This is also sometimes implemented as make test depending on the software package, and is used to run any built-in tests that the developers provided for the software. When available you should always run this step to see if there are any issues that can be automatically identified with the software you just built. These tests are seldom comprehensive, but they can be a great way to find glaring issues before actually using the software. We won’t be running it today due to time constraints, but I would recommend doing so in general.

The configure step is where we will set any customizations we want for our software installation. At a minimum this will usually include specifying the installation directory, but you can also set the compilers to be used and enable/disable features to build. You can see the full set of options available for a particular configure by running

$ ./configure --help

`configure' configures HMMER 3.2.1 to adapt to many kinds of systems.

Usage: ./configure [OPTION]... [VAR=VALUE]...

To assign environment variables (e.g., CC, CFLAGS...), specify them as
VAR=VALUE.  See below for descriptions of some of the useful variables.

Defaults for the options are specified in brackets.

Configuration:
  -h, --help              display this help and exit
      --help=short        display options specific to this package
      --help=recursive    display the short help of all the included packages
  -V, --version           display version information and exit
  -q, --quiet, --silent   do not print `checking ...' messages
      --cache-file=FILE   cache test results in FILE [disabled]
  -C, --config-cache      alias for `--cache-file=config.cache'
  -n, --no-create         do not create output files
      --srcdir=DIR        find the sources in DIR [configure dir or `..']

Installation directories:
  --prefix=PREFIX         install architecture-independent files in PREFIX
                          [/usr/local]
  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]

By default, `make install' will install all the files in
`/usr/local/bin', `/usr/local/lib' etc.  You can specify
an installation prefix other than `/usr/local' using `--prefix',
for instance `--prefix=$HOME'.

For better control, use the options below.

Fine tuning of the installation directories:
  --bindir=DIR            user executables [EPREFIX/bin]
  --sbindir=DIR           system admin executables [EPREFIX/sbin]
  --libexecdir=DIR        program executables [EPREFIX/libexec]
  --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
  --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
  --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
  --libdir=DIR            object code libraries [EPREFIX/lib]
  --includedir=DIR        C header files [PREFIX/include]
  --oldincludedir=DIR     C header files for non-gcc [/usr/include]
  --datarootdir=DIR       read-only arch.-independent data root [PREFIX/share]
  --datadir=DIR           read-only architecture-independent data [DATAROOTDIR]
  --infodir=DIR           info documentation [DATAROOTDIR/info]
  --localedir=DIR         locale-dependent data [DATAROOTDIR/locale]
  --mandir=DIR            man documentation [DATAROOTDIR/man]
  --docdir=DIR            documentation root [DATAROOTDIR/doc/hmmer]
  --htmldir=DIR           html documentation [DOCDIR]
  --dvidir=DIR            dvi documentation [DOCDIR]
  --pdfdir=DIR            pdf documentation [DOCDIR]
  --psdir=DIR             ps documentation [DOCDIR]

System types:
  --build=BUILD     configure for building on BUILD [guessed]
  --host=HOST       cross-compile to build programs to run on HOST [BUILD]

Optional Features:
  --disable-option-checking  ignore unrecognized --enable/--with options
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]

      --enable-debugging      include debugging code
      --enable-debugging=x    also set diagnostics verbosity level to <x> (1-3)

  --enable-gcov           compile for code coverage testing
  --enable-gprof          compile for gcc code profiling
  --enable-sse            enable our SSE vector code
  --enable-vmx            enable our Altivec/VMX vector code
  --enable-threads        enable POSIX threads parallelization
  --enable-mpi            enable MPI parallelization
  --disable-largefile     omit support for large files

Optional Packages:
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-gsl              use the GSL, GNU Scientific Library

Some influential environment variables:
  CC          C compiler command
  CFLAGS      C compiler flags
  LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
              nonstandard directory <lib dir>
  LIBS        libraries to pass to the linker, e.g. -l<library>
  CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
              you have headers in a nonstandard directory <include dir>
  CPP         C preprocessor
  MPICC       MPI C compiler command

Use these variables to override the choices made by `configure' or to help
it to find libraries and programs with nonstandard names/locations.

Report bugs to <sean@eddylab.org>.

As you can see there are many options we can potentially change during the configure step for this software, but we will focus on just a couple: the installation directory and the compiler to use. The former is set using the --prefix= flag, and the latter is set by setting the value of key environment variables. It is good practice to specify the compiler explicitly during this step, even if most configure scripts will auto-detect working compilers for you. If you specify the compilers manually, you will be able to precisely re-create the build procedure in the future, which may not be possible if you don’t know which compiler was auto-detected.

Since we can set the compilers via environment variables, let’s do that first. We will load a compiler module to use for this example, which is also considered good practice. If you don’t manually load a compiler then you will rely on the system-installed compilers, which may change depending on necessary updates and can’t be relied upon to be available over the long-term.

$ module load gcc/7.2.0
$ which gcc

/common/software/install/migrated/gcc/7.2.0/bin/gcc

I’m using the GNU compiler suite for this build, but you could also use the Intel compiler suite, LLVM, or any other compilers that are recommended by the developers. Those compilers will be named differently, so you will need to look in their documentation to see which executables to use.

Referencing the configure --help output from before, we can see that the environment variable CC controls the compiler command that is run. We will set it to the location of the gcc executable that we just loaded:

$ export CC=$(which gcc)

Now when we run configure it will be forced to use this compiler rather than any others that may be present in the environment.

Setting the installation directory is done as an argument when running configure, so let’s do that now:

$ ./configure --prefix=$HOME/software.install/hmmer/3.2.1

Using the --prefix we specify the installation directory, which doesn’t have to exist yet (it will be created automatically during make install if needed). We are targeting our software.install directory, and are specifying a subdirectory named after the software package, and a further subdirectory for the specific version of that package. Again, this mirrors the directory structure of the modulefiles.local directory. When we actually run the command it will produce a lot of output and take a moment to run as it checks what is available on our system. If it completes without errors, we can proceed immediately to the next step by running

$ make

For more complex software packages you may want to run make inside an interactive job, as it can take a long time to build them and the process could be killed if running for too long on the login nodes. In those cases you will also likely want to run make -jN, replacing N with a number of available processors. This will allow make to run in a multithreaded mode that can be significantly faster.

We would run the make check or make test step here if this wasn’t a live demo, to make sure our new software is working as expected. Instead, for the purposes of time we will proceed directly to the installation step:

$ make install

Once this step is complete, the software is installed and (almost) ready to use. We can check the resulting directory to see what was actually installed:

$ ls ~/software.install/hmmer/3.2.1

bin/  share/

In this case the directory structure is fairly simple - we have a bin directory than contains all of the executables for this package, and a share directory that contains documentation. At this point, we are ready to create a new modulefile to interact with the newly installed software:

$ mkdir ~/modulefiles.local/hmmer
$ nano ~/modulefiles.local/hmmer/3.2.1

#%Module
module load gcc/7.2.0
prepend-path PATH /users/3/dunn0404/software.install/hmmer/3.2.1/bin

We’ve added an extra element to this modulefile - loading a dependency. Including the module load command here makes it so that when we load our new hmmer module, the gcc module we used to build it will also be loaded. This isn’t always necessary, but becomes important when the new module may refer to libraries or executables in the dependency module. Here we’re loading the gcc module as an example, as the hmmer module doesn’t refer to it directly. You’ll need to read the documentation for the package you are installing to know whether you need to load a dependency or not, but it generally doesn’t hurt anything to load everything that you used to build with. So if you don’t know, load away!

Other Paths

For some software packages you may need to make some additional adjustments to your environment. The most common additional changes would be to the LIBRARY_PATH and INCLUDE variables for software you are using as a build dependency for other software, and to the LD_LIBRARY_PATH for software that creates dynamic libraries that are needed at runtime. These will often be hinted at by the inclusion of an include and lib or lib64 directory alongside the bin directory in your install location. For an example of a modulefile that modifies additional environment variables, you can run module show gcc on one of the MSI login nodes.

With the new modulefile in place, we can see our module is available for loading:

$ module avail hmmer

-------------------------------------------- /users/3/dunn0404/modulefiles.local --------------------------------------
hmmer/3.2.1
-------------------------------------------- /common/software/modulefiles/migrated/common ---------------------------------------------
hmmer/2.3.2  hmmer/3.0  hmmer/3.1-snap20121016  hmmer/3.1b2(default)  hmmer/3.2.1

We can see that we’ve installed a version of hmmer that was already available as a system module. In cases like this where there are two modules with the same name/version string, the module in the topmost modulefile directory will be loaded when using that name. So our custom module will override the system module if we run

$ module load hmmer/3.2.1
$ which nhmmer

/users/3/dunn0404/software.install/hmmer/3.2.1/bin/nhmmer

This is usually the behavior we want, otherwise we wouldn’t have bothered to install the local module and could have just used the existing module. If you do want to load the existing module, the best practice would be to rename your local module to something like hmmer/3.2.1-local by renaming the modulefile accordingly. You can also load a module by specifying the full path to the modulefile, so you could load the global module by running:

$ module load /common/software/modulefiles/migrated/common/hmmer/3.2.1

This is a lot of typing if you need to do it regularly, so it is generally better to just rename your local module.

Now that we’ve installed the software and created the module, let’s document the steps we took to install it so we can reproduce the exact installation procedure in the future. We can see the recently run commands for the current session by running history, then we will copy and paste the relevant commands into a README file in the source directory:

$ nano ~/software.source/hmmer/README.recipes

# dunn0404 - April 20, 2022
# hmmer 3.2.1 with gcc/7.2.0

wget http://eddylab.org/software/hmmer/hmmer-3.2.1.tar.gz
tar xvf hmmer-3.2.1.tar.gz
cd hmmer-3.2.1/

module load gcc/7.2.0

export CC=$(which gcc)

./configure --prefix=$HOME/software.install/hmmer/3.2.1

make
# make check
make install

That’s all there is to it! You’ve created your first full custom software module. Now, let’s take a look at how we can work with other common build systems to do the same.

Key Points

• The configure; make; make install workflow is one of the most common installation methods for software.

• Most of the customizations to the workflow happen during the configure step.

• Saving the source code and install recipes for your packages and environments will make your life easier.