Separating Packages Using Environment Modules
Overview
Teaching: 25 min
Exercises: 10 minQuestions
What is a software environment?
How do I use environment modules?
How do I create my own environment modules?
Objectives
Understand the basics of interacting with environment modules.
Create our first ‘hello world’ environment module.
Software environments in the bash shell
When you open a new bash session, you are automatically dropped into a software environment. There are
a set of programs that you are able to run to interact with the system - cd, ls, ssh, and echo
are a few examples of programs that are immediately available in a new session. The shell locates these
programs in the same way that it will locate research software that we install - by checking relevant
environment variables.
The key environment variable for locating executable files like these is PATH, which contains a colon-
separated list of directories where the shell will look for executable files when you run a command. We
can see the default value for this variable by running:
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
The exact value of PATH will depend on where you run the command, but you should see something similar
to the output above. We can look at the contents of one of these directories to see what programs are
installed there:
$ ls /usr/bin
These are where the operating system installs all of the tools that it provides for you to use on the command line or as part of shell scripts, and there may be dozens or hundreds of executable files installed in each one.
If we want to know where a specific executable is located, we can use the which command to get the
location of the executable that the shell uses when we run the command. Let’s see where the ls
executable is located:
$ which ls
/usr/bin/ls
So we can see that the shell is finding the ls executable located in the /usr/bin directory from
our PATH variable. We can make new programs available on the command line by adding new directories
to PATH, and we can hide programs from a shell session by removing the directory containing them
from PATH. We do this by setting a new value for PATH in the shell. Let’s demonstrate this by
modifying PATH to include an existing mothur installation on the cluster. First, let’s demonstrate
that mothur isn’t currently available in our environment using which:
$ which mothur
/usr/bin/which: no mothur in (/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin)
Now, let’s modify PATH:
$ export PATH=$PATH:/common/software/install/manual/mothur/1.48.1/bin
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/common/software/install/manual/mothur/1.48.1/bin
Now, mothur is available in our shell session:
$ which mothur
/common/software/install/manual/mothur/1.48.1/bin/mothur
$ mothur --version
Linux
Mothur version=1.48.1
Release Date=5/15/24
However, we don’t want to manually modify PATH when we can avoid it, because:
- It gets unweildy when we need to add a lot of directories to the environment
- It can be difficult to undo changes when you’ve added more than a couple of directories
- Mistakes can break your shell session
To demonstrate that last point, let’s take a look at a common error that can occur with this type of modification:
$ export PATH=/common/software/install/manual/mothur/1.48.1/bin
$ ls $PATH
bash: ls: command not found
Uh oh, where did ls go?
$ which ls
bash: which: command not found
Whoops.
We can fix this by starting a new shell session, but we can also avoid the issue entirely if we use a more robust approach to modifying our environment.
Environment modules automate environment changes
Adding a layer of automation to our environment modifications can help to guard against typos,
and make it easier to see what software is available for use to use. This automation is provided
by the module command, which is the executable interface to a system known as environment modules.
We can use this command to bring new software into our environment:
$ module load mothur/1.48.1-rocky8
$ which mothur
/common/software/install/manual/mothur/1.48.1/bin/mothur
And to remove the software from our environment again:
$ module unload mothur
$ which mothur
/usr/bin/which: no mothur in (/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin)
We can see which versions of mothur are currently installed:
$ module avail mothur
----------------------------------- /common/software/modulefiles/migrated/hpc ------------------------------------
mothur/1.37.6_intel_ompi mothur/1.44.0 mothur/1.44.3 mothur/1.48.0
mothur/1.42.3 mothur/1.44.1 mothur/1.45.3
----------------------------------- /common/software/modulefiles/manual/common -----------------------------------
mothur/1.48.1-rocky8
---------------------------------- /common/software/modulefiles/migrated/common ----------------------------------
mothur/1.27.0 mothur/1.29.2 mothur/1.33.3(default) mothur/1.34.0 mothur/1.35.1 mothur/1.39.5 mothur/1.41.1
Or what other modules are available:
$ module avail
--------------------------------------------- /common/software/modulefiles/migrated/mangi ---------------------------------------------
gaussian/g09.e01 gaussian/g16.c01(default)
...
libunistring/0.9.3 wgs/wgs-8.1(default)
libxml2/2.9.10 workbench/1.4.2
libxz/5.0.8 xz-utils/5.2.2_gcc6.1.0
llvm/5.0.0(default) xz-utils/5.2.2_intel2015update3
llvm/9.0.1 xz-utils/5.2.3_gcc7.2.0
lmdb/0.9.18 zlib/1.2.3.3
lmdb/0.9.22 zlib/1.2.7
longranger/1.1.0 zlib/1.2.8(default)
longranger/2.1.6 zlib/1.2.8_gcc6.1.0
longranger/2.2.2 zlib/1.2.11_gcc7.2.0
ltr_retriever/2.9.0 zsh/5.5.1
The modules we are seeing through the command right now are installed system-wide, and are available for use by any researcher at MSI, with some exceptions due to licensing restrictions. This points out one case where you might not want to install software yourself - if it would be used by other research groups as well. In those cases it can save time and effort for MSI staff to install the software centrally. If you aren’t sure if a software package would be a good fit for central installation, you can submit a request on the MSI webpage, and we will let you know if this should be a local or central installation:
https://www.msi.umn.edu/swrequest
Creating Modulefiles
Let’s create our first modulefile. Modulefiles are specialized scripts in the tcl language, so they are
somewhat flexible in what they can do. To start, let’s begin in the tradition of all programming languages
and create a modulefile that says Hello world.
The first thing we need to do will be to create a directory where our modulefiles will be stored. The module tool looks in specified directories for modulefiles, and the structure of those directories is important for labeling in the module tool. Let’s create a folder to hold all of our custom modulefiles:
$ mkdir ~/modulefiles.local
Then, let’s create a directory to contain the specific example modulefile. We’ll name it after the ‘program’ we plan to install:
$ mkdir ~/modulefiles.local/hello-world
Then, let’s create the modulefile itself in that directory. We’ll name the modulefile as the version of the
program we’re installing:
$ nano ~/modulefiles.local/hello-world/1.0
A modulefile starts with a special string called a magic cookie that identifies it as a modulefile:
#%Module
Files in this directory without that cookie will be ignored by the module command, so be sure to include it. Next, let’s add the tcl statement to print our statement:
#%Module
system echo Hello world
Now, we need to tell the module command that we have a new directory with modulefiles to use:
$ module use ~/modulefiles.local
And we can see our new module via the module command, and load and unload it:
$ module avail hello-world
-------------------------------------------- /users/3/dunn0404/modulefiles.local --------------------------------------------
hello-world/1.0
$ module load hello-world
Hello world
This demonstrates the basic structure of a modulefile and the directory structure that the module tool expects. Modules are labeled according to the subdirectory they are located in, usually named after the software package they represent, and their filename, usually the version of the software you are installing. When you load or unload the file, the tcl script inside the modulefile is run.
This isn’t a very useful module, however, since it doesn’t change any features of our environment. Let’s modify it to
add mothur to our PATH as we had done manually earlier:
$ nano ~/modulefiles.local/hello-world/1.0
#%Module
prepend-path PATH /common/software/install/manual/mothur/1.48.1/bin
The prepend-path command is a special statement for modulefiles. When you load the module it will add the directory
at the end of the line to the environment variable provided as the first argument. This change will be undone when
you unload the module without the need for us to add any other logic. Let’s try the new module:
$ module load hello-world
$ mothur --version
Linux
Mothur version=1.45.3
Release Date=5/15/24
The prepend-path command is the most common command we use in modulefiles. You can also use append-path to add the
new directory to the end of the specified variable for cases where the order matters in complex environments.
With these basics in-hand, let’s move on to a few examples of installing our own software and using modulefiles to manage that software, rather than pre-existing software installations.
Key Points
A software environment is defined by a set of environment variables and the directories they refer to.
Environment modules are tcl (or lua) scripts specialized for modifying software environments.
The
modulecommand lets you quickly swap between software environments.