The Shell Environment and Software Modules

Overview

Teaching: 15 min
Exercises: 0 min

Questions

How do I access properties of my current shell session?

How do I use the software modules provided by MSI?

Objectives

Explain how to get, set, and explore environment variables for a shell session.

Explain how software modules change the environment to allow different software packages to run.

Load a software module and observe the environment changes.

A single shell window is known as a ‘session’, and has session-specific properties that are defined as variables within the session. These variables are known as ‘environment variables’ because they define the shell environment for the session. These variables control the behavior of various command line tools in the session. Examples of environment variables include:

HOME defines the home directory
TMPDIR specifies where temporary files are written
LANG specifies the language of the current session

Note that environment variables are usually named in all caps, which is a convention used to help distinguish them from other variables you may set in bash. We can see what variables are set for our current session:

$ env | less

MODULE_VERSION_STACK=3.2.6
MANPATH=:/opt/puppetlabs/puppet/share/man
HOSTNAME=ahl03
SELINUX_ROLE_REQUESTED=
MODULEPATH=/users/3/dunn0404/modulefiles.local:/common/software/modulefiles/manual/mangi:/common/software/modulefiles/migrated/mangi:/common/software/modulefiles/spack.tcl/linux-centos7-ivybridge:/common/software/modulefiles/singularity-hpc:/common/software/modulefiles/manual/k40:/common/software/modulefiles/migrated/k40:/common/software/modulefiles/manual/v100:/common/software/modulefiles/migrated/v100:/common/software/modulefiles/manual/hpc:/common/software/modulefiles/migrated/hpc:/common/software/modulefiles/migrated/intel:/common/software/modulefiles/manual/common:/common/software/modulefiles/migrated/common:/common/software/modulefiles/migrated/centos7
MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl
TERM=xterm-256color
SHELL=/bin/bash
HISTSIZE=1000
...

There are a lot of settings here, and we won’t be getting into them all. If you are interested, most of these variables have standard names that you can research on your own. We will be focusing on a few environment variables related to software availability within the shell. The most important of these is PATH, which defines the locations where the shell looks for executables when you run a command:

$ echo $PATH

/usr/local/bin:/usr/local/sbin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

Your output may be a little different from mine, depending on where you are logged in and what customization you have done to your account, if any. Note that the contents of PATH are a series of directories separated by colons. Each of those paths contains executables that we may have been using in the workshop so far. Since there may be many executables present in any of these folders, it can be more useful to look for the location of a specific executable rather than listing the contents of the PATH directories:

$ which ls

/usr/bin/ls

So we can see that the ls command is associated with an executable file that lives in /usr/bin. An important thing to note is that the shell will look for commands that you run in each of the directories in PATH, in the order they appear. So if you are changing the value of PATH it is usually best to add your modifications to the end of the variable, rather than the beginning. Otherwise you run the risk of overriding a system tool with a custom one, which can cause problems if you are not careful. For instance, if I had a directory in my home folder called bin where I stored some of my own programs, I might modify PATH to include it.

$ mkdir ~/bin
$ echo hostname > ~/bin/hello.sh
$ chmod +x ~/bin/hello.sh
$ export PATH=$PATH:~/bin

Then, I can run my new hello.sh script just like any of the built-in command line tools:

$ which hello.sh
$ hello.sh

~/bin/hello.sh
ahl03

This can be very useful when you are writing your own scripts and programs and want to reference them frequently from the command line. You can make such a modification permanent by adding the line export PATH=$PATH:~/bin to the end of the file ~/.bashrc, which is where some of the environment variables can be customized.

MSI also provides hundreds of software packages that you can use by modifying your environment. Luckily, there is a tool that makes working with these kinds of environment changes more streamlined: the module command.

You can see all of the software modules that are available on the current system by running the command

$ module avail

...
qiime/1.9.1_centos7
readline/8.0
relion/3.0_beta_cpu
relion/testing.3.0_beta
repeatmasker/4.0.5(default)
ruby/2.6.0_gcc8.2.0
samtools/1.9(default)
scalapack/2.0.2-gcc-7.2.0
singularity/current
snappy/1.1.7
sprng/5.0_gcc8.2.0
sprng/5.0_gcc8.2.0_ompi4.0.0
sprng/5.0_intel2019update1
star/2.7.1a
structure/2.3.4-console-CentOS7
swig/3.0.12_gcc8.2.0
vtk/8.2.0_gcc8.2.0
xerces/3.2.2_intel
zsh/5.5.1.CentOS7

You can also be more specific and see what versions of a particular software package are available:

$ module avail rclone

-------------------------------- /common/software/modulefiles/manual/common --------------------------------
rclone/1.74.4

-------------------------------- /common/software/modulefiles/bundle/views/common --------------------------
rclone/1.74.4-r0

MSI also maintains a searchable directory of available software at https://www.msi.umn.edu/software.

We can look at how a particular module modifies the environment using the module show command:

$ module show rclone/1.74.4

-------------------------------------------------------------------
/common/software/modulefiles/manual/common/rclone/1.74.4:

prepend-path    PATH /common/software/install/manual/rclone/1.74.4/bin
-------------------------------------------------------------------

The scripting language inside module files is a bit different than bash, but we can read it given the context of knowning about environment variables. For this example, we can see that this module add a directoriy to PATH. If we want to use this version of rclone in our session, we can run:

$ module load rclone/1.74.4

We can demonstrate that the module has loaded successfully in a couple of ways:

$ module list

Currently Loaded Modulefiles:
  1) rclone/1.74.4

will show the list of currently loaded module files, while

$ which rclone

shows that the rclone executable is present in our environment now, and shows its location:

/common/software/install/manual/rclone/1.74.4/bin/rclone

We can also load many modules at once:

$ module load gcc python ompi
$ module list

Currently Loaded Modulefiles:
  1) rclone/1.74.4           4) mpc/1.1.0_gcc8.1.0      7) python/3.7.1_anaconda
  2) gmp/6.1.2_gcc8.1.0      5) isl/0.19_gcc8.1.0       8) java/openjdk-11.0.2
  3) mpfr/4.0.1_gcc8.1.0     6) gcc/8.2.0               9) ompi/gnu.mesabi

Note that there are more modules currently listed than we explicitly loaded. This is because some modules load other modules that they depend on:

$ module show gcc

-------------------------------------------------------------------
/common/software/modulefiles/migrated/hpc/gcc/8.2.0:

module          load gmp/6.1.2_gcc8.1.0
module          load mpfr/4.0.1_gcc8.1.0
module          load mpc/1.1.0_gcc8.1.0
module          load isl/0.19_gcc8.1.0
prepend-path    PATH /common/software/install/migrated/gcc/8.2.0/bin
prepend-path    INCLUDE /common/software/install/migrated/gcc/8.2.0/include
prepend-path    CPATH /common/software/install/migrated/gcc/8.2.0/include
prepend-path    C_INCLUDE_PATH /common/software/install/migrated/gcc/8.2.0/include
prepend-path    CPLUS_INCLUDE_PATH /common/software/install/migrated/gcc/8.2.0/include
prepend-path    FPATH /common/software/install/migrated/gcc/8.2.0/include
prepend-path    LD_LIBRARY_PATH /common/software/install/migrated/gcc/8.2.0/lib64
prepend-path    LD_RUN_PATH /common/software/install/migrated/gcc/8.2.0/lib64
prepend-path    LIBRARY_PATH /common/software/install/migrated/gcc/8.2.0/lib64
prepend-path    MANPATH /common/software/install/migrated/gcc/8.2.0/share/man
prepend-path    INFOPATH /common/software/install/migrated/gcc/8.2.0/share/info
-------------------------------------------------------------------

Also note that while we asked to load gcc, we got gcc/8.2.0 instead. This is because that is the version of gcc that is currently set as default, and gets loaded when you don’t specify a module version:

$ module avail gcc

---------------------- /common/software/modulefiles/spack.tcl/linux-centos7-ivybridge -----------------------
gcc/4.9.2-krax5se  gcc/7.2.0-kjvrzfs  gcc/8.2.0-hisn3fl  gcc/13.1.0-5z64cho
gcc/4.9.2-ugw6raf  gcc/7.2.0-tvrcc2c  gcc/8.2.0-ztp76o3  gcc/13.1.0-mptekim

--------------------------------- /common/software/modulefiles/migrated/hpc ---------------------------------
gcc/8.2.0(default)  gcc/9.2.0

------------------------------- /common/software/modulefiles/migrated/common --------------------------------
gcc/4.9.2  gcc/5.1.0  gcc/5.4.0  gcc/6.1.0  gcc/6.3.0  gcc/7.2.0  gcc/8.1.0  gcc/11.3.0

There are some special rules for how module defaults are defined. The first (default) that you see in the module avail acts as the default, and if no default is defined, then the last version listed under the first heading is used as the default. It is configured this way so that we can specify different default versions for different systems, but still use many of the same module files.

If we want to remove a module from our environment, we can do so via:

$ module unload gcc/8.2.0
$ module list

Currently Loaded Modulefiles:
  1) rclone/1.74.4           3) java/openjdk-11.0.2
  2) python/3.7.1_anaconda   4) ompi/gnu.mesabi

Or, if we want to clear all modules from our environment, we can do so using the module purge command:

$ module purge

No Modulefiles Currently Loaded.

Similar to how you can save changes to environment variables, you can make it so that a module is always loaded when you log in by adding module load MODULENAME to your ~/.bashrc file.

Key Points

env prints all of the currently defined environment variables.

export VAR=newvalue can be used to change the value of environment variables, or to set new ones.

module load MODULE changes the shell environment to allow use of new software packages.

module unload MODULE undoes the changes to the environment from a single module

module purge undoes the changes to the environment from all loaded modules

module avail and module avail MODULE can be used to find what software and versions are available.

module list will list all currently loaded modules.

module show MODULE will show all of the changes that the module MODULE will make to the environment when it is loaded.

previous episode

Intro to Linux

lesson home

The Shell Environment and Software Modules

Overview

Key Points

previous episode

lesson home