Overview

Teaching: 25 min
Exercises: 10 min

Questions

• How does anaconda install software?

• How can I use anaconda to create independent modules?

• How can I keep my pip-installed packages separate?

Objectives

• Learn how to use conda/mamba to create a module-compatible software environment

• Learn how to install software into a conda environment using pip

License Issues

The company Anaconda that develops conda and provides several package repositories for it has changed their Terms of Service as of Spring 2024 to no longer accommodate free use of their licensed components by academic researchers. Discussion on how to proceed under these ToS within the community and within Anaconda, as well as between the community and Anaconda, is still ongoing as of Nov 2024.

In the meantime, avoid using the package channels defaults, main, anaconda, msys2, or r. You will want to make sure that these package channels do not appear in your ~/.condarc file, and that you do not reference them in your conda install commands. Finally, use miniforge over other modules that provide conda, as it avoids using forbidden channels by default.

Forcing Anaconda to work with environment modules

We have been creating modules by building from source up to this point, but you can also create modules for pre-built executables that you download. One common way of managing and obtaining pre-built software for your computer is to use Anaconda, a tool originally developed for managing python environments that has been extended to work as a general-purpose package manager.

The most basic conda installation procedure looks like the following for pandas (a python toolkit for working with tabular data):

$ module load miniforge
$ conda install pandas

However, Anaconda has its own directory structure that it will impose by default, and it tries to save disk space by linking to files rather than copying them to each individual environment. This makes it difficult to work with conda environments over the long-term, since these aren’t compatible with our module approach. Anaconda does include a tool for creating independent environments, but we have to use some special options to make it behave as we need. Further, the conda tool itself is notoriously slow at creating these environments, so we will be using an alternative tool known as mamba, which re-implements the conda environment management functionality in a more performant way.

So to quickly create a module-friendly environment for pandas, we would run the following:

$ today_date=$(date +%d%m%y)
$ module load mamba
$ mamba create --copy -p $HOME/software.install/pandas/$today_date pandas

This command has a few elements worth noting:

• create - This tells mamba to create an independent environment instead of using its default environment
• --copy - This forces mamba to copy in full all of the packages to the install location, which prevents it from depending on any other environments.
• -p DIR - This specifies the install location for the new environment. We use a date here because it is important to note when this environment was ‘solved’ by mamba, since the ‘solution’ may not be the same tomorrow as packages and dependencies are updated.

Since anaconda doesn’t work with source code, we don’t have source code to store. Further, we just running the same build command again in the future isn’t guaranteed to recreate the same environment because updates to packages may change how the environment is ‘solved’. We can take a snapshot of the exact package versions used for this environment by running:

$ mkdir ~/software.source/pandas
$ source activate $HOME/software.install/pandas/$today_date
$ conda env export --no-builds | grep -v "prefix" > ~/software.source/pandas/${today_date}.yml
$ source deactivate

The file ${today_date}.yml contains the complete specification for this environment, and you can use it to reinstall an identical environment in the future via, e.g. for a hypothetical environment I made on Jan 5, 2021:

$ module load mamba
$ mamba env create -f ~/software.source/pandas/010521.yml

If you need to change the install location for some reason for this reinstall, you will need to edit the first line of the yml file with the directory you’d like to use before recreating the environment.

Now, while we did use anaconda’s built-in environment management tool above via the source activate command, we don’t want to rely on this when using the environment. The reason for this is that anaconda environments don’t behave well in scripts, and attempting to fix this behavior can cause more issues than it solves. So we will use modulefiles once again, completely sidestepping the issue.

For most anaconda environments you will only need to modify PATH, so your modulefile would look like:

$ mkdir ~/modulefiles.local/pandas
$ nano ~/modulefiles.local/pandas/$today_date

#%Module
prepend-path PATH /users/3/dunn0404/software.install/pandas/032123/bin

Occasionally you will have to also modify the LD_LIBRARY_PATH variable to include the lib directory in the environment, but this will depend on the specific software being installed. In this case we’re OK with just modifying PATH.

Conda caches packages in your home directory

If you frequently use conda or use it to solve large and complex environments, you may find that you run out of storage space more quickly than expected. This might be because conda caches packages to ~/.conda, a hidden directory under your home directory, and this cache can become quite large in terms of both disk space and number of files over time.

The cache makes it faster to repeat conda installations that share common dependencies, but it is necessary to clean it out every once and a while. To check what will be removed by a cache cleaning, you can run the command:

$ conda clean --all --dry-run

And to actually execute the cleaning, you can remove the --dry-run option:

$ conda clean --all

Alternatively, if you want to keep your conda cache around for longer than it is convenient to store in your home directory, you can choose another location to use for the cache by changing the pkg_dirs option in your ~/.condarc file, e.g.

pkgs_dirs:
  - /nfs/volume/user/conda_pkgs

You can read more about this in the official conda documentation on the topic.

Anaconda environment tips

There are a few miscellaneous things to keep in mind when working with anaconda environments to make things work more smoothly and predictably:

• Don’t modify an existing conda environment. If you need to add packages to an environment, you are better served by re-creating the environment from scratch with the additional packages, as the anaconda environment solver can get confused by dependency trees that may have changed over time.
• Many packages will use an anaconda environment to fill dependencies, and will expect the environment to be active when running the software. For most of these, you can set up the anaconda environment with the dependencies as a module like we have here, then copy the executables of the package into the bin directory of the anaconda environment.
• Some python packages are only available pip. In these cases, you can still use anaconda to create an isolated environment, but the actual anaconda environment will be minimal. This could look like:

# create a conda environment with only pip and its dependencies
$ today_date=$(date +%d%m%y)
$ module load mamba
$ mamba create --copy -p $HOME/software.install/numpy/$today_date python pip

# use conda's environment management to force pip to install there
$ source activate $HOME/software.install/numpy/$today_date
$ pip install numpy
$ source deactivate

The packages you install via pip won’t be captured by conda env export, so you will need to record these manually in a recipe file alongside the yml file.

Key Points

• Conda environments are complex and finicky, but we can avoid most issues by forcing them to install into our module framework.

• We can use conda to isolate pip installations.