Apptainer: Practical Considerations

Overview

Teaching: 30 min
Exercises: 20 min

Questions

• What does apptainer download and store when you run an image?

• How does a apptainer container interact with my files and data?

• Where can I find images to use in my workflows?

Objectives

• Learn about how to view and manage the apptainer image cache.

• How do I make data available in a apptainer container?

• Learn how to run apptainer containers based on Docker images.

Apptainer’s image cache

While Apptainer doesn’t have a local image repository in the same way as Docker, it does cache downloaded image files. As we saw in the previous episode, images are simply .sif files stored on your local disk.

If you delete a local .sif image that you have pulled from a remote image repository and then pull it again, if the image is unchanged from the version you previously pulled, you will be given a copy of the image file from your local cache rather than the image being downloaded again from the remote source. This removes unnecessary network transfers and is particularly useful for large images which may take some time to transfer over the network. To demonstrate this, remove the hello-world.sif file stored in your test directory and then issue the pull command again:

$ rm hello-world.sif
$ apptainer pull hello-world.sif shub://vsoch/hello-world

INFO:    Use image from cache

As we can see in the above output, the image has been returned from the cache and we don’t see the output that we saw previously showing the image being downloaded from Singularity Hub.

How do we know what is stored in the local cache? We can find out using the apptainer cache command:

$ apptainer cache list

There are 1 container file(s) using 62.65 MB and 0 oci blob file(s) using 0.00 kB of space
Total space used: 62.65 MB

This tells us how many container files are stored in the cache and how much disk space the cache is using but it doesn’t tell us what is actually being stored. To find out more information we can add the -v verbose flag to the list command:

$ apptainer cache list -v

NAME                     DATE CREATED           SIZE             TYPE
hello-world_latest.sif   2020-04-03 13:20:44    62.65 MB         shub

There are 1 container file(s) using 62.65 MB and 0 oci blob file(s) using 0.00 kB of space
Total space used: 62.65 MB

This provides us with some more useful information about the actual images stored in the cache. In the TYPE column we can see that our image type is shub because it’s a SIF image that has been pulled from Singularity Hub.

Cleaning the Apptainer image cache

We can remove images from the cache using the apptainer cache clean command. Running the command without any options will display a warning and ask you to confirm that you want to remove everything from your cache.

You can also remove specific images or all images of a particular type. Look at the output of apptainer cache clean --help for more information.

Cache location

By default, apptainer uses $HOME/.apptainer/cache as the location for the cache. You can change the location of the cache by setting the APPTAINER_CACHEDIR environment variable to the cache location you want to use.

Files and directories within as apptainer container

Apptainer also binds some directories from the host system where you are running the apptainer command into the container that you’re starting. Note that this bind process is not copying files into the running container, it is making an existing directory on the host system visible and accessible within the container environment. If you write files to this directory within the running container, when the container shuts down, those changes will persist in the relevant location on the host system.

There is a default configuration of which files and directories are bound into the container but ultimate control of how things are set up on the system where you are running apptainer is determined by the system administrator. As a result, this section provides an overview but you may find that things are a little different on the system that you’re running on.

One directory that is likely to be accessible within a container that you start is your home directory. You may also find that the directory from which you issued the apptainer command (the current working directory) is also mapped.

The mapping of file content and directories from a host system into an apptainer container is illustrated in the example below showing a subset of the directories on the host Linux system and in an apptainer container:

Host system:                                                      Apptainer container:
-------------                                                     ----------------------
/                                                                 /
├── bin                                                           ├── bin
├── etc                                                           ├── etc
│   ├── ...                                                       │   ├── ...
│   ├── group  ─> user's group added to group file in container ─>│   ├── group
│   └── passwd ──> user info added to passwd file in container ──>│   └── passwd
├── home                                                          ├── usr
│   └── jc1000 ───> user home directory made available ──> ─┐     ├── sbin
├── usr                 in container via bind mount         │     ├── home
├── sbin                                                    └────────>└── jc1000
└── ...                                                           └── ...

Questions and exercises: Files in apptainer containers

Q1: What do you notice about the ownership of files in a container started from the hello-world image? (e.g. take a look at the ownership of files in the root directory (/))

Exercise 1: In this container, try editing (for example using the editor vi which should be avaiable in the container) the /rawr.sh file. What do you notice?

If you’re not familiar with vi there are many quick reference pages online showing the main commands for using the editor, for example this one.

Exercise 2: In your home directory within the container shell, try and create a simple text file. Is it possible to do this? If so, why? If not, why not?! If you can successfully create a file, what happens to it when you exit the shell and the container shuts down?

Answers

A1: Use the ls -l command to see a detailed file listing including file ownership and permission details. You should see that most of the files in the / directory are owned by root, as you’d probably expect on any Linux system. If you look at the files in your home directory, they should be owned by you.

A Ex1: We’ve already seen from the previous answer that the files in / are owned by root so we wouldn’t expect to be able to edit them if we’re not the root user. However, if you tried to edit /rawr.sh you probably saw that the file was read only and, if you tried for example to delete the file you would have seen an error similar to the following: cannot remove '/rawr.sh': Read-only file system. This tells us something else about the filesystem. It’s not just that we don’t have permission to delete the file, the filesystem itself is read-only so even the root user wouldn’t be able to edit/delete this file. We’ll look at this in more detail shortly.

A Ex2: Within your home directory, you should be able to successfully create a file. Since you’re seeing your home directory on the host system which has been bound into the container, when you exit and the container shuts down, the file that you created within the container should still be present when you look at your home directory on the host system.

Binding additional host system directories to the container

You will sometimes need to bind additional host system directories into a container you are using over and above those bound by default. For example:

• There may be a shared dataset in a shard location that you need access to in the container
• You may require executables and software libraries in the container

The -B option to the apptainer command is used to specify additonal binds. For example, to bind the /work/z19/shared directory into a container you could use (note this directory is unlikely to exist on the host system you are using so you’ll need to test this using a different directory):

$ apptainer shell -B /work/z19/shared hello-world.sif
Singularity> ls /work/z19/shared

CP2K-regtest	    cube	     eleanor		   image256x192.pgm		kevin		    pblas			    q-e-qe-6.7 
ebe		    evince.simg	     image512x384.pgm	   low_priority.slurm           pblas.tar.gz	                                    q-qe
Q1529568	    edge192x128.pgm  extrae		   image768x1152.pgm		mkdir		    petsc			    regtest-ls-rtp_forCray
adrianj		    edge256x192.pgm  gnuplot-5.4.1.tar.gz  image768x768.pgm		moose.job	    petsc-hypre			    udunits-2.2.28.tar.gz
antlr-2.7.7.tar.gz  edge512x384.pgm  hj			   job-defmpi-cpe-21.03-robust	mrb4cab		    petsc-hypre-cpe21.03	    xios-2.5
cdo-archer2.sif     edge768x768.pgm  image192x128.pgm	   jsindt			paraver		    petsc-hypre-cpe21.03-gcc10.2.0

Note that, by default, a bind is mounted at the same path in the container as on the host system. You can also specify where a host directory is mounted in the container by separating the host path from the container path by a colon (:) in the option:

$ apptainer shell -B /work/z19/shared:/shared-data hello-world.sif
Singularity> ls /shared-data

CP2K-regtest	    cube	     eleanor		   image256x192.pgm		kevin		    pblas			    q-e-qe-6.7 
ebe		    evince.simg	     image512x384.pgm	   low_priority.slurm           pblas.tar.gz	                                    q-qe
Q1529568	    edge192x128.pgm  extrae		   image768x1152.pgm		mkdir		    petsc			    regtest-ls-rtp_forCray
adrianj		    edge256x192.pgm  gnuplot-5.4.1.tar.gz  image768x768.pgm		moose.job	    petsc-hypre			    udunits-2.2.28.tar.gz
antlr-2.7.7.tar.gz  edge512x384.pgm  hj			   job-defmpi-cpe-21.03-robust	mrb4cab		    petsc-hypre-cpe21.03	    xios-2.5
cdo-archer2.sif     edge768x768.pgm  image192x128.pgm	   jsindt			paraver		    petsc-hypre-cpe21.03-gcc10.2.0

You can also specify multiple binds to -B by separating them by commas (,).

You can also copy data into a container image at build time if there is some static data required in the image. We cover this later in the section on building apptainer containers.

Using Docker images with Apptainer

Apptainer can also start containers directly from Docker images, opening up access to a huge number of existing container images available on Docker Hub and other registries.

While apptainer doesn’t actually run a container using the Docker image (it first converts it to a format suitable for use by apptainer), the approach used provides a seamless experience for the end user. When you direct apptainer to run a container based on pull a Docker image, apptainer pulls the slices or layers that make up the Docker image and converts them into a single-file apptainer SIF image.

For example, moving on from the simple Hello World examples that we’ve looked at so far, let’s pull one of the official Docker Python images. We’ll use the image with the tag 3.9.6-slim-buster which has Python 3.9.6 installed on Debian’s Buster (v10) Linux distribution:

$ apptainer pull python-3.9.6.sif docker://python:3.9.6-slim-buster

INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 33847f680f63 done  
Copying blob b693dfa28d38 done  
Copying blob ef8f1a8cefd1 done  
Copying blob 248d7d56b4a7 done  
Copying blob 478d2dfa1a8d done  
Copying config c7d70af7c3 done  
Writing manifest to image destination
Storing signatures
2021/07/27 17:23:38  info unpack layer: sha256:33847f680f63fb1b343a9fc782e267b5abdbdb50d65d4b9bd2a136291d67cf75
2021/07/27 17:23:40  info unpack layer: sha256:b693dfa28d38fd92288f84a9e7ffeba93eba5caff2c1b7d9fe3385b6dd972b5d
2021/07/27 17:23:40  info unpack layer: sha256:ef8f1a8cefd144b4ee4871a7d0d9e34f67c8c266f516c221e6d20bca001ce2a5
2021/07/27 17:23:40  info unpack layer: sha256:248d7d56b4a792ca7bdfe866fde773a9cf2028f973216160323684ceabb36451
2021/07/27 17:23:40  info unpack layer: sha256:478d2dfa1a8d7fc4d9957aca29ae4f4187bc2e5365400a842aaefce8b01c2658
INFO:    Creating SIF file...

Note how we see apptainer saying that it’s “Converting OCI blobs to SIF format”. We then see the layers of the Docker image being downloaded and unpacked and written into a single SIF file. Once the process is complete, we should see the python-3.9.6.sif image file in the current directory.

We can now run a container from this image as we would with any other apptainer image.

Some image repositories you may be interested in are:

• NVIDIA’s NGC catalog for CUDA-accelerated software: https://catalog.ngc.nvidia.com/containers
• Singularity’s curated list of HPC-relevant containers: https://singularityhub.github.io/singularity-hpc

Building an image from a definition file

Building images can become quite in-depth depending on the features you are looking to include in your image file. We won’t be able to cover all of these details here today, but you can refer to MSI’s Apptainer Tutorial for more details.

As an illustrative example, let’s look at how we might build an image for the hmmer software we created a module for earlier in today’s material. Let’s further say that the specific tool we are most interested in using from this package is nhmmer, so we want our image to run that executable by default when using apptainer run.

An example definition file for this purpose might look like the following:

BootStrap: docker
From: ubuntu:20.04

%post
        apt update && apt install build-essential wget -y

        mkdir /opt/hmmer && cd /opt/hmmer
        wget http://eddylab.org/software/hmmer/hmmer.tar.gz
        tar zxf hmmer.tar.gz
        ln -s hmmer*/ src
        cd src
        ./configure --prefix=/opt/hmmer/install
        make
        make install

%environment
        export PATH=$PATH:/opt/hmmer/install/bin

%runscript
        nhmmer -h

%labels
    Author dunn0404
    Version v0.0.1

%help
    Example container running the nhmmer help documentation

Let’s take a look at the definition file:

• The first two lines still define the base image. In this case, the image ubuntu:20.04 from Docker Hub is used.
• %post are lines to execute inside the container to modify the base image. These commands are run once when the image is built, and never again. In this example, we are listing the steps that we would follow to install hmmer from source. Note that we use the -y flag for apt, to auto-accept the y/n prompt to install the proposed packages. We need to do this because the post section is run non-interactively.
• %environment is used to define environment variables available inside the container when it is run. Here we are setting the env variables required to find the executables we install for hmmer.
• Apptainer containers can be executable. %runscript define the actions to take when the container is executed, or when the apptainer run command is used. To illustrate the functionality, we will just run nhmmer -h to print the help text for nhmmer.
• %labels add custom metadata to the container.
• %help it is the container documentation: what it is and how to use it. Can be displayed using apptainer run-help

Note that we installed this software into the /opt directory. We could have installed the software anywhere we liked, as we are able to freely modify the contents of the eventual image. However, /opt is a good choice for installing software into most images because in the Linux filesystem, the /opt directory is intended as a place to install optional or add-on software packages. By using /opt we avoid the chance of accidentally overwriting critical files in other directories, and other users who might use this image later will know to look in /opt for user-installed software.

Save this definition file as hmmerInUbuntu.def. To build the container, just provide the definition file as argument:

apptainer build hmmerInUbuntu.sif hmmerInUbuntu.def

Now we can use the resulting image file using our familiar apptainer commands:

apptainer run ./hmmerInUbuntu.sif

# nhmmer :: search a DNA model, alignment, or sequence against a DNA database
# HMMER 3.4 (Aug 2023); http://hmmer.org/
# Copyright (C) 2023 Howard Hughes Medical Institute.
# Freely distributed under the BSD open source license.
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Usage: nhmmer [options] <query hmmfile|alignfile|seqfile> <target seqfile>

Basic options:
  -h : show brief help on version and usage

Options directing output:
  -o <f>             : direct output to file <f>, not stdout
  -A <f>             : save multiple alignment of all hits to file <f>
  --tblout <f>       : save parseable table of hits to file <f>
  --dfamtblout <f>   : save table of hits to file, in Dfam format <f>
  --aliscoresout <f> : save scores for each position in each alignment to <f>
  --hmmout <f>       : if input is alignment(s), write produced hmms to file <f>
  --acc              : prefer accessions over names in output
  --noali            : don't output alignments, so output is smaller
  --notextw          : unlimit ASCII text output line width
  --textw <n>        : set max width of ASCII text output lines  [120]  (n>=120)
...

If we want to run a different hmmer executable (or any other command available in the image), we can do so using exec:

apptainer exec ./hmmerInUbuntu.sif phmmer -h

# phmmer :: search a protein sequence against a protein database
# HMMER 3.2.1 (June 2018); http://hmmer.org/
# Copyright (C) 2018 Howard Hughes Medical Institute.
# Freely distributed under the BSD open source license.
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Usage: phmmer [-options] <seqfile> <seqdb>

Basic options:
  -h : show brief help on version and usage

Options directing output:
  -o <f>           : direct output to file <f>, not stdout
  -A <f>           : save multiple alignment of hits to file <f>
  --tblout <f>     : save parseable table of per-sequence hits to file <f>
  --domtblout <f>  : save parseable table of per-domain hits to file <f>
  --pfamtblout <f> : save table of hits and domains to file, in Pfam format <f>
...

To save the image and the definition file, we will need to copy hmmerInUbuntu.sif and hmmerInUbuntu.def to a safe location. This can be alongside the rest of your software and documentation installed into ~/software.install and ~/software.source in appropriate subdirectories, where the image is the software and the definition file is stored as the source.

References

• [1] Gregory M. Kurzer, Containers for Science, Reproducibility and Mobility: Singularity P2. Intel HPC Developer Conference, 2017. Available at: https://www.intel.com/content/dam/www/public/us/en/documents/presentation/hpc-containers-singularity-advanced.pdf
• [2] J. Cohen and A. Turner. Reproducible computational environments using containers: Introduction to Singularity. Version 2020.08a, August 2020. Carpentries Incubator. Available at: https://github.com/carpentries-incubator/singularity-introduction

Key Points

• Apptainer caches downloaded images so that an unchanged image isn’t downloaded again when it is requested using the apptainer pull command.

• You can free up space in the cache by removing all locally cached images or by specifying individual images to remove.

• Your current directory and home directory are usually available by default in a container.

• You have the same username and permissions in a container as on the host system.

• You can specify additional host system directories to be available in the container.

• Apptainer can start a container from a Docker image which can be pulled directly from Docker Hub.