Using Containers in HPC with Apptainer (formerly Singularity)
Quick Start
After having successfully installed Apptainer on your system the initial step is to run your first container.
apptainer run docker://ghcr.io/apptainer/lolcow
Output
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob 5ca731fc36c2 skipped: already exists
Copying blob 16ec32c2132b skipped: already exists
Copying config fd0daa4d89 done |
Writing manifest to image destination
2024/04/03 14:21:08 info unpack layer: sha256:16ec32c2132b43494832a05f2b02f7a822479f8250c173d0ab27b3de78b2f058
2024/04/03 14:21:10 info unpack layer: sha256:5ca731fc36c28789c5ddc3216563e8bfca2ab3ea10347e07554ebba1c953242e
INFO: Creating SIF file...
______________________________
< Wed Apr 3 14:21:16 CEST 2024 >
------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
The leading docker://
part instructs Apptainer to look for the image in
Docker Hub.
With a leading library://
in the image name Apptainer looks for the image in
the Singularity Cloud Library.
This is the equivalent to Docker Hub in the
Apptainer world.
Let's try another example by running an Ubuntu image using Apptainer. Therefore, let us use the
apptainer run docker://ubuntu:22.04
Output
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob bccd10f490ab done |
Copying config ca2b0f2696 done |
Writing manifest to image destination
2024/04/03 14:22:05 info unpack layer: sha256:bccd10f490ab0f3fba61b193d1b80af91b17ca9bdca9768a16ed05ce16552fcb
INFO: Creating SIF file...
Apptainer>
Using this command we are presented a shell inside the Ubuntu Apptainer
container.
We could also pull a similar Ubuntu image
library/default/ubuntu:22.04
from the Singularity Cloud Library:
apptainer run library://ubuntu:22.04
Output
INFO: Downloading library image
28.4MiB / 28.4MiB [============================================================] 100 % 5.6 MiB/s 0s
Singularity>
In the following we will keep using Docker Hub for images.
Once inside the container, you are the same user as you are on the host system.
Apptainer> whoami
christianhueser
Apptainer automatically makes your home directory (/home/$USER
) available
in the container.
Unlike with Podman, it is not necessary to explicitly mount the directories.
They are made available by default.
With Apptainer, you can easily reuse existing images from Docker Hub
and run them as an Apptainer container.
Let us run the python:3.11
Docker Hub
image as an Apptainer container.
Therefore, we pull
the image first.
apptainer pull docker://python:3.12
Output
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob 63941d09e532 done |
Copying blob 71215d55680c done |
Copying blob 5f899db30843 done |
Copying blob 567db630df8d done |
Copying blob d68cd2123173 done |
Copying blob 3cb8f9c23302 done |
Copying blob 097431623722 done |
Copying blob 09527fa4de8d done |
Copying config 65324f103f done |
Writing manifest to image destination
2024/04/03 14:48:54 info unpack layer: sha256:71215d55680cf0ab2dcc0e1dd65ed76414e3fb0c294249b5b9319a8fa7c398e4
2024/04/03 14:48:58 info unpack layer: sha256:3cb8f9c23302e175d87a827f0a1c376bd59b1f6949bd3bc24ab8da0d669cdfa0
2024/04/03 14:48:59 info unpack layer: sha256:5f899db30843f8330d5a40d1acb26bb00e93a9f21bff253f31c20562fa264767
2024/04/03 14:49:04 info unpack layer: sha256:567db630df8d441ffe43e050ede26996c87e3b33c99f79d4fba0bf6b7ffa0213
2024/04/03 14:49:16 info unpack layer: sha256:d68cd2123173935e339e3feb56980a0aefd7364ad43ca2b9750699e60fbf74c6
2024/04/03 14:49:17 info unpack layer: sha256:63941d09e5322b88281f3a325eff9ced5bf2ee45b691aaf8ec2f829bafbd8021
2024/04/03 14:49:18 info unpack layer: sha256:097431623722383300c03bb41fd162d32346bf6a02a053263f51969eb9746e3d
2024/04/03 14:49:18 info unpack layer: sha256:09527fa4de8dd73399164c307942cc43652a01fc2bb370e38ae0f806b42b4b18
INFO: Creating SIF file...
This command downloads the image from Docker Hub (docker://
) and converts it
into the Apptainer specific image format called SIF file
.
A file called python_3.12.sif
was created in your current directory.
Run it as shown below.
apptainer run python_3.12.sif
Output
Python 3.12.2 (main, Mar 12 2024, 11:02:14) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
Apptainer Definition Files
As demonstrated in the Podman build lesson we want to build our first custom
Apptainer image as well.
Therefore, we need to create the Apptainer definition file, i.e. the
equivalent to Dockerfile
.
We will recreate the lolcow image used for the first run
command in this episode.
Therefore, we create a file called lolcow.def
using the editor of your choice.
This file contains the construction manual for the Apptainer image.
Bootstrap: docker
From: ubuntu:22.04
%post
apt-get -qy update
apt-get -qy install fortune cowsay lolcat
%environment
export LC_ALL=C
export PATH=/usr/games:$PATH
%runscript
fortune | cowsay | lolcat
Build the image using the apptainer build
command.
apptainer build lolcow.sif lolcow.def
Output
INFO: Starting build...
Getting image source signatures
Copying blob bccd10f490ab done |
Copying config ca2b0f2696 done |
Writing manifest to image destination
2024/04/03 15:00:41 info unpack layer: sha256:bccd10f490ab0f3fba61b193d1b80af91b17ca9bdca9768a16ed05ce16552fcb
INFO: Running post scriptlet
+ apt-get -qy update
[...]
+ apt-get -qy install fortune cowsay lolcat
[...]
INFO: Adding environment to container
INFO: Adding runscript
INFO: Creating SIF file...
INFO: Build complete: lolcow.sif
This command creates the file lolcow.sif
.
It is run using the apptainer run
command.
apptainer run lolcow.sif
Output
_______________________________________
/ If you sow your wild oats, hope for a \
\ crop failure. /
---------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Let's take apart the Apptainer definition file.
Header
Each Apptainer definition file needs to start with the header part consisting
of at least the Bootstrap
keyword.
In our example we use the docker
bootstrap agent telling Apptainer to
get the image from a Docker registry.
This agent needs to be combined with the From
keyword to let Apptainer know,
which base image to use.
Bootstrap: docker
From: ubuntu:22.04
Usage of Other Registries
If you want to use another registry, e.g. the Helmholtz Codebase GitLab Container Registry, it is easily possible. Specify the full name of the image according to the GitLab Container Registry naming convention as described in the documentation.
Generic example:
Bootstrap: docker
From: <registry URL>/<namespace>/<project>/<image>
Helmholtz Codebase GitLab example:
Bootstrap: docker
From: hcr.helmholtz.cloud/hueser93/my-helloworld-image-project
apptainer remote login -u hueser93 docker://hcr.helmholtz.cloud
apptainer build hello.sif hello.def
apptainer run hello.sif
A list of preferred bootstrap agents is available here.
Sections
The main content of the definition file is broken into sections. In our example we used three different sections:
%post
- In this section you can download files from the internet with tools like
git
,wget
orpip
, - You can install new software and libraries,
- Create configuration files,
- Create files and directories, etc.
- In this section you can download files from the internet with tools like
%environment
- This section allows you to define environment variables which are set at runtime.
- These variables are not set at build time,
when running the
apptainer build
command.
%runscript
- The commands specified in this section are executed when the container
image is run. (
apptainer run
)
- The commands specified in this section are executed when the container
image is run. (
Please refer to the official documentation for a complete list of available sections and their usage.
Apptainer vs. Podman
Podman | Apptainer | |
---|---|---|
Isolation from host | Shares little by default. | Isolation not the primary focus. By default shares most everything. |
Supported Host Operating Systems (OS) | Windows, Mac, Linux | Linux |
Data Persistence | No host filesystem available by default. | Writable bind-mounts of the user home directory are automatically created. |
Primary target group | Developers, DevOps | Scientific Application Users/Developers |
HPC | Not suitable for HPC: requires elevated permissions | Integrates well with MPI, GPUs, Infiniband and Schedulers (e.g. SLURM) |
Ecosystem | Larger ecosystem; more sophisticated registries and preconfigured images | Smaller ecosystem; is able to use Docker images. |