ALERT! Warning: your browser isn't supported. Please install a modern one, like Firefox, Opera, Safari, Chrome or the latest Internet Explorer. Thank you!
Startseite » ... » Zentrale Einrichtungen  » ZIH  » Wiki
phone prefix: +49 351 463.....

HPC Support

Operation Status

Ulf Markwardt: 33640
Claudia Schmidt: 39833 hpcsupport@zih.tu-dresden.de

Login and project application

Phone: 40000
Fax: 42328
servicedesk@tu-dresden.de

You are here: Compendium » Containers

Singularity

If you wish to containerize your workflow/applications, you can use Singularity containers on Taurus. As opposed to Docker, this solution is much more suited to being used in an HPC environment, existing Docker containers can easily be converted.

Website: http://singularity.lbl.gov

FAQ: http://singularity.lbl.gov/faq

It is available on Taurus after loading the module singularity.

Container creation

Since bootstrapping a new container requires access to system-level tools and thus root privileges, it is not possible for users to generate new custom containers on Taurus directly. You can, however, create container image files and import an existing container from e.g. Docker. In case you wish to bootstrap a new container, you can do so on your own local machine where you have the necessary privileges and then simply copy your container file to Taurus and use it there.

Creating an image file

singularity create --size 8192 my-container.img

This will create an empty 8 GiB image file in the current directory.

Bootstrapping a container

NOTE: this must be done on your own machine with root rights

The following example bootstraps a basic CentOS 7 into your image file. First, you have to create a definition file and save it e.g. as "centos7.def":
BootStrap: yum
OSVersion: 7
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
Include: yum

# If you want the updates (available at the bootstrap date) to be installed
# inside the container during the bootstrap instead of the General Availability
# point release (7.x) then uncomment the following line
#UpdateURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/updates/$basearch/

%runscript
    echo "This is what happens when you run the container..."

%post
    echo "Hello from inside the container"
    yum -y install vim-minimal

Then you can call:
singularity bootstrap my-container.img centos7.def

And it will install CentOS7 inside your container according to your definition file.

More examples of definition files can be found at https://github.com/singularityware/singularity/tree/master/examples

Importing a docker container

You first have to export your Docker container to a .tar archive:

docker export my_docker_container > my-docker-container.tar

Then you can import the tar into your empty singularity image file:

cat my-docker-container.tar | singularity import my-container.img

Note that you can also import an image from the Docker repository (Docker Hub):

singularity import my-container.img docker://ubuntu:latest

As opposed to bootstrapping a container, importing from Docker does not require root privileges and so works on Taurus directly.

Using the containers

Entering a shell in your container

A read-only shell can be entered as follows:

singularity shell my-container.img

If you wish, for instance, to install additional packages, you have to use the -w parameter to enter your container with it being writable. This, again, must be done on a system where you have the necessary privileges, otherwise you can only edit files that your user has the permissions for. E.g:

singularity shell -w my-container.img
Singularity.my-container.img> yum install htop

The -w parameter should only be used to make permanent changes to your container, not for your productive runs (it can only be used writeable by one user at the same time). You should write your output to the usual Taurus file systems like /scratch.

By default, Singularity already binds your home directory. In order to make paths like /scratch available inside your container, you can use the -B (bind) parameter. Note that the directory (mount-point) must already exist in your image file, so you best include a "mkdir /scratch" in your bootstrapping process.
singularity shell -B /scratch:/scratch my-container.img

Note that writing to directories bound with -B does not require the -w parameter since they are not part of your container image.

Also note that we already defined bind paths for /scratch, /projects and /sw in our global singularity.conf, so you needn't use the -B parameter for those. Just make sure you create the empty directory in your image, if necessary.

Launching applications in your container

While the "shell" command can be useful for tests and setup, you can also launch your applications inside the container directly using "exec":
singularity exec my-container.img /opt/myapplication/bin/run_myapp

This can be useful if you wish to create a wrapper script that transparently calls a containerized application for you. E.g.:
#!/bin/bash

X=`which singularity 2>/dev/null`
if [ "z$X" = "z" ] ; then
        echo "Singularity not found. Is the module loaded?"
        exit 1
fi

singularity exec /scratch/p_myproject/my-container.img /opt/myapplication/run_myapp "$@"

Use-cases

One common use-case for containers is that you need an operating system with a newer GLIBC version than what is available on Taurus. Since the bullx Linux on Taurus is still based on RHEL6 having a rather dated GLIBC version 2.12, some binary-distributed applications won't work on that anymore. You can use one of our pre-made CentOS 7 container images (/scratch/singularity/centos7.img) to circumvent this problem. Example:

$ singularity exec /scratch/singularity/centos7.img ldd --version
ldd (GNU libc) 2.17