Skip to content

Workspaces

Storage systems differ in terms of capacity, streaming bandwidth, IOPS rate, etc. Price and efficiency don't allow to have it all in one. That is why fast parallel filesystems at ZIH have restrictions with regards to lifetime and volume quota. The mechanism of using workspaces enables you to better manage your HPC data. It is common and used at a large number of HPC centers.

Note

A workspace is a directory, with an associated expiration date, created on behalf of a user in a certain filesystem.

Once the workspace has reached its expiration date, it gets moved to a hidden directory and enters a grace period. Once the grace period ends, the workspace is deleted permanently. The maximum lifetime of a workspace depends on the storage system. All workspaces can be extended a certain amount of times.

Tip

Use the faster filesystems if you need to write temporary data in your computations, and use the capacity oriented filesystems if you only need to read data for your computations. Please keep track of your data and move it to a capacity oriented filesystem after the end of your computations.

Workspace Management

Workspace Lifetimes

Since the workspace filesystems are intended for different use cases and thus differ in performance, their granted timespans differ accordingly. The maximum lifetime and number of renewals are provided in the following table.

Filesystem (use with parameter --filesystem=<filesystem>) Max. Duration in Days Extensions Keeptime Filesystem Feature
horse 100 10 30
walrus 100 10 60
beegfs_global0 (deprecated) 30 2 30 fs_beegfs_global0
beegfs 30 2 30 fs_beegfs
Settings for Workspace Filesystems.

Note

Currently, not all filesystems are available on all of our five clusters. The page Working Filesystems provides the necessary information.

End-of-life filesystems

The filesystems warm_archive, ssd and scratch will be switched off end of 2023. Do not use them anymore!

Filesystem (use with parameter --filesystem=<filesystem>) Duration, days Extensions Filesystem Feature Remarks
scratch (default) 100 10 fs_lustre_scratch2 Scratch filesystem (/lustre/scratch2, symbolic link: /scratch) with high streaming bandwidth, based on spinning disks
ssd 30 2 fs_lustre_ssd High-IOPS filesystem (/lustre/ssd, symbolic link: /ssd) on SSDs.
warm_archive 365 2 30 fs_warm_archive_ws

List Available Filesystems

To list all available filesystems for using workspaces, you can either invoke ws_list -l or ws_find --list. Since not all workspace filesystems are available on all HPC systems, the concrete output differs depending on the system you are logged in. The page Working Filesystems provides information which filesystem is available on which cluster.

marie@login.barnard$ ws_list -l
available filesystems:
horse (default)
walrus
marie@login.alpha$ ws_list -l
available filesystems:
ssd
beegfs_global0
beegfs (default)
marie@login.romeo$ ws_list -l
available filesystems:
horse
marie@login.taurus$ ws_list -l
available filesystems:
scratch (default)
warm_archive
ssd
beegfs_global0
beegfs

Default filesystem

The output of the commands ws_find --list and ws_list -l will indicate the default filesystem. If you prefer another filesystem (cf. section List Available Filesystems), you have to explictly provide the option --filesystem=<filesystem> to the workspace commands. If the default filesystems is the one you want to work with, you can omit this option.

List Current Workspaces

The command ws_list lists all your currently active (,i.e, not expired) workspaces, e.g.

marie@login$ ws_list
id: test-workspace
    workspace directory  : /data/horse/ws/marie-test-workspace
    remaining time       : 89 days 23 hours
    creation time        : Wed Dec  6 14:46:12 2023
    expiration date      : Tue Mar  5 14:46:12 2024
    filesystem name      : horse
    available extensions : 10

The output of ws_list can be customized via several options. The following switch tab provides a overview of some of these options. All available options can be queried by ws_list --help.

marie@login$ ws_list --filesystem=walrus
id: marie-numbercrunch
    workspace directory  : /data/walrus/ws/marie-numbercrunch
    remaining time       : 89 days 23 hours
    creation time        : Wed Dec  6 14:49:55 2023
    expiration date      : Tue Mar  5 14:49:55 2024
    filesystem name      : walrus
    available extensions : 2
marie@login$ ws_list -v
id: test-workspace
    workspace directory  : /data/horse/ws/marie-test-workspace
    remaining time       : 89 days 23 hours
    creation time        : Wed Dec  6 14:46:12 2023
    expiration date      : Tue Mar  5 14:46:12 2024
    filesystem name      : scratch
    available extensions : 10
    acctcode             : p_numbercrunch
    reminder             : Tue Feb 27 14:46:12 2024
    mailaddress          : marie@tu-dresden.de
marie@login$ ws_list -t
id: test-workspace
    workspace directory  : /data/horse/ws/marie-test-workspace
    remaining time       : 89 days 23 hours
    available extensions : 10
id: numbercrunch
    workspace directory  : /data/walrus/ws/marie-numbercrunch
    remaining time       : 89 days 23 hours
    available extensions : 2
marie@login$ ws_list -s
test-workspace
numbercrunch

You can list your currently allocated workspace by remaining time. This is especially useful for housekeeping tasks, such as extending soon expiring workspaces if necessary.

marie@login$ ws_list -R -t
id: test-workspace
     workspace directory  : /data/horse/ws/marie-test-workspace
     remaining time       : 89 days 23 hours
     available extensions : 10
id: marie-numbercrunch
    workspace directory  : /data/walrus/ws/marie-numbercrunch
    remaining time       : 89 days 23 hours
    available extensions : 2

Allocate a Workspace

To allocate a workspace in one of the listed filesystems, use ws_allocate. It is necessary to specify a unique name and the duration (in days) of the workspace.

ws_allocate: [options] workspace_name duration

Options:
  -h [ --help]               produce help message
  -V [ --version ]           show version
  -d [ --duration ] arg (=1) duration in days
  -n [ --name ] arg          workspace name
  -F [ --filesystem ] arg    filesystem
  -r [ --reminder ] arg      reminder to be sent n days before expiration
  -m [ --mailaddress ] arg   mailaddress to send reminder to (works only with tu-dresden.de mails)
  -x [ --extension ]         extend workspace
  -u [ --username ] arg      username
  -g [ --group ]             group workspace
  -c [ --comment ] arg       comment

Name of a workspace

The workspace name should help you to remember the experiment and data stored here. It has to be unique on a certain filesystem. On the other hand it is possible to use the very same name for workspaces on different filesystems.

The simple way to allocate a workspace is calling ws_allocate command with two arguments, where the first specifies the workspace name and the second the duration. This allocates a workspace on the default filesystem with no e-mail reminder.

marie@login$ ws_allocate test-workspace 90
Info: creating workspace.
/data/horse/ws/marie-test-workspace
remaining extensions  : 10
remaining time in days: 90

In order to allocate a workspace on a non-default filesystem, the option --filesystem=<filesystem> is required.

marie@login$ ws_allocate --filesystem=walrus test-workspace 99
Info: creating workspace.
/lustre/ssd/ws/marie-test-workspace
remaining extensions  : 2
remaining time in days: 99

This command will create a workspace with the name test-workspace on the /horse filesystem (default) with a duration of 99 days and send an e-mail reminder. The e-mail reminder will be sent every day starting 7 days prior to expiration. We strongly recommend setting this e-mail reminder.

marie@login$ ws_allocate --reminder=7 --mailaddress=marie@tu-dresden.de test-workspace 99
Info: creating workspace.
/horse/ws/marie-test-workspace
remaining extensions  : 10
remaining time in days: 99

Please refer to the section Cooperative Usage for group workspaces.

Extension of a Workspace

The lifetime of a workspace is finite and different filesystems (storage systems) have different maximum durations. The life time of a workspace can be adjusted multiple times, depending on the filesystem. You can find the concrete values in the section settings for workspaces.

Use the command ws_extend [-F filesystem] workspace days to extend your workspace:

marie@login$ ws_extend -F scratch test-workspace 100
Info: extending workspace.
/data/horse/ws/marie-test-workspace
remaining extensions  : 1
remaining time in days: 100

E-mail reminder settings are retained. I.e., previously set e-mail alerts apply to the extended workspace, too.

Attention

With the ws_extend command, a new duration for the workspace is set. The new duration is not added to the remaining lifetime!

This means when you extend a workspace that expires in 90 days with the command

marie@login$ ws_extend -F scratch test-workspace 40

it will now expire in 40 days, not in 130 days!

Send Reminder for Workspace Expiration Date

We strongly recommend using one of the two provided ways to ensure that the expiration date of a workspace is not forgotten.

Send Daily Reminder

An e-mail reminder can be set at workspace allocation using

ws_allocate --reminder=<N> --mailaddress=<mail> [...]

This will send an e-mail every day starting N days prior to the expiration date. See the example above for reference.

If you missed setting an e-mail reminder at workspace allocation, you can add a reminder later, e.g.

# initial allocation
marie@login$ ws_allocate --name=FancyExp --duration=17
[...]
# add e-mail reminder
marie@login$ ws_allocate --name=FancyExp --duration=17 --reminder=7 --mailaddress=marie@dlr.de
--extension

This will reallocate the workspace, which counts against your maximum number of reallocations (Note: No data is deleted, but the database entry is modified).

Send Calender Invitation

The command ws_send_ical sends you an ical event on the expiration date of a specified workspace. This calender invitation can be further managed according to your personal preferences. The syntax is as follows:

ws_send_ical [--filesystem <filesystem>] --mail <e-mail-address> --workspace <workspace name>

Deletion of a Workspace

To delete a workspace use the ws_release command. It is mandatory to specify the name of the workspace and the filesystem in which it is allocated:

marie@login$ ws_release --filesystem=horse --name=test-workspace

You can list your already released or expired workspaces using the ws_restore --list command.

marie@login$ ws_restore --list
horse:
marie-test-workspace-1701873807
    unavailable since Wed Dec  6 15:43:27 2023
walrus:
marie-numbercrunch-1701873907
    unavailable since Wed Dec  6 15:45:07 2023

In this example, the user marie has two inactive, i.e., expired, workspaces namely test-workspace in horse, as well as numbercrunch in the walrus filesystem. The command ws_restore --list lists the name of the workspace and its expiration date. As you can see, the expiration date is added to the workspace name as Unix timestamp.

Deleting data in in an expired workspace

If you are short on quota, you might want to delete data in expired workspaces since it counts to your quota. Expired workspaces are moved to a hidden directory named .removed. The access rights remain unchanged. I.e., you can delete the data inside the workspace directory but you must not delete the workspace directory itself!

Expire Process

The clean up process of expired workspaces is automatically handled by a so-called expirer process. It performs the following steps once per day and filesystem:

  • Check for remaining life time of all workspaces.
    • If the workspaces expired, move it to a hidden directory so that it becomes inactive.
  • Send reminder e-mails to users if the reminder functionality was configured for their particular workspaces.
  • Scan through all workspaces in grace period.
    • If a workspace exceeded the grace period, the workspace and its data are permanently deleted.

Restoring Expired Workspaces

At expiration time your workspace will be moved to a special, hidden directory. For a month, you can still restore your data into an existing workspace.

Warning

When you release a workspace by hand, it will not receive a grace period and be permanently deleted the next day. The advantage of this design is that you can create and release workspaces inside jobs and not flood the filesystem with data no one needs anymore in the hidden directories (when workspaces are in the grace period).

Use

marie@login$ ws_restore --list
horse:
marie-test-workspace-1701873807
    unavailable since Wed Dec  6 15:43:27 2023
walrus:
marie-numbercrunch-1701873907
    unavailable since Wed Dec  6 15:45:07 2023

to get a list of your expired workspaces, and then restore them like that into an existing, active workspace 'new_ws':

marie@login$ ws_restore --filesystem=horse marie-test-workspace-1701873807 new_ws

The expired workspace has to be specified by its full name as listed by ws_restore --list, including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified). The target workspace, on the other hand, must be given with just its short name, as listed by ws_list, without the username prefix.

Both workspaces must be on the same filesystem. The data from the old workspace will be moved into a directory in the new workspace with the name of the old one. This means a fresh workspace works as well as a workspace that already contains data.

Linking Workspaces in HOME

It might be valuable to have links to personal workspaces within a certain directory, e.g., your home directory. The command ws_register DIR will create and manage links to all personal workspaces within in the directory DIR. Calling this command will do the following:

  • The directory DIR will be created if necessary.
  • Links to all personal workspaces will be managed:
    • Create links to all available workspaces if not already present.
    • Remove links to released workspaces.

Remark: An automatic update of the workspace links can be invoked by putting the command ws_register DIR in your personal shell configuration file (e.g., .bashrc).

How to use Workspaces

There are three typical options for the use of workspaces:

Per-Job Storage

The idea of a "workspace per-job storage" addresses the need of a batch job for a directory for temporary data which can be deleted afterwards. To help you to write your own (Slurm) job file, suited to your needs, we came up with the following example (which works for the program g16).

Hint

Please do not blind copy the example, but rather take the essential idea and concept and adjust it to your needs and workflow, e.g.

  • adopt Slurm options for ressource specification,
  • insert the path to your input file,
  • specify what software you want to load,
  • and call the actual software to do your computation.

Using temporary workspaces for I/O intensive tasks

#!/bin/bash

#SBATCH --time=48:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
## The optional constraint for the filesystem feature depends
## on the filesystem on which you want to use a workspace.
## Documentation here https://compendium.hpc.tu-dresden.de/jobs_and_resources/slurm/#filesystem-features
#SBATCH --constraint=fs_lustre_ssd
#SBATCH --cpus-per-task=24

# Load the software you need here
module purge
module load <modules>

# The path to where your input file is located
INPUTFILE="/path/to/my/inputfile.data"
test ! -f "${INPUTFILE}" && echo "Error: Could not find the input file ${INPUTFILE}" && exit 1

# The workspace where results from multiple expirements will be saved for later analysis
RESULT_WSDIR="/path/to/workspace-experiments-results"
test -z "${RESULT_WSDIR}" && echo "Error: Cannot find workspace ${RESULT_WSDIR}" && exit 1

# Allocate workspace for this job. Adjust time span to time limit of the job (-d <N>).
WSNAME=computation_$SLURM_JOB_ID
export WSDDIR=$(ws_allocate --filesystem=ssd --name=${WSNAME} --duration=2)
echo ${WSDIR}

# Check allocation
test -z "${WSDIR}" && echo "Error: Cannot allocate workspace ${WSDIR}" && exit 1

# Change to workspace directory
cd ${WSDIR}

# Adjust the following line to invoke the program you want to run
srun <application> < "${INPUTFILE}" > logfile.log

# Move result and log files of interest to directory named 'results'. This directory and its
# content will be saved in another storage location for later analysis. All files and
# directories will be deleted right away at the end of this job file.
mkdir results
cp <results and log files> results/

# Save result files in a general workspace (RESULT_WSDIR, s.a.) holding results from several
# experiments.
# Compress results with bzip2 (which includes CRC32 Checksums).
bzip2 --compress --stdout -4 "${WSDIR}/results" > ${RESULT_WSDIR}/gaussian_job-${SLURM_JOB_ID}.bz2
RETURN_CODE=$?
COMPRESSION_SUCCESS="$(if test ${RETURN_CODE} -eq 0; then echo 'TRUE'; else echo 'FALSE'; fi)"

# Clean up workspace
if [ "TRUE" = ${COMPRESSION_SUCCESS} ]; then
    test -d ${WSDIR} && rm -rf ${WSDIR}/*
    # Reduces grace period to 1 day!
    ws_release -F ssd ${WSNAME}
else
    echo "Error with compression and writing of results"
    echo "Please check the folder \"${WSDIR}\" for any partial(?) results."
    exit 1
fi

Data for a Campaign

For a series of jobs or calculations that work on the same data, you should allocate a workspace once, e.g., in horse for 100 days:

marie@login$ ws_allocate --filesystem=horse my_scratchdata 100
Info: creating workspace.
/data/horse/ws/marie-my_scratchdata
remaining extensions  : 10
remaining time in days: 99

You can grant your project group access rights:

marie@login$ chmod g+wrx /data/horse/ws/marie-my_scratchdata

And verify it with:

marie@login$ ls -la /data/horse/ws/marie-my_scratchdata
total 8
drwxrwx--- 2 marie    hpcsupport 4096 Jul 10 09:03 .
drwxr-xr-x 5 operator adm        4096 Jul 10 09:01 ..

Mid-Term Storage

For data that rarely changes but consumes a lot of space, the walrus filesystem can be used. Note that this is mounted read-only on the compute nodes, so you cannot use it as a work directory for your jobs!

marie@login$ ws_allocate --filesystem=walrus my_inputdata 100
/data/walrus/ws/marie-my_inputdata
remaining extensions  : 2
remaining time in days: 100

Cooperative Usage (Group Workspaces)

When a workspace is created with the option -g, --group, it gets a group workspace that is visible to others (if in the same group) via ws_list -g.

Choose group

If you are member of multiple groups, then the group workspace is visible for your primary group. You can list all groups you belong to via groups, and the first entry is your primary group.

Nevertheless, you can create a group workspace for any of your groups following these two steps:

  1. Change to the desired group using newgrp <other-group>.
  2. Create the group workspace as usual, i.e., ws_allocate --group [...]

The page on Sharing Data provides information on how to grant access to certain colleagues and whole project groups.

Allocate and list group workspaces

If Marie wants to share results and scripts in a workspace with all of her colleagues in the project p_number_crunch, she can allocate a so-called group workspace.

marie@login$ ws_allocate --group --name=numbercrunch --duration=30
Info: creating workspace.
/data/horse/ws/marie-numbercrunch
remaining extensions  : 10
remaining time in days: 30

This workspace directory is readable for the group, e.g.,

marie@login$ ls -ld /data/horse/ws/marie-numbercrunch
drwxr-x--- 2 marie p_number_crunch 4096 Mar  2 15:24 /data/horse/ws/marie-numbercrunch

All members of the project group p_number_crunch can now list this workspace using ws_list -g and access the data (read-only).

martin@login$ ws_list -g -t
id: numbercrunch
     workspace directory  : /data/horse/ws/marie-numbercrunch
     remaining time       : 29 days 23 hours
     available extensions : 10

FAQ and Troubleshooting

Q: I am getting the error Error: could not create workspace directory!

A: Please check the "locale" setting of your SSH client. Some clients (e.g. the one from Mac) set values that are not valid on our ZIH systems. You should overwrite LC_CTYPE and set it to a valid locale value like export LC_CTYPE=de_DE.UTF-8.

A list of valid locales can be retrieved via locale -a.

Please use language_CountryCode.UTF-8 (or plain) settings. Avoid "iso" codepages!

Examples:

Language Code
Chinese - Simplified zh_CN.UTF-8
English en_US.UTF-8
French fr_FR.UTF-8
German de_DE.UTF-8

Q: I am getting the error Error: target workspace does not exist! when trying to restore my workspace.

A: The workspace you want to restore into is either not on the same filesystem or you used the wrong name. Use only the short name that is listed after id: when using ws_list. See section restoring expired workspaces.


Q: I forgot to specify an e-mail reminder when allocating my workspace. How can I add the e-mail alert functionality to an existing workspace?

A: You can add the e-mail alert by "overwriting" the workspace settings via

marie@login$ ws_allocate --extension --mailaddress=<mail address> --reminder=<days> \
             --name=<workspace-name> --duration=<duration> --filesystem=<filesystem>

This will lower the remaining extensions by one.