Skip to content


Storage systems differ in terms of capacity, streaming bandwidth, IOPS rate, etc. Price and efficiency don't allow to have it all in one. That is why fast parallel filesystems at ZIH have restrictions with regards to age of files and quota. The mechanism of workspaces enables you to better manage your HPC data. It is common and used at a large number of HPC centers.


A workspace is a directory, with an associated expiration date, created on behalf of a user in a certain filesystem.

Once the workspace has reached its expiration date, it gets moved to a hidden directory and enters a grace period. Once the grace period ends, the workspace is deleted permanently. The maximum lifetime of a workspace depends on the storage system. All workspaces can be extended a certain amount of times.


Use the faster filesystems if you need to write temporary data in your computations, and use the capacity oriented filesystems if you only need to read data for your computations. Please keep track of your data and move it to a capacity oriented filesystem after the end of your computations.

Workspace Management

List Available Filesystems

To list all available filesystems for using workspaces, you can either invoke ws_list -l or ws_find -l, e.g.,

marie@login$ ws_find -l
available filesystems:
scratch (default)

Default is scratch

The default filesystem is scratch. If you prefer another filesystem (cf. section List Available Filesystems), you have to explictly provide the option -F <fs> to the workspace commands.

List Current Workspaces

The command ws_list lists all your currently active (,i.e, not expired) workspaces, e.g.

marie@login$ ws_list
id: test-workspace
     workspace directory  : /scratch/ws/0/marie-test-workspace
     remaining time       : 89 days 23 hours
     creation time        : Thu Jul 29 10:30:04 2021
     expiration date      : Wed Oct 27 10:30:04 2021
     filesystem name      : scratch
     available extensions : 10

The output of ws_list can be customized via several options. The following switch tab provides a overview of some of these options. All available options can be queried by ws_list --help.

marie@login$ ws_list --filesystem scratch_fast
id: numbercrunch
     workspace directory  : /lustre/ssd/ws/marie-numbercrunch
     remaining time       : 2 days 23 hours
     creation time        : Thu Mar  2 14:15:33 2023
     expiration date      : Sun Mar  5 14:15:33 2023
     filesystem name      : ssd
     available extensions : 2
marie@login$ ws_list -v
id: test-workspace
     workspace directory  : /scratch/ws/0/marie-test-workspace
     remaining time       : 89 days 23 hours
     creation time        : Thu Jul 29 10:30:04 2021
     expiration date      : Wed Oct 27 10:30:04 2021
     filesystem name      : scratch
     available extensions : 10
     acctcode             : p_numbercrunch
     reminder             : Sat Oct 20 10:30:04 2021
     mailaddress          :
marie@login$ ws_list -t
id: test-workspace
     workspace directory  : /scratch/ws/0/marie-test-workspace
     remaining time       : 89 days 23 hours
     available extensions : 10
id: foo
     workspace directory  : /scratch/ws/0/marie-foo
     remaining time       : 3 days 22 hours
     available extensions : 10
marie@login$ ws_list -s

You can list your currently allocated workspace by remaining time. This is especially useful for housekeeping tasks, such as extending soon expiring workspaces if necessary.

marie@login$ ws_list -R -t
id: test-workspace
     workspace directory  : /scratch/ws/0/marie-test-workspace
     remaining time       : 89 days 23 hours
     available extensions : 10
id: foo
     workspace directory  : /scratch/ws/0/marie-foof
     remaining time       : 3 days 22 hours
     available extensions : 10

Allocate a Workspace

To allocate a workspace in one of the listed filesystems, use ws_allocate. It is necessary to specify a unique name and the duration of the workspace.

ws_allocate: [options] workspace_name duration

  -h [ --help]               produce help message
  -V [ --version ]           show version
  -d [ --duration ] arg (=1) duration in days
  -n [ --name ] arg          workspace name
  -F [ --filesystem ] arg    filesystem
  -r [ --reminder ] arg      reminder to be sent n days before expiration
  -m [ --mailaddress ] arg   mailaddress to send reminder to (works only with mails)
  -x [ --extension ]         extend workspace
  -u [ --username ] arg      username
  -g [ --group ]             group workspace
  -c [ --comment ] arg       comment


marie@login$ ws_allocate -F scratch -r 7 -m test-workspace 90
Info: creating workspace.
remaining extensions  : 10
remaining time in days: 90

This will create a workspace with the name test-workspace on the /scratch filesystem for 90 days with an email reminder for 7 days before the expiration.

Email reminder

Setting the reminder to 7 means you will get a reminder email on every day starting 7 days prior to expiration date. We strongly recommend to set this email reminder.

Name of a workspace

The workspace name should help you to remember the experiment and data stored here. It has to be unique on a certain filesystem. On the other hand it is possible to use the very same name for workspaces on different filesystems.

Please refer to the section section Cooperative Usage for group workspaces.

Extension of a Workspace

The lifetime of a workspace is finite and different filesystems (storage systems) have different maximum durations. A workspace can be extended multiple times, depending on the filesystem.

Filesystem (use with parameter -F <fs>) Duration, days Extensions Filesystem Feature Remarks
scratch (default) 100 10 fs_lustre_scratch2 Scratch filesystem (/lustre/scratch2, symbolic link: /scratch) with high streaming bandwidth, based on spinning disks
ssd 30 2 fs_lustre_ssd High-IOPS filesystem (/lustre/ssd, symbolic link: /ssd) on SSDs.
beegfs_global0 (deprecated) 30 2 fs_beegfs_global0 High-IOPS filesystem (/beegfs/global0) on NVMes.
beegfs 30 2 fs_beegfs High-IOPS filesystem (/beegfs) on NVMes.
warm_archive 365 2 fs_warm_archive_ws Capacity filesystem based on spinning disks
Settings for Workspace Filesystem.

Use the command ws_extend to extend your workspace:

marie@login$ ws_extend -F scratch test-workspace 100
Info: extending workspace.
remaining extensions  : 1
remaining time in days: 100

Mail reminder settings are retained. I.e., previously set mail alerts apply to the extended workspace, too.


With the ws_extend command, a new duration for the workspace is set. The new duration is not added to the remaining lifetime!

This means when you extend a workspace that expires in 90 days with the command

marie@login$ ws_extend -F scratch my-workspace 40

it will now expire in 40 days not 130 days.

Send Reminder for Workspace Expiry Date

Send a calendar invitation by Email to ensure that the expiration date of a workspace is not forgotten

marie@login$ ws_send_ical -F scratch my-workspace -m

Deletion of a Workspace

To delete a workspace use the ws_release command. It is mandatory to specify the name of the workspace and the filesystem in which it is located:

marie@login$ ws_release -F scratch my-workspace

You can list your already released or expired workspaces using the ws_restore -l command.

marie@login$ ws_restore -l
  unavailable since Thu Oct  6 02:01:26 2022
  unavailable since Sat Mar 12 12:42:00 2022
  unavailable since Wen Jun 1 11:11:00 2022

In this example, the user marie has three inactive, i.e., expired, workspaces namely my-workspace in scratch, as well as foo and bar in ssd filesystem. The command ws_restore -l lists the name of the workspace and the expiration date. As you can see, the expiration date is added to the workspace name as Unix timestamp.

Deleting data in in an expired workspace

If you are short on quota, you might want to delete data in expired workspaces since it counts to your quota. Expired workspaces are moved to a hidden directory named .removed. The access rights remain unchanged. I.e., you can delete the data inside the workspace directory but you must not delete the workspace directory itself!

Expirer Process

The clean up process of expired workspaces is automatically handled by a so-called expirer process. It performs the following steps once per day and filesystem:

  • Check for remaining life time of all workspaces.
  • If the workspaces expired, move it to a hidden directory so that it becomes inactive.
  • Send reminder Emails to users if the reminder functionality was configured for their particular workspaces.
  • Scan through all workspaces in grace period.
  • If a workspace exceeded the grace period, the workspace and its data are deleted.

Restoring Expired Workspaces

At expiration time your workspace will be moved to a special, hidden directory. For a month (in warm_archive: 2 months), you can still restore your data into an existing workspace.


When you release a workspace by hand, it will not receive a grace period and be permanently deleted the next day. The advantage of this design is that you can create and release workspaces inside jobs and not swamp the filesystem with data no one needs anymore in the hidden directories (when workspaces are in the grace period).


marie@login$ ws_restore -l -F scratch
  unavailable since Thu Oct  6 02:01:26 2022

to get a list of your expired workspaces, and then restore them like that into an existing, active workspace 'new_ws':

marie@login$ ws_restore -F scratch marie-my-workspace-1665014486 new_ws

The expired workspace has to be specified by its full name as listed by ws_restore -l, including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified). The target workspace, on the other hand, must be given with just its short name, as listed by ws_list, without the username prefix.

Both workspaces must be on the same filesystem. The data from the old workspace will be moved into a directory in the new workspace with the name of the old one. This means a fresh workspace works as well as a workspace that already contains data.

Linking Workspaces in HOME

It might be valuable to have links to personal workspaces within a certain directory, e.g., your home directory. The command ws_register DIR will create and manage links to all personal workspaces within in the directory DIR. Calling this command will do the following:

  • The directory DIR will be created if necessary.
  • Links to all personal workspaces will be managed:
    • Create links to all available workspaces if not already present.
    • Remove links to released workspaces.

Remark: An automatic update of the workspace links can be invoked by putting the command ws_register DIR in your personal shell configuration file (e.g., .bashrc).

How to use Workspaces

There are three typical options for the use of workspaces:

Per-Job Storage

The idea of a "workspace per-job storage" addresses the need of a batch job for a directory for temporary data which can be deleted afterwards. To help you to write your own (Slurm) job file, suited to your needs, we came up with the following example (which works for the program g16).


Please do not blind copy the example, but rather take the essential idea and concept and adjust it to your needs and workflow, e.g.

  • adopt Slurm options for ressource specification,
  • inserting the path to your input file,
  • what software you want to load,
  • and calling the actual software to do your computation.

Using temporary workspaces for I/O intensive tasks


#SBATCH --partition=haswell
#SBATCH --time=48:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
## The optional constraint for the filesystem feature depends
## on the filesystem on which you want to use a workspace.
## Documentation here
#SBATCH --constraint=fs_lustre_ssd
#SBATCH --cpus-per-task=24

# Load the software you need here
module purge
module load <modules>

# The path to where your input file is located
test ! -f "${INPUTFILE}" && echo "Error: Could not find the input file ${INPUTFILE}" && exit 1

# The workspace where results from multiple expirements will be saved for later analysis
test -z "${RESULT_WSDIR}" && echo "Error: Cannot find workspace ${RESULT_WSDIR}" && exit 1

# Allocate workspace for this job. Adjust time span to time limit of the job (-d <N>).
export WSDDIR=$(ws_allocate -F ssd -n ${WSNAME} -d 2)
echo ${WSDIR}

# Check allocation
test -z "${WSDIR}" && echo "Error: Cannot allocate workspace ${WSDIR}" && exit 1

# Change to workspace directory
cd ${WSDIR}

# Adjust the following line to invoke the program you want to run
srun <application> < "${INPUTFILE}" > logfile.log

# Move result and log files of interest to directory named 'results'. This directory and its
# content will be saved in another storage location for later analysis. All files and
# directories will be deleted right away at the end of this job file.
mkdir results
cp <results and log files> results/

# Save result files in a general workspace (RESULT_WSDIR, s.a.) holding results from several
# experiments.
# Compress results with bzip2 (which includes CRC32 Checksums).
bzip2 --compress --stdout -4 "${WSDIR}/results" > ${RESULT_WSDIR}/gaussian_job-${SLURM_JOB_ID}.bz2
COMPRESSION_SUCCESS="$(if test ${RETURN_CODE} -eq 0; then echo 'TRUE'; else echo 'FALSE'; fi)"

# Clean up workspace
if [ "TRUE" = ${COMPRESSION_SUCCESS} ]; then
    test -d ${WSDIR} && rm -rf ${WSDIR}/*
    # Reduces grace period to 1 day!
    ws_release -F ssd ${WSNAME}
    echo "Error with compression and writing of results"
    echo "Please check the folder \"${WSDIR}\" for any partial(?) results."
    exit 1

Data for a Campaign

For a series of jobs or calculations that work on the same data, you should allocate a workspace once, e.g., in scratch for 100 days:

marie@login$ ws_allocate -F scratch my_scratchdata 100
Info: creating workspace.
remaining extensions  : 2
remaining time in days: 99

You can grant your project group access rights:

chmod g+wrx /scratch/ws/marie-my_scratchdata

And verify it with:

marie@login$ ls -la /scratch/ws/marie-my_scratchdata
total 8
drwxrwx--- 2 marie    hpcsupport 4096 Jul 10 09:03 .
drwxr-xr-x 5 operator adm        4096 Jul 10 09:01 ..

Mid-Term Storage

For data that seldom changes but consumes a lot of space, the warm archive can be used. Note that this is mounted read-only on the compute nodes, so you cannot use it as a work directory for your jobs!

marie@login$ ws_allocate -F warm_archive my_inputdata 365
remaining extensions  : 2
remaining time in days: 365


The warm archive is not built for billions of files. There is a quota for 100.000 files per group. Please archive data.

To see your active quota use

marie@login$ qinfo quota /warm_archive/ws/

Note that the workspaces reside under the mountpoint /warm_archive/ws/ and not /warm_archive anymore.

Cooperative Usage (Group Workspaces)

When a workspace is created with the option -g, --group, it gets a group workspace that is visible to others (if in the same group) via ws_list -g.

Chose group

If you are member of multiple groups, than the group workspace is visible for your primary group. You can list all groups you belong to via groups, and the first entry is your primary group.

Nevertheless, you can create a group workspace for any of your groups following these two steps:

  1. Change to the desired group using newgrp <other-group>.
  2. Create the group workspace as usual, i.e., ws_allocate --group [...]

The page on Sharing Data provides information on how to grant access to certain colleagues and whole project groups.

Allocate and list group workspaces

If Marie wants to share results and scripts in a workspace with all of her colleagues in the project p_number_crunch, she can allocate a so-called group workspace.

marie@login$ ws_allocate --group --name numbercrunch --duration 30
Info: creating workspace.
remaining extensions  : 10
remaining time in days: 30

This workspace directory is readable for the group, e.g.,

marie@login$ ls -ld /scratch/ws/0/marie-numbercrunch
drwxr-x--- 2 marie p_number_crunch 4096 Mar  2 15:24 /scratch/ws/0/marie-numbercrunch

All members of the project group p_number_crunch can now list this workspace using ws_list -g and access the data (read-only).

martin@login$ ws_list -g -t
id: numbercrunch
     workspace directory  : /scratch/ws/0/marie-numbercrunch
     remaining time       : 29 days 23 hours
     available extensions : 10

FAQ and Troubleshooting

Q: I am getting the error Error: could not create workspace directory!

A: Please check the "locale" setting of your SSH client. Some clients (e.g. the one from Mac) set values that are not valid on our ZIH systems. You should overwrite LC_CTYPE and set it to a valid locale value like export LC_CTYPE=de_DE.UTF-8.

A list of valid locales can be retrieved via locale -a.

Please use language_CountryCode.UTF-8 (or plain) settings. Avoid "iso" codepages!


Language Code
Chinese - Simplified zh_CN.UTF-8
English en_US.UTF-8
French fr_FR.UTF-8
German de_DE.UTF-8

Q: I am getting the error Error: target workspace does not exist! when trying to restore my workspace.

A: The workspace you want to restore into is either not on the same filesystem or you used the wrong name. Use only the short name that is listed after id: when using ws_list.

Q: Man, I've missed to specify mail alert when allocating my workspace. How can I add the mail alert functionality to an existing workspace?

A: You can add the mail alert by "overwriting" the workspace settings via ws_allocate -x -m <mail address> -r <days> -n <ws-name> -d <duration> -F <fs>. (This will lower the remaining extensions by one.)