ALERT! Warning: your browser isn't supported. Please install a modern one, like Firefox, Opera, Safari, Chrome or the latest Internet Explorer. Thank you!
Startseite » ... » Zentrale Einrichtungen  » ZIH  » Wiki
phone prefix: +49 351 463.....

HPC Support

Operation Status

Ulf Markwardt: 33640
Claudia Schmidt: 39833

hpcsupport@zih.tu-dresden.de

Login and project application

Phone: 40000
Fax: 42328
servicedesk@tu-dresden.de

You are here: Compendium » DataManagement » WorkSpaces

Workspaces

Storage systems come with different flavours in terms of
  • size
  • streaming bandwidth
  • IOPS rate
With a limited price one cannot have all in one. That is the reason why our fast parallel file systems have restrictions wrt. age of files (see TermsOfUse). The mechanism of workspaces enables users to better manage the data life cycle of their HPC data. The tool concept of "workspaces" is common in a large number of HPC centers. The idea is to request for a workspace directory in a certain storage system - connected with an expiry date. After a grace period the data is deleted automatically. The lifetime of a workspace depends on the storage system:
  • ssd: 30 days,
  • scratch: 100 days,
  • warm_archive: 1 year.
All workspaces can be extended twice. There is no problem to use the fastest file systems we have, but keep track on your data and move it to a cheaper system once you have done your computations.

Workspace commands

To create a workspace, specify a unique name and its life time like this:
mark@tauruslogin6:~> ws_allocate -F scratch SPECint 50
Info: creating workspace.
/scratch/ws/mark-SPECint
remaining extensions  : 2
remaining time in days: 50

Important: You can (and should) also add your email address and a relative date for notification:

mark@tauruslogin6:~> ws_allocate -F scratch -r 7 -m name.lastname@tu-dresden.de SPECint 50
mark@tauruslogin6:~> ws_allocate: [options] workspace_name duration
Options:
  -h [ --help ]              produce help message
  -V [ --version ]           show version
  -d [ --duration ] arg (=1) duration in days
  -n [ --name ] arg          workspace name
  -F [ --filesystem ] arg    filesystem
  -r [ --reminder ] arg      reminder to be sent n days before expiration
  -m [ --mailaddress ] arg   mailaddress to send reminder to
  -x [ --extension ]         extend workspace
  -u [ --username ] arg      username
  -g [ --group ]             group workspace
  -c [ --comment ] arg       comment

The maximum duration depends on the storage system:

Storage system ( use with parameter -F )
Duration Remarks
ssd 30 days High-IOPS file system (/lustre/ssd) on SSDs.
scratch 100 days Scratch file system (/scratch) with high streaming bandwidth, based on spinning disks.
warm_archive 1 year Capacity file system based on spinning disks.
A workspace can be extended twice. With this command, a new duration for the workspace is set (not cumulative):

mark@tauruslogin6:~> ws_extend -F scratch SPECint 100
Info: extending workspace.
/scratch/ws/mark-SPECint
remaining extensions  : 1
remaining time in days: 100

For email notification, you can either use the option -m in the ws_allocate command line or use ws_send_ical to get an entry in your calendar. (This works only with tu-dresden.de addresses. Please configure email redirection if you want to use another address.)

mark@tauruslogin6:~> ws_send_ical -m ulf.markwardt@tu-dresden.de -F scratch SPECint

You can easily get an overview of your currently used workspaces with ws_list.

mark@tauruslogin6:~> ws_list
id: benchmark_storage
     workspace directory  : /warm_archive/ws/mark-benchmark_storage
     remaining time       : 364 days 23 hours
     creation time        : Thu Jul  4 13:40:31 2019
     expiration date      : Fri Jul  3 13:40:30 2020
     filesystem name      : warm_archive
     available extensions : 2
id: SPECint
     workspace directory  : /scratch/ws/mark-SPECint
     remaining time       : 99 days 23 hours
     creation time        : Thu Jul  4 13:36:51 2019
     expiration date      : Sat Oct 12 13:36:51 2019
     filesystem name      : scratch
     available extensions : 1

With ws_release -F <file system> <workspace name>, you can delete your workspace.

Restoring expired workspaces

At expiration time (or when you manually release your workspace), your workspace will be moved to a special, hidden directory. For a month (in warm_archive: 2 months), you can still restore your data into a valid workspace. For that, use

mark@tauruslogin6:~> ws_restore -l -F scratch

to get a list of your expired workspaces, and then restore them like that into an existing, active workspace newws:

mark@tauruslogin6:~> ws_restore -F scratch myuser-myws-1234567 newws

Linking workspaces in home

It might be valuable to have links to personal workspaces within a certain directory, e.g., the user home directory. The command `ws_register DIR` will create and manage links to all personal workspaces within in the directory `DIR`. Calling this command will do the following:
  • The directory `DIR` will be created if necessary
  • Links to all personal workspaces will be managed:
    • Creates links to all available workspaces if not already present
    • Removes links to released workspaces

Remark: An automatic update of the workspace links can be invoked by putting the command `ws_register DIR` in the user's personal shell configuration file (e.g., .bashrc, .zshrc).

How to Use Workspaces

We see three typical use cases for the use of workspaces:

Per-Job-Storage

A batch job needs a directory for temporary data. This can be deleted afterwards.

Here an example for the use with Gaussian:

#!/bin/bash
#SBATCH --partition=haswell
#SBATCH --time=96:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=24

module load modenv/classic
module load gaussian

COMPUTE_DIR=gaussian_$SLURM_JOB_ID
export GAUSS_SCRDIR=$(ws_allocate -F ssd $COMPUTE_DIR 7)
echo $GAUSS_SCRDIR

srun g16 inputfile.gjf logfile.log

test -d $GAUSS_SCRDIR && rm -rf $GAUSS_SCRDIR/*
ws_release -F ssd $COMPUTE_DIR

In a similar manner, other jobs can make use of temporary workspaces.

Data for a Campaign

For a series of calculations that works on the same data, you could allocate a workspace in the scratch for e.g. 100 days:

mark@tauruslogin6:~> ws_allocate -F scratch my_scratchdata 100
Info: creating workspace.
/scratch/ws/mark-my_scratchdata
remaining extensions  : 2
remaining time in days: 99

If you want to share it with your project group, set the correct access attributes, eg.

mark@tauruslogin6:~> chmod g+wrx /scratch/ws/mark-my_scratchdata

And verify it with:

mark@tauruslogin6:~> ls -la /scratch/ws/mark-my_scratchdata 
total 8
drwxrwx--- 2 mark hpcsupport 4096 Jul 10 09:03 .
drwxr-xr-x 5 operator adm 4096 Jul 10 09:01 ..

Mid-Term Storage

For data that seldomly changes but consumes a lot of space, the warm archive can be used.
Note that this is mounted read-only on the compute nodes, so you cannot use it as a work directory for your jobs!

mark@tauruslogin6:~> ws_allocate -F warm_archive my_inputdata 365
/warm_archive/ws/mark-my_inputdata
remaining extensions  : 2
remaining time in days: 365

Attention: The warm archive is not built for billions of files. There is a quota active of 100.000 files per group. Maybe you might want to tar your data. To see your active quota use:

mark@tauruslogin6:~> qinfo quota /warm_archive/ws/
Consuming Entity                              Type                Limit   Current Usage  
GROUP: hpcsupport                             LOGICAL_DISK_SPACE  100 TB  51 GB (0%)     
GROUP: hpcsupport                             FILE_COUNT          100000  4 (0%)         
GROUP: swtest                                 LOGICAL_DISK_SPACE  100 TB  5 GB (0%)      
GROUP: swtest                                 FILE_COUNT          100000  38459 (38%)    
TENANT: 8a2373d6-7aaf-4df3-86f5-a201281afdbb  LOGICAL_DISK_SPACE  5 PB    1 TB (0%)    

Note that the workspaces reside under the mountpoint /warm_archive/ws/ and not /warm_archiveanymore.

Troubleshooting

If you are getting the error:

Error: could not create workspace directory!

you should check the localesetting of your ssh client. Some clients (e.g. the one from MacOSX) set values that are not valid on Taurus. You should overwrite LC_CTYPE and set it to a valid locale value like:

export LC_CTYPE=de_DE.UTF-8

A list of valid locales can be retrieved via

locale -a