Skip to content

BeeGFS Filesystem

Warning

This documentation page is outdated. Please see the new BeeGFS page.

Prerequisites: To work with TensorFlow you obviously need a login to the ZIH systems and basic knowledge about Linux, mounting, and batch system Slurm.

Aim of this page is to introduce users how to start working with the BeeGFS filesystem - a high-performance parallel filesystem.

Mount Point

Understanding of mounting and the concept of the mount point is important for using filesystems and object storage. A mount point is a directory (typically an empty one) in the currently accessible filesystem on which an additional filesystem is mounted (i.e., logically attached). The default mount points for a system are the directories in which filesystems will be automatically mounted unless told by the user to do otherwise. All partitions are attached to the system via a mount point. The mount point defines the place of a particular data set in the filesystem. Usually, all partitions are connected through the root partition. On this partition, which is indicated with the slash (/), directories are created.

BeeGFS Introduction

BeeGFS is the parallel cluster filesystem. BeeGFS spreads data across multiple servers to aggregate capacity and performance of all servers to provide a highly scalable shared network filesystem with striped file contents. This is made possible by the separation of metadata and file contents.

BeeGFS is fast, flexible, and easy to manage storage if for your issue filesystem plays an important role use BeeGFS. It addresses everyone, who needs large and/or fast file storage.

Create BeeGFS Filesystem

To reserve nodes for creating BeeGFS filesystem you need to create a batch job

#!/bin/bash
#SBATCH -p nvme
#SBATCH -N 4
#SBATCH --exclusive
#SBATCH --time=1-00:00:00
#SBATCH --beegfs-create=yes

srun sleep 1d  # sleep for one day

## when finished writing, submit with:  sbatch <script_name>

Example output with job id:

Submitted batch job 11047414   #Job id n.1

Check the status of the job with squeue -u \<username>.

Mount BeeGFS Filesystem

You can mount BeeGFS filesystem on the partition ml (PowerPC architecture) or on the partition haswell (x86_64 architecture), more information about partitions.

Mount BeeGFS Filesystem on the Partition ml

Job submission can be done with the command (use job id (n.1) from batch job used for creating BeeGFS system):

srun -p ml --beegfs-mount=yes --beegfs-jobid=11047414 --pty bash                #Job submission on ml nodes
```console

Example output:

```console
srun: job 11054579 queued and waiting for resources         #Job id n.2
srun: job 11054579 has been allocated resources

Mount BeeGFS Filesystem on the Haswell Nodes (x86_64)

Job submission can be done with the command (use job id (n.1) from batch job used for creating BeeGFS system):

srun --constrain=DA --beegfs-mount=yes --beegfs-jobid=11047414 --pty bash       #Job submission on the Haswell nodes

Example output:

srun: job 11054580 queued and waiting for resources          #Job id n.2
srun: job 11054580 has been allocated resources

Working with BeeGFS files for both types of nodes

Show contents of the previously created file, for example, beegfs_11054579 (where 11054579 - job id n.2 of srun job):

cat .beegfs_11054579

Note: don't forget to go over to your home directory where the file located

Example output:

#!/bin/bash

export BEEGFS_USER_DIR="/mnt/beegfs/<your_id>_<name_of_your_job>/<your_id>"
export BEEGFS_PROJECT_DIR="/mnt/beegfs/<your_id>_<name_of_your_job>/<name of your project>"

Execute the content of the file:

source .beegfs_11054579

Show content of user's BeeGFS directory with the command:

ls -la ${BEEGFS_USER_DIR}

Example output:

total 0
drwx--S--- 2 <username> swtest  6 21. Jun 10:54 .
drwxr-xr-x 4 root        root  36 21. Jun 10:54 ..

Show content of the user's project BeeGFS directory with the command:

ls -la ${BEEGFS_PROJECT_DIR}

Example output:

total 0
drwxrws--T 2 root swtest  6 21. Jun 10:54 .
drwxr-xr-x 4 root root   36 21. Jun 10:54 ..

Note

If you want to mount the BeeGFS filesystem on an x86 instead of an ML (power) node, you can either choose the partition "interactive" or the partition haswell64, but for the partition haswell64 you have to add the parameter --exclude=taurusi[4001-4104,5001-5612] to your job. This is necessary because the BeeGFS client is only installed on the 6000 island.