Lustre Filesystems

Large Files in /scratch

The data containers in Lustre are called object storage targets (OST). The capacity of one OST is about 21 TB. All files are striped over a certain number of these OSTs. For small and medium files, the default number is 2. As soon as a file grows above ~1 TB it makes sense to spread it over a higher number of OSTs, e.g. 16. Once the filesystem is used >75%, the average space per OST is only 5 GB. So, it is essential to split your larger files so that the chunks can be saved!

Lets assume you have a directory where you tar your results, e.g. /scratch/marie/tar. Now, simply set the stripe count to a higher number in this directory with:

marie@login$ lfs setstripe -c 20  /scratch/ws/marie-stripe20/tar

Note

This does not affect existing files. But all files that will be created in this directory will be distributed over 20 OSTs.

Useful Commands for Lustre

These commands work for /scratch and /ssd.

Listing Disk Usages per OST and MDT

marie@login$ lfs quota -h -u username /path/to/my/data

It is possible to display the usage on each OST by adding the argument -v.

Listing Space Usage per OST and MDT

marie@login$ lfs df -h /path/to/my/data

Listing inode usage for an specific path

marie@login$ lfs df -i /path/to/my/data

Listing OSTs

marie@login$ lfs osts /path/to/my/data

View Striping Information

marie@login$ lfs getstripe myfile
marie@login$ lfs getstripe -d mydirectory

The argument -d will also display striping for all files in the directory.