Introduction HPC Resources and Jobs¶
ZIH operates high performance computing (HPC) systems with more than 90.000 cores, 500 GPUs, and a flexible storage hierarchy with about 20 PB total capacity. The HPC system provides an optimal research environment especially in the area of data analytics, artificial intelligence methods and machine learning as well as for processing extremely large data sets. Moreover it is also a perfect platform for highly scalable, data-intensive and compute-intensive applications and has extensive capabilities for energy measurement and performance monitoring. Therefore provides ideal conditions to achieve the ambitious research goals of the users and the ZIH.
The HPC system, redesigned in December 2023, consists of five homogeneous clusters with their own Slurm instances and cluster specific login nodes. The clusters share one filesystem which enables users to easily switch between the components.
Selection of Suitable Hardware¶
The five clusters
differ, among others, in number of nodes, cores per node, and GPUs and memory. The particular
characteristica qualify them for different applications.
Which Cluster Do I Need?¶
The majority of the basic tasks can be executed on the conventional nodes like on
log in to ZIH systems, you are placed on a login node where you can execute short tests and compile
moderate projects. The login nodes cannot be used for real experiments and computations. Long and
extensive computational work and experiments have to be encapsulated into so called jobs and
scheduled to the compute nodes.
There is no such thing as free lunch at ZIH systems. Since compute nodes are operated in multi-user node by default, jobs of several users can run at the same time at the very same node sharing resources, like memory (but not CPU). On the other hand, a higher throughput can be achieved by smaller jobs. Thus, restrictions w.r.t. memory and runtime limits have to be respected when submitting jobs.
The following questions may help to decide which cluster to use
- my application
- Do I need GPUs?
- How much run time do I need?
- How many cores do I need?
- How much memory do I need?
- Which software is required?
|Cores per Node
|Threads per Core
|Memory per Node [in MB]
|Memory per Core [in MB]
|GPUs per Node
|single SMP system
|IBM Power/GPU system
Interactive or Batch Mode¶
Interactive jobs: An interactive job is the best choice for testing and development. See
Slurm can forward your X11 credentials to the first node (or even all) for a job
--x11 option. To use an interactive job you have to specify
-X flag for the ssh login.
srun directly on the Shell will lead to blocking and launch an interactive job.
Apart from short test runs, it is recommended to encapsulate your experiments and computational
tasks into batch jobs and submit them to the batch system. For that, you can conveniently put the
parameters directly into the job file which you can submit using
sbatch [options] <job file>.
MPI jobs: For MPI jobs typically allocates one core per task. Several nodes could be allocated if it is necessary. The batch system Slurm will automatically find suitable hardware.
OpenMP jobs: SMP-parallel applications can only run within a node, so it is necessary to
include the batch system options
Slurm will start one task and you will have
N CPUs. The maximum number of processors for an
SMP-parallel program is 896 on cluster
Julia (be aware that the application has to be
developed with that large number of threads in mind).
Partitions with GPUs are best suited for repetitive and highly-parallel computing tasks. If
you have a task with potential data parallelism most likely that
you need the GPUs. Beyond video rendering, GPUs excel in tasks such as machine learning, financial
simulations and risk modeling. Use the cluster
power only if you need GPUs! Otherwise
using the x86-based partitions most likely would be more beneficial.
Some cluster/nodes have Simultaneous Multithreading (SMT) enabled, e.g
request for this additional threads using the Slurm option
--hint=multithread or by setting the
SLURM_HINT=multithread. Besides the usage of the threads to speed up the
computations, the memory of the other threads is allocated implicitly, too, and you will always get
Memory per Core*
number of threads as memory pledge.
What do I need, a CPU or GPU?¶
If an application is designed to run on GPUs this is normally announced unmistakable since the efforts of adapting an existing software to make use of a GPU can be overwhelming. And even if the software was listed in NVIDIA's list of GPU-Accelerated Applications only certain parts of the computations may run on the GPU.
To answer the question: The easiest way is to compare a typical computation on a normal node and on a GPU node. (Make sure to eliminate the influence of different CPU types and different number of cores.) If the execution time with GPU is better by a significant factor then this might be the obvious choice.
Difference in Architecture
The main difference between CPU and GPU architecture is that a CPU is designed to handle a wide range of tasks quickly, but are limited in the concurrency of tasks that can be running. While GPUs can process data much faster than a CPU due to massive parallelism (but the amount of data which a single GPU's core can handle is small), GPUs are not as versatile as CPUs.
How much time do I need?¶
Runtime limits on login nodes
There is a time limit of 600 seconds set for processes on login nodes. Each process running longer than this time limit is automatically killed. The login nodes are shared ressources between all users of ZIH system and thus, need to be available and cannot be used for productive runs.
CPU time limit exceeded
Please submit extensive application runs to the compute nodes using the batch system.
Runtime limits are enforced.
A job is canceled as soon as it exceeds its requested limit. Currently, the maximum run time limit is 7 days.
Shorter jobs come with multiple advantages:
- lower risk of loss of computing time,
- shorter waiting time for scheduling,
- higher job fluctuation; thus, jobs with high priorities may start faster.
To bring down the percentage of long running jobs we restrict the number of cores with jobs longer than 2 days to approximately 50% and with jobs longer than 24 to 75% of the total number of cores. (These numbers are subject to change.) As best practice we advise a run time of about 8h.
Please always try to make a good estimation of your needed time limit.
For this, you can use a command line like this to compare the requested timelimit with the elapsed time for your completed jobs that started after a given date:
marie@login$ sacct -X -S 2021-01-01 -E now --format=start,JobID,jobname,elapsed,timelimit -s COMPLETED
Instead of running one long job, you should split it up into a chain job. Even applications that are not capable of checkpoint/restart can be adapted. Please refer to the section Checkpoint/Restart for further documentation.
How many cores do I need?¶
ZIH systems are focused on data-intensive computing. They are meant to be used for highly parallelized code. Please take that into account when migrating sequential code from a local machine to our HPC systems. To estimate your execution time when executing your previously sequential program in parallel, you can use Amdahl's law. Think in advance about the parallelization strategy for your project and how to effectively use HPC resources.
However, this is highly depending on the used software, investigate if your application supports a parallel execution.
How much memory do I need?¶
Memory limits are enforced.
Jobs which exceed their per-node memory limit are killed automatically by the batch system.
Memory requirements for your job can be specified via the
--mem=<MB> (which is "memory per node"). The default limit regardless
of the partition it runs on is quite low at 300 MB per CPU. If you need more memory, you need
to request it.
ZIH systems comprise different sets of nodes with different amount of installed memory which affect where your job may be run. To achieve the shortest possible waiting time for your jobs, you should be aware of the limits shown in the Slurm resource limits table.
Which software is required?¶
Pre-installed software on our HPC systems is managed via modules. You can see the list of software that's already installed and accessible via modules. However, there are many different variants of these modules available. Each cluster has its own set of installed modules, depending on their purpose.
Specific modules can be found with:
marie@compute$ module spider <software_name>
Processing of Data for Input and Output¶
Pre-processing and post-processing of the data is a crucial part for the majority of data-dependent projects. The quality of this work influence on the computations. However, pre- and post-processing in many cases can be done completely or partially on a local system and then transferred to ZIH systems. Please use ZIH systems primarily for the computation-intensive tasks.
Exclusive Reservation of Hardware¶
If you need for some special reasons, e.g., for benchmarking, a project or paper deadline, parts of our machines exclusively, we offer the opportunity to request and reserve these parts for your project.
Please send your request 7 working days before the reservation should start (as that's our maximum time limit for jobs and it is therefore not guaranteed that resources are available on shorter notice) with the following information to the HPC support:
Project:Which project will be credited for the reservation?
Reservation owner:Who should be able to run jobs on the reservation? I.e., name of an individual user or a group of users within the specified project.
System:Which machine should be used?
Hardware requirements:How many nodes and cores do you need? Do you have special requirements, e.g., minimum on main memory, equipped with a graphic card, special placement within the network topology?
Time window:Begin and end of the reservation in the form
Reason:Reason for the reservation.
Please note that your project CPU hour budget will be credited for the reserved hardware even if you don't use it.