Transfer Data to/from ZIH Systems via Dataport Nodes¶

To copy large data to/from ZIH systems, the so-called dataport nodes should be used. While it is possible to transfer small files directly via the login nodes, they are not intended to be used that way. Furthermore, longer transfers will hit the CPU time limit on the login nodes, i.e. the process get killed. The dataport nodes have a better uplink (10 GBit/s) allowing for higher bandwidth. Note that you cannot log in via SSH to the dataport nodes, but only use

rsync or sftp (incl. FTP-clients like e.g. FileZilla) on them.

The dataport nodes are reachable under the hostnames

dataport1.hpc.tu-dresden.de (IP: 141.30.73.4)
dataport2.hpc.tu-dresden.de (IP: 141.30.73.5)

Through the usage of these dataport nodes, you can bring your data to ZIH HPC systems or get data from there - they have access to the different HPC filesystems. As of 11/2023, the following directories are accessible:

/home
/projects
/data/horse
/data/walrus
/data/archiv
(/data/software)
(/data/old/home)
(/data/old/software)

Access From Linux¶

There are at least three tools to exchange data between your local workstation and ZIH systems. They are explained in the following section in more detail.

Important

The following explanations require that you have already set up your SSH configuration.

SFTP¶

The tool sftp (OpenSSH secure file transfer) is a file transfer program, which performs all operations over an encrypted SSH transport. It may use compression to increase performance.

sftp is basically a virtual command line, which you could access and exit as follows.

Note

It is important from which point in your directory tree you 'enter' sftp! The current working directory (double ckeck with pwd) will be the target folder on your local machine from/to which remote files from the ZIH system will be put/get by sftp. The local folder might also be changed during a session with special commands. During the sftp session, you can use regular commands like ls, cd, pwd etc. But if you wish to access your local workstation, these must be prefixed with the letter l (local), e.g., lls, lcd or lpwd.

# Enter virtual command line
marie@local$ sftp dataport
# Exit virtual command line
sftp> exit
# or
sftp> <Ctrl+D>

Example: Copy a file from your workstation to ZIH systems

marie@local$ cd my/local/work
marie@local$ sftp dataport
# Copy file
sftp> put <file>
# Copy directory
sftp> put -r <directory>

Example: Copy a file from ZIH systems to your local workstation

marie@local$ sftp dataport
# Copy file
sftp> get <file>
# change local (target) directory
sftp> lcd /my/local/work
# Copy directory
sftp> get -r <directory>

Rsync¶

Rsync, is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.

Type following commands in the terminal when you are in the directory of the local machine.

Example: Copy a file from your workstation to ZIH systems

# Copy file
marie@local$ rsync <file> dataport:<target-location>
# Copy directory
marie@local$ rsync -r <directory> dataport:<target-location>

Example: Copy a file from ZIH systems to your local workstation

# Copy file
marie@local$ rsync dataport:<file> <target-location>
# Copy directory
marie@local$ rsync -r dataport:<directory> <target-location>

Access From Windows¶

Command Line¶

Windows 10 (1809 and higher) comes with a built-in OpenSSH support including the above described SFTP.

GUI - Using WinSCP¶

First you have to install WinSCP.

Then you have to execute the WinSCP application and configure some option as described below.

After your connection succeeded, you can copy files from your local workstation to ZIH systems and the other way around.