Transfer Data Inside ZIH Systems with Datamover¶
With the Datamover, we provide special data transfer machines for transferring data between the ZIH filesystems with best transfer speed. The Datamover machine is not accessible through SSH as it is dedicated to data transfers. To move or copy files from one filesystem to another, you have to use the following commands after logging in to any of the ZIH HPC systems:
dtcp
,dtls
,dtmv
,dtrm
,dtrsync
,dttar
, anddtwget
These special commands submit a batch job to the data transfer
machines performing the selected command. Their syntax and behavior is the very same as the
well-known shell commands without the prefix dt
, except for the following options.
Additional Option | Description |
---|---|
--account=ACCOUNT |
Assign data transfer job to specified account. |
--blocking |
Do not return until the data transfer job is complete. (default for dtls ) |
--time=TIME |
Job time limit (default: 18 h). |
Managing Transfer Jobs¶
There are the commands dtinfo
, dtqueue
, dtq
, and dtcancel
to manage your transfer commands
and jobs.
dtinfo
shows information about the nodes of the data transfer machine (likesinfo
).dtqueue
anddtq
show all your data transfer jobs (likesqueue --me
).dtcancel
signals data transfer jobs (likescancel
).
To identify the mount points of the different filesystems on the data transfer machine, use
dtinfo
. It shows an output like this:
Directory on Datamover | Mounting Clusters | Directory on Cluster |
---|---|---|
/home |
Alpha,Barnard,Julia,Power9,Romeo | /home |
/projects |
Alpha,Barnard,Julia,Power9,Romeo | /projects |
/data/horse |
Alpha,Barnard,Julia,Power9,Romeo | /data/horse |
/data/walrus |
Alpha,Barnard,Julia,Power9 | /data/walrus |
/data/octopus |
Alpha,Barnard,Power9,Romeo | /data/octopus |
/data/archiv |
Usage of Datamover¶
Copy data from /data/horse
to /projects
filesystem.
marie@login$ dtcp -r /data/horse/ws/marie-workdata/results /projects/p_number_crunch/.
Move data from /data/horse
to /data/walrus
filesystem.
marie@login$ dtmv /data/horse/ws/marie-workdata/results /data/walrus/ws/marie-archive/.
Archive data from /data/walrus
to /archiv
filesystem.
marie@login$ dttar -czf /archiv/p_number_crunch/results.tgz /data/walrus/ws/marie-workdata/results
Warning
Do not generate files in the /archiv
filesystem much larger that 500 GB!
Note
The projects
filesystem is not writable from within batch jobs.
However, you can store the data in the walrus
filesystem
using the Datamover nodes via dt*
commands.
Transferring Files Between ZIH Systems and Group Drive¶
In order to let the datamover have access to your group drive, copy your public SSH key from ZIH
system to login1.zih.tu-dresden.de
, first.
marie@login$ ssh-copy-id -i ~/.ssh/id_rsa.pub login1.zih.tu-dresden.de
# Export the name of your group drive for reuse of example commands
marie@login$ export GROUP_DRIVE_NAME=<my-drive-name>
Copy data from your group drive to /data/horse
filesystem.
marie@login$ dtrsync -av dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/inputfile /data/horse/ws/marie-workdata/.
Copy data from /data/horse
filesystem to your group drive.
marie@login$ dtrsync -av /data/horse/ws/marie-workdata/resultfile dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/.