In order to transfer files to and from the supercomputer, an SSH-based file transfer utility is required. There are a variety of Options that you can use.
If you are using Windows Subsystem for Linux, we recommend that you follow the instructions for Linux. The Linux utilities are generally superior to the Windows equivalents, and are more widely supported.
Mac and Linux
Copying a file or directory via SSH¶
The best way to copy a file to or from the supercomputer is to use the
rsync is preferred over
scp because it designed to handle multiple files, and because it preserves permissions on directories.
Please use the data-mover nodes (
data-mover[01-04]) instead of the login nodes for file transfers. The login nodes are shared between all users, who may be running tests or compiling code, whereas the data-movers are dedicated to file transfers. You can access the data-mover nodes externally by their addresses: data-mover[01-04].hpc.swin.edu.au
Copying a local file to the OzSTAR supercomputer:
rsync -avPxH --no-g --chmod=Dg+s <local files and dirs> email@example.com:/fred/<project dir>/<somewhere>/
rsync has many options, but these are the recommendations. Please refer to rsync's man page for more information:
-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) -v, --verbose increase verbosity -P same as --partial --progress -x, --one-file-system don't cross filesystem boundaries -H, --hard-links preserve hard links -g, --group preserve group --chmod=CHMOD affect file and/or directory permissions
You can also copy a file from the supercomputer to your local machine (i.e. download) by swapping the source and destination arguments:
rsync -avPxH --no-g --chmod=Dg+s firstname.lastname@example.org:/fred/<project dir>/<somewhere>/ <local destination>
When transferring large files, it may be useful to use the
-z option of
rsync to first compress the file, send it, and then decompress it. This is especially useful if your network connection is slow.
Resuming interrupted transfers¶
If a transfer is interrupted, you might end up with part of the files being transferred. Rather than restarting the transfer from scratch, rsync will compare the source and destination directories and only transfer what needs to be transferred (missing files, modified files, etc.). Simply run the rsync command again (with the same source and destination arguments) to resume the transfer.
The best way to transfer code from one computer to another is to host the code in a source code repository using a versioning system such as git and clone the repository to the supercomputer.
Synchronising with a local directory¶
If you want to keep two directories (one on your local computer, and one on the supercomputer) in sync, you can do that with rsync using its
--delete option. But that is only one-way so you need to really think in what direction you do it, and it does not scale beyond two synchronized directory trees.
Please note that the supercomputer is NOT a place to use as a backup for your laptop or workstation. Storage on the Lustre file system is expensive to maintain, and must not be used for purposes unrelated to compute jobs.