In order to transfer files to and from the supercomputer (OzSTAR), a ssh-based file transfer utility is required. There are a variety of Options that you can use.
|Mac||“scp” (secure copy) command can be used via the terminal or use Cyberduck as a GUI alternative|
|Linux||“scp” (secure copy) command _ can be used via the terminal|
|Windows||The recommended software is WinSCP|
Copying a file or directory via SSH¶
The simplest way to copy a file to or from the supercomputer is to use the
In the following examples, we assume an
ozstar alias set to
Copying a local file to the OzSTAR supercomputer:
scp ./file.txt ozstar:destination/path/
You can also copy a file from the supercomputer to your local machine (e.g. download) as follows:
scp ozstar:path/to/file.txt .
Copy a directory and its content is done by using the
scp -r path/to/copy/ ozstar:destination/path/
The above command will initiate a connection from your local environment to OzSTAR directly.
Transferring a large number of small files¶
Transferring lots of small files can take a long time with
scp due to the overhead of copying each file individually. Instead, it is advised to package your files into a single
tar file using the tar command to significantly reduce the transfer time. This is done by create a tar archive, copying it using
scp, and finally
untar it to retrieve your individual files.
The best way to transfer code from one computer to another is to host the code in a source code repository using a versioning system such as git or mercurial and clone the repository from your local computer, or a cloud service (e.g. bitbucket, github, ...), to the supercomputer.
Resuming interrupted transfers¶
If a transfer is interrupted, you might end up with part of the files being transferred. Rather than restarting the transfer from scratch, you should then use the rsync command. The rsync command will compare the source and destination directories and only transfer what needs to be transferred (missing files, modified files, etc.)
rsync -va ./source_dir ozstar:destination/path
Make sure not to leave trailing slashes in your path names (e.g.
destination/path rather than
destination/path/); else you will have a full copy of the directory inside the existing, partial, one. Use the
-n (dry-run) option of
rsync to check what will happen before you run the actual command.
If one large file is left partially transferred, you can resume it using the
Synchronising with a local directory¶
If you want to keep two directories (one on your local computer, and one on the supercomputer) in sync, you can do that with rsync using its
--delete option. But that is only one-way so you need to really think in what direction you do it, and it does not scale beyond two synchronized directory trees.
Another potential option is to use Unison, a piece of software that can detect and handle conflicts (incompatible changes made to the same file in the two directories that must be kept in sync.)
Please note that the supercomputer is NOT a place to use as a backup for your laptop or workstation's holiday snaps and emails.