File Transfer¶
In order to transfer files to and from the supercomputer (OzSTAR), a ssh-based file transfer utility is required. There are a variety of Options that you can use.
Operating System | Terminal |
Mac | “scp” (secure copy) command can be used via the terminal or use Cyberduck as a GUI alternative |
Linux | “scp” (secure copy) command _ can be used via the terminal |
Windows | The recommended software is WinSCP |
As an alternative option, you can use a SFTP Client (e.g FileZilla Client – more details about using the SFTP mode in FileZilla can be found on https://it.unh.edu/sftp/filezilla.html)
Copying a file or directory via SSH¶
The simplest way to copy a file to or from the supercomputer is to use the scp
command.
Note
In the following examples, we assume an ozstar
alias set to [your-username]@ozstar.swin.edu.au
.
Copying a local file to the OzSTAR supercomputer:
scp ./file.txt ozstar:destination/path/
You can also copy a file from the supercomputer to your local machine (e.g. download) as follows:
scp ozstar:path/to/file.txt .
Copy a directory and its content is done by using the -r
option:
scp -r path/to/copy/ ozstar:destination/path/
The above command will initiate a connection from your local environment to OzSTAR directly.
Transferring a large number of small files¶
Transferring lots of small files can take a long time with scp
due to the overhead of copying each file individually. Instead, it is advised to package your files into a single tar
file using the tar command to significantly reduce the transfer time. This is done by create a tar archive, copying it using scp
, and finally untar
it to retrieve your individual files.
Transferring code¶
The best way to transfer code from one computer to another is to host the code in a source code repository using a versioning system such as git or mercurial and clone the repository from your local computer, or a cloud service (e.g. bitbucket, github, ...), to the supercomputer.
Resuming interrupted transfers¶
If a transfer is interrupted, you might end up with part of the files being transferred. Rather than restarting the transfer from scratch, you should then use the rsync command. The rsync command will compare the source and destination directories and only transfer what needs to be transferred (missing files, modified files, etc.)
rsync -va ./source_dir ozstar:destination/path
Warning
Make sure not to leave trailing slashes in your path names (e.g. destination/path
rather than destination/path/
); else you will have a full copy of the directory inside the existing, partial, one. Use the -n
(dry-run) option of rsync
to check what will happen before you run the actual command.
Tip
If one large file is left partially transferred, you can resume it using the --partial
.
Synchronising with a local directory¶
If you want to keep two directories (one on your local computer, and one on the supercomputer) in sync, you can do that with rsync using its --delete
option. But that is only one-way so you need to really think in what direction you do it, and it does not scale beyond two synchronized directory trees.
Another potential option is to use Unison, a piece of software that can detect and handle conflicts (incompatible changes made to the same file in the two directories that must be kept in sync.)
Please note that the supercomputer is NOT a place to use as a backup for your laptop or workstation's holiday snaps and emails.