Transferring data between your local computer and HPC

Transferring data to Artemis using FileZilla

FileZilla is a freely available SFTP client. You can download a copy from their website: http://filezilla-project.org.

To transfer data from your local computer to HPC using FileZilla, follow these steps:

  1. Go to File -> Site Manager in the menu bar at the top of the program.
  2. Click New Site in the lower left-hand corner and give this connection a name. The name is for your reference only.
  3. Click the General tab in Site Manager.
  4. Type hpc.sydney.edu.au in the Host field.
  5. Leave the port blank.
  6. Select the SFTP protocol in the protocol field.
  7. Select Normal in the logon type field.
  8. Type your UniKey in the User field.
  9. Type your UniKey password in the Password field.
  10. Click Connect.
  11. An Unknown host key box will pop up. Click Always trust this host, add this key to the cache box and then click OK.

If you successfully logged on, you will see a message saying “Directory listing of “/home/abcd1234” successful”.

To transfer data between HPC and your local computer, drag and drop files between the left-hand side (local files) and the right-hand side (HPC files).

Transferring data to Artemis using CyberDuck

Download Cyberduck from https://cyberduck.io, then open Cyberduck and connect to HPC:

  1. Click Open Connection.
  2. Select SFTP (SSH File Transfer Protocol) from the drop-down menu at the top of the Open Connection box.
  3. In the Server field, type hpc.sydney.edu.au.
  4. In the Username field, type your UniKey.
  5. In the Password field, type your UniKey password.
  6. Click Connect.
  7. When the Unknown fingerprint box appears, click the Always check box in the lower-left hand corner, then click Allow.

If you successfully logged in, you will see a directory listing of /home/abcd1234.

You can then transfer data to and from your local computer and HPC by dragging and dropping files between your computer’s file explorer and the Cyberduck window.

Transferring data to Artemis using command-line SFTP

If you have access to a Linux terminal on your local computer, you can use the command line program sftp to transfer files between your local computer and HPC. Open a Linux Terminal, such as Cygwin on Windows, or the built-in Terminal apps on MacOS or Linux, and type the following to connect to HPC:

sftp abcd1234@hpc.sydney.edu.au

This will start a command-line SFTP session. Your terminal will change to look like this:

sftp>

You can use SFTP commands to navigate the filesystem on your local and remote computers. A summary of SFTP commands is shown in the table below. If you connected using the SFTP command above, your local computer is your computer and the remote computer is Artemis.

SFTP Command Description
cd data Change directory to the directory called “data” on the remote computer
ls List files on the remote computer
lcd data Change directory to the directory called “data” on your local computer
lls List files on your local computer
pwd Present working directory on the remote computer
lpwd Present working directory on your local computer
mkdir data Make a directory called “data” on the remote computer
lmkdir data Make a directory called data on your local computer
put data Copy a file called “data” on the local computer to the remote computer
get data Copy a file called “data” on the remote computer to the local computer

Using scp or rsync

An example rsync command to transfer data from your local computer to Artemis is:

rsync -tvxPr /path/to/my/files abcd1234@hpc.sydney.edu.au:/scratch/PANDORA

Important tip for Windows users

Any files you transfer from Windows to Artemis HPC may be incompatible with Artemis due to differences in line endings between Windows and Linux. For example, if you submit a PBS script to Artemis that was created on Windows, it will submit normally, but it will fail with the following error:

-bash: /var/spool/PBS/mom_priv/jobs/1488497.pbsserver.SC: /bin/bash^M: bad interpreter: No such file or directory

The key here is the ^M character that appears in the error message, which indicates the file uses Windows line endings.

To fix this error, convert line endings from Windows format to an Artemis HPC compatible format by either:

  • Running dos2unix on files transferred to Artemis HPC from Windows
  • Saving files created on Windows in “Unix” format

Converting line endings using dos2unix

After copying files created on Windows to Artemis, run them through dos2unix:

[abcd1234@login4 ~]$ file input.txt
input.txt: ASCII English text, with CRLF line terminators

[abcd1234@login4 ~]$ dos2unix input.txt
dos2unix: converting file input.txt to UNIX format ...
[abcd1234@login4 ~]$ file input.txt
input.txt: ASCII English text

Converting line endings using Notepad++

Using Notepad++, you can request your file is saved with Artemis HPC compatible line endings. Click Edit -> EOL Conversion -> Unix (LF), then click Save. If you have followed this procedure, you can then copy this saved file to Artemis and it will be compatible.