Transferring data between your local computer and HPC¶
Transferring data to Artemis using FileZilla¶
FileZilla is a freely available SFTP client. You can download a copy from their website: http://filezilla-project.org.
To transfer data from your local computer to HPC using FileZilla, follow these steps:
- Go to File -> Site Manager in the menu bar at the top of the program.
- Click New Site in the lower left-hand corner and give this connection a name. The name is for your reference only.
- Click the General tab in Site Manager.
hpc.sydney.edu.auin the Host field.
- Leave the port blank.
- Select the SFTP protocol in the protocol field.
- Select Normal in the logon type field.
- Type your UniKey in the User field.
- Type your UniKey password in the Password field.
- Click Connect.
- An Unknown host key box will pop up. Click Always trust this host, add this key to the cache box and then click OK.
If you successfully logged on, you will see a message saying “Directory listing of “/home/abcd1234” successful”.
To transfer data between HPC and your local computer, drag and drop files between the left-hand side (local files) and the right-hand side (HPC files).
Transferring data to Artemis using CyberDuck¶
Download Cyberduck from https://cyberduck.io, then open Cyberduck and connect to HPC:
- Click Open Connection.
- Select SFTP (SSH File Transfer Protocol) from the drop-down menu at the top of the Open Connection box.
- In the Server field, type
- In the Username field, type your UniKey.
- In the Password field, type your UniKey password.
- Click Connect.
- When the Unknown fingerprint box appears, click the Always check box in the lower-left hand corner, then click Allow.
If you successfully logged in, you will see a directory listing of
You can then transfer data to and from your local computer and HPC by dragging and dropping files between your computer’s file explorer and the Cyberduck window.
Transferring data to Artemis using command-line SFTP¶
If you have access to a Linux terminal on your local computer, you can use the command line program
sftp to transfer
files between your local computer and HPC. Open a Linux Terminal, such as Cygwin
on Windows, or the built-in Terminal apps on MacOS or Linux, and type the following to connect to HPC:
This will start a command-line SFTP session. Your terminal will change to look like this:
You can use SFTP commands to navigate the filesystem on your local and remote computers. A summary of SFTP commands is shown in the table below. If you connected using the SFTP command above, your local computer is your computer and the remote computer is Artemis.
||Change directory to the directory called “data” on the remote computer|
||List files on the remote computer|
||Change directory to the directory called “data” on your local computer|
||List files on your local computer|
||Present working directory on the remote computer|
||Present working directory on your local computer|
||Make a directory called “data” on the remote computer|
||Make a directory called data on your local computer|
||Copy a file called “data” on the local computer to the remote computer|
||Copy a file called “data” on the remote computer to the local computer|
Using scp or rsync¶
An example rsync command to transfer data from your local computer to Artemis is:
rsync -tvxPr /path/to/my/files email@example.com:/scratch/PANDORA
Important tip for Windows users¶
Any files you transfer from Windows to Artemis HPC may be incompatible with Artemis due to differences in line endings between Windows and Linux. For example, if you submit a PBS script to Artemis that was created on Windows, it will submit normally, but it will fail with the following error:
-bash: /var/spool/PBS/mom_priv/jobs/1488497.pbsserver.SC: /bin/bash^M: bad interpreter: No such file or directory
The key here is the
^M character that appears in the error message, which
indicates the file uses Windows line endings.
To fix this error, convert line endings from Windows format to an Artemis HPC compatible format by either:
dos2unixon files transferred to Artemis HPC from Windows
- Saving files created on Windows in “Unix” format
Converting line endings using dos2unix¶
After copying files created on Windows to Artemis, run them through
[abcd1234@login4 ~]$ file input.txt input.txt: ASCII English text, with CRLF line terminators [abcd1234@login4 ~]$ dos2unix input.txt dos2unix: converting file input.txt to UNIX format ... [abcd1234@login4 ~]$ file input.txt input.txt: ASCII English text
Converting line endings using Notepad++¶
Using Notepad++, you can request your file is saved with Artemis HPC compatible line endings. Click Edit -> EOL Conversion -> Unix (LF), then click Save. If you have followed this procedure, you can then copy this saved file to Artemis and it will be compatible.