Research Computing Optimised Storage (RCOS)

RCOS is a Linux-style data storage service, accessible via SFTP from an RCOS server:

  • rcos.sydney.edu.au
  • rcos-int.sydney.edu.au
  • hpc.sydney.edu.au

All three servers grant access to the same RCOS storage, so it does not matter which server you use.

rcos.sydney.edu.au and rcos-int.sydney.edu.au are servers whose sole purpose is to grant access to RCOS. hpc.sydney.edu.au is Artemis HPC, which has RCOS mounted on the login nodes and on the data transfer nodes. If your project has access to RCOS and Artemis HPC, you can transfer data to RCOS using any RCOS server. If you do not have Artemis HPC access, use rcos.sydney.edu.au and/or rcos-int.sydney.edu.au.

The path to your RCOS folder on all three servers is /rds/PRJ-<Project>, where <Project> is your short project name as specified in your DashR project (do not include the angle brackets).

Note

You cannot access RCOS using a web browser or by “mapping a network drive”. You must use the SSH or SFTP protocols. Instructions to connect to RCOS via SFTP are provided further down this page.

There are many ways of establishing a SFTP session. You only have to choose one of the below connection methods to access RCOS, so use whatever works best for you. If you need to access to an internal service (rcos-int or hpc) from off-campus, connect to the University VPN first, then connect to the relevant internal service.

  External (rcos) Internal (rcos-int or hpc)
FileZilla
../_images/tick.svg
../_images/tick.svg
Cyberduck
../_images/tick.svg
../_images/tick.svg
SFTP client
../_images/tick.svg
../_images/tick.svg
Interactive SSH
../_images/cross.svg
../_images/tick.svg
scp
../_images/cross.svg
../_images/tick.svg
rsync
../_images/cross.svg
../_images/tick.svg

Important

In all examples below, whenever you see angle brackets < >, replace whatever is inside the angle brackets AND the angle brackets with the content relevant to you. For example, if you see <UniKey>, replace it with your UniKey. If your UniKey was abcd1234, replace <UniKey> with abcd1234.

Connecting to RCOS using FileZilla

FileZilla is a freely available SFTP client. You can download a copy from their website: http://filezilla-project.org. Recently, the FileZilla installer comes bundled with affiliate software. Ensure you select “no” when asked if you wish to install affiliate software. Only say “yes” to installing FileZilla.

To transfer data from your local computer to RCOS using FileZilla, follow these steps:

  1. Go to File -> Site Manager in the menu bar at the top of the program.
  2. Click New Site in the lower left-hand corner and give this connection a name. The name is for your reference only.
  3. Click the General tab in Site Manager.
  4. Type rcos.sydney.edu.au in the Host field.
  5. Leave the port blank.
  6. Select the SFTP protocol in the protocol field.
  7. Select Normal in the logon type field.
  8. Type your UniKey in the User field.
  9. Type your UniKey password in the Password field.
  10. Click Connect.
  11. An Unknown host key box will pop up. Click Always trust this host, add this key to the cache box and then click OK.

If you successfully logged on, you will see a message saying “Directory listing of “/home/<unikey>” successful”.

  1. Type /rds/PRJ-<Project> in the remote site field, remembering to replace <Project> with your short project name, as specified in your DashR project.

To transfer data between RCOS and your local computer, drag and drop files between the left-hand side (local files) and the right-hand side (RCOS files).

Note

If you do not complete step 12, your data will be saved in /home/<UniKey>, which is not accessible to Artemis HPC or your group members.

Important

Do not store research data in your home directories on rcos.sydney.edu.au or rcos-int.sydney.edu.au. These directories have limited storage and are provided to store environment configuration files and SSH keys only. We will ask you to move your data to /rds/PRJ-<Project> if you store excessive amounts of data in your RCOS home directory.

Connecting to RCOS using CyberDuck

Download Cyberduck from https://cyberduck.io, then open Cyberduck and connect to RCOS:

  1. Click Open Connection.
  2. Select SFTP (SSH File Transfer Protocol) from the drop-down menu at the top of the Open Connection box.
  3. In the Server field, type rcos.sydney.edu.au.
  4. In the Username field, type your UniKey.
  5. In the Password field, type your UniKey password.
  6. Click Connect.
  7. When the Unknown fingerprint box appears, click the Always check box in the lower-left hand corner, then click Allow.

If you successfully logged in, you will see a directory listing of /home/<UniKey>.

  1. Press Ctrl + g, then type /rds/PRJ-<Project>, remembering to replace <Project> with your short project name, as specified in your DashR project.

You can then transfer data to and from your local computer and RCOS by dragging and dropping files between your computer’s file explorer and the Cyberduck window.

Important

Do not store research data in your home directories on rcos.sydney.edu.au or rcos-int.sydney.edu.au. These directories have limited storage and are provided to store environment configuration files and SSH keys only. We will ask you to move your data to /rds/PRJ-<Project> if you store excessive amounts of data in your RCOS home directory.

Connecting to RCOS using command-line SFTP

If you have access to a Linux terminal, you can use the command line program sftp to transfer files between your local computer and RCOS. Open a Linux Terminal, such as Cygwin on Windows, or the built-in Terminal apps on MacOS or Linux, and type the following to connect to RCOS:

sftp <UniKey>@rcos.sydney.edu.au

remembering to replace <UniKey> with your UniKey. This will start a command-line SFTP session. Your terminal will change to look like this:

sftp>

You can use SFTP commands to navigate the filesystem on your local and remote computers. A summary of SFTP commands is shown in the table below. If you connected using the SFTP command above, your local computer is your computer and the remote computer is RCOS.

SFTP Command Description
cd <directory> Change directory on the remote computer
ls List files on the remote computer
lcd Change directory on your local computer
lls List files on your local computer
pwd Present working directory on the remote computer
lpwd Present working directory on your local computer
mkdir <directory> Make a directory on the remote computer
lmkdir <directory> Make a directory on your local computer
put <file(s)> Copy files from your local computer to the remote computer
get <file(s)> Copy files from the remote computer to your local computer

Important

Do not store research data in your home directories on rcos.sydney.edu.au or rcos-int.sydney.edu.au. These directories have limited storage and are provided to store environment configuration files and SSH keys only. We will ask you to move your data to /rds/PRJ-<Project> if you store excessive amounts of data in your RCOS home directory.

Using scp or rsync

scp and rsync can be used on rcos-int.sydney.edu and hpc.sydney.edu.au only. rcos.sydney.edu.au only allows SFTP connections. An example rsync command to transfer data from your local computer to RCOS is:

rsync -tvxPr /path/to/my/files <UniKey>@rcos-int.sydney.edu.au:/rds/PRJ-<Project>/

Important

Do not store research data in your home directories on rcos.sydney.edu.au or rcos-int.sydney.edu.au. These directories have limited storage and are provided to store environment configuration files and SSH keys only. We will ask you to move your data to /rds/PRJ-<Project> if you store excessive amounts of data in your RCOS home directory.