Transferring data between HPC and the Research data store (RDS)

Artemis’s storage is tuned for performance. Therefore, it is not backed up, space is limited, and the available space is shared among all Artemis users. In the interests of keeping your data safely backed up and to ensure there is free space available for active Artemis users, we encourage you to move any data you don’t need on Artemis to the Research Data Store.

There are two different flavours of RDS: Classic RDS and Research Computing Optimised Storage (RCOS). Of these two options, RCOS is the easiest to use with Artemis.

The remainder of this page explains how to move data from Artemis to Classic RDS, and from Artemis to RCOS.

Transferring data between RCOS and Artemis

RCOS is a linux-based file store, and can be accessed from Artemis login nodes and the data transfer queue in the directory:

/rds/PRJ-PANDORA

remembering to replace PANDORA with your short project name.

If you are logged into Artemis, you can use standard Linux commands, such as cp, mv and rsync to move data to and from Artemis and RCOS.

Note

Only the Artemis login nodes and data transfer nodes can access /rds. If you need to analyse data on Artemis, you must transfer your data from /rds to either /project or /scratch before submitting your job.

Classic RDS

Transferring data between Classic RDS and Artemis using a GUI

The most straightforward way to transfer data between Artemis and Classic RDS is to mount Classic RDS as a network drive, then connect to Artemis using FileZilla and drag-and-drop data between the Artemis (on the right-hand side of the FileZilla window) and your Classic RDS network drive (on the left-hand side of the FileZilla window).

Note

This method transfers data from Classic RDS to Artemis HPC via your local computer, so this method of data transfer is only recommended if you have a fast internet connection, preferably on-campus, with unlimited downloads and uploads.

Transferring data between Classic RDS and Artemis using the command line

Alternatively, it is possible to transfer data from Artemis to Classic RDS via the Artemis command line using smbclient. To connect to Classic RDS from Artemis, type the following command:

smbclient //research-data.shared.sydney.edu.au/RDS-01 -U abcd1234 -W SHARED

where RDS-01 is the volume where your Classic RDS drive is stored. If your project was created before May 2017, replace RDS-01 with your faculty code.

If you successfully logged in, your command prompt will change to this:

smb: \>

Once logged into smbclient, your command prompt will allow you to navigate the Classic RDS filesystem using commands like cd and ls. To move a file from Artemis to Classic RDS, use put. To move a file from Classic RDS to Artemis, use get. For example, to move a file called my_data.out from Artemis to Classic RDS, type:

smb: \> put my_data.out

To move a file called other_data.txt from Classic RDS to Artemis, type:

smb: \> get other_data.txt

To move multiple files using a single command, use the mput and mget commands (multiple put and multiple get). By default, smbclient will ask to confirm whether you want to transfer each individual file. To disable this functionality, type prompt off before using mput or mget commands:

smb: \> prompt off
smb: \> mput my\_data\*

To run commands on Artemis while logged into smbclient, place an exclamation mark before your command. For example, to view files in your current directory on Artemis, type:

smb: \> !ls

To transfer data recursively from Artemis to Classic RDS (that is, from the current directory and all sub-directories), use the recurse command:

smb: \> recurse on
smb: \> prompt off
smb: \> mput directory

where directory is the directory you want to copy to Classic RDS. The recurse on and prompt off commands toggle recursive mode and disable prompting for every file transfer, so these only need to be run once. To move data from Classic RDS to Artemis, replace mput with mget.