Transferring data between HPC and the Research data store (RDS)¶
Artemis’s storage is tuned for performance. Therefore, it is not backed up, space is limited, and the available space is shared among all Artemis users. In the interests of keeping your data safely backed up and to ensure there is free space available for active Artemis users, we encourage you to move any data you don’t need on Artemis to the Research Data Store.
The remainder of this page explains how to move data from Artemis to Classic RDS, and from Artemis to RCOS.
Transferring data between RCOS and Artemis¶
RCOS is a linux-based file store, and can be accessed from Artemis login nodes and the data transfer queue in the directory:
remembering to replace PANDORA with your short project name.
If you are logged into Artemis, you can use standard Linux
commands, such as
rsync to move data to and from Artemis and RCOS.
Transferring data between Classic RDS and Artemis using a GUI¶
The most straightforward way to transfer data between Artemis and Classic RDS is to mount Classic RDS as a network drive, then connect to Artemis using FileZilla and drag-and-drop data between the Artemis (on the right-hand side of the FileZilla window) and your Classic RDS network drive (on the left-hand side of the FileZilla window).
This method transfers data from Classic RDS to Artemis HPC via your local computer, so this method of data transfer is only recommended if you have a fast internet connection, preferably on-campus, with unlimited downloads and uploads.
Transferring data between Classic RDS and Artemis using the command line¶
Alternatively, it is possible to transfer data from Artemis to Classic RDS via the Artemis command line using smbclient. To connect to Classic RDS from Artemis, type the following command:
smbclient //research-data.shared.sydney.edu.au/RDS-01 -U abcd1234 -W SHARED
where RDS-01 is the volume where your Classic RDS drive is stored. If your project was created before May 2017, replace RDS-01 with your faculty code.
If you successfully logged in, your command prompt will change to this:
Once logged into smbclient, your command prompt will allow you to
navigate the Classic RDS filesystem using commands like
move a file from Artemis to Classic RDS, use
put. To move a file
from Classic RDS to Artemis, use
get. For example, to move a file called
my_data.out from Artemis to Classic RDS, type:
smb: \> put my_data.out
To move a file called
other_data.txt from Classic RDS to Artemis, type:
smb: \> get other_data.txt
To move multiple files using a single command, use the
commands (multiple put and multiple get). By default, smbclient will ask
to confirm whether you want to transfer each individual file. To disable
this functionality, type
prompt off before using
smb: \> prompt off smb: \> mput my\_data\*
To run commands on Artemis while logged into smbclient, place an exclamation mark before your command. For example, to view files in your current directory on Artemis, type:
smb: \> !ls
To transfer data recursively from Artemis to Classic RDS (that is, from
the current directory and all sub-directories), use the
smb: \> recurse on smb: \> prompt off smb: \> mput directory
directory is the directory you want to copy to Classic RDS. The
recurse on and
prompt off commands toggle recursive mode and disable
prompting for every file transfer, so these only need to be run once. To
move data from Classic RDS to Artemis, replace