Hardware

Artemis is a 7636-core InfiniBand cluster comprised of the following nodes. Some of these nodes are reserved for research groups who won a “Grand Challenge” node allocation, or who purchased compute nodes for their research group or school and are having them hosted in Artemis. See the below table for a summary of available resources in Artemis.

  Up to 12 hour jobs Up to 24 hour jobs Up to 1 week jobs Up to 3 week jobs GPU jobs High memory jobs Data transfer jobs1 Interactive jobs2 “Scavenger” jobs3
Number of Nodes 1 24 41 49 7 3 2 1 via PBS Pro, 2 via “NoMachine” 61
Available cores 24 576 1968 1568 252 cores, 28 GPUs 192 24 24 1800
Maximum requestable cores per node (or chunk) 4 24 32 32 36 cores, 4 GPUs 64 2 4 24
RAM requestable per node (GB) 16 123 123 123 185 6100 16 16 123

Table footnotes:

  1. Jobs that are for I/O workloads or data transfer. Compute jobs will be terminated.
  2. Interactive jobs are jobs where you have interactive access to a compute node. The PBS Pro node is exactly like a compute node, and the NoMachine nodes are for opening programs with GUIs where graphics processing is done server-side.
  3. “Scavenger” jobs are low-priority jobs that run using idle resources in Artemis Grand Challenge scheme winners allocations. All Artemis users are welcome to submit scavenger jobs, however they will be terminated before finishing if a Grand Challenge allocation member submits work to their allocation and requires resources being used by a scavenger job.

Core allocations

Some Artemis nodes are reserved for certain groups. These nodes are either granted to researchers who won dedicated access to compute nodes as part of the Artemis Grand Challenge scheme, or who have ownership of compute nodes and choose to have them hosted in Artemis. As of March 2018, 1800 cores are allocated to Artemis Grand Challenge scheme winners, and 1136 cores and 80 V100 GPUs are owned by groups who have chosen to host compute resources in Artemis.

Detailed Technical Specifications

The information here is of a detailed, technical nature and is not required knowledge to use Artemis. If you are an advanced user, the information here may prove useful if you wish to compile specifically for Artemis.

Artemis is made up of a number of components, including the login nodes, compute nodes, storage system, and management nodes. The compute nodes, Lustre storage and management nodes are all connected by a high performance, low latency “interconnect” based on Mellanox InfiniBand (IB), which is a proprietary networking infrastructure. There is also a 10 Gbps Ethernet management network interconnecting the compute, login and management nodes for compute node and batch job management.

Artemis has a mix of Haswell, Broadwell and Skylake compute nodes based on dual socket servers.

The key features of the Haswell nodes are:

Attribute Value
Base vendor model Dell PowerEdge R630 Server
CPU model Intel Xeon E5-2680 V3 @ 2.50 GHz
CPU generation Haswell
Number of cores per node 24 (2 x 12)
Resident RAM 128 GB (8 x 16 GB) DDR3 DIMMs
Disk storage 2 x 1 TB 7k NL-SAS in RAID 1
Ethernet interface 10 Gbps
InfiniBand interface 56 Gbps FDR InfiniBand
Queues that use these nodes small, small-express, scavenger

The key features of the Broadwell nodes are:

Attribute Value
Base vendor model Dell PowerEdge C6320 Server
CPU model Intel Xeon E5-2697A V4 @ 2.6 GHz
CPU generation Broadwell
Number of cores per node 32 (2 x 16)
Resident RAM 128 GB (8 x 16 GB) DDR3 DIMMs
Disk storage 2 x 1 TB 10k SAS in RAID 1
Ethernet interface 10 Gbps
InfiniBand interface 56 Gbps FDR InfiniBand
Queues that use these nodes large

The key features of the Skylake nodes are:

Attribute Value
Base vendor model Dell PowerEdge C6420 Server
CPU model Intel Xeon Platinum 8160 @ 2.1 GHz
CPU generation Skylake
Number of cores per node 48 (2 x 24)
Resident RAM 192 GB
Disk storage  
Ethernet interface 10 Gbps
InfiniBand interface 56 Gbps FDR InfiniBand
Queues that use these nodes normal

High Memory Nodes

Artemis has three very high memory compute nodes based on a quad socket server with 6 TB of RAM:

Attribute Value
Base vendor model Dell PowerEdge R930 Server
CPU model Intel Xeon E7-8860 V3 @ 2.20 GHz
CPU generation Haswell
Number of cores per node 64 (4 x 16)
Resident RAM 6144 GB (96 x 64 GB) DDR4 DIMMs
Disk storage 2 x 200 GB (12 Gbps) SAS SSD, 5 x 2 TB (2 Gbps) SAS SSD
Ethernet interface 10 Gbps
InfiniBand interface 56 Gbps FDR InfiniBand
Queues that use these nodes highmem

GPU Nodes

There are seven Dell C4140 GPU nodes with four Nvidia V100 SXM2 16 GB GPGPUs and two x 16 core CPUs available to all Artemis users. The four GPUs in each compute node are connected using 300 Gbit/s NVLINK2. Technical details about these nodes are shown below:

Attribute Value
Base vendor model Dell EMC PowerEdge C4140
CPU model Intel Xeon Gold 6140 @ 2.3 GHz
CPU generation Skylake
Number of cores per node 36 (2 x 18)
Resident RAM 192 GB
GPUs 4 x NVIDIA V100 SXM2
GPU RAM 16 GB HBM2 per GPU
Disk storage  
Ethernet interface 10 Gbps
InfiniBand interface 56 Gbps FDR InfiniBand
Queues that use these nodes gpu

High Performance File System

Artemis has globally accessible, high-performance “Lustre” file system with 56 Gbps FDR InfiniBand interconnect to the rest of the cluster.

The /home, /scratch and /project directories are all stored on Lustre.

Note

Lustre performs best when doing I/O in large chunks (greater than 1 MB). If you are performing I/O constantly on very small files (less than 1 MB), the performance of Lustre filesystem will be poor for yourself and all other all Artemis users. You may want to consider using /tmp, which uses local node disk if you have a program that does lots of small I/O.

Management and Control Nodes

The management nodes are accessible to the vendor’s system administrators only and are used to manage workflow within the cluster.

These nodes consist of two Dell R630 servers with dual 16 core CPUs and 512 GB of memory with a Compellant storage subsystem running virtualised VMware ESXI 6.

Node Interconnect

All nodes and the Lustre filesystem are connected using 56 Gbps FDR InfiniBand interconnect in a 2:1 blocking fat tree configuration. The InfiniBand interconnect provides low latency communications between compute nodes for maximum MPI bandwidth.

Ethernet Networking

10 Gbps redundant switched connections are available between the MPLS switches provided by the University and the Ethernet fabric. The 10 Gbps Ethernet provides access to the login nodes from AARnet and the management network. The 1 Gbps Ethernet provides out-of-band connectivity to the compute nodes.