Hardware

Artemis is a 4264-core InfiniBand cluster comprised of the following nodes:

  Artemis Phase 1 Nodes Artemis Phase 2 Nodes
  Standard Memory High Memory GPU Standard Memory High Memory
Number of Nodes 56 2 5 80 3
Cores per node 2 x 12 2 x 12

2 x 12 CPUs

2 GPUs

2 x 16 4 x 16
RAM per node 128 GB 512 GB 128 GB 128 GB 6 TB

Core allocations

Some Artemis nodes are reserved for certain groups:

  • Civil Engineering have ownership of 13 nodes (416 cores); they have exclusive access to these nodes.
  • Some researchers were granted strategic allocations on Artemis. These nodes are reserved for these projects. Currently, 936 cores are reserved for strategic allocations.

The remaining nodes (2912 cores in total) are available to all users of the system.

Detailed Technical Specifications

The information here is of a detailed, technical nature and is not required knowledge to use Artemis. If you are an advanced user, the information here may prove useful if you wish to compile specifically for Artemis.

Artemis is made up of a number of components, including the login nodes, compute nodes, storage system, and management nodes. The compute nodes, Lustre storage and management nodes are all connected by a high performance, low latency “interconnect” based on Mellanox InfiniBand (IB), which is a proprietary networking infrastructure. There is also a 10 Gbps Ethernet management network interconnecting the compute, login and management nodes for compute node and batch job management.

Artemis has 56 “Standard” Haswell compute nodes and 80 “Standard” Broadwell compute nodes based on dual socket servers.

The key features of the Haswell nodes are:

Attribute Value
Base vendor model Dell PowerEdge R630 Server
CPU model Intel Xeon E5-2680 V3 @ 2.50 GHz
CPU generation Haswell
Number of cores per node 24 (2 x 12)
Resident RAM 128 GB (8 x 16 GB) DDR3 DIMMs
Disk storage 2 x 1 TB 7k NL-SAS in RAID 1
Number of 10 Gbps interfaces 2
Number of 1 Gbps interfaces 2
InfiniBand interface FDR InfiniBand

The key features of the Broadwell nodes are:

Attribute Value
Base vendor model Dell PowerEdge C6320 Server
CPU model Intel Xeon E5-2697A V4 @ 2.6 GHz
CPU generation Broadwell
Number of cores per node 32 (2 x 16)
Resident RAM 128 GB (8 x 16 GB) DDR3 DIMMs
Disk storage 2 x 1 TB 10k SAS in RAID 1
Number of 10 Gbps interfaces 2
Number of 1 Gbps interfaces 2
InfiniBand interface FDR InfiniBand

High Memory Nodes

Artemis has three very high memory compute nodes based on a quad socket server with 6 TB of RAM:

Attribute Value
Base vendor model Dell PowerEdge R930 Server
CPU model Intel Xeon E7-8860 V3 @ 2.20 GHz
CPU generation Haswell
Number of cores per node 64 (4 x 16)
Resident RAM 6144 GB (96 x 64 GB) DDR4 DIMMs
Disk storage 2 x 200 GB (12 Gbps) SAS SSD, 5 x 2 TB (2 Gbps) SAS SSD
Number of 10 Gbps interfaces 2
Number of 1 Gbps interfaces 2
InfiniBand interface FDR InfiniBand

There are also two high memory nodes based on a dual socket server with 512 GB of RAM:

Attribute Value
Base vendor model Dell PowerEdge R630 Server
CPU model Intel Xeon E5-2680 V3 @ 2.50 GHz
CPU generation Haswell
Number of cores per node 24 (2 x 12)
Resident RAM 512 GB (16 x 32 GB) DDR4 DIMMs
Disk storage 4 x 1 TB 7k NL-SAS in RAID 10
Number of 10 Gbps interfaces 2
Number of 1 Gbps interfaces 2
InfiniBand interface FDR InfiniBand

GPU Nodes

There are 5 GPU compute nodes, based on Dell PowerEdge R730 servers. Each is fitted with dual 12 core CPUs and 2 NVIDIA K40 GPUs, giving 10 GPUs in total.

High Performance File System

Artemis has globally accessible, high-performance “Lustre” file system with 56 Gbps FDR InfiniBand interconnect to the rest of the cluster.

The /home, /scratch and /project directories are all stored on Lustre.

Note

Lustre performs best when doing I/O in large chunks (greater than 1 MB). If you are performing I/O constantly on very small files (less than 1 MB), the performance of Lustre filesystem will be poor for yourself and all other all Artemis users. You may want to consider using /tmp, which uses local node disk if you have a program that does lots of small I/O.

Management and Control Nodes

The management nodes are accessible to the vendor’s system administrators only and are used to manage workflow within the cluster.

These nodes consist of two Dell R630 servers with dual 16 core CPUs and 512 GB of memory with a Compellant storage subsystem running virtualised VMware ESXI 6.

Node Interconnect

All nodes and the Lustre filesystem are connected using 56 Gbps FDR InfiniBand interconnect in a 2:1 blocking fat tree configuration. The InfiniBand interconnect provides low latency communications between compute nodes for maximum MPI bandwidth.

Ethernet Networking

10 Gbps redundant switched connections are available between the MPLS switches provided by the University and the Ethernet fabric. The 10 Gbps Ethernet provides access to the login nodes from AARnet and the management network. The 1 Gbps Ethernet provides out-of-band connectivity to the compute nodes.