OWL - Water-cooled AMD CPU
The OWL cluster equipment was acquired in FY23 but full commissioning of the cluster has been delayed by prerequisite datacenter renovation to integrate the direct water-cooling system with the building and datacenter where it is housed. As of February 2024, it is in the late stages of deployment and testing.
Direct water-cooling of the base compute nodes allows for running at boost speeds (3.8GHz) indefinitely which is 40% higher than the base clock rate. Tinkercliffs base compute nodes run at 2.0GHz.
AMD’s “Genoa” codename architecture is the first to feature AVX-512 instructions which provides 512-bit width vectorization (ie. eight-way FP64 SIMD in each clock-cycle). Tinkercliffs base compute nodes support the previous generation AVX2 instructions which has 256-bit width
12 memory channels per socket (24 per node) provide much higher aggregate memory bandwidth and increased granularity which should provide substantial speedup for memory-bandwidth constrained workload such as finite-element analysis.
DDR5-4800 memory provides a nominal 50% speed increase over DDR4-3200 on Tinkercliffs
768GB memory per node provides 8GB memory per core compared to Tinkercliffs which has 2GB/core
Three nodes are equipped with very-large memory (4TB or 8TB) enabling computational workloads for which we have never had sufficient memory resources.
The large memory nodes were not available in the AMD "Genoa" package at the time of acquisition and equipped with different processors (detail below) and are not water-cooled.
Overview
Base Compute Nodes |
Large Memory |
Huge Memory |
Totals |
|
|---|---|---|---|---|
Vendor |
Lenovo |
Lenovo |
Lenovo |
|
Chip |
||||
Nodes |
84 |
2 |
1 |
87 |
Cores/Node |
96 |
128 |
128 |
|
Memory (GiB)/Node |
768 DDR5-4800 |
4019 DDR4-3200 |
8038 DDR4-3200 |
|
Local Disk |
2.9TB NVMe |
2.9TB NVMe |
2.9TB NVMe |
|
Interconnect |
shared 200Gbps HDR Infiniband: |
shared 200Gbps HDR Infiniband: |
shared 200Gbps HDR Infiniband: |
|
Total Memory |
64512 |
8038 |
8038 |
|
Total Cores |
8064 |
256 |
128 |
8448 |
Theoretical Peak |
245.1456 TFLOPS |