(owl)=

# OWL - Water-cooled AMD CPU

The OWL cluster equipment was acquired in FY23 but full commissioning of the cluster has been delayed by prerequisite datacenter renovation to integrate the direct water-cooling system with the building and datacenter where it is housed. As of February 2024, it is in the late stages of deployment and testing.

 - Direct water-cooling of the base compute nodes allows for running at boost speeds (3.8GHz) indefinitely which is 40% higher than the base clock rate. Tinkercliffs base compute nodes run at 2.0GHz.
 - AMD's "Genoa" codename architecture is the first to feature AVX-512 instructions which provides 512-bit width vectorization (ie. eight-way FP64 SIMD in each clock-cycle). Tinkercliffs base compute nodes support the previous generation AVX2 instructions which has 256-bit width
 - 12 memory channels per socket (24 per node) provide much higher aggregate memory bandwidth and increased granularity which should provide substantial speedup for memory-bandwidth constrained workload such as finite-element analysis.
 - DDR5-4800 memory provides a nominal 50% speed increase over DDR4-3200 on Tinkercliffs
 - 768GB memory per node provides 8GB memory per core compared to Tinkercliffs which has 2GB/core
 - Three nodes are equipped with very-large memory (4TB or 8TB) enabling computational workloads for which we have never had sufficient memory resources. 

 ```Note
The large memory nodes were not available in the AMD "Genoa" package at the time of acquisition and equipped with different processors (detail below) and are not water-cooled.
 ```

## Overview ##

|                   | Base Compute Nodes                                                  | Large Memory                                                      | Huge Memory                                                       | Totals |
|-------------------|---------------------------------------------------------------------|-------------------------------------------------------------------|-------------------------------------------------------------------|--------|
| Vendor            | Lenovo                                                              | Lenovo                                                            | Lenovo                                                            |        |
| Chip              | [AMD EPYC 9454 - Genoa](https://en.wikichip.org/wiki/amd/epyc/9454) | [AMD EPYC 7763 Milan](https://en.wikichip.org/wiki/amd/epyc/7763) | [AMD EPYC 7763 Milan](https://en.wikichip.org/wiki/amd/epyc/7763) |        |
| Nodes             | 84                                                                  | 2                                                                 | 1                                                                 | 87     |
| Cores/Node        | 96                                                                  | 128                                                               | 128                                                               |        |
| Memory (GiB)/Node | 768 DDR5-4800                                                                | 4019 DDR4-3200                                                             | 8038 DDR4-3200                                                             |        |
| Local Disk        | 2.9TB NVMe                                                          | 2.9TB NVMe                                                        | 2.9TB NVMe                                                        |        |
| Interconnect      | shared 200Gbps HDR Infiniband:<br> 100Gbps effective data rate                                                  | shared 200Gbps HDR Infiniband:<br> 100Gbps effective data rate                                                | shared 200Gbps HDR Infiniband:<br> 100Gbps effective data rate                                                |        |
| Total Memory      | 64512                                                               | 8038                                                              | 8038                                                              |        |
| Total Cores       | 8064                                                                | 256                                                               | 128                                                               | 8448   |
| Theoretical Peak | 245.1456 TFLOPS | 

## AMD Resources
[Tuning Guide AMD EPYC 9004](https://www.amd.com/content/dam/amd/en/documents/epyc-technical-docs/tuning-guides/58002_amd-epyc-9004-tg-hpc.pdf)

[AOCC User Guide](https://www.amd.com/content/dam/amd/en/documents/pdfs/developer/aocc/aocc-v4.0-ga-user-guide.pdf)

## Benchmarks
### STREAM
### HPL
### HPCG