Biomed - Biomedical research

Overview

The biomed cluster has 3 nodes, 192 CPU cores, 3 TB RAM, and 8 NVIDIA A100 GPUs. Biomed hardware is summarized in the table below.

Node Type	CPU	GPU	Total
Chip	AMD EPYC 7542 Milan	AMD EPYC 7542 Milan	-
Architecture	Zen 2	Zen 2	-
Slurm features	-	-	-
Nodes	2	1	3
GPUs	-	8x NVIDIA A100-80G	8
Cores/Node	64	64	-
Memory (GB)/Node	512	2,048	-
Maximum Memory for Slurm (GB)/Node	495	2,007	-
Total Cores	128	64	192
Total Memory (GB)	1,536	2,048	3,584
Local Disk	240GB SSD	240GB SSD	-
Interconnect	HDR-100 IB	HDR-100 IB	-

Access

The biomed cluster is set up to host projects which require some computational scale but are subject to restrictions such as NIST SP 800-171 required by NIH or other agencies. Researchers planning to use the Biomedical Cluster for NIH controlled-access datasets (e.g., dbGaP, ABCD, and other NIH repositories) should review the Accessing NIH Controlled-Access Datasets guidance for information about institutional approvals, security requirements, and the process for obtaining access. Researchers will also need to consult with Advanced Research Computing (ARC) personnel to establish access to the cluster and receive instructions for its use.

The cluster may also support other research projects with specialized security or compliance requirements, subject to applicable approvals.

For questions about accessing the Biomedical Cluster for sensitive health or controlled-access research data, contact the Privacy and Research Data Protection Program at prdp@vt.edu. For technical assistance with the cluster, contact Advanced Research Computing.

Get Started

Biomed can be accessed via the login node using your VT credentials:

• biomed1.arc.vt.edu

Access is limited to university-managed devices and by authorized researchers subject to a user agreement. Access from personal devices is not permitted.

Partitions

Users submit jobs to partitions of the cluster depending on the type of resources needed (for example, CPUs or GPUs). Features are optional restrictions users can indicate in their job submission to restrict the execution of their job to nodes meeting specific requirements. If users do not specify the amount of memory requested for a job, the parameter DefMemPerCPU will automatically determine the amount of memory for the job based on the number of CPU cores requested. If the users do not specify the number of CPU cores on a GPU job, the parameter DepCpuPerGPU will automatically determine the number of CPU cores based on the number of GPUs requested. Jobs will be billed against the user’s allocation accounting for the utilization of number of CPU cores, memory, and GPU time. Consult the Slurm configuration to understand how to specify the parameters for your job.

Partition	normal_q	a100_normal_q
Node Type	CPU	GPU
Features	-	-
Number of Nodes	2	1
DefMemPerCPU (MB)	7920	32112
DefCpuPerGPU	-	8
PreemptMode	OFF	OFF
DefaultTime	1h	1h
MaxTime	UNLIMITED	UNLIMITED

Optimization

The performance of jobs can be greatly enhanced by appropriate optimizations being applied. Not only does this reduce the execution time of jobs but it also makes more efficient use of the resources for the benefit of all.

See the tuning guides available at https://developer.amd.com and https://www.intel.com/content/www/us/en/developer/

General principles of optimization:

• Cache locality really matters - process pinning can make a big difference on performance.

• Hybrid programming often pays off - one MPI process per L3 cache with 4 threads is often optimal.

• Use the appropriate -march flag to optimize the compiled code and -gencode flag when using the NVCC compiler.

Suggested optimization parameters:

Node Type	CPU	GPU
CPU arch	Zen 2	Zen 2
Compiler flags	-march=znver2	-march=znver2
GPU arch	-	NVIDIA A100
Compute Capability	-	8.0
NVCC flags	-	-gencode=arch=compute_80,code=sm_80