StarCCM+

Specifying nodes and processes per node on Slurm systems

This is in a job which had “nodes=3” and “ntasks-per-node=2”:

[brownm12@ca028 ~]$ STARNL=$(srun hostname | sort -u | awk -v NT=$SLURM_NTASKS_PER_NODE '{printf $1":"NT","}' | sed s/.$//)
[brownm12@ca028 ~]$ echo $STARNL
ca028:2,ca099:2,ca100:2

Breakdown:

  • STARNL=$( ... ) set bash variable STARNL to the output of the command enclosed in parentheses:

  • cmd1 | cmd2 | cmd3 send output of cmd1 as input to cmd2 and the resulting output of that to cmd3 (pipes)

  • srun hostname run SLURM_NTASKS instances of hostname command, one on each task (process) allocated to the job

  • sort -u sort the input and remove duplicate lines

  • awk -v NT=$SLURM_NTASKS_PER_NODE '{ ... }' run an awk script on the input, additionaly set the awk variable NT to a value

  • printf $1":"NT"," awk script to print the first column of each line of input, followed by a colon, the value stored in NT, and then a comma

  • sed s/.$// remove the last character of the input line using sed’s “swap” capability

Example usage in a script

So in your job script, prior to the starccm+ command, you would have:

STARNL=$(srun hostname | sort -u | awk -v NT=$SLURM_NTASKS_PER_NODE '{printf $1":"NT","}' | sed s/.$//)
echo "Using nodelist: `echo $STARNL`"

So an example job script would have:

#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=6

module load STAR-CCM+
# Get hostlist and number of processes for each host and save in a variable:
STARNL=$(srun hostname | sort -u | awk -v NT=$SLURM_NTASKS_PER_NODE '{printf $1":"NT","}' | sed s/.$//)
# Print value of the variable for diagnostics
echo "Using nodelist: `echo $STARNL`"
starccm+ <your options and settings> -np $SLURM_NTASKS -on $STARNL <simfile>

Do not user parallel i/o on ARC clusters

We have found that using the StarCCM+ option -pio which enables parallel input/output for reading simulation files and writing output causes extreme performance degredation for some jobs. This option is intended to be used in conjunction with certain parallel files systems which are not available on ARC systems and it seems that it can create severe contention. For this reason, we suggest that -pio should not be used on ARC systems.