Your Responsibilities For Running Jobs on Compute Nodes

Background and Motivation

We address here a universal problem: wasting of cluster resources. It is a problem across universities (VT is not immune to it). It is well-known and discussed among cluster administrators across universities.

Big picture view:

  1. Significant resources get wasted daily on compute clusters.

  2. We should all try to eliminate this waste.

  3. It is every user’s responsibility to do so.

  4. Every user benefits from the diligence of each user inspecting their jobs throughout the jobs’ lives.

    • This means at the job’s start and every few hours while the job is running. Not while you are asleep, of course.

  5. The effort required to inspect your jobs is quite minimal. While the instructions below may seem long, once you do them a couple of times, they will become very natural and very fast.

NOTE:

  1. The purpose of these tasks is NOT to put pressure on you to reduce your resource usage.

  2. You should use the resources that you need to complete your work.

A Sense of the Issue

To build a sense of shared responsibility, see this Waiting Jobs Dashboard that shows jobs that are queued and are awaiting resources to run. What you see will vary with time, but across all clusters, it is not uncommon to have some 1200 to 1300 jobs waiting to run on GPU nodes and 1700-2000 jobs waiting to run on CPU nodes. There is a lot of demand.

How to Use This Page

You may have been contacted by ARC about one of your (slurm) jobs. That email will have stated the problem.

These are the steps you follow.

  1. Identify the problem as one entry in one of the two tables in this section below.

  2. In this row, look at the problem and adverse effect.

  3. In the last column, there are steps listed to correct the problem.

  4. Those steps are detailed in the section below.

  5. Execute those steps.

By following these procedures, we can reduce the numbers of waiting jobs and reduce their wait times.

Common Problems Leading to Resource Wastage

We divide interactive jobs and batch jobs.

  1. Interactive jobs.

    • There are two distinct steps for an interactive job:

      • Request resources from the Slurm resource manager.

      • Use the provided resources to complete your work.

  2. Batch jobs.

    • There is one distinct step for a batch job:

      • You prepare the resource request AND the work to be done at one time and submit that information to Slurm.

    • Slurm provides resources AND executes your tasks with no additional action from you.

Because there are two steps for interactive jobs, there is greater potential for wasting resources.

We first address interactive job-specific sources of waste and then we address sources of waste common to both interactive and batch jobs.

In each table below, the issue/problem is stated and the adverse implication of the problem is given in the next column. The final column lists one or more integers that correspond to the solution approaches for that problem. These solution steps are described near the end of this page.

Problems For Interactive Jobs

Interactive jobs both from the command line and through OOD [Open OnDemand]:

Problem Number Undesirable Action or Inaction Adverse Outcome Corrective Steps
P1 Not keeping track of your interactive job requests to determine when they start, i.e., when Slurm provides you the resources that you can start using. Once resources are allocated to you, no one else can use them, even if you do not use them. If you do not use them, the resources are wasted.
  1. Use tool #1 to track your jobs and to identify the particular slurm job ID(s) from your list of jobs.
  2. Use tool #2 to cancel the job if you no longer need it.
P2 Not giving resources back when you are done with them. Once resources are allocated to you, if you do not give them back when you are done with them, then you possess them until the specified interactive job time expires, so this time difference is the duration of resource idling (wastage).
  1. Use tool #1 to identify the particular slurm job ID from your list of jobs.
  2. Use tool #2 to cancel the job; this will relinquish the resources.

NOTES:

  1. In the first problem above, we understand that resources are sometimes provided to you when you are asleep, and you can do nothing about this. (We are trying to mitigate that problem.)

  2. But you should check on your submitted interactive jobs before you go to sleep and upon waking. And periodically during the day.

  3. The last row in the table above means, for example, if you request resources for two days, and finish your work in seven hours, then if you do (2), the resources can be used by someone else immediately after seven hours. But if instead you leave the terminal and do not give back the resources, then for 48 hours only you can use the resources, meaning that 41 hours of those 48 hours are completely wasted.

Problems For Both Batch and Interactive Jobs

Problem Number Undesirable Action or Inaction Adverse Outcome Corrective Steps
P3 Requesting a GPU node and then not using the GPU; or, requesting too many GPUs. Once GPU resources are allocated to you, no one else can use them, even if you do not use them. If no one uses the resources, they are wasted. Wastage also occurs, for example, if you are using five GPUs on one job, and four GPUs are being used, but one GPU is not being used at all. In this case, the job must be killed because we cannot have idle GPUs. If your situation is unclear, please contact ARC.
  1. Use tool #1 to identify the particular slurm job ID from your list of jobs.
  2. Use tool #3 with the Slurm job ID to identify the GPU indexes of each compute node that your job is using.
  3. Use tool #4b for the cluster on which your job is running; choose the compute nodes, one a time, on which your job is running.
  4. Use tool #4c for each compute node, inspect the utilization time history plot for each GPU to ensure it is being used appropriately.
  5. Use tool #2 to cancel the job if the work is not using the GPUs correctly.
  6. In a separate line of examination, if you are running a particular code many times, it might be useful to employ tool #9 to obtain detailed GPU utilization traces.
P4 Requesting more CPUs than your job requires. Once CPU resources are allocated to you, no one else can use them, even if you do not use them. If no one uses the resources, they are wasted.
  1. Use tool #1 to identify the particular slurm job ID from your list of jobs.
  2. Use tool #3 with the Slurm job ID to identify the CPU numbers/indexes of each compute node that your job is using.
  3. Use tool #5 to ssh into each compute node, one at time.
  4. Use tool #6 (both top and htop) to examine the instantaneous utilization of CPUs that your job is using.
  5. Use tool #2 to cancel the job if the work is not using the CPUs correctly.
  6. In a separate line of examination, if you are running a particular code many times, it might be useful to employ tool #10 to get detailed traces of CPU utilization over the job life.
P5 Requesting more memory than your job requires. Once memory resources are allocated to you, no one else can use them, even if you do not use them. If no one uses the resources, they are wasted. Also, one can "starve" other jobs of CPUs and GPUs by specifying large amounts of meory. For example, say there are 96 cores on a compute node. You have a serial job, so you only use one core, but you need 90% of the compute node's memory. So what is left is 95 cores but only 20% of memory. So it could be that some/many of the cores are not used because there is insufficient remaining memory. _If you need 90% of memory, then use it._
  1. Use the same steps as for problem P4, but look at memory usage in top and htop tools instead of CPU utilization.
  2. In a separate line of examination, if you are running a particular code many times, it might be useful to employ tool #11 to get detailed memory usage over the job life.
P6 Not looking at resource utilization while your job is running to confirm that resources are well used. This is not a repeat of Problems P3 and P4 above. P3 and P4 deal with resources that are never used. Here, we deal with resources whose usage patterns change in time over the job duration. We have seen the following many times: A GPU is used for the first nine hours of a 4-day job. Then, after nine hours, the GPU is never used again. There are many other similar examples where resource usage change over a job's life.
  1. For investigating GPU-related issues, execute the corrective steps of problem P3.
  2. For investigating CPU-related issues, execute the corrective steps of problem P4.
  3. In addition to these first two items, for investigating both GPU- and CPU-related issues, tools #7 and #8 can give you instantaneous information about both CPUs and GPUs.
P7 Not looking at resource utilization when (after) your job completes. Doing so enables you to build intuition about how your code operates. In not doing so, you miss this opportunity.
  1. Use tool #12 to list CPU and CPU-memory average and peak values over the life of the job.
  2. Also, use tools #10 and #11 to collect data that can then be used to compute average and peak CPU and CPU memory use.
  3. Use tool #9 to collect data that can then be used to compute average and peak GPU and GPU memory use.

Tools and Procedures You Can Use to Solve Problems

These tools can tell you whether you are using your requested resources, and can give you insights into your code’s performance.

The number in front of each command, in the headings below, corresponds to the numbers in column 4 of the tables above, which identify the procedures and commands that can be used to address the problem.

Many of the tools are command line tools. In the text below, statements such as “at the command line” are stated. This means that you must be logged onto a login node of the cluster where your slurm jobs of interest are executing. And you must have a terminal window that you have used to log into that login node.

Tool 1. squeue Command to List Your Jobs

You can list all of your Slurm interactive and batch jobs, those running and those queued (i.e., waiting to run). Type:

squeue

Each row is a separate job. For each job, the Slurm job ID, job-name, state (e.g., queued or running), elapsed execution time (if running), and the compute node(s) on which the job is running.

Note that straight-away, this tool can tell you how long a batch job has been running. It can also tell you—for interactive jobs that you have not yet engaged with—how much time the resources have set idle.

Tool 2. scancel Command to Terminate a Job

Let SJID be a slurm job ID that you can obtain from Step 1 above, then typing

scancel SJID

at the command line will terminate the slurm job whose ID is SJID.

You can enter squeue again to verify that the slurm job SJID has been terminated; it will no longer be in the listing.

Hence, squeue and scancel are often used together.

Tool 3. scontrol Command to List Detailed Job Resources

Let SJID be the slurm job ID for the slurm job of interest. Enter the command

scontrol show job --details SJID

Many types of data will be displayed for the specified SJID. Among these are the amount of memory requested, numbers of compute nodes and CPUs, numbers of GPUs, allotted wall clock time to run your job, the compute nodes on which your job is running (if it is currently running), and myriad other data.

Note that the compute nodes are also given by squeue, again, if the job is running.

Because of --details, the specific CPU numbers on the compute nodes will be given and the index(es) for the GPU(s) specified are also given.

NOTE: This command lists CPU IDs starting from 0. If there are 128 CPUs on a compute node, then they are numbered 0 through 127. In some tools, like htop (below), CPU IDs are numbered from 1. In this case, you must make the mental conversion, e.g., CPU 63 from the scontrol show job command is node 64 within htop.

Hence, squeue and scontrol are often used together.

Also, scontrol and all of ssh, top, and htop are often used together.

Tool 4. Grafana Dashboards Showing Job Resource Usage

Grafana is accessed through your internet browser.

There are several different categories of dashboards. We highlight a few here.

Tool 4a. The VT Root Grafana Dashboard Page

The VT Grafana landing page provides links to all other dashboards.

Tool 4b. Summary Tables of All Compute Nodes Per Cluster

Each summary page is a table listing each compute node for the cluster, along with current performance/load metrics.

From the landing page above, you can select these pages directly:

  • Falcon cluster: ARC Cluster Efficiency Falcon

  • Owl cluster: ARC Cluster Efficiency Owl

  • Tinkercliffs cluster: ARC Cluster Efficiency Tinkercliffs

Or, you can go to the pages directly:

Tool 4c. Inspecting a Particular Compute Node

From step 4b, once you are on the desired page, you can hone in on the compute nodes of a particular partition, by selecting the partition at the top of the page.

Regardless, the first entry in each row is the compute node name, and clicking it shows data for that particular compute node.

On the page for a particular compute node, there are two rows of three plots each. For GPU performance, the most useful plot is often the plot in row 1, column 3.

You can see the GPU utilization for a particular GPU index (click on desired index in the plot legend). You can also specify the time window over which data are displayed by clicking the duration drop-down above the plot.

Tool 5. ssh Command to Move Between Cluster Nodes

The ssh command can be used on the command line in a terminal window on a login node.

Using the squeue command as specified above as tool #1, one can obtain the compute node(s) on which a particular slurm job is running.

A user has access to all compute nodes on which she is running jobs.

From a login node, you can ssh into a compute node that is running (part of) your job. Let cnid be the compute node ID or name. To go from the login node to the compute node, type:

ssh cnid

When this command completes, you will be on the compute node cnid and your command prompt in your terminal window will reflect this change.

At this point, you can use commands such as top and htop described in the next tool.

Once you are done working on a compute node, it is recommended to end this ssh session by typing exit, which will take you back to the login node. From there, you can ssh into another compute node.

Tool 6. top and htop Commands to Provide CPU and Memory Resource Usage

top and htop are command line tools that provide similar data and information. But there are differences. Each is designed to give information about your processes that are running on CPUs of a particular compute node.

Steps for setup:

  1. To get the compute nodes that your job is running on, see tools #1.

  2. To see the CPU numbers (and GPU indexes) for the resources that you are using on a particular compute node, see tool #3.

  3. To access a particular compute node, see tool #5.

Now you are on the desired compute node and are ready to issue the commands.

top is the more concise command

top -u $USER

or

htop -u $USER

top

top displays your running processes on the node that you are logged into. Since this is often a compute node, you can see the processes that are running your code and operating on data. Each line of the output is one process. You can see CPU utilization and memory utilization associated with each process. Make sure you identify your processes and not someone else’s, since all users who are running jobs on the particular compute node have processes running. For example, suppose a Python code, C++ code, MPI process or some other task is running. You should be able to find the name of the code in the last column. If this code is expected to be running at all times, then look at the column %CPU and determine what the instantaneous utilization is of that CPU. If it is near 100%, that is good. If it is at, say, 15% for a prolonged period, this might indicate an error. Similarly, you can see the process’ memory consumption.

htop

htop provides more detail than top.

At the top of the htop output, each CPU is listed. There is a bar to indicate analogue utilization, and a percentage is there, too. Similarly, there are memory utilization data. This is where the CPU IDs that you are using, listed by tool #3, are indispensible.

Down below, the individual processes are given. These processes are typically accompanied by the full command used to start the process, and hence provide more detail than does the top command.

For example, if a user is running VS Code, it is typically easier to identify this through htop rather than top.

Tool 7. shownodeusage Command to Show Jobs Running on a Compute Node

This tool enables you to see all of your jobs that are running on a particular compute node.

You issue this command from a login node of the cluster on which your job is running.

Steps for setup:

  1. Login to a login node of the cluster of interest.

Suppose the compute node of this cluster is cnid. In reality, cnid will have names like tc032, tc-xe006, tc-dgx003 tc-gpu002, owl006, or fal153.

Issue the command:

shownodeusage cnid

You will see instantaneous CPU and memory utilization, along with processes running. If this compute node also has GPUs and you are using them, you will see instantaneous GPU utilization. Output from the command will also include the GPU index(es) of GPUs assigned to your jobs.

Tool 8. showjobusage Command to Show Resource Usage by a Job

This tool provides you with information for compute nodes for a specified Slurm job.

You issue this command from a login node of the cluster on which your job is running.

Steps for setup:

  1. Login to a login node of the cluster of interest.

Suppose the Slurm job ID is sjid.

Issue the command:

showjobusage sjid

You will see instantaneous CPU and memory utilization, along with processes running. If your job also uses GPUs, you will see instantaneous GPU utilization. Output from the command will also include the GPU index(es) of GPUs assigned to your job. If your job uses more than one compute node, then the output of this command will span compute nodes.

Tool 9. nvidia-smi Command Inside the Sbatch Slurm Script to Obtain and Record GPU and GPU-Memory Utilization

This command can be used only on Nvidia GPUs.

The command writes to a specified output file, via redirection, characteristics/performance of all GPUs of a job.

You enter this command into the sbatch slurm script and a detailed example use is given for Tinkercliffs here.

One common form of the command is

nvidia-smi --query-gpu=timestamp,name,pci.bus_id,driver_version,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used --format=csv -l 3 > $SLURM_JOBID.gpu.log &

where the three means log data every three seconds (and should be specified with your total job duration in mind).

This command will write data to file $SLURM_JOBID.gpu.log, for post-processing.

Tool 10. mpstat Command Inside the Sbatch Slurm Script to Obtain and Record CPU Utilization

This command, like tool #9, is specified in an sbatch Slurm script. Unlike tool #9, it is not restricted to Nvidia GPU nodes or any GPU nodes. That is, CPU-only nodes (e.g., many-core nodes) can use this tool, too.

The command writes to a specified output file, via redirection, characteristics/performance of all CPUs of a job.

You enter this command into the sbatch slurm script and a detailed example use is given for Tinkercliffs here.

Some comments about the mpstat command.

  1. Note that there are two commands associated with mpstat: one to turn it on and one to stop it.

  2. The command to turn on mpstat must come before the commands to start your work.

  • For example, if you have a C++ code or Ansys invocation, you must start mpstat before invoking your codes.

  1. After your code and application invocations are finished, you must stop mpstat.

One common form of the command, to start data gathering and recording, is:

# Logging interval in seconds.
mpstat_logging_interval=4
mpstat_output_file=mpstat.dat.out
mpstat_error_file=mpstat.err
mpstat -P ALL ${mpstat_logging_interval}  >${mpstat_output_file} 2>${mpstat_error_file}  &

Notes:

  1. ALL means record data for all CPUs.

  2. The & at the end of the command so that it starts immediately and runs in the background.

The logging interval should be specified with your total job duration in mind so that the data file does not become huge.

The common form of the command, to stop data gathering, is:

kill %NUMBER

where NUMBER is the background job number.

The example link in this section uses mpstat, iostat, and vmstat for a total of three background jobs, and hence we have associated with these tasks, respectively, kill %1, kill  %2, and kill %3.

Tool 11. vmstat Command Inside the Sbatch Slurm Script to Obtain and Record CPU-Based Memory Utilization

This command, like tools #9 and #10, is specified in an sbatch Slurm script. Like tool #10, it is not restricted to Nvidia GPU nodes or any GPU nodes. That is, CPU-only nodes (e.g., many-core nodes) can use this tool, too. The memory usage recorded is that for the CPU memory, not the GPU memory. GPU memory data are acquired through tool #9.

The command writes to a specified output file, via redirection, characteristics/performance of all CPU memory of a job.

You enter this command into the sbatch slurm script and a detailed example use is given for Tinkercliffs here.

Some comments about the vmstat command.

  1. Note that there are two commands associated with vmstat: one to turn it on and one to stop it.

  2. The command to turn on vmstat must come before the commands to start your work.

  • For example, if you have a C++ code or Ansys invocation, you must start vmstat before invoking your codes.

  1. After your code and application invocations are finished, you must stop vmstat.

One common form of the command, to start data gathering and recording, is:

# Logging interval in seconds.
vmstat_logging_interval=4
vmstat_output_file=mpstat.dat.out
vmstat_error_file=mpstat.err
vmstat  ${vmstat_logging_interval}  >${vmstat_output_file} 2>${vmstat_error_file}  &

Notes:

  1. The & at the end of the command so that it starts immediately and runs in the background.

The logging interval should be specified with your total job duration in mind so that the data file does not become huge.

The common form of the command, to stop data gathering, is:

kill %NUMBER

where NUMBER is the background job number.

The example link in this section uses mpstat, iostat, and vmstat for a total of three background jobs, and hence we have associated with these tasks, respectively, kill %1, kill  %2, and kill %3.

Tool 12. seff Command to Give Average CPU and Memory Resource Usage

Usage statistics are available for CPU utilization and CPU-based memory use over an entire job.

Given a slurm job ID of SJID, type

seff SJID

to see CPU and CPU-based memory usage aggregated over your job.

This enables you to build up intuition about the resources that your jobs need.

We are not saying that you should cut resources down to the bone, but if, say, you use 20 CPUs and from seff your CPU utilization is 20%, then you should consider whether you can decrease the number of CPUs for the job and increase the utilization of those.

There is no analogous command for GPUs.