Video Tutorials

ARC provides a number of video tutorials on our channel on video.vt.edu. In particular, the following sequence walks a user through the fundamentals of ARC usage in less than an hour:

Accessing Software

The following videos will walk the user through accessing software that ARC has installed or through setting up your own packages:

Note that these videos require a VT Login to access. Also, each video has a table of contents that can be used to skip between sections; this can be accessed by clicking the “hamburger” (three horizontal bars) button at the top left of the video.

We are actively updating this page.

There are many types of videos linked on this page. Generally, they fall into two classes:

General overviews.
Narrow focus on how to accomplish one thing.

With respect to bullet #2, sometimes, one video suffices for a topic. In other cases, multiple videos are required. An example of the latter are virtual environments (VEs). There are videos on why they are needed, how to use modules (in setting up general environments), how to build and use VEs of various types, and how to use VEs across programming languages. Rather than throw all of these topics into one (longer) video, we carve the issues up into several videos so one can more directly satisfy their needs. You might, for example, only want to know how to build a VE and use it in a Jupyter notebook.

We try to make very few assumptions about the experience of the user so that videos are of value to novices and those with experience. All users of all types are welcome on ARC resources and are encouraged to use them.

General Overview

These videos provides a general overview of use of ARC systems.

Overview of ARC systems.
- Overview of ARC systems and jobs
How to get help with and how to learn about ARC resources.
- Help and learn

Connecting to ARC Clusters

There are multiple ways to connect to ARC clusters.

How to access ARC clusters using a terminal window on your laptop and ssh (and streamlining access for subsequent logins). This approaches uses the command line for working on clusters.
- Login with SSH
- Using SSH Keys and Agent to simplify logins
How to connect to ARC clusters using VS Code. This approach uses an IDE for working on clusters.
- Connect to ARC Systems with VS Code
How to access ARC clusters using Open OnDemand (OOD). This approach uses an assortment of ways for working on clusters (e.g., command line, UIs).
- Open OnDemand

How to Run Codes—Your Own or Commercial/Open Software

Whether you run your own code, e.g., built using some programming language, or a commercial/open source code such as Ansys, you will be interacting with the scheduler. Our scheduler on ARC clusters is the Slurm Scheduler. The first video is an excellent and concise introduction for running interactive and batch jobs. The subsequent videos are motivated by cluster configurations that increase throughput of jobs on the clusters and signficantly reduce wait time. To specify more precisely job resources, concepts such as constraints are needed that are not covered in the first video.

How to run both interactive and batch jobs.
- Interactive and batch jobs
Learn the different types of compute nodes on clusters and how to specify them for your jobs. Watch this before the later videos of this section.
- Compute node partitions on clusters
A more detaile example of an interactive job that uses the material in the second video—that is, how to use constraints to specify compute node types.
- TODO: Interactive jobs
A more detaile example of a batch job that uses the material in the second video—that is, how to use constraints to specify compute node types.
- Batch jobs with constraints

Self-Monitoring Your Activities and Understanding How Your Code Is Performing

ARC clusters are communal resources. They work best when everyone knows the purposes for different types of computer nodes (i.e., computers) on a cluster. These videos describe appropriate (and inappropriate) uses, and just as importantly, how to monitor your own use of the computing resources. And monitoring your (computational) jobs can give you insights about your codes and their performance.

Appropriate use of login (head) nodes of ARC clusters.
- Appropriate uses of login and compute nodes
How you can self-monitor your use of login (head) nodes.
- How to self-monitor login nodes
How you can self-monitor your Slurm jobs and understand your code’s performance.
- How to self-monitor compute nodes and your Slurm jobs

File Compression and Archiving

File compression and file archiving are two separate activities. However, they can also be used in combination. These videos describe how to compress and archive files.

How to compress files.
- Using file compression tools
How to create an archive (a single file) that contains many files, and how to compress this resulting file.
- Creating a single file (archive) containing many other files

File Transfer: Transferring files onto and off of clusters

One can use these tools to transfer files between and within machines/computers; here, our focus is ARC clusters.

Overview of utilities/tools for copying files onto and off of ARC clusters.
- Overview of file copying/movement tools
How to use scp (secure copy protocol) for copying files onto and off of ARC clusters.
- Using scp to transfer files
How to use sftp (secure file transfer protocol) for copying files onto and off of ARC clusters.
- Using sftp to transfer files
How to use rsync for copying files onto and off of ARC clusters.
- Using rsync to transfer files
How to use Globus for copying files onto and off of ARC clusters.
- TODO: Using Globus to transfer files

Different Drives on ARC Clusters

This video describes the various mounts/directories for doing different types of work. For example, these drives vary in the desired longevity of files contained in them.

Drives for Use on Clusters

Environments

Environments are collections of software that tailor an otherwise “basic” computing ecosystem into one that supports your particular computing needs for a particular type of task. You may need different environments for your different tasks. There is a series of videos here on

motivation,
how to use modules,
how to structure directories to house your virtual environments,
how to construct and use virtual environments (VEs) for
- command-line execution of code and
- use with applications like Jupyter notebooks.

Motivation: why we need environments.
- Motivation for environments
Modules–a backbone of customizing your environments.
- How to use modules
Ways to think about structuring the locations of virtual environments to organize the (cluster, compute node type) for which they are used.
- Organizing your virtual environments
How to create and use Conda virtual environments on Owl (and other) clusters.
- Create a Conda VE on Owl for Python
- Run a python code using a Conda VE on Owl
How to create and use a Python pip-venv virtual environment (VE) on Owl (and other clusters).
- Create a Pip-Venv VE on Owl for Python
- Run a python code using a Pip-Venv VE on Owl
How to create and use Python Conda virtual environments with Jupyter notebooks (through OOD [Open OnDemand]).
- Create a Conda VE on Owl for Jupyter notebooks and Python
- Run a Jupyter notebook with a Conda VE on Owl
How to create and use Julia virtual environments on clusters.

Running Jobs on ARC Clusters

The two dominant ways to run jobs on ARC clusters are:

interactive jobs
batch jobs.

Interactive Jobs

This video provides an example of an interactive job.

Batch Jobs

This video provides an example of a batch job.