Video Tutorials
ARC provides a number of video tutorials on our channel on video.vt.edu. In particular, the following sequence walks a user through the fundamentals of ARC usage in less than an hour:
Login
How to access ARC clusters for the first time (and streamlining access for subsequent logins):
Accessing Software
The following videos will walk the user through accessing software that ARC has installed or through setting up your own packages:
Note that these videos require a VT Login to access. Also, each video has a table of contents that can be used to skip between sections; this can be accessed by clicking the “hamburger” (three horizontal bars) button at the top left of the video.
We are actively updating this page.
There are many types of videos linked on this page. Generally, they fall into two classes:
General overviews.
Narrow focus on how to accomplish one thing.
With respect to bullet #2, sometimes, one video suffices for a topic. In other cases, multiple videos are required. An example of the latter are virtual environments (VEs). There are videos on why they are needed, how to use modules (in setting up general environments), how to build and use VEs of various types, and how to use VEs across programming languages. Rather than throw all of these topics into one (longer) video, we carve the issues up into several videos so one can more directly satisfy their needs. You might, for example, only want to know how to build a VE and use it in a Jupyter notebook.
We try to make very few assumptions about the experience of the user so that videos are of value to novices and those with experience. All users of all types are welcome on ARC resources and are encouraged to use them.
General Overview
These videos provides a general overview of use of ARC systems.
Overview of ARC systems.
How to get help with and how to learn about ARC resources.
Connecting to ARC Clusters
There are multiple ways to connect to ARC clusters.
How to access ARC clusters using a terminal window on your laptop and ssh (and streamlining access for subsequent logins). This approaches uses the command line for working on clusters.
How to connect to ARC clusters using VS Code. This approach uses an IDE for working on clusters.
How to access ARC clusters using Open OnDemand (OOD). This approach uses an assortment of ways for working on clusters (e.g., command line, UIs).
How to Run Codes—Your Own or Commercial/Open Software
Whether you run your own code, e.g., built using some programming language, or a commercial/open source code such as Ansys, you will be interacting with the scheduler. Our scheduler on ARC clusters is the Slurm Scheduler. The first video is an excellent and concise introduction for running interactive and batch jobs. The subsequent videos are motivated by cluster configurations that increase throughput of jobs on the clusters and signficantly reduce wait time. To specify more precisely job resources, concepts such as constraints are needed that are not covered in the first video.
How to run both interactive and batch jobs.
Learn the different types of compute nodes on clusters and how to specify them for your jobs. Watch this before the later videos of this section.
A more detaile example of an interactive job that uses the material in the second video—that is, how to use constraints to specify compute node types.
A more detaile example of a batch job that uses the material in the second video—that is, how to use constraints to specify compute node types.
Self-Monitoring Your Activities and Understanding How Your Code Is Performing
ARC clusters are communal resources. They work best when everyone knows the purposes for different types of computer nodes (i.e., computers) on a cluster. These videos describe appropriate (and inappropriate) uses, and just as importantly, how to monitor your own use of the computing resources. And monitoring your (computational) jobs can give you insights about your codes and their performance.
Appropriate use of login (head) nodes of ARC clusters.
How you can self-monitor your use of login (head) nodes.
How you can self-monitor your Slurm jobs and understand your code’s performance.
File Compression and Archiving
File compression and file archiving are two separate activities. However, they can also be used in combination. These videos describe how to compress and archive files.
How to compress files.
How to create an archive (a single file) that contains many files, and how to compress this resulting file.
File Transfer: Transferring files onto and off of clusters
One can use these tools to transfer files between and within machines/computers; here, our focus is ARC clusters.
Overview of utilities/tools for copying files onto and off of ARC clusters.
How to use scp (secure copy protocol) for copying files onto and off of ARC clusters.
How to use sftp (secure file transfer protocol) for copying files onto and off of ARC clusters.
How to use rsync for copying files onto and off of ARC clusters.
How to use Globus for copying files onto and off of ARC clusters.
Different Drives on ARC Clusters
This video describes the various mounts/directories for doing different types of work. For example, these drives vary in the desired longevity of files contained in them.
Environments
Environments are collections of software that tailor an otherwise “basic” computing ecosystem into one that supports your particular computing needs for a particular type of task. You may need different environments for your different tasks. There is a series of videos here on
motivation,
how to use modules,
how to structure directories to house your virtual environments,
how to construct and use virtual environments (VEs) for
command-line execution of code and
use with applications like Jupyter notebooks.
Motivation: why we need environments.
Modules–a backbone of customizing your environments.
Ways to think about structuring the locations of virtual environments to organize the (cluster, compute node type) for which they are used.
How to create and use Conda virtual environments on Owl (and other) clusters.
How to create and use a Python pip-venv virtual environment (VE) on Owl (and other clusters).
How to create and use Python Conda virtual environments with Jupyter notebooks (through OOD [Open OnDemand]).
How to create and use Julia virtual environments on clusters.
Running Jobs on ARC Clusters
The two dominant ways to run jobs on ARC clusters are:
interactive jobs
batch jobs.
Interactive Jobs
This video provides an example of an interactive job.
Batch Jobs
This video provides an example of a batch job.