Storage Resources
Overview
ARC offers several different storage options for users’ data:
Persistent Storage
Name |
Intent |
File System |
Environment Variable |
Per User Maximum |
Data Lifespan |
Available On |
---|---|---|---|---|---|---|
Long-term storage of files |
Qumulo |
$HOME |
640 GB 1 million files |
As long as the user account is active |
Login and Compute Nodes |
|
Long-term storage of shared, group files |
GPFS |
- n/a - |
50 TB, ~10 million files per faculty researcher (Expandable) |
As long as the project account is active |
Login and Compute Nodes |
|
Long-term storage for infrequently-accessed files |
IBM Cloud Object Storage |
- n/a - |
Length of the purchase agreement |
Managed by ARC staff |
Scratch (temporary) storage
Name |
Intent |
Per User Maximum |
Data Lifespan |
File System |
Environment Variable |
Available On |
|
---|---|---|---|---|---|---|---|
Short-term access to working files. Automatic deletion. |
No size limits enforced |
90 days |
Vast |
- n/a - |
Login and compute nodes |
||
Fast, temporary storage. Auto-deleted when job ends |
Size of node hard drive |
Length of Job |
Local disk hard drives, usually spinning disk or SSD |
$TMPDIR |
Compute Nodes |
||
Fast, temporary storage. Auto-deleted when job ends |
Size of node hard drive |
Length of Job |
Local disk hard drives, usually spinning disk or SSD |
$TMPDIR |
Compute Nodes |
||
Very fast I/O |
Size of node memory allocated to job |
Length of Job |
Memory (RAM) |
$TMPFS |
Compute Nodes |
Home
Home provides long-term storage for system-specific data or files, such as installed programs or compiled executables. Home can be reached the variable $HOME
, so if a user wishes to navigate to their Home directory, they can simply type cd $HOME
. Each user is provided a maximum of 640 GB in their Home directories (across all systems). Home directories are not allowed to exceed this limit. Note that running jobs fail if they try to write to a Home directory when the hard limit is reached.
Note
Avoid reading/writing data to/from HOME in a job or using it as a working directory. Stage files into a “scratch” location to keep unnecessary I/O off of the HOME filesystem and improve performance, use /scratch and Local Scratch
Project
Project provide long-term storage for files shared among a research project or group, facilitating collaboration and data exchange within the group. Each Virginia Tech faculty member can request group storage up to the prescribed limit at no cost by requesting a storage allocation via ColdFront. Additional storage may be purchased through the investment computing or cost center programs.
Best Practices for long-term storage
ARC policy limits the number of files and the total size of data stored in these filesystems which are designed for long-term storage. In order to avoid violating these policies, users should tar and compress data stored in these filesystems.
Tip
To “tar” means to package into single file, thereby reducing inodes. We “extract” data from a tar file. To “compress” is to reduce byte size. We “decompress/unzip” data from a compressed/zipped file.
Consider the fundamental tradeoff between compression and time. The best protocol depends heavily on the research workflow. Some questions to ask yourself:
How compressible is the data?
Because compression algorithms take advantage of redundancy to reduce size, some data are more compressible than others. Additionally there are many options for compressing that affect the runtime and the size reduction.
Take the following example:
uncompressed plaintext
gzip
gzip -2
gzip -9
xz
xz -2
xz -9
xz -T 24
xz -T 24 -9
Size
954M
309M
347M
308M
220M
266M
204M
223M
206M
Time
N/A
0m44.706s
0m20.155s
1m0.805s
7m59.689s
1m35.004s
11m39.037s
0m34.547s
2m40.020s
Here we are testing different tools, gzip and xz, with different options. We are varying the “compression value”, where a higher value means more compression (and more time). We also see that parallelizing reduces the time cost.
[slima2@tc049 text_data]$ time bash -c "xz -T 24 enwik9.tar" real 0m34.547s user 10m21.351s sys 0m1.509s
While these operations are acceptable on a login node, a compute node may be desirable, especially if parallelizing across more than the limits on the login node.
Every case should appreciate the gains in size over time cost. See the figure below for differences between a selection of example datasets.
How often do you plan to extract/transfer the data?
If your workflow involves copying to /localscratch for better IO performance, it may be worthwhile to compress the data.
Some workflows use software that work directly on compressed files. This is especially the case in many bioinformatics software such as cellranger and bedtools with gzipped files in particular. Luckily, the sequencing core probably prepared your data correctly before they sent it to you.
Archive
Archive provides users with long-term storage for data that does not need to be frequently accessed i.e. storing important/historical results and for data preservation purposes to comply with mandates of federal grants (data retention policies). Archive is not mounted on the clusters. Archive is accessible only through ARC staff. Researchers can compress their datasets on the clusters and ARC staff will transfer to the archive.
Archive storage may be purchased through the investment computing or cost center programs. Please reach out to us to acquire archive storage.
Scratch Filesystems
VAST - Scratch
Note
Files and directories stored here are subject to automatic deletion. Do not use it for long term storage.
Local scratch storage options (see below) generally provide the best performance, but are constrained to the duration of a job and are strictly local to the compute node(s) allocated to a job. In constrast, we also have a VAST storage system which provides storage for temporary staging and working space with better performance characteristics than HOME or PROJECTS. It is “global” in the sense that it is accessible from any node on the cluster.
It is a shared resource and has limited capacity, but individual use at any point in time is unlimited provided it the space is available. A strict automatical deletion policy is in place wherein any file will be automatically deleted when it has reached an age of 90 days on /scratch
.
Best practices
Create a directory for yourself
mkdir /scratch/<username>
Stage files for a job or set of jobs
Keep the number of files and directories relatively small (ie. less than 10,000). It is a network-attached filesystem and incurs the same performance overhead for file operations that you would get with
/home
or/projects
.Immediately copy any files you want to keep to a permanent location to avoid accidental deletion
Always remember the 90-day automatic deletion policy
Automatic Deletion Details
As mentioned above, files and directories in /scratch
will be automatically deleted based on aging policies. Here is how that works:
The storage system runs an hourly job to identify files which have exceeding the aging policy (90 days) and adds these to the deletion queue.
The storage system runs an automated job at 12:00am UTC (7:00PM EST) every day to process the deletion queue.
Additionally, the storage system will detect and delete all empty directories regardless of age.
Restoring files
In some situations, deleted files and directories may be restored from “snapshots”. Snapshots are an efficient way to keep several instances of the status of a file system at regular points in time.
For the /scratch
file system, these are kept in the “hidden directory” /scratch/.snapshots
which contains a set of snapshots named according to the type (daily, weekly, or monthly) and the date-time when they were recorded. For example:
/scratch/.snapshot/week_2023-11-13_12_00_00_UTC
is an instance of a weekly snapshot which was recorded on 2023-11-13 at 12:00:00PM UTC.
Snapshots may be recorded in daily, weekly, and monthly cycles, but ARC reserves the right to adjust the frequencies and quantities of snapshots which are retained. Changes in the frequencies and quantities may occasionally be needed to adjust how much of the storage system capacity is dedicated to snapshot retentions.
Note
While snapshots provide some level of protection against data loss, they should not be viewed as a “backup” or as part of a data retention plan.
Local Scratch
Running jobs are given a workspace on the local drives on each compute node which are allocated to the job. The path to this space is specified in the $TMPDIR
environment variable. This provides a higher performing option for I/O which is a bottleneck for some tasks that involve either handling a large volume of data or a large number of file operations.
Note
Any files in local scratch are removed at the end of a job, so any results or files to be kept after the job ends must be copied to another location as part of the job. /scratch is a good choice for most people.
Local Drives
Running jobs are given a workspace on the local drives on each compute node. The path to this space is specified in the $TMPDIR
environment variable.
Solid State Drives (SSDs)
Solid state drives do not use rotational media (spinning disks/platters) but memory-like flash storage which gives it better performance characteristics. The environment variable $TMPSSD
is set to a directory on an SSD accessible to the owner of a job when SSD is available on compute nodes allocated to a job.
Memory as storage
Running jobs have access to an in-memory mount on compute nodes via the $TMPFS
environment variable. This should provide very fast read/write speeds for jobs doing I/O to files that fit in memory. Please note that these files are removed at the end of a job, so any results or files to be kept after the job ends must be copied to Work or Home.
NVMe Drives
Same idea as Local Scratch, but on NVMe media which “has been designed to capitalize on the low latency and internal parallelism of solid-state storage devices.” Running jobs are given a workspace on the local NVMe drive on each compute node if it is so equipped. The path to this space is specified in the $TMPNVME
environment variable. This provides another option for users who would prefer to do I/O to local disk (such as for some kinds of big data tasks). Please note that any files in local scratch are automatically removed at the end of a job, so any results or files to be kept after the job ends must be copied to Home or Project.
Checking Usage
You can check your current storage usage (in addition to your compute allocation usage) with the quota
command:
[mypid@tinkercliffs2 ~]$ quota
USER FILESYS/SET DATA (GiB) QUOTA (GiB) FILES QUOTA NOTE
mypid /home 584.2 596 - -
GPFS
mypid /projects/myproject1 109.3 931
mypid /projects/myproject2 2648.4 25600