Managing Virtual Environments Using pip and venv on ARC

This page describes how to build, organize, and use Python virtual environments (VEs) with pip and venv on ARC systems. This method is lightweight and does not rely on Conda or Miniforge. ARC suggests the use of Miniforge as the preferred way to build conda virtual environments (CVEs). We recommend not to install andaconda on your home as suggested on traditional tutorials. ARC’s provided Miniforge and Miniconda will work faster than user-installed anaconda in $HOME.

1. When to Use pip/venv

Use pip and venv if:

  • You only need a few Python packages.

  • You prefer the built-in Python tools.

  • You do not need Conda’s dependency management or GPU toolkit integrations.

This method creates a fully isolated Python environment using only standard library tools.

2. Steps to Create a pip/venv Virtual Environment

Step 1 — Start an Interactive Session

Always create environments on the same node type where they’ll be used.

interact --account=<account> --partition=<partition_name>

Step 2 — Load a Python Module

Identify available Python versions, then load one.

module spider Python/3
module load Python/3.12.3-GCCcore-13.3.0

Step 3 — Create the Virtual Environment

python -m venv ~/venvs/<env_name>

Step 4 — Activate the Environment

source ~/venvs/<env_name>/bin/activate

Step 5 — Install Packages

Install or upgrade packages using pip.

pip install numpy pandas
pip install --upgrade pip

Step 6 — Check Installed Packages

pip list
python --version

Step 7 — Deactivate the Environment

deactivate

3. Organizing and Managing Virtual Environments

Path

Description

~/venvs/<name>

Default per-user environment directory

/projects/<group>/envs/<name>

Shared environment directory for group use

Example structure:

~/venvs/
 ├── tc_normal-ml/
 ├── tc_a100-nlp/
 └── bioinfo-tools/

List environments:

ls ~/venvs

4. Using pip/venv Environments in Jupyter OnDemand

You can use a pip/venv environment as a Jupyter kernel to run notebooks on Open OnDemand.

Step 1 — Install ipykernel

Activate your environment, then install the ipykernel package.

source ~/venvs/<env_name>/bin/activate
pip install ipykernel

Step 2 — Register the Kernel

python -m ipykernel install --user --name <env_name> --display-name "Python (<env_name>)"

Step 3 — Select the Kernel in Jupyter

  1. Launch Jupyter via Open OnDemand.

  2. From the top menu, select Kernel → Change Kernel → Python (<env_name>).

5. Using pip/venv Environments in Slurm Jobs

pip/venv environments can be used in Slurm batch jobs as long as the environment was created on the same node type and the same Python module is loaded.

Example Slurm batch script:

#!/bin/bash
#SBATCH --job-name=my_pip_job
#SBATCH --partition=<partition>
#SBATCH --nodes=1
#SBATCH --time=01:00:00
#SBATCH --account=<account>

module load Python/3.12.3-GCCcore-13.3.0
source ~/venvs/<env_name>/bin/activate

python my_script.py

Important notes:

  • Do not create virtual environments inside batch jobs.

  • Always load the same Python module used during environment creation.

  • The environment path must match the one created interactively.

6. Summary

  • Use pip and venv for simple, lightweight Python environments.

  • Always create environments on the correct node type.

  • Register environments using ipykernel for Jupyter OnDemand.

  • For complex dependency chains or GPU use, prefer Miniforge/Miniconda instead.

7. Troubleshooting

Issue

Solution

Module not found

Verify the package is installed in the active environment.

Kernel not showing in Jupyter

Re-run the ipykernel install command.

Permission denied

Avoid using shared directories unless permissions are set.

Python version mismatch

Load the same Python module that was used to create the environment.

8. Common Questions

When should I prefer pip/venv over Conda?

Use pip/venv for lightweight, pure-Python, CPU-only workflows where fast setup is preferred.

Where should I store pip/venv environments?

Recommended locations:

  • ~/venvs/<env_name>

  • /projects/<group>/envs/<env_name>

Can I use pip/venv environments in Slurm jobs?

Yes. Create the environment interactively on the target node type and activate it in Slurm jobs using the same Python module.

Can pip/venv environments be used in Jupyter?

Yes. Install and register an IPython kernel from within the environment.