Virtual Environments - Conda and Pip/Venv

ARC supports two primary methods for creating isolated Python environments:

• Conda environments using Miniforge or Miniconda

• Python virtual environments using pip and venv

Each method has strengths depending on workload type, complexity, and GPU requirements.

Contents:

• Managing Conda Environments on ARC
  • 1. Why Miniforge or Miniconda (and Not Anaconda)
  • 2. Create a conda environment specifically for the type of node where it will be used
  • 3. GPU and CUDA Compatibility
  • 4. Use a Jupyter kernel to load a conda environment in a Jupyter application
  • 5. Using Conda Environments Inside Slurm Batch Scripts
  • 6. Compatibility Note: Miniforge and Miniconda
  • 7. Summary
  • 8. Troubleshooting
  • 9. Common Questions
• Managing Virtual Environments Using pip and venv on ARC
  • 1. When to Use pip/venv
  • 2. Steps to Create a pip/venv Virtual Environment
  • 3. Organizing and Managing Virtual Environments
  • 4. Using pip/venv Environments in Jupyter OnDemand
  • 5. Using pip/venv Environments in Slurm Jobs
  • 6. Summary
  • 7. Troubleshooting
  • 8. Common Questions

ARC Specific Notes

Store user-managed environments in visible directories for easier, intentional management:

• Conda: ~/envs/<name> (or /projects/<group>/envs/<name> for shared environments)

• pip/venv: ~/venvs/<name> (or /projects/<group>/envs/<name> for shared environments)

Avoid storing user environments under ~/.conda/, which Conda uses for internal caches and metadata.

Environments are not portable between node types or partitions.

Always build your Conda or pip/venv environment on the same node type where it will be used.

Common Questions

1. Do I need a virtual environment for every project?

   Yes. ARC strongly recommends using virtual environments to avoid package conflicts and ensure reproducibility.

2. Can I use the same environment in Slurm and Jupyter?

   Yes. Once created on the correct node type, environments can be used in Slurm interactive jobs, Slurm batch jobs, and Jupyter (via IPython kernels).

3. Are environments portable across partitions?

   No. Environments must be created on the same node type where they will be used.

4. Should I use a Conda environment or pip/venv?

   Use Conda for machine learning, GPU workflows, scientific computing, large research projects, and anything requiring compiled dependencies or CUDA.

   Use pip/venv for small tools, scripts, lightweight projects, teaching workflows, or CPU-only tasks that involve few dependencies.