Quick Install

This tutorial will get you started with a minimal setup of Optexp.

Down the line, see the installation and configuration guide to customize the installation for your system and integrate with Slurm clusters.

Prerequisites

  1. Python 3.10

    Make sure you have an installation of Python 3.10 or later. Using venv or conda is recommended.

  2. Pytorch 2.3.0

    The Pytorch installation is os and hardware dependent, follow Pytorch Installation.

Installation

Installing from source is recommended as the library is not yet stable.

  1. Clone the repository

  2. Install the dependencies

  3. Install the library in editable mode

git clone https://github.com/fKunstner/optexp optexp
cd optexp
pip install -r requirements/main.txt
pip install -e .

Once installed, use git pull to update to the latest version.

Configuration

Optexp needs to know a few things about your system to run experiments.

  1. Where to store dataset files and logs

  2. Where to upload the results of the experiments

  3. If using a Slurm cluster, the configuration for the cluster

Those are configured through environment variables and best managed with an environment file. Here is a template you can copy and fill in.

env.sh
 1# Basic configuration
 2export OPTEXP_WORKSPACE=~/workspace
 3
 4# Wandb configuration
 5export OPTEXP_WANDB_ENABLED=true
 6export OPTEXP_WANDB_AUTOSYNC=true
 7export OPTEXP_WANDB_MODE=offline
 8export OPTEXP_WANDB_PROJECT=$your_wandb_project
 9export OPTEXP_WANDB_ENTITY=$your_wandb_entity
10export WANDB_API_KEY=$your_api_key
11
12# Slurm configuration (optional)
13# export OPTEXP_SLURM_NOTIFICATION_EMAIL=$your_mail@example.com
14# export OPTEXP_SLURM_ACCOUNT=$your_account_name

Once complete, you can load the environment variables with source env.sh (Unix/OSX) or call env.bat (Windows).

The environment variables are only loaded for the current terminal session and you will need to load the environment variables again with source env.sh or call env.bat on your next session. To avoid the hassle, add source /path/to/env.sh to your .bashrc (Unix) or .bash_profile (OSX).

Optexp configuration

OPTEXP_WORKSPACE is the directory where the logs and datasets will be stored. It will fill up, so avoid putting it in a directory with limited space (eg a university home directory with 5mb of space). For testing purposes, something like ~/workspace should work fine.

Wandb configuration

Optexp uses Weights and Biases (wandb) to log the results of the experiments. If you do not have a wandb account, create a one at wandb.ai.

To log to your account, create a project at https://wandb.ai/$your_username/projects and set

  • OPTEXP_WANDB_ENTITY to your username

  • OPTEXP_WANDB_PROJECT to your project.

The logs will appear at https://wandb.ai/$your_username/$your_project

Next steps

The library is installed and configured! You can already try to run the tutorials.

The Slurm configuration is not necessary for local runs.

Slurm configuration

To run experiments using a Slurm cluster, Optexp and its requirements need to be installed on the login node with the following tweaks:

  • The Pytorch installation on your cluster is likely different, check your cluster’s documentation.

  • OPTEXP_WORKSPACE needs to be accessible from the login node and the compute nodes.

  • OPTEXP_SLURM_ACCOUNT is the account to charge the jobs to (--account in sbatch).

  • OPTEXP_SLURM_NOTIFICATION_EMAIL is the email to receive notifications (--mail-user in sbatch). Leave it undefined to not receive emails.