Overview

OptExp is a library to manage experiments in optimization for ML. Defining and running experiments can be tedious and error-prone, and Optexp aims to make those tasks easier and reduce wasted time due to common errors. The library provides utilities to

Define experiments, directly in Python code
Prepare the necessary components, such as downloading the datasets
Run experiments
- locally or on a slurm cluster
- log the results to wandb
- with support for multiple GPUs
Check that all experiments have succeeded and logged their data
Download the data locally to make plots

Defining experiments

Optexp defines experiments directly in Python code. For example, to run a grid search for the step-size of SGD:

from optexp import Experiment, Problem, cli
from optexp.datasets import MNIST
from optexp.metrics import Accuracy, CrossEntropy
from optexp.models import LeNet5
from optexp.optim import SGD

experiments = [
    Experiment(
        problem=Problem(
            dataset=MNIST(),
            model=LeNet5(),
            batch_size=100,
            lossfunc=CrossEntropy(),
            metrics=[  # Monitor both the accuracy and the loss on the training and validation sets
                Accuracy(),
                CrossEntropy(),
            ],
        ),
        group="first_gridsearch",
        optim=SGD(lr=lr),
        steps=10,  # 2 epochs
        eval_every=5,  # Evaluate every epoch
    )
    for lr in [0.1, 0.01, 0.001]
]

if __name__ == "__main__":
    cli(experiments)

Running experiments

Optexp provides a command line interface to manage experiments.

Prepare experiments and make sure the datasets are downloaded
```
python experiments.py prepare
```
Run experiments either locally or on a slurm cluster. The results will be logged to wandb.
```
python experiments.py run --local
python experiments.py run --slurm
```
Check that all experiments have succeeded and logged their data.
```
python experiments.py check
```
Download the data locally to make plots
```
python experiments.py download
```

Load the results

All the experiments have run and it’s time to make plots?

from experiments import experiments

exp_data = [load_data(exp) for exp in experiments]

Filter experiments directly in Python

filtered_data = [
    load_data(exp) for exp in experiments
    if exp.optimizer.lr == 0.01
]