Experiment tracking

After a fortnight of experiments you will have a result you cannot reproduce and a model file you cannot account for. This section is about the record that prevents it: what to log for every run so it can be traced back, how to compare runs meaningfully, where the resulting artifacts live, and why exact determinism is harder than it sounds and usually not what you actually need.

The problem#

Three weeks ago a run scored well. Which configuration was it? Which data? Without tracking, that result is gone, and everyone learns this the hard way exactly once.

Why it is in the notebook does not count#

The usual answer to where a result lives is that it is in the notebook. That fails for four separate reasons, and it is worth having all four to hand because people will argue the first three individually.

• A notebook records the last state of each cell, not the sequence that produced the number. Cells run out of order leave a printed 0.93 under code that would now print something else, and nothing in the file says which.
• It records the output and not the inputs. The cell that read the data three cells up has since been edited; the number below it has not.
• It is one run. The comparison you actually need is across twenty, and twenty notebooks with the numbers in different cells is not a table.
• It is not searchable by result. Six months later the question is which run got above 0.88, and the only way to answer it from notebooks is to open all of them.

The fix is not to stop using notebooks. It is that the run's record has to be written by the code, to a place outside the notebook, every time, without anyone deciding this run is worth keeping. Runs are only interesting in retrospect, and the ones you did not expect to matter are exactly the ones you will want back.

What a tracker actually is#

MLflow and Weights and Biases look like infrastructure, and that impression makes people put off adopting one and misjudge what they are getting. The whole of it is a directory per run, a JSON file of what the run was, and an append-only file of what happened during it. Here it is, in about forty lines, so the tools stop being magic.

import hashlib, json, platform, subprocess, sys, time, uuid
from pathlib import Path

RUNS = Path("runs")

def git_sha():
    try:
        sha = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True,
                                      stderr=subprocess.DEVNULL).strip()
        dirty = subprocess.check_output(["git", "status", "--porcelain"], text=True,
                                        stderr=subprocess.DEVNULL).strip()
        return sha + ("-dirty" if dirty else "")
    except Exception:
        return "no-git"

def file_hash(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()[:12]

def freeze():
    out = subprocess.check_output([sys.executable, "-m", "pip", "freeze"], text=True)
    return sorted(l for l in out.splitlines() if l.split("==")[0].lower() in
                  {"numpy", "pandas", "scikit-learn", "torch", "scipy"})

class Run:
    """One directory per run: meta.json for what it was, metrics.jsonl for what happened."""

    def __init__(self, name, config, data_path=None):
        self.id = time.strftime("%Y%m%d-%H%M%S") + "-" + uuid.uuid4().hex[:6]
        self.dir = RUNS / self.id
        self.dir.mkdir(parents=True)
        self.meta = {
            "id": self.id, "name": name, "config": config,
            "git_sha": git_sha(),
            "data": {"path": str(data_path), "sha256_12": file_hash(data_path)}
                    if data_path else None,
            "python": sys.version.split()[0], "platform": platform.platform(),
            "packages": freeze(), "started": time.time(),
        }
        (self.dir / "meta.json").write_text(json.dumps(self.meta, indent=2))
        self.metrics = open(self.dir / "metrics.jsonl", "a")

    def log(self, step, **values):
        self.metrics.write(json.dumps({"step": step, **values}) + "\n")
        self.metrics.flush()

    def finish(self, **summary):
        self.metrics.close()
        self.meta["summary"] = summary
        self.meta["finished"] = time.time()
        (self.dir / "meta.json").write_text(json.dumps(self.meta, indent=2))

Five fields in that class are the ones that make a run reproducible six months later, and every one of them is a thing people omit. The git sha, with the dirty flag, because a commit hash with uncommitted changes on top of it is a lie. The content hash of the data, because a path is a name and names get reused. The full config, including the defaults nobody changed, because the default will change in a library upgrade. The interpreter and package versions, because a result that only reproduces on the old numpy is still worth knowing about. The seed, inside the config, so that the run-to-run variation can be separated from the effect you are measuring.

What the real tools add on top of this is a server so runs from several machines land in one place, a user interface for plotting, artifact storage that is not your laptop, and a registry that connects a run to the model serving in production. None of that changes what a run is. If you understand the forty lines you can pick a tool in an afternoon and you will not be confused by any of them.

Comparing runs, and what a sweep is for#

The record only pays for itself when it is read across runs. Twelve runs through the tracker above, four values of a regularisation strength crossed with three seeds, loading every meta.json into one table at the end.

import numpy as np, pandas as pd
from sklearn.datasets import make_classification
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

def load_runs():
    rows = []
    for meta_path in sorted(RUNS.glob("*/meta.json")):
        m = json.loads(meta_path.read_text())
        rows.append({"id": m["id"], "name": m["name"], "git": m["git_sha"][:7],
                     **{f"p.{k}": v for k, v in m["config"].items()},
                     **{f"m.{k}": v for k, v in (m.get("summary") or {}).items()}})
    return pd.DataFrame(rows)

X, y = make_classification(n_samples=8000, n_features=20, n_informative=8, random_state=0)
np.save("dataset.npy", np.column_stack([X, y]))
Xtr, Xva, ytr, yva = train_test_split(X, y, test_size=2000, random_state=0)

for alpha in [1e-5, 1e-4, 1e-3, 1e-2]:
    for seed in [0, 1, 2]:
        cfg = {"alpha": alpha, "seed": seed, "loss": "log_loss", "epochs": 15}
        r = Run(f"sgd-a{alpha}", cfg, data_path="dataset.npy")
        m = SGDClassifier(loss="log_loss", alpha=alpha, random_state=seed)
        best = 0.0
        for epoch in range(cfg["epochs"]):
            m.partial_fit(Xtr, ytr, classes=[0, 1])
            auc = roc_auc_score(yva, m.decision_function(Xva))
            r.log(epoch, val_auc=auc)
            best = max(best, auc)
        r.finish(val_auc=round(best, 4))

df = load_runs()
print(df.sort_values(["p.alpha", "p.seed"])[["p.alpha", "p.seed", "m.val_auc"]]
        .to_string(index=False))
print(df.groupby("p.alpha")["m.val_auc"].agg(["mean", "std", "count"]).round(4))

 p.alpha  p.seed  m.val_auc
 0.00001       0     0.9801
 0.00001       1     0.9808
 0.00001       2     0.9823
 0.00010       0     0.9809
 0.00010       1     0.9804
 0.00010       2     0.9840
 0.00100       0     0.9844
 0.00100       1     0.9832
 0.00100       2     0.9836
 0.01000       0     0.9843
 0.01000       1     0.9842
 0.01000       2     0.9840

           mean     std  count
p.alpha
0.00001  0.9811  0.0011      3
0.00010  0.9818  0.0020      3
0.00100  0.9837  0.0006      3
0.01000  0.9842  0.0002      3

Read the two halves against each other. The best single number in the sweep is 0.9844, at alpha 0.001 seed 0. The group means say alpha 0.01 is better, 0.9842 against 0.9837. Both cannot drive the decision, and the one that should is the mean, because the spread within a group is 0.0002 to 0.0020 and the gap between the best two groups is 0.0005. A sweep that ran one seed per setting would have reported alpha 0.001 as the winner with total confidence, on a difference smaller than the noise it never measured.

That is the discipline that makes a sweep interpretable, and it is three rules. Vary one thing at a time, or accept that you cannot attribute the result. Repeat every setting on at least three seeds, so the table carries its own error bar. Compare group means and not best runs, because the maximum of a noisy set is biased upward and the bias grows with how many settings you tried. None of that requires a tool; the tool only makes it cheap enough that you keep doing it on the twentieth run.

Determinism has limits#

This matters when comparing runs: a difference smaller than the run-to-run variance is not a result. Run the baseline more than once to find out how large that variance is.

Six months after a model reached production, someone will ask what it was trained on, and the answer has to come from a record rather than anyone's memory. This section sets that record up in an afternoon: a few lines around a training loop logging the git commit, the data version, the whole config and the metrics per epoch. The habit that makes it work is keeping settings in a config file instead of editing the script between runs. The gap that catches almost everyone is data versioning, where the code stays findable and the table it read has since been overwritten.

The smallest useful setup#

Whole afternoon, and this is the afternoon. A SQLite file for the backend, six added lines around a training loop, and a query at the end that reads runs back as a table. There is no server to run and nothing to install beyond the library.

import mlflow, subprocess
from sklearn.datasets import make_classification
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

mlflow.set_tracking_uri("sqlite:///mlflow.db")
config = {"lr": 0.01, "epochs": 15, "seed": 0, "model": "sgd-log"}

def current_sha():
    try:
        return subprocess.check_output(["git", "rev-parse", "HEAD"], text=True,
                                       stderr=subprocess.DEVNULL).strip()
    except Exception:
        return "unversioned"

X, y = make_classification(n_samples=6000, n_features=20, random_state=0)
Xtr, Xva, ytr, yva = train_test_split(X, y, test_size=2000, random_state=0)

with mlflow.start_run(run_name="lr-sweep-03"):
    mlflow.set_tag("git_commit", current_sha())
    mlflow.set_tag("data_version", "orders_2025_06_v2")
    mlflow.log_params(config)
    m = SGDClassifier(loss="log_loss", eta0=config["lr"], learning_rate="constant",
                      random_state=config["seed"])
    for epoch in range(config["epochs"]):
        m.partial_fit(Xtr, ytr, classes=[0, 1])
        val_auc = roc_auc_score(yva, m.decision_function(Xva))
        mlflow.log_metric("val_auc", val_auc, step=epoch)
    mlflow.log_metric("best_val_auc", val_auc)

runs = mlflow.search_runs(order_by=["metrics.best_val_auc DESC"])
print(runs[["tags.mlflow.runName", "params.lr", "tags.data_version",
            "metrics.best_val_auc"]].to_string(index=False))

tags.mlflow.runName params.lr tags.data_version  metrics.best_val_auc
        lr-sweep-03      0.01 orders_2025_06_v2              0.951398

Weights and Biases and similar tools work the same way. The tool matters less than the habit: every run logs its code version, data version, config and results, automatically. Two details in that code are the ones people leave out and regret. The commit is read by the program rather than typed by a person, because a person types the commit they think they are on. And log_params takes the whole config dictionary rather than the three settings that seemed interesting, because the interesting one is always the fourth.

Config files, not edited scripts#

If you change hyperparameters by editing the training script, the run record cannot tell you what the script said at the time. Keep settings in a config file or command line arguments, log the whole config, and commit code before running anything you might want to keep.

The reason this is a rule and not a preference is that a config in a file can be logged as data, and a config inside the script can only be logged as code. Once the settings are a dictionary loaded from disk, the run record contains the exact object the program used, comparing two runs is a dictionary diff, and rerunning one is passing the recorded config back in. When the settings are constants edited between runs, the record has to reconstruct them from the commit, which only works if every single run was committed first, which nobody does at four in the afternoon on the twelfth attempt.

# what the run record contains

with a config file      config: {lr: 0.01, epochs: 15, seed: 0, model: sgd-log}
                        git_commit: a3f21e (clean)
                        -> rerun by passing the config back in

with edited constants   git_commit: a3f21e-dirty
                        -> rerun by guessing what was uncommitted

The same argument applies to the dirty flag, which is why it is worth computing rather than only recording the hash. A run tagged with a clean commit can be reproduced from that commit. A run tagged dirty cannot be reproduced at all, and it is much better to know that in the table than to discover it six months later after an afternoon of trying.

The gap is always the data#

Of the five things a run needs, four are easy: the code is in git, the config is a dictionary, the environment is a lockfile, the seed is an integer. The data is the one that defeats people, because the natural way to record it is a name — orders_2025_06_v2, or a warehouse table, or a path on shared storage — and a name is a pointer to something mutable. Six months later the code is exactly findable and the table it read has been rebuilt four times.

• Best: an immutable snapshot with a content hash. Write the training extract to a file, hash it, log the hash. Reproduction is then verifiable rather than hopeful.
• Good: a warehouse with time travel, where a query can be pinned to a timestamp or a version number and replayed exactly.
• Acceptable: a write-once path convention, where a dated directory is never modified after it is written and the pipeline creates a new one instead.
• Not enough: a table name and a date in a tag. That is what most teams have, and it records intent rather than content.

The cheap version of the best option is one line: hash the file and log the hash. It costs a second on a gigabyte, it proves two runs saw the same bytes, and it turns the question of whether the data changed from an argument into a comparison.

Reproducing a run six months later#

1. 1.Code: the commit hash, with no uncommitted changes.
2. 2.Data: a version or snapshot id, not 'the table as it is today'.
3. 3.Environment: locked dependency versions, and the hardware type.
4. 4.Config: every setting, including defaults you did not change.
5. 5.Seeds: logged, knowing that GPU nondeterminism can still cause small differences.

Reproducing does not mean bit-identical, and aiming for that wastes time. It means landing within the run-to-run variance you measured by rerunning the baseline on several seeds. If the rebuilt run lands inside that spread, the record was good enough. If it lands outside, one of the five items above is wrong, and the order to check them in is data, environment, config, code, seed — which is roughly the order of how often each one is the culprit.

What to skip#

• Skip the hosted tracking server until several people or several machines produce runs. SQLite on a laptop does everything described here.
• Skip logging model weights for every run in a sweep. Log metrics and config for all of them and the artefact only for runs you promote, or the storage bill arrives before the insight does.
• Skip chasing bit-exact determinism. Measure the seed spread instead and use it as the bar.
• Worth memorising: the five things a run needs, that the data is the one that will be missing, and that a dirty commit means not reproducible.