Knowing which run produced which result

After a fortnight of experiments you will have a result you cannot reproduce and a model file you cannot account for. This section is about the record that prevents it: what to log for every run so it can be traced back, how to compare runs meaningfully, where the resulting artifacts live, and why exact determinism is harder than it sounds and usually not what you actually need.

The problem#

Three weeks ago a run scored well. Which configuration was it? Which data? Without tracking, that result is gone, and everyone learns this the hard way exactly once.

Why it is in the notebook does not count#

The usual answer to where a result lives is that it is in the notebook. That fails for four separate reasons, and it is worth having all four to hand because people will argue the first three individually.

• A notebook records the last state of each cell, not the sequence that produced the number. Cells run out of order leave a printed 0.93 under code that would now print something else, and nothing in the file says which.
• It records the output and not the inputs. The cell that read the data three cells up has since been edited; the number below it has not.
• It is one run. The comparison you actually need is across twenty, and twenty notebooks with the numbers in different cells is not a table.
• It is not searchable by result. Six months later the question is which run got above 0.88, and the only way to answer it from notebooks is to open all of them.

The fix is not to stop using notebooks. It is that the run's record has to be written by the code, to a place outside the notebook, every time, without anyone deciding this run is worth keeping. Runs are only interesting in retrospect, and the ones you did not expect to matter are exactly the ones you will want back.

What a tracker actually is#

MLflow and Weights and Biases look like infrastructure, and that impression makes people put off adopting one and misjudge what they are getting. The whole of it is a directory per run, a JSON file of what the run was, and an append-only file of what happened during it. Here it is, in about forty lines, so the tools stop being magic.

import hashlib, json, platform, subprocess, sys, time, uuid
from pathlib import Path

RUNS = Path("runs")

def git_sha():
    try:
        sha = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True,
                                      stderr=subprocess.DEVNULL).strip()
        dirty = subprocess.check_output(["git", "status", "--porcelain"], text=True,
                                        stderr=subprocess.DEVNULL).strip()
        return sha + ("-dirty" if dirty else "")
    except Exception:
        return "no-git"

def file_hash(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()[:12]

def freeze():
    out = subprocess.check_output([sys.executable, "-m", "pip", "freeze"], text=True)
    return sorted(l for l in out.splitlines() if l.split("==")[0].lower() in
                  {"numpy", "pandas", "scikit-learn", "torch", "scipy"})

class Run:
    """One directory per run: meta.json for what it was, metrics.jsonl for what happened."""

    def __init__(self, name, config, data_path=None):
        self.id = time.strftime("%Y%m%d-%H%M%S") + "-" + uuid.uuid4().hex[:6]
        self.dir = RUNS / self.id
        self.dir.mkdir(parents=True)
        self.meta = {
            "id": self.id, "name": name, "config": config,
            "git_sha": git_sha(),
            "data": {"path": str(data_path), "sha256_12": file_hash(data_path)}
                    if data_path else None,
            "python": sys.version.split()[0], "platform": platform.platform(),
            "packages": freeze(), "started": time.time(),
        }
        (self.dir / "meta.json").write_text(json.dumps(self.meta, indent=2))
        self.metrics = open(self.dir / "metrics.jsonl", "a")

    def log(self, step, **values):
        self.metrics.write(json.dumps({"step": step, **values}) + "\n")
        self.metrics.flush()

    def finish(self, **summary):
        self.metrics.close()
        self.meta["summary"] = summary
        self.meta["finished"] = time.time()
        (self.dir / "meta.json").write_text(json.dumps(self.meta, indent=2))

Five fields in that class are the ones that make a run reproducible six months later, and every one of them is a thing people omit. The git sha, with the dirty flag, because a commit hash with uncommitted changes on top of it is a lie. The content hash of the data, because a path is a name and names get reused. The full config, including the defaults nobody changed, because the default will change in a library upgrade. The interpreter and package versions, because a result that only reproduces on the old numpy is still worth knowing about. The seed, inside the config, so that the run-to-run variation can be separated from the effect you are measuring.

What the real tools add on top of this is a server so runs from several machines land in one place, a user interface for plotting, artifact storage that is not your laptop, and a registry that connects a run to the model serving in production. None of that changes what a run is. If you understand the forty lines you can pick a tool in an afternoon and you will not be confused by any of them.

Comparing runs, and what a sweep is for#

The record only pays for itself when it is read across runs. Twelve runs through the tracker above, four values of a regularisation strength crossed with three seeds, loading every meta.json into one table at the end.

import numpy as np, pandas as pd
from sklearn.datasets import make_classification
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

def load_runs():
    rows = []
    for meta_path in sorted(RUNS.glob("*/meta.json")):
        m = json.loads(meta_path.read_text())
        rows.append({"id": m["id"], "name": m["name"], "git": m["git_sha"][:7],
                     **{f"p.{k}": v for k, v in m["config"].items()},
                     **{f"m.{k}": v for k, v in (m.get("summary") or {}).items()}})
    return pd.DataFrame(rows)

X, y = make_classification(n_samples=8000, n_features=20, n_informative=8, random_state=0)
np.save("dataset.npy", np.column_stack([X, y]))
Xtr, Xva, ytr, yva = train_test_split(X, y, test_size=2000, random_state=0)

for alpha in [1e-5, 1e-4, 1e-3, 1e-2]:
    for seed in [0, 1, 2]:
        cfg = {"alpha": alpha, "seed": seed, "loss": "log_loss", "epochs": 15}
        r = Run(f"sgd-a{alpha}", cfg, data_path="dataset.npy")
        m = SGDClassifier(loss="log_loss", alpha=alpha, random_state=seed)
        best = 0.0
        for epoch in range(cfg["epochs"]):
            m.partial_fit(Xtr, ytr, classes=[0, 1])
            auc = roc_auc_score(yva, m.decision_function(Xva))
            r.log(epoch, val_auc=auc)
            best = max(best, auc)
        r.finish(val_auc=round(best, 4))

df = load_runs()
print(df.sort_values(["p.alpha", "p.seed"])[["p.alpha", "p.seed", "m.val_auc"]]
        .to_string(index=False))
print(df.groupby("p.alpha")["m.val_auc"].agg(["mean", "std", "count"]).round(4))

 p.alpha  p.seed  m.val_auc
 0.00001       0     0.9801
 0.00001       1     0.9808
 0.00001       2     0.9823
 0.00010       0     0.9809
 0.00010       1     0.9804
 0.00010       2     0.9840
 0.00100       0     0.9844
 0.00100       1     0.9832
 0.00100       2     0.9836
 0.01000       0     0.9843
 0.01000       1     0.9842
 0.01000       2     0.9840

           mean     std  count
p.alpha
0.00001  0.9811  0.0011      3
0.00010  0.9818  0.0020      3
0.00100  0.9837  0.0006      3
0.01000  0.9842  0.0002      3

Read the two halves against each other. The best single number in the sweep is 0.9844, at alpha 0.001 seed 0. The group means say alpha 0.01 is better, 0.9842 against 0.9837. Both cannot drive the decision, and the one that should is the mean, because the spread within a group is 0.0002 to 0.0020 and the gap between the best two groups is 0.0005. A sweep that ran one seed per setting would have reported alpha 0.001 as the winner with total confidence, on a difference smaller than the noise it never measured.

That is the discipline that makes a sweep interpretable, and it is three rules. Vary one thing at a time, or accept that you cannot attribute the result. Repeat every setting on at least three seeds, so the table carries its own error bar. Compare group means and not best runs, because the maximum of a noisy set is biased upward and the bias grows with how many settings you tried. None of that requires a tool; the tool only makes it cheap enough that you keep doing it on the twentieth run.

Determinism has limits#

This matters when comparing runs: a difference smaller than the run-to-run variance is not a result. Run the baseline more than once to find out how large that variance is.