Tracking that people actually use

Six months after a model reached production, someone will ask what it was trained on, and the answer has to come from a record rather than anyone's memory. This section sets that record up in an afternoon: a few lines around a training loop logging the git commit, the data version, the whole config and the metrics per epoch. The habit that makes it work is keeping settings in a config file instead of editing the script between runs. The gap that catches almost everyone is data versioning, where the code stays findable and the table it read has since been overwritten.

The smallest useful setup#

Whole afternoon, and this is the afternoon. A SQLite file for the backend, six added lines around a training loop, and a query at the end that reads runs back as a table. There is no server to run and nothing to install beyond the library.

import mlflow, subprocess
from sklearn.datasets import make_classification
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

mlflow.set_tracking_uri("sqlite:///mlflow.db")
config = {"lr": 0.01, "epochs": 15, "seed": 0, "model": "sgd-log"}

def current_sha():
    try:
        return subprocess.check_output(["git", "rev-parse", "HEAD"], text=True,
                                       stderr=subprocess.DEVNULL).strip()
    except Exception:
        return "unversioned"

X, y = make_classification(n_samples=6000, n_features=20, random_state=0)
Xtr, Xva, ytr, yva = train_test_split(X, y, test_size=2000, random_state=0)

with mlflow.start_run(run_name="lr-sweep-03"):
    mlflow.set_tag("git_commit", current_sha())
    mlflow.set_tag("data_version", "orders_2025_06_v2")
    mlflow.log_params(config)
    m = SGDClassifier(loss="log_loss", eta0=config["lr"], learning_rate="constant",
                      random_state=config["seed"])
    for epoch in range(config["epochs"]):
        m.partial_fit(Xtr, ytr, classes=[0, 1])
        val_auc = roc_auc_score(yva, m.decision_function(Xva))
        mlflow.log_metric("val_auc", val_auc, step=epoch)
    mlflow.log_metric("best_val_auc", val_auc)

runs = mlflow.search_runs(order_by=["metrics.best_val_auc DESC"])
print(runs[["tags.mlflow.runName", "params.lr", "tags.data_version",
            "metrics.best_val_auc"]].to_string(index=False))

tags.mlflow.runName params.lr tags.data_version  metrics.best_val_auc
        lr-sweep-03      0.01 orders_2025_06_v2              0.951398

Weights and Biases and similar tools work the same way. The tool matters less than the habit: every run logs its code version, data version, config and results, automatically. Two details in that code are the ones people leave out and regret. The commit is read by the program rather than typed by a person, because a person types the commit they think they are on. And log_params takes the whole config dictionary rather than the three settings that seemed interesting, because the interesting one is always the fourth.

Config files, not edited scripts#

If you change hyperparameters by editing the training script, the run record cannot tell you what the script said at the time. Keep settings in a config file or command line arguments, log the whole config, and commit code before running anything you might want to keep.

The reason this is a rule and not a preference is that a config in a file can be logged as data, and a config inside the script can only be logged as code. Once the settings are a dictionary loaded from disk, the run record contains the exact object the program used, comparing two runs is a dictionary diff, and rerunning one is passing the recorded config back in. When the settings are constants edited between runs, the record has to reconstruct them from the commit, which only works if every single run was committed first, which nobody does at four in the afternoon on the twelfth attempt.

# what the run record contains

with a config file      config: {lr: 0.01, epochs: 15, seed: 0, model: sgd-log}
                        git_commit: a3f21e (clean)
                        -> rerun by passing the config back in

with edited constants   git_commit: a3f21e-dirty
                        -> rerun by guessing what was uncommitted

The same argument applies to the dirty flag, which is why it is worth computing rather than only recording the hash. A run tagged with a clean commit can be reproduced from that commit. A run tagged dirty cannot be reproduced at all, and it is much better to know that in the table than to discover it six months later after an afternoon of trying.

The gap is always the data#

Of the five things a run needs, four are easy: the code is in git, the config is a dictionary, the environment is a lockfile, the seed is an integer. The data is the one that defeats people, because the natural way to record it is a name — orders_2025_06_v2, or a warehouse table, or a path on shared storage — and a name is a pointer to something mutable. Six months later the code is exactly findable and the table it read has been rebuilt four times.

• Best: an immutable snapshot with a content hash. Write the training extract to a file, hash it, log the hash. Reproduction is then verifiable rather than hopeful.
• Good: a warehouse with time travel, where a query can be pinned to a timestamp or a version number and replayed exactly.
• Acceptable: a write-once path convention, where a dated directory is never modified after it is written and the pipeline creates a new one instead.
• Not enough: a table name and a date in a tag. That is what most teams have, and it records intent rather than content.

The cheap version of the best option is one line: hash the file and log the hash. It costs a second on a gigabyte, it proves two runs saw the same bytes, and it turns the question of whether the data changed from an argument into a comparison.

Reproducing a run six months later#

1. 1.Code: the commit hash, with no uncommitted changes.
2. 2.Data: a version or snapshot id, not 'the table as it is today'.
3. 3.Environment: locked dependency versions, and the hardware type.
4. 4.Config: every setting, including defaults you did not change.
5. 5.Seeds: logged, knowing that GPU nondeterminism can still cause small differences.

Reproducing does not mean bit-identical, and aiming for that wastes time. It means landing within the run-to-run variance you measured by rerunning the baseline on several seeds. If the rebuilt run lands inside that spread, the record was good enough. If it lands outside, one of the five items above is wrong, and the order to check them in is data, environment, config, code, seed — which is roughly the order of how often each one is the culprit.

What to skip#

• Skip the hosted tracking server until several people or several machines produce runs. SQLite on a laptop does everything described here.
• Skip logging model weights for every run in a sweep. Log metrics and config for all of them and the artefact only for runs you promote, or the storage bill arrives before the insight does.
• Skip chasing bit-exact determinism. Measure the seed spread instead and use it as the bar.
• Worth memorising: the five things a run needs, that the data is the one that will be missing, and that a dirty commit means not reproducible.