Experiment tracking

Maturity: verified

Experiment tracking owns the durable identity, lineage, metrics, artifacts, and terminal status needed to inspect and compare work across a logical run and its execution attempts. A run is the stable experiment identity; each attempt is a distinct execution that contributes evidence without overwriting another attempt.

Attempt evidence

The current bounded application implements tracking as attempt-local files on local or mounted storage. It does not integrate TensorBoard, MLflow, Weights & Biases, or a tracker plugin.

Evidence set

One successful attempt publishes:

  • the complete resolved YAML config in a content-named file;

  • metrics.jsonl, containing one record per completed optimizer step;

  • scheduled PyTorch checkpoints;

  • run_manifest.json, published last as the terminal inventory.

The manifest records run, attempt, config and training-contract identities; dataset name, snapshot, split and split fingerprint; selected implementations; device and optional provider provenance; resume lineage; status and final step; and byte counts plus SHA-256 hashes for referenced artifacts.

On a non-finite training error, the application also writes non_finite_training_error.json. The failed manifest identifies the failing stage and attempted step, and includes only checkpoints completed by the current attempt.

What metrics mean

The current step record contains event_type="optimizer_step", global_step, loss, num_target_patches, and the ordered sample_ids. It is completed-step evidence, not a validation or benchmark result.

Provider provenance

Managed callers may supply the exact source revision, immutable container image URI, provider job ID, launch-config URI, and launch-config SHA-256. The application records these values but does not independently attest that the provider ran them.

See Also