Storage layout

Maturity: verified

Storage is organized by a durable logical run, then by distinct execution attempts within that run, then by the application that owns each artifact. This hierarchy preserves lineage and prevents a retry or resume from overwriting evidence from an earlier attempt.

Pretraining artifact namespace

For either a filesystem output root or an executor-mounted gs:// root, the current pretraining application owns this exact namespace:

<artifact-root>/
  <run-id>/
    attempts/
      <attempt-id>/
        application/
          resolved_config.sha256_<64-hex-digest>.yaml
          metrics.jsonl
          run_manifest.json
          checkpoints/
            step_<eight-digit-global-step>.pt
          non_finite_training_error.json  # failed non-finite attempt only

The application creates application/ with exist_ok=False; it will not reuse or silently overwrite an attempt namespace. The checkpoint directory name is configuration-owned but must be one safe path component. The shipped value is checkpoints.

Publication order

For success, the application writes config and checkpoints, atomically writes completed-step metrics, and publishes the terminal manifest last. The manifest content-addresses the final config, metrics, and terminal checkpoint.

For a non-finite failure, it attempts to publish completed metrics and the diagnostic before a failed terminal manifest. If evidence publication itself fails, those errors are attached to the original training exception instead of replacing it.

URI resolution

Scheme-less paths resolve directly. A gs://bucket/path URI resolves under the executor-supplied mounted namespace as <mount-root>/bucket/path. The application does not invoke a cloud SDK or own authentication and mounting.

See Also