Storage layout

Maturity: verified for application artifacts; runtime-isolated for provider log archives

Storage is organized by a durable logical run, then by distinct execution attempts within that run, then by the owner of each artifact. This hierarchy preserves lineage and prevents a retry or resume from overwriting evidence from an earlier attempt.

Pretraining artifact namespace

For the canonical managed procedure, the run evidence has this complete layout. The launch URI is supplied to the provider adapter; the application and provider evidence paths are derived from the configured artifact root, run ID, and attempt ID.

<artifact-root>/
  <run-id>/
    corrections/
      <content-sha256>.md
    attempts/
      <attempt-id>/
        launch/                              # provider-adapter-owned
          vertex_ai_l4.yaml                  # selected launch config; L4 example
        application/
          resolved_config.sha256_<64-hex-digest>.yaml
          metrics.jsonl
          run_manifest.json
          checkpoints/
            step_<eight-digit-global-step>.pt
          non_finite_training_error.json  # failed non-finite attempt only
        provider/                         # failed managed attempt only
          vertex_custom_job_logs.jsonl.gz
          provider_evidence.json

For this managed layout, supply the selected launch configuration’s immutable attempt URI as LAUNCH_CONFIG_URI, for example <artifact-root>/<run-id>/attempts/<attempt-id>/launch/vertex_ai_l4.yaml, vertex_ai_l4_flex_start.yaml, or vertex_ai_a100.yaml. The submission adapter uploads that exact file create-only before creating the job. The terminal manifest records the supplied URI and its SHA-256; provider evidence collection verifies both. The adapter accepts an explicit URI rather than deriving it, so callers using a different location must still preserve and supply that exact immutable object.

The application creates application/ with exist_ok=False; it will not reuse or silently overwrite an attempt namespace. The checkpoint directory name is configuration-owned but must be one safe path component. The shipped value is checkpoints.

After a failed managed attempt, provider tooling may create the sibling provider/ directory without modifying application/. Completed human-and-agent correction records are content-addressed under the logical run’s corrections/ directory. Both later surfaces use create-only publication.

Publication order

For success, the application writes config and checkpoints, atomically writes completed-step metrics, and publishes the terminal manifest last. The manifest content-addresses the final config, metrics, and terminal checkpoint.

For a non-finite failure, it attempts to publish completed metrics and the diagnostic before a failed terminal manifest. If evidence publication itself fails, those errors are attached to the original training exception instead of replacing it.

The provider log archive is published before provider_evidence.json. A correction record is published only after investigation, human decision, implementation, and validation are complete.

URI resolution

Scheme-less paths resolve directly. A gs://bucket/path URI resolves under the executor-supplied mounted namespace as <mount-root>/bucket/path. The application does not invoke a cloud SDK or own authentication and mounting.

See Also