Resume training

Maturity: verified

Resume continues a logical training run from persisted state while preserving the identity and lineage of both the earlier and later execution attempts.

Resume behavior

The current path starts a new bounded attempt in the same logical run, restores a compatible completed-step checkpoint, and advances to a larger training.max_steps.

Prerequisites

  • a checkpoint URI produced by a successful bounded attempt;

  • the same canonical dataset content and selected component implementations;

  • a new attempt ID;

  • a target maximum step greater than the checkpoint step;

  • the same mounted bucket namespace required by the dataset, artifact root, and checkpoint URI.

Run locally or in an existing worker

Use the same application command as the parent attempt and add two overrides:

PYTHONPATH=src:. uv run --project analysis/eda \
  python -m rffm.cli.main pretrain \
  --config-dir configs \
  --experiment radioml2018_one_step_npp \
  --run-id <same-logical-run-id> \
  --attempt-id <new-attempt-id> \
  --rffm-repository-root . \
  --gcs-bucket-namespace-root /gcs \
  --override training.max_steps=2 \
  --override checkpoint.resume_from=<parent-checkpoint-uri>

Do not reuse the parent attempt ID. Do not point resume_from at a checkpoint whose run, training-contract, dataset, or component identities differ.

Submit through the Gemini Enterprise Agent Platform adapter

The submission script exposes the same pair as environment variables:

RESUME_FROM=<parent-checkpoint-uri> \
TARGET_MAX_STEPS=2 \
infra/runtime/submit_vertex_ai_training.sh

Supply the required display name, run and new attempt IDs, experiment, image digest, source revision, and launch-config variables described in the Gemini Enterprise Agent Platform reference. The script rejects cases where only one of RESUME_FROM and TARGET_MAX_STEPS is set.

Verify the result

The new run_manifest.json must record the parent checkpoint URI, parent config hash, and parent attempt ID. Its checkpoint metadata must record the new completed global step and the same parent URI. A step-two run’s metrics file contains only the step completed by that new attempt.

This procedure proves the bounded continuation contract. It does not restore loader position or random-number-generator state and is not a general exact resume recipe.

See Also