Resume training

Maturity: verified

Resume continues a logical training run from persisted state while preserving the identity and lineage of both the earlier and later execution attempts.

Resume behavior

The current path starts a new bounded attempt in the same logical run, restores a compatible completed-step checkpoint, and advances to a larger training.max_steps.

Prerequisites

  • the latest checkpoint URI recorded by a completed attempt;

  • the same canonical dataset content and selected component implementations;

  • a new attempt ID;

  • a target maximum step greater than the checkpoint step;

  • the same mounted bucket namespace required by the dataset, artifact root, and checkpoint URI.

Run locally or in an existing worker

Use the same application command as the parent attempt and add two overrides:

PYTHONPATH=src:. uv run --project analysis/eda \
  python -m rffm.cli.main pretrain \
  --config-dir configs \
  --experiment radioml2018_one_step_npp \
  --run-id <same-logical-run-id> \
  --attempt-id <new-attempt-id> \
  --rffm-repository-root . \
  --gcs-bucket-namespace-root /gcs \
  --override training.max_steps=2 \
  --override checkpoint.resume_from=<parent-checkpoint-uri>

Do not reuse the parent attempt ID. Do not point resume_from at a checkpoint whose run, training-contract, dataset, or component identities differ.

Submit through a managed adapter

The submission script exposes the same pair as environment variables:

RESUME_FROM=<parent-checkpoint-uri> \
TARGET_MAX_STEPS=2 \
infra/runtime/submit_vertex_ai_training.sh

Use submit_cloud_batch_training.sh instead for configured GPU training with the dataset cache. Supply the required provider job name, run and new attempt IDs, experiment, image digest, source revision, and launch-config variables described by the selected provider reference. The script rejects cases where only one of RESUME_FROM and TARGET_MAX_STEPS is set.

Verify the result

The new run_manifest.json records the parent checkpoint URI, parent config hash, and parent attempt ID. Its latest checkpoint records the new completed global step and the same parent URI. The child metric file begins with the exact checkpoint-bound parent prefix and appends the child attempt’s records.

The application restores model, optimizer, scheduler, loader, loop, metric, and CPU/GPU random-number state before continuing. The deterministic CPU integration test requires the resumed training state and metrics to match uninterrupted execution exactly; attempt metadata intentionally differs. CUDA uses the declared deterministic or numerical-tolerance qualification.

See Also