Checkpointing¶
Maturity: verified
Checkpointing persists training state at a completed optimizer step so a later attempt can validate lineage and continue compatible work.
Checkpointing scope¶
The current checkpoint is a strict PyTorch payload for one completed optimizer step. It supports the demonstrated bounded continuation from step one to step two; it is not an exact-resume contract for general training.
Saved state¶
Every checkpoint has exactly three top-level fields:
training_module_state_dict;optimizer_state_dict;metadata.
Metadata identifies the completed global step, full resolved-config hash, training-contract hash, logical run and producing attempt, selected model, objective and optimizer implementations, dataset split fingerprint, and optional parent-checkpoint URI.
The current payload has no scheduler, scaler, sampler, dataloader position, or
Python, NumPy, PyTorch CPU, or accelerator random-number-generator state. The
current accelerator path uses an NVIDIA CUDA GPU selected by the exact cuda
configuration literal. Consequently, the supported claim is compatible
bounded continuation, not bit-identical replay.
Restoring a checkpoint¶
The application constructs a program from the target config, reads and validates checkpoint metadata, then restores model and optimizer state. Compatibility is checked before those objects are mutated. Resume requires:
the same logical run ID;
the same training-contract hash;
the same model, objective, and optimizer implementation keys;
the same dataset split fingerprint;
a new attempt ID;
a target maximum step greater than the restored completed step.
The resolved-config hash may differ because the training-contract hash excludes the output root, target maximum step, and resume URI. That is what permits a new attempt namespace and a larger bounded target without weakening training semantics.