Training loop

Maturity: verified

The training loop owns the ordered transition from a batch to a completed optimizer step.

Bounded pretraining loop

The current pretraining application owns a small, explicit PyTorch loop. It is bounded by training.max_steps; it has no epoch, validation, scheduler, gradient-accumulation, mixed-precision, callback, or distributed lifecycle. The application records completed-step evidence and publishes terminal attempt artifacts around that loop.

One completed step

For each global step, the application:

  1. reads the next deterministic batch, restarting the non-empty dataset when it reaches the end;

  2. moves tensor fields to the configured CPU or NVIDIA CUDA GPU, selected by the exact training.device literal cpu or cuda;

  3. normalizes and patches the batch with prepare_iq_input;

  4. calls the selected program’s single optimizer-step primitive;

  5. records the completed-step loss, target-patch count, and sample IDs;

  6. saves a checkpoint when the cadence divides the step or the step is the configured final step.

The shipped loader uses shuffle=False. torch.manual_seed(config.seed) runs once before model construction. These facts make the bounded evidence path inspectable; they do not establish exact replay for a general training system.

Handling training failures

The reusable optimizer step checks finite inputs, parameters before the step, forward outputs, loss, gradients, and parameters after the step. If a check fails, the application excludes the attempted step from completed metrics, writes a non-finite diagnostic, and publishes a failed terminal manifest when all referenced failure artifacts are available.

A successful manifest is published only after the resolved config, metrics, and terminal checkpoint exist and have byte counts and SHA-256 hashes.

Retry and provider responsibilities

run_pretraining_attempt creates exactly one attempt namespace. Its caller owns retries and cross-attempt orchestration. The application does not build a container, submit a provider job, monitor that job, or delete remote resources.

See Also