Canonical datasets

Maturity: verified

Training reads canonical bundles, never raw dataset formats. Dataset-specific preprocessing converts raw sources into one HDF5, Parquet, JSON, and persisted split contract; the training reader then joins every artifact by canonical sample ID.

Bundle flow

raw source
  -> dataset-specific preprocessing adapter
  -> common canonical bundle writer and validator
  -> manifest.json + metadata.parquet + signals.h5 + splits/*.parquet
  -> one CanonicalRFDataset split
  -> semantic samples
  -> DataLoader with rf_collate

The reader validates the manifest, unique metadata IDs, unique ordered split membership, signal-row IDs, and cross-artifact membership before returning a sample.

Dataset readiness

Dataset

Registry readiness

Training status

RadioML 2018.01A

validated

selected by the shipped bounded experiment

Daytona ADS-B

validated

readable through the common canonical path; not selected by bounded pretraining

RadioML2016, HisarMod, TorchSig Sig53, NIST Radar, DroneRF, WiSig, LoRa-OSU

present

raw mirror present; canonical training bundle not validated

present is not a weaker spelling of ready. It records only that raw source material exists. A training guide must not point at those entries until their canonical bundles are generated and validated.

Reader boundary

CanonicalRFDataset is a map-style PyTorch dataset over one bundle and one split. It owns file reading, joins, sample validation, and the split fingerprint. It does not own remote downloads, batching, normalization, patching, model construction, or training lifecycle.

The pretraining application resolves the selected registry URI to a path visible to the current process. Scheme-less paths work directly. A gs:// URI requires an executor-supplied mounted bucket namespace.

See Also