cli.py and scripts/train.py duplicated ~140 lines of training setup and had
drifted (scripts/train.py forgot to save model_config, breaking predict on
those checkpoints). Extract shared logic into giant/constants.py (X_DIM,
target names), giant/config.py (device/git/TOML/seeding helpers, run
metadata), and giant/pipeline.py (the actual training-job orchestration),
so both entry points become thin CLI wrappers around the same code path.
Also adds --seed/--resume support (checkpoints now carry optimizer/scheduler
state, epoch, and best_val_loss), a richer [meta] section in the saved
config.toml (git hash, seed, versions, timestamp, invocation, dataset
stats), and a metrics.csv (train/val loss, lr, epoch time) written every
epoch and append-safe across resumes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
step_length already encodes |post_pos - pre_pos| by definition, so a raw
post_pos target would duplicate that magnitude and could drift inconsistent
with step_length during sampling. Instead add travel_dir, a unit vector
(local frame) giving only the direction of pre_pos->post_pos; post_pos is
reconstructed at inference as pre_pos + step_length * travel_dir, keeping
the two self-consistent. Target grows from 6D to 9D.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The dataset yielded one row at a time, forcing DataLoader's default
collate to Python-loop over every row to assemble each batch. That
loop scales with batch size and was pinning a CPU core at 100% while
the GPU sat idle. Now the dataset yields whole batches via vectorized
numpy slicing, used with DataLoader(batch_size=None).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Streaming pipeline: row-group-level parquet reading (PyArrow) so
large files never fully land in RAM; Welford online algorithm for
normalizer fitting; StreamingStepsDataset with shuffle buffer and
multi-worker file striping; event-ID scan and vocab scan via cheap
single-column reads
- giant/cli.py: typer-based CLI with `giant train` subcommand, mirroring
scripts/train.py; --shuffle-buffer flag for RAM control
- pyproject.toml: add typer>=0.12 dependency and giant entry point
- train.py: replace len(loader.dataset) with local counters (compatible
with IterableDataset which has no __len__)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lars