lars/giant
1
0
Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
12 Commits 5 Branches 1 Tag
c3b7b2744c502129228712cffeb4cf4af82ad885
Commit Graph

5 Commits

Author SHA1 Message Date
lars c3b7b2744c Batch StreamingStepsDataset internally instead of per-row collate 
The dataset yielded one row at a time, forcing DataLoader's default
collate to Python-loop over every row to assemble each batch. That
loop scales with batch size and was pinning a CPU core at 100% while
the GPU sat idle. Now the dataset yields whole batches via vectorized
numpy slicing, used with DataLoader(batch_size=None).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-18 10:08:36 +02:00
lars 2e4b4d91d1 Add ROOT-to-parquet conversion script with convert dependency group 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-17 11:27:49 +02:00
lars 93c4d6b74d Add streaming data pipeline and giant CLI entry point 
- Streaming pipeline: row-group-level parquet reading (PyArrow) so
  large files never fully land in RAM; Welford online algorithm for
  normalizer fitting; StreamingStepsDataset with shuffle buffer and
  multi-worker file striping; event-ID scan and vocab scan via cheap
  single-column reads
- giant/cli.py: typer-based CLI with `giant train` subcommand, mirroring
  scripts/train.py; --shuffle-buffer flag for RAM control
- pyproject.toml: add typer>=0.12 dependency and giant entry point
- train.py: replace len(loader.dataset) with local counters (compatible
  with IterableDataset which has no __len__)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-17 11:03:48 +02:00
lars 9277d79dff Implement Phase 1: full data pipeline, model, training, and config support 
- Data pipeline: loader (parquet→numpy), transforms (log, local-frame
  Rodrigues rotation, Normalizer), StepsDataset with event-ID-based split
- Model: SinusoidalEmbedding, ConditionEncoder, ResBlock, DenoisingMLP
- Schedule: cosine DDPM and conditional flow matching loss (Lipman 2022)
- Samplers: flow (Euler ODE), DDPM ancestral, DDIM deterministic
- Training loop: AdamW + cosine LR, grad clipping, best-val checkpoint
- Validation: per-dimension marginal summary (normalised space)
- CLI: TOML config support with CLI-overrides; hyperparam-encoded output
  directory; config.toml with git hash saved into each run's checkpoint dir
- 21 unit tests covering transforms, network, flow/DDPM losses, dataset splits

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-17 10:48:03 +02:00
lars 943a044608 Initial commit: giant surrogate model with two-phase roadmap in README 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-17 09:29:19 +02:00