Add streaming data pipeline and giant CLI entry point

- Streaming pipeline: row-group-level parquet reading (PyArrow) so
  large files never fully land in RAM; Welford online algorithm for
  normalizer fitting; StreamingStepsDataset with shuffle buffer and
  multi-worker file striping; event-ID scan and vocab scan via cheap
  single-column reads
- giant/cli.py: typer-based CLI with `giant train` subcommand, mirroring
  scripts/train.py; --shuffle-buffer flag for RAM control
- pyproject.toml: add typer>=0.12 dependency and giant entry point
- train.py: replace len(loader.dataset) with local counters (compatible
  with IterableDataset which has no __len__)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-06-17 11:03:48 +02:00
parent 9277d79dff
commit 93c4d6b74d
8 changed files with 538 additions and 30 deletions
+4
View File
@@ -9,6 +9,7 @@ dependencies = [
"numpy>=1.26",
"pandas>=2.2",
"pyarrow>=16",
"typer>=0.12",
]
[project.optional-dependencies]
@@ -16,6 +17,9 @@ dev = [
"pytest>=8",
]
[project.scripts]
giant = "giant.cli:app"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"