Add streaming data pipeline and giant CLI entry point
- Streaming pipeline: row-group-level parquet reading (PyArrow) so large files never fully land in RAM; Welford online algorithm for normalizer fitting; StreamingStepsDataset with shuffle buffer and multi-worker file striping; event-ID scan and vocab scan via cheap single-column reads - giant/cli.py: typer-based CLI with `giant train` subcommand, mirroring scripts/train.py; --shuffle-buffer flag for RAM control - pyproject.toml: add typer>=0.12 dependency and giant entry point - train.py: replace len(loader.dataset) with local counters (compatible with IterableDataset which has no __len__) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -9,6 +9,7 @@ dependencies = [
|
||||
"numpy>=1.26",
|
||||
"pandas>=2.2",
|
||||
"pyarrow>=16",
|
||||
"typer>=0.12",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
@@ -16,6 +17,9 @@ dev = [
|
||||
"pytest>=8",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
giant = "giant.cli:app"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
Reference in New Issue
Block a user