v0.3.0 post-implementation audit: resolve all 9 tracked discrepancies
CI / Lint (ruff check) (push) Successful in 27s
CI / Format (ruff format) (push) Successful in 28s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 36s
CI / Type check (ty) (push) Successful in 39s
CI / Format (ruff format) (pull_request) Successful in 30s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 30s
CI / Tests (pull_request) Successful in 2m50s
CI / Tests (push) Successful in 2m58s

Works through docs/v0.3.0-followups.md item by item, closing the gap
between the design doc and the shipped v0.3.0-stage2-autoregressive code:

1. validate.py: 7-tuple batch unpacking, sample_stage1/sample_stage2
   dispatch, stage-2 particle-type-class marginal.
2. Stage-prefixed --stage1-*/--stage2-* CLI flags for train/new-run.
3. Thread stage2_model.k_max through loader/transforms/dataset/pipeline/
   train instead of the hardcoded K_MAX constant.
4. Mixed conditioning.particle.type / conditioning.material.type support
   end-to-end (data pipeline + dwarf warm-cache).
5. conditioning.share_stages = true: one shared ConditionEncoder instance
   across both stages.
6. stage2_model.generator = "ddpm" formally deferred into design doc §11.2
   (was silently unimplemented).
7. giant predict/rollout: implement conditioning.*.type = "onehot" via the
   checkpoint's saved pdg_topn_map/mat_topn_map.
8. network.py's checkpoint-path model_config migration now fails loudly on
   non-zero legacy expert_hidden_dim/expert_n_blocks, matching config.py's
   TOML-load path (§4.2).
9. validate_config now rejects stage2_model.n_sec.mode = "truth" for a
   rollout-capable checkpoint (§9).

Also cleared all pre-existing `ty check` noise (44 -> 0 diagnostics),
mostly a test-helper dict-unpack pattern that made every unrelated
constructor keyword look like a type error.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-07 16:12:58 +02:00
parent 200c6d243b
commit da7cde3ef9
31 changed files with 1536 additions and 499 deletions
+19 -11
View File
@@ -6,6 +6,8 @@ import numpy as np
import pandas as pd
import pyarrow.parquet as pq
from giant.constants import K_MAX
# A manifest is a plain text file listing one parquet path per line, used to
# name a curated subset of files (e.g. a train/holdout pool) without copying
# or symlinking the underlying parquet files. Lines are resolved relative to
@@ -113,9 +115,9 @@ def _pad_dir_col(dx: pd.Series, dy: pd.Series, dz: pd.Series, K: int) -> np.ndar
return out
def _df_to_dict(df: pd.DataFrame, offset: int = 0) -> dict[str, np.ndarray]:
from giant.constants import K_MAX
def _df_to_dict(
df: pd.DataFrame, offset: int = 0, k_max: int = K_MAX
) -> dict[str, np.ndarray]:
has_sec_lists = "sec_E_list" in df.columns
d: dict[str, np.ndarray] = {
@@ -147,17 +149,19 @@ def _df_to_dict(df: pd.DataFrame, offset: int = 0) -> dict[str, np.ndarray]:
}
if has_sec_lists:
d["sec_E_list"] = _pad_list_col(df["sec_E_list"], K_MAX)
d["sec_pdg_list"] = _pad_list_col_int(df["sec_pdg_list"], K_MAX)
d["sec_E_list"] = _pad_list_col(df["sec_E_list"], k_max)
d["sec_pdg_list"] = _pad_list_col_int(df["sec_pdg_list"], k_max)
d["sec_dir_list"] = _pad_dir_col(
df["sec_dx_list"], df["sec_dy_list"], df["sec_dz_list"], K_MAX
df["sec_dx_list"], df["sec_dy_list"], df["sec_dz_list"], k_max
)
return d
def load_steps(path: str | Path, offset: int = 0) -> dict[str, np.ndarray]:
return _df_to_dict(pd.read_parquet(path), offset=offset)
def load_steps(
path: str | Path, offset: int = 0, k_max: int = K_MAX
) -> dict[str, np.ndarray]:
return _df_to_dict(pd.read_parquet(path), offset=offset, k_max=k_max)
def load_event_ids(path: str | Path, offset: int = 0) -> np.ndarray:
@@ -167,12 +171,16 @@ def load_event_ids(path: str | Path, offset: int = 0) -> np.ndarray:
def iter_file_chunks(
path: str | Path, offset: int = 0
path: str | Path, offset: int = 0, k_max: int = K_MAX
) -> Iterator[dict[str, np.ndarray]]:
"""Yield one parquet row-group at a time so a large file never fully loads."""
"""Yield one parquet row-group at a time so a large file never fully loads.
`k_max` sets the padded width of the sec_*_list columns (should match
`stage2_model.k_max` — see docs/v0.3.0-design.md §8); defaults to the
module constant for callers that don't care (e.g. Stage-1-only reads)."""
pf = pq.ParquetFile(path)
for i in range(pf.num_row_groups):
yield _df_to_dict(pf.read_row_group(i).to_pandas(), offset=offset)
yield _df_to_dict(pf.read_row_group(i).to_pandas(), offset=offset, k_max=k_max)
_COND_COLS = [