v0.3.0 post-implementation audit: resolve all 9 tracked discrepancies
CI / Lint (ruff check) (push) Successful in 27s
CI / Format (ruff format) (push) Successful in 28s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 36s
CI / Type check (ty) (push) Successful in 39s
CI / Format (ruff format) (pull_request) Successful in 30s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 30s
CI / Tests (pull_request) Successful in 2m50s
CI / Tests (push) Successful in 2m58s
CI / Lint (ruff check) (push) Successful in 27s
CI / Format (ruff format) (push) Successful in 28s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 36s
CI / Type check (ty) (push) Successful in 39s
CI / Format (ruff format) (pull_request) Successful in 30s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 30s
CI / Tests (pull_request) Successful in 2m50s
CI / Tests (push) Successful in 2m58s
Works through docs/v0.3.0-followups.md item by item, closing the gap between the design doc and the shipped v0.3.0-stage2-autoregressive code: 1. validate.py: 7-tuple batch unpacking, sample_stage1/sample_stage2 dispatch, stage-2 particle-type-class marginal. 2. Stage-prefixed --stage1-*/--stage2-* CLI flags for train/new-run. 3. Thread stage2_model.k_max through loader/transforms/dataset/pipeline/ train instead of the hardcoded K_MAX constant. 4. Mixed conditioning.particle.type / conditioning.material.type support end-to-end (data pipeline + dwarf warm-cache). 5. conditioning.share_stages = true: one shared ConditionEncoder instance across both stages. 6. stage2_model.generator = "ddpm" formally deferred into design doc §11.2 (was silently unimplemented). 7. giant predict/rollout: implement conditioning.*.type = "onehot" via the checkpoint's saved pdg_topn_map/mat_topn_map. 8. network.py's checkpoint-path model_config migration now fails loudly on non-zero legacy expert_hidden_dim/expert_n_blocks, matching config.py's TOML-load path (§4.2). 9. validate_config now rejects stage2_model.n_sec.mode = "truth" for a rollout-capable checkpoint (§9). Also cleared all pre-existing `ty check` noise (44 -> 0 diagnostics), mostly a test-helper dict-unpack pattern that made every unrelated constructor keyword look like a type error. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
+19
-11
@@ -6,6 +6,8 @@ import numpy as np
|
||||
import pandas as pd
|
||||
import pyarrow.parquet as pq
|
||||
|
||||
from giant.constants import K_MAX
|
||||
|
||||
# A manifest is a plain text file listing one parquet path per line, used to
|
||||
# name a curated subset of files (e.g. a train/holdout pool) without copying
|
||||
# or symlinking the underlying parquet files. Lines are resolved relative to
|
||||
@@ -113,9 +115,9 @@ def _pad_dir_col(dx: pd.Series, dy: pd.Series, dz: pd.Series, K: int) -> np.ndar
|
||||
return out
|
||||
|
||||
|
||||
def _df_to_dict(df: pd.DataFrame, offset: int = 0) -> dict[str, np.ndarray]:
|
||||
from giant.constants import K_MAX
|
||||
|
||||
def _df_to_dict(
|
||||
df: pd.DataFrame, offset: int = 0, k_max: int = K_MAX
|
||||
) -> dict[str, np.ndarray]:
|
||||
has_sec_lists = "sec_E_list" in df.columns
|
||||
|
||||
d: dict[str, np.ndarray] = {
|
||||
@@ -147,17 +149,19 @@ def _df_to_dict(df: pd.DataFrame, offset: int = 0) -> dict[str, np.ndarray]:
|
||||
}
|
||||
|
||||
if has_sec_lists:
|
||||
d["sec_E_list"] = _pad_list_col(df["sec_E_list"], K_MAX)
|
||||
d["sec_pdg_list"] = _pad_list_col_int(df["sec_pdg_list"], K_MAX)
|
||||
d["sec_E_list"] = _pad_list_col(df["sec_E_list"], k_max)
|
||||
d["sec_pdg_list"] = _pad_list_col_int(df["sec_pdg_list"], k_max)
|
||||
d["sec_dir_list"] = _pad_dir_col(
|
||||
df["sec_dx_list"], df["sec_dy_list"], df["sec_dz_list"], K_MAX
|
||||
df["sec_dx_list"], df["sec_dy_list"], df["sec_dz_list"], k_max
|
||||
)
|
||||
|
||||
return d
|
||||
|
||||
|
||||
def load_steps(path: str | Path, offset: int = 0) -> dict[str, np.ndarray]:
|
||||
return _df_to_dict(pd.read_parquet(path), offset=offset)
|
||||
def load_steps(
|
||||
path: str | Path, offset: int = 0, k_max: int = K_MAX
|
||||
) -> dict[str, np.ndarray]:
|
||||
return _df_to_dict(pd.read_parquet(path), offset=offset, k_max=k_max)
|
||||
|
||||
|
||||
def load_event_ids(path: str | Path, offset: int = 0) -> np.ndarray:
|
||||
@@ -167,12 +171,16 @@ def load_event_ids(path: str | Path, offset: int = 0) -> np.ndarray:
|
||||
|
||||
|
||||
def iter_file_chunks(
|
||||
path: str | Path, offset: int = 0
|
||||
path: str | Path, offset: int = 0, k_max: int = K_MAX
|
||||
) -> Iterator[dict[str, np.ndarray]]:
|
||||
"""Yield one parquet row-group at a time so a large file never fully loads."""
|
||||
"""Yield one parquet row-group at a time so a large file never fully loads.
|
||||
|
||||
`k_max` sets the padded width of the sec_*_list columns (should match
|
||||
`stage2_model.k_max` — see docs/v0.3.0-design.md §8); defaults to the
|
||||
module constant for callers that don't care (e.g. Stage-1-only reads)."""
|
||||
pf = pq.ParquetFile(path)
|
||||
for i in range(pf.num_row_groups):
|
||||
yield _df_to_dict(pf.read_row_group(i).to_pandas(), offset=offset)
|
||||
yield _df_to_dict(pf.read_row_group(i).to_pandas(), offset=offset, k_max=k_max)
|
||||
|
||||
|
||||
_COND_COLS = [
|
||||
|
||||
Reference in New Issue
Block a user