13 Commits

Author	SHA1	Message	Date
lars	51790d3e0a	feat(predict): enrich YAML sidecar with provenance and timing ... `giant predict`'s sidecar previously stopped at kind/prediction_id/ output/dataset/checkpoint/timestamp, unlike `giant rollout`'s, which carries full run provenance (model_config, training_epoch, training_config, timing, ...) that flows into analysis gallery metadata. `analyze --prediction` consumed the same thin sidecar, so a prediction series in an analysis run was nearly unlabeled compared to its rollout counterparts. - `_write_prediction_ref` takes an `extra: dict | None` merged into the sidecar; `giant rollout` now uses it instead of a load/update/rewrite round trip (identical output). - New `_build_predict_timing`, key-compatible with `_build_rollout_timing`, from timers now wrapping predict's setup/ sample/write phases. - `giant predict` writes coord, has_truth, schema_version, steps, weights, device, batch_size(+auto), row/skip/unknown-pdg counts, timing, and the checkpoint's model_config/config_overrides/ training_epoch/best_val_loss/training_config/training_meta. - `giant/analysis/condor.py`'s `_PLOT_META_KEYS` forwards the new predict-only keys (plus rollout's previously-unforwarded config_overrides) into each plot's gallery metadata.yaml. - Fixes a `ty` regression from the prior commit in tests/test_cli_predict.py (Command has no static `.commands`). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PpxE9nij3ujg9XcuzvQ26q	2026-09-07 11:52:45 +02:00
lars	51f9dad3b0	fix(tests): make predict --truth flag test robust to terminal rendering ... The --help-text assertion was brittle to CI's terminal width/color settings (rich can wrap or re-color the flag name mid-word), causing a false CI failure even though the flag itself is fine. Inspect the click command's registered option directly instead of parsing rendered --help output. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PpxE9nij3ujg9XcuzvQ26q	2026-09-07 11:35:34 +02:00
lars	ac01966a1f	feat(analyze): add paired truth/pred plots from giant predict ... Adds a `prediction` plot family to `giant analyze`, alongside the existing rollout-vs-reference comparison, and extends `giant predict` to make it possible: - `giant predict --coord global` gains schema v3 (`--truth/--no-truth`, default on): writes true_* physical columns and true secondary lists alongside the predictions, so the output is fully paired. - New `giant/analysis/prediction.py` builds one canonical true/pred frame (`paired_frame`) from either predict coord mode. - `catalog.py` gains 35 `pred_*` specs: marginals, 2D truth-vs-pred scatter (new `heatmap2d` kind), residuals/relative-residuals/calibration profiles, KS/bias/RMSE scorecards, n_sec + secondary-species confusion matrices, direction-alignment and constraint-violation checks, and a correlation delta. Two new Reduced kinds (`paired_hist`, `heatmap2d`) get renderers. Every spec degrades to kind="unavailable" with no --prediction given. - `condor.py`/`cli.py`: `--prediction`/`--prediction-label` on `analyze prep`/`submit`, threaded through RunMeta and every compute job. Full test suite (1162 tests), ruff, and ty all pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PpxE9nij3ujg9XcuzvQ26q	2026-09-07 11:19:19 +02:00
lars	1ec333ff6d	Merge remote-tracking branch 'origin/master' into fix/issue-87 ... # Conflicts: # giant/config.py	2026-08-28 11:31:05 +02:00
lars	bd255419e1	Add inference-time model_config overrides with a sampling-key allowlist (gitea #87) ... giant predict/rollout rebuilt models straight from ckpt["model_config"] with no way to change sampling-only keys (e.g. stage2_model.n_sec.stop_sampling) without retraining. Adds config_overrides to load_for_inference, validated against giant.config.INFERENCE_OVERRIDES so a typo or shape-bearing key raises CheckpointCompatibilityError up front instead of an opaque load_state_dict mismatch. Wired as a repeatable --set dotted.path=value on both CLI commands, recorded in the rollout YAML sidecar, and surfaced in `giant model summary`'s output. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-28 11:21:36 +02:00
lars	818c380fd0	Extract predict/rollout's duplicated inference bootstrap into giant.checkpoint_io (issues.md Issue 5) ... giant predict and giant rollout each carried a ~65-line, independently drifting copy of "load checkpoint -> validate -> resolve conditioning axes -> restore normalizers/vocab maps -> build models -> load weights", plus a third partial copy of _conditioning_axes in analysis/router_gating.py. A silent divergence there doesn't crash, it makes the two commands run different physics from the same checkpoint with no test coverage anywhere along that path. giant/checkpoint_io.py now holds the single implementation: load_for_inference() + an InferenceContext dataclass, raising CheckpointCompatibilityError (verbatim message text preserved) instead of calling typer directly, so it can be unit-tested and imported from non-Typer code. router_gating.py's load_router imports conditioning_axes from it lazily, keeping its "no torch at module scope" contract intact. Adds 17 direct unit tests for load_for_inference/conditioning_axes/stage_cfg plus CLI smoke tests confirming the error surfaces as typer.Exit(1) through predict and rollout — previously zero coverage on this path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 15:31:14 +02:00
lars	55332db67a	Bump ruff line-length to 120 and reformat ... Rejoins lines that only wrapped because they exceeded the old 88-char limit; ruff check and the full test suite (725 passed) are unaffected.	2026-08-12 13:33:09 +02:00
lars	da7cde3ef9	v0.3.0 post-implementation audit: resolve all 9 tracked discrepancies ... Works through docs/v0.3.0-followups.md item by item, closing the gap between the design doc and the shipped v0.3.0-stage2-autoregressive code: 1. validate.py: 7-tuple batch unpacking, sample_stage1/sample_stage2 dispatch, stage-2 particle-type-class marginal. 2. Stage-prefixed --stage1-*/--stage2-* CLI flags for train/new-run. 3. Thread stage2_model.k_max through loader/transforms/dataset/pipeline/ train instead of the hardcoded K_MAX constant. 4. Mixed conditioning.particle.type / conditioning.material.type support end-to-end (data pipeline + dwarf warm-cache). 5. conditioning.share_stages = true: one shared ConditionEncoder instance across both stages. 6. stage2_model.generator = "ddpm" formally deferred into design doc §11.2 (was silently unimplemented). 7. giant predict/rollout: implement conditioning.*.type = "onehot" via the checkpoint's saved pdg_topn_map/mat_topn_map. 8. network.py's checkpoint-path model_config migration now fails loudly on non-zero legacy expert_hidden_dim/expert_n_blocks, matching config.py's TOML-load path (§4.2). 9. validate_config now rejects stage2_model.n_sec.mode = "truth" for a rollout-capable checkpoint (§9). Also cleared all pre-existing `ty check` noise (44 -> 0 diagnostics), mostly a test-helper dict-unpack pattern that made every unrelated constructor keyword look like a type error. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 16:12:58 +02:00
lars	93b19911f8	v0.3.0 step 6: sample.py/rollout.py AR generation + class->PDG decode ... - giant/sample.py: fix every sampler's call convention against Stage1Model/Stage2OneShot's actual forward signatures (was still calling model(x, t, cond_cont, cond_cat) positionally); add sample_secondaries_ar (free-running AR loop, unsnapped history feature) and sample_stage1/sample_stage2/resolve_n_sec dispatch helpers that read each stage's generator_kind/decoder off the model instance itself. - giant/particles.py: decode_topn_class (argmax + other_policy) and decode_embedding_nearest (L1-snap + distance) turn a secondary's "onehot"/"embedding" type prediction into a concrete PDG. - giant/rollout.py: decode_secondary_identity routes all three particle_type.target values to real mass/charge; per-stage generator dispatch (drops the single shared `mode` string, adds ddpm support); L1DistCollector accumulates the §11.3 embedding-distance diagnostic. - giant/cli.py: drop the onehot/embedding-target rejection gate (narrowed to the still-unimplemented conditioning.particle/material.type=onehot axis); fix the dead model_cfg.get("mode") bug in predict/rollout. - giant/analysis/: new type_embedding_l1_distance PlotSpec, wired through the rollout YAML sidecar (no live-model call needed, unlike router_gating -- the histogram is already pre-aggregated at rollout time). - Un-xfail every test that was blocked on this step (test_rollout.py, test_flow.py, test_wgan.py, test_phase2.py, test_router.py, test_validate.py); add test_sample.py, test_type_embedding_distance.py. Known follow-up: giant/validate.py still unpacks the training val-batch as a stale 6-tuple and doesn't use the new per-stage dispatch, so marginal validation during training degrades gracefully with a warning rather than working -- not in this step's scope. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 10:37:57 +02:00
lars	4fc15ecdfc	v0.3.0 step 4: type map + particle_type.target = "onehot"/"embedding" ... Builds the shared top-N-plus-other PDG/material maps (pooling both primary and secondary occurrences for PDG, directly targeting the meeting's species-collapse failure mode) and wires up conditioning.{particle,material} = "onehot" plus stage2_model.particle_type.target in ("onehot", "embedding") end-to-end: setup-cache persistence, Stage2OneShot's type_head (flow/ddpm) vs. folded+ST-Gumbel-relaxed adversarial slice (wgan), and the corresponding CE/MSE training losses. particle_type.target = "physical" stays byte-for-byte unchanged, keeping the v0.2 migration shim's bit-identical guarantee intact. giant predict/rollout fail loudly on a onehot/embedding checkpoint until full decode support lands in step 6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 15:43:48 +02:00
lars	f0cb41477f	Add --comment option to predict, recorded in YAML sidecar ... Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-06 09:37:48 +02:00
lars	25718f175e	Fix ruff, ty, and pytest failures; apply ruff format ... Removes unused imports and an ambiguous variable name, narrows Optional types before use so ty's flow analysis is satisfied, swaps sum() over polars expressions for pl.sum_horizontal to avoid the Literal[0] fallback type, and converts numpy bin edges to plain lists before passing to matplotlib's hist (whose stub only accepts Sequence[float]). Also applies ruff format across the repo, which had drifted out of sync with the formatter. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-02 16:59:22 +02:00
lars	c627142135	Route predict output to UUID-named parquet with YAML reference sidecar ... When no --out is given, predictions are written to a UUID-named parquet (/ceph central store for ceph inputs, sibling directory otherwise) and a YAML reference file is created next to the checkpoint recording the output path, dataset, checkpoint, and timestamp. Adds pyyaml as a core dependency and unit tests for the two new helper functions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-29 09:57:14 +02:00