feat(analyze): add paired truth/pred plots from giant predict #98

Merged

lars merged 3 commits from feature/predict-analysis-plots into master 2026-09-07 11:57:37 +02:00

Conversation 0 Commits 3 Files Changed 15

+2425 -67

lars commented 2026-09-07 11:21:14 +02:00

Owner

Copy Link

Copy Source

Adds a prediction plot family to giant analyze, alongside the existing
rollout-vs-reference comparison, and extends giant predict to make it
possible:

• giant predict --coord global gains schema v3 (--truth/--no-truth,
  default on): writes true_* physical columns and true secondary lists
  alongside the predictions, so the output is fully paired.
• New giant/analysis/prediction.py builds one canonical true/pred frame
  (paired_frame) from either predict coord mode.
• catalog.py gains 35 pred_* specs: marginals, 2D truth-vs-pred scatter
  (new heatmap2d kind), residuals/relative-residuals/calibration profiles,
  KS/bias/RMSE scorecards, n_sec + secondary-species confusion matrices,
  direction-alignment and constraint-violation checks, and a correlation
  delta. Two new Reduced kinds (paired_hist, heatmap2d) get renderers.
  Every spec degrades to kind="unavailable" with no --prediction given.
• condor.py/cli.py: --prediction/--prediction-label on
  analyze prep/submit, threaded through RunMeta and every compute job.

Full test suite (1162 tests), ruff, and ty all pass.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01PpxE9nij3ujg9XcuzvQ26q

Adds a `prediction` plot family to `giant analyze`, alongside the existing rollout-vs-reference comparison, and extends `giant predict` to make it possible: - `giant predict --coord global` gains schema v3 (`--truth/--no-truth`, default on): writes true_* physical columns and true secondary lists alongside the predictions, so the output is fully paired. - New `giant/analysis/prediction.py` builds one canonical true/pred frame (`paired_frame`) from either predict coord mode. - `catalog.py` gains 35 `pred_*` specs: marginals, 2D truth-vs-pred scatter (new `heatmap2d` kind), residuals/relative-residuals/calibration profiles, KS/bias/RMSE scorecards, n_sec + secondary-species confusion matrices, direction-alignment and constraint-violation checks, and a correlation delta. Two new Reduced kinds (`paired_hist`, `heatmap2d`) get renderers. Every spec degrades to kind="unavailable" with no --prediction given. - `condor.py`/`cli.py`: `--prediction`/`--prediction-label` on `analyze prep`/`submit`, threaded through RunMeta and every compute job. Full test suite (1162 tests), ruff, and ty all pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PpxE9nij3ujg9XcuzvQ26q

lars added 1 commit 2026-09-07 11:21:15 +02:00

feat(analyze): add paired truth/pred plots from giant predict ... ac01966a1f

Adds a `prediction` plot family to `giant analyze`, alongside the existing
rollout-vs-reference comparison, and extends `giant predict` to make it
possible:

- `giant predict --coord global` gains schema v3 (`--truth/--no-truth`,
  default on): writes true_* physical columns and true secondary lists
  alongside the predictions, so the output is fully paired.
- New `giant/analysis/prediction.py` builds one canonical true/pred frame
  (`paired_frame`) from either predict coord mode.
- `catalog.py` gains 35 `pred_*` specs: marginals, 2D truth-vs-pred scatter
  (new `heatmap2d` kind), residuals/relative-residuals/calibration profiles,
  KS/bias/RMSE scorecards, n_sec + secondary-species confusion matrices,
  direction-alignment and constraint-violation checks, and a correlation
  delta. Two new Reduced kinds (`paired_hist`, `heatmap2d`) get renderers.
  Every spec degrades to kind="unavailable" with no --prediction given.
- `condor.py`/`cli.py`: `--prediction`/`--prediction-label` on
  `analyze prep`/`submit`, threaded through RunMeta and every compute job.

Full test suite (1162 tests), ruff, and ty all pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PpxE9nij3ujg9XcuzvQ26q

lars added 1 commit 2026-09-07 11:35:41 +02:00

fix(tests): make predict --truth flag test robust to terminal rendering ... 51f9dad3b0

The --help-text assertion was brittle to CI's terminal width/color
settings (rich can wrap or re-color the flag name mid-word), causing a
false CI failure even though the flag itself is fine. Inspect the
click command's registered option directly instead of parsing rendered
--help output.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PpxE9nij3ujg9XcuzvQ26q

lars added 1 commit 2026-09-07 11:52:49 +02:00

feat(predict): enrich YAML sidecar with provenance and timing ... 51790d3e0a

`giant predict`'s sidecar previously stopped at kind/prediction_id/
output/dataset/checkpoint/timestamp, unlike `giant rollout`'s, which
carries full run provenance (model_config, training_epoch,
training_config, timing, ...) that flows into analysis gallery
metadata. `analyze --prediction` consumed the same thin sidecar, so a
prediction series in an analysis run was nearly unlabeled compared to
its rollout counterparts.

- `_write_prediction_ref` takes an `extra: dict | None` merged into
  the sidecar; `giant rollout` now uses it instead of a
  load/update/rewrite round trip (identical output).
- New `_build_predict_timing`, key-compatible with
  `_build_rollout_timing`, from timers now wrapping predict's setup/
  sample/write phases.
- `giant predict` writes coord, has_truth, schema_version, steps,
  weights, device, batch_size(+auto), row/skip/unknown-pdg counts,
  timing, and the checkpoint's model_config/config_overrides/
  training_epoch/best_val_loss/training_config/training_meta.
- `giant/analysis/condor.py`'s `_PLOT_META_KEYS` forwards the new
  predict-only keys (plus rollout's previously-unforwarded
  config_overrides) into each plot's gallery metadata.yaml.
- Fixes a `ty` regression from the prior commit in
  tests/test_cli_predict.py (Command has no static `.commands`).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PpxE9nij3ujg9XcuzvQ26q

lars merged commit 48208e6d18 into master 2026-09-07 11:57:37 +02:00

lars deleted branch feature/predict-analysis-plots 2026-09-07 11:57:43 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

architecture

bug

Silent/wrong behavior, no crash

experiment

feature

New feature matching the v0.3 redesign

investigation

modularity

Making the network genuinely pluggable

performance

phase2

validation

wontfix

Not being fixed right now

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

lars (Lars Bogner)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lars/giant#98