Feature/wandb integration #18

Merged

lars merged 10 commits from feature/wandb-integration into master 2026-07-29 10:52:52 +02:00

Conversation 0 Commits 10 Files Changed 13

+826 -78

lars commented 2026-07-27 16:13:14 +02:00

Owner

Copy Link

Copy Source

No description provided.

lars added 3 commits 2026-07-27 16:13:14 +02:00

Add configs for router energy (embedding/physical) and WGAN baseline runs ... db0f12be58

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Add opt-in Weights & Biases logging for the training loop ... 0778a61360

giant train --wandb logs the same per-epoch metrics already written to
metrics.csv, so wandb stays an optional extra (`uv sync --extra wandb`)
that nothing else depends on. A run's id is derived from the checkpoint
out_dir so --resume reattaches to the existing run instead of starting
a new one.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Apply ruff format ... a05837f918

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars added 1 commit 2026-07-27 16:20:07 +02:00

Log batch-level metrics to W&B, not just per-epoch summaries ... 29459ab1f7

giant train --wandb now also logs loss/grad_norm/lr every N optimizer
steps (--wandb-log-every, default 50) so W&B shows within-epoch trends,
not just one point per epoch. Both share global_step as a single
monotonic step axis (wandb.Run.log requires step to never decrease
across calls), which also fixes global_step previously only advancing
in wgan mode.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars added 1 commit 2026-07-27 16:34:29 +02:00

Fix router experts silently ignoring --hidden-dim/--n-blocks ... 969c5c6e9a

expert_hidden_dim/expert_n_blocks were hardcoded to 128/3 in
DEFAULT_CONFIG, independent of model.hidden_dim/n_blocks, so a routed
run always got fixed 128/3-wide experts no matter what --hidden-dim/
--n-blocks was passed. They now default to 0 ("unset"), which
resolve_expert_dims() resolves by inheriting the model dims; an
explicit override still works and now warns when it diverges from
model.hidden_dim/n_blocks, since the checkpoint dir name won't
reflect it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars added 1 commit 2026-07-29 10:21:07 +02:00

Log router health, WGAN grad-norm split, n_sec accuracy, GPU/throughput to W&B ... a986f96ba3

Adds Router.gate_stats (per-router gate entropy + per-expert utilization),
logged both per-batch (entropy only, train loop) and per-epoch (full
stats, over the whole val set) — the router-collapse failure mode from
the roadmap's rollout postmortem is now visible during training instead
of only after a full rollout+analysis run. Also splits WGAN critic/
generator grad norms instead of summing them, logs critic LR, n_sec head
accuracy, GPU peak memory + samples/sec, model parameter counts (in
wandb.config), and an is_best flag — all wired into both metrics.csv and
W&B.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars added 3 commits 2026-07-29 10:42:47 +02:00

Persist global_step across --resume so W&B step stays monotonic ... bb699d41b2

Previously global_step always reset to 0, even on --resume. Since the
W&B run reattaches to the same run id on resume, logging with
step=global_step after a restart passed step values below what was
already recorded, silently dropping the resumed portion's metrics.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Timestamp default checkpoint dir to avoid W&B run-id collisions ... f427d3384f

out_dir (and thus the W&B run id, which is derived from out_dir.name)
was previously date-only, so two fresh runs on the same day with
identical hyperparams silently shared one W&B run history. Default
out_dir is now timestamped to the second. --resume without an explicit
--out now reuses the checkpoint's own parent directory instead of
recomputing a hyperparam-derived name, which both preserves the old
continue-in-place behavior and fixes a latent bug where a resumed run
with a changed hyperparam (e.g. --lr) would silently start writing to
a new directory.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Add test coverage for resolve_expert_dims ... 539b6f61e1

Covers the default-config 0-sentinel inheritance path (the exact bug
fixed by 969c5c6, previously untested since every router test in
test_router.py passes expert_hidden_dim/expert_n_blocks explicitly),
plus missing-key inheritance, full override, and partial override.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars added 1 commit 2026-07-29 10:49:15 +02:00

Skip empty-slice mean/std in sec phys validation print ... 4c19072724

The per-dim print loop lacked the empty-array guard already used for
the KL computation right above it and the sec-slot loop further down,
so an all-zero-secondaries validation batch (e.g. early/unstable
training) triggered numpy RuntimeWarnings from .mean()/.std() on
empty arrays.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars marked the pull request as ready for review 2026-07-29 10:52:21 +02:00

lars merged commit f2f89023d5 into master 2026-07-29 10:52:52 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

architecture

experiment

investigation

performance

phase2

validation

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

lars (Lars Bogner)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lars/giant#18