The per-dim print loop lacked the empty-array guard already used for
the KL computation right above it and the sec-slot loop further down,
so an all-zero-secondaries validation batch (e.g. early/unstable
training) triggered numpy RuntimeWarnings from .mean()/.std() on
empty arrays.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Covers the default-config 0-sentinel inheritance path (the exact bug
fixed by 969c5c6, previously untested since every router test in
test_router.py passes expert_hidden_dim/expert_n_blocks explicitly),
plus missing-key inheritance, full override, and partial override.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
out_dir (and thus the W&B run id, which is derived from out_dir.name)
was previously date-only, so two fresh runs on the same day with
identical hyperparams silently shared one W&B run history. Default
out_dir is now timestamped to the second. --resume without an explicit
--out now reuses the checkpoint's own parent directory instead of
recomputing a hyperparam-derived name, which both preserves the old
continue-in-place behavior and fixes a latent bug where a resumed run
with a changed hyperparam (e.g. --lr) would silently start writing to
a new directory.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously global_step always reset to 0, even on --resume. Since the
W&B run reattaches to the same run id on resume, logging with
step=global_step after a restart passed step values below what was
already recorded, silently dropping the resumed portion's metrics.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds Router.gate_stats (per-router gate entropy + per-expert utilization),
logged both per-batch (entropy only, train loop) and per-epoch (full
stats, over the whole val set) — the router-collapse failure mode from
the roadmap's rollout postmortem is now visible during training instead
of only after a full rollout+analysis run. Also splits WGAN critic/
generator grad norms instead of summing them, logs critic LR, n_sec head
accuracy, GPU peak memory + samples/sec, model parameter counts (in
wandb.config), and an is_best flag — all wired into both metrics.csv and
W&B.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
expert_hidden_dim/expert_n_blocks were hardcoded to 128/3 in
DEFAULT_CONFIG, independent of model.hidden_dim/n_blocks, so a routed
run always got fixed 128/3-wide experts no matter what --hidden-dim/
--n-blocks was passed. They now default to 0 ("unset"), which
resolve_expert_dims() resolves by inheriting the model dims; an
explicit override still works and now warns when it diverges from
model.hidden_dim/n_blocks, since the checkpoint dir name won't
reflect it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant train --wandb now also logs loss/grad_norm/lr every N optimizer
steps (--wandb-log-every, default 50) so W&B shows within-epoch trends,
not just one point per epoch. Both share global_step as a single
monotonic step axis (wandb.Run.log requires step to never decrease
across calls), which also fixes global_step previously only advancing
in wgan mode.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant train --wandb logs the same per-epoch metrics already written to
metrics.csv, so wandb stays an optional extra (`uv sync --extra wandb`)
that nothing else depends on. A run's id is derived from the checkpoint
out_dir so --resume reattaches to the existing run instead of starting
a new one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lars