- Graceful shutdown (SIGINT/SIGTERM) now actually saves a checkpoint of
in-progress weights before exiting mid-epoch — it previously broke
out of the epoch loop before reaching the checkpoint-save block,
contradicting its own printed "saving a checkpoint" message and
losing all progress since the last completed epoch. Checkpoint-dict
construction is factored into a shared _build_checkpoint() helper
used by both the mid-epoch and end-of-epoch save paths.
- WGAN LR-schedule steps_per_epoch used the wrong denominator
(n_critic + 1 instead of n_critic), causing the schedule to exhaust
early and LR to floor to 0 before training completed.
- --critic-lr override was silently dropped on WGAN --resume (only the
generator optimizer's LR was made authoritative again after
load_state_dict; optimizer_d's was not).
- WGAN secondary gradient-penalty forced x_hat/grad to zero for
fully-masked rows (n_sec == 0, common in a shower), adding a
constant ~1.0 bias into the batch-mean GP term; such rows are now
excluded from the mean.
- run_train_job warns (never blocks) when --num-workers exceeds ~1/4
of the machine's CPUs, per this repo's shared-portal-machine
etiquette (see CLAUDE.md's Compute environment section).
Each fix has a regression test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lars
ad0341a9d4