dwarf warm-cache silently under-warms any config that sets particle_type.n_classes #59

New Issue

2026-08-14T17:49:37+02:00

lars commented

2026-08-14 17:49:37 +02:00

dwarf warm-cache builds its config from DEFAULT_CONFIG with only the conditioning and router axes overridable (giant/tools/warm_setup_cache.py), so it has no way to express stage2_model.particle_type.n_classes. Under the default particle_type.target = "onehot" it therefore always resolves resolve_type_n_classes to conditioning.particle.emb_dim (16) and warms the species top-N map under topn_key("pdg", 16).

Any config that sets n_classes to something else — configs/baseline.toml sets 32 — then misses that entry and rebuilds it during giant train, which is a full scan of every parquet file (build_pdg_topn_map_from_files). That is the most expensive single piece of the setup stage, and it is exactly the work warm-cache exists to eliminate. Nothing fails and nothing warns; the run just quietly pays for the scan it was told it wouldn't.

Reproduced on a synthetic dataset:

after `dwarf warm-cache`     : ['pdg:16']
after baseline-config warm   : ['pdg:16', 'pdg:32']   # echoes "building pdg top-N map …"

The other cache keys (val_fraction, seed, both conditioning axes) happen to line up with warm-cache's defaults for this config, so n_classes is the only mismatch — but it is also the only one the CLI cannot be told about.

The fix is to let warm-cache take the same --config a training run takes and warm from the merged config, rather than re-deriving a partial one from flags. run_setup_stage already accepts the full merged v0.3 cfg, so this is a wrapper change:

cfg = gconfig.merge_cli_overrides(gconfig.DEFAULT_CONFIG, config_path, overrides)
gconfig.validate_config(cfg)
run_setup_stage(data, val_fraction=cfg["train"]["val_fraction"],
                seed=cfg["train"]["seed"], cfg=cfg, cache_setup=True)

Driving it that way is verifiably complete — a second pass over the same data is a clean hit on every section:

event index: cache hit (40 unique events)
vocabulary maps: cache hit
pdg top-N map: cache hit (32 classes)
normalizer: cache hit (key='valfrac=0.1_seed=0_pcond=physical_mcond=physical')

Worth keeping the existing flags working for the no-config case, and worth considering whether --val-fraction/--seed/--particle-conditioning/--material-conditioning should be rejected alongside --config rather than silently layered over it, since a mismatch there produces the same silent under-warm through the normalizer key instead of the top-N key.

`dwarf warm-cache` builds its config from `DEFAULT_CONFIG` with only the conditioning and router axes overridable (`giant/tools/warm_setup_cache.py`), so it has no way to express `stage2_model.particle_type.n_classes`. Under the default `particle_type.target = "onehot"` it therefore always resolves `resolve_type_n_classes` to `conditioning.particle.emb_dim` (16) and warms the species top-N map under `topn_key("pdg", 16)`. Any config that sets `n_classes` to something else — `configs/baseline.toml` sets 32 — then misses that entry and rebuilds it during `giant train`, which is a full scan of every parquet file (`build_pdg_topn_map_from_files`). That is the most expensive single piece of the setup stage, and it is exactly the work `warm-cache` exists to eliminate. Nothing fails and nothing warns; the run just quietly pays for the scan it was told it wouldn't. Reproduced on a synthetic dataset: ``` after `dwarf warm-cache` : ['pdg:16'] after baseline-config warm : ['pdg:16', 'pdg:32'] # echoes "building pdg top-N map …" ``` The other cache keys (`val_fraction`, `seed`, both conditioning axes) happen to line up with `warm-cache`'s defaults for this config, so `n_classes` is the only mismatch — but it is also the only one the CLI cannot be told about. The fix is to let `warm-cache` take the same `--config` a training run takes and warm from the merged config, rather than re-deriving a partial one from flags. `run_setup_stage` already accepts the full merged v0.3 cfg, so this is a wrapper change: ```python cfg = gconfig.merge_cli_overrides(gconfig.DEFAULT_CONFIG, config_path, overrides) gconfig.validate_config(cfg) run_setup_stage(data, val_fraction=cfg["train"]["val_fraction"], seed=cfg["train"]["seed"], cfg=cfg, cache_setup=True) ``` Driving it that way is verifiably complete — a second pass over the same data is a clean hit on every section: ``` event index: cache hit (40 unique events) vocabulary maps: cache hit pdg top-N map: cache hit (32 classes) normalizer: cache hit (key='valfrac=0.1_seed=0_pcond=physical_mcond=physical') ``` Worth keeping the existing flags working for the no-config case, and worth considering whether `--val-fraction`/`--seed`/`--particle-conditioning`/`--material-conditioning` should be rejected alongside `--config` rather than silently layered over it, since a mismatch there produces the same silent under-warm through the normalizer key instead of the top-N key.

lars added the bug label 2026-08-14 17:49:37 +02:00

lars commented

2026-08-17 08:54:26 +02:00

Fixed in d25dfc0 on fix/issue-59.

warm-cache now accepts --config and resolves everything run_setup_stage needs (val_fraction/seed, conditioning types, both stages' router, stage2_model.particle_type.n_classes, ...) from one gconfig.merge_cli_overrides + validate_config pass — the same pipeline giant train itself uses — instead of a hand-rolled partial override dict. Verified end to end against configs/baseline.toml: a second warm-cache --config configs/baseline.toml run now shows 'pdg top-N map: cache hit (... 32 classes)' instead of rebuilding.

Decision made during planning: when --config is given, the individual --val-fraction/--seed/--particle-conditioning/--material-conditioning/--router*/flags are rejected outright (not silently layered on top) — a hardcoded CLI default overriding an unset config value would reproduce the same silent-mismatch bug one level down. Also removed a hardcoded stage2_model.router/k_max override in the same function that was a no-op today but had the identical latent-clobber problem.

Adding validate_config to warm-cache's path surfaced that the pre-existing test_warm_cache_router_process_warms_proc_map test was exercising a router.type=process + conditioning.particle.type=physical combination that giant train's own validate_config already rejects as incompatible — fixed the test to pass --particle-conditioning embedding, which is what a real process-router run requires. No follow-up filed; this was a same-commit test correction, not a new bug.

Fixed in d25dfc0 on fix/issue-59. warm-cache now accepts --config and resolves everything run_setup_stage needs (val_fraction/seed, conditioning types, both stages' router, stage2_model.particle_type.n_classes, ...) from one gconfig.merge_cli_overrides + validate_config pass — the same pipeline giant train itself uses — instead of a hand-rolled partial override dict. Verified end to end against configs/baseline.toml: a second warm-cache --config configs/baseline.toml run now shows 'pdg top-N map: cache hit (... 32 classes)' instead of rebuilding. Decision made during planning: when --config is given, the individual --val-fraction/--seed/--particle-conditioning/--material-conditioning/--router*/flags are rejected outright (not silently layered on top) — a hardcoded CLI default overriding an unset config value would reproduce the same silent-mismatch bug one level down. Also removed a hardcoded stage2_model.router/k_max override in the same function that was a no-op today but had the identical latent-clobber problem. Adding validate_config to warm-cache's path surfaced that the pre-existing test_warm_cache_router_process_warms_proc_map test was exercising a router.type=process + conditioning.particle.type=physical combination that giant train's own validate_config already rejects as incompatible — fixed the test to pass --particle-conditioning embedding, which is what a real process-router run requires. No follow-up filed; this was a same-commit test correction, not a new bug.

lars closed this issue

2026-08-17 08:54:29 +02:00

lars referenced this issue from a commit

2026-08-17 09:15:52 +02:00

Let dwarf warm-cache take --config so it can't under-warm a config's cache keys (gitea #59)

lars referenced this issue

2026-08-17 09:16:04 +02:00

Let dwarf warm-cache take --config so it can't under-warm a config's cache keys (gitea #59) #60

lars referenced this issue from a commit

2026-08-17 09:24:45 +02:00

Merge pull request 'Let dwarf warm-cache take --config so it can't under-warm a config's cache keys (gitea #59)' (#60) from fix/issue-59 into master

Sign in to join this conversation.