Experiments · E123

Does reporting the best find from each niche, instead of the top five, reveal more?

Yes. It surfaced 18 compositions instead of 5 at no cost to the best find, but only one of the new ones qualified.

In the log: Promotion by niche instead of top-k

confirmedDate not stated in the log; it was written between the commit of 2026-09-16 04:53 and the first commit that contains it, 2026-09-16 05:56unclassified0 predictions · 1 result paragraphEXPERIMENTS.md lines 7320–7346, lines 7348–7377
exp E123 diagram
What E123 did and how it came out, drawn from this record and the files it names (book/assets/diagrams/exp/E123.svg).

Results

EXPERIMENTS.md · line 7348

E123 result: all three predictions confirmed — and the fix reveals diversity, it does not create it.

One converged CrossEntropy run, 600 proposals, both policies reading the identical trace:

policy       promoted   distinct niches   best find
top-5               5                 2   -81.0 meV  Ta0.44 Mo0.39 W0.10 V0.06
per-niche          18                18   -81.0 meV  Ta0.44 Mo0.39 W0.10 V0.06

Prediction 1 confirmed: the top five span two niches, and four of the five are the same Mo-Ta-W-Nb composition at slightly different mixing ratios. Prediction 2 confirmed: 18 promoted against the predicted 8, and 16 of them are absent from the top five entirely. Prediction 3 confirmed: the best find is identical under both, to the digit - the niche policy costs nothing at the top, because the global best is always the elite of its own cell.

The caveat the data forces, and it matters more than the result. Of the 16 compositions the niche policy surfaces, only one is below the -40 meV/atom qualifying threshold (Mo0.71 Nb0.24 at -25.0 is the next best, and does not qualify). The rest run from +16 to +181. The niche policy surfaces what the run found; a CrossEntropy run did not find much.

That is exactly E115's finding seen from the other end. The scalar arm's trace contains two useful niches because the scalar arm only ever visits two; MAP-Elites at 20,000 evaluations covers 86.9 per cent of the space and finds a reproducible Mo-Ti-Ta basin at -47 meV/atom that CrossEntropy never reaches. The reporting fix and the generator fix are complementary and neither is sufficient alone: top-k reporting would have hidden the Mo-Ti-Ta basin even if the archive had found it, and niche reporting over a mode-seeking trace surfaces mostly mediocre compositions.

So the promotion change is correct and it is not the win. The win requires the archive arm to be the one generating, which is a separate change to stage_b's default and has not been made.

The full record

This entry is written in 2 separate places in the log, shown here in log order.

EXPERIMENTS.md · lines 7320–7346

E123 — Promotion by niche instead of top-k

stage_b.py reports sorted(trace, key=-p)[:5] - the five highest-scoring compositions of a run. E114 and E116 measured what the generator does to a trace: element-mass entropy falls from 2.37 to 1.30 and distinct element supports from 72 to 19. A converged optimiser's top five are therefore five near-copies of one composition, and E115's reproducible Mo-Ti-Ta basin at -47 meV/atom - which the scalar arm never reaches - would not appear in a report of this shape even if the archive had found it.

This is a reporting change and nothing else. Both policies read the same trace, so no new sampling is involved and the comparison is exact rather than statistical. The niche is the MAP-Elites cell already defined in forager.search.generate.MapElites: valence electron concentration against atomic size mismatch.

Predicted, on one stage_b run, comparing the two policies over the identical trace:

  1. The top-5 policy returns compositions from at most 2 distinct niches, because the generator concentrates and the five best are near-copies.
  2. One-elite-per-niche returns at least 8 distinct compositions, and at least one of them is absent from the top 5 entirely - that is the whole claim: the report is currently hiding finds the run already made.
  3. The single best composition is identical under both policies. A niche policy must not cost the best find; if it does, it is not a reporting change but a worse ranking.

Falsified if the top-5 already spans many niches, which would mean the trace is more diverse than E114/E116 measured and the reporting was never the bottleneck - in which case the promotion problem is upstream, in what the generator proposes, not in what gets reported.

EXPERIMENTS.md · lines 7348–7377

E123 result: all three predictions confirmed — and the fix reveals diversity, it does not create it.

One converged CrossEntropy run, 600 proposals, both policies reading the identical trace:

policy       promoted   distinct niches   best find
top-5               5                 2   -81.0 meV  Ta0.44 Mo0.39 W0.10 V0.06
per-niche          18                18   -81.0 meV  Ta0.44 Mo0.39 W0.10 V0.06

Prediction 1 confirmed: the top five span two niches, and four of the five are the same Mo-Ta-W-Nb composition at slightly different mixing ratios. Prediction 2 confirmed: 18 promoted against the predicted 8, and 16 of them are absent from the top five entirely. Prediction 3 confirmed: the best find is identical under both, to the digit - the niche policy costs nothing at the top, because the global best is always the elite of its own cell.

The caveat the data forces, and it matters more than the result. Of the 16 compositions the niche policy surfaces, only one is below the -40 meV/atom qualifying threshold (Mo0.71 Nb0.24 at -25.0 is the next best, and does not qualify). The rest run from +16 to +181. The niche policy surfaces what the run found; a CrossEntropy run did not find much.

That is exactly E115's finding seen from the other end. The scalar arm's trace contains two useful niches because the scalar arm only ever visits two; MAP-Elites at 20,000 evaluations covers 86.9 per cent of the space and finds a reproducible Mo-Ti-Ta basin at -47 meV/atom that CrossEntropy never reaches. The reporting fix and the generator fix are complementary and neither is sufficient alone: top-k reporting would have hidden the Mo-Ti-Ta basin even if the archive had found it, and niche reporting over a mode-seeking trace surfaces mostly mediocre compositions.

So the promotion change is correct and it is not the win. The win requires the archive arm to be the one generating, which is a separate change to stage_b's default and has not been made.

Related entries

Built with PRISMWebsite and visualizations made using Claude