Should the diversity-seeking search become the default at the usual budget of 240 evaluations?
No. It samples at random for its first 600 evaluations, so at 240 it never starts; its best was +129 against −48 meV/atom.
In the log: Should the archive arm be the default generator?
confirmedDate not stated in the log; it was written between the commit of 2026-09-16 05:56 and the first commit that contains it, 2026-09-16 06:55search baselines0 predictions · 1 result paragraphEXPERIMENTS.md lines 7379–7408, lines 7410–7444
What E124 did and how it came out, drawn from this record and the files it names (book/assets/diagrams/exp/E124.svg).
Results
EXPERIMENTS.md · line 7410
E124 result: all three predictions confirmed. The archive must NOT be the default at this
budget, and the reason is structural rather than statistical.
MapElites is now registered as the archive arm. Three arms, three seeds, stage_b's
default 60 rounds x 4 proposals = 240 evaluations per run:
arm AUC_Q distinct found best meV/atom
uniform 0.00 +- 0.00 0.00 +- 0.00 +132
cem 0.36 +- 0.29 2.67 +- 1.76 -48
archive 0.00 +- 0.00 0.00 +- 0.00 +129
Prediction 1 confirmed, and more strongly than it was stated. The archive arm does not
merely score within the seed spread of uniform - it scores identically to two decimals on
both AUC_Q and distinct-found, because at this budget it is uniform sampling. Verified
directly rather than inferred: MapElites.init is 600, the run gives 240, and instrumenting
the arm shows the archive is consulted zero times in 240 evaluations.
Prediction 2 confirmed: CrossEntropy takes AUC_Q 0.36 and a best of -48 meV/atom
against the archive arm's +129 - a gap of 177 meV/atom, and the difference between
qualifying and not qualifying at all.
Prediction 3 confirmed. The default is unchanged.E115's result stands and is not
contradicted here: it was measured at 20,000 evaluations per arm, eighty times this budget.
What E124 establishes is that the result does not transfer to the budget stage_b runs at,
and that switching the default would have been a 177 meV/atom regression presented as an
improvement.
What was changed instead of the default.MapElites.viable_budget reports the fewest
evaluations at which the arm is doing what its name says - init plus four per reachable
cell, 1080 on the twelve-element space - and stage_b now prints a warning when the
archive arm is run below it, saying in as many words that its score is not a verdict on
quality-diversity. An arm that silently degrades into a different algorithm is the kind of
thing that produces a confident wrong conclusion, which is what E114 was.
229 passing.
The full record
This entry is written in 2 separate places in the log, shown here in log order.
EXPERIMENTS.md · lines 7379–7408
E124 — Should the archive arm be the default generator?
E115 showed MAP-Elites matching CrossEntropy's best to 0.5 meV/atom while covering 86.9 per
cent of the behaviour space against 18.1, and finding a Mo-Ti-Ta basin at -47 meV/atom in all
three seeds that the scalar arm never reaches. E123 made the reporting able to show such a
basin. The obvious next step is to make the archive the default arm. It is not registered as
an arm in stage_b at all.
Before switching anything, the budget.E115 ran 20,000 evaluations per arm.
stage_b's default is 60 rounds x 4 proposals = 240 evaluations per run, eighty times
smaller. And MapElites.init is 600: the arm samples the simplex uniformly until it holds
that many observations before the archive is ever consulted.
240 is less than 600. So at stage_b's default budget the archive arm never leaves its
seeding phase, and is not MAP-Elites at all - it is uniform random sampling wearing the name.
Predicted:
At the default budget the archive arm scores statistically indistinguishably from
uniform, because it is uniform there. Their AUC_Q means should differ by less than
the seed-to-seed spread of either.
CrossEntropy beats both on AUC_Q, because AUC_Q rewards qualifying quickly and a
converged Gaussian on a unimodal landscape does that better than random sampling.
So the archive arm must NOT become the default at this budget. The honest conclusion
would be that E115's result is real but lives at a budget stage_b does not run at, and
that switching the default would be a regression dressed as an improvement.
Falsified if the archive arm is competitive on AUC_Q at 240 evaluations, which would mean
either that init is not doing what it appears to, or that AUC_Q is insensitive to the
difference, and either of those needs understanding before any default is changed.
EXPERIMENTS.md · lines 7410–7444
E124 result: all three predictions confirmed. The archive must NOT be the default at this
budget, and the reason is structural rather than statistical.
MapElites is now registered as the archive arm. Three arms, three seeds, stage_b's
default 60 rounds x 4 proposals = 240 evaluations per run:
arm AUC_Q distinct found best meV/atom
uniform 0.00 +- 0.00 0.00 +- 0.00 +132
cem 0.36 +- 0.29 2.67 +- 1.76 -48
archive 0.00 +- 0.00 0.00 +- 0.00 +129
Prediction 1 confirmed, and more strongly than it was stated. The archive arm does not
merely score within the seed spread of uniform - it scores identically to two decimals on
both AUC_Q and distinct-found, because at this budget it is uniform sampling. Verified
directly rather than inferred: MapElites.init is 600, the run gives 240, and instrumenting
the arm shows the archive is consulted zero times in 240 evaluations.
Prediction 2 confirmed: CrossEntropy takes AUC_Q 0.36 and a best of -48 meV/atom
against the archive arm's +129 - a gap of 177 meV/atom, and the difference between
qualifying and not qualifying at all.
Prediction 3 confirmed. The default is unchanged.E115's result stands and is not
contradicted here: it was measured at 20,000 evaluations per arm, eighty times this budget.
What E124 establishes is that the result does not transfer to the budget stage_b runs at,
and that switching the default would have been a 177 meV/atom regression presented as an
improvement.
What was changed instead of the default.MapElites.viable_budget reports the fewest
evaluations at which the arm is doing what its name says - init plus four per reachable
cell, 1080 on the twelve-element space - and stage_b now prints a warning when the
archive arm is run below it, saying in as many words that its score is not a verdict on
quality-diversity. An arm that silently degrades into a different algorithm is the kind of
thing that produces a confident wrong conclusion, which is what E114 was.
229 passing.
Related entries
E115 — The same head-to-head, with the implementation defects fixed