Experiments · E124

Should the diversity-seeking search become the default at the usual budget of 240 evaluations?

No. It samples at random for its first 600 evaluations, so at 240 it never starts; its best was +129 against −48 meV/atom.

In the log: Should the archive arm be the default generator?

confirmedDate not stated in the log; it was written between the commit of 2026-09-16 05:56 and the first commit that contains it, 2026-09-16 06:55search baselines0 predictions · 1 result paragraphEXPERIMENTS.md lines 7379–7408, lines 7410–7444
exp E124 diagram
What E124 did and how it came out, drawn from this record and the files it names (book/assets/diagrams/exp/E124.svg).

Results

EXPERIMENTS.md · line 7410

E124 result: all three predictions confirmed. The archive must NOT be the default at this budget, and the reason is structural rather than statistical.

MapElites is now registered as the archive arm. Three arms, three seeds, stage_b's default 60 rounds x 4 proposals = 240 evaluations per run:

arm              AUC_Q          distinct found     best meV/atom
uniform    0.00 +- 0.00        0.00 +- 0.00                 +132
cem        0.36 +- 0.29        2.67 +- 1.76                  -48
archive    0.00 +- 0.00        0.00 +- 0.00                 +129

Prediction 1 confirmed, and more strongly than it was stated. The archive arm does not merely score within the seed spread of uniform - it scores identically to two decimals on both AUC_Q and distinct-found, because at this budget it is uniform sampling. Verified directly rather than inferred: MapElites.init is 600, the run gives 240, and instrumenting the arm shows the archive is consulted zero times in 240 evaluations.

Prediction 2 confirmed: CrossEntropy takes AUC_Q 0.36 and a best of -48 meV/atom against the archive arm's +129 - a gap of 177 meV/atom, and the difference between qualifying and not qualifying at all.

Prediction 3 confirmed. The default is unchanged. E115's result stands and is not contradicted here: it was measured at 20,000 evaluations per arm, eighty times this budget. What E124 establishes is that the result does not transfer to the budget stage_b runs at, and that switching the default would have been a 177 meV/atom regression presented as an improvement.

What was changed instead of the default. MapElites.viable_budget reports the fewest evaluations at which the arm is doing what its name says - init plus four per reachable cell, 1080 on the twelve-element space - and stage_b now prints a warning when the archive arm is run below it, saying in as many words that its score is not a verdict on quality-diversity. An arm that silently degrades into a different algorithm is the kind of thing that produces a confident wrong conclusion, which is what E114 was.

229 passing.

The full record

This entry is written in 2 separate places in the log, shown here in log order.

EXPERIMENTS.md · lines 7379–7408

E124 — Should the archive arm be the default generator?

E115 showed MAP-Elites matching CrossEntropy's best to 0.5 meV/atom while covering 86.9 per cent of the behaviour space against 18.1, and finding a Mo-Ti-Ta basin at -47 meV/atom in all three seeds that the scalar arm never reaches. E123 made the reporting able to show such a basin. The obvious next step is to make the archive the default arm. It is not registered as an arm in stage_b at all.

Before switching anything, the budget. E115 ran 20,000 evaluations per arm. stage_b's default is 60 rounds x 4 proposals = 240 evaluations per run, eighty times smaller. And MapElites.init is 600: the arm samples the simplex uniformly until it holds that many observations before the archive is ever consulted.

240 is less than 600. So at stage_b's default budget the archive arm never leaves its seeding phase, and is not MAP-Elites at all - it is uniform random sampling wearing the name.

Predicted:

  1. At the default budget the archive arm scores statistically indistinguishably from uniform, because it is uniform there. Their AUC_Q means should differ by less than the seed-to-seed spread of either.
  2. CrossEntropy beats both on AUC_Q, because AUC_Q rewards qualifying quickly and a converged Gaussian on a unimodal landscape does that better than random sampling.
  3. So the archive arm must NOT become the default at this budget. The honest conclusion would be that E115's result is real but lives at a budget stage_b does not run at, and that switching the default would be a regression dressed as an improvement.

Falsified if the archive arm is competitive on AUC_Q at 240 evaluations, which would mean either that init is not doing what it appears to, or that AUC_Q is insensitive to the difference, and either of those needs understanding before any default is changed.

EXPERIMENTS.md · lines 7410–7444

E124 result: all three predictions confirmed. The archive must NOT be the default at this budget, and the reason is structural rather than statistical.

MapElites is now registered as the archive arm. Three arms, three seeds, stage_b's default 60 rounds x 4 proposals = 240 evaluations per run:

arm              AUC_Q          distinct found     best meV/atom
uniform    0.00 +- 0.00        0.00 +- 0.00                 +132
cem        0.36 +- 0.29        2.67 +- 1.76                  -48
archive    0.00 +- 0.00        0.00 +- 0.00                 +129

Prediction 1 confirmed, and more strongly than it was stated. The archive arm does not merely score within the seed spread of uniform - it scores identically to two decimals on both AUC_Q and distinct-found, because at this budget it is uniform sampling. Verified directly rather than inferred: MapElites.init is 600, the run gives 240, and instrumenting the arm shows the archive is consulted zero times in 240 evaluations.

Prediction 2 confirmed: CrossEntropy takes AUC_Q 0.36 and a best of -48 meV/atom against the archive arm's +129 - a gap of 177 meV/atom, and the difference between qualifying and not qualifying at all.

Prediction 3 confirmed. The default is unchanged. E115's result stands and is not contradicted here: it was measured at 20,000 evaluations per arm, eighty times this budget. What E124 establishes is that the result does not transfer to the budget stage_b runs at, and that switching the default would have been a 177 meV/atom regression presented as an improvement.

What was changed instead of the default. MapElites.viable_budget reports the fewest evaluations at which the arm is doing what its name says - init plus four per reachable cell, 1080 on the twelve-element space - and stage_b now prints a warning when the archive arm is run below it, saying in as many words that its score is not a verdict on quality-diversity. An arm that silently degrades into a different algorithm is the kind of thing that produces a confident wrong conclusion, which is what E114 was.

229 passing.

Related entries

Built with PRISMWebsite and visualizations made using Claude