Given a budget big enough to start, does the diversity-seeking search find more qualifying alloys?
No. At 1200 evaluations it found 2.67 qualifying alloys per run against 54 for the standard search, though its best reached −76 meV/atom.
In the log: Raise the budget, or stop calling the archive an arm?
mixedDate not stated in the log; it was written between the commit of 2026-09-16 06:55 and the first commit that contains it, 2026-09-16 07:54search baselines0 predictions · 1 result paragraphEXPERIMENTS.md lines 7446–7481, lines 7483–7534
What E125 did and how it came out, drawn from this record and the files it names (book/assets/diagrams/exp/E125.svg).
Results
EXPERIMENTS.md · line 7483
E125 result: predictions 1 and 2 confirmed, prediction 3 FALSIFIED — and the falsification
overturns the framing, not just the number.
1200 evaluations per run, above viable_budget 1080, three seeds:
arm AUC_Q distinct qualifiers best meV/atom
uniform 0.00 +- 0.00 0.00 +- 0.00 +84
cem 21.21 +- 6.28 54.00 +- 7.21 -80
archive 0.51 +- 0.26 2.67 +- 0.67 -76
Prediction 1 confirmed: CrossEntropy takes AUC_Q by a factor of 42.
Prediction 2 confirmed: the archive arm's best find goes from +129 meV/atom at 240
evaluations to -76 at 1200. Past init it does what E115 measured, and lands 4 meV/atom
from CrossEntropy's best - close to E115's 0.5 meV at twenty thousand. viable_budget is set
about right.
Prediction 3 falsified, and badly. I predicted the archive would find more distinct
compositions than CrossEntropy, since that is what it optimises for. It found 2.67 against
54.00 - twenty times fewer.
The error was mine and it was a conflation.distinct counts distinct confirmed
qualifiers - compositions past the -40 meV/atom bar and at least 0.15 apart. E115's 86.9
per cent "coverage" counts behaviour cells filled, including every cell filled with an
unstable composition. Those are different quantities and I treated them as the same. E117
already established that only 3.2 per cent of the composition grid passes both feasibility
tests, so an algorithm that spreads evenly over behaviour space spends almost all of its
budget in infeasible regions - by design.
So the decision rule I wrote before the numbers does not fire on either branch, and I am not
going to retrofit it. It said: if CrossEntropy wins AUC_Q and the archive finds more
distinct compositions, then AUC_Q is the wrong instrument. The premise failed. And the
opposite of what that rule was guarding against turned out to be true: AUC_Q is not unfair
to the archive. It agrees with the distinct-qualifier count, which is the very thing I
claimed AUC_Q could not see. I suspected the metric was biased; the metric was right.
Decision: stage_b's default budget and default arm both stand. CrossEntropy is not
merely winning a metric that suits it - it finds twenty times more qualifying compositions
and a better best. The archive is not a better search at any budget tested; it is a
different sampler whose coverage is mostly coverage of infeasible space.
What is NOT withdrawn.E115's specific finding stands: at 20,000 evaluations MAP-Elites
found a reproducible Mo-Ti-Ta basin at -47 meV/atom, in all three seeds, that CrossEntropy
never reached. The archive does reach places the scalar search does not, and one of those
places held something real. What E125 removes is the inference that this makes it a better
generator. It finds different things, and far fewer of them per unit budget.
E123's promotion change is unaffected and still correct: reporting one elite per niche costs
nothing at the top and surfaces what a run found. It simply cannot surface what a run did not
find, and the archive arm finds little.
230 passing.
The full record
This entry is written in 2 separate places in the log, shown here in log order.
EXPERIMENTS.md · lines 7446–7481
E125 — Raise the budget, or stop calling the archive an arm?
E124 left one decision. Either stage_b's default rises above MapElites.viable_budget
(1080 on twelve elements) so the archive arm is actually itself, or the archive is accepted as
a separate large-budget tool and stops being scored alongside the others. Leaving it
registered-but-useless is not an option.
The thing to notice before measuring is what AUC_Q is. It is the area under the
qualification curve - it rewards qualifying early. The archive arm spends its first init
= 600 evaluations sampling uniformly before it consults its archive once. At a budget of 1200
that is half the run producing nothing AUC_Q can credit, by construction, whatever the
method's merit.
Predicted, at 1200 evaluations per run, three seeds, against E124's 240-evaluation
baseline:
CrossEntropy still wins AUC_Q, and by a wide margin, because 600 of the archive arm's
1200 evaluations cannot qualify anything. Raising the budget to the viable threshold does
not make the archive competitive on this metric.
The archive arm's best find improves sharply and goes below zero - from +129 meV/atom
at 240 evaluations to better than -40, because past 600 it is finally doing what E115
measured. If it does not, viable_budget is set too low and the threshold is wrong.
The archive arm finds more distinct compositions than CrossEntropy, since that is the
one thing it optimises for and the one thing AUC_Q does not measure.
The decision this forces, stated before the numbers so it cannot be fitted to them: if 1
and 3 both hold, then AUC_Q is simply the wrong instrument for this arm - it measures
time-to-qualify and the archive's value is coverage of the feasible set - and the honest
outcome is to keep stage_b's default where it is, keep CrossEntropy as the default arm, and
score the archive separately on coverage and on the feasible set it returns. If instead
prediction 1 fails and the archive wins AUC_Q at 1200, the default budget should rise and
the arm comparison becomes meaningful as it stands.
Falsified if the archive arm's best find does not improve at all between 240 and 1200
evaluations, which would mean the problem is not budget and E115's result does not
reproduce through stage_b at any budget - a much more serious finding about the arm wiring.
EXPERIMENTS.md · lines 7483–7534
E125 result: predictions 1 and 2 confirmed, prediction 3 FALSIFIED — and the falsification
overturns the framing, not just the number.
1200 evaluations per run, above viable_budget 1080, three seeds:
arm AUC_Q distinct qualifiers best meV/atom
uniform 0.00 +- 0.00 0.00 +- 0.00 +84
cem 21.21 +- 6.28 54.00 +- 7.21 -80
archive 0.51 +- 0.26 2.67 +- 0.67 -76
Prediction 1 confirmed: CrossEntropy takes AUC_Q by a factor of 42.
Prediction 2 confirmed: the archive arm's best find goes from +129 meV/atom at 240
evaluations to -76 at 1200. Past init it does what E115 measured, and lands 4 meV/atom
from CrossEntropy's best - close to E115's 0.5 meV at twenty thousand. viable_budget is set
about right.
Prediction 3 falsified, and badly. I predicted the archive would find more distinct
compositions than CrossEntropy, since that is what it optimises for. It found 2.67 against
54.00 - twenty times fewer.
The error was mine and it was a conflation.distinct counts distinct confirmed
qualifiers - compositions past the -40 meV/atom bar and at least 0.15 apart. E115's 86.9
per cent "coverage" counts behaviour cells filled, including every cell filled with an
unstable composition. Those are different quantities and I treated them as the same. E117
already established that only 3.2 per cent of the composition grid passes both feasibility
tests, so an algorithm that spreads evenly over behaviour space spends almost all of its
budget in infeasible regions - by design.
So the decision rule I wrote before the numbers does not fire on either branch, and I am not
going to retrofit it. It said: if CrossEntropy wins AUC_Q and the archive finds more
distinct compositions, then AUC_Q is the wrong instrument. The premise failed. And the
opposite of what that rule was guarding against turned out to be true: AUC_Q is not unfair
to the archive. It agrees with the distinct-qualifier count, which is the very thing I
claimed AUC_Q could not see. I suspected the metric was biased; the metric was right.
Decision: stage_b's default budget and default arm both stand. CrossEntropy is not
merely winning a metric that suits it - it finds twenty times more qualifying compositions
and a better best. The archive is not a better search at any budget tested; it is a
different sampler whose coverage is mostly coverage of infeasible space.
What is NOT withdrawn.E115's specific finding stands: at 20,000 evaluations MAP-Elites
found a reproducible Mo-Ti-Ta basin at -47 meV/atom, in all three seeds, that CrossEntropy
never reached. The archive does reach places the scalar search does not, and one of those
places held something real. What E125 removes is the inference that this makes it a better
generator. It finds different things, and far fewer of them per unit budget.
E123's promotion change is unaffected and still correct: reporting one elite per niche costs
nothing at the top and surfaces what a run found. It simply cannot surface what a run did not
find, and the archive arm finds little.
230 passing.
Related entries
E124 — Should the archive arm be the default generator?
E115 — The same head-to-head, with the implementation defects fixed
E117 — Is rung 0's reward anti-aligned with the actual requirement?