Experiments · E125

Given a budget big enough to start, does the diversity-seeking search find more qualifying alloys?

No. At 1200 evaluations it found 2.67 qualifying alloys per run against 54 for the standard search, though its best reached −76 meV/atom.

In the log: Raise the budget, or stop calling the archive an arm?

mixedDate not stated in the log; it was written between the commit of 2026-09-16 06:55 and the first commit that contains it, 2026-09-16 07:54search baselines0 predictions · 1 result paragraphEXPERIMENTS.md lines 7446–7481, lines 7483–7534
exp E125 diagram
What E125 did and how it came out, drawn from this record and the files it names (book/assets/diagrams/exp/E125.svg).

Results

EXPERIMENTS.md · line 7483

E125 result: predictions 1 and 2 confirmed, prediction 3 FALSIFIED — and the falsification overturns the framing, not just the number.

1200 evaluations per run, above viable_budget 1080, three seeds:

arm              AUC_Q          distinct qualifiers      best meV/atom
uniform    0.00 +- 0.00           0.00 +- 0.00                    +84
cem       21.21 +- 6.28          54.00 +- 7.21                    -80
archive    0.51 +- 0.26           2.67 +- 0.67                    -76

Prediction 1 confirmed: CrossEntropy takes AUC_Q by a factor of 42.

Prediction 2 confirmed: the archive arm's best find goes from +129 meV/atom at 240 evaluations to -76 at 1200. Past init it does what E115 measured, and lands 4 meV/atom from CrossEntropy's best - close to E115's 0.5 meV at twenty thousand. viable_budget is set about right.

Prediction 3 falsified, and badly. I predicted the archive would find more distinct compositions than CrossEntropy, since that is what it optimises for. It found 2.67 against 54.00 - twenty times fewer.

The error was mine and it was a conflation. distinct counts distinct confirmed qualifiers - compositions past the -40 meV/atom bar and at least 0.15 apart. E115's 86.9 per cent "coverage" counts behaviour cells filled, including every cell filled with an unstable composition. Those are different quantities and I treated them as the same. E117 already established that only 3.2 per cent of the composition grid passes both feasibility tests, so an algorithm that spreads evenly over behaviour space spends almost all of its budget in infeasible regions - by design.

So the decision rule I wrote before the numbers does not fire on either branch, and I am not going to retrofit it. It said: if CrossEntropy wins AUC_Q and the archive finds more distinct compositions, then AUC_Q is the wrong instrument. The premise failed. And the opposite of what that rule was guarding against turned out to be true: AUC_Q is not unfair to the archive. It agrees with the distinct-qualifier count, which is the very thing I claimed AUC_Q could not see. I suspected the metric was biased; the metric was right.

Decision: stage_b's default budget and default arm both stand. CrossEntropy is not merely winning a metric that suits it - it finds twenty times more qualifying compositions and a better best. The archive is not a better search at any budget tested; it is a different sampler whose coverage is mostly coverage of infeasible space.

What is NOT withdrawn. E115's specific finding stands: at 20,000 evaluations MAP-Elites found a reproducible Mo-Ti-Ta basin at -47 meV/atom, in all three seeds, that CrossEntropy never reached. The archive does reach places the scalar search does not, and one of those places held something real. What E125 removes is the inference that this makes it a better generator. It finds different things, and far fewer of them per unit budget.

E123's promotion change is unaffected and still correct: reporting one elite per niche costs nothing at the top and surfaces what a run found. It simply cannot surface what a run did not find, and the archive arm finds little.

230 passing.

The full record

This entry is written in 2 separate places in the log, shown here in log order.

EXPERIMENTS.md · lines 7446–7481

E125 — Raise the budget, or stop calling the archive an arm?

E124 left one decision. Either stage_b's default rises above MapElites.viable_budget (1080 on twelve elements) so the archive arm is actually itself, or the archive is accepted as a separate large-budget tool and stops being scored alongside the others. Leaving it registered-but-useless is not an option.

The thing to notice before measuring is what AUC_Q is. It is the area under the qualification curve - it rewards qualifying early. The archive arm spends its first init = 600 evaluations sampling uniformly before it consults its archive once. At a budget of 1200 that is half the run producing nothing AUC_Q can credit, by construction, whatever the method's merit.

Predicted, at 1200 evaluations per run, three seeds, against E124's 240-evaluation baseline:

  1. CrossEntropy still wins AUC_Q, and by a wide margin, because 600 of the archive arm's 1200 evaluations cannot qualify anything. Raising the budget to the viable threshold does not make the archive competitive on this metric.
  2. The archive arm's best find improves sharply and goes below zero - from +129 meV/atom at 240 evaluations to better than -40, because past 600 it is finally doing what E115 measured. If it does not, viable_budget is set too low and the threshold is wrong.
  3. The archive arm finds more distinct compositions than CrossEntropy, since that is the one thing it optimises for and the one thing AUC_Q does not measure.

The decision this forces, stated before the numbers so it cannot be fitted to them: if 1 and 3 both hold, then AUC_Q is simply the wrong instrument for this arm - it measures time-to-qualify and the archive's value is coverage of the feasible set - and the honest outcome is to keep stage_b's default where it is, keep CrossEntropy as the default arm, and score the archive separately on coverage and on the feasible set it returns. If instead prediction 1 fails and the archive wins AUC_Q at 1200, the default budget should rise and the arm comparison becomes meaningful as it stands.

Falsified if the archive arm's best find does not improve at all between 240 and 1200 evaluations, which would mean the problem is not budget and E115's result does not reproduce through stage_b at any budget - a much more serious finding about the arm wiring.

EXPERIMENTS.md · lines 7483–7534

E125 result: predictions 1 and 2 confirmed, prediction 3 FALSIFIED — and the falsification overturns the framing, not just the number.

1200 evaluations per run, above viable_budget 1080, three seeds:

arm              AUC_Q          distinct qualifiers      best meV/atom
uniform    0.00 +- 0.00           0.00 +- 0.00                    +84
cem       21.21 +- 6.28          54.00 +- 7.21                    -80
archive    0.51 +- 0.26           2.67 +- 0.67                    -76

Prediction 1 confirmed: CrossEntropy takes AUC_Q by a factor of 42.

Prediction 2 confirmed: the archive arm's best find goes from +129 meV/atom at 240 evaluations to -76 at 1200. Past init it does what E115 measured, and lands 4 meV/atom from CrossEntropy's best - close to E115's 0.5 meV at twenty thousand. viable_budget is set about right.

Prediction 3 falsified, and badly. I predicted the archive would find more distinct compositions than CrossEntropy, since that is what it optimises for. It found 2.67 against 54.00 - twenty times fewer.

The error was mine and it was a conflation. distinct counts distinct confirmed qualifiers - compositions past the -40 meV/atom bar and at least 0.15 apart. E115's 86.9 per cent "coverage" counts behaviour cells filled, including every cell filled with an unstable composition. Those are different quantities and I treated them as the same. E117 already established that only 3.2 per cent of the composition grid passes both feasibility tests, so an algorithm that spreads evenly over behaviour space spends almost all of its budget in infeasible regions - by design.

So the decision rule I wrote before the numbers does not fire on either branch, and I am not going to retrofit it. It said: if CrossEntropy wins AUC_Q and the archive finds more distinct compositions, then AUC_Q is the wrong instrument. The premise failed. And the opposite of what that rule was guarding against turned out to be true: AUC_Q is not unfair to the archive. It agrees with the distinct-qualifier count, which is the very thing I claimed AUC_Q could not see. I suspected the metric was biased; the metric was right.

Decision: stage_b's default budget and default arm both stand. CrossEntropy is not merely winning a metric that suits it - it finds twenty times more qualifying compositions and a better best. The archive is not a better search at any budget tested; it is a different sampler whose coverage is mostly coverage of infeasible space.

What is NOT withdrawn. E115's specific finding stands: at 20,000 evaluations MAP-Elites found a reproducible Mo-Ti-Ta basin at -47 meV/atom, in all three seeds, that CrossEntropy never reached. The archive does reach places the scalar search does not, and one of those places held something real. What E125 removes is the inference that this makes it a better generator. It finds different things, and far fewer of them per unit budget.

E123's promotion change is unaffected and still correct: reporting one elite per niche costs nothing at the top and surfaces what a run found. It simply cannot surface what a run did not find, and the archive arm finds little.

230 passing.

Related entries

Built with PRISMWebsite and visualizations made using Claude