Was the search exploring all twelve elements, and does a diversity-seeking search do better?

arm           cells   coverage   best drive   best off-corner
cem            26.7      22.2%    -86.0 meV        +31.7 meV
map-elites     33.0      27.5%    +12.8 meV        +64.5 meV

EXPERIMENTS.md · lines 6661–6715

E114 — The search space was eight elements, and the search collapses inside it

Two separate defects, found by measuring the generator rather than reasoning about it.

First, and worse: the twelve-element expansion was never switched on. stage_b.build() reads os.environ.get("FORAGER_CE", "ce_8element"). data/ce_12element.npz, its hull offlattice_hull12.json and its references references_equilibrium12.npz were all built and all sit behind an environment variable nobody sets. Every result in this project, including every measurement made today, ran on eight elements. Cr, Ni, Cu and Co - the four the operator explicitly asked for - were not in the search space at all, so no amount of fixing the search would ever have produced them. Under FORAGER_CE=ce_12element the path works untouched: CrNiCoCu screens at +72.7 meV/atom, MoNbTaW at -68.5.

Second, the search collapses inside whatever space it is given. CrossEntropy, the strongest model-free arm, fits a Gaussian to the top twelve elites and shrinks its covariance each round. Thirty rounds, eight proposals each, on the eight-element space:

element        early     late           early -> late
W              0.160    0.403     element-mass entropy   2.026 -> 1.443  (uniform 2.079)
Ta             0.120    0.338     distinct supports         45 -> 16
Hf             0.086    0.008     W + Ta, atomic mass      28% -> 74%
Zr             0.090    0.006

This is not a bug in the optimiser. The landscape's global optimum genuinely is Mo-Ta, the strongest bond in the system at -186 meV/atom (Huhn & Widom, JOM 65, 1772, 2013), and a hill climber on a scalar reward returns it. Nothing in the objective ever rewarded being different.

The fix is to change the objective, not the optimiser. forager.generate.MapElites partitions a behaviour space and keeps the best candidate in each cell, so a composition competes only against others that resemble it (Mouret & Clune 2015). Wolińska, Walsh & Cully (Digital Discovery 2024, DOI 10.1039/D4DD00054D) do exactly this for crystal structure prediction - fitness the energy, behaviour two computed properties - and recover the ground state plus a spread of distinct polymorphs. The behaviour axes here are the two an HEA phase diagram is drawn on and both are already in descriptors.py: VEC, which separates bcc refractories near 5 from fcc Cr-Ni-Co-Cu near 9, and delta, atomic size mismatch.

Predicted, before the head-to-head on twelve elements at equal evaluation budget:

1. MAP-Elites covers several times more behaviour cells than CrossEntropy. True by construction and not on its own interesting - it is the control, and if it fails the implementation is broken.
2. The interesting one: MAP-Elites finds at least one composition outside the Mo-Nb-Ta-W corner whose driving force is within 30 meV/atom of the best CrossEntropy finds. If diversity is free at the top, the archive is strictly better than the scalar.
3. CrossEntropy's single best driving force is as good as or better than MAP-Elites', because it optimises that scalar directly and MAP-Elites spends budget elsewhere. A gap larger than 30 meV/atom is the real price of diversity and must be reported as such.
4. Both stay inside the refractory corner for their best-scoring find, because the hull says so - CrNiCoCu is +72.7 meV/atom, genuinely unstable, and a generator that reported it as a win would be wrong. Diversity must not be scored as success on its own.

Falsified if MAP-Elites' archive is full of high-VEC cells whose occupants are all positive driving force, in which case coverage is measuring the space's emptiness rather than finding anything, and the behaviour axes are the wrong ones.

EXPERIMENTS.md · lines 6717–6753

E114 result: all four predictions falsified, and MAP-Elites as first built LOSES.

Twelve elements, 1200 screens per arm, three seeds:

arm           cells   coverage   best drive   best off-corner
cem            26.7      22.2%    -86.0 meV        +31.7 meV
map-elites     33.0      27.5%    +12.8 meV        +64.5 meV

Prediction 1 falsified: coverage is 1.24x, not "several times". Prediction 2 falsified: the best off-corner find is 150 meV/atom worse than CrossEntropy's best, not 30. Prediction 3's costly branch is what happened: the price of diversity here is 98.8 meV/atom, far past the 30 declared as the threshold for reporting it as a real cost. Prediction 4 holds only trivially - both arms' best finds are refractory, but MAP-Elites' are bad ones (Mo0.52 Ti0.23, Nb0.52 W0.22 Mo0.17, Nb0.70 Ta0.13).

The diagnosis is in the number I nearly skipped: both arms cover about a quarter of the grid. That is either an empty grid or a search that does not reach, and those demand opposite fixes. Measured directly - 54,000 compositions, dense Dirichlet plus the sparse 2- to 5-element supports a generator actually proposes - 115 of 120 cells are reachable, 96 per cent. So the behaviour axes are sound and the search is failing to reach.

Three defects in the implementation, not in the idea:

1. The grid ranges were guessed and wrong. delta reaches 12.37 per cent against an assumed ceiling of 8, so the entire tail collapsed into one bin - and the bulk of the space sits at delta 7.6 to 8.4, right on that bin's edge. VEC reaches 11.0 against an assumed 10.5. Now 4.0-11.0 and 0-13.
2. init=64 cannot seed 115 cells. MAP-Elites selects from what it has, so a thin archive makes it mutate the same few parents forever. Now 600.
3. 1200 evaluations is not a MAP-Elites budget. The method's premise, and the explicit argument of Wolińska, Walsh & Cully, is that surrogates make high evaluation counts affordable. At 57-106 ms per screen, 20,000 evaluations is 35 minutes.

Also changed: Differential MAP-Elites variation (Storn & Price operators inside the archive, arXiv 2107.04964) instead of a fixed Gaussian step. A Gaussian needs its step size chosen in advance and is wrong everywhere else; DE takes its step from the spread of the archive itself. Reported to beat Gaussian CVT-MAP-Elites on both quality and coverage.