With two goals at once, does the fly beat the standard statistical optimiser?
No. In a pre-registered test the optimiser won by 5.03 standard errors and the fly fell below random; the benchmark also had little power.
In the log: The multi-objective test: a pre-registered loss, on a benchmark with no power
falsifiedDate 2026-09-12, as written in the loggenerator · fly brain0 predictions · 1 result paragraphEXPERIMENTS.md lines 1748–1796
What E35 did and how it came out, drawn from this record and the files it names (book/assets/diagrams/exp/E35.svg).
Pre-registration
The pre-registration, as written
E35 — The multi-objective test: a pre-registered loss, on a benchmark with no power
Date 2026-09-12 · Question A mushroom body is many compartments reading one shared
code, each taught its own valence. Does that beat a Gaussian process when there is more
than one objective? · ProvenancePREREGISTER_multiobjective.md,
…/{multiobj,multiobj2,head_check,audit}.py
Pre-registered, design and seeds fixed before the code ran, because the operator wanted
a particular outcome and wanting one is how one gets manufactured.
-5.03 standard errors, and below random. The verdict stands as recorded.
An explanation I offered and then disproved. The heads were undertrained: at the budget
used, their scores correlated with their own objectives at only 0.139 and 0.311, against
0.706 for a single head at the same budget - each head owns roughly half the plastic
synapses (33,110 and 28,100 of 61,210), so each learns at roughly half the rate. Doubling
the budget raises those correlations to 0.482 and 0.662 and changes the result not at
all: 15.715 against 15.727. Training was not the cause, and the tidy explanation was
wrong.
The benchmark had almost no power, which is a fault in its design. Audited afterwards:
94.4% of the hypervolume comes from the shared 20-point random start. Every method
begins at 15.54; the whole contest happens inside a band 0.9 wide.
The objectives are not commensurate. Free energy spans 193 to 476 meV/atom, a factor
of 2.5. The solid-solution fraction spans 0.922 to 0.992 - 7% of its mean. Against a
reference point at the minimum of each, hypervolume is dominated by the first, so a test
built to measure multi-objective behaviour was close to single-objective.
Ceiling from the full 4,000-draw sweep: 18.110. A random 60 reaches 16.465.
What is nonetheless real. The fly lands below random by 4.9 standard errors, and that
is explicable rather than noise: it selects greedily from a model whose error is bias
(E29), so it concentrates on a systematically wrong region while random at least spreads.
An additional asymmetry, introduced by me and not by the architecture: the process was
given an uncertainty-aware acquisition and the circuit pure greed, so part of what this
measured was the acquisition rule rather than the model.
Not repaired by re-running. A metric changed after seeing results is not a metric. The
properly powered version - objectives normalised before the hypervolume, improvement over
the shared start rather than absolute, matched acquisitions - is a new experiment and needs
its own pre-registration.
E35 — The multi-objective test: a pre-registered loss, on a benchmark with no power
Date 2026-09-12 · Question A mushroom body is many compartments reading one shared
code, each taught its own valence. Does that beat a Gaussian process when there is more
than one objective? · ProvenancePREREGISTER_multiobjective.md,
…/{multiobj,multiobj2,head_check,audit}.py
Pre-registered, design and seeds fixed before the code ran, because the operator wanted
a particular outcome and wanting one is how one gets manufactured.
-5.03 standard errors, and below random. The verdict stands as recorded.
An explanation I offered and then disproved. The heads were undertrained: at the budget
used, their scores correlated with their own objectives at only 0.139 and 0.311, against
0.706 for a single head at the same budget - each head owns roughly half the plastic
synapses (33,110 and 28,100 of 61,210), so each learns at roughly half the rate. Doubling
the budget raises those correlations to 0.482 and 0.662 and changes the result not at
all: 15.715 against 15.727. Training was not the cause, and the tidy explanation was
wrong.
The benchmark had almost no power, which is a fault in its design. Audited afterwards:
94.4% of the hypervolume comes from the shared 20-point random start. Every method
begins at 15.54; the whole contest happens inside a band 0.9 wide.
The objectives are not commensurate. Free energy spans 193 to 476 meV/atom, a factor
of 2.5. The solid-solution fraction spans 0.922 to 0.992 - 7% of its mean. Against a
reference point at the minimum of each, hypervolume is dominated by the first, so a test
built to measure multi-objective behaviour was close to single-objective.
Ceiling from the full 4,000-draw sweep: 18.110. A random 60 reaches 16.465.
What is nonetheless real. The fly lands below random by 4.9 standard errors, and that
is explicable rather than noise: it selects greedily from a model whose error is bias
(E29), so it concentrates on a systematically wrong region while random at least spreads.
An additional asymmetry, introduced by me and not by the architecture: the process was
given an uncertainty-aware acquisition and the circuit pure greed, so part of what this
measured was the acquisition rule rather than the model.
Not repaired by re-running. A metric changed after seeing results is not a metric. The
properly powered version - objectives normalised before the hypervolume, improvement over
the shared start rather than absolute, matched acquisitions - is a new experiment and needs
its own pre-registration.
Related entries
E29 — The fly's error is bias, so no variance-based uncertainty can see it