Experiments · E35

With two goals at once, does the fly beat the standard statistical optimiser?

No. In a pre-registered test the optimiser won by 5.03 standard errors and the fly fell below random; the benchmark also had little power.

In the log: The multi-objective test: a pre-registered loss, on a benchmark with no power

falsifiedDate 2026-09-12, as written in the loggenerator · fly brain0 predictions · 1 result paragraphEXPERIMENTS.md lines 1748–1796
exp E35 diagram
What E35 did and how it came out, drawn from this record and the files it names (book/assets/diagrams/exp/E35.svg).

Pre-registration

The pre-registration, as written

E35 — The multi-objective test: a pre-registered loss, on a benchmark with no power

Date 2026-09-12 · Question A mushroom body is many compartments reading one shared code, each taught its own valence. Does that beat a Gaussian process when there is more than one objective? · Provenance PREREGISTER_multiobjective.md, …/{multiobj,multiobj2,head_check,audit}.py

Pre-registered, design and seeds fixed before the code ran, because the operator wanted a particular outcome and wanting one is how one gets manufactured.

Result: LOSS. Hypervolume, 60 evaluations, seeds 700-709:

method hypervolume standard error
Gaussian process + ParEGO 16.930 0.094
random 16.445 0.096
the fly, two compartment heads 15.727 0.220

-5.03 standard errors, and below random. The verdict stands as recorded.

An explanation I offered and then disproved. The heads were undertrained: at the budget used, their scores correlated with their own objectives at only 0.139 and 0.311, against 0.706 for a single head at the same budget - each head owns roughly half the plastic synapses (33,110 and 28,100 of 61,210), so each learns at roughly half the rate. Doubling the budget raises those correlations to 0.482 and 0.662 and changes the result not at all: 15.715 against 15.727. Training was not the cause, and the tidy explanation was wrong.

The benchmark had almost no power, which is a fault in its design. Audited afterwards:

  • 94.4% of the hypervolume comes from the shared 20-point random start. Every method begins at 15.54; the whole contest happens inside a band 0.9 wide.
  • The objectives are not commensurate. Free energy spans 193 to 476 meV/atom, a factor of 2.5. The solid-solution fraction spans 0.922 to 0.992 - 7% of its mean. Against a reference point at the minimum of each, hypervolume is dominated by the first, so a test built to measure multi-objective behaviour was close to single-objective.
  • Ceiling from the full 4,000-draw sweep: 18.110. A random 60 reaches 16.465.

What is nonetheless real. The fly lands below random by 4.9 standard errors, and that is explicable rather than noise: it selects greedily from a model whose error is bias (E29), so it concentrates on a systematically wrong region while random at least spreads. An additional asymmetry, introduced by me and not by the architecture: the process was given an uncertainty-aware acquisition and the circuit pure greed, so part of what this measured was the acquisition rule rather than the model.

Not repaired by re-running. A metric changed after seeing results is not a metric. The properly powered version - objectives normalised before the hypervolume, improvement over the shared start rather than absolute, matched acquisitions - is a new experiment and needs its own pre-registration.


Results

EXPERIMENTS.md · line 1758

Result: LOSS. Hypervolume, 60 evaluations, seeds 700-709:

The full record

EXPERIMENTS.md · lines 1748–1796

E35 — The multi-objective test: a pre-registered loss, on a benchmark with no power

Date 2026-09-12 · Question A mushroom body is many compartments reading one shared code, each taught its own valence. Does that beat a Gaussian process when there is more than one objective? · Provenance PREREGISTER_multiobjective.md, …/{multiobj,multiobj2,head_check,audit}.py

Pre-registered, design and seeds fixed before the code ran, because the operator wanted a particular outcome and wanting one is how one gets manufactured.

Result: LOSS. Hypervolume, 60 evaluations, seeds 700-709:

method hypervolume standard error
Gaussian process + ParEGO 16.930 0.094
random 16.445 0.096
the fly, two compartment heads 15.727 0.220

-5.03 standard errors, and below random. The verdict stands as recorded.

An explanation I offered and then disproved. The heads were undertrained: at the budget used, their scores correlated with their own objectives at only 0.139 and 0.311, against 0.706 for a single head at the same budget - each head owns roughly half the plastic synapses (33,110 and 28,100 of 61,210), so each learns at roughly half the rate. Doubling the budget raises those correlations to 0.482 and 0.662 and changes the result not at all: 15.715 against 15.727. Training was not the cause, and the tidy explanation was wrong.

The benchmark had almost no power, which is a fault in its design. Audited afterwards:

  • 94.4% of the hypervolume comes from the shared 20-point random start. Every method begins at 15.54; the whole contest happens inside a band 0.9 wide.
  • The objectives are not commensurate. Free energy spans 193 to 476 meV/atom, a factor of 2.5. The solid-solution fraction spans 0.922 to 0.992 - 7% of its mean. Against a reference point at the minimum of each, hypervolume is dominated by the first, so a test built to measure multi-objective behaviour was close to single-objective.
  • Ceiling from the full 4,000-draw sweep: 18.110. A random 60 reaches 16.465.

What is nonetheless real. The fly lands below random by 4.9 standard errors, and that is explicable rather than noise: it selects greedily from a model whose error is bias (E29), so it concentrates on a systematically wrong region while random at least spreads. An additional asymmetry, introduced by me and not by the architecture: the process was given an uncertainty-aware acquisition and the circuit pure greed, so part of what this measured was the acquisition rule rather than the model.

Not repaired by re-running. A metric changed after seeing results is not a metric. The properly powered version - objectives normalised before the hypervolume, improvement over the shared start rather than absolute, matched acquisitions - is a new experiment and needs its own pre-registration.

Related entries

Built with PRISMWebsite and visualizations made using Claude