Can any of the fly's own signals tell where its predictions are wrong?

EXPERIMENTS.md · lines 1394–1457

E29 — The fly's error is bias, so no variance-based uncertainty can see it

Date 2026-09-12 · Question Bayesian optimisation needs an uncertainty. Does the circuit have one? · Provenance …/{novelty_test,uncertainty_chain, uc2,uc3,uc4,uc5}.py, forager/mushroom.py

Three signals, all tried, all failed.

All three negative, which is worse than useless: an exploration bonus built on any of them sends the search where the circuit is already right. It also explains why fly+explore scored identically to fly in the search benchmark, to the decimal.

Two false trails, both corrected.

• A first test found error uncorrelated with distance from the training data (+0.007) and concluded there was nothing to predict. That test had no power: in eight dimensions 100 random points sit at nearly equal nearest-neighbour distance (spread/median 0.25). Restricting the training region widened it to 0.42 and the correlation rose to +0.215. The effect was real; the measurement could not see it.
• An earlier attempt compared two regions with different reward distributions while standardising across both, so a distribution shift entered the residual and the trained region appeared harder than the untrained one. Standardising on training statistics only removes it.

The shared-confound hypothesis was tested and rejected. All three signals grow with prediction magnitude, and a model can be most accurate where it predicts most strongly, which would give every one of them the same spurious sign. Controlling for |prediction| changes almost nothing (-0.137 to -0.177, -0.213 to -0.204, -0.077 to -0.068). Not it.

What it actually is. Train the same circuit on the same 100 observations eight times, varying only the order they arrive in:

• agreement between runs on which compositions are hard: +0.998
• variance of the error explained by the composition: 99.8%
• variance left to the run itself: 0.2%

The error is bias, not variance. The circuit is wrong in the same places every time, whatever path its learning takes. Novelty, disagreement and bootstrap ensembles all measure variance - how much the answer would move if the data or the fit were resampled

• and a quantity that is identical across every resampling is invisible to all of them by construction. Three failures, one cause, and it was not a shortage of ideas for signals.

Consequences.

• No variance-based uncertainty can work here. Not a limitation of the three tried; a statement about what they measure.
• Exploration is not the fly's problem. Its 99.8%-reproducible error is the gap between what its local plasticity rule converges to and what its own code can support - ridge regression on that same code reaches rank 0.6 where the rule reaches 14 (E28). The lever is the rule, not the acquisition function.
• Bayesian optimisation over this environment should take its uncertainty from the cluster expansion, whose bootstrap ensemble is measured, unbiased and covers every held-out point (E24). Using a validated uncertainty rather than inventing one that correlates at -0.1 is the honest construction.

Caveat. This concerns the error of a trained readout at fixed sparsity 0.12 on one problem. It does not show that connectome circuits have no usable uncertainty in general; it shows this one's error, here, is reproducible to 99.8% and therefore not a variance.