Can the fly be given honest error bars even though its error is bias?

EXPERIMENTS.md · lines 2012–2069

E40 — Calibrated uncertainty, by not assuming the error is a variance

Date 2026-09-12 · Question (operator) The circuit has to learn uncertainty properly - with it, it is interchangeable with any other network in a Bayesian optimisation or active-learning loop. · Provenance forager/uncertainty.py, …/uq_test.py

Why three previous attempts failed, in one sentence. A bootstrap ensemble, the spread across compartments and a raw novelty score all measure variance - how far an answer would move under resampling - and E29 established this circuit's error is reproducible to 99.8% across runs on identical data. A quantity identical under every resampling is invisible to every method that resamples.

The method that does not care. Conformal prediction is distribution-free and gives a finite-sample coverage guarantee for any model and any heuristic: calibrate on held-out residuals, take the empirical quantile at level ceil((n+1)(1-alpha))/n, and the interval covers at least 1-alpha of the time. The model may be biased, badly calibrated or plainly wrong; the heuristic only decides how sharp the interval is, never whether it is honest. Following Hu, Musielewicz, Ulissi and Kitchin, Mach. Learn.: Sci. Technol. 4, 015031 (2023), who pair it with latent-space distance for machine-learned interatomic potentials.

Result, 160 training / 100 calibration / 100 held out, nominal coverage 90%:

Coverage is achieved by every row, which is the guarantee doing its work and is the first honest uncertainty this circuit has had. Wrapping the failed signals in conformal prediction makes them all truthful, so what separates them is sharpness alone - which turns "whose uncertainty is real" from an argument into a measurement.

The circuit's own sparse code is the best signal available: sharpest at nominal coverage, 16% tighter than knowing nothing. So the Kenyon expansion does carry information about where the readout is unreliable - the earlier novelty attempt was not wrong in kind, only raw and uncalibrated.

The margin is modest and is reported as such. 0.940 against 1.124 is useful, not decisive, and the width tracks the actual error at only +0.071. This is an uncertainty that is honest and mildly informative, not one that is sharp.

The bootstrap ensemble is worse than knowing nothing - 1.763 against 1.124, and anti-correlated with the error. Exactly what E29 predicts for an error that is bias, and consistent with the literature finding ensembles overconfident on neural potentials, their "magnitude of uncertainty lower than the error by orders of magnitude" (npj Comput. Mater. 2025).

What this unlocks. An acquisition function needs a calibrated interval, not a correct one. With coverage guaranteed the circuit can be dropped into a Bayesian-optimisation or active-learning loop on the same footing as any other surrogate - and swapped for one.

Also corrected here: the first test of this asserted nominal coverage on a single calibration split. Conformal's guarantee is marginal - in expectation over splits - and one split of 500 points missed by 8% purely because its halves differed in spread. The implementation was right and the test was wrong; it now averages over forty splits.