P1 — cross-scene calibrator transfer and certificate operating studies¶

P1 hardens the P0 killer property (calibrated, certified, exported per-carrier confidence) with two measurements the P0 paper explicitly declined for lack of data:

P1a — cross-scene calibrator transfer. Does a calibrator fit on one scene work on another, or must every asset carry its own?
P1b — certificate operating study, all scenes. The P0 paper reported the conformal certificate at a single ε = 0.6 and sketched the ε-sweep only on Truck; P1b sweeps ε on all four scenes and both labels and locates where the certificate first becomes selective.

Both are pure CPU/numpy over the committed outputs/reliability_*.npz, seed-0 deterministic, no retraining and no GPU. Every number below traces to outputs/cross_scene_transfer.json (P1a) or outputs/cert_sweep.json (P1b). The diagonal of P1a and the ε = 0.6 slice of P1b reproduce the committed outputs/calib_<scene>{,_depth}.json P0 reports exactly (verified in-script).

Reproduce:

.gpu_venv/bin/python experiments/cross_scene_transfer.py   # -> outputs/cross_scene_transfer.json
.gpu_venv/bin/python experiments/cert_sweep.py             # -> outputs/cert_sweep.json

Scenes: Truck (Tanks & Temples), Garden, Kitchen, Room (Mip-NeRF 360). Labels: color (colour-agreement) and depth (occlusion-aware).

P1a — cross-scene calibrator transfer¶

Protocol¶

For each ordered pair (source A → target B) over the four scenes and each label:

Diagonal (A == B) = the in-scene reference: fit the isotonic calibrator on B's 50/50 seed-0 cal half and evaluate on B's eval half — the exact protocol of experiments/calibrate_confidence.py, so the diagonal reproduces outputs/calib_<scene>.json.
Off-diagonal (A ≠ B) = the transfer: fit the calibrator on all labeled carriers of source A (the export-time train_agree feature), then evaluate on target B's eval half — the identical held-out set the diagonal uses, so transferred and in-scene ECE/AUC are directly comparable.
Certificate: ε = 0.6, α = 0.1, computed with the transferred confidence on B's own calibration split (cal_idx). That local conformal set is the point of the experiment (see below).

The key structural fact: selection transfers for free, calibration is the real question¶

An isotonic calibrator is a monotone map, and selection AUC is a ranking metric (sort carriers by score, measure retained reliability across budgets). So the order B's carriers are scored in is the same whether the calibrator was fit on A or on B — only the absolute probability values differ. Consequently:

Selection AUC transfers essentially perfectly. Across all 24 off-diagonal pairs the transferred-vs-in-scene AUC delta is within ±0.0004 (color mean −0.0000, depth mean −0.0001; range [−0.0004, +0.0001]). Pruning quality is scene-portable for free — a transferred calibrator prunes as well as a native one.
The real transfer question is ECE — whether A's absolute probability scale is calibrated on B. That is what the matrices below measure.

ECE transfer matrix (rows = source `A`, columns = target `B`)¶

color label (diagonal = in-scene reference; raw view-count heuristic ECE for reference: Truck 0.586, Garden 0.551, Kitchen 0.557, Room 0.464):

A ↓ \ B →	Truck	Garden	Kitchen	Room
Truck	0.0014	0.0044	0.0123	0.0173
Garden	0.0047	0.0015	0.0084	0.0127
Kitchen	0.0113	0.0083	0.0006	0.0027
Room	0.0094	0.0074	0.0024	0.0016

depth label (raw ECE: Truck 0.620, Garden 0.537, Kitchen 0.550, Room 0.471):

A ↓ \ B →	Truck	Garden	Kitchen	Room
Truck	0.0037	0.0350	0.0485	0.0579
Garden	0.0286	0.0017	0.0144	0.0132
Kitchen	0.0407	0.0141	0.0009	0.0062
Room	0.0346	0.0083	0.0062	0.0024

Summary (off-diagonal, 12 pairs per label):

	color	depth
mean raw ECE (no calibration)	0.539	0.545
mean in-scene ECE (diagonal)	0.0013	0.0022
mean transferred ECE (off-diagonal)	0.0084	0.0256
median transferred ECE	0.0083	0.0215
max transferred ECE (worst pair)	0.0173	0.0579
best transfer pair	kitchen→room 0.0027	kitchen→room 0.0062
worst transfer pair	truck→room 0.0173	truck→room 0.0579

Certificate under transfer — validity is restored by the local split¶

The certificate is computed with the transferred confidence but on B's own calibration split for the conformal threshold. Because the calibrator is only a monotone re-scoring, the split-conformal guarantee depends on the labeled conformal set, not on where the calibrator came from — so a small local conformal set on B restores distribution-free validity even under a foreign calibrator. Result: all 24 off-diagonal transferred certificates are valid (certified = True) at ε = 0.6, α = 0.1. Kept fraction: color 1.00 on every pair; depth 0.914–1.00 (mean 0.979) — i.e. transfer never inflates the certified prune beyond what the local labels support.

P1a verdict — transfer HOLDS for ranking, DEGRADES GRACEFULLY for calibration¶

Selection / pruning quality transfers for free (AUC within ±0.0004 of in-scene on every pair). A single cross-scene calibrator prunes as well as a scene-native one — because the export-time feature ranking is scene-portable.
Absolute calibration degrades gracefully, never catastrophically. A transferred calibrator raises ECE from the in-scene ~0.001–0.002 to ~0.008 (color) / ~0.026 (depth) on average, worst case 0.017 / 0.058 — still 1–2 orders of magnitude below the uncalibrated raw heuristic (~0.54). The depth (occlusion-aware) label transfers roughly 3× worse than color, and Truck is the outlier source and target: every worst pair involves Truck, whose heavy floater population (the depth label is sparsest there) gives it the most idiosyncratic feature→reliability map. Neighbouring indoor Mip-360 scenes (kitchen↔room) transfer best.
The certificate stays valid under transfer as long as a small local conformal set is kept on the target scene.

Honest deployment story: transfer the calibrator (selection quality comes for free and ECE stays 1–2 orders below uncalibrated), but keep a small local conformal split on each scene to restore the certificate — a modest labeling cost that buys back distribution-free validity. A single global calibrator is a viable default; a scene-native calibrator is only worth fitting when absolute-probability ECE (not ranking) is the binding requirement, and it matters most when the source and target scenes are dissimilar (e.g. an object-centric capture like Truck vs an indoor room).

P1b — certificate operating study, all scenes × both labels¶

Sweeps ε over 0.30 → 0.65 (step 0.01, 36 points) at α = 0.1, using the P0 in-scene protocol (calibrator fit on the seed-0 cal half, certificate evaluated on the eval half with calibrated confidence). kept_fraction is monotone non-decreasing in ε (a looser risk budget keeps more), so each scene has a threshold ε* below which the certificate first becomes selective (kept < 1.0).

Kept fraction vs `ε`¶

color label

scene	0.65	0.60	0.55	0.50	0.45	0.40	0.35	0.30
Truck	1.00	1.00	0.86	0.73	0.60	0.46	0.35	0.22
Garden	1.00	1.00	0.93	0.79	0.66	0.53	0.40	0.29
Kitchen	1.00	1.00	0.97	0.84	0.72	0.59	0.47	0.35
Room	1.00	1.00	1.00	1.00	0.95	0.84	0.73	0.60

depth label

scene	0.65	0.60	0.55	0.50	0.45	0.40	0.35	0.30
Truck	1.00	0.90	0.74	0.59	0.43	0.25	0.11	0.00
Garden	1.00	1.00	0.95	0.83	0.69	0.56	0.42	0.29
Kitchen	1.00	1.00	0.99	0.87	0.74	0.62	0.49	0.37
Room	1.00	1.00	1.00	1.00	0.93	0.83	0.69	0.57

Selectivity onset (largest `ε` with `kept < 1.0`)¶

scene	color `ε*`	depth `ε*`	mean eval reliability (color / depth)
Truck	0.59	0.62	0.41 / 0.38
Garden	0.57	0.56	0.43 / 0.44
Kitchen	0.56	0.55	0.44 / 0.45
Room	0.47	0.48	0.53 / 0.52

The onset tracks scene reliability exactly as the theory predicts: ε* sits just above each scene's mean unreliability (1 − mean reliability) plus the Hoeffding safety margin. Room, the most reliable scene (mean reliability ~0.53), only starts pruning at ε ≈ 0.47; Truck, the least reliable (~0.40) and floater-heavy, is the first to become selective (ε ≈ 0.59 color, 0.62 depth — Truck-depth is already selective at the P0 headline ε = 0.6, keeping 90%). Above these onsets the whole set is certified; below them the certificate trades kept fraction for a tighter reliability bound, degrading smoothly (e.g. Truck-color 0.86 → 0.22 as ε goes 0.55 → 0.30). The certificate is doing real work — it is never vacuous once ε drops below the scene's natural reliability floor, and it never over-prunes above it.

Bottom line¶

P1a: a single calibrator transfers — selection/pruning quality for free (rank-invariant), absolute calibration degrades gracefully (ECE stays 1–2 orders below uncalibrated, worst case 0.017 color / 0.058 depth), and the conformal certificate stays valid when a small local conformal split is kept on the target scene. Not a negative: the honest deployment recipe is "ship one calibrator + a small per-scene conformal set."
P1b: the certificate's selective regime is now mapped on all four scenes × both labels; the onset ε* (0.47–0.62) tracks scene reliability, confirming the certificate is neither vacuous nor over-eager.