Open question · OPEN-RETENTION-UNCERTAINTY

How uncertain is a finite-sample retention estimate?

Openpublication: open

Read the full question →

Meaning

Beyond frozen-rule asymptotics, finite-sample guarantees for retention estimates remain open.

Scientific context

Retention is reported as a point estimate. Confidence intervals for it — handling the non-smoothness of hard assignment at cell boundaries — do not exist yet.

Statement

Develop influence-function or bootstrap confidence intervals for reported retention functionals (e.g. geometric-mean retention), handling the non-smoothness of hard assignment at cell boundaries, so retention is reported with error bars rather than as a point estimate. Settled special cases under a frozen rule with an iid equally weighted oracle-score evaluation sample: scalar true score (5 Sep 2026, RETENTION-PLUGIN-CLT-FROZEN-SCALAR, independently audited) and d-dimensional true score with the geometric-mean D-retention (det I_Z / det V)^(1/d) (6 Sep 2026, RETENTION-PLUGIN-CLT-FROZEN-VECTOR, independently audited the same day) - the plug-in is asymptotically normal with an explicit matrix influence function, a consistent plug-in variance and a valid Wald interval iff sigma^2 > 0, where sigma^2 = 0 iff every cell's conditional law sits on an ellipsoid (CE-O7-ELLIPSOID-ZERO-VARIANCE-001 shows atomless such laws exist for d >= 2), and at singular I_Z the plug-in is biased upward at rate n^{-(d-r)/d} (RETENTION-PLUGIN-SINGULAR-ENDPOINT-RATE). Still open: rules refitted on the evaluation sample, where the boundary non-smoothness enters; weighted samples; any uncertainty statement without an oracle score; the profiled D_s retention (a Schur-complement functional); the degenerate limits of n(eta_hat - eta) when sigma^2 = 0 and the joint limit of (n^{(d-r)/d} eta_hat_D, n^{(d-r)/d} sigma_hat/sqrt(n)) at the singular endpoint (the endpoint coverage is measured to stabilise, not proved to converge); and a second-order-corrected or transformed interval for heavy-tailed scores, where the first-order interval is measured to be liberal at n <= 3000.

Closest prior work

Nearest prior work

Relationships

Related settled results

The full chain beneath it

Where it comes from

Raised by

Local graph
Local map of Finite-sample uncertainty for retention estimates4 prerequisites to the left, 0 results it enables or questions it raises to the right, 0 boundary cases below.True retained Fisher information under a score proxyTrue retained Fisher…Normalized retained-information spectrumNormalized retained-i…Frozen-rule scalar retention: plug-in CLT with a consistent influence-function varianceFrozen-rule scalar re…Frozen-rule vector retention: plug-in CLT for the geometric-mean D-retention with a consistent influence-function varianceFrozen-rule vector re…Finite-sample uncertainty for retention estimatesFinite-sample uncerta…
Prerequisites to the left, what it enables or raises to the right, boundary cases below. Open in the map.

The question in full

Finite-sample uncertainty for retention estimates

Every retention number the library reports is a point estimate. Develop influence-function or bootstrap intervals for retention functionals, handling the non-smoothness of hard assignment at cell boundaries. Settled for a frozen rule on an iid oracle-score sample: scalar O6 (RETENTION-PLUGIN-CLT-FROZEN-SCALAR, audited) and vector O7 (RETENTION-PLUGIN-CLT-FROZEN-VECTOR, audited 6 Sep 2026, AUDIT-RETENTION-PLUGIN-VECTOR), both in KNOWN_RESULTS/10-oracle.md. Shipped as the library diagnostic retention_uncertainty (closure step 1, 8 Sep 2026, ADR 0039). Remaining: rules refitted on the evaluation sample (closure step 2); weights; no oracle; the profiled DsD_s retention; the degenerate limits when σ2=0\sigma^2=0; a second-order-corrected interval for heavy-tailed scores. Target claim: OPEN-RETENTION-UNCERTAINTY.

From OPEN_PROBLEMS.md