Open question · OPEN-RETENTION-UNCERTAINTY
How uncertain is a finite-sample retention estimate?
Openpublication: open
Meaning
Beyond frozen-rule asymptotics, finite-sample guarantees for retention estimates remain open.
Scientific context
Retention is reported as a point estimate. Confidence intervals for it — handling the non-smoothness of hard assignment at cell boundaries — do not exist yet.
Statement
Develop influence-function or bootstrap confidence intervals for reported retention functionals (e.g. geometric-mean retention), handling the non-smoothness of hard assignment at cell boundaries, so retention is reported with error bars rather than as a point estimate. Settled special cases under a frozen rule with an iid equally weighted oracle-score evaluation sample: scalar true score (5 Sep 2026, RETENTION-PLUGIN-CLT-FROZEN-SCALAR, independently audited) and d-dimensional true score with the geometric-mean D-retention (det I_Z / det V)^(1/d) (6 Sep 2026, RETENTION-PLUGIN-CLT-FROZEN-VECTOR, independently audited the same day) - the plug-in is asymptotically normal with an explicit matrix influence function, a consistent plug-in variance and a valid Wald interval iff sigma^2 > 0, where sigma^2 = 0 iff every cell's conditional law sits on an ellipsoid (CE-O7-ELLIPSOID-ZERO-VARIANCE-001 shows atomless such laws exist for d >= 2), and at singular I_Z the plug-in is biased upward at rate n^{-(d-r)/d} (RETENTION-PLUGIN-SINGULAR-ENDPOINT-RATE). Still open: rules refitted on the evaluation sample, where the boundary non-smoothness enters; weighted samples; any uncertainty statement without an oracle score; the profiled D_s retention (a Schur-complement functional); the degenerate limits of n(eta_hat - eta) when sigma^2 = 0 and the joint limit of (n^{(d-r)/d} eta_hat_D, n^{(d-r)/d} sigma_hat/sqrt(n)) at the singular endpoint (the endpoint coverage is measured to stabilise, not proved to converge); and a second-order-corrected or transformed interval for heavy-tailed scores, where the first-order interval is measured to be liberal at n <= 3000.
Closest prior work
- Radhakrishnan, Kshirsagar (1981)Influence functions for certain parameters in multivariate analysis
- Romanazzi (1992)Influence in canonical correlation analysis
- Muirhead, Waternaux (1980)Asymptotic distributions in canonical correlation analysis and other multivariate procedures for nonnormal populations
- Fang, Krishnaiah (1982)Asymptotic distributions of functions of the eigenvalues of some random matrices for nonnormal populations
- Seo, Kanda, Fujikoshi (1995)The effects of nonnormality on tests for dimensionality in canonical correlation and MANOVA models
- Sugiura, Fujikoshi (1969)Asymptotic expansions of the non-null distributions of the likelihood ratio criteria for multivariate linear hypothesis and independence
- Magnus, Neudecker (1999)Matrix Differential Calculus with Applications in Statistics and Econometrics
- Anderson (2003)An Introduction to Multivariate Statistical Analysis
- Cramér, Nicewander (1979)Some symmetric, invariant measures of multivariate association
Nearest prior work
- Wishart (1932)A note on the distribution of the correlation ratio
- Cramér (1946)Mathematical Methods of Statistics
- Kendall, Stuart (1961)The Advanced Theory of Statistics, Vol. 2: Inference and Relationship
- Serfling (1980)Approximation Theorems of Mathematical Statistics
- Hampel, Ronchetti, Rousseeuw, Stahel (1986)Robust Statistics: The Approach Based on Influence Functions
- van der Vaart (1998)Asymptotic Statistics
- Kelley (2007)Confidence intervals for standardized effect sizes: theory, application, and implementation
Relationships
Related settled results
- True retained Fisher information under a score proxyPROXY-TRUE-RETAINED-FI
- Normalized retained-information spectrumINFO-RETENTION-SPECTRUM
- Frozen-rule scalar retention: plug-in CLT with a consistent influence-function varianceRETENTION-PLUGIN-CLT-FROZEN-SCALAR
- Frozen-rule vector retention: plug-in CLT for the geometric-mean D-retention with a consistent influence-function varianceRETENTION-PLUGIN-CLT-FROZEN-VECTOR
The full chain beneath it
- Quantized Fisher-information identityFI-QUANT-IDENTITY
- Exact Fisher-information loss decompositionFI-LOSS-DECOMPOSITION
- Rank ceiling for K hard categoriesFI-RANK-CEILING
- D-efficiency relative to unbinned inferenceINFO-D-EFFICIENCY
- D-optimal partition invariance under invertible reparameterizationD-REPARAM-INVARIANCE
Where it comes from
Raised by
Local graph
The question in full
Finite-sample uncertainty for retention estimates
Every retention number the library reports is a point estimate. Develop influence-function or
bootstrap intervals for retention functionals, handling the non-smoothness of hard assignment
at cell boundaries. Settled for a frozen rule on an iid oracle-score sample: scalar O6
(RETENTION-PLUGIN-CLT-FROZEN-SCALAR, audited) and vector O7
(RETENTION-PLUGIN-CLT-FROZEN-VECTOR, audited 6 Sep 2026, AUDIT-RETENTION-PLUGIN-VECTOR), both in KNOWN_RESULTS/10-oracle.md.
Shipped as the library diagnostic retention_uncertainty (closure step 1, 8 Sep 2026, ADR 0039).
Remaining: rules refitted on the evaluation sample (closure step 2); weights; no oracle; the
profiled retention; the degenerate limits when ; a second-order-corrected
interval for heavy-tailed scores. Target claim: OPEN-RETENTION-UNCERTAINTY.
From OPEN_PROBLEMS.md