ScoreQuant

Result · FI-QUANT-IDENTITY

A hard label's information is the covariance of cell mean scores

Establishedpublication: published

Read the proof and evidence →

Meaning

Cell masses and conditional score means determine the retained Fisher information of a fixed rule in a regular model.

Scientific context

If each observation is replaced by its category label, the label carries its own likelihood, and the Fisher information it carries is the variance of the conditional mean of the score given the label. It is a sum over cells of one outer product each — the cell's weighted score sum against itself, divided by the cell's weight — with no approximation and no asymptotics involved.

The closest theoretical predecessor is Barnes, Han and Özgür, who characterized quantized Fisher information geometrically in terms of conditional score means and solved the one-bit Gaussian location problem exactly; the retained-information identity here is attributed to that line of work.

Statement

For Z=q(S), the categorical score is E[S|Z] and I_q=Var(E[S|Z])=Σ_b W_b μ_b μ_b^T=Σ_b m_b m_b^T/W_b.

Assumptions

  • regular model

  • quantizer fixed with respect to theta when differentiating

foundational information representation

Closest prior work

Nearest prior work

Relationships

Enables

In the library

Carried by DOptimality. Reference.

Every object and refusal

Local graph
Local map of Quantized Fisher-information identity0 prerequisites to the left, 8 results it enables or questions it raises to the right, 0 boundary cases below.D leverage inequalityD leverage inequalityD-optimal partition invariance under invertible reparameterizationD-optimal partition i…Full-data efficient-score dominationFull-data efficient-s…Exact Fisher-information loss decompositionExact Fisher-informat…Rank ceiling for K hard categoriesRank ceiling for K ha…Refinement monotonicityRefinement monotonici…Infinitesimal cell-transfer identityInfinitesimal cell-tr…True retained Fisher information under a score proxyTrue retained Fisher…Quantized Fisher-information identityQuantized Fisher-info…
Prerequisites to the left, what it enables or raises to the right, boundary cases below. Open in the map.

Proof

U1. Quantized Fisher-information identity

For Z=q(S)Z=q(S),

∇θlog⁡Pθ(Z=b)∣θ0=E[S∣Z=b]=μb,\nabla_\theta\log P_\theta(Z=b)|_{\theta_0} =E[S\mid Z=b]=\mu_b,

hence

Iq=Var⁡(E[S∣Z])=∑bWbμbμb⊤=∑bmbmb⊤Wb.\boxed{ I_q =\operatorname{Var}(E[S\mid Z]) =\sum_bW_b\mu_b\mu_b^\top =\sum_b\frac{m_bm_b^\top}{W_b}. }

Closest direct literature: score-function quantization; Barnes–Han–Özgür.

From Universal information structure · chapter contents