1 Score Function

As a motivation, we consider the exponential family $$p(x;\theta )={\mathrm{e}}^{\eta (\theta {)}^{\mathrm{T}}T(x)-A(\eta (\theta ))}h(x),\eta :\mathbb{R}\to {\mathbb{R}}^2,\mathrm{\Xi }=\{\eta (\theta )|\theta \in \mathbb{R}\}.$$For this family,

• ${\displaystyle T(X)}$ is complete sufficient.
• ${\displaystyle T(X)}$ is minimal.
• ${\displaystyle {\mathbb{P}}_{\theta }(T(X)=t)={\mathrm{e}}^{{\theta }^{\mathrm{T}}t-A(\theta )}}$.
• ${\displaystyle {\mathbb{E}}_{\theta }[T(X)]=A^{\prime }(\theta )}$.

If ${\displaystyle \eta (\theta )}$ is non-linear, then ${\displaystyle T(X)}$ is minimal. The tangent vector is ${\displaystyle \dot{\eta }({\theta }_0)=\frac{\mathrm{d}\eta }{\mathrm{d}\theta }({\theta }_0)}$. So fix ${\displaystyle {\theta }_0}$, we define the tangent family ${\displaystyle \mathrm{\Xi }=\{\eta ({\theta }_0)+\epsilon \dot{\eta }({\theta }_0)|\epsilon \in \mathbb{R}\}}$, where density will become $$\begin{array}{rl}{q}_{\epsilon }(x)& ={\mathrm{e}}^{(\eta ({\theta }_0)+\epsilon \dot{\eta }({\theta }_0){)}^{\mathrm{T}}T(x)-A(\eta ({\theta }_0)+\epsilon \dot{\eta }({\theta }_0))}h(x)\\ & ={\mathrm{e}}^{\epsilon \dot{\eta }({\theta }_0{)}^{\mathrm{T}}(T(x)-{\mathbb{E}}_{{\theta }_0}T(x))-B(\epsilon )}k(x).\end{array}$$
Here denote ${\displaystyle S_{{\theta }_0}(x)=\dot{\eta }({\theta }_0{)}^{\mathrm{T}}(T(x)-{\mathbb{E}}_{{\theta }_0}T(x))}$ is complete sufficient for tangent family at ${\displaystyle {\theta }_0}$. We derive score function from here.

Assume ${\displaystyle \mathcal{P}}$ has densities ${\displaystyle p_{\theta }}$ w.r.t ${\displaystyle \mu }$, ${\displaystyle \mathrm{\Theta }\subset {\mathbb{R}}^d}$. The common support ${\displaystyle \{x|p_{\theta }(x)>0\}}$ is the same for all ${\displaystyle \theta }$. Recall ${\displaystyle l(\theta ;x)=\log p_{\theta }(x)}$.

It plays a key role in many areas of statistics, especially in asymptotics.

We can think of as "local complete sufficient statistic", i.e. $$p_{{\theta }_0+\eta }(x)={\mathrm{e}}^{l({\theta }_0+\eta ;x)}{p}_{{\theta }_0}(x)\approx {\mathrm{e}}^{{\eta }^{\mathrm{T}}\mathrm{\nabla }l({\theta }_0;x)}{p}_{{\theta }_0}(x),\eta \approx 0.$$

2 Differential Identities and Fisher Information

We assume as usual, enough regularity so that differentiation and integration are exchangeable.
Since $$1=\int {\mathrm{e}}^{l(\theta ;x)}\mathrm{d}\mu (x),$$ we take partial derivative w.r.t ${\displaystyle {\theta }_j}$: $$\begin{array}{}\text{(1.1)}& 0=\int \frac{\partial l}{\partial {\theta }_j}(\theta ){\mathrm{e}}^{l(\theta )}\mathrm{d}\mu ={\mathbb{E}}_{\theta }[\mathrm{\nabla }l(\theta ;x)].\end{array}$$
(this is only true if ${\displaystyle \theta }$ is the same). Then take partial derivative w.r.t ${\displaystyle {\theta }_k}$: $$\begin{array}{rl}0& =\int (\frac{{\partial }^{2}l}{\partial {\theta }_j\partial {\theta }_k}(\theta )+\frac{\partial l}{\partial {\theta }_j}\frac{\partial l}{\partial {\theta }_k}(\theta )){\mathrm{e}}^{l(\theta )}\mathrm{d}\mu \\ & ={\mathbb{E}}_{\theta }\left[\frac{{\partial }^{2}l}{\partial {\theta }_j\partial {\theta }_k}\right]+{\mathbb{E}}_{\theta }\left[\frac{\partial l}{\partial {\theta }_j}\frac{\partial l}{\partial {\theta }_k}\right].\end{array}$$
The second term here is the covariance, which leads to $$\begin{array}{}\text{(1.2)}& {\mathrm{Var}}_{\theta }[\mathrm{\nabla }l(\theta ;x)]=-{\mathbb{E}}_{\theta }[{\mathrm{\nabla }}^{2}l(\theta ;x)]=J(\theta ).\end{array}$$

It is possible to extend this definition to certain cases where ${\displaystyle l}$ is not even differentiable, like Laplace location family, but for our purposes we can just assume "sufficient regularity".

Now we try with another statistic ${\displaystyle \delta (X)}$. Let ${\displaystyle g(\theta )={\mathbb{E}}_{\theta }[\delta (X)]=\int \delta {\mathrm{e}}^{l(\theta )}\mathrm{d}\mu }$ (i.e. ${\displaystyle \delta }$ is an unbiased estimator), then $$\mathrm{\nabla }g(\theta )=\int \delta \mathrm{\nabla }l{\mathrm{e}}^l\mathrm{d}\mu ={\mathbb{E}}_{\theta }[\delta (X)\mathrm{\nabla }l(\theta ;X)]={\mathrm{Cov}}_{\theta }(\delta (X),\mathrm{\nabla }l(\theta ;X)).$$ (Since ${\displaystyle \mathbb{E}\mathrm{\nabla }l=0}$.)

3 CRLB

Now we combine the results with Cauchy-Schwarz inequality.

• When ${\displaystyle d=1}$, $$\begin{array}{rl}& {\mathrm{Var}}_{\theta }(\delta )\cdot {\mathrm{Var}}_{\theta }(\dot{l}(\theta ;x))\ge {\mathrm{Cov}}_{\theta }(\delta ,\dot{l}(\theta ;x){)}^2\\ \text{(2.1)}& \Rightarrow & {\mathrm{Var}}_{\theta }(\delta )\ge \frac{\dot{g}(\theta {)}^2}{J(\theta )}.\end{array}$$

• When ${\displaystyle d>1}$, ${\displaystyle \theta \in {\mathbb{R}}^d}$, ${\displaystyle g(\theta ),\delta (x)\in \mathbb{R}}$, $$\begin{array}{}\text{(2.2)}& {\mathrm{Var}}_{\theta }(\delta )\ge \mathrm{\nabla }g(\theta {)}^{\mathrm{T}}J(\theta {)}^{-1}\mathrm{\nabla }g(\theta ).\end{array}$$

Interpretation: if ${\displaystyle g(\theta )}$ is estimand, no unbiased estimator can have smaller variance than ${\displaystyle \mathrm{\nabla }g(\theta {)}^{\mathrm{T}}J(\theta {)}^{-1}\mathrm{\nabla }g(\theta )}$.

The upper bound combined, is called Cramer-Rao Lower Bound (CRLB).

4 Efficiency

CRLB is not necessarily attainable. We define the efficiency of an unbiased estimator as $${\mathrm{eff}}_{\theta }(\delta )=\frac{\mathrm{CRLB}}{{\mathrm{Var}}_{\theta }(\delta )}\le 1.$$
If ${\displaystyle g(\theta )=\theta }$, by (2.1), ${\displaystyle \mathrm{CRLB}=\frac{1}{J(\theta )}}$, so $${\mathrm{eff}}_{\theta }(\delta )=\frac{1}{J(\theta ){\mathrm{Var}}_{\theta }(\delta )}.$$
We say ${\displaystyle \delta (X)}$ is efficient if ${\displaystyle {\mathrm{eff}}_{\theta }(\delta )=1,\mathrm{\forall }\theta }$.
Plug in CRLB's expression: $${\mathrm{eff}}_{\theta }(\delta (X))=\frac{{\mathrm{Cov}}^2(\delta (X),\dot{l}(\theta ,X))}{{\mathrm{Var}}_{\theta }(\delta (X)){\mathrm{Var}}_{\theta }(\dot{l}(\theta ))}={\mathrm{Corr}}_{\theta }^2(\delta (X),\dot{l}(\theta ;X)).$$
So ${\displaystyle \delta (X)}$ is efficient iff ${\displaystyle {\mathrm{Corr}}_{\theta }(\delta (X),\mathrm{\nabla }l(\theta ;X))=1,\mathrm{\forall }\theta }$.

This is rarely achieved in finite samples but we can approach it asymptotically as ${\displaystyle n\to \mathrm{\infty }}$.

5 Fisher Information as Local Metric

Recall KL divergence $$D_{\mathrm{KL}}(p||q)={\mathbb{E}}_p[\log p(X)-\log q(X)]=\int \log \left(\frac{p}{q}\right)p\mathrm{d}\mu$$ is used to describe distance between two distributions.
For parametric model, $$D_{\mathrm{KL}}({\theta }^{\ast }||\theta )=D_{\mathrm{KL}}(p_{{\theta }^{\ast }}||p_{\theta })=\int (l({\theta }^{\ast })-l(\theta )){\mathrm{e}}^{l({\theta }^{\ast })}\mathrm{d}\mu .$$ Since $$\begin{array}{rl}\frac{\partial }{\partial {\theta }_j}{D}_{\mathrm{KL}}({\theta }^{\ast }||\theta )& =-\int \frac{\partial l}{\partial {\theta }_j}(\theta ){\mathrm{e}}^{l({\theta }^{\ast })}\mathrm{d}\mu =0\text{ at }\theta ={\theta }^{\ast },\\ \frac{{\partial }^2}{\partial {\theta }_j\partial {\theta }_k}{D}_{\mathrm{KL}}({\theta }^{\ast }||\theta )& =-\int \frac{{\partial }^{2}l}{\partial {\theta }_j\partial {\theta }_k}(\theta ){\mathrm{e}}^{l({\theta }^{\ast })}\mathrm{d}\mu \\ & =J({\theta }^{\ast }{)}_{jk}>0\text{ at }\theta ={\theta }^{\ast },\end{array}$$ KL divergence is maximized at ${\displaystyle \theta ={\theta }^{\ast }}$.

We can use Taylor expansion to show that $$D_{\mathrm{KL}}({\theta }^{\ast }||\theta )\approx \frac{(\theta -{\theta }^{\ast }{)}^2}{2}J({\theta }^{\ast }),\theta \approx {\theta }^{\ast }.$$