3.1 Unbiased Estimation

#UnbiasedEstimation #Convex #RaoBlackwellization #UMVUE #JensenInequality #TowerProperty

1 Unbiased Estimation

Recall the principle for estimator picking, we can focus on a certain kind of estimators, that is unbiased estimators.

Unbiased Estimation

$g (θ)$ is an unbiased estimation if $E_{θ} δ = g (θ), \forall θ$ .

Unbiased estimation is especially convenient in models with a complete sufficient $T (X)$ . In that case

There is at most one unbiased $δ (T (X))$ :
If $δ_{1}, δ_{2}$ are both unbiased, then $δ_{1} - δ_{2} \overset{a . s .}{=} 0$ .
If an unbiased estimator exists, it uniformsly minimizes risk for any convex loss function.

2 Convex Loss

We have defined convex functions here. And we have derived that if $f$ is convex, for any RV $X$ , $f (E X) \leq E f (X),$ and if $f$ is strictly convex, " $=$ " holds iff $X$ is constant.
The proof can be found here with Jensen's inequality.

We say a loss function $L (θ, d)$ is (strictly) convex if it is a (strictly) convex function of the estimand $d$ , given that parameter $θ$ is fixed.

#MSE is a best-known convex loss function. Recall the bias-variance tradeoff ${MSE}_{θ} (δ) = {Bias}_{θ} (δ)^{2} + {Var}_{θ} (δ (X)) .$
If $δ (X)$ is unbiased, then ${MSE}_{θ} (δ) = {Var}_{θ} (δ (X))$ . So minimizing the risk is equal to finding one with the least variance.

3 Rao-Blackwell Theorem

Theorem (Rao-Blackwell)

$T (X)$ is sufficient, and $δ (X)$ is any estimator. Let $\overset{―}{δ} (T (X)) = E [δ (X) | T (X)]$ (no $θ$ under $E$ by sufficiency). If $L (θ, δ)$ is convex, then $R (θ; \overset{―}{δ}) \leq R (θ; δ)$ .
If strictly convex, then "=" iff $δ (X) \overset{a . s .}{=} f (T)$ .

Proof

We can simply use Jensen's inequality and tower property to show that $\begin{aligned} R (θ; \overset{―}{δ}) & = E_{θ} [L (θ, E [δ | T])] \leq E_{θ} [E [L (θ, δ) | T]] \\ = E_{θ} [L (θ, δ)] = R (θ; \overset{―}{δ}), \end{aligned}$

$\overset{―}{δ}$ is called Rao-Blackwellization of $δ$ .
So for convex loss, the Rao-Blackwell Theorem lets us to focus only the estimators running through $T (X)$ . This offers us a way to construct the improved estimator.

4 UMVU Estimators

4.1 Existence

U-Estimable

$g (θ)$ is U-estimable if $\exists δ (X)$ with $E_{θ} δ (X) = g (θ)$ . I.e. there exists an unbiased estimator.

Theorem

For a model $P = {P_{θ} | θ \in Θ}$ , assume

$T (X)$ is complete sufficient;
$g (θ)$ is U-estimable;
$L (θ, d)$ is convex.

Then there is a unique unbiased estimator of the form $δ^{*} (T (X))$ , which uniformly minimizes $R$ among (or, dominates) all unbiased estimators.^[1]

Proof

Existence: $\forall$ unbiased $δ_{0} (X)$ for $g (θ)$ , then its Rao-Blackwellization $δ^{*} (T) = E [δ_{0} | T]$ is also unbiased, because $E_{θ} δ^{*} (T) = E_{θ} [E [δ_{0} | T]] = E_{θ} δ_{0} = g (θ) .$
Uniqueness: Any other estimator of the form $\tilde{δ} (T)$ must be almost surely equal to $δ (T)$ , by completeness: if $f (t) = δ (t) - \tilde{δ} (t)$ , then both estimators being unbiased means $E_{θ} f (T) = g (θ) - g (θ) = 0 \Rightarrow f (T (X)) \overset{a . s .}{=} 0,$ so $δ^{*}$ is unique.
Optimality: The result above implies that every unbiased estimator has the same Rao-Blackwellization $δ (T)$ . So by Rao-Blackwell theorem, $δ (T)$ dominates every other unbiased estimator for any strictly convex loss function, unless the estimator $\tilde{δ} \overset{a . s .}{=} δ$ .

We give a definition for such an estimator:

UMVUE

$δ (X)$ is uniform minimum-variance unbiased estimator (UMVUE) if

$δ$ is unbiased,
$\forall$ other unbiased $\tilde{δ}$ , ${Var}_{θ} δ \leq {Var}_{θ} \tilde{δ}, \forall θ$ .

Since $MSE (θ; δ) = {Var}_{θ} (δ (X))$ for any unbiased estimator $δ$ , and the square loss is strictly convex, then the above theorem implies the existence of a unique UMVUE for any U-estimable $g (θ)$ , whenever we have a complete sufficient statistic.

4.2 Ways of Finding UMVUE

The above theorem suggests two strategies for finding UMVUE:

@ Solve directly for an unbiased estimator based on $T$ .
@ Find any unbiased estimator, and then Rao-Blackwellize it.

Example (Poisson)

Let $X_{1}, \dots, X_{n} \sim Poisson (θ)$ , $g (θ) = e^{- θ}$ .
We've shown here that the sufficient statistic for the model is $T (X) = \sum_{i = 1}^{n} X_{i} \sim Poisson (n θ)$ . Since full-rank, it is complete sufficient. So the p.m.f is $p_{θ} (t) = \frac{e^{- n θ} (n θ)^{t}}{t!}$ .

Strategy 1: any unbiased estimator $δ (t)$ , by definition $e^{- θ} = E_{θ} δ (T) = \sum_{t = 0}^{\infty} δ (t) \frac{e^{- n θ} (n θ)^{t}}{t!},$ then $\sum_{t = 0}^{\infty} δ (t) \frac{n^{t} θ^{t}}{t!} = e^{(n - 1) θ} = \sum_{t = 0}^{\infty} \frac{((n - 1) θ)^{t}}{t!} .$ Compare the coefficients, we have $δ (t) = {(\frac{n - 1}{n})}^{t}, \forall t \in N$ , so UMVUE is $δ (T) = {(\frac{n - 1}{n})}^{T}$ .
Strategy 2: notice that $g (θ) = e^{- θ} = P_{θ} (X_{1} = 0)$ , so an unbiased estimator is $δ_{0} (X) = 1 {X_{1} = 0}$ .^[2] Now we Rao-Blackwellize it: $E [1 {X_{1} = 0} | T (X) = t] = \sum_{\sum_{i = 2}^{n} X_{i} = t} {(\frac{1}{n})}^{t} \frac{t!}{x_{2}! \dots x_{n}!} = {(\frac{n - 1}{n})}^{t} .$ Here we use the fact that $(X_{1}, \dots, X_{n}) | \sum_{i = 1}^{n} X_{i} = t \sim Multinomial (t, (n^{- 1}, \dots, n^{- 1}))$ . (Check Poissonization of multinomial distribution) So $δ (T) = {(\frac{n - 1}{n})}^{T}$ .

Example (Uniform Scale Family)

$X_{1}, \dots, X_{n} \sim U [0, θ], θ > 0$ . We want to estimate $θ$ . $T = X_{(n)}$ is complete sufficient.

Strategy 1: the p.d.f for $X_{(n)}$ is $p_{θ} (t) = \frac{n t^{n - 1}}{θ^{n}} \cdot 1 {0 < t < θ}$ . So $E_{θ} [T] = \int_{0}^{θ} \frac{n t^{n}}{θ^{n}} d t = \frac{n θ}{n + 1},$ so UMVUE is $\frac{(n + 1) T}{n}$ .
Strategy 2: $2 X_{1}$ is unbiased. So $E [2 X_{1} | T] = 2 T \cdot \frac{n + 1}{2 n} = \frac{(n + 1) T}{n} .$

As usual "uniqueness" here is in the sense of $P - a . s$ . ↩︎
Check that $E [δ_{0} (X_{1})] = e^{- θ} \cdot 1 + 0 \cdot P_{θ} (X_{1} \neq 0) = e^{- θ}$ . ↩︎