1 Bayes Risk and Bayes Estimator

1.1 Frequentist Motivation

Consider model ${\displaystyle \mathcal{P}=\{P_{\theta }|\theta \in \mathrm{\Theta }\}}$ for data ${\displaystyle X}$. Loss ${\displaystyle \mathcal{L}(\theta ,d)}$, risk ${\displaystyle \mathcal{R}(\theta ;\delta )={\mathbb{E}}_{\theta }[\mathcal{L}(\theta ,\delta (X))]}$.
The Bayes risk is the average-case risk, integrated w.r.t some measure ${\displaystyle \mathrm{\Lambda }}$, called prior.
For now, assume ${\displaystyle \mathrm{\Lambda }(\mathrm{\Theta })=1}$ (probability measure). Later we will allow it to be improper (${\displaystyle \mathrm{\Lambda }(\mathrm{\Theta })=\mathrm{\infty }}$).

• ${\displaystyle \mathrm{\Lambda }}$ and ${\displaystyle c\mathrm{\Lambda }}$ for ${\displaystyle c>0}$ are functionally equivalent.
• Average risk makes sense even if we don't "believe" ${\displaystyle \theta \sim \mathrm{\Lambda }}$.

(if we assume ${\displaystyle \theta \sim \mathrm{\Lambda },X|\theta \sim P_{\theta }}$) ${\displaystyle \mathbb{E}}$ is now mean w.r.t the joint distribution of ${\displaystyle (\theta ,X)}$.

An estimator ${\displaystyle \delta }$ minimizing ${\displaystyle r_{\mathrm{\Lambda }}(\cdot )}$ is called a Bayes estimator. It depends on ${\displaystyle \mathcal{P},\mathrm{\Lambda },\mathcal{L}}$ (by tower property again, $$r_{\mathrm{\Lambda }}(\delta )=\mathbb{E}[\mathbb{E}[\mathcal{L}(\theta ,\delta (X))|X]].$$)

1.2 Prior and Posterior

• Usual interpretation of ${\displaystyle \mathrm{\Lambda }}$ is prior belief about ${\displaystyle \theta }$ before seeing the data.
• Conditional distribution ${\displaystyle \mathrm{\Lambda }(\theta |X)}$ is called posterior distribution: "belief after seeing the data".

Now we can explicitly define the densities:

Bayes estimator depends on posterior:

Solving Bayes estimator should be "one ${\displaystyle x}$ at a time".

2 Posterior Mean

2.1 Square Error Loss

If ${\displaystyle \mathcal{L}(\theta ,d)=(g(\theta )-d{)}^2}$, then the Bayes estimator is the posterior mean: $$\begin{array}{rl}\mathbb{E}[(g(\theta )-d{)}^2|X]& =\mathbb{E}[(g(\theta )-\mathbb{E}[g(\theta )|X]+\mathbb{E}[g(\theta )|X]-d{)}^2|X]\\ & =\mathrm{Var}(g(\theta )|X)+(\mathbb{E}[g(\theta )|X]-d{)}^2,\end{array}$$ so ${\displaystyle {\delta }_{\mathrm{\Lambda }}(X)=\mathbb{E}[g(\theta )|X]}$.

2.2 Weighted Square Error

If ${\displaystyle \mathcal{L}(\theta ,d)=w(\theta )(g(\theta )-d{)}^2}$ (like ${\displaystyle {\left(\frac{\theta -d}{\theta }\right)}^2}$), then $$\mathbb{E}[(d-g(\theta ){)}^{2}w(\theta )|X]=d^2\mathbb{E}[w(\theta )|X]-2d\mathbb{E}[w(\theta )g(\theta )|X]+\mathbb{E}[w(\theta )g(\theta {)}^2|X],$$ which is minimized at ${\displaystyle d=\frac{\mathbb{E}[w(\theta )g(\theta )|X]}{\mathbb{E}[w(\theta )|X]}}$.

2.3 Other Examples

We can see from several examples that posterior mean is weighted sum of sample mean and prior mean. When ${\displaystyle n}$ is huge, the weight of sample is huge.

If the posterior is from the same family as the prior, we say that prior is conjugate to the likelihood. This is most common in exponential families.

3 Conjugate Priors

Suppose ${\displaystyle X_i|\eta \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }{p}_{\eta }(x)={\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x)-A(\eta )}h(x),\eta \in \mathrm{\Xi }\subset {\mathbb{R}}^s}$, ${\displaystyle i=1,\cdots ,n}$.
For any carrier ${\displaystyle {\lambda }_0(\eta )}$, define ${\displaystyle (s+1)}$ -dim family $${\lambda }_{\mu ,k}(\eta )={\mathrm{e}}^{k{\mu }^{\mathrm{T}}\eta -kA(\eta )-B(\mu ,k)}{\lambda }_0(\eta ),$$ so sufficient statistic is ${\displaystyle \left(\begin{array}{c}\eta \\ -A(\eta )\end{array}\right)}$, and natural parameter is ${\displaystyle \left(\begin{array}{c}k\mu \\ k\end{array}\right)}$.
So $$\begin{array}{rl}\lambda (\eta |x_1,\cdots ,x_n)& {\propto }_{\eta }(\prod _{i=1}^n{\mathrm{e}}^{{\eta }^{\mathrm{T}}T(x_i)-A(\eta )}h(x_i))\cdot {\mathrm{e}}^{k{\mu }^{\mathrm{T}}\eta -kA(\eta )-B(k\mu ,k)}{\lambda }_0(\eta )\\ & {\propto }_{\eta }{\mathrm{e}}^{{(k\mu +\sum _{i=1}^{n}T(x_i))}^{\mathrm{T}}\eta -(k+n)A(\eta )}{\lambda }_0(\eta )\\ & ={\lambda }_{{\mu }_{\mathrm{post}},n+k}(\eta ),\end{array}$$ where ${\displaystyle {\mu }_{\mathrm{post}}=\frac{k\mu +n\bar{T}}{k+n},\bar{T}(X)=\frac{1}{n}\sum _{i=1}^{n}T(X_i)}$. Then $${\mu }_{\mathrm{post}}=\underset{\text{UMVUE from data}}{\underbrace{\bar{T}\cdot \frac{n}{k+n}}}+\underset{\text{“UMVUE” from “pseudo data”}}{\underbrace{\mu \cdot \frac{k}{k+n}}}.$$

3.1 Conjugate Prior Examples