1 Hypothesis Testing

Assume we have model ${\displaystyle \mathcal{P}=\{P_{\theta }:\theta \in \mathrm{\Theta }\}}$ (as usual, may be nonparametric). We have two competing hypothesis about ${\displaystyle \theta }$:

• Null hypothesis: ${\displaystyle H_0:\theta \in {\mathrm{\Theta }}_0}$.
• Alternative hypothesis: ${\displaystyle H_1:\theta \in {\mathrm{\Theta }}_1}$.

Here ${\displaystyle {\mathrm{\Theta }}_0\cap {\mathrm{\Theta }}_1=\varnothing ,{\mathrm{\Theta }}_0\cup {\mathrm{\Theta }}_1=\mathrm{\Theta }}$ (${\displaystyle {\mathrm{\Theta }}_0,{\mathrm{\Theta }}_1}$ are disjoint).

A hypothesis is called simple if it fully specifies data distribution (like ${\displaystyle \theta ={\theta }_0}$ in the above examples).
Now we want to use the data ${\displaystyle X\sim P_{\theta }}$ to determine whether ${\displaystyle H_0}$ or ${\displaystyle H_1}$ is true.

• The Bayesian answer is simple: $$\begin{array}{r}\mathrm{\Lambda }({\mathrm{\Theta }}_0|X)=\mathbb{P}(\theta \in {\mathrm{\Theta }}_0|X),\\ \mathrm{\Lambda }({\mathrm{\Theta }}_1|X)=\mathbb{P}(\theta \in {\mathrm{\Theta }}_1|X).\end{array}$$
  But this is undesirable if prior information is rare (like for drug companies, scientists, etc.)

• The frequentist answer is to come up with a decision rule based on data. We can either

  • Reject ${\displaystyle H_0}$ (conclude that ${\displaystyle H_0}$ is implausible and ${\displaystyle H_1}$ must be true), or
  • Accept ${\displaystyle H_0}$ (go on believing ${\displaystyle H_0}$)

  Normally we set ${\displaystyle H_0}$ the choice to be disconfirmed. Like when convicting a crime, ${\displaystyle H_0}$: suspect is innocent.

2 The Test Function

We describe a test by its critical/test function $$\begin{array}{r}\varphi (x)=\{\begin{array}{rl}& 0,\text{accept }{H}_0,\\ & \gamma \in (0,1),\text{reject }{H}_0\text{ with probability }\gamma ,\\ & 1,\text{reject }{H}_0.\end{array}\end{array}$$

${\displaystyle \gamma }$ part is to "top off" the Type I error rate if ${\displaystyle T(X)}$ is discrete and ${\displaystyle {\mathbb{P}}_{{\theta }_0}(T(X)>c_{\alpha })<\alpha }$. In practice, we will skip the random ${\displaystyle \gamma }$ part.

Therefore we partition ${\displaystyle \mathcal{X}}$ into rejection region ${\displaystyle R=\{x\in \mathcal{X}|\varphi (x)=1\}}$, and acceptance region ${\displaystyle A=\{x\in \mathcal{X}|\varphi (x)=0\}}$.
We will define a test statistic ${\displaystyle T(X)}$ and some critical threshold ${\displaystyle c\in \mathbb{R}}$. So ${\displaystyle \varphi }$ rejects for large ${\displaystyle T(X)}$, if $$\varphi (x)=\{\begin{array}{rl}& 0,T(x)<c,\\ & \gamma \in (0,1),T(x)=c,\\ & 1,T(x)>c.\end{array}$$
Or ${\displaystyle \varphi }$ rejects for extreme ${\displaystyle T(X)}$, if $$\varphi (x)=\{\begin{array}{rl}& 0,|T(x)|<c,\\ & \gamma \in (0,1),|T(x)|=c,\\ & 1,|T(x)|>c.\end{array}$$

2.1 Significance Level, Power

Inevitably deduction produces error. There are two types of error we make:

• Type I error (False Positive): we reject ${\displaystyle H_0}$ when it is actually true.

• Type II error (False Negative): we fail to reject ${\displaystyle H_0}$ when it is false.

Our goal is to minimize type II error as small as we can, while control type I below a pre-specified value ${\displaystyle \alpha \in [0,1]}$.
Define power function $$\beta (\theta )={\mathbb{E}}_{\theta }[\varphi (X)]={\mathbb{P}}_{\theta }(\text{reject }{H}_0).$$
We can explicitly combine power function with type I/II error:

So our goal can be expressed as $$\begin{array}{r}\underset{\varphi }{\max }{\beta }_{\varphi }(\theta ),\theta \in {\mathrm{\Theta }}_1,\\ \mathrm{s}.\mathrm{t}.{\beta }_{\varphi }(\theta )\le \alpha ,\theta \in {\mathrm{\Theta }}_0.\end{array}$$
We say ${\displaystyle \varphi }$ is a level ${\displaystyle -\alpha }$ test if ${\displaystyle \underset{\theta \in {\mathrm{\Theta }}_0}{sup}{\beta }_{\varphi }(\theta )\le \alpha }$. (if strictly below ${\displaystyle \alpha }$, we say it is conservative). We commonly use ${\displaystyle \alpha =0.05}$.

2.2 Example: ${\displaystyle {\displaystyle Z-}}$ test

Assume we observe ${\displaystyle Z(X)\sim \mathcal{N}(\theta ,1)}$. We use the right-tailed test ${\displaystyle {\varphi }_1(z)=\mathbf{1}\{z>z_{\alpha }\}}$ that rejects for large ${\displaystyle Z}$. Here ${\displaystyle z_{\alpha }={\mathrm{\Phi }}^{-1}(1-\alpha )}$ is the upper ${\displaystyle \alpha }$ quantile of the ${\displaystyle \mathcal{N}(0,1)}$ distribution (top xx%).

If we want to test the two-sided hypothesis, we use two-tailed test ${\displaystyle {\varphi }_2(z)=\mathbf{1}\{|z|>z_{\alpha /2}\}}$.
#？

3 Optimal Testing

3.1 Likelihood Ratio Test

We start with the simplest case: $$\begin{array}{}\text{(3.1)}& H_0:X\sim P_0\text{ vs }{H}_1:X\sim P_1.\end{array}$$ WLOG assume ${\displaystyle P_0,P_1}$ have densities ${\displaystyle p_0,p_1}$ w.r.t common dominating measure ${\displaystyle \mu }$.
Define $$\mathrm{LR}(X)=\frac{p_1(X)}{p_0(X)}.$$

We should set ${\displaystyle c_{\alpha }}$ as the upper- ${\displaystyle \alpha }$ quantile of the distribution of ${\displaystyle \mathrm{LR}(X)}$, i.e. $${\mathbb{P}}_0(\mathrm{LR}(X)>c_{\alpha })<\alpha \le {\mathbb{P}}_0(\mathrm{LR}(X)\ge c_{\alpha }).$$ So we can set $$\gamma =\frac{\alpha -{\mathbb{P}}_0(\mathrm{LR}(X)>c)}{{\mathbb{P}}_0(\mathrm{LR}(X)=c)}.$$

By discreteness, we can't exactly have Type I error be ${\displaystyle \alpha =0.05}$. For example, ${\displaystyle n=100,\alpha =0.05}$, take ${\displaystyle c_{\alpha }=58}$ (${\displaystyle 0.95}$ quantile under the null). But ${\displaystyle {\mathbb{P}}_{0.5}(X>58)=0.044<0.05}$. So we randomize at the boundary by setting $$\gamma =\frac{0.05-0.044}{{\mathbb{P}}_{0.5}(X=58)}=0.26,$$ and ${\displaystyle {\varphi }^{\ast }}$ rejects with probability ${\displaystyle 0.26}$ if ${\displaystyle X=58}$.
But in practice we seldom use randomized tests. So consider the conservative test ${\displaystyle {\varphi }^{\ast }(X)=\mathbf{1}\{X>58\}}$.

3.2 UMP Tests

MLR is sufficient for finding a UMP test: