1 Introduction

In the above example, we want to test ${\displaystyle H_0:{\beta }_1=0}$. So we need to condition on ${\displaystyle X_{\mathrm{\setminus }1}^{\mathrm{T}}y}$ (by discuss in testing with nuisance parameters). But that would condition on ${\displaystyle Y}$.
If we want to estimate ${\displaystyle \beta }$, UMVU generically doesn't exist; Bayes needs prior on ${\displaystyle \beta \in {\mathbb{R}}^d}$.
Software packages use general purpose asymptotic methods: $$\begin{array}{rl}{\hat{\beta }}_{\mathrm{MLE}}(x,y)& =\arg \underset{\beta \in {\mathbb{R}}^d}{max}{p}_{\beta }(y|x)\\ & =\arg \underset{\beta \in {\mathbb{R}}^d}{max}{\beta }^{\mathrm{T}}{X}^{\mathrm{T}}y-A(\beta ;x_i).\end{array}$$
Asymptotically, ${\displaystyle {\hat{\beta }}_{\mathrm{MLE}}\approx \mathcal{N}(\beta ,J(\beta {)}^{-1})}$ (recall Fisher information).

${\displaystyle \beta }$ is derived because ${\displaystyle {\hat{\beta }}_{\mathrm{MLE}}}$ is unbiased, and we have $${\mathrm{\nabla }}^{2}l(\hat{\beta };X,y)\approx {\mathbb{E}}_{\beta }[{\mathrm{\nabla }}^{2}l(\beta ;X,y)]=J(\beta )$$ and $$\hat{\mathrm{\Sigma }}=(-{\mathrm{\nabla }}^{2}l(\hat{\beta }{)}^{-1})\approx \mathrm{\Sigma }(\beta )=J(\beta {)}^{-1}.$$

So ${\displaystyle Z_j=\frac{{\hat{\beta }}_j-{\beta }_j}{{\hat{\sigma }}_j}\approx \mathcal{N}(0,1)}$. And for test ${\displaystyle H_0:{\beta }_j=0}$, reject if ${\displaystyle \frac{{\hat{\beta }}_j}{{\hat{\sigma }}_j}}$ is large/small/extreme.
We can invert ^[1]: ${\displaystyle |Z_j|<z_{\alpha /2}⟺{\beta }_j\in {\hat{\beta }}_j\pm z_{\alpha /2}{\hat{\sigma }}_j}$.

So far, everything has finite-sample, often using special properties of model ${\displaystyle \mathcal{P}}$ (like exponential family) to do exact calculations.
For "general" models, exact calculations may be intractable or impossible. But we may be able to approximate our problem with a simpler problem in which calculations are easy.
Typically approximate by Gaussian, by taking limit of number of observations. But this is only interesting if approximation is good for "reasonable" sample size.

2 Probability Recall

2.1 Convergence

Let ${\displaystyle X_1,X_2,\cdots \in {\mathbb{R}}^d}$ be a sequence of random vectors. We care about two kinds of convergence:

• Convergence in probability: ${\displaystyle X_n\approx \mathrm{Const}}$;
• Convergence in distribution: ${\displaystyle X_n\approx {\mathcal{N}}_d(0,I_d)}$ usually.

In a sequence of statistical models ${\displaystyle {\mathcal{P}}_n=\{P_{n,\theta }:\theta \in \mathrm{\Theta }\}}$ with ${\displaystyle X_n\sim P_{n,\theta }}$, we say ${\displaystyle {\delta }_n(X_n)}$ is consistent for ${\displaystyle g(\theta )}$ if $${\delta }_n(X_n)\stackrel{P_{\theta }}{\to }g(\theta ),$$ meaning $${\mathbb{P}}_{\theta }(||{\delta }_n(X_n)-g(\theta )||>\epsilon )\to 0.$$ Usually we omit the index ${\displaystyle n}$, because sequence can be implicit.

2.2 Limit Theorems

Denote ${\displaystyle {\bar{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i}$. We are familiar with

• WLLN : ${\displaystyle {\bar{X}}_n\stackrel{p}{\to }\mu }$; & SLLN: ${\displaystyle {\bar{X}}_n\stackrel{\mathrm{a}.\mathrm{s}.}{\to }\mu }$.
• CLT: ${\displaystyle \sqrt{n}({\bar{X}}_n-\mu )\Rightarrow \mathcal{N}(0,\epsilon )}$.

3 Continuous Mapping, Delta Method

In general, we can do higher-order Taylor expansions for delta method if derivatives is ${\displaystyle 0}$.

If ${\displaystyle f^{\prime }(\mu )=0}$, use second-order term $$n(f(X_n)-f(\mu ))\approx \frac{f^{″}(\mu )}{2}(\sqrt{n}(X_n-\mu ){)}^2\approx \frac{f^{″}(\mu ){\sigma }^2}{2}{\chi }_1^2.$$