1.1 Nonparametric Estimation

Setting: ${\displaystyle X_1,\cdots ,X_n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }P}$, ${\displaystyle P}$ is any distribution. Want to do inference on some "parameter" (functional) ${\displaystyle \theta (P)}$, e.g.

• ${\displaystyle \theta (P)=\mathrm{median}(P),\mathcal{X}\subset \mathbb{R}}$,
• ${\displaystyle \theta (P)={\lambda }_{max}({\mathrm{Var}}_P(X_i)),\mathcal{X}\subset {\mathbb{R}}^d}$.
• ${\displaystyle \theta (P)=\arg \underset{\theta \in {\mathbb{R}}^d}{min}{\mathbb{E}}_p[(Y_i-{\theta }^{\mathrm{T}}{X}_i{)}^2]}$, ${\displaystyle (X_i,Y_i)\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }P}$.
• ${\displaystyle \theta (P)=\arg \underset{\theta \in \mathrm{\Theta }}{min}{D}_{\mathrm{KL}}(P||P_{\theta })=\arg \underset{\theta \in \mathrm{\Theta }}{max}{\mathbb{E}}_P[l_1(\theta ;X_i)]}$.

Recall the empirical distribution of ${\displaystyle X_1,\cdots ,X_n}$ is ${\displaystyle {\hat{P}}_n=\frac{1}{n}\sum _{i=1}^n{\delta }_{X_i}}$ (${\displaystyle {\hat{P}}_n(A)=\frac{\mathrm{\#}\{i:X_i\in A\}}{n}}$). The plug-in estimator of ${\displaystyle \theta (P)}$ is ${\displaystyle \hat{\theta }=\theta ({\hat{P}}_n)}$:

• Sample median.
• ${\displaystyle {\lambda }_{max}}$ (sample variance)
• OLS estimator
• MLE for ${\displaystyle \{P_{\theta }:\theta \in \mathrm{\Theta }\}}$.

Does plug-in estimator work? Depends.
Does ${\displaystyle {\hat{P}}_n\stackrel{p}{\to }P}$? Depends on what sense of convergence.
Does ${\displaystyle {\hat{P}}_n(A)\stackrel{p}{\to }P(A)}$ for all ${\displaystyle A}$? Yes.
By Glivenko-Cantelli, $$\underset{x}{sup}|{\hat{P}}_n((-\mathrm{\infty },x])-P((-\mathrm{\infty },x])|\stackrel{p}{\to }0$$ for ${\displaystyle \mathcal{X}\subset \mathbb{R}}$. But

${\displaystyle {\hat{P}}_n}$ is non-parametric estimator for ${\displaystyle P}$, e.g. empirical distribution.
Plug-in estimator ${\displaystyle \hat{\theta }=\theta ({\hat{P}}_n)}$ is called bootstrap estimator.

${\displaystyle \theta (P)={\mathrm{median}}_P\left[\frac{1}{\underset{i\ne j}{min}|X_i-X_j|}\right]}$, or ${\displaystyle \theta (P)=\mathbb{E}\left[\underset{1\le i\le n}{max}{X}_i\right]}$.
Say determine the dam's height that has a ${\displaystyle 99\mathrm{\%}}$ chance higher than the storm that year.

1.2 Bootstrap CI

${\displaystyle R_n(X,P)={\hat{\theta }}_n(X)-\theta (P)}$, ${\displaystyle G_{n,P}(r)={\mathbb{P}}_P(\hat{\theta }(X)-\theta (P)\le r)}$. ${\displaystyle r_1,r_2}$ are ${\displaystyle \alpha /2,1-\alpha /2}$ quantiles of ${\displaystyle G_{n,p}}$. Then ${\displaystyle C(X)=[\hat{\theta }(X)-{\hat{r}}_2,\hat{\theta }(X)-{\hat{r}}_1]}$.
We can take ${\displaystyle {\hat{r}}_1=G_{n,{\hat{P}}_n}^{-1}\left(\frac{\alpha }{2}\right)}$.

2 Double Bootstrap

Might have theory telling us, e.g., ${\displaystyle \underset{a<b}{sup}|G_{n,{\hat{P}}_n}([a,b])-G_{n,P}([a,b])|\stackrel{\mathrm{p}}{\to }0}$.
Let ${\displaystyle {\gamma }_{n,P}(\alpha )={\mathbb{P}}_P(C_{n,\alpha }(X)\ni \theta (P))\to 1-\alpha }$. But in finite samples might have ${\displaystyle {\gamma }_{n,P}(\alpha )<1-\alpha }$ (or more).
If ${\displaystyle {\gamma }_{n,P}(0,1)=0.87<0.9}$, double bootstrap:

1. Estimate ${\displaystyle {\gamma }_{n,P}(\cdot )}$ with plug-in ${\displaystyle {\gamma }_{n,{\hat{P}}_n}(\cdot )}$.
2. Use ${\displaystyle C_{n,\hat{\alpha }}(X)}$ where ${\displaystyle \hat{\gamma }(\hat{\alpha })=1-\alpha }$.
   ${\displaystyle \hat{\alpha }={\hat{\gamma }}^{-1}(1-\alpha )}$, ${\displaystyle {\hat{\gamma }}_{n,{\hat{P}}_n}(\cdot )=\hat{\gamma }(\cdot )}$.

${\displaystyle G_{n,{\hat{P}}_n}(r)=\frac{1}{B}\sum _{b=1}^B\mathbf{1}\{R_n^{\ast b}\le r\}}$.
${\displaystyle {\hat{r}}_1=G_{n,{\hat{P}}_n}^{-1}\left(\frac{\alpha }{2}\right)}$.
${\displaystyle C(X)=[\hat{\theta }(X)-{\hat{r}}_2,\hat{\theta }(X)-{\hat{r}}_1]}$.

2.1 Double Bootstrap CI