1 Setting

Setting: ${\displaystyle (x_1,y_1),\cdots ,(x_n,y_n)}$.
For a time series setting, for example, we may take ${\displaystyle x_i=i}$ as the index, and ${\displaystyle y_i}$ be the population at a certain time.
For a linear model, we assume $$y_i={\beta }_0+{\beta }_1{x}_i+{\epsilon }_i,$$ where ${\displaystyle {\beta }_0}$ is the intercept, ${\displaystyle {\beta }_1}$ is the slope, ${\displaystyle {\epsilon }_i}$ is the error term, and assume ${\displaystyle {\epsilon }_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\sigma }^2)}$.
The parameters here are ${\displaystyle {\beta }_0,{\beta }_1,\sigma }$. (${\displaystyle \sigma }$ measures the noise scale). We want to do estimation and uncertainty quantification.

2 Frequentist Inference

The frequentist approach is the MLE . The basic process is

• Calculate MLE, and get estimated parameters
• Determine the distribution of the MLE
• Confidence intervals

• Step 1: The likelihood for a single observation is $$f_{(x_i,y_i)}(x_i,y_i)=f_{y_i|x_i}(y_i)f_{x_i}(x_i)\propto \frac{1}{\sqrt{2\pi }\sigma }\exp [-\frac{(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2}{2{\sigma }^2}].$$ (From the model ${\displaystyle y_i|x_i\sim \mathcal{N}({\beta }_0+{\beta }_1{x}_i,{\sigma }^2)}$) so the general likelihood is $$\begin{array}{rl}& \prod _{i=1}^n\frac{1}{\sqrt{2\pi }\sigma }\exp [-\frac{(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2}{2{\sigma }^2}]\\ =& (2\pi {)}^{-\frac{n}{2}}{\sigma }^{-n}\exp [-\sum _{i=1}^n\frac{(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2}{2{\sigma }^2}].\end{array}$$
  If ${\displaystyle ({\hat{\beta }}_0,{\hat{\beta }}_1)}$ minimize ${\displaystyle S({\beta }_0,{\beta }_1)=\sum _{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2}$, then by taking derivative we have $${\hat{\beta }}_1=\frac{\sum _{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{\sum _{i=1}^n(x_i-\bar{x}{)}^2}=\frac{\sum _{i=1}^n(x_i-\bar{x})y_i}{\sum _{i=1}^n(x_i-\bar{x}{)}^2},{\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}.$$
  Then MLE for ${\displaystyle \sigma ={\hat{\sigma }}_{\mathrm{MLE}}=\sqrt{\frac{1}{n}\sum _{i=1}^n(y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i{)}^2}=\sqrt{\frac{S({\hat{\beta }}_0,\widehat{{\beta }_1})}{n}}}$.
• Step 2: for ${\displaystyle {\hat{\beta }}_1}$, taking all ${\displaystyle x_i}$ as deterministic, since ${\displaystyle y_i\sim \mathcal{N}({\beta }_0+{\beta }_1{x}_i,{\sigma }^2)}$, we have $$\begin{array}{rl}{\hat{\beta }}_1& \sim \mathcal{N}(\frac{\sum _{i=1}^n(x_i-\bar{x})({\beta }_0+{\beta }_1{x}_i)}{\sum _{i=1}^n(x_i-\bar{x}{)}^2},\frac{\sum _{i=1}^n(x_i-\bar{x}{)}^2{\sigma }^2}{{(\sum _{i=1}^n(x_i-\bar{x}{)}^2)}^2})\\ & =\mathcal{N}({\beta }_1,\frac{{\sigma }^2}{\sum _{i=1}^n(x_i-\bar{x}{)}^2}).\end{array}$$

• Step 3: ${\displaystyle {\hat{\beta }}_0,{\hat{\beta }}_1}$ are unbiased, but ${\displaystyle \hat{\sigma }}$ is biased: ${\displaystyle \mathbb{E}({\hat{\sigma }}^2)={\sigma }^2\cdot \frac{n-2}{n}}$. Based on this, ${\displaystyle {\hat{\sigma }}_{\text{unbiased}}^2=\frac{n}{n-2}{\hat{\sigma }}_{\mathrm{MLE}}^2=\frac{S(\widehat{{\beta }_0},\widehat{{\beta }_1})}{n-2}}$.
  Now we have ${\displaystyle \frac{\sqrt{\sum _{i=1}^n(x_i-\bar{x}{)}^2}({\hat{\beta }}_1-{\beta }_1)}{\sigma }\sim \mathcal{N}(0,1)}$, and ${\displaystyle \frac{\sqrt{\sum _{i=1}^n(x_i-\bar{x}{)}^2}({\hat{\beta }}_1-{\beta }_1)}{{\hat{\sigma }}_{\mathrm{unbiased}}}\sim t_{n-2}}$, so 95% interval is $$[{\hat{\beta }}_1-\frac{t_{0.025}{\hat{\sigma }}_{\mathrm{unbiased}}}{\sqrt{\sum _{i=1}^n(x_i-\bar{x}{)}^2}},{\hat{\beta }}_1+\frac{t_{0.025}{\hat{\sigma }}_{\mathrm{unbiased}}}{\sqrt{\sum _{i=1}^n(x_i-\bar{x}{)}^2}}].$$

3 Bayesian Inference

Bayesian involves posterior density $$f_{({\beta }_0,{\beta }_1,\sigma )|(x_i,y_i)}({\beta }_0,{\beta }_1,\sigma )\propto \underset{\text{likelihood}}{\underbrace{f_{(x_i,y_i)|({\beta }_0,{\beta }_1,\sigma )}(x_i,y_i)}}\underset{\text{prior}}{\underbrace{f_{{\beta }_0,{\beta }_1,\sigma }({\beta }_0,{\beta }_1,\sigma )}}.$$
The additional information we need to provide is the prior. We can assume $${\beta }_0,{\beta }_1,\log \sigma \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathrm{Uniform}[-C,C].$$
By transformation formula, ${\displaystyle f_{\sigma }(\sigma )=f_{\log \sigma }(\log \sigma )\frac{1}{\sigma }}$. So the prior density is $$\begin{array}{rl}& f_{{\beta }_0,{\beta }_1,\sigma }({\beta }_0,{\beta }_1,\sigma )=f_{{\beta }_0}({\beta }_0)f_{{\beta }_1}({\beta }_1)f_{\sigma }(\sigma )\\ =& \frac{I\{-C<{\beta }_0<C\}}{2C}\frac{I\{-C<{\beta }_1<C\}}{2C}\frac{I\{-C<\log \sigma <C\}}{2C\sigma }\\ \propto & \frac{1}{\sigma }I\{-C<{\beta }_0,{\beta }_1,\log \sigma <C\}.\end{array}$$
So $$\begin{array}{rl}{f}_{{\beta }_0,{\beta }_1,\sigma |\mathrm{data}}({\beta }_0,{\beta }_1,\sigma )& \propto (2\pi {)}^{-\frac{n}{2}}{\sigma }^{-n}\exp [-\frac{S({\beta }_0,{\beta }_1)}{2{\sigma }^2}]\frac{I\{-C<{\beta }_0,{\beta }_1,\log \sigma <C\}}{\sigma }\\ & \propto {\sigma }^{-n-1}\exp [-\frac{S({\beta }_0,{\beta }_1)}{2{\sigma }^2}]I\{-C<{\beta }_0,{\beta }_1,\log \sigma <C\}.\end{array}$$
If we want to get posterior for ${\displaystyle {\beta }_0,{\beta }_1|\mathrm{data}}$, we integrate over ${\displaystyle \sigma }$: $$\begin{array}{rl}& f_{{\beta }_0,{\beta }_1|\mathrm{data}}({\beta }_0,{\beta }_1)=\int f_{{\beta }_0,{\beta }_1,\sigma |\mathrm{data}}({\beta }_0,{\beta }_1,\sigma )\mathrm{d}\sigma \\ \propto & \mathbf{1}\{-C<{\beta }_0,{\beta }_1<C\}{\int }_{{\mathrm{e}}^{-C}}^{{\mathrm{e}}^C}{\sigma }^{-n-1}\exp (-\frac{S({\beta }_0,{\beta }_1)}{2{\sigma }^2})\mathrm{d}\sigma .\end{array}$$
When ${\displaystyle C}$ is large, ${\displaystyle ({\mathrm{e}}^{-C},{\mathrm{e}}^C)}$ goes to ${\displaystyle (0,\mathrm{\infty })}$. Use change of variable $$s=\frac{\sigma }{\sqrt{S({\beta }_0,{\beta }_1)}},$$ then the integral is $$\begin{array}{rl}& {\int }_0^{\mathrm{\infty }}{\sigma }^{-n-1}\exp (-\frac{S({\beta }_0,{\beta }_1)}{2{\sigma }^2})\mathrm{d}\sigma \\ =& S({\beta }_0,{\beta }_1{)}^{-\frac{n}{2}}{\int }_0^{\mathrm{\infty }}{s}^{-n-1}\exp (-\frac{1}{2s^2})\mathrm{d}s{\propto }_{\beta }S({\beta }_0,{\beta }_1{)}^{-\frac{n}{2}},\end{array}$$ and $$\begin{array}{}\text{(3.1)}& f_{{\beta }_0,{\beta }_1|\mathrm{data}}({\beta }_0,{\beta }_1)\propto \mathbf{1}\{-C<{\beta }_0,{\beta }_1<C\}S({\beta }_0,{\beta }_1{)}^{-\frac{n}{2}}.\end{array}$$
Since ${\displaystyle S({\beta }_0,{\beta }_1)}$ in practical problems are extremely large, to handle numerical issue, we rewrite as $$\begin{array}{}\text{(3.2)}& f_{{\beta }_0,{\beta }_1|\mathrm{data}}({\beta }_0,{\beta }_1)\propto {\left(\frac{S({\hat{\beta }}_0,{\hat{\beta }}_1)}{S({\beta }_0,{\beta }_1)}\right)}^{\frac{n}{2}}\mathbf{1}\{-C<{\beta }_0,{\beta }_1<C\}.\end{array}$$
The density will be concentrated around values s.t. ${\displaystyle S({\beta }_0,{\beta }_1)}$ is close to ${\displaystyle S({\hat{\beta }}_0,{\hat{\beta }}_1)}$.