1 Time Series Models, Stationarity

For a stationary ${\displaystyle \{y_t\}}$, we can define ${\displaystyle \gamma (h)=\mathrm{Cov}(y_t,y_{t+h})}$. Call ${\displaystyle \gamma (h)}$ the ACVF (AutoCovariance Function). Observe that $$\gamma (0)=\mathrm{Cov}(y_t,y_t),\gamma (h)=\gamma (-h),$$ so ${\displaystyle \gamma (h)}$ is a symmetric function of ${\displaystyle h}$, so we only consider nonnegative ${\displaystyle h}$.

Define ACF (AutoCorrelation Function): $$\rho (h)=\frac{\mathrm{Cov}(y_t,y_{t+h})}{\sqrt{\mathrm{Var}(y_t)\mathrm{Var}(y_{t+h})}}=\frac{\gamma (h)}{\gamma (0)}.$$
So ${\displaystyle \rho (0)=1,\rho (h)=\rho (-h)}$.

Note that

• Stationarity refers to the model not the data.
• Not all time series models are stationary.
• ACVF and ACF are only defined for stationary models.

2 MA Models

The Moving Average Model (MA) with order ${\displaystyle q}$ is defined by $$\begin{array}{}\text{(2.1)}& y_t=\mu +{\epsilon }_t+{\theta }_1{\epsilon }_{t-1}+\cdots +{\theta }_q{\epsilon }_{t-q},\end{array}$$ where ${\displaystyle {\epsilon }_t\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\sigma }^2)}$. Denote as ${\displaystyle \mathrm{MA}(q)}$. There are ${\displaystyle q+2}$ unknown parameters: ${\displaystyle \mu ,{\theta }_1,\cdots ,{\theta }_q,\sigma }$.

For MA model $$\begin{array}{rl}& \mathrm{Cov}(y_t,y_{t+h})\\ =& \mathrm{Cov}(\mu +\sum _{j=0}^q{\theta }_j{\epsilon }_{t-j},\mu +\sum _{k=0}^q{\theta }_k{\epsilon }_{t+h-k})\\ =& \sum _{j=0}^q\sum _{k=0}^q{\theta }_j{\theta }_k\mathrm{Cov}({\epsilon }_{t-j},{\epsilon }_{t+h-k}).\end{array}$$
(take ${\displaystyle {\theta }_0=1}$). Since ${\displaystyle \{{\epsilon }_t\}}$ is Gaussian white noise, ${\displaystyle \mathrm{Cov}({\epsilon }_{t-j},{\epsilon }_{t+h-k})=0}$, unless ${\displaystyle t-j=t+h-k\Rightarrow k=j+h}$. So we need ${\displaystyle 0\le j\le q,0\le k\le q,k=j+h}$. Then $$\mathrm{Cov}(y_t,y_{t+h})=\{\begin{array}{rl}& {\sigma }^2\sum _{j=0}^{q-h}{\theta }_j{\theta }_{j+h},0\le h\le q.\\ & 0,h>q.\end{array}$$
It does not depend on ${\displaystyle t}$, so ${\displaystyle \mathrm{MA}(q)}$ is stationary, and ${\displaystyle \mathrm{Cov}(y_t,y_{t+h})=\gamma (h)}$, and $$\rho (h)=\{\begin{array}{rl}& \frac{\sum _{j=0}^{q-h}{\theta }_j{\theta }_{j+h}}{\sum _{j=0}^q{\theta }_j^2},0\le h\le q,\\ & 0,h>q.\end{array}$$
For ${\displaystyle \mathrm{MA}(1)}$, ${\displaystyle y_t=\mu +{\epsilon }_t+\theta {\epsilon }_{t-1}}$, and $$\rho (h)=\{\begin{array}{rl}& 1,h=0,\\ & \frac{{\theta }_1}{1+{\theta }_1^2},h=1,\\ & 0,h>1.\end{array}$$

3 Sample ACF

For fixed ${\displaystyle h}$, the sample ACF at lag ${\displaystyle h}$ is defined as: $$\frac{\sum _{t=1}^{n-h}(a_t-\bar{a})(b_t-\bar{b})}{\sqrt{\sum _{t=1}^{n-h}(a_t-\bar{a}{)}^2\sum _{t=1}^{n-h}(b_t-\bar{b}{)}^2}}=\frac{\sum _{t=1}^{n-h}(y_t-\bar{a})(y_{t+h}-\bar{b})}{\sqrt{\sum _{t=1}^{n-h}(y_t-\bar{a}{)}^2\sum _{t=1}^{n-h}(y_{n+h}-\bar{b}{)}^2}},$$ where $$\bar{a}=\frac{1}{n-h}\sum _{t=1}^{n-h}{y}_t,\bar{b}=\frac{1}{n-h}\sum _{t=1}^{n-h}{y}_{t+h}.$$
We can simplify by ${\displaystyle \bar{a}\approx \bar{y},\bar{b}\approx \bar{y}}$, and $$\sum _{t=1}^{n-h}(y_t-\bar{a}{)}^2\approx \sum _{t=1}^n(y_t-\bar{y}{)}^2,\sum _{t=1}^{n-h}(y_{t+h}-\bar{b}{)}^2\approx \sum _{t=1}^n(y_t-\bar{y}{)}^2.$$ (reasonable when ${\displaystyle h}$ is small compared to ${\displaystyle n}$). Then define sample ACF: $$r_h=\frac{\sum _{t=1}^{n-h}(y_t-\bar{y})(y_{t+h}-\bar{y})}{\sum _{t=1}^n(y_t-\bar{y}{)}^2},h=0,1,2,\cdots$$
Note that ${\displaystyle r_0=1}$.

Although sample ACF can be computed for any time series, it is only useful for stationary ones.

Sample ACF is useful in determining ${\displaystyle q}$: sample ACF after lag ${\displaystyle q}$ is very small/close to 0.

4 Sample PACF

Define sample PACF (Partial AutoCorrelation) of ${\displaystyle h}$ as ${\displaystyle {\hat{\varphi }}_h}$ (estimate of ${\displaystyle {\varphi }_h}$) when ${\displaystyle \mathrm{AR}(h)}$ is fit to the data.

Sample PACF is useful in determining ${\displaystyle p}$ in ${\displaystyle \mathrm{AR}(p)}$: sample PACF after ${\displaystyle p}$ is very small/close to 0.

Why PACF? Suppose we have data ${\displaystyle (x_1,y_1),\cdots ,(x_n,y_n)}$. The correlation is then $$\mathrm{Corr}(x,y)=\frac{\sum _{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum _{i=1}^n(x_i-\bar{x}{)}^2\sum _{i=1}^n(y_i-\bar{y}{)}^2}}.$$
Under usual OLS (see here), $${\hat{\beta }}_1=\frac{\sum _{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{\sum _{i=1}^n(x_i-\bar{x}{)}^2}=\mathrm{Corr}(x,y)\sqrt{\frac{\mathrm{Var}(y)}{\mathrm{Var}(x)}}.$$
Now we also have data on other variables ${\displaystyle z_1,\cdots ,z_k}$, and dataset becomes ${\displaystyle (y_i,x_i,z_{i1},\cdots ,z_{ik}),i=1,\cdots ,n}$. The partial correlation between ${\displaystyle x,y}$ given ${\displaystyle z_1,\cdots ,z_k}$ is given by ${\displaystyle \mathrm{Corr}(x,y|z_1,\cdots ,z_k)}$: define residual of ${\displaystyle x}$ given ${\displaystyle z_1,\cdots ,z_k}$ as the residual in linear regression $$e_i^{x|z_1,\cdots ,z_k}=x_i-{\hat{\beta }}_0^x-{\hat{\beta }}_1^x{z}_{i1}-\cdots -{\hat{\beta }}_k^x{z}_{ik},$$ and $$\mathrm{Corr}(x,y|z_1,\cdots ,z_k)=\mathrm{Corr}(e^{x|z_1,\cdots ,z_k},e^{y|z_1,\cdots ,z_k}).$$
And for multiple linear regression, denote coefficients (RSS minimizer) as ${\displaystyle {\hat{\beta }}_0,{\hat{\beta }}_x,{\hat{\beta }}_1,\cdots ,{\hat{\beta }}_k}$, then $${\hat{\beta }}_x=\mathrm{Corr}(x,y|z_1,\cdots ,z_k)\sqrt{\frac{\mathrm{Var}(e^{y|z_1,\cdots ,z_k})}{\mathrm{Var}(e^{x|z_1,\cdots ,z_k})}}.$$
Now for time series setting with ${\displaystyle y_1,\cdots ,y_n}$ with AR(p), we can write $${\hat{\varphi }}_p=\mathrm{Corr}(y_{t-p},y_t|y_{t-1},\cdots ,y_{t-p+1})\sqrt{\frac{\mathrm{Var}(e^{y_t|y_{t-1},\cdots ,y_{t-p+1}})}{\mathrm{Var}(e^{y_{t-p}|y_{t-1},\cdots ,y_{t-p+1}})}}.$$
When ${\displaystyle \mathrm{AR}(p)}$ is stationary, $${\hat{\varphi }}_p\approx \mathrm{Corr}(y_{t-p},y_t|y_{t-1},\cdots ,y_{t-p+1}).$$