1 RNN Motivation

First we recap some models we have discussed.

1.1 Regression with ${\displaystyle {\displaystyle t}}$ as Covariate

Consider the simplest linear regression model: $$\begin{array}{}\text{(1.1)}& y_t={\beta }_0+{\beta }_{1}t+{\epsilon }_t,{\epsilon }_t\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\sigma }^2).\end{array}$$
Then consider nonlinear model ( change of slope model): $$\begin{array}{}\text{(1.2)}& y_t=\text{be}|t{a}_0+{\beta }_{1}t+{\beta }_2(t-c_1{)}_{+}+\cdots +{\beta }_{k+1}(t-c_k{)}_{+}+{\epsilon }_t.\end{array}$$ We can also notate as ${\displaystyle \sigma (u)=\mathrm{ReLU}(u)=u_{+}}$. Unknown parameters here are ${\displaystyle {\beta }_0,\cdots ,{\beta }_{k+1},c_1,\cdots ,c_k}$ and ${\displaystyle \sigma }$. This is a linear model for feature vector ${\displaystyle (1,t,(t-c_1{)}_{+},\cdots ,(t-c_k{)}_{+}{)}^{\mathrm{T}}}$.

Rewrite (1.2): let ${\displaystyle x_t=t}$, and ${\displaystyle {\mu }_t}$ for the mean of ${\displaystyle y_t}$. Also denote $$\begin{array}{rl}{r}_t& =(\sigma (x_t-c_1),\cdots ,\sigma (x_t-c_k){)}^{\mathrm{T}},\\ s_t& =(x_t-c_1,\cdots ,x_t-c_k{)}^{\mathrm{T}}.\end{array}$$ Now (1.2) becomes $$\begin{array}{rl}{x}_t& =t,\\ s_t& =(x_t-c_1,\cdots ,x_t-c_k{)}^{\mathrm{T}},\\ \text{(1.3)}& r_t& =\sigma (s_t),{\mu }_t& ={\beta }_0+{\beta }^{\mathrm{T}}{r}_t,\\ y_t& ={\mu }_t+{\epsilon }_t.\end{array}$$
In words, ${\displaystyle x_t}$ is first converted to ${\displaystyle s_t}$ by linear function, then we apply ${\displaystyle \sigma (\cdot )}$ to generate ${\displaystyle r_t}$. Then ${\displaystyle {\mu }_t}$ is a linear function of ${\displaystyle r_t}$ which serves as the mean to ${\displaystyle y_t}$.

1.2 AR

Consider AR model ${\displaystyle \mathrm{AR}(1)}$: ${\displaystyle x_t=y_{t-1}}$. This is simply (1.1) with ${\displaystyle t}$ replaced by ${\displaystyle x_t=y_{t-1}}$: $$y_t={\beta }_0+{\beta }_1{x}_t+{\epsilon }_t,{\epsilon }_t\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\sigma }^2).$$
One can create a nonlinear version by using (1.3) with ${\displaystyle x_t=y_{t-1}}$. Call this ${\displaystyle \mathrm{NAR}(1)}$: $$\begin{array}{rl}{x}_t& =y_{t-1},\\ s_t& =(x_t-c_1,\cdots ,x_t-c_k{)}^{\mathrm{T}},\\ \text{(1.4)}& r_t& =\sigma (s_t),{\mu }_t& ={\beta }_0+{\beta }^{\mathrm{T}}{r}_t,\\ y_t& ={\mu }_t+{\epsilon }_t.\end{array}$$
Now let's consider ${\displaystyle \mathrm{AR}(p)}$: $$\begin{array}{rl}{x}_t& =(y_{t-1},\cdots ,y_{t-p}{)}^{\mathrm{T}},\\ \text{(1.5)}& {\mu }_t& ={\beta }_0+{\beta }^{\mathrm{T}}{x}_t,y_t& ={\mu }_t+{\epsilon }_t.\end{array}$$ Note that (1.5) can be written in compressed form ${\displaystyle y_t={\beta }_0+{\beta }_1{y}_{t-1}+\cdots +{\beta }_p{y}_{t-p}+{\epsilon }_t}$.

Now we extend to nonlinear version like (1.4): $$\begin{array}{rl}{x}_t& =(y_{t-1},\cdots ,y_{t-p}{)}^{\mathrm{T}},\\ s_t& =(x_{t1}-c_1^{(1)},\cdots ,x_{t1}-c_k^{(1)},x_{t2}-c_1^{(2)},\cdots ,x_{t2}-c_k^{(2)},\cdots ,x_{tp}-c_1^{(p)},\cdots ,x_{tp}-c_k^{(p)}{)}^{\mathrm{T}},\\ r_t& =\sigma (s_t),\\ {\mu }_t& ={\beta }_0+{\beta }^{\mathrm{T}}{r}_t,\\ y_t& ={\mu }_t+{\epsilon }_t.\end{array}$$
Here ${\displaystyle x_{t1}=y_{t-1},\cdots x_{tp}=y_{t-p}}$ denote the components of ${\displaystyle x_t}$. With this choice of ${\displaystyle s_t}$, note that $${\mu }_t={\beta }_0+{\beta }^{\mathrm{T}}{r}_t={\beta }_0+{\beta }^{\mathrm{T}}\sigma (s_t)={\beta }_0+\sum _{j=1}^p{g}_j(x_{tj}),$$ where ${\displaystyle g_j(x)=\sum _{i=1}^k{\beta }_{i,j}\sigma (x_{tj}-c_j^{(i)})}$.
In other words, we are fitting an additive model for ${\displaystyle y_t}$ in terms of covariates ${\displaystyle x_{t1}=y_{t-1},\cdots ,x_{tp}=y_{t-p}}$. However, it can't handle interactions between covariates.

Instead of the above additive model, we shall use ${\displaystyle \mathrm{NAR}(p)}$: $$\begin{array}{rl}{x}_t& =(y_{t-1},\cdots ,y_{t-p}{)}^{\mathrm{T}},\\ s_t& =W{x}_t+b,\\ \text{(1.7)}& r_t& =\sigma (s_t),{\mu }_t& ={\beta }_0+{\beta }^{\mathrm{T}}{r}_t,\\ y_t& ={\mu }_t+{\epsilon }_t.\end{array}$$
Here ${\displaystyle W\in {\mathbb{R}}^{k\times p}}$ and ${\displaystyle b\in {\mathbb{R}}^{k\times 1}}$. Parameters here are ${\displaystyle W}$, ${\displaystyle b}$, ${\displaystyle {\beta }_0}$, ${\displaystyle \beta }$ and ${\displaystyle \sigma }$.

As for neural network, (1.7) is called a single-hidden layer neural network (see here).

2 RNNs

In (1.7), we do one modification: $$\begin{array}{rl}{x}_t& =(y_{t-1},\cdots ,y_{t-p}{)}^{\mathrm{T}},\\ s_t& =W_r{r}_{t-1}+W{x}_t+b,\\ \text{(2.1)}& r_t& =\sigma (s_t),{\mu }_t& ={\beta }_0+{\beta }^{\mathrm{T}}{r}_t,\\ y_t& ={\mu }_t+{\epsilon }_t.\end{array}$$
Here ${\displaystyle r_t}$ involves not just current ${\displaystyle x_t}$ but also previous hidden layer output ${\displaystyle r_{t-1}}$ through ${\displaystyle W_r{r}_{t-1}}$, ${\displaystyle W_r\in {\mathbb{R}}^{k\times k}}$. Parameters now are ${\displaystyle W_r,W,b,{\beta }_0,\beta }$ and ${\displaystyle \sigma }$. Typically ${\displaystyle k>p}$. (2.1) also requires an initialization of ${\displaystyle r_t}$, usually ${\displaystyle r_0=0}$.

In (1.7), ${\displaystyle r_t}$ depends only on ${\displaystyle x_t}$. While in (2.1), ${\displaystyle r_t}$ depends on ${\displaystyle x_t,x_{t-1},\cdots ,x_1}$. Now we assume ${\displaystyle x_t}$ are defined for all ${\displaystyle t=1,2,\cdots }$ WLOG. To see how ${\displaystyle r_t}$ depends, note that $$\begin{array}{rl}{r}_1& =\sigma (W{x}_1+b),(r_0=0),\\ r_2& =\sigma (W_r\sigma (W{x}_1+b)+W{x}_2+b),\\ r_3& =\sigma (W_r\sigma (W_r\sigma (W{x}_1+b)+W{x}_2+b)+W{x}_3+b),\\ r_4& =\sigma (W_r\sigma (W_r\sigma (W_r\sigma (W{x}_1+b)+W{x}_2+b)+W{x}_3+b)+W{x}_4+b).\end{array}$$
From the above, ${\displaystyle r_t}$ clearly depends on ${\displaystyle x_1,\cdots ,x_t}$, but the strength varies.

RNN can have stability issues, causing either gratitude explosion or vanishing. To see this, observe that $$\frac{\partial r_t}{\partial x_u}={\sigma }^{\prime }(s_t)W_r{\sigma }^{\prime }(s_{t-1})W_r\cdots {\sigma }^{\prime }(s_{u+1})W_r{\sigma }^{\prime }(s_u)W,u\le t.$$
To solve this, it is customary to take ${\displaystyle \sigma }$ as the hyperbolic tangent function ${\displaystyle {\sigma }_{\tanh }(u)=\frac{{\mathrm{e}}^u-{\mathrm{e}}^{-u}}{{\mathrm{e}}^u+{\mathrm{e}}^{-u}}}$.
Unlike ReLU, this function takes values between ${\displaystyle -1}$ and ${\displaystyle 1}$: ${\displaystyle {\sigma }^{\prime }(u)=1-{\sigma }^2(u)\in (0,1]}$.

Further, we want ${\displaystyle r_t}$ to represent the ideal summary of ${\displaystyle x_1,\cdots ,x_t}$. But here in RNN, ${\displaystyle r_t}$ only depends on the inputs closer to ${\displaystyle t}$. So RNN doesn't have a very long memory.