1 GRU (Gated Recurrent Unit)

Recall formula for RNN. The basic problem is that ${\displaystyle r_t}$ depends on ${\displaystyle r_{t-1}}$ through ${\displaystyle W_r{r}_{t-1}}$. If ${\displaystyle W_r}$ is a matrix with spectral radius less than ${\displaystyle 1}$, ${\displaystyle W_r{r}_{t-1}}$ can be thought of as "reducing" ${\displaystyle r_{t-1}}$ by a factor of ${\displaystyle W_r}$. Applied repeatedly, ${\displaystyle r_t,r_u}$ 's dependence will be very small. So ${\displaystyle r_t}$ should connect to ${\displaystyle r_{t-1}}$ not only through ${\displaystyle W_r{r}_{t-1}}$.

We first construct a potential version ${\displaystyle {\tilde{r}}_t}$ of ${\displaystyle r_t}$: $$\begin{array}{}\text{(1.1)}& {\tilde{r}}_t=\sigma (W_r{r}_{t-1}+W{x}_t+b).\end{array}$$ Two natural options:

• ${\displaystyle r_t={\tilde{r}}_t}$. We are back to RNN.
• ${\displaystyle r_t=r_{t-1}}$: ${\displaystyle r_t}$ is exactly equal to ${\displaystyle r_{t-1}}$, which means ${\displaystyle x_t}$ is ignored.

The idea behind GRU is to combine them $$r_t=z_t{r}_{t-1}+(1-z_t){\tilde{r}}_t.$$ This would exactly be a convex combination if ${\displaystyle z_t}$ were a scalar in ${\displaystyle [0,1]}$, but ${\displaystyle z_t}$ is allowed to be a vector, so we'd better write $$r_t=z_t\odot r_{t-1}+(1-z_t)\odot {\tilde{r}}_t.$$
For ${\displaystyle z_t}$, we take $$\begin{array}{}\text{(1.2)}& z_t={\sigma }_{\mathrm{sigmoid}}(W_r^z{r}_{t-1}+W^z{x}_t+b^z),\end{array}$$, where ${\displaystyle {\sigma }_{\mathrm{sigmoid}}(u)=\frac{1}{1+{\mathrm{e}}^{-u}}}$. We sometimes refer to ${\displaystyle z_t}$ as a gate. It controls the closeness of ${\displaystyle r_t}$ to ${\displaystyle r_{t-1}}$ and ${\displaystyle {\tilde{r}}_t}$.

${\displaystyle r_{t-1}}$ appears in both ${\displaystyle z_t\odot r_{t-1}}$ and ${\displaystyle {\tilde{r}}_t}$. It might be redundant. So GRU modifies (1.1) by using one more gate: $${\tilde{r}}_t=\sigma (W_r(r_{t-1}\odot g_t)+W{x}_t+b),$$ where ${\displaystyle g_t}$ controls the extent to which ${\displaystyle r_{t-1}}$ is used in the formula for ${\displaystyle {\tilde{r}}_t}$. Similar to (1.2), $$g_t={\sigma }_{\mathrm{sigmoid}}(W_r^g{r}_{t-1}+W^g{x}_t+b^g).$$
Putting all the formulae together: $$\begin{array}{rl}{r}_0& =0,\\ g_t& ={\sigma }_{\mathrm{sigmoid}}(W_r^g{r}_{t-1}+W^g{x}_t+b^g),\\ z_t& ={\sigma }_{\mathrm{sigmoid}}(W_r^z{r}_{t-1}+W^z{x}_t+b^z),\\ \text{(1.3)}& {\tilde{r}}_t& ={\sigma }_{\tanh }(W_r(r_{t-1}\odot g_t)+W{x}_t+b),r_t& =z_t\odot r_{t-1}+(1-z_t)\odot {\tilde{r}}_t,\\ {\mu }_t& ={\beta }_0+{\beta }^{\mathrm{T}}{r}_t.\end{array}$$
${\displaystyle z_t}$ is called the update gate while ${\displaystyle g_t}$ is called the reset gate. Unknown parameters are ${\displaystyle W_r^g,W^g,b^g,W_r^z,W^z,b^z,W_r,W,b,{\beta }_0,\beta }$.

LSTM (Long Short Term Memory)

This is another modification to the basic RNN for enabling long memory. It has one more gate compared with GRU. Instead of a recursion directly between ${\displaystyle r_{t-1}}$ and ${\displaystyle r_t}$, LSTM recursions are between ${\displaystyle (s_{t-1},r_{t-1})\to (s_t,r_t)}$.

We again construct a potential version: $$\begin{array}{}\text{(2.1)}& {\tilde{r}}_t=\sigma (W_r{r}_{t-1}+W{x}_t+b).\end{array}$$
In LSTM, ${\displaystyle s_t}$ is taken to be a linear combination of ${\displaystyle s_{t-1}}$ and ${\displaystyle {\tilde{r}}_t}$ with gates controlling both coefficients of the linear combination: $$s_t=f_t\odot s_{t-1}+i_t\odot {\tilde{r}}_t,$$ where ${\displaystyle f_t,i_t}$ denote gates. ${\displaystyle r_t}$ is defined usually as ${\displaystyle {\sigma }_{\tanh }(s_t)}$. In LSTM, we add a gate to ${\displaystyle r_t}$: $$r_t=o_t\odot {\sigma }_{\tanh }(s_t).$$
Putting everything together, we obtain the full LSTM model: $$\begin{array}{rl}{r}_0& =0,\\ f_t& ={\sigma }_{\mathrm{sigmoid}}(W_r^f{r}_{t-1}+W^f{x}_t+b^f),\\ i_t& ={\sigma }_{\mathrm{sigmoid}}(W_r^i{r}_{t-1}+W^i{x}_t+b^i),\\ o_t& ={\sigma }_{\mathrm{sigmoid}}(W_r^o{r}_{t-1}+W^o{x}_t+b^o),\\ \text{(2.2)}& {\tilde{r}}_t& ={\sigma }_{\tanh }(W_r{r}_{t-1}+W{x}_t+b),s_t& =f_t\odot s_{t-1}+i_t\odot {\tilde{r}}_t,\\ r_t& =o_t\odot {\sigma }_{\tanh }(s_t),\\ {\mu }_t& ={\beta }_0+{\beta }^{\mathrm{T}}{r}_t.\end{array}$$
${\displaystyle f_t}$ is called the forget gate, ${\displaystyle i_t}$ is called the input gate and ${\displaystyle o_t}$ is called the output gate. Unknown parameters are ${\displaystyle W_r^f,W^f,b^f,W_r^i,W^i,b^i,W_r^o,W^o,b^o,W_r,W,b,{\beta }_0,\beta }$.