.. _sec_rnn:
Recurrent Neural Networks
=========================
In :numref:`sec_language-model` we described Markov models and
:math:`n`-grams for language modeling, where the conditional probability
of token :math:`x_t` at time step :math:`t` only depends on the
:math:`n-1` previous tokens. If we want to incorporate the possible
effect of tokens earlier than time step :math:`t-(n-1)` on :math:`x_t`,
we need to increase :math:`n`. However, the number of model parameters
would also increase exponentially with it, as we need to store
:math:`|\mathcal{V}|^n` numbers for a vocabulary set
:math:`\mathcal{V}`. Hence, rather than modeling
:math:`P(x_t \mid x_{t-1}, \ldots, x_{t-n+1})` it is preferable to use a
latent variable model,
.. math:: P(x_t \mid x_{t-1}, \ldots, x_1) \approx P(x_t \mid h_{t-1}),
where :math:`h_{t-1}` is a *hidden state* that stores the sequence
information up to time step :math:`t-1`. In general, the hidden state at
any time step :math:`t` could be computed based on both the current
input :math:`x_{t}` and the previous hidden state :math:`h_{t-1}`:
.. math:: h_t = f(x_{t}, h_{t-1}).
:label: eq_ht_xt
For a sufficiently powerful function :math:`f` in :eq:`eq_ht_xt`,
the latent variable model is not an approximation. After all,
:math:`h_t` may simply store all the data it has observed so far.
However, it could potentially make both computation and storage
expensive.
Recall that we have discussed hidden layers with hidden units in
:numref:`chap_perceptrons`. It is noteworthy that hidden layers and
hidden states refer to two very different concepts. Hidden layers are,
as explained, layers that are hidden from view on the path from input to
output. Hidden states are technically speaking *inputs* to whatever we
do at a given step, and they can only be computed by looking at data at
previous time steps.
*Recurrent neural networks* (RNNs) are neural networks with hidden
states. Before introducing the RNN model, we first revisit the MLP model
introduced in :numref:`sec_mlp`.
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import torch
from d2l import torch as d2l
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
from mxnet import np, npx
from d2l import mxnet as d2l
npx.set_np()
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import jax
from jax import numpy as jnp
from d2l import jax as d2l
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import tensorflow as tf
from d2l import tensorflow as d2l
.. raw:: html
.. raw:: html
Neural Networks without Hidden States
-------------------------------------
Let’s take a look at an MLP with a single hidden layer. Let the hidden
layer’s activation function be :math:`\phi`. Given a minibatch of
examples :math:`\mathbf{X} \in \mathbb{R}^{n \times d}` with batch size
:math:`n` and :math:`d` inputs, the hidden layer output
:math:`\mathbf{H} \in \mathbb{R}^{n \times h}` is calculated as
.. math:: \mathbf{H} = \phi(\mathbf{X} \mathbf{W}_{\textrm{xh}} + \mathbf{b}_\textrm{h}).
:label: rnn_h_without_state
In :eq:`rnn_h_without_state`, we have the weight parameter
:math:`\mathbf{W}_{\textrm{xh}} \in \mathbb{R}^{d \times h}`, the bias
parameter :math:`\mathbf{b}_\textrm{h} \in \mathbb{R}^{1 \times h}`, and
the number of hidden units :math:`h`, for the hidden layer. So armed, we
apply broadcasting (see :numref:`subsec_broadcasting`) during the
summation. Next, the hidden layer output :math:`\mathbf{H}` is used as
input of the output layer, which is given by
.. math:: \mathbf{O} = \mathbf{H} \mathbf{W}_{\textrm{hq}} + \mathbf{b}_\textrm{q},
where :math:`\mathbf{O} \in \mathbb{R}^{n \times q}` is the output
variable, :math:`\mathbf{W}_{\textrm{hq}} \in \mathbb{R}^{h \times q}`
is the weight parameter, and
:math:`\mathbf{b}_\textrm{q} \in \mathbb{R}^{1 \times q}` is the bias
parameter of the output layer. If it is a classification problem, we can
use :math:`\mathrm{softmax}(\mathbf{O})` to compute the probability
distribution of the output categories.
This is entirely analogous to the regression problem we solved
previously in :numref:`sec_sequence`, hence we omit details. Suffice
it to say that we can pick feature-label pairs at random and learn the
parameters of our network via automatic differentiation and stochastic
gradient descent.
.. _subsec_rnn_w_hidden_states:
Recurrent Neural Networks with Hidden States
--------------------------------------------
Matters are entirely different when we have hidden states. Let’s look at
the structure in some more detail.
Assume that we have a minibatch of inputs
:math:`\mathbf{X}_t \in \mathbb{R}^{n \times d}` at time step :math:`t`.
In other words, for a minibatch of :math:`n` sequence examples, each row
of :math:`\mathbf{X}_t` corresponds to one example at time step
:math:`t` from the sequence. Next, denote by
:math:`\mathbf{H}_t \in \mathbb{R}^{n \times h}` the hidden layer output
of time step :math:`t`. Unlike with MLP, here we save the hidden layer
output :math:`\mathbf{H}_{t-1}` from the previous time step and
introduce a new weight parameter
:math:`\mathbf{W}_{\textrm{hh}} \in \mathbb{R}^{h \times h}` to describe
how to use the hidden layer output of the previous time step in the
current time step. Specifically, the calculation of the hidden layer
output of the current time step is determined by the input of the
current time step together with the hidden layer output of the previous
time step:
.. math:: \mathbf{H}_t = \phi(\mathbf{X}_t \mathbf{W}_{\textrm{xh}} + \mathbf{H}_{t-1} \mathbf{W}_{\textrm{hh}} + \mathbf{b}_\textrm{h}).
:label: rnn_h_with_state
Compared with :eq:`rnn_h_without_state`,
:eq:`rnn_h_with_state` adds one more term
:math:`\mathbf{H}_{t-1} \mathbf{W}_{\textrm{hh}}` and thus instantiates
:eq:`eq_ht_xt`. From the relationship between hidden layer outputs
:math:`\mathbf{H}_t` and :math:`\mathbf{H}_{t-1}` of adjacent time
steps, we know that these variables captured and retained the sequence’s
historical information up to their current time step, just like the
state or memory of the neural network’s current time step. Therefore,
such a hidden layer output is called a *hidden state*. Since the hidden
state uses the same definition of the previous time step in the current
time step, the computation of :eq:`rnn_h_with_state` is
*recurrent*. Hence, as we said, neural networks with hidden states based
on recurrent computation are named *recurrent neural networks*. Layers
that perform the computation of :eq:`rnn_h_with_state` in RNNs are
called *recurrent layers*.
There are many different ways for constructing RNNs. Those with a hidden
state defined by :eq:`rnn_h_with_state` are very common. For time
step :math:`t`, the output of the output layer is similar to the
computation in the MLP:
.. math:: \mathbf{O}_t = \mathbf{H}_t \mathbf{W}_{\textrm{hq}} + \mathbf{b}_\textrm{q}.
Parameters of the RNN include the weights
:math:`\mathbf{W}_{\textrm{xh}} \in \mathbb{R}^{d \times h}, \mathbf{W}_{\textrm{hh}} \in \mathbb{R}^{h \times h}`,
and the bias :math:`\mathbf{b}_\textrm{h} \in \mathbb{R}^{1 \times h}`
of the hidden layer, together with the weights
:math:`\mathbf{W}_{\textrm{hq}} \in \mathbb{R}^{h \times q}` and the
bias :math:`\mathbf{b}_\textrm{q} \in \mathbb{R}^{1 \times q}` of the
output layer. It is worth mentioning that even at different time steps,
RNNs always use these model parameters. Therefore, the parametrization
cost of an RNN does not grow as the number of time steps increases.
:numref:`fig_rnn` illustrates the computational logic of an RNN at
three adjacent time steps. At any time step :math:`t`, the computation
of the hidden state can be treated as: (i) concatenating the input
:math:`\mathbf{X}_t` at the current time step :math:`t` and the hidden
state :math:`\mathbf{H}_{t-1}` at the previous time step :math:`t-1`;
(ii) feeding the concatenation result into a fully connected layer with
the activation function :math:`\phi`. The output of such a fully
connected layer is the hidden state :math:`\mathbf{H}_t` of the current
time step :math:`t`. In this case, the model parameters are the
concatenation of :math:`\mathbf{W}_{\textrm{xh}}` and
:math:`\mathbf{W}_{\textrm{hh}}`, and a bias of
:math:`\mathbf{b}_\textrm{h}`, all from :eq:`rnn_h_with_state`. The
hidden state of the current time step :math:`t`, :math:`\mathbf{H}_t`,
will participate in computing the hidden state :math:`\mathbf{H}_{t+1}`
of the next time step :math:`t+1`. What is more, :math:`\mathbf{H}_t`
will also be fed into the fully connected output layer to compute the
output :math:`\mathbf{O}_t` of the current time step :math:`t`.
.. _fig_rnn:
.. figure:: ../img/rnn.svg
An RNN with a hidden state.
We just mentioned that the calculation of
:math:`\mathbf{X}_t \mathbf{W}_{\textrm{xh}} + \mathbf{H}_{t-1} \mathbf{W}_{\textrm{hh}}`
for the hidden state is equivalent to matrix multiplication of the
concatenation of :math:`\mathbf{X}_t` and :math:`\mathbf{H}_{t-1}` and
the concatenation of :math:`\mathbf{W}_{\textrm{xh}}` and
:math:`\mathbf{W}_{\textrm{hh}}`. Though this can be proven
mathematically, in the following we just use a simple code snippet as a
demonstration. To begin with, we define matrices ``X``, ``W_xh``, ``H``,
and ``W_hh``, whose shapes are (3, 1), (1, 4), (3, 4), and (4, 4),
respectively. Multiplying ``X`` by ``W_xh``, and ``H`` by ``W_hh``, and
then adding these two products, we obtain a matrix of shape (3, 4).
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X, W_xh = torch.randn(3, 1), torch.randn(1, 4)
H, W_hh = torch.randn(3, 4), torch.randn(4, 4)
torch.matmul(X, W_xh) + torch.matmul(H, W_hh)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[ 1.2526, 0.0580, -3.3460, -0.2519],
[-1.3064, 1.4132, -0.1435, 0.3482],
[ 3.1495, 0.8172, 1.5167, -0.9038]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X, W_xh = np.random.randn(3, 1), np.random.randn(1, 4)
H, W_hh = np.random.randn(3, 4), np.random.randn(4, 4)
np.dot(X, W_xh) + np.dot(H, W_hh)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[22:07:37] ../src/storage/storage.cc:196: Using Pooled (Naive) StorageManager for CPU
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[-0.21952915, 4.256434 , 4.5812645 , -5.344988 ],
[ 3.447858 , -3.0177274 , -1.6777471 , 7.535347 ],
[ 2.2390068 , 1.4199957 , 4.744728 , -8.421293 ]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X, W_xh = jax.random.normal(d2l.get_key(), (3, 1)), jax.random.normal(
d2l.get_key(), (1, 4))
H, W_hh = jax.random.normal(d2l.get_key(), (3, 4)), jax.random.normal(
d2l.get_key(), (4, 4))
jnp.matmul(X, W_xh) + jnp.matmul(H, W_hh)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Array([[-4.270068 , 2.114032 , 1.7267488 , -0.34792683],
[ 0.90870035, 1.4999211 , 1.0609093 , 1.4550787 ],
[-0.08191824, -4.367585 , 0.48806643, 2.5333326 ]], dtype=float32)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
X, W_xh = tf.random.normal((3, 1)), tf.random.normal((1, 4))
H, W_hh = tf.random.normal((3, 4)), tf.random.normal((4, 4))
tf.matmul(X, W_xh) + tf.matmul(H, W_hh)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
Now we concatenate the matrices ``X`` and ``H`` along columns (axis 1),
and the matrices ``W_xh`` and ``W_hh`` along rows (axis 0). These two
concatenations result in matrices of shape (3, 5) and of shape (5, 4),
respectively. Multiplying these two concatenated matrices, we obtain the
same output matrix of shape (3, 4) as above.
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
torch.matmul(torch.cat((X, H), 1), torch.cat((W_xh, W_hh), 0))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[ 1.2526, 0.0580, -3.3460, -0.2519],
[-1.3064, 1.4132, -0.1435, 0.3482],
[ 3.1495, 0.8172, 1.5167, -0.9038]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
np.dot(np.concatenate((X, H), 1), np.concatenate((W_xh, W_hh), 0))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
array([[-0.21952918, 4.256434 , 4.5812645 , -5.344988 ],
[ 3.4478583 , -3.0177271 , -1.677747 , 7.535347 ],
[ 2.2390068 , 1.4199957 , 4.744728 , -8.421294 ]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
jnp.matmul(jnp.concatenate((X, H), 1), jnp.concatenate((W_xh, W_hh), 0))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Array([[-4.270068 , 2.114032 , 1.7267488 , -0.3479268 ],
[ 0.9087004 , 1.4999211 , 1.0609093 , 1.4550787 ],
[-0.08191825, -4.3675847 , 0.48806655, 2.5333326 ]], dtype=float32)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
tf.matmul(tf.concat((X, H), 1), tf.concat((W_xh, W_hh), 0))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. raw:: html
.. raw:: html
RNN-Based Character-Level Language Models
-----------------------------------------
Recall that for language modeling in :numref:`sec_language-model`, we
aim to predict the next token based on the current and past tokens; thus
we shift the original sequence by one token as the targets (labels).
:cite:t:`Bengio.Ducharme.Vincent.ea.2003` first proposed to use a
neural network for language modeling. In the following we illustrate how
RNNs can be used to build a language model. Let the minibatch size be
one, and the sequence of the text be “machine”. To simplify training in
subsequent sections, we tokenize text into characters rather than words
and consider a *character-level language model*.
:numref:`fig_rnn_train` demonstrates how to predict the next character
based on the current and previous characters via an RNN for
character-level language modeling.
.. _fig_rnn_train:
.. figure:: ../img/rnn-train.svg
A character-level language model based on the RNN. The input and
target sequences are “machin” and “achine”, respectively.
During the training process, we run a softmax operation on the output
from the output layer for each time step, and then use the cross-entropy
loss to compute the error between the model output and the target.
Because of the recurrent computation of the hidden state in the hidden
layer, the output, :math:`\mathbf{O}_3`, of time step 3 in
:numref:`fig_rnn_train` is determined by the text sequence “m”, “a”,
and “c”. Since the next character of the sequence in the training data
is “h”, the loss of time step 3 will depend on the probability
distribution of the next character generated based on the feature
sequence “m”, “a”, “c” and the target “h” of this time step.
In practice, each token is represented by a :math:`d`-dimensional
vector, and we use a batch size :math:`n>1`. Therefore, the input
:math:`\mathbf X_t` at time step :math:`t` will be an :math:`n\times d`
matrix, which is identical to what we discussed in
:numref:`subsec_rnn_w_hidden_states`.
In the following sections, we will implement RNNs for character-level
language models.
Summary
-------
A neural network that uses recurrent computation for hidden states is
called a recurrent neural network (RNN). The hidden state of an RNN can
capture historical information of the sequence up to the current time
step. With recurrent computation, the number of RNN model parameters
does not grow as the number of time steps increases. As for
applications, an RNN can be used to create character-level language
models.
Exercises
---------
1. If we use an RNN to predict the next character in a text sequence,
what is the required dimension for any output?
2. Why can RNNs express the conditional probability of a token at some
time step based on all the previous tokens in the text sequence?
3. What happens to the gradient if you backpropagate through a long
sequence?
4. What are some of the problems associated with the language model
described in this section?
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html
`Discussions `__
.. raw:: html
.. raw:: html