# 4.1. Multilayer Perceptron¶

In the previous chapters, we showed how you could implement multiclass
logistic regression (also called softmax regression) for classifying
images of clothing into the 10 possible categories. To get there, we had
to learn how to wrangle data, coerce our outputs into a valid
probability distribution (via `softmax`

), how to apply an appropriate
loss function, and how to optimize over our parameters. Now that we’ve
covered these preliminaries, we are free to focus our attention on the
more exciting enterprise of designing powerful models using deep neural
networks.

## 4.1.2. Activation Functions¶

Because they are so fundamental to deep learning, before going further, let’s take a brief look at some common activation functions.

### 4.1.2.1. ReLU Function¶

As stated above, the most popular choice, due to its simplicity of implementation and its efficacy in training is the rectified linear unit (ReLU). ReLUs provide a very simple nonlinear transformation. Given the element \(z\), the function is defined as the maximum of that element and 0.

It can be understood that the ReLU function retains only positive elements and discards negative elements (setting those nodes to 0). To get a better idea of what it looks like, we can plot it.

Because it is used so commonly, NDarray supports the `relu`

function
as a basic native operator. As you can see, the activation function is
piece-wise linear.

```
x = nd.arange(-8.0, 8.0, 0.1)
x.attach_grad()
with autograd.record():
y = x.relu()
d2l.set_figsize((4, 2.5))
d2l.plot(x, y, 'x', 'relu(x)')
```

When the input is negative, the derivative of ReLU function is 0 and
when the input is positive, the derivative of ReLU function is 1. Note
that the ReLU function is not differentiable when the input takes value
precisely equal to 0. In these cases, we go with the left-hand-side
(LHS) derivative and say that the derivative is 0 when the input is 0.
We can get away with this because the input may never actually be zero.
There’s an old adage that if subtle boundary conditions matter, we are
probably doing (*real*) mathematics, not engineering. That conventional
wisdom may apply here. See the derivative of the ReLU function plotted
below.

```
y.backward()
d2l.plot(x, x.grad, 'x', 'grad of relu')
```

Note that there are many variants to the ReLU function, such as the parameterized ReLU (pReLU) of He et al., 2015. This variation adds a linear term to the ReLU, so some information still gets through, even when the argument is negative.

The reason for using the ReLU is that its derivatives are particularly well behaved - either they vanish or they just let the argument through. This makes optimization better behaved and it reduces the issue of the vanishing gradient problem (more on this later).

### 4.1.2.2. Sigmoid Function¶

The sigmoid function transforms its inputs which take values in
\(\mathbb{R}\) to the interval \((0,1)\). For that reason, the
sigmoid is often called a *squashing* function: it squashes any input in
the range (-inf, inf) to some value in the range (0,1).

In the earliest neural networks, scientists were interested in modeling
biological neurons which either *fire* or *don’t fire*. Thus the
pioneers of this field, going all the way back to McCulloch and Pitts in
the 1940s, were focused on thresholding units. A thresholding function
takes either value \(0\) (if the input is below the threshold) or
value \(1\) (if the input exceeds the threshold)

When attention shifted to gradient based learning, the sigmoid function was a natural choice because it is a smooth, differentiable approximation to a thresholding unit. Sigmoids are still common as activation functions on the output units, when we want to interpret the outputs as probabilities for binary classification problems (you can think of the sigmoid as a special case of the softmax) but the sigmoid has mostly been replaced by the simpler and easier to train ReLU for most use in hidden layers. In the “Recurrent Neural Network” chapter, we will describe how sigmoid units can be used to control the flow of information in a neural network thanks to its capacity to transform the value range between 0 and 1.

See the sigmoid function plotted below. When the input is close to 0, the sigmoid function approaches a linear transformation.

```
with autograd.record():
y = x.sigmoid()
d2l.plot(x, y, 'x', 'sigmoid(x)')
```

The derivative of sigmoid function is given by the following equation:

The derivative of sigmoid function is plotted below. Note that when the input is 0, the derivative of the sigmoid function reaches a maximum of 0.25. As the input diverges from 0 in either direction, the derivative approaches 0.

```
y.backward()
d2l.plot(x, x.grad, 'x', 'grad of sigmoid')
```

### 4.1.2.3. Tanh Function¶

Like the sigmoid function, the tanh (Hyperbolic Tangent) function also squashes its inputs, transforms them into elements on the interval between -1 and 1:

We plot the tanh function blow. Note that as the input nears 0, the tanh function approaches a linear transformation. Although the shape of the function is similar to the sigmoid function, the tanh function exhibits point symmetry about the origin of the coordinate system.

```
with autograd.record():
y = x.tanh()
d2l.plot(x, y, 'x', 'tanh(x)')
```

The derivative of the Tanh function is:

The derivative of tanh function is plotted below. As the input nears 0, the derivative of the tanh function approaches a maximum of 1. And as we saw with the sigmoid function, as the input moves away from 0 in either direction, the derivative of the tanh function approaches 0.

```
y.backward()
d2l.plot(x, x.grad, 'x', 'grad of tanh')
```

In summary, we now know how to incorporate nonlinearities to build expressive multilayer neural network architectures. As a side note, your knowledge now already puts you in command of the state of the art in deep learning, circa 1990. In fact, you have an advantage over anyone working the 1990s, because you can leverage powerful open-source deep learning frameworks to build models rapidly, using only a few lines of code. Previously, getting these nets training required researchers to code up thousands of lines of C and Fortran.

## 4.1.3. Summary¶

- The multilayer perceptron adds one or multiple fully-connected hidden layers between the output and input layers and transforms the output of the hidden layer via an activation function.
- Commonly-used activation functions include the ReLU function, the sigmoid function, and the tanh function.

## 4.1.4. Exercises¶

- Compute the derivative of the tanh and the pReLU activation function.
- Show that a multilayer perceptron using only ReLU (or pReLU) constructs a continuous piecewise linear function.
- Show that \(\mathrm{tanh}(x) + 1 = 2 \mathrm{sigmoid}(2x)\).
- Assume we have a multilayer perceptron
*without*nonlinearities between the layers. In particular, assume that we have \(d\) input dimensions, \(d\) output dimensions and that one of the layers had only \(d/2\) dimensions. Show that this network is less expressive (powerful) than a single layer perceptron. - Assume that we have a nonlinearity that applies to one minibatch at a time. What kinds of problems do you expect this to cause?