mhjensen
571 lines
23 KiB
Plaintext
571 lines
23 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- dom:TITLE: Data Analysis and Machine Learning: Elements of machine learning -->\n",
|
|
"# Data Analysis and Machine Learning: Elements of machine learning\n",
|
|
"<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
|
|
"<!-- Author: --> \n",
|
|
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
|
"\n",
|
|
"Date: **Nov 22, 2017**\n",
|
|
"\n",
|
|
"Copyright 1999-2017, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"## What is Machine Learning?\n",
|
|
"\n",
|
|
"Machine learning is the science of giving computers the ability to learn without being explicitly programmed. \n",
|
|
"The idea is that there exist generic algorithms which can be used to find patterns in a broad class of data sets without \n",
|
|
"having to write code specifically for each problem. The algorithm will build its own logic based on the data. \n",
|
|
"\n",
|
|
"Machine learning is a subfield of computer science, and is closely related to computational statistics. \n",
|
|
"It evolved from the study of pattern recognition in artificial intelligence (AI) research, and has made contributions to\n",
|
|
"AI tasks like computer vision, natural language processing \n",
|
|
"and speech recognition. It has also, especially in later years, \n",
|
|
"found applications in a wide variety of other areas, including bioinformatics, economy, physics, finance and marketing. \n",
|
|
"\n",
|
|
"## Types of Machine Learning\n",
|
|
"\n",
|
|
"\n",
|
|
"The approaches to machine learning are many, but are often split into two main categories. \n",
|
|
"In *supervised learning* we know the answer to a problem,\n",
|
|
"and let the computer deduce the logic behind it. On the other hand, *unsupervised learning*\n",
|
|
"is a method for finding patterns and relationship in data sets without any prior knowledge of the system.\n",
|
|
"Some authours also operate with a third category, namely *reinforcement learning*. This is a paradigm \n",
|
|
"of learning inspired by behavioural psychology, where learning is achieved by trial-and-error, \n",
|
|
"solely from rewards and punishment.\n",
|
|
"\n",
|
|
"Another way to categorize machine learning tasks is to consider the desired output of a system.\n",
|
|
"Some of the most common tasks are:\n",
|
|
"\n",
|
|
" * Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.\n",
|
|
"\n",
|
|
" * Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.\n",
|
|
"\n",
|
|
" * Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.\n",
|
|
"\n",
|
|
"## Artificial neurons\n",
|
|
"The field of artificial neural networks has a long history of development, and is closely connected with \n",
|
|
"the advancement of computer science and computers in general. A model of artificial neurons \n",
|
|
"was first developed by McCulloch and Pitts in 1943 to study signal processing in the brain and \n",
|
|
"has later been refined by others. The general idea is to mimic neural networks in the human brain, which\n",
|
|
"is composed of billions of neurons that communicate with each other by sending electrical signals. \n",
|
|
"Each neuron accumulates its incoming signals, \n",
|
|
"which must exceed an activation threshold to yield an output. If the threshold is not overcome, the neuron\n",
|
|
"remains inactive, i.e. has zero output. \n",
|
|
"\n",
|
|
"This behaviour has inspired a simple mathematical model for an artificial neuron."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"artificialNeuron\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" y = f\\left(\\sum_{i=1}^n w_ix_i\\right) = f(u)\n",
|
|
"\\label{artificialNeuron} \\tag{1}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here, the output $y$ of the neuron is the value of its activation function, which have as input\n",
|
|
"a weighted sum of signals $x_i, \\dots ,x_n$ received by $n$ other neurons.\n",
|
|
"\n",
|
|
"## Neural network types\n",
|
|
"\n",
|
|
"An artificial neural network (NN), is a computational model that consists of layers of connected neurons, or *nodes*. \n",
|
|
"It is supposed to mimic a biological nervous system by letting each neuron interact with other neurons\n",
|
|
"by sending signals in the form of mathematical functions between layers. \n",
|
|
"A wide variety of different NNs have\n",
|
|
"been developed, but most of them consist of an input layer, an output layer and eventual layers in-between, called\n",
|
|
"*hidden layers*. All layers can contain an arbitrary number of nodes, and each connection between two nodes\n",
|
|
"is associated with a weight variable. \n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"## Feed-forward neural networks\n",
|
|
"The feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network, \n",
|
|
"the information moves in only one direction: forward through the layers.\n",
|
|
"\n",
|
|
"Nodes are represented by circles, while the arrows display the connections between the nodes, including the \n",
|
|
"direction of information flow. Additionally, each arrow corresponds to a weight variable, not displayed here. \n",
|
|
"We observe that each node in a layer is connected to *all* nodes in the subsequent layer, \n",
|
|
"making this a so-called *fully-connected* FFNN. \n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"A different variant of FFNNs are *convolutional neural networks* (CNNs), which have a connectivity pattern\n",
|
|
"inspired by the animal visual cortex. Individual neurons in the visual cortex only respond to stimuli from\n",
|
|
"small sub-regions of the visual field, called a receptive field. This makes the neurons well-suited to exploit the strong\n",
|
|
"spatially local correlation present in natural images. The response of each neuron can be approximated mathematically \n",
|
|
"as a convolution operation. \n",
|
|
"\n",
|
|
"CNNs emulate the behaviour of neurons in the visual cortex by enforcing a *local* connectivity pattern\n",
|
|
"between nodes of adjacent layers: Each node\n",
|
|
"in a convolutional layer is connected only to a subset of the nodes in the previous layer, \n",
|
|
"in contrast to the fully-connected FFNN.\n",
|
|
"Often, CNNs \n",
|
|
"consist of several convolutional layers that learn local features of the input, with a fully-connected layer at the end, \n",
|
|
"which gathers all the local data and produces the outputs. They have wide applications in image and video recognition\n",
|
|
"\n",
|
|
"## Recurrent neural networks\n",
|
|
"\n",
|
|
"So far we have only mentioned NNs where information flows in one direction: forward. *Recurrent neural networks* on\n",
|
|
"the other hand, have connections between nodes that form directed *cycles*. This creates a form of \n",
|
|
"internal memory which are able to capture information on what has been calculated before; the output is dependent \n",
|
|
"on the previous computations. Recurrent NNs make use of sequential information by performing the same task for \n",
|
|
"every element in a sequence, where each element depends on previous elements. An example of such information is \n",
|
|
"sentences, making recurrent NNs especially well-suited for handwriting and speech recognition.\n",
|
|
"\n",
|
|
"## Other types of networks\n",
|
|
"\n",
|
|
"There are many other kinds of NNs that have been developed. One type that is specifically designed for interpolation\n",
|
|
"in multidimensional space is the radial basis function (RBF) network. RBFs are typically made up of three layers: \n",
|
|
"an input layer, a hidden layer with non-linear radial symmetric activation functions and a linear output layer (''linear'' here\n",
|
|
"means that each node in the output layer has a linear activation function). The layers are normally fully-connected and \n",
|
|
"there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as\n",
|
|
"a separate type of NN due the unusual activation functions.\n",
|
|
"\n",
|
|
"\n",
|
|
"Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description\n",
|
|
"of how a fully-connected FFNN works, and how it can be used to interpolate data sets. \n",
|
|
"\n",
|
|
"## Multilayer perceptrons\n",
|
|
"\n",
|
|
"One use often so-called fully-connected feed-forward neural networks with three\n",
|
|
"or more layers (an input layer, one or more hidden layers and an output layer)\n",
|
|
"consisting of neurons that have non-linear activation functions.\n",
|
|
"\n",
|
|
"Such networks are often called *multilayer perceptrons* (MLPs)\n",
|
|
"\n",
|
|
"## Why multilayer perceptrons?\n",
|
|
"\n",
|
|
"According to the *Universal approximation theorem*, a feed-forward neural network with just a single hidden layer containing \n",
|
|
"a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, \n",
|
|
"assuming the activation function for the hidden layer is a **non-constant, bounded and monotonically-increasing continuous function**.\n",
|
|
"Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always\n",
|
|
"assumed to be linear, so as to not restrict the range of output values. \n",
|
|
"\n",
|
|
"We note that this theorem is only applicable to a NN with *one* hidden layer. \n",
|
|
"Therefore, we can easily construct an NN \n",
|
|
"that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer\n",
|
|
"with activation functions that *do*. Furthermore, although the universal approximation theorem\n",
|
|
"lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: \n",
|
|
"A neural network can still be able to approximate a given function reasonably well without having the flexibility to fit *all other*\n",
|
|
"functions. \n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"## Mathematical model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"artificialNeuron2\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" y = f\\left(\\sum_{i=1}^n w_ix_i + b_i\\right) = f(u)\n",
|
|
"\\label{artificialNeuron2} \\tag{2}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"In an FFNN of such neurons, the *inputs* $x_i$\n",
|
|
"are the *outputs* of the neurons in the preceding layer. Furthermore, an MLP is fully-connected, \n",
|
|
"which means that each neuron receives a weighted sum of the outputs of *all* neurons in the previous layer. \n",
|
|
"\n",
|
|
"## Mathematical model\n",
|
|
"\n",
|
|
"First, for each node $i$ in the first hidden layer, we calculate a weighted sum $u_i^1$ of the input coordinates $x_j$,"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto1\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" u_i^1 = \\sum_{j=1}^2 w_{ij}^1 x_j + b_i^1 \n",
|
|
"\\label{_auto1} \\tag{3}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"This value is the argument to the activation function $f_1$ of each neuron $i$,\n",
|
|
"producing the output $y_i^1$ of all neurons in layer 1,"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"outputLayer1\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" y_i^1 = f_1(u_i^1) = f_1\\left(\\sum_{j=1}^2 w_{ij}^1 x_j + b_i^1\\right)\n",
|
|
"\\label{outputLayer1} \\tag{4}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"where we assume that all nodes in the same layer have identical activation functions, hence the notation $f_l$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"generalLayer\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" y_i^l = f_l(u_i^l) = f_l\\left(\\sum_{j=1}^{N_{l-1}} w_{ij}^l y_j^{l-1} + b_i^l\\right)\n",
|
|
"\\label{generalLayer} \\tag{5}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"where $N_l$ is the number of nodes in layer $l$. When the output of all the nodes in the first hidden layer are computed,\n",
|
|
"the values of the subsequent layer can be calculated and so forth until the output is obtained. \n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"## Mathematical model\n",
|
|
"\n",
|
|
"The output of neuron $i$ in layer 2 is thus,"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto2\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" y_i^2 = f_2\\left(\\sum_{j=1}^3 w_{ij}^2 y_j^1 + b_i^2\\right) \n",
|
|
"\\label{_auto2} \\tag{6}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"outputLayer2\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation} \n",
|
|
" = f_2\\left[\\sum_{j=1}^3 w_{ij}^2f_1\\left(\\sum_{k=1}^2 w_{jk}^1 x_k + b_j^1\\right) + b_i^2\\right]\n",
|
|
"\\label{outputLayer2} \\tag{7}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"where we have substituted $y_m^1$ with. Finally, the NN output yields,"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto3\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" y_1^3 = f_3\\left(\\sum_{j=1}^3 w_{1m}^3 y_j^2 + b_1^3\\right) \n",
|
|
"\\label{_auto3} \\tag{8}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto4\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation} \n",
|
|
" = f_3\\left[\\sum_{j=1}^3 w_{1j}^3 f_2\\left(\\sum_{k=1}^3 w_{jk}^2 f_1\\left(\\sum_{m=1}^2 w_{km}^1 x_m + b_k^1\\right) + b_j^2\\right)\n",
|
|
" + b_1^3\\right]\n",
|
|
"\\label{_auto4} \\tag{9}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Mathematical model\n",
|
|
"\n",
|
|
"We can generalize this expression to an MLP with $l$ hidden layers. The complete functional form\n",
|
|
"is,"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"completeNN\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
"y^{l+1}_1\\! = \\!f_{l+1}\\!\\left[\\!\\sum_{j=1}^{N_l}\\! w_{1j}^3 f_l\\!\\left(\\!\\sum_{k=1}^{N_{l-1}}\\! w_{jk}^2 f_{l-1}\\!\\left(\\!\n",
|
|
" \\dots \\!f_1\\!\\left(\\!\\sum_{n=1}^{N_0} \\!w_{mn}^1 x_n\\! + \\!b_m^1\\!\\right)\n",
|
|
" \\!\\dots \\!\\right) \\!+ \\!b_k^2\\!\\right)\n",
|
|
" \\!+ \\!b_1^3\\!\\right] \n",
|
|
"\\label{completeNN} \\tag{10}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"which illustrates a basic property of MLPs: The only independent variables are the input values $x_n$. \n",
|
|
"\n",
|
|
"## Mathematical model\n",
|
|
"\n",
|
|
"This confirms that an MLP,\n",
|
|
"despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a \n",
|
|
"mapping of real-valued vectors $\\vec{x} \\in \\mathbb{R}^n \\rightarrow \\vec{y} \\in \\mathbb{R}^m$. \n",
|
|
"In our example, $n=2$ and $m=1$. Consequentially, \n",
|
|
"the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP. \n",
|
|
"\n",
|
|
"Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that \n",
|
|
"the expression is essentially a nested sum of scaled activation functions of the form"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto5\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" h(x) = c_1 f(c_2 x + c_3) + c_4\n",
|
|
"\\label{_auto5} \\tag{11}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"where the parameters $c_i$ are weights and biases. By adjusting these parameters, the activation functions\n",
|
|
"can be shifted up and down or left and right, change slope or be rescaled \n",
|
|
"which is the key to the flexibility of a neural network. \n",
|
|
"\n",
|
|
"### Matrix-vector notation\n",
|
|
"\n",
|
|
"We can introduce a more convenient notation for the activations in a NN. \n",
|
|
"\n",
|
|
"Additionally, we can represent the biases and activations\n",
|
|
"as layer-wise column vectors $\\vec{b}_l$ and $\\vec{y}_l$, so that the $i$-th element of each vector \n",
|
|
"is the bias $b_i^l$ and activation $y_i^l$ of node $i$ in layer $l$ respectively. \n",
|
|
"\n",
|
|
"We have that $\\mathrm{W}_l$ is a $N_{l-1} \\times N_l$ matrix, while $\\vec{b}_l$ and $\\vec{y}_l$ are $N_l \\times 1$ column vectors. \n",
|
|
"With this notation, the sum in becomes a matrix-vector multiplication, and we can write\n",
|
|
"the equation for the activations of hidden layer 2 in"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto6\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" \\vec{y}_2 = f_2(\\mathrm{W}_2 \\vec{y}_{1} + \\vec{b}_{2}) = \n",
|
|
" f_2\\left(\\left[\\begin{array}{ccc}\n",
|
|
" w^2_{11} &w^2_{12} &w^2_{13} \\\\\n",
|
|
" w^2_{21} &w^2_{22} &w^2_{23} \\\\\n",
|
|
" w^2_{31} &w^2_{32} &w^2_{33} \\\\\n",
|
|
" \\end{array} \\right] \\cdot\n",
|
|
" \\left[\\begin{array}{c}\n",
|
|
" y^1_1 \\\\\n",
|
|
" y^1_2 \\\\\n",
|
|
" y^1_3 \\\\\n",
|
|
" \\end{array}\\right] + \n",
|
|
" \\left[\\begin{array}{c}\n",
|
|
" b^2_1 \\\\\n",
|
|
" b^2_2 \\\\\n",
|
|
" b^2_3 \\\\\n",
|
|
" \\end{array}\\right]\\right).\n",
|
|
"\\label{_auto6} \\tag{12}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Matrix-vector notation and activation\n",
|
|
"\n",
|
|
"The activation of node $i$ in layer 2 is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto7\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" y^2_i = f_2\\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\\Bigr) = \n",
|
|
" f_2\\left(\\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\\right).\n",
|
|
"\\label{_auto7} \\tag{13}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"This is not just a convenient and compact notation, but also \n",
|
|
"a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications\n",
|
|
"and vector additions that are used as input to the activation functions. For each operation \n",
|
|
"$\\mathrm{W}_l \\vec{y}_{l-1}$ we move forward one layer. \n",
|
|
"\n",
|
|
"\n",
|
|
"### Activation functions\n",
|
|
"\n",
|
|
"A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s). \n",
|
|
"As described in, the following restrictions are imposed on an activation function for a FFNN\n",
|
|
"to fulfill the universal approximation theorem\n",
|
|
"\n",
|
|
" * Non-constant\n",
|
|
"\n",
|
|
" * Bounded\n",
|
|
"\n",
|
|
" * Monotonically-increasing\n",
|
|
"\n",
|
|
" * Continuous\n",
|
|
"\n",
|
|
"### Activation functions, Logistic and Hyperbolic ones\n",
|
|
"\n",
|
|
"The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each \n",
|
|
"layer simply performs a linear transformation of its inputs.\n",
|
|
"\n",
|
|
"Regardless of the number of layers, \n",
|
|
"the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of \n",
|
|
"non-linearity to the NN to be able to fit non-linear functions\n",
|
|
"Typical examples are the logistic *Sigmoid*"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"sigmoidActivationFunction\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" f(x) = \\frac{1}{1 + e^{-x}},\n",
|
|
"\\label{sigmoidActivationFunction} \\tag{14}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"and the *hyperbolic tangent* function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"tanhActivationFunction\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" f(x) = \\tanh(x)\n",
|
|
"\\label{tanhActivationFunction} \\tag{15}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Relevance\n",
|
|
"\n",
|
|
"The *sigmoid* function are more biologically plausible because \n",
|
|
"the output of inactive neurons are zero. Such activation function are called *one-sided*. However,\n",
|
|
"it has been shown that the hyperbolic tangent \n",
|
|
"performs better than the sigmoid for training MLPs. \n",
|
|
"has become the most popular for *deep neural networks*"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|