Morten Hjorth-Jensen
442 lines
17 KiB
HTML
442 lines
17 KiB
HTML
<!--
|
|
Automatically generated HTML file from DocOnce source
|
|
(https://github.com/hplgit/doconce/)
|
|
-->
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
|
<meta name="description" content="Data Analysis and Machine Learning: Elements of machine learning">
|
|
|
|
<title>Data Analysis and Machine Learning: Elements of machine learning</title>
|
|
|
|
|
|
<style type="text/css">
|
|
/* bloodish style */
|
|
|
|
body {
|
|
font-family: Helvetica, Verdana, Arial, Sans-serif;
|
|
color: #404040;
|
|
background: #ffffff;
|
|
}
|
|
h1 { font-size: 1.8em; color: #8A0808; }
|
|
h2 { font-size: 1.6em; color: #8A0808; }
|
|
h3 { font-size: 1.4em; color: #8A0808; }
|
|
h4 { color: #8A0808; }
|
|
a { color: #8A0808; text-decoration:none; }
|
|
tt { font-family: "Courier New", Courier; }
|
|
/* pre style removed because it will interfer with pygments */
|
|
p { text-indent: 0px; }
|
|
hr { border: 0; width: 80%; border-bottom: 1px solid #aaa}
|
|
p.caption { width: 80%; font-style: normal; text-align: left; }
|
|
hr.figure { border: 0; width: 80%; border-bottom: 1px solid #aaa}
|
|
|
|
div { text-align: justify; text-justify: inter-word; }
|
|
</style>
|
|
|
|
|
|
</head>
|
|
|
|
<!-- tocinfo
|
|
{'highest level': 2,
|
|
'sections': [('What is Machine Learning?', 2, None, '___sec0'),
|
|
('Types of Machine Learning', 2, None, '___sec1'),
|
|
('Artificial neurons', 2, None, '___sec2'),
|
|
('Neural network types', 2, None, '___sec3'),
|
|
('Feed-forward neural networks', 2, None, '___sec4'),
|
|
('Recurrent neural networks', 2, None, '___sec5'),
|
|
('Other types of networks', 2, None, '___sec6'),
|
|
('Mathematical model', 2, None, '___sec7'),
|
|
('Mathematical model', 2, None, '___sec8'),
|
|
('Mathematical model', 2, None, '___sec9'),
|
|
('Mathematical model', 2, None, '___sec10'),
|
|
('Mathematical model', 2, None, '___sec11'),
|
|
('Matrix-vector notation', 3, None, '___sec12')]}
|
|
end of tocinfo -->
|
|
|
|
<body>
|
|
|
|
|
|
|
|
<script type="text/x-mathjax-config">
|
|
MathJax.Hub.Config({
|
|
TeX: {
|
|
equationNumbers: { autoNumber: "AMS" },
|
|
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
|
}
|
|
});
|
|
</script>
|
|
<script type="text/javascript"
|
|
src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
|
</script>
|
|
|
|
|
|
|
|
|
|
<!-- ------------------- main content ---------------------- -->
|
|
|
|
|
|
|
|
<center><h1>Data Analysis and Machine Learning: Elements of machine learning</h1></center> <!-- document title -->
|
|
|
|
<p>
|
|
<!-- author(s): Morten Hjorth-Jensen -->
|
|
|
|
<center>
|
|
<b>Morten Hjorth-Jensen</b> [1, 2]
|
|
</center>
|
|
|
|
<p>
|
|
<!-- institution(s) -->
|
|
|
|
<center>[1] <b>Department of Physics, University of Oslo</b></center>
|
|
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
|
<br>
|
|
<p>
|
|
<center><h4>Nov 2, 2017</h4></center> <!-- date -->
|
|
<br>
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec0">What is Machine Learning? </h2>
|
|
|
|
<p>
|
|
Machine learning is the science of giving computers the ability to learn without being explicitly programmed.
|
|
The idea is that there exist generic algorithms which can be used to find patterns in a broad class of data sets without
|
|
having to write code specifically for each problem. The algorithm will build its own logic based on the data.
|
|
|
|
<p>
|
|
Machine learning is a subfield of computer science, and is closely related to computational statistics.
|
|
It evolved from the study of pattern recognition in artificial intelligence (AI) research, and has made contributions to
|
|
AI tasks like computer vision, natural language processing
|
|
and speech recognition. It has also, especially in later years,
|
|
found applications in a wide variety of other areas, including bioinformatics, economy, physics, finance and marketing.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec1">Types of Machine Learning </h2>
|
|
|
|
<p>
|
|
The approaches to machine learning are many, but are often split into two main categories.
|
|
In <em>supervised learning</em> we know the answer to a problem,
|
|
and let the computer deduce the logic behind it. On the other hand, <em>unsupervised learning</em>
|
|
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
|
|
Some authours also operate with a third category, namely <em>reinforcement learning</em>. This is a paradigm
|
|
of learning inspired by behavioural psychology, where learning is achieved by trial-and-error,
|
|
solely from rewards and punishment.
|
|
|
|
<p>
|
|
Another way to categorize machine learning tasks is to consider the desired output of a system.
|
|
Some of the most common tasks are:
|
|
|
|
<ul>
|
|
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
|
|
<li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
|
|
<li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
|
|
</ul>
|
|
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec2">Artificial neurons </h2>
|
|
The field of artificial neural networks has a long history of development, and is closely connected with
|
|
the advancement of computer science and computers in general. A model of artificial neurons
|
|
was first developed by McCulloch and Pitts in 1943 to study signal processing in the brain and
|
|
has later been refined by others. The general idea is to mimic neural networks in the human brain, which
|
|
is composed of billions of neurons that communicate with each other by sending electrical signals.
|
|
Each neuron accumulates its incoming signals,
|
|
which must exceed an activation threshold to yield an output. If the threshold is not overcome, the neuron
|
|
remains inactive, i.e. has zero output.
|
|
|
|
<p>
|
|
This behaviour has inspired a simple mathematical model for an artificial neuron.
|
|
|
|
$$
|
|
\begin{equation}
|
|
y = f\left(\sum_{i=1}^n w_ix_i\right) = f(u)
|
|
\label{artificialNeuron}
|
|
\end{equation}
|
|
$$
|
|
|
|
Here, the output \( y \) of the neuron is the value of its activation function, which have as input
|
|
a weighted sum of signals \( x_i, \dots ,x_n \) received by \( n \) other neurons.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec3">Neural network types </h2>
|
|
|
|
<p>
|
|
An artificial neural network (NN), is a computational model that consists of layers of connected neurons, or <em>nodes</em>.
|
|
It is supposed to mimic a biological nervous system by letting each neuron interact with other neurons
|
|
by sending signals in the form of mathematical functions between layers.
|
|
A wide variety of different NNs have
|
|
been developed, but most of them consist of an input layer, an output layer and eventual layers in-between, called
|
|
<em>hidden layers</em>. All layers can contain an arbitrary number of nodes, and each connection between two nodes
|
|
is associated with a weight variable.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec4">Feed-forward neural networks </h2>
|
|
The feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network,
|
|
the information moves in only one direction: forward through the layers.
|
|
|
|
<p>
|
|
Nodes are represented by circles, while the arrows display the connections between the nodes, including the
|
|
direction of information flow. Additionally, each arrow corresponds to a weight variable, not displayed here.
|
|
We observe that each node in a layer is connected to <em>all</em> nodes in the subsequent layer,
|
|
making this a so-called <em>fully-connected</em> FFNN.
|
|
|
|
<p>
|
|
A different variant of FFNNs are <em>convolutional neural networks</em> (CNNs), which have a connectivity pattern
|
|
inspired by the animal visual cortex. Individual neurons in the visual cortex only respond to stimuli from
|
|
small sub-regions of the visual field, called a receptive field. This makes the neurons well-suited to exploit the strong
|
|
spatially local correlation present in natural images. The response of each neuron can be approximated mathematically
|
|
as a convolution operation.
|
|
|
|
<p>
|
|
CNNs emulate the behaviour of neurons in the visual cortex by enforcing a <em>local</em> connectivity pattern
|
|
between nodes of adjacent layers: Each node
|
|
in a convolutional layer is connected only to a subset of the nodes in the previous layer,
|
|
in contrast to the fully-connected FFNN.
|
|
Often, CNNs
|
|
consist of several convolutional layers that learn local features of the input, with a fully-connected layer at the end,
|
|
which gathers all the local data and produces the outputs. They have wide applications in image and video recognition
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec5">Recurrent neural networks </h2>
|
|
|
|
<p>
|
|
So far we have only mentioned NNs where information flows in one direction: forward. <em>Recurrent neural networks</em> on
|
|
the other hand, have connections between nodes that form directed <em>cycles</em>. This creates a form of
|
|
internal memory which are able to capture information on what has been calculated before; the output is dependent
|
|
on the previous computations. Recurrent NNs make use of sequential information by performing the same task for
|
|
every element in a sequence, where each element depends on previous elements. An example of such information is
|
|
sentences, making recurrent NNs especially well-suited for handwriting and speech recognition.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec6">Other types of networks </h2>
|
|
|
|
<p>
|
|
There are many other kinds of NNs that have been developed. One type that is specifically designed for interpolation
|
|
in multidimensional space is the radial basis function (RBF) network. RBFs are typically made up of three layers:
|
|
an input layer, a hidden layer with non-linear radial symmetric activation functions and a linear output layer (''linear'' here
|
|
means that each node in the output layer has a linear activation function). The layers are normally fully-connected and
|
|
there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as
|
|
a separate type of NN due the unusual activation functions.
|
|
|
|
<p>
|
|
Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description
|
|
of how a fully-connected FFNN works, and how it can be used to interpolate data sets.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec7">Mathematical model </h2>
|
|
|
|
$$
|
|
\begin{equation}
|
|
y = f\left(\sum_{i=1}^n w_ix_i + b_i\right) = f(u)
|
|
\label{artificialNeuron2}
|
|
\end{equation}
|
|
$$
|
|
|
|
In an FFNN of such neurons, the <em>inputs</em> \( x_i \)
|
|
are the <em>outputs</em> of the neurons in the preceding layer. Furthermore, a MLP is fully-connected,
|
|
which means that each neuron receives a weighted sum of the outputs of <em>all</em> neurons in the previous layer.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec8">Mathematical model </h2>
|
|
|
|
<p>
|
|
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( u_i^1 \) of the input coordinates \( x_j \),
|
|
|
|
$$
|
|
\begin{equation}
|
|
u_i^1 = \sum_{j=1}^2 w_{ij}^1 x_j + b_i^1
|
|
\label{_auto1}
|
|
\end{equation}
|
|
$$
|
|
|
|
This value is the argument to the activation function \( f_1 \) of each neuron \( i \),
|
|
producing the output \( y_i^1 \) of all neurons in layer 1,
|
|
|
|
$$
|
|
\begin{equation}
|
|
y_i^1 = f_1(u_i^1) = f_1\left(\sum_{j=1}^2 w_{ij}^1 x_j + b_i^1\right)
|
|
\label{outputLayer1}
|
|
\end{equation}
|
|
$$
|
|
|
|
where we assume that all nodes in the same layer have identical activation functions, hence the notation \( f_l \)
|
|
|
|
$$
|
|
\begin{equation}
|
|
y_i^l = f_l(u_i^l) = f_l\left(\sum_{j=1}^{N_{l-1}} w_{ij}^l y_j^{l-1} + b_i^l\right)
|
|
\label{generalLayer}
|
|
\end{equation}
|
|
$$
|
|
|
|
where \( N_l \) is the number of nodes in layer \( l \). When the output of all the nodes in the first hidden layer are computed,
|
|
the values of the subsequent layer can be calculated and so forth until the output is obtained.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec9">Mathematical model </h2>
|
|
|
|
<p>
|
|
The output of neuron \( i \) in layer 2 is thus,
|
|
|
|
$$
|
|
\begin{align}
|
|
y_i^2 &= f_2\left(\sum_{j=1}^3 w_{ij}^2 y_j^1 + b_i^2\right)
|
|
\label{_auto2}\\
|
|
&= f_2\left[\sum_{j=1}^3 w_{ij}^2f_1\left(\sum_{k=1}^2 w_{jk}^1 x_k + b_j^1\right) + b_i^2\right]
|
|
\label{outputLayer2}
|
|
\end{align}
|
|
$$
|
|
|
|
where we have substituted \( y_m^1 \) with. Finally, the NN output yields,
|
|
|
|
$$
|
|
\begin{align}
|
|
y_1^3 &= f_3\left(\sum_{j=1}^3 w_{1m}^3 y_j^2 + b_1^3\right)
|
|
\label{_auto3}\\
|
|
&= f_3\left[\sum_{j=1}^3 w_{1j}^3 f_2\left(\sum_{k=1}^3 w_{jk}^2 f_1\left(\sum_{m=1}^2 w_{km}^1 x_m + b_k^1\right) + b_j^2\right)
|
|
+ b_1^3\right]
|
|
\label{_auto4}
|
|
\end{align}
|
|
$$
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec10">Mathematical model </h2>
|
|
|
|
<p>
|
|
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
|
|
is,
|
|
$$
|
|
\begin{align}
|
|
&y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\!
|
|
\dots \!f_1\!\left(\!\sum_{n=1}^{N_0} \!w_{mn}^1 x_n\! + \!b_m^1\!\right)
|
|
\!\dots \!\right) \!+ \!b_k^2\!\right)
|
|
\!+ \!b_1^3\!\right] &&
|
|
\label{completeNN}
|
|
\end{align}
|
|
$$
|
|
|
|
which illustrates a basic property of MLPs: The only independent variables are the input values \( x_n \).
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec11">Mathematical model </h2>
|
|
|
|
<p>
|
|
This confirms that an MLP,
|
|
despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a
|
|
mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m \).
|
|
In our example, \( n=2 \) and \( m=1 \). Consequentially,
|
|
the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP.
|
|
|
|
<p>
|
|
Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that
|
|
the expression is essentially a nested sum of scaled activation functions of the form
|
|
|
|
$$
|
|
\begin{equation}
|
|
h(x) = c_1 f(c_2 x + c_3) + c_4
|
|
\label{_auto5}
|
|
\end{equation}
|
|
$$
|
|
|
|
where the parameters \( c_i \) are weights and biases. By adjusting these parameters, the activation functions
|
|
can be shifted up and down or left and right, change slope or be rescaled
|
|
which is the key to the flexibility of a NN.
|
|
|
|
$$
|
|
\begin{equation}
|
|
f_o = f(u_o) = u_o
|
|
\label{outputActivation}
|
|
\end{equation}
|
|
$$
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h3 id="___sec12">Matrix-vector notation </h3>
|
|
|
|
We can introduce a more convenient notation for the activations in a NN.
|
|
|
|
<p>
|
|
Additionally, we can represent the biases and activations
|
|
as layer-wise column vectors \( \vec{b}_l \) and \( \vec{y}_l \), so that the \( i \)-th element of each vector
|
|
is the bias \( b_i^l \) and activation \( y_i^l \) of node \( i \) in layer \( l \) respectively.
|
|
|
|
<p>
|
|
We have that \( \mathrm{W}_l \) is a \( N_{l-1} \times N_l \) matrix, while \( \vec{b}_l \) and \( \vec{y}_l \) are \( N_l \times 1 \) column vectors.
|
|
With this notation, the sum in becomes a matrix-vector multiplication, and we can write
|
|
the equation for the activations of hidden layer 2 in
|
|
|
|
$$
|
|
\begin{equation}
|
|
\vec{y}_2 = f_2(\mathrm{W}_2 \vec{y}_{1} + \vec{b}_{2}) =
|
|
f_2\left(\left[\begin{array}{ccc}
|
|
w^2_{11} &w^2_{12} &w^2_{13} \\
|
|
w^2_{21} &w^2_{22} &w^2_{23} \\
|
|
w^2_{31} &w^2_{32} &w^2_{33} \\
|
|
\end{array} \right] \cdot
|
|
\left[\begin{array}{c}
|
|
y^1_1 \\
|
|
y^1_2 \\
|
|
y^1_3 \\
|
|
\end{array}\right] +
|
|
\left[\begin{array}{c}
|
|
b^2_1 \\
|
|
b^2_2 \\
|
|
b^2_3 \\
|
|
\end{array}\right]\right)
|
|
\label{_auto6}
|
|
\end{equation}
|
|
$$
|
|
|
|
and we see that the activation of node \( i \) in layer 2 is
|
|
|
|
$$
|
|
\begin{equation}
|
|
y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) =
|
|
f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right)
|
|
\label{_auto7}
|
|
\end{equation}
|
|
$$
|
|
|
|
which is in accordance with. Note that
|
|
This is not just a convenient and compact notation, but also
|
|
a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
|
|
and vector additions that are used as input to the activation functions. For each operation
|
|
\( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
|
|
|
|
<p>
|
|
|
|
<!-- ------------------- end of main content --------------- -->
|
|
|
|
|
|
<center style="font-size:80%">
|
|
<!-- copyright --> © 1999-2017, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
|
|
</center>
|
|
|
|
|
|
</body>
|
|
</html>
|
|
|
|
|