lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on neural nets

Browse Source
This commit is contained in:
mhjensen
2018-09-28 11:14:05 +02:00
parent a21be0fb51
commit 4f225ce503
13 changed files with 455 additions and 1528 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/NeuralNet/html/NeuralNet.html
+57 -75
View File
@@ -39,28 +39,31 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('What is Machine Learning?', 2, None, '___sec0'),
('Types of Machine Learning', 2, None, '___sec1'),
('Artificial neurons', 2, None, '___sec2'),
('Neural network types', 2, None, '___sec3'),
('Feed-forward neural networks', 2, None, '___sec4'),
('Recurrent neural networks', 2, None, '___sec5'),
('Other types of networks', 2, None, '___sec6'),
('Multilayer perceptrons', 2, None, '___sec7'),
('Why multilayer perceptrons?', 2, None, '___sec8'),
'sections': [('Neural networks', 2, None, '___sec0'),
('Artificial neurons', 2, None, '___sec1'),
('Neural network types', 2, None, '___sec2'),
('Feed-forward neural networks', 2, None, '___sec3'),
('Recurrent neural networks', 2, None, '___sec4'),
('Other types of networks', 2, None, '___sec5'),
('Multilayer perceptrons', 2, None, '___sec6'),
('Why multilayer perceptrons?', 2, None, '___sec7'),
('Mathematical model', 2, None, '___sec8'),
('Mathematical model', 2, None, '___sec9'),
('Mathematical model', 2, None, '___sec10'),
('Mathematical model', 2, None, '___sec11'),
('Mathematical model', 2, None, '___sec12'),
('Mathematical model', 2, None, '___sec13'),
('Matrix-vector notation', 3, None, '___sec14'),
('Matrix-vector notation and activation', 3, None, '___sec15'),
('Activation functions', 3, None, '___sec16'),
('Matrix-vector notation', 3, None, '___sec13'),
('Matrix-vector notation and activation', 3, None, '___sec14'),
('Activation functions', 3, None, '___sec15'),
('Activation functions, Logistic and Hyperbolic ones',
3,
None,
'___sec17'),
('Relevance', 3, None, '___sec18'),
'___sec16'),
('Relevance', 3, None, '___sec17'),
('Setting up a Multi-layer perceptron model',
2,
None,
'___sec18'),
('Two-layer Neural Network', 2, None, '___sec19')]}
end of tocinfo -->
@@ -103,56 +106,29 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>May 30, 2018</h4></center> <!-- date -->
<center><h4>Sep 28, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">What is Machine Learning? </h2>
<p>
Machine learning is the science of giving computers the ability to
learn without being explicitly programmed. The idea is that there
exist generic algorithms which can be used to find patterns in a broad
class of data sets without having to write code specifically for each
problem. The algorithm will build its own logic based on the data.
<p>
Machine learning is a subfield of computer science, and is closely
related to computational statistics. It evolved from the study of
pattern recognition in artificial intelligence (AI) research, and has
made contributions to AI tasks like computer vision, natural language
processing and speech recognition. It has also, especially in later
years, found applications in a wide variety of other areas, including
bioinformatics, economy, physics, finance and marketing.
<!-- add own code for DNN -->
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Types of Machine Learning </h2>
<h2 id="___sec0">Neural networks </h2>
<p>
The approaches to machine learning are many, but are often split into two main categories.
In <em>supervised learning</em> we know the answer to a problem,
and let the computer deduce the logic behind it. On the other hand, <em>unsupervised learning</em>
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
Some authours also operate with a third category, namely <em>reinforcement learning</em>. This is a paradigm
of learning inspired by behavioural psychology, where learning is achieved by trial-and-error,
solely from rewards and punishment.
Artificial neural networks are computational systems that can learn to
perform tasks by considering examples, generally without being
programmed with any task-specific rules. It is supposed to mimic a
biological system, wherein neurons interact by sending signals in the
form of mathematical functions between layers. All layers can contain
an arbitrary number of neurons, and each connection is represented by
a weight variable.
<p>
Another way to categorize machine learning tasks is to consider the desired output of a system.
Some of the most common tasks are:
<ul>
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
<li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Artificial neurons </h2>
<h2 id="___sec1">Artificial neurons </h2>
<p>
The field of artificial neural networks has a long history of
@@ -191,13 +167,13 @@ categories:
<li> neural networks for unsupervised learning such as Deep Boltzmann Machines.</li>
</ol>
In physics, DNNs and CNNs have already found numerous applications. In
In natural science, DNNs and CNNs have already found numerous applications. In
statistical physics, they have been applied to detect phase
transitions in 2D Ising and Potts models, lattice gauge theories, and
different phases of polymers.
different phases of polymers, or solving the Navier-Stokes equation in weather forecasting.
Deep learning has also found interesting applications in quantum
physics. Various quantum phase transitions can be detected and studied
using DNNs and CNNs, including the transverse-field Ising model,
using DNNs and CNNs,
topological phases, and even non-equilibrium many-body
localization. Representing quantum states as DNNs quantum state
tomography are among some of the impressive
@@ -208,14 +184,15 @@ of quantum systems.
In quantum information theory, it has been shown that one can perform
gate decompositions with the help of neural. In lattice quantum chromodynamics,
DNNs have been used to learn action parameters in regions of parameter
space where PCA fails. Last but not least,
DNNs also found place in the study of quantum, and in scattering theory to learn
\( s \)-wave scattering length of potentials.
space where PCA fails.
<p>
The applications are not limited to the natural sciences. There is a plethora of applications in essentially all disciplines, from the humanities to life science and medicine.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Neural network types </h2>
<h2 id="___sec2">Neural network types </h2>
<p>
An artificial neural network (NN), is a computational model that
@@ -238,7 +215,7 @@ methods.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Feed-forward neural networks </h2>
<h2 id="___sec3">Feed-forward neural networks </h2>
<p>
The feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network,
@@ -269,7 +246,7 @@ which gathers all the local data and produces the outputs. They have wide applic
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Recurrent neural networks </h2>
<h2 id="___sec4">Recurrent neural networks </h2>
<p>
So far we have only mentioned NNs where information flows in one direction: forward. <em>Recurrent neural networks</em> on
@@ -282,7 +259,7 @@ sentences, making recurrent NNs especially well-suited for handwriting and speec
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Other types of networks </h2>
<h2 id="___sec5">Other types of networks </h2>
<p>
There are many other kinds of NNs that have been developed. One type that is specifically designed for interpolation
@@ -299,7 +276,7 @@ of how a fully-connected FFNN works, and how it can be used to interpolate data
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Multilayer perceptrons </h2>
<h2 id="___sec6">Multilayer perceptrons </h2>
<p>
One use often so-called fully-connected feed-forward neural networks with three
@@ -312,7 +289,7 @@ Such networks are often called <em>multilayer perceptrons</em> (MLPs)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Why multilayer perceptrons? </h2>
<h2 id="___sec7">Why multilayer perceptrons? </h2>
<p>
According to the <em>Universal approximation theorem</em>, a feed-forward neural network with just a single hidden layer containing
@@ -333,7 +310,7 @@ functions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Mathematical model </h2>
<h2 id="___sec8">Mathematical model </h2>
$$
\begin{equation}
@@ -349,7 +326,7 @@ which means that each neuron receives a weighted sum of the outputs of <em>all</
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Mathematical model </h2>
<h2 id="___sec9">Mathematical model </h2>
<p>
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( u_i^1 \) of the input coordinates \( x_j \),
@@ -386,7 +363,7 @@ the values of the subsequent layer can be calculated and so forth until the outp
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">Mathematical model </h2>
<h2 id="___sec10">Mathematical model </h2>
<p>
The output of neuron \( i \) in layer 2 is thus,
@@ -415,7 +392,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Mathematical model </h2>
<h2 id="___sec11">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
@@ -435,7 +412,7 @@ which illustrates a basic property of MLPs: The only independent variables are t
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Mathematical model </h2>
<h2 id="___sec12">Mathematical model </h2>
<p>
This confirms that an MLP,
@@ -462,7 +439,7 @@ which is the key to the flexibility of a neural network.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec14">Matrix-vector notation </h3>
<h3 id="___sec13">Matrix-vector notation </h3>
We can introduce a more convenient notation for the activations in a NN.
@@ -500,7 +477,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec15">Matrix-vector notation and activation </h3>
<h3 id="___sec14">Matrix-vector notation and activation </h3>
The activation of node \( i \) in layer 2 is
@@ -520,7 +497,7 @@ and vector additions that are used as input to the activation functions. For eac
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec16">Activation functions </h3>
<h3 id="___sec15">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
@@ -536,7 +513,7 @@ to fulfill the universal approximation theorem
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec17">Activation functions, Logistic and Hyperbolic ones </h3>
<h3 id="___sec16">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
@@ -565,7 +542,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec18">Relevance </h3>
<h3 id="___sec17">Relevance </h3>
The <em>sigmoid</em> function are more biologically plausible because
the output of inactive neurons are zero. Such activation function are called <em>one-sided</em>. However,
@@ -648,6 +625,11 @@ ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec18">Setting up a Multi-layer perceptron model </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->