lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on neural nets

Browse Source
This commit is contained in:
mhjensen
2018-09-28 11:14:05 +02:00
parent a21be0fb51
commit 4f225ce503
13 changed files with 455 additions and 1528 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/LogReg/ipynb/LogReg.ipynb
+21 -7
View File
@@ -388,9 +388,7 @@
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -425,9 +423,7 @@
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
@@ -482,7 +478,25 @@
]
}
],
"metadata": {},
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
doc/pub/NeuralNet/html/NeuralNet-bs.html
+74 -92
View File
@@ -40,28 +40,31 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('What is Machine Learning?', 2, None, '___sec0'),
('Types of Machine Learning', 2, None, '___sec1'),
('Artificial neurons', 2, None, '___sec2'),
('Neural network types', 2, None, '___sec3'),
('Feed-forward neural networks', 2, None, '___sec4'),
('Recurrent neural networks', 2, None, '___sec5'),
('Other types of networks', 2, None, '___sec6'),
('Multilayer perceptrons', 2, None, '___sec7'),
('Why multilayer perceptrons?', 2, None, '___sec8'),
'sections': [('Neural networks', 2, None, '___sec0'),
('Artificial neurons', 2, None, '___sec1'),
('Neural network types', 2, None, '___sec2'),
('Feed-forward neural networks', 2, None, '___sec3'),
('Recurrent neural networks', 2, None, '___sec4'),
('Other types of networks', 2, None, '___sec5'),
('Multilayer perceptrons', 2, None, '___sec6'),
('Why multilayer perceptrons?', 2, None, '___sec7'),
('Mathematical model', 2, None, '___sec8'),
('Mathematical model', 2, None, '___sec9'),
('Mathematical model', 2, None, '___sec10'),
('Mathematical model', 2, None, '___sec11'),
('Mathematical model', 2, None, '___sec12'),
('Mathematical model', 2, None, '___sec13'),
('Matrix-vector notation', 3, None, '___sec14'),
('Matrix-vector notation and activation', 3, None, '___sec15'),
('Activation functions', 3, None, '___sec16'),
('Matrix-vector notation', 3, None, '___sec13'),
('Matrix-vector notation and activation', 3, None, '___sec14'),
('Activation functions', 3, None, '___sec15'),
('Activation functions, Logistic and Hyperbolic ones',
3,
None,
'___sec17'),
('Relevance', 3, None, '___sec18'),
'___sec16'),
('Relevance', 3, None, '___sec17'),
('Setting up a Multi-layer perceptron model',
2,
None,
'___sec18'),
('Two-layer Neural Network', 2, None, '___sec19')]}
end of tocinfo -->
@@ -100,25 +103,25 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;"><b>What is Machine Learning?</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;"><b>Types of Machine Learning</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;"><b>Artificial neurons</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;"><b>Neural network types</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;"><b>Feed-forward neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;"><b>Recurrent neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;"><b>Other types of networks</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;"><b>Multilayer perceptrons</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;"><b>Why multilayer perceptrons?</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;"><b>Neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;"><b>Artificial neurons</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;"><b>Neural network types</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;"><b>Feed-forward neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;"><b>Recurrent neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;"><b>Other types of networks</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;"><b>Multilayer perceptrons</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;"><b>Why multilayer perceptrons?</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec10" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec11" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec12" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec13" style="font-size: 80%;"><b>Mathematical model</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec14" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Matrix-vector notation</a></li>
<!-- navigation toc: --> <li><a href="#___sec15" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Matrix-vector notation and activation</a></li>
<!-- navigation toc: --> <li><a href="#___sec16" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Activation functions</a></li>
<!-- navigation toc: --> <li><a href="#___sec17" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Activation functions, Logistic and Hyperbolic ones</a></li>
<!-- navigation toc: --> <li><a href="#___sec18" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Relevance</a></li>
<!-- navigation toc: --> <li><a href="#___sec13" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Matrix-vector notation</a></li>
<!-- navigation toc: --> <li><a href="#___sec14" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Matrix-vector notation and activation</a></li>
<!-- navigation toc: --> <li><a href="#___sec15" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Activation functions</a></li>
<!-- navigation toc: --> <li><a href="#___sec16" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Activation functions, Logistic and Hyperbolic ones</a></li>
<!-- navigation toc: --> <li><a href="#___sec17" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Relevance</a></li>
<!-- navigation toc: --> <li><a href="#___sec18" style="font-size: 80%;"><b>Setting up a Multi-layer perceptron model</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec19" style="font-size: 80%;"><b>Two-layer Neural Network</b></a></li>
</ul>
@@ -153,59 +156,32 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>May 30, 2018</h4></center> <!-- date -->
<center><h4>Sep 28, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- add own code for DNN -->
<p>
<!-- potential-jumbotron-button -->
</div> <!-- end jumbotron -->
<!-- !split -->
<h2 id="___sec0" class="anchor">What is Machine Learning? </h2>
<h2 id="___sec0" class="anchor">Neural networks </h2>
<p>
Machine learning is the science of giving computers the ability to
learn without being explicitly programmed. The idea is that there
exist generic algorithms which can be used to find patterns in a broad
class of data sets without having to write code specifically for each
problem. The algorithm will build its own logic based on the data.
<p>
Machine learning is a subfield of computer science, and is closely
related to computational statistics. It evolved from the study of
pattern recognition in artificial intelligence (AI) research, and has
made contributions to AI tasks like computer vision, natural language
processing and speech recognition. It has also, especially in later
years, found applications in a wide variety of other areas, including
bioinformatics, economy, physics, finance and marketing.
Artificial neural networks are computational systems that can learn to
perform tasks by considering examples, generally without being
programmed with any task-specific rules. It is supposed to mimic a
biological system, wherein neurons interact by sending signals in the
form of mathematical functions between layers. All layers can contain
an arbitrary number of neurons, and each connection is represented by
a weight variable.
<p>
<!-- !split -->
<h2 id="___sec1" class="anchor">Types of Machine Learning </h2>
<p>
The approaches to machine learning are many, but are often split into two main categories.
In <em>supervised learning</em> we know the answer to a problem,
and let the computer deduce the logic behind it. On the other hand, <em>unsupervised learning</em>
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
Some authours also operate with a third category, namely <em>reinforcement learning</em>. This is a paradigm
of learning inspired by behavioural psychology, where learning is achieved by trial-and-error,
solely from rewards and punishment.
<p>
Another way to categorize machine learning tasks is to consider the desired output of a system.
Some of the most common tasks are:
<ul>
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
<li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
</ul>
<!-- !split -->
<h2 id="___sec2" class="anchor">Artificial neurons </h2>
<h2 id="___sec1" class="anchor">Artificial neurons </h2>
<p>
The field of artificial neural networks has a long history of
@@ -244,13 +220,13 @@ categories:
<li> neural networks for unsupervised learning such as Deep Boltzmann Machines.</li>
</ol>
In physics, DNNs and CNNs have already found numerous applications. In
In natural science, DNNs and CNNs have already found numerous applications. In
statistical physics, they have been applied to detect phase
transitions in 2D Ising and Potts models, lattice gauge theories, and
different phases of polymers.
different phases of polymers, or solving the Navier-Stokes equation in weather forecasting.
Deep learning has also found interesting applications in quantum
physics. Various quantum phase transitions can be detected and studied
using DNNs and CNNs, including the transverse-field Ising model,
using DNNs and CNNs,
topological phases, and even non-equilibrium many-body
localization. Representing quantum states as DNNs quantum state
tomography are among some of the impressive
@@ -261,14 +237,15 @@ of quantum systems.
In quantum information theory, it has been shown that one can perform
gate decompositions with the help of neural. In lattice quantum chromodynamics,
DNNs have been used to learn action parameters in regions of parameter
space where PCA fails. Last but not least,
DNNs also found place in the study of quantum, and in scattering theory to learn
\( s \)-wave scattering length of potentials.
space where PCA fails.
<p>
The applications are not limited to the natural sciences. There is a plethora of applications in essentially all disciplines, from the humanities to life science and medicine.
<p>
<!-- !split -->
<h2 id="___sec3" class="anchor">Neural network types </h2>
<h2 id="___sec2" class="anchor">Neural network types </h2>
<p>
An artificial neural network (NN), is a computational model that
@@ -291,7 +268,7 @@ methods.
<p>
<!-- !split -->
<h2 id="___sec4" class="anchor">Feed-forward neural networks </h2>
<h2 id="___sec3" class="anchor">Feed-forward neural networks </h2>
<p>
The feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network,
@@ -322,7 +299,7 @@ which gathers all the local data and produces the outputs. They have wide applic
<p>
<!-- !split -->
<h2 id="___sec5" class="anchor">Recurrent neural networks </h2>
<h2 id="___sec4" class="anchor">Recurrent neural networks </h2>
<p>
So far we have only mentioned NNs where information flows in one direction: forward. <em>Recurrent neural networks</em> on
@@ -335,7 +312,7 @@ sentences, making recurrent NNs especially well-suited for handwriting and speec
<p>
<!-- !split -->
<h2 id="___sec6" class="anchor">Other types of networks </h2>
<h2 id="___sec5" class="anchor">Other types of networks </h2>
<p>
There are many other kinds of NNs that have been developed. One type that is specifically designed for interpolation
@@ -352,7 +329,7 @@ of how a fully-connected FFNN works, and how it can be used to interpolate data
<p>
<!-- !split -->
<h2 id="___sec7" class="anchor">Multilayer perceptrons </h2>
<h2 id="___sec6" class="anchor">Multilayer perceptrons </h2>
<p>
One use often so-called fully-connected feed-forward neural networks with three
@@ -365,7 +342,7 @@ Such networks are often called <em>multilayer perceptrons</em> (MLPs)
<p>
<!-- !split -->
<h2 id="___sec8" class="anchor">Why multilayer perceptrons? </h2>
<h2 id="___sec7" class="anchor">Why multilayer perceptrons? </h2>
<p>
According to the <em>Universal approximation theorem</em>, a feed-forward neural network with just a single hidden layer containing
@@ -386,7 +363,7 @@ functions.
<p>
<!-- !split -->
<h2 id="___sec9" class="anchor">Mathematical model </h2>
<h2 id="___sec8" class="anchor">Mathematical model </h2>
$$
\begin{equation}
@@ -402,7 +379,7 @@ which means that each neuron receives a weighted sum of the outputs of <em>all</
<p>
<!-- !split -->
<h2 id="___sec10" class="anchor">Mathematical model </h2>
<h2 id="___sec9" class="anchor">Mathematical model </h2>
<p>
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( u_i^1 \) of the input coordinates \( x_j \),
@@ -439,7 +416,7 @@ the values of the subsequent layer can be calculated and so forth until the outp
<p>
<!-- !split -->
<h2 id="___sec11" class="anchor">Mathematical model </h2>
<h2 id="___sec10" class="anchor">Mathematical model </h2>
<p>
The output of neuron \( i \) in layer 2 is thus,
@@ -468,7 +445,7 @@ $$
<p>
<!-- !split -->
<h2 id="___sec12" class="anchor">Mathematical model </h2>
<h2 id="___sec11" class="anchor">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
@@ -488,7 +465,7 @@ which illustrates a basic property of MLPs: The only independent variables are t
<p>
<!-- !split -->
<h2 id="___sec13" class="anchor">Mathematical model </h2>
<h2 id="___sec12" class="anchor">Mathematical model </h2>
<p>
This confirms that an MLP,
@@ -515,7 +492,7 @@ which is the key to the flexibility of a neural network.
<p>
<!-- !split -->
<h3 id="___sec14" class="anchor">Matrix-vector notation </h3>
<h3 id="___sec13" class="anchor">Matrix-vector notation </h3>
We can introduce a more convenient notation for the activations in a NN.
@@ -553,7 +530,7 @@ $$
<p>
<!-- !split -->
<h3 id="___sec15" class="anchor">Matrix-vector notation and activation </h3>
<h3 id="___sec14" class="anchor">Matrix-vector notation and activation </h3>
The activation of node \( i \) in layer 2 is
@@ -573,7 +550,7 @@ and vector additions that are used as input to the activation functions. For eac
<p>
<!-- !split -->
<h3 id="___sec16" class="anchor">Activation functions </h3>
<h3 id="___sec15" class="anchor">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
@@ -589,7 +566,7 @@ to fulfill the universal approximation theorem
<!-- !split -->
<h3 id="___sec17" class="anchor">Activation functions, Logistic and Hyperbolic ones </h3>
<h3 id="___sec16" class="anchor">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
@@ -618,7 +595,7 @@ $$
<p>
<!-- !split -->
<h3 id="___sec18" class="anchor">Relevance </h3>
<h3 id="___sec17" class="anchor">Relevance </h3>
The <em>sigmoid</em> function are more biologically plausible because
the output of inactive neurons are zero. Such activation function are called <em>one-sided</em>. However,
@@ -701,6 +678,11 @@ ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec18" class="anchor">Setting up a Multi-layer perceptron model </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
doc/pub/NeuralNet/html/NeuralNet-reveal.html
+42 -66
View File
@@ -148,8 +148,11 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>May 30, 2018</h4></center> <!-- date -->
<center><h4>Sep 28, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- add own code for DNN -->
<p>
<center style="font-size:80%">
@@ -159,55 +162,21 @@ MathJax.Hub.Config({
<section>
<h2 id="___sec0">What is Machine Learning? </h2>
<h2 id="___sec0">Neural networks </h2>
<p>
Machine learning is the science of giving computers the ability to
learn without being explicitly programmed. The idea is that there
exist generic algorithms which can be used to find patterns in a broad
class of data sets without having to write code specifically for each
problem. The algorithm will build its own logic based on the data.
<p>
Machine learning is a subfield of computer science, and is closely
related to computational statistics. It evolved from the study of
pattern recognition in artificial intelligence (AI) research, and has
made contributions to AI tasks like computer vision, natural language
processing and speech recognition. It has also, especially in later
years, found applications in a wide variety of other areas, including
bioinformatics, economy, physics, finance and marketing.
Artificial neural networks are computational systems that can learn to
perform tasks by considering examples, generally without being
programmed with any task-specific rules. It is supposed to mimic a
biological system, wherein neurons interact by sending signals in the
form of mathematical functions between layers. All layers can contain
an arbitrary number of neurons, and each connection is represented by
a weight variable.
</section>
<section>
<h2 id="___sec1">Types of Machine Learning </h2>
<p>
The approaches to machine learning are many, but are often split into two main categories.
In <em>supervised learning</em> we know the answer to a problem,
and let the computer deduce the logic behind it. On the other hand, <em>unsupervised learning</em>
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
Some authours also operate with a third category, namely <em>reinforcement learning</em>. This is a paradigm
of learning inspired by behavioural psychology, where learning is achieved by trial-and-error,
solely from rewards and punishment.
<p>
Another way to categorize machine learning tasks is to consider the desired output of a system.
Some of the most common tasks are:
<ul>
<p><li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
<p><li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<p><li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
</ul>
</section>
<section>
<h2 id="___sec2">Artificial neurons </h2>
<h2 id="___sec1">Artificial neurons </h2>
<p>
The field of artificial neural networks has a long history of
@@ -249,13 +218,13 @@ categories:
</ol>
<p>
In physics, DNNs and CNNs have already found numerous applications. In
In natural science, DNNs and CNNs have already found numerous applications. In
statistical physics, they have been applied to detect phase
transitions in 2D Ising and Potts models, lattice gauge theories, and
different phases of polymers.
different phases of polymers, or solving the Navier-Stokes equation in weather forecasting.
Deep learning has also found interesting applications in quantum
physics. Various quantum phase transitions can be detected and studied
using DNNs and CNNs, including the transverse-field Ising model,
using DNNs and CNNs,
topological phases, and even non-equilibrium many-body
localization. Representing quantum states as DNNs quantum state
tomography are among some of the impressive
@@ -266,14 +235,15 @@ of quantum systems.
In quantum information theory, it has been shown that one can perform
gate decompositions with the help of neural. In lattice quantum chromodynamics,
DNNs have been used to learn action parameters in regions of parameter
space where PCA fails. Last but not least,
DNNs also found place in the study of quantum, and in scattering theory to learn
\( s \)-wave scattering length of potentials.
space where PCA fails.
<p>
The applications are not limited to the natural sciences. There is a plethora of applications in essentially all disciplines, from the humanities to life science and medicine.
</section>
<section>
<h2 id="___sec3">Neural network types </h2>
<h2 id="___sec2">Neural network types </h2>
<p>
An artificial neural network (NN), is a computational model that
@@ -296,7 +266,7 @@ methods.
<section>
<h2 id="___sec4">Feed-forward neural networks </h2>
<h2 id="___sec3">Feed-forward neural networks </h2>
<p>
The feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network,
@@ -327,7 +297,7 @@ which gathers all the local data and produces the outputs. They have wide applic
<section>
<h2 id="___sec5">Recurrent neural networks </h2>
<h2 id="___sec4">Recurrent neural networks </h2>
<p>
So far we have only mentioned NNs where information flows in one direction: forward. <em>Recurrent neural networks</em> on
@@ -340,7 +310,7 @@ sentences, making recurrent NNs especially well-suited for handwriting and speec
<section>
<h2 id="___sec6">Other types of networks </h2>
<h2 id="___sec5">Other types of networks </h2>
<p>
There are many other kinds of NNs that have been developed. One type that is specifically designed for interpolation
@@ -357,7 +327,7 @@ of how a fully-connected FFNN works, and how it can be used to interpolate data
<section>
<h2 id="___sec7">Multilayer perceptrons </h2>
<h2 id="___sec6">Multilayer perceptrons </h2>
<p>
One use often so-called fully-connected feed-forward neural networks with three
@@ -370,7 +340,7 @@ Such networks are often called <em>multilayer perceptrons</em> (MLPs)
<section>
<h2 id="___sec8">Why multilayer perceptrons? </h2>
<h2 id="___sec7">Why multilayer perceptrons? </h2>
<p>
According to the <em>Universal approximation theorem</em>, a feed-forward neural network with just a single hidden layer containing
@@ -391,7 +361,7 @@ functions.
<section>
<h2 id="___sec9">Mathematical model </h2>
<h2 id="___sec8">Mathematical model </h2>
<p>&nbsp;<br>
$$
@@ -409,7 +379,7 @@ which means that each neuron receives a weighted sum of the outputs of <em>all</
<section>
<h2 id="___sec10">Mathematical model </h2>
<h2 id="___sec9">Mathematical model </h2>
<p>
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( u_i^1 \) of the input coordinates \( x_j \),
@@ -452,7 +422,7 @@ the values of the subsequent layer can be calculated and so forth until the outp
<section>
<h2 id="___sec11">Mathematical model </h2>
<h2 id="___sec10">Mathematical model </h2>
<p>
The output of neuron \( i \) in layer 2 is thus,
@@ -485,7 +455,7 @@ $$
<section>
<h2 id="___sec12">Mathematical model </h2>
<h2 id="___sec11">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
@@ -507,7 +477,7 @@ which illustrates a basic property of MLPs: The only independent variables are t
<section>
<h2 id="___sec13">Mathematical model </h2>
<h2 id="___sec12">Mathematical model </h2>
<p>
This confirms that an MLP,
@@ -537,7 +507,7 @@ which is the key to the flexibility of a neural network.
<section>
<h3 id="___sec14">Matrix-vector notation </h3>
<h3 id="___sec13">Matrix-vector notation </h3>
We can introduce a more convenient notation for the activations in a NN.
@@ -578,7 +548,7 @@ $$
<section>
<h3 id="___sec15">Matrix-vector notation and activation </h3>
<h3 id="___sec14">Matrix-vector notation and activation </h3>
The activation of node \( i \) in layer 2 is
@@ -601,7 +571,7 @@ and vector additions that are used as input to the activation functions. For eac
<section>
<h3 id="___sec16">Activation functions </h3>
<h3 id="___sec15">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
@@ -623,7 +593,7 @@ to fulfill the universal approximation theorem
<section>
<h3 id="___sec17">Activation functions, Logistic and Hyperbolic ones </h3>
<h3 id="___sec16">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
@@ -657,7 +627,7 @@ $$
<section>
<h3 id="___sec18">Relevance </h3>
<h3 id="___sec17">Relevance </h3>
The <em>sigmoid</em> function are more biologically plausible because
the output of inactive neurons are zero. Such activation function are called <em>one-sided</em>. However,
@@ -740,6 +710,12 @@ ax.set_title(<span style="color: #CD5555">&#39;Rectified linear unit&#39;</span>
plt.show()
</pre></div>
</section>
<section>
<h2 id="___sec18">Setting up a Multi-layer perceptron model </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
doc/pub/NeuralNet/html/NeuralNet-solarized.html
+57 -75
View File
@@ -34,28 +34,31 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('What is Machine Learning?', 2, None, '___sec0'),
('Types of Machine Learning', 2, None, '___sec1'),
('Artificial neurons', 2, None, '___sec2'),
('Neural network types', 2, None, '___sec3'),
('Feed-forward neural networks', 2, None, '___sec4'),
('Recurrent neural networks', 2, None, '___sec5'),
('Other types of networks', 2, None, '___sec6'),
('Multilayer perceptrons', 2, None, '___sec7'),
('Why multilayer perceptrons?', 2, None, '___sec8'),
'sections': [('Neural networks', 2, None, '___sec0'),
('Artificial neurons', 2, None, '___sec1'),
('Neural network types', 2, None, '___sec2'),
('Feed-forward neural networks', 2, None, '___sec3'),
('Recurrent neural networks', 2, None, '___sec4'),
('Other types of networks', 2, None, '___sec5'),
('Multilayer perceptrons', 2, None, '___sec6'),
('Why multilayer perceptrons?', 2, None, '___sec7'),
('Mathematical model', 2, None, '___sec8'),
('Mathematical model', 2, None, '___sec9'),
('Mathematical model', 2, None, '___sec10'),
('Mathematical model', 2, None, '___sec11'),
('Mathematical model', 2, None, '___sec12'),
('Mathematical model', 2, None, '___sec13'),
('Matrix-vector notation', 3, None, '___sec14'),
('Matrix-vector notation and activation', 3, None, '___sec15'),
('Activation functions', 3, None, '___sec16'),
('Matrix-vector notation', 3, None, '___sec13'),
('Matrix-vector notation and activation', 3, None, '___sec14'),
('Activation functions', 3, None, '___sec15'),
('Activation functions, Logistic and Hyperbolic ones',
3,
None,
'___sec17'),
('Relevance', 3, None, '___sec18'),
'___sec16'),
('Relevance', 3, None, '___sec17'),
('Setting up a Multi-layer perceptron model',
2,
None,
'___sec18'),
('Two-layer Neural Network', 2, None, '___sec19')]}
end of tocinfo -->
@@ -98,56 +101,29 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>May 30, 2018</h4></center> <!-- date -->
<center><h4>Sep 28, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">What is Machine Learning? </h2>
<p>
Machine learning is the science of giving computers the ability to
learn without being explicitly programmed. The idea is that there
exist generic algorithms which can be used to find patterns in a broad
class of data sets without having to write code specifically for each
problem. The algorithm will build its own logic based on the data.
<p>
Machine learning is a subfield of computer science, and is closely
related to computational statistics. It evolved from the study of
pattern recognition in artificial intelligence (AI) research, and has
made contributions to AI tasks like computer vision, natural language
processing and speech recognition. It has also, especially in later
years, found applications in a wide variety of other areas, including
bioinformatics, economy, physics, finance and marketing.
<!-- add own code for DNN -->
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Types of Machine Learning </h2>
<h2 id="___sec0">Neural networks </h2>
<p>
The approaches to machine learning are many, but are often split into two main categories.
In <em>supervised learning</em> we know the answer to a problem,
and let the computer deduce the logic behind it. On the other hand, <em>unsupervised learning</em>
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
Some authours also operate with a third category, namely <em>reinforcement learning</em>. This is a paradigm
of learning inspired by behavioural psychology, where learning is achieved by trial-and-error,
solely from rewards and punishment.
Artificial neural networks are computational systems that can learn to
perform tasks by considering examples, generally without being
programmed with any task-specific rules. It is supposed to mimic a
biological system, wherein neurons interact by sending signals in the
form of mathematical functions between layers. All layers can contain
an arbitrary number of neurons, and each connection is represented by
a weight variable.
<p>
Another way to categorize machine learning tasks is to consider the desired output of a system.
Some of the most common tasks are:
<ul>
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
<li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Artificial neurons </h2>
<h2 id="___sec1">Artificial neurons </h2>
<p>
The field of artificial neural networks has a long history of
@@ -186,13 +162,13 @@ categories:
<li> neural networks for unsupervised learning such as Deep Boltzmann Machines.</li>
</ol>
In physics, DNNs and CNNs have already found numerous applications. In
In natural science, DNNs and CNNs have already found numerous applications. In
statistical physics, they have been applied to detect phase
transitions in 2D Ising and Potts models, lattice gauge theories, and
different phases of polymers.
different phases of polymers, or solving the Navier-Stokes equation in weather forecasting.
Deep learning has also found interesting applications in quantum
physics. Various quantum phase transitions can be detected and studied
using DNNs and CNNs, including the transverse-field Ising model,
using DNNs and CNNs,
topological phases, and even non-equilibrium many-body
localization. Representing quantum states as DNNs quantum state
tomography are among some of the impressive
@@ -203,14 +179,15 @@ of quantum systems.
In quantum information theory, it has been shown that one can perform
gate decompositions with the help of neural. In lattice quantum chromodynamics,
DNNs have been used to learn action parameters in regions of parameter
space where PCA fails. Last but not least,
DNNs also found place in the study of quantum, and in scattering theory to learn
\( s \)-wave scattering length of potentials.
space where PCA fails.
<p>
The applications are not limited to the natural sciences. There is a plethora of applications in essentially all disciplines, from the humanities to life science and medicine.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Neural network types </h2>
<h2 id="___sec2">Neural network types </h2>
<p>
An artificial neural network (NN), is a computational model that
@@ -233,7 +210,7 @@ methods.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Feed-forward neural networks </h2>
<h2 id="___sec3">Feed-forward neural networks </h2>
<p>
The feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network,
@@ -264,7 +241,7 @@ which gathers all the local data and produces the outputs. They have wide applic
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Recurrent neural networks </h2>
<h2 id="___sec4">Recurrent neural networks </h2>
<p>
So far we have only mentioned NNs where information flows in one direction: forward. <em>Recurrent neural networks</em> on
@@ -277,7 +254,7 @@ sentences, making recurrent NNs especially well-suited for handwriting and speec
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Other types of networks </h2>
<h2 id="___sec5">Other types of networks </h2>
<p>
There are many other kinds of NNs that have been developed. One type that is specifically designed for interpolation
@@ -294,7 +271,7 @@ of how a fully-connected FFNN works, and how it can be used to interpolate data
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Multilayer perceptrons </h2>
<h2 id="___sec6">Multilayer perceptrons </h2>
<p>
One use often so-called fully-connected feed-forward neural networks with three
@@ -307,7 +284,7 @@ Such networks are often called <em>multilayer perceptrons</em> (MLPs)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Why multilayer perceptrons? </h2>
<h2 id="___sec7">Why multilayer perceptrons? </h2>
<p>
According to the <em>Universal approximation theorem</em>, a feed-forward neural network with just a single hidden layer containing
@@ -328,7 +305,7 @@ functions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Mathematical model </h2>
<h2 id="___sec8">Mathematical model </h2>
$$
\begin{equation}
@@ -344,7 +321,7 @@ which means that each neuron receives a weighted sum of the outputs of <em>all</
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Mathematical model </h2>
<h2 id="___sec9">Mathematical model </h2>
<p>
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( u_i^1 \) of the input coordinates \( x_j \),
@@ -381,7 +358,7 @@ the values of the subsequent layer can be calculated and so forth until the outp
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">Mathematical model </h2>
<h2 id="___sec10">Mathematical model </h2>
<p>
The output of neuron \( i \) in layer 2 is thus,
@@ -410,7 +387,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Mathematical model </h2>
<h2 id="___sec11">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
@@ -430,7 +407,7 @@ which illustrates a basic property of MLPs: The only independent variables are t
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Mathematical model </h2>
<h2 id="___sec12">Mathematical model </h2>
<p>
This confirms that an MLP,
@@ -457,7 +434,7 @@ which is the key to the flexibility of a neural network.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec14">Matrix-vector notation </h3>
<h3 id="___sec13">Matrix-vector notation </h3>
We can introduce a more convenient notation for the activations in a NN.
@@ -495,7 +472,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec15">Matrix-vector notation and activation </h3>
<h3 id="___sec14">Matrix-vector notation and activation </h3>
The activation of node \( i \) in layer 2 is
@@ -515,7 +492,7 @@ and vector additions that are used as input to the activation functions. For eac
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec16">Activation functions </h3>
<h3 id="___sec15">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
@@ -531,7 +508,7 @@ to fulfill the universal approximation theorem
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec17">Activation functions, Logistic and Hyperbolic ones </h3>
<h3 id="___sec16">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
@@ -560,7 +537,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec18">Relevance </h3>
<h3 id="___sec17">Relevance </h3>
The <em>sigmoid</em> function are more biologically plausible because
the output of inactive neurons are zero. Such activation function are called <em>one-sided</em>. However,
@@ -643,6 +620,11 @@ ax.set_title(<span style="color: #CD5555">&#39;Rectified linear unit&#39;</span>
plt.show()
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec18">Setting up a Multi-layer perceptron model </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
doc/pub/NeuralNet/html/NeuralNet.html
+57 -75
View File
@@ -39,28 +39,31 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('What is Machine Learning?', 2, None, '___sec0'),
('Types of Machine Learning', 2, None, '___sec1'),
('Artificial neurons', 2, None, '___sec2'),
('Neural network types', 2, None, '___sec3'),
('Feed-forward neural networks', 2, None, '___sec4'),
('Recurrent neural networks', 2, None, '___sec5'),
('Other types of networks', 2, None, '___sec6'),
('Multilayer perceptrons', 2, None, '___sec7'),
('Why multilayer perceptrons?', 2, None, '___sec8'),
'sections': [('Neural networks', 2, None, '___sec0'),
('Artificial neurons', 2, None, '___sec1'),
('Neural network types', 2, None, '___sec2'),
('Feed-forward neural networks', 2, None, '___sec3'),
('Recurrent neural networks', 2, None, '___sec4'),
('Other types of networks', 2, None, '___sec5'),
('Multilayer perceptrons', 2, None, '___sec6'),
('Why multilayer perceptrons?', 2, None, '___sec7'),
('Mathematical model', 2, None, '___sec8'),
('Mathematical model', 2, None, '___sec9'),
('Mathematical model', 2, None, '___sec10'),
('Mathematical model', 2, None, '___sec11'),
('Mathematical model', 2, None, '___sec12'),
('Mathematical model', 2, None, '___sec13'),
('Matrix-vector notation', 3, None, '___sec14'),
('Matrix-vector notation and activation', 3, None, '___sec15'),
('Activation functions', 3, None, '___sec16'),
('Matrix-vector notation', 3, None, '___sec13'),
('Matrix-vector notation and activation', 3, None, '___sec14'),
('Activation functions', 3, None, '___sec15'),
('Activation functions, Logistic and Hyperbolic ones',
3,
None,
'___sec17'),
('Relevance', 3, None, '___sec18'),
'___sec16'),
('Relevance', 3, None, '___sec17'),
('Setting up a Multi-layer perceptron model',
2,
None,
'___sec18'),
('Two-layer Neural Network', 2, None, '___sec19')]}
end of tocinfo -->
@@ -103,56 +106,29 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>May 30, 2018</h4></center> <!-- date -->
<center><h4>Sep 28, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">What is Machine Learning? </h2>
<p>
Machine learning is the science of giving computers the ability to
learn without being explicitly programmed. The idea is that there
exist generic algorithms which can be used to find patterns in a broad
class of data sets without having to write code specifically for each
problem. The algorithm will build its own logic based on the data.
<p>
Machine learning is a subfield of computer science, and is closely
related to computational statistics. It evolved from the study of
pattern recognition in artificial intelligence (AI) research, and has
made contributions to AI tasks like computer vision, natural language
processing and speech recognition. It has also, especially in later
years, found applications in a wide variety of other areas, including
bioinformatics, economy, physics, finance and marketing.
<!-- add own code for DNN -->
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Types of Machine Learning </h2>
<h2 id="___sec0">Neural networks </h2>
<p>
The approaches to machine learning are many, but are often split into two main categories.
In <em>supervised learning</em> we know the answer to a problem,
and let the computer deduce the logic behind it. On the other hand, <em>unsupervised learning</em>
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
Some authours also operate with a third category, namely <em>reinforcement learning</em>. This is a paradigm
of learning inspired by behavioural psychology, where learning is achieved by trial-and-error,
solely from rewards and punishment.
Artificial neural networks are computational systems that can learn to
perform tasks by considering examples, generally without being
programmed with any task-specific rules. It is supposed to mimic a
biological system, wherein neurons interact by sending signals in the
form of mathematical functions between layers. All layers can contain
an arbitrary number of neurons, and each connection is represented by
a weight variable.
<p>
Another way to categorize machine learning tasks is to consider the desired output of a system.
Some of the most common tasks are:
<ul>
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
<li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Artificial neurons </h2>
<h2 id="___sec1">Artificial neurons </h2>
<p>
The field of artificial neural networks has a long history of
@@ -191,13 +167,13 @@ categories:
<li> neural networks for unsupervised learning such as Deep Boltzmann Machines.</li>
</ol>
In physics, DNNs and CNNs have already found numerous applications. In
In natural science, DNNs and CNNs have already found numerous applications. In
statistical physics, they have been applied to detect phase
transitions in 2D Ising and Potts models, lattice gauge theories, and
different phases of polymers.
different phases of polymers, or solving the Navier-Stokes equation in weather forecasting.
Deep learning has also found interesting applications in quantum
physics. Various quantum phase transitions can be detected and studied
using DNNs and CNNs, including the transverse-field Ising model,
using DNNs and CNNs,
topological phases, and even non-equilibrium many-body
localization. Representing quantum states as DNNs quantum state
tomography are among some of the impressive
@@ -208,14 +184,15 @@ of quantum systems.
In quantum information theory, it has been shown that one can perform
gate decompositions with the help of neural. In lattice quantum chromodynamics,
DNNs have been used to learn action parameters in regions of parameter
space where PCA fails. Last but not least,
DNNs also found place in the study of quantum, and in scattering theory to learn
\( s \)-wave scattering length of potentials.
space where PCA fails.
<p>
The applications are not limited to the natural sciences. There is a plethora of applications in essentially all disciplines, from the humanities to life science and medicine.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Neural network types </h2>
<h2 id="___sec2">Neural network types </h2>
<p>
An artificial neural network (NN), is a computational model that
@@ -238,7 +215,7 @@ methods.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Feed-forward neural networks </h2>
<h2 id="___sec3">Feed-forward neural networks </h2>
<p>
The feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network,
@@ -269,7 +246,7 @@ which gathers all the local data and produces the outputs. They have wide applic
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Recurrent neural networks </h2>
<h2 id="___sec4">Recurrent neural networks </h2>
<p>
So far we have only mentioned NNs where information flows in one direction: forward. <em>Recurrent neural networks</em> on
@@ -282,7 +259,7 @@ sentences, making recurrent NNs especially well-suited for handwriting and speec
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Other types of networks </h2>
<h2 id="___sec5">Other types of networks </h2>
<p>
There are many other kinds of NNs that have been developed. One type that is specifically designed for interpolation
@@ -299,7 +276,7 @@ of how a fully-connected FFNN works, and how it can be used to interpolate data
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Multilayer perceptrons </h2>
<h2 id="___sec6">Multilayer perceptrons </h2>
<p>
One use often so-called fully-connected feed-forward neural networks with three
@@ -312,7 +289,7 @@ Such networks are often called <em>multilayer perceptrons</em> (MLPs)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Why multilayer perceptrons? </h2>
<h2 id="___sec7">Why multilayer perceptrons? </h2>
<p>
According to the <em>Universal approximation theorem</em>, a feed-forward neural network with just a single hidden layer containing
@@ -333,7 +310,7 @@ functions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Mathematical model </h2>
<h2 id="___sec8">Mathematical model </h2>
$$
\begin{equation}
@@ -349,7 +326,7 @@ which means that each neuron receives a weighted sum of the outputs of <em>all</
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Mathematical model </h2>
<h2 id="___sec9">Mathematical model </h2>
<p>
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( u_i^1 \) of the input coordinates \( x_j \),
@@ -386,7 +363,7 @@ the values of the subsequent layer can be calculated and so forth until the outp
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">Mathematical model </h2>
<h2 id="___sec10">Mathematical model </h2>
<p>
The output of neuron \( i \) in layer 2 is thus,
@@ -415,7 +392,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Mathematical model </h2>
<h2 id="___sec11">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden layers. The complete functional form
@@ -435,7 +412,7 @@ which illustrates a basic property of MLPs: The only independent variables are t
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Mathematical model </h2>
<h2 id="___sec12">Mathematical model </h2>
<p>
This confirms that an MLP,
@@ -462,7 +439,7 @@ which is the key to the flexibility of a neural network.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec14">Matrix-vector notation </h3>
<h3 id="___sec13">Matrix-vector notation </h3>
We can introduce a more convenient notation for the activations in a NN.
@@ -500,7 +477,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec15">Matrix-vector notation and activation </h3>
<h3 id="___sec14">Matrix-vector notation and activation </h3>
The activation of node \( i \) in layer 2 is
@@ -520,7 +497,7 @@ and vector additions that are used as input to the activation functions. For eac
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec16">Activation functions </h3>
<h3 id="___sec15">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its connectivity, is the choice of activation function(s).
@@ -536,7 +513,7 @@ to fulfill the universal approximation theorem
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec17">Activation functions, Logistic and Hyperbolic ones </h3>
<h3 id="___sec16">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
@@ -565,7 +542,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec18">Relevance </h3>
<h3 id="___sec17">Relevance </h3>
The <em>sigmoid</em> function are more biologically plausible because
the output of inactive neurons are zero. Such activation function are called <em>one-sided</em>. However,
@@ -648,6 +625,11 @@ ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec18">Setting up a Multi-layer perceptron model </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
+24 -39
View File
@@ -10,47 +10,24 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **May 30, 2018**\n",
"Date: **Sep 28, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
"\n",
"\n",
"## What is Machine Learning?\n",
"<!-- add own code for DNN -->\n",
"\n",
"Machine learning is the science of giving computers the ability to\n",
"learn without being explicitly programmed. The idea is that there\n",
"exist generic algorithms which can be used to find patterns in a broad\n",
"class of data sets without having to write code specifically for each\n",
"problem. The algorithm will build its own logic based on the data.\n",
"## Neural networks\n",
"\n",
"Machine learning is a subfield of computer science, and is closely\n",
"related to computational statistics. It evolved from the study of\n",
"pattern recognition in artificial intelligence (AI) research, and has\n",
"made contributions to AI tasks like computer vision, natural language\n",
"processing and speech recognition. It has also, especially in later\n",
"years, found applications in a wide variety of other areas, including\n",
"bioinformatics, economy, physics, finance and marketing.\n",
"Artificial neural networks are computational systems that can learn to\n",
"perform tasks by considering examples, generally without being\n",
"programmed with any task-specific rules. It is supposed to mimic a\n",
"biological system, wherein neurons interact by sending signals in the\n",
"form of mathematical functions between layers. All layers can contain\n",
"an arbitrary number of neurons, and each connection is represented by\n",
"a weight variable.\n",
"\n",
"## Types of Machine Learning\n",
"\n",
"\n",
"The approaches to machine learning are many, but are often split into two main categories. \n",
"In *supervised learning* we know the answer to a problem,\n",
"and let the computer deduce the logic behind it. On the other hand, *unsupervised learning*\n",
"is a method for finding patterns and relationship in data sets without any prior knowledge of the system.\n",
"Some authours also operate with a third category, namely *reinforcement learning*. This is a paradigm \n",
"of learning inspired by behavioural psychology, where learning is achieved by trial-and-error, \n",
"solely from rewards and punishment.\n",
"\n",
"Another way to categorize machine learning tasks is to consider the desired output of a system.\n",
"Some of the most common tasks are:\n",
"\n",
" * Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.\n",
"\n",
" * Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.\n",
"\n",
" * Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.\n",
"\n",
"## Artificial neurons\n",
"\n",
@@ -101,13 +78,13 @@
"\n",
"4. neural networks for unsupervised learning such as Deep Boltzmann Machines.\n",
"\n",
"In physics, DNNs and CNNs have already found numerous applications. In\n",
"In natural science, DNNs and CNNs have already found numerous applications. In\n",
"statistical physics, they have been applied to detect phase\n",
"transitions in 2D Ising and Potts models, lattice gauge theories, and\n",
"different phases of polymers.\n",
"different phases of polymers, or solving the Navier-Stokes equation in weather forecasting.\n",
"Deep learning has also found interesting applications in quantum\n",
"physics. Various quantum phase transitions can be detected and studied\n",
"using DNNs and CNNs, including the transverse-field Ising model,\n",
"using DNNs and CNNs,\n",
"topological phases, and even non-equilibrium many-body\n",
"localization. Representing quantum states as DNNs quantum state\n",
"tomography are among some of the impressive\n",
@@ -117,9 +94,9 @@
"In quantum information theory, it has been shown that one can perform\n",
"gate decompositions with the help of neural. In lattice quantum chromodynamics,\n",
"DNNs have been used to learn action parameters in regions of parameter\n",
"space where PCA fails. Last but not least,\n",
"DNNs also found place in the study of quantum, and in scattering theory to learn\n",
"$s$-wave scattering length of potentials.\n",
"space where PCA fails. \n",
"\n",
"The applications are not limited to the natural sciences. There is a plethora of applications in essentially all disciplines, from the humanities to life science and medicine.\n",
"\n",
"## Neural network types\n",
"\n",
@@ -692,6 +669,14 @@
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- !split -->\n",
"## Setting up a Multi-layer perceptron model"
]
},
{
"cell_type": "code",
"execution_count": 2,
doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/NeuralNet/pdf/NeuralNet-beamer-handouts2x3.pdf
BIN
View File
Binary file not shown.
doc/pub/NeuralNet/pdf/NeuralNet-beamer.pdf
BIN
View File
Binary file not shown.
doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf
BIN
View File
Binary file not shown.
doc/pub/Splines/ipynb/Splines.ipynb
+161 -80
View File
File diff suppressed because one or more lines are too long
doc/src/NeuralNet/NeuralNet.do.txt
+19 -47
View File
@@ -5,42 +5,15 @@ DATE: today
# add own code for DNN
!split
===== What is Machine Learning? =====
===== Neural networks =====
Machine learning is the science of giving computers the ability to
learn without being explicitly programmed. The idea is that there
exist generic algorithms which can be used to find patterns in a broad
class of data sets without having to write code specifically for each
problem. The algorithm will build its own logic based on the data.
Machine learning is a subfield of computer science, and is closely
related to computational statistics. It evolved from the study of
pattern recognition in artificial intelligence (AI) research, and has
made contributions to AI tasks like computer vision, natural language
processing and speech recognition. It has also, especially in later
years, found applications in a wide variety of other areas, including
bioinformatics, economy, physics, finance and marketing.
!split
===== Types of Machine Learning =====
The approaches to machine learning are many, but are often split into two main categories.
In *supervised learning* we know the answer to a problem,
and let the computer deduce the logic behind it. On the other hand, *unsupervised learning*
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
Some authours also operate with a third category, namely *reinforcement learning*. This is a paradigm
of learning inspired by behavioural psychology, where learning is achieved by trial-and-error,
solely from rewards and punishment.
Another way to categorize machine learning tasks is to consider the desired output of a system.
Some of the most common tasks are:
* Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.
* Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.
* Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.
Artificial neural networks are computational systems that can learn to
perform tasks by considering examples, generally without being
programmed with any task-specific rules. It is supposed to mimic a
biological system, wherein neurons interact by sending signals in the
form of mathematical functions between layers. All layers can contain
an arbitrary number of neurons, and each connection is represented by
a weight variable.
!split
@@ -77,13 +50,13 @@ o neural networks for sequential data such as Recurrent Neural Networks (RNNs),
o neural networks for unsupervised learning such as Deep Boltzmann Machines.
In physics, DNNs and CNNs have already found numerous applications. In
In natural science, DNNs and CNNs have already found numerous applications. In
statistical physics, they have been applied to detect phase
transitions in 2D Ising and Potts models, lattice gauge theories, and
different phases of polymers.
different phases of polymers, or solving the Navier-Stokes equation in weather forecasting.
Deep learning has also found interesting applications in quantum
physics. Various quantum phase transitions can be detected and studied
using DNNs and CNNs, including the transverse-field Ising model,
using DNNs and CNNs,
topological phases, and even non-equilibrium many-body
localization. Representing quantum states as DNNs quantum state
tomography are among some of the impressive
@@ -93,9 +66,9 @@ of quantum systems.
In quantum information theory, it has been shown that one can perform
gate decompositions with the help of neural. In lattice quantum chromodynamics,
DNNs have been used to learn action parameters in regions of parameter
space where PCA fails. Last but not least,
DNNs also found place in the study of quantum, and in scattering theory to learn
$s$-wave scattering length of potentials.
space where PCA fails.
The applications are not limited to the natural sciences. There is a plethora of applications in essentially all disciplines, from the humanities to life science and medicine.
!split
===== Neural network types =====
@@ -471,6 +444,11 @@ plt.show()
!ec
!split
===== Setting up a Multi-layer perceptron model =====
!bc pycod
from scipy import optimize
@@ -828,9 +806,3 @@ net.SGD(training_data,30,10,3,test_data=test_data)
!ec
!split
===== Using Tensorflow =====
!split
===== Adding deep learning =====
Convulotional neural networks as well
doc/src/NeuralNet/diffeq.tex
-1047
View File
File diff suppressed because it is too large Load Diff