lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

small update on neural networks

Browse Source
This commit is contained in:
mhjensen
2018-10-16 13:54:54 +02:00
parent ce0c90eccd
commit 9bdb28895e
75 changed files with 6318 additions and 5703 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/NeuralNet/html/._NeuralNet-bs053.html
+103 -74
View File
@@ -92,54 +92,55 @@ Automatically generated HTML file from DocOnce source
'___sec30'),
('Defining the cost function', 2, None, '___sec31'),
('Example: binary classification problem', 2, None, '___sec32'),
('The Softmax function', 2, None, '___sec33'),
('Developing a code for doing neural networks with back '
'propagation',
2,
None,
'___sec33'),
('Collect and pre-process data', 2, None, '___sec34'),
('Train and test datasets', 2, None, '___sec35'),
('Define model and architecture', 2, None, '___sec36'),
('Layers', 2, None, '___sec37'),
('Weights and biases', 2, None, '___sec38'),
('Feed-forward pass', 2, None, '___sec39'),
('Matrix multiplications', 2, None, '___sec40'),
('Choose cost function and optimizer', 2, None, '___sec41'),
('Optimizing the cost function', 2, None, '___sec42'),
('Regularization', 2, None, '___sec43'),
('Matrix multiplication', 2, None, '___sec44'),
('Improving performance', 2, None, '___sec45'),
('Full object-oriented implementation', 2, None, '___sec46'),
('Evaluate model performance on test data', 2, None, '___sec47'),
('Adjust hyperparameters', 2, None, '___sec48'),
('Visualization', 2, None, '___sec49'),
('scikit-learn implementation', 2, None, '___sec50'),
('Visualization', 2, None, '___sec51'),
'___sec34'),
('Collect and pre-process data', 2, None, '___sec35'),
('Train and test datasets', 2, None, '___sec36'),
('Define model and architecture', 2, None, '___sec37'),
('Layers', 2, None, '___sec38'),
('Weights and biases', 2, None, '___sec39'),
('Feed-forward pass', 2, None, '___sec40'),
('Matrix multiplications', 2, None, '___sec41'),
('Choose cost function and optimizer', 2, None, '___sec42'),
('Optimizing the cost function', 2, None, '___sec43'),
('Regularization', 2, None, '___sec44'),
('Matrix multiplication', 2, None, '___sec45'),
('Improving performance', 2, None, '___sec46'),
('Full object-oriented implementation', 2, None, '___sec47'),
('Evaluate model performance on test data', 2, None, '___sec48'),
('Adjust hyperparameters', 2, None, '___sec49'),
('Visualization', 2, None, '___sec50'),
('scikit-learn implementation', 2, None, '___sec51'),
('Visualization', 2, None, '___sec52'),
('Building neural networks in Tensorflow and Keras',
2,
None,
'___sec52'),
('Tensorflow', 2, None, '___sec53'),
('Collect and pre-process data', 2, None, '___sec54'),
('Using TensorFlow backend', 2, None, '___sec55'),
('Optimizing and using gradient descent', 2, None, '___sec56'),
('Using Keras', 2, None, '___sec57'),
('Which activation function should I use?', 2, None, '___sec58'),
'___sec53'),
('Tensorflow', 2, None, '___sec54'),
('Collect and pre-process data', 2, None, '___sec55'),
('Using TensorFlow backend', 2, None, '___sec56'),
('Optimizing and using gradient descent', 2, None, '___sec57'),
('Using Keras', 2, None, '___sec58'),
('Which activation function should I use?', 2, None, '___sec59'),
('Is the Logistic activation function (Sigmoid) our choice?',
2,
None,
'___sec59'),
('The derivative of the Logistic funtion', 2, None, '___sec60'),
('The RELU function family', 2, None, '___sec61'),
('Which activation function should we use?', 2, None, '___sec62'),
'___sec60'),
('The derivative of the Logistic funtion', 2, None, '___sec61'),
('The RELU function family', 2, None, '___sec62'),
('Which activation function should we use?', 2, None, '___sec63'),
('A top-down perspective on Neural networks',
2,
None,
'___sec63'),
'___sec64'),
('Limitations of supervised learning with deep networks',
2,
None,
'___sec64')]}
'___sec65')]}
end of tocinfo -->
<body>
@@ -210,38 +211,39 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._NeuralNet-bs031.html#___sec30" style="font-size: 80%;"><b>Setting up a Multi-layer perceptron model for classification</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs032.html#___sec31" style="font-size: 80%;"><b>Defining the cost function</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs033.html#___sec32" style="font-size: 80%;"><b>Example: binary classification problem</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs034.html#___sec33" style="font-size: 80%;"><b>Developing a code for doing neural networks with back propagation</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs035.html#___sec34" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs036.html#___sec35" style="font-size: 80%;"><b>Train and test datasets</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs037.html#___sec36" style="font-size: 80%;"><b>Define model and architecture</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs038.html#___sec37" style="font-size: 80%;"><b>Layers</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs039.html#___sec38" style="font-size: 80%;"><b>Weights and biases</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs040.html#___sec39" style="font-size: 80%;"><b>Feed-forward pass</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs041.html#___sec40" style="font-size: 80%;"><b>Matrix multiplications</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs042.html#___sec41" style="font-size: 80%;"><b>Choose cost function and optimizer</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs043.html#___sec42" style="font-size: 80%;"><b>Optimizing the cost function</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs044.html#___sec43" style="font-size: 80%;"><b>Regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs045.html#___sec44" style="font-size: 80%;"><b>Matrix multiplication</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs046.html#___sec45" style="font-size: 80%;"><b>Improving performance</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs047.html#___sec46" style="font-size: 80%;"><b>Full object-oriented implementation</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs048.html#___sec47" style="font-size: 80%;"><b>Evaluate model performance on test data</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs049.html#___sec48" style="font-size: 80%;"><b>Adjust hyperparameters</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs050.html#___sec49" style="font-size: 80%;"><b>Visualization</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs051.html#___sec50" style="font-size: 80%;"><b>scikit-learn implementation</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs052.html#___sec51" style="font-size: 80%;"><b>Visualization</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec52" style="font-size: 80%;"><b>Building neural networks in Tensorflow and Keras</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs054.html#___sec53" style="font-size: 80%;"><b>Tensorflow</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs055.html#___sec54" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs056.html#___sec55" style="font-size: 80%;"><b>Using TensorFlow backend</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs057.html#___sec56" style="font-size: 80%;"><b>Optimizing and using gradient descent</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs058.html#___sec57" style="font-size: 80%;"><b>Using Keras</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs059.html#___sec58" style="font-size: 80%;"><b>Which activation function should I use?</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs060.html#___sec59" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs061.html#___sec60" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs062.html#___sec61" style="font-size: 80%;"><b>The RELU function family</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs063.html#___sec62" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs064.html#___sec63" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs065.html#___sec64" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs034.html#___sec33" style="font-size: 80%;"><b>The Softmax function</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs035.html#___sec34" style="font-size: 80%;"><b>Developing a code for doing neural networks with back propagation</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs036.html#___sec35" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs037.html#___sec36" style="font-size: 80%;"><b>Train and test datasets</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs038.html#___sec37" style="font-size: 80%;"><b>Define model and architecture</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs039.html#___sec38" style="font-size: 80%;"><b>Layers</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs040.html#___sec39" style="font-size: 80%;"><b>Weights and biases</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs041.html#___sec40" style="font-size: 80%;"><b>Feed-forward pass</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs042.html#___sec41" style="font-size: 80%;"><b>Matrix multiplications</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs043.html#___sec42" style="font-size: 80%;"><b>Choose cost function and optimizer</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs044.html#___sec43" style="font-size: 80%;"><b>Optimizing the cost function</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs045.html#___sec44" style="font-size: 80%;"><b>Regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs046.html#___sec45" style="font-size: 80%;"><b>Matrix multiplication</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs047.html#___sec46" style="font-size: 80%;"><b>Improving performance</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs048.html#___sec47" style="font-size: 80%;"><b>Full object-oriented implementation</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs049.html#___sec48" style="font-size: 80%;"><b>Evaluate model performance on test data</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs050.html#___sec49" style="font-size: 80%;"><b>Adjust hyperparameters</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs051.html#___sec50" style="font-size: 80%;"><b>Visualization</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs052.html#___sec51" style="font-size: 80%;"><b>scikit-learn implementation</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec52" style="font-size: 80%;"><b>Visualization</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs054.html#___sec53" style="font-size: 80%;"><b>Building neural networks in Tensorflow and Keras</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs055.html#___sec54" style="font-size: 80%;"><b>Tensorflow</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs056.html#___sec55" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs057.html#___sec56" style="font-size: 80%;"><b>Using TensorFlow backend</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs058.html#___sec57" style="font-size: 80%;"><b>Optimizing and using gradient descent</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs059.html#___sec58" style="font-size: 80%;"><b>Using Keras</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs060.html#___sec59" style="font-size: 80%;"><b>Which activation function should I use?</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs061.html#___sec60" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs062.html#___sec61" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs063.html#___sec62" style="font-size: 80%;"><b>The RELU function family</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs064.html#___sec63" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs065.html#___sec64" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs066.html#___sec65" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
</ul>
</li>
@@ -257,18 +259,45 @@ MathJax.Hub.Config({
<a name="part0053"></a>
<!-- !split -->
<h2 id="___sec52" class="anchor">Building neural networks in Tensorflow and Keras </h2>
<h2 id="___sec52" class="anchor">Visualization </h2>
<p>
Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn
and use it to construct a neural network in Tensorflow. Once we have constructed a neural network in NumPy
and Tensorflow, building one in Keras is really quite trivial, though the performance may suffer.
<p>
In our previous example we used only one hidden layer, and in this we will use two. From this it should be quite
clear how to build one using an arbitrary number of hidden layers, using data structures such as Python lists or
NumPy arrays.
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># optional</span>
<span style="color: #408080; font-style: italic"># visual representation of grid search</span>
<span style="color: #408080; font-style: italic"># uses seaborn heatmap, could probably do this in matplotlib</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">seaborn</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">sns</span>
sns<span style="color: #666666">.</span>set()
train_accuracy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)))
test_accuracy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)))
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">len</span>(eta_vals)):
<span style="color: #008000; font-weight: bold">for</span> j <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">len</span>(lmbd_vals)):
dnn <span style="color: #666666">=</span> DNN_scikit[i][j]
train_pred <span style="color: #666666">=</span> dnn<span style="color: #666666">.</span>predict(X_train)
test_pred <span style="color: #666666">=</span> dnn<span style="color: #666666">.</span>predict(X_test)
train_accuracy[i][j] <span style="color: #666666">=</span> accuracy_score(Y_train, train_pred)
test_accuracy[i][j] <span style="color: #666666">=</span> accuracy_score(Y_test, test_pred)
fig, ax <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>subplots(figsize <span style="color: #666666">=</span> (<span style="color: #666666">10</span>, <span style="color: #666666">10</span>))
sns<span style="color: #666666">.</span>heatmap(train_accuracy, annot<span style="color: #666666">=</span><span style="color: #008000">True</span>, ax<span style="color: #666666">=</span>ax, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&quot;viridis&quot;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&quot;Training Accuracy&quot;</span>)
ax<span style="color: #666666">.</span>set_ylabel(<span style="color: #BA2121">&quot;$\eta$&quot;</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&quot;$\lambda$&quot;</span>)
plt<span style="color: #666666">.</span>show()
fig, ax <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>subplots(figsize <span style="color: #666666">=</span> (<span style="color: #666666">10</span>, <span style="color: #666666">10</span>))
sns<span style="color: #666666">.</span>heatmap(test_accuracy, annot<span style="color: #666666">=</span><span style="color: #008000">True</span>, ax<span style="color: #666666">=</span>ax, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&quot;viridis&quot;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&quot;Test Accuracy&quot;</span>)
ax<span style="color: #666666">.</span>set_ylabel(<span style="color: #BA2121">&quot;$\eta$&quot;</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&quot;$\lambda$&quot;</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
@@ -295,7 +324,7 @@ NumPy arrays.
<li><a href="._NeuralNet-bs061.html">62</a></li>
<li><a href="._NeuralNet-bs062.html">63</a></li>
<li><a href="">...</a></li>
<li><a href="._NeuralNet-bs065.html">66</a></li>
<li><a href="._NeuralNet-bs066.html">67</a></li>
<li><a href="._NeuralNet-bs054.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->