lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-10-13 22:28:17 +02:00
parent a462464a08
commit d70ccf7b9a
11 changed files with 1332 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week41/html/week41-bs.html
+46 -41
View File
@@ -128,6 +128,10 @@ Automatically generated HTML file from DocOnce source
2,
None,
'the-full-network-for-the-various-gates'),
('And the same using Scikit-Learn',
2,
None,
'and-the-same-using-scikit-learn'),
('Building neural networks in Tensorflow and Keras',
2,
None,
@@ -273,7 +277,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week41-bs006.html#example-binary-classification-problem" style="font-size: 80%;">Example: binary classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs007.html#the-softmax-function" style="font-size: 80%;">The Softmax function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs008.html#developing-a-code-for-doing-neural-networks-with-back-propagation" style="font-size: 80%;">Developing a code for doing neural networks with back propagation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs036.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs037.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs010.html#train-and-test-datasets" style="font-size: 80%;">Train and test datasets</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs011.html#define-model-and-architecture" style="font-size: 80%;">Define model and architecture</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs012.html#layers" style="font-size: 80%;">Layers</a></li>
@@ -297,45 +301,46 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week41-bs030.html#setting-up-the-neural-network" style="font-size: 80%;">Setting up the Neural Network</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs031.html#then-the-first-feed-forward-pass" style="font-size: 80%;">Then the first Feed Forward pass</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs032.html#the-full-network-for-the-various-gates" style="font-size: 80%;">The full Network for the Various Gates</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs033.html#building-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">Building neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs034.html#tensorflow" style="font-size: 80%;">Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs035.html#using-keras" style="font-size: 80%;">Using Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs036.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs037.html#the-breast-cancer-data-now-with-keras" style="font-size: 80%;">The Breast Cancer Data, now with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs038.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;">Fine-tuning neural network hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs039.html#hidden-layers" style="font-size: 80%;">Hidden layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs040.html#which-activation-function-should-i-use" style="font-size: 80%;">Which activation function should I use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs041.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;">Is the Logistic activation function (Sigmoid) our choice?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs042.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;">The derivative of the Logistic funtion</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs043.html#the-relu-function-family" style="font-size: 80%;">The RELU function family</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs044.html#which-activation-function-should-we-use" style="font-size: 80%;">Which activation function should we use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs045.html#more-on-activation-functions-output-layers" style="font-size: 80%;">More on activation functions, output layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs046.html#batch-normalization" style="font-size: 80%;">Batch Normalization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs047.html#dropout" style="font-size: 80%;">Dropout</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs048.html#gradient-clipping" style="font-size: 80%;">Gradient Clipping</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs049.html#a-very-nice-website-on-neural-networks" style="font-size: 80%;">A very nice website on Neural Networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs050.html#a-top-down-perspective-on-neural-networks" style="font-size: 80%;">A top-down perspective on Neural networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs051.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;">Limitations of supervised learning with deep networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs052.html#overarching-views-a-personal-note" style="font-size: 80%;">Overarching Views, a personal note</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs053.html#from-a-spherical-cow-to-a-real-one" style="font-size: 80%;">From a Spherical Cow to a real one</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs054.html#convolutional-neural-networks-recognizing-images" style="font-size: 80%;">Convolutional Neural Networks (recognizing images)</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs055.html#regular-nns-don-t-scale-well-to-full-images" style="font-size: 80%;">Regular NNs don’t scale well to full images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs056.html#3d-volumes-of-neurons" style="font-size: 80%;">3D volumes of neurons</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs057.html#layers-used-to-build-cnns" style="font-size: 80%;">Layers used to build CNNs</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs058.html#transforming-images" style="font-size: 80%;">Transforming images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs059.html#cnns-in-brief" style="font-size: 80%;">CNNs in brief</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs060.html#cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs061.html#setting-it-up" style="font-size: 80%;">Setting it up</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs062.html#the-mnist-dataset-again" style="font-size: 80%;">The MNIST dataset again</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs063.html#strong-correlations" style="font-size: 80%;">Strong correlations</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs064.html#layers-of-a-cnn" style="font-size: 80%;">Layers of a CNN</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs065.html#systematic-reduction" style="font-size: 80%;">Systematic reduction</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs066.html#prerequisites-collect-and-pre-process-data" style="font-size: 80%;">Prerequisites: Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs067.html#importing-keras-and-tensorflow" style="font-size: 80%;">Importing Keras and Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs068.html#running-with-keras" style="font-size: 80%;">Running with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs069.html#final-part" style="font-size: 80%;">Final part</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs070.html#final-visualization" style="font-size: 80%;">Final visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs071.html#fun-links" style="font-size: 80%;">Fun links</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs033.html#and-the-same-using-scikit-learn" style="font-size: 80%;">And the same using Scikit-Learn</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs034.html#building-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">Building neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs035.html#tensorflow" style="font-size: 80%;">Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs036.html#using-keras" style="font-size: 80%;">Using Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs037.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs038.html#the-breast-cancer-data-now-with-keras" style="font-size: 80%;">The Breast Cancer Data, now with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs039.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;">Fine-tuning neural network hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs040.html#hidden-layers" style="font-size: 80%;">Hidden layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs041.html#which-activation-function-should-i-use" style="font-size: 80%;">Which activation function should I use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs042.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;">Is the Logistic activation function (Sigmoid) our choice?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs043.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;">The derivative of the Logistic funtion</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs044.html#the-relu-function-family" style="font-size: 80%;">The RELU function family</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs045.html#which-activation-function-should-we-use" style="font-size: 80%;">Which activation function should we use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs046.html#more-on-activation-functions-output-layers" style="font-size: 80%;">More on activation functions, output layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs047.html#batch-normalization" style="font-size: 80%;">Batch Normalization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs048.html#dropout" style="font-size: 80%;">Dropout</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs049.html#gradient-clipping" style="font-size: 80%;">Gradient Clipping</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs050.html#a-very-nice-website-on-neural-networks" style="font-size: 80%;">A very nice website on Neural Networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs051.html#a-top-down-perspective-on-neural-networks" style="font-size: 80%;">A top-down perspective on Neural networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs052.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;">Limitations of supervised learning with deep networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs053.html#overarching-views-a-personal-note" style="font-size: 80%;">Overarching Views, a personal note</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs054.html#from-a-spherical-cow-to-a-real-one" style="font-size: 80%;">From a Spherical Cow to a real one</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs055.html#convolutional-neural-networks-recognizing-images" style="font-size: 80%;">Convolutional Neural Networks (recognizing images)</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs056.html#regular-nns-don-t-scale-well-to-full-images" style="font-size: 80%;">Regular NNs don’t scale well to full images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs057.html#3d-volumes-of-neurons" style="font-size: 80%;">3D volumes of neurons</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs058.html#layers-used-to-build-cnns" style="font-size: 80%;">Layers used to build CNNs</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs059.html#transforming-images" style="font-size: 80%;">Transforming images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs060.html#cnns-in-brief" style="font-size: 80%;">CNNs in brief</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs061.html#cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs062.html#setting-it-up" style="font-size: 80%;">Setting it up</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs063.html#the-mnist-dataset-again" style="font-size: 80%;">The MNIST dataset again</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs064.html#strong-correlations" style="font-size: 80%;">Strong correlations</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs065.html#layers-of-a-cnn" style="font-size: 80%;">Layers of a CNN</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs066.html#systematic-reduction" style="font-size: 80%;">Systematic reduction</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs067.html#prerequisites-collect-and-pre-process-data" style="font-size: 80%;">Prerequisites: Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs068.html#importing-keras-and-tensorflow" style="font-size: 80%;">Importing Keras and Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs069.html#running-with-keras" style="font-size: 80%;">Running with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs070.html#final-part" style="font-size: 80%;">Final part</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs071.html#final-visualization" style="font-size: 80%;">Final visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs072.html#fun-links" style="font-size: 80%;">Fun links</a></li>
</ul>
</li>
@@ -394,7 +399,7 @@ MathJax.Hub.Config({
<li><a href="._week41-bs008.html">9</a></li>
<li><a href="._week41-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week41-bs071.html">72</a></li>
<li><a href="._week41-bs072.html">73</a></li>
<li><a href="._week41-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week41/html/week41-reveal.html
+70 -3
View File
@@ -1551,7 +1551,7 @@ while the vector of outputs is \( \boldsymbol{y}^T=[0,1,1,0] \) for the XOR gate
<h2 id="setting-up-the-neural-network">Setting up the Neural Network </h2>
<p>
We define first our design matrix and the various input vectors.
We define first our design matrix and the various output vectors for the different gates.
<p>
@@ -1613,7 +1613,7 @@ np.random.seed(<span style="color: #B452CD">0</span>)
<span style="color: #228B22"># Defining the neural network</span>
n_inputs, n_features = X.shape
n_hidden_neurons = <span style="color: #B452CD">2</span>
n_categories = <span style="color: #B452CD">1</span>
n_categories = <span style="color: #B452CD">2</span>
n_features = <span style="color: #B452CD">2</span>
<span style="color: #228B22"># we make the weights normally distributed using numpy.random.randn</span>
@@ -1634,7 +1634,7 @@ predictions = predict(X)
<span style="color: #658b00">print</span>(predictions)
</pre></div>
<p>
Not an impressive result. Let us now add the full network with the back-propagation algorithm discussed above.
Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above.
</section>
@@ -1648,6 +1648,73 @@ Not an impressive result. Let us now add the full network with the back-propagat
</section>
<section>
<h2 id="and-the-same-using-scikit-learn">And the same using Scikit-Learn </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># import necessary packages</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.neural_network</span> <span style="color: #8B008B; font-weight: bold">import</span> MLPClassifier
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.metrics</span> <span style="color: #8B008B; font-weight: bold">import</span> accuracy_score
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">seaborn</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sns</span>
<span style="color: #228B22"># ensure the same random numbers appear every time</span>
np.random.seed(<span style="color: #B452CD">0</span>)
<span style="color: #228B22"># Design matrix</span>
X = np.array([ [<span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span>], [<span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>],[<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>]],dtype=np.float64)
<span style="color: #228B22"># The XOR gate</span>
yXOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>])
<span style="color: #228B22"># The OR gate</span>
yOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>])
<span style="color: #228B22"># The AND gate</span>
yAND = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span> ,<span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>])
<span style="color: #228B22"># Defining the neural network</span>
n_inputs, n_features = X.shape
n_hidden_neurons = <span style="color: #B452CD">2</span>
n_categories = <span style="color: #B452CD">2</span>
n_features = <span style="color: #B452CD">2</span>
eta_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">7</span>)
lmbd_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">7</span>)
<span style="color: #228B22"># store models for later use</span>
DNN_scikit = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
epochs = <span style="color: #B452CD">100</span>
<span style="color: #8B008B; font-weight: bold">for</span> i, eta <span style="color: #8B008B">in</span> <span style="color: #658b00">enumerate</span>(eta_vals):
<span style="color: #8B008B; font-weight: bold">for</span> j, lmbd <span style="color: #8B008B">in</span> <span style="color: #658b00">enumerate</span>(lmbd_vals):
dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation=<span style="color: #CD5555">&#39;logistic&#39;</span>,
alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
dnn.fit(X, yXOR)
DNN_scikit[i][j] = dnn
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Learning rate = &quot;</span>, eta)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Lambda = &quot;</span>, lmbd)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Accuracy score on data set: &quot;</span>, dnn.score(X, yXOR))
<span style="color: #658b00">print</span>()
sns.set()
test_accuracy = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)))
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #658b00">len</span>(eta_vals)):
<span style="color: #8B008B; font-weight: bold">for</span> j <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #658b00">len</span>(lmbd_vals)):
dnn = DNN_scikit[i][j]
test_pred = dnn.predict(X)
test_accuracy[i][j] = accuracy_score(yXOR, test_pred)
fig, ax = plt.subplots(figsize = (<span style="color: #B452CD">10</span>, <span style="color: #B452CD">10</span>))
sns.heatmap(test_accuracy, annot=<span style="color: #8B008B; font-weight: bold">True</span>, ax=ax, cmap=<span style="color: #CD5555">&quot;viridis&quot;</span>)
ax.set_title(<span style="color: #CD5555">&quot;Test Accuracy&quot;</span>)
ax.set_ylabel(<span style="color: #CD5555">&quot;$\eta$&quot;</span>)
ax.set_xlabel(<span style="color: #CD5555">&quot;$\lambda$&quot;</span>)
plt.show()
</pre></div>
</section>
<section>
<h2 id="building-neural-networks-in-tensorflow-and-keras">Building neural networks in Tensorflow and Keras </h2>
doc/pub/week41/html/week41-solarized.html
+73 -3
View File
@@ -148,6 +148,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'the-full-network-for-the-various-gates'),
('And the same using Scikit-Learn',
2,
None,
'and-the-same-using-scikit-learn'),
('Building neural networks in Tensorflow and Keras',
2,
None,
@@ -1564,7 +1568,7 @@ while the vector of outputs is \( \boldsymbol{y}^T=[0,1,1,0] \) for the XOR gate
<h2 id="setting-up-the-neural-network">Setting up the Neural Network </h2>
<p>
We define first our design matrix and the various input vectors.
We define first our design matrix and the various output vectors for the different gates.
<p>
@@ -1625,7 +1629,7 @@ np.random.seed(<span style="color: #B452CD">0</span>)
<span style="color: #228B22"># Defining the neural network</span>
n_inputs, n_features = X.shape
n_hidden_neurons = <span style="color: #B452CD">2</span>
n_categories = <span style="color: #B452CD">1</span>
n_categories = <span style="color: #B452CD">2</span>
n_features = <span style="color: #B452CD">2</span>
<span style="color: #228B22"># we make the weights normally distributed using numpy.random.randn</span>
@@ -1646,7 +1650,7 @@ predictions = predict(X)
<span style="color: #658b00">print</span>(predictions)
</pre></div>
<p>
Not an impressive result. Let us now add the full network with the back-propagation algorithm discussed above.
Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1660,6 +1664,72 @@ Not an impressive result. Let us now add the full network with the back-propagat
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="and-the-same-using-scikit-learn">And the same using Scikit-Learn </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># import necessary packages</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.neural_network</span> <span style="color: #8B008B; font-weight: bold">import</span> MLPClassifier
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.metrics</span> <span style="color: #8B008B; font-weight: bold">import</span> accuracy_score
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">seaborn</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sns</span>
<span style="color: #228B22"># ensure the same random numbers appear every time</span>
np.random.seed(<span style="color: #B452CD">0</span>)
<span style="color: #228B22"># Design matrix</span>
X = np.array([ [<span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span>], [<span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>],[<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>]],dtype=np.float64)
<span style="color: #228B22"># The XOR gate</span>
yXOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>])
<span style="color: #228B22"># The OR gate</span>
yOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>])
<span style="color: #228B22"># The AND gate</span>
yAND = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span> ,<span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>])
<span style="color: #228B22"># Defining the neural network</span>
n_inputs, n_features = X.shape
n_hidden_neurons = <span style="color: #B452CD">2</span>
n_categories = <span style="color: #B452CD">2</span>
n_features = <span style="color: #B452CD">2</span>
eta_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">7</span>)
lmbd_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">7</span>)
<span style="color: #228B22"># store models for later use</span>
DNN_scikit = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
epochs = <span style="color: #B452CD">100</span>
<span style="color: #8B008B; font-weight: bold">for</span> i, eta <span style="color: #8B008B">in</span> <span style="color: #658b00">enumerate</span>(eta_vals):
<span style="color: #8B008B; font-weight: bold">for</span> j, lmbd <span style="color: #8B008B">in</span> <span style="color: #658b00">enumerate</span>(lmbd_vals):
dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation=<span style="color: #CD5555">&#39;logistic&#39;</span>,
alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
dnn.fit(X, yXOR)
DNN_scikit[i][j] = dnn
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Learning rate = &quot;</span>, eta)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Lambda = &quot;</span>, lmbd)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Accuracy score on data set: &quot;</span>, dnn.score(X, yXOR))
<span style="color: #658b00">print</span>()
sns.set()
test_accuracy = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)))
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #658b00">len</span>(eta_vals)):
<span style="color: #8B008B; font-weight: bold">for</span> j <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #658b00">len</span>(lmbd_vals)):
dnn = DNN_scikit[i][j]
test_pred = dnn.predict(X)
test_accuracy[i][j] = accuracy_score(yXOR, test_pred)
fig, ax = plt.subplots(figsize = (<span style="color: #B452CD">10</span>, <span style="color: #B452CD">10</span>))
sns.heatmap(test_accuracy, annot=<span style="color: #8B008B; font-weight: bold">True</span>, ax=ax, cmap=<span style="color: #CD5555">&quot;viridis&quot;</span>)
ax.set_title(<span style="color: #CD5555">&quot;Test Accuracy&quot;</span>)
ax.set_ylabel(<span style="color: #CD5555">&quot;$\eta$&quot;</span>)
ax.set_xlabel(<span style="color: #CD5555">&quot;$\lambda$&quot;</span>)
plt.show()
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="building-neural-networks-in-tensorflow-and-keras">Building neural networks in Tensorflow and Keras </h2>
<p>
doc/pub/week41/html/week41.html
+73 -3
View File
@@ -153,6 +153,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'the-full-network-for-the-various-gates'),
('And the same using Scikit-Learn',
2,
None,
'and-the-same-using-scikit-learn'),
('Building neural networks in Tensorflow and Keras',
2,
None,
@@ -1569,7 +1573,7 @@ while the vector of outputs is \( \boldsymbol{y}^T=[0,1,1,0] \) for the XOR gate
<h2 id="setting-up-the-neural-network">Setting up the Neural Network </h2>
<p>
We define first our design matrix and the various input vectors.
We define first our design matrix and the various output vectors for the different gates.
<p>
@@ -1630,7 +1634,7 @@ np<span style="color: #666666">.</span>random<span style="color: #666666">.</spa
<span style="color: #408080; font-style: italic"># Defining the neural network</span>
n_inputs, n_features <span style="color: #666666">=</span> X<span style="color: #666666">.</span>shape
n_hidden_neurons <span style="color: #666666">=</span> <span style="color: #666666">2</span>
n_categories <span style="color: #666666">=</span> <span style="color: #666666">1</span>
n_categories <span style="color: #666666">=</span> <span style="color: #666666">2</span>
n_features <span style="color: #666666">=</span> <span style="color: #666666">2</span>
<span style="color: #408080; font-style: italic"># we make the weights normally distributed using numpy.random.randn</span>
@@ -1651,7 +1655,7 @@ predictions <span style="color: #666666">=</span> predict(X)
<span style="color: #008000">print</span>(predictions)
</pre></div>
<p>
Not an impressive result. Let us now add the full network with the back-propagation algorithm discussed above.
Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1665,6 +1669,72 @@ Not an impressive result. Let us now add the full network with the back-propagat
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="and-the-same-using-scikit-learn">And the same using Scikit-Learn </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># import necessary packages</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.neural_network</span> <span style="color: #008000; font-weight: bold">import</span> MLPClassifier
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> accuracy_score
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">seaborn</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">sns</span>
<span style="color: #408080; font-style: italic"># ensure the same random numbers appear every time</span>
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">0</span>)
<span style="color: #408080; font-style: italic"># Design matrix</span>
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([ [<span style="color: #666666">0</span>, <span style="color: #666666">0</span>], [<span style="color: #666666">0</span>, <span style="color: #666666">1</span>], [<span style="color: #666666">1</span>, <span style="color: #666666">0</span>],[<span style="color: #666666">1</span>, <span style="color: #666666">1</span>]],dtype<span style="color: #666666">=</span>np<span style="color: #666666">.</span>float64)
<span style="color: #408080; font-style: italic"># The XOR gate</span>
yXOR <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array( [ <span style="color: #666666">0</span>, <span style="color: #666666">1</span> ,<span style="color: #666666">1</span>, <span style="color: #666666">0</span>])
<span style="color: #408080; font-style: italic"># The OR gate</span>
yOR <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array( [ <span style="color: #666666">0</span>, <span style="color: #666666">1</span> ,<span style="color: #666666">1</span>, <span style="color: #666666">1</span>])
<span style="color: #408080; font-style: italic"># The AND gate</span>
yAND <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array( [ <span style="color: #666666">0</span>, <span style="color: #666666">0</span> ,<span style="color: #666666">0</span>, <span style="color: #666666">1</span>])
<span style="color: #408080; font-style: italic"># Defining the neural network</span>
n_inputs, n_features <span style="color: #666666">=</span> X<span style="color: #666666">.</span>shape
n_hidden_neurons <span style="color: #666666">=</span> <span style="color: #666666">2</span>
n_categories <span style="color: #666666">=</span> <span style="color: #666666">2</span>
n_features <span style="color: #666666">=</span> <span style="color: #666666">2</span>
eta_vals <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-5</span>, <span style="color: #666666">1</span>, <span style="color: #666666">7</span>)
lmbd_vals <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-5</span>, <span style="color: #666666">1</span>, <span style="color: #666666">7</span>)
<span style="color: #408080; font-style: italic"># store models for later use</span>
DNN_scikit <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)), dtype<span style="color: #666666">=</span><span style="color: #008000">object</span>)
epochs <span style="color: #666666">=</span> <span style="color: #666666">100</span>
<span style="color: #008000; font-weight: bold">for</span> i, eta <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(eta_vals):
<span style="color: #008000; font-weight: bold">for</span> j, lmbd <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(lmbd_vals):
dnn <span style="color: #666666">=</span> MLPClassifier(hidden_layer_sizes<span style="color: #666666">=</span>(n_hidden_neurons), activation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;logistic&#39;</span>,
alpha<span style="color: #666666">=</span>lmbd, learning_rate_init<span style="color: #666666">=</span>eta, max_iter<span style="color: #666666">=</span>epochs)
dnn<span style="color: #666666">.</span>fit(X, yXOR)
DNN_scikit[i][j] <span style="color: #666666">=</span> dnn
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Learning rate = &quot;</span>, eta)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Lambda = &quot;</span>, lmbd)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Accuracy score on data set: &quot;</span>, dnn<span style="color: #666666">.</span>score(X, yXOR))
<span style="color: #008000">print</span>()
sns<span style="color: #666666">.</span>set()
test_accuracy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)))
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">len</span>(eta_vals)):
<span style="color: #008000; font-weight: bold">for</span> j <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">len</span>(lmbd_vals)):
dnn <span style="color: #666666">=</span> DNN_scikit[i][j]
test_pred <span style="color: #666666">=</span> dnn<span style="color: #666666">.</span>predict(X)
test_accuracy[i][j] <span style="color: #666666">=</span> accuracy_score(yXOR, test_pred)
fig, ax <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>subplots(figsize <span style="color: #666666">=</span> (<span style="color: #666666">10</span>, <span style="color: #666666">10</span>))
sns<span style="color: #666666">.</span>heatmap(test_accuracy, annot<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>, ax<span style="color: #666666">=</span>ax, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&quot;viridis&quot;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&quot;Test Accuracy&quot;</span>)
ax<span style="color: #666666">.</span>set_ylabel(<span style="color: #BA2121">&quot;$\eta$&quot;</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&quot;$\lambda$&quot;</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="building-neural-networks-in-tensorflow-and-keras">Building neural networks in Tensorflow and Keras </h2>
<p>
doc/pub/week41/ipynb/ipynb-week41-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week41/ipynb/week41.ipynb
+79 -3
View File
@@ -1458,7 +1458,7 @@
"\n",
"## Setting up the Neural Network\n",
"\n",
"We define first our design matrix and the various input vectors."
"We define first our design matrix and the various output vectors for the different gates."
]
},
{
@@ -1537,7 +1537,7 @@
"# Defining the neural network\n",
"n_inputs, n_features = X.shape\n",
"n_hidden_neurons = 2\n",
"n_categories = 1\n",
"n_categories = 2\n",
"n_features = 2\n",
"\n",
"# we make the weights normally distributed using numpy.random.randn\n",
@@ -1562,11 +1562,87 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Not an impressive result. Let us now add the full network with the back-propagation algorithm discussed above.\n",
"Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above.\n",
"\n",
"## The full Network for the Various Gates"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## And the same using Scikit-Learn"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"\n",
"# import necessary packages\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.neural_network import MLPClassifier\n",
"from sklearn.metrics import accuracy_score\n",
"import seaborn as sns\n",
"\n",
"# ensure the same random numbers appear every time\n",
"np.random.seed(0)\n",
"\n",
"# Design matrix\n",
"X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n",
"\n",
"# The XOR gate\n",
"yXOR = np.array( [ 0, 1 ,1, 0])\n",
"# The OR gate\n",
"yOR = np.array( [ 0, 1 ,1, 1])\n",
"# The AND gate\n",
"yAND = np.array( [ 0, 0 ,0, 1])\n",
"\n",
"# Defining the neural network\n",
"n_inputs, n_features = X.shape\n",
"n_hidden_neurons = 2\n",
"n_categories = 2\n",
"n_features = 2\n",
"\n",
"eta_vals = np.logspace(-5, 1, 7)\n",
"lmbd_vals = np.logspace(-5, 1, 7)\n",
"# store models for later use\n",
"DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n",
"epochs = 100\n",
"\n",
"for i, eta in enumerate(eta_vals):\n",
" for j, lmbd in enumerate(lmbd_vals):\n",
" dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',\n",
" alpha=lmbd, learning_rate_init=eta, max_iter=epochs)\n",
" dnn.fit(X, yXOR)\n",
" DNN_scikit[i][j] = dnn\n",
" print(\"Learning rate = \", eta)\n",
" print(\"Lambda = \", lmbd)\n",
" print(\"Accuracy score on data set: \", dnn.score(X, yXOR))\n",
" print()\n",
"\n",
"sns.set()\n",
"test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n",
"for i in range(len(eta_vals)):\n",
" for j in range(len(lmbd_vals)):\n",
" dnn = DNN_scikit[i][j]\n",
" test_pred = dnn.predict(X)\n",
" test_accuracy[i][j] = accuracy_score(yXOR, test_pred)\n",
"\n",
"fig, ax = plt.subplots(figsize = (10, 10))\n",
"sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n",
"ax.set_title(\"Test Accuracy\")\n",
"ax.set_ylabel(\"$\\eta$\")\n",
"ax.set_xlabel(\"$\\lambda$\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
doc/src/week41/programs/FFpart.py
+67
View File
@@ -0,0 +1,67 @@
"""
Simple code that tests XOR, OR and AND gates with linear regression
"""
# import necessary packages
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
def sigmoid(x):
return 1/(1 + np.exp(-x))
def feed_forward(X):
# weighted sum of inputs to the hidden layer
z_h = np.matmul(X, hidden_weights) + hidden_bias
# activation in the hidden layer
a_h = sigmoid(z_h)
# weighted sum of inputs to the output layer
z_o = np.matmul(a_h, output_weights) + output_bias
# softmax output
# axis 0 holds each input and axis 1 the probabilities of each category
probabilities = sigmoid(z_o)
return probabilities
# we obtain a prediction by taking the class with the highest likelihood
def predict(X):
probabilities = feed_forward(X)
return np.argmax(probabilities, axis=1)
# ensure the same random numbers appear every time
np.random.seed(0)
# Design matrix
X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)
# The XOR gate
yXOR = np.array( [ 0, 1 ,1, 0])
# The OR gate
yOR = np.array( [ 0, 1 ,1, 1])
# The AND gate
yAND = np.array( [ 0, 0 ,0, 1])
# Defining the neural network
n_inputs, n_features = X.shape
n_hidden_neurons = 2
n_categories = 2
n_features = 2
# we make the weights normally distributed using numpy.random.randn
# weights and bias in the hidden layer
hidden_weights = np.random.randn(n_features, n_hidden_neurons)
hidden_bias = np.zeros(n_hidden_neurons) + 0.01
# weights and bias in the output layer
output_weights = np.random.randn(n_hidden_neurons, n_categories)
output_bias = np.zeros(n_categories) + 0.01
probabilities = feed_forward(X)
print(probabilities)
predictions = predict(X)
print(predictions)
doc/src/week41/nn.py → doc/src/week41/programs/FFpart.py~
View File
doc/src/week41/programs/ffnn.py
+428
View File
@@ -0,0 +1,428 @@
# import necessary packages
import numpy as np
import matplotlib.pyplot as plt
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score
import seaborn as sns
# ensure the same random numbers appear every time
np.random.seed(0)
# Design matrix
X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)
# The XOR gate
yXOR = np.array( [ 0, 1 ,1, 0])
# The OR gate
yOR = np.array( [ 0, 1 ,1, 1])
# The AND gate
yAND = np.array( [ 0, 0 ,0, 1])
# Defining the neural network
n_inputs, n_features = X.shape
n_hidden_neurons = 2
n_categories = 2
n_features = 2
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
# store models for later use
DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
epochs = 100
batch_size = 100
for i, eta in enumerate(eta_vals):
for j, lmbd in enumerate(lmbd_vals):
dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
dnn.fit(X, yXOR)
DNN_scikit[i][j] = dnn
print("Learning rate = ", eta)
print("Lambda = ", lmbd)
print("Accuracy score on data set: ", dnn.score(X, yXOR))
print()
sns.set()
test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
for i in range(len(eta_vals)):
for j in range(len(lmbd_vals)):
dnn = DNN_scikit[i][j]
test_pred = dnn.predict(X)
test_accuracy[i][j] = accuracy_score(yXOR, test_pred)
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Test Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
"""
# one-hot in numpy
def to_categorical_numpy(integer_vector):
n_inputs = len(integer_vector)
n_categories = np.max(integer_vector) + 1
onehot_vector = np.zeros((n_inputs, n_categories))
onehot_vector[range(n_inputs), integer_vector] = 1
return onehot_vector
#Y_train_onehot, Y_test_onehot = to_categorical(Y_train), to_categorical(Y_test)
Y_train_onehot, Y_test_onehot = to_categorical_numpy(Y_train), to_categorical_numpy(Y_test)
def feed_forward_train(X):
# weighted sum of inputs to the hidden layer
z_h = np.matmul(X, hidden_weights) + hidden_bias
# activation in the hidden layer
a_h = sigmoid(z_h)
# weighted sum of inputs to the output layer
z_o = np.matmul(a_h, output_weights) + output_bias
# softmax output
# axis 0 holds each input and axis 1 the probabilities of each category
exp_term = np.exp(z_o)
probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
# for backpropagation need activations in hidden and output layers
return a_h, probabilities
def backpropagation(X, Y):
a_h, probabilities = feed_forward_train(X)
# error in the output layer
error_output = probabilities - Y
# error in the hidden layer
error_hidden = np.matmul(error_output, output_weights.T) * a_h * (1 - a_h)
# gradients for the output layer
output_weights_gradient = np.matmul(a_h.T, error_output)
output_bias_gradient = np.sum(error_output, axis=0)
# gradient for the hidden layer
hidden_weights_gradient = np.matmul(X.T, error_hidden)
hidden_bias_gradient = np.sum(error_hidden, axis=0)
return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient
print("Old accuracy on training data: " + str(accuracy_score(predict(X_train), Y_train)))
eta = 0.01
lmbd = 0.01
for i in range(1000):
# calculate gradients
dWo, dBo, dWh, dBh = backpropagation(X_train, Y_train_onehot)
# regularization term gradients
dWo += lmbd * output_weights
dWh += lmbd * hidden_weights
# update weights and biases
output_weights -= eta * dWo
output_bias -= eta * dBo
hidden_weights -= eta * dWh
hidden_bias -= eta * dBh
print("New accuracy on training data: " + str(accuracy_score(predict(X_train), Y_train)))
!ec
!split
===== Improving performance =====
As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image.
In order to obtain a network that does something useful, we will have to do a bit more work.
The choice of *hyperparameters* such as learning rate and regularization parameter is hugely influential for the performance of the network. Typically a *grid-search* is performed, wherein we test different hyperparameters separated by orders of magnitude. For example we could test the learning rates $\eta = 10^{-6}, 10^{-5},...,10^{-1}$ with different regularization parameters $\lambda = 10^{-6},...,10^{-0}$.
Next, we haven't implemented minibatching yet, which introduces stochasticity and is though to act as an important regularizer on the weights. We call a feed-forward + backward pass with a minibatch an *iteration*, and a full training period
going through the entire dataset ($n/M$ batches) an *epoch*.
If this does not improve network performance, you may want to consider altering the network architecture, adding more neurons or hidden layers.
Andrew Ng goes through some of these considerations in this "video":"https://youtu.be/F1ka6a13S9I". You can find a summary of the video "here":"https://kevinzakka.github.io/2016/09/26/applying-deep-learning/".
!split
===== Full object-oriented implementation =====
It is very natural to think of the network as an object, with specific instances of the network
being realizations of this object with different hyperparameters. An implementation using Python classes provides a clean structure and interface, and the full implementation of our neural network is given below.
!bc pycod
class NeuralNetwork:
def __init__(
self,
X_data,
Y_data,
n_hidden_neurons=50,
n_categories=10,
epochs=10,
batch_size=100,
eta=0.1,
lmbd=0.0):
self.X_data_full = X_data
self.Y_data_full = Y_data
self.n_inputs = X_data.shape[0]
self.n_features = X_data.shape[1]
self.n_hidden_neurons = n_hidden_neurons
self.n_categories = n_categories
self.epochs = epochs
self.batch_size = batch_size
self.iterations = self.n_inputs // self.batch_size
self.eta = eta
self.lmbd = lmbd
self.create_biases_and_weights()
def create_biases_and_weights(self):
self.hidden_weights = np.random.randn(self.n_features, self.n_hidden_neurons)
self.hidden_bias = np.zeros(self.n_hidden_neurons) + 0.01
self.output_weights = np.random.randn(self.n_hidden_neurons, self.n_categories)
self.output_bias = np.zeros(self.n_categories) + 0.01
def feed_forward(self):
# feed-forward for training
self.z_h = np.matmul(self.X_data, self.hidden_weights) + self.hidden_bias
self.a_h = sigmoid(self.z_h)
self.z_o = np.matmul(self.a_h, self.output_weights) + self.output_bias
exp_term = np.exp(self.z_o)
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
def feed_forward_out(self, X):
# feed-forward for output
z_h = np.matmul(X, self.hidden_weights) + self.hidden_bias
a_h = sigmoid(z_h)
z_o = np.matmul(a_h, self.output_weights) + self.output_bias
exp_term = np.exp(z_o)
probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
return probabilities
def backpropagation(self):
error_output = self.probabilities - self.Y_data
error_hidden = np.matmul(error_output, self.output_weights.T) * self.a_h * (1 - self.a_h)
self.output_weights_gradient = np.matmul(self.a_h.T, error_output)
self.output_bias_gradient = np.sum(error_output, axis=0)
self.hidden_weights_gradient = np.matmul(self.X_data.T, error_hidden)
self.hidden_bias_gradient = np.sum(error_hidden, axis=0)
if self.lmbd > 0.0:
self.output_weights_gradient += self.lmbd * self.output_weights
self.hidden_weights_gradient += self.lmbd * self.hidden_weights
self.output_weights -= self.eta * self.output_weights_gradient
self.output_bias -= self.eta * self.output_bias_gradient
self.hidden_weights -= self.eta * self.hidden_weights_gradient
self.hidden_bias -= self.eta * self.hidden_bias_gradient
def predict(self, X):
probabilities = self.feed_forward_out(X)
return np.argmax(probabilities, axis=1)
def predict_probabilities(self, X):
probabilities = self.feed_forward_out(X)
return probabilities
def train(self):
data_indices = np.arange(self.n_inputs)
for i in range(self.epochs):
for j in range(self.iterations):
# pick datapoints with replacement
chosen_datapoints = np.random.choice(
data_indices, size=self.batch_size, replace=False
)
# minibatch training data
self.X_data = self.X_data_full[chosen_datapoints]
self.Y_data = self.Y_data_full[chosen_datapoints]
self.feed_forward()
self.backpropagation()
!ec
!split
===== Evaluate model performance on test data =====
To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data.
We measure the performance of the network using the *accuracy* score.
The accuracy is as you would expect just the number of images correctly labeled divided by the total number of images. A perfect classifier will have an accuracy score of $1$.
$$ \text{Accuracy} = \frac{\sum_{i=1}^n I(\hat{y}_i = y_i)}{n} ,$$
where $I$ is the indicator function, $1$ if $\hat{y}_i = y_i$ and $0$ otherwise.
!bc pycod
epochs = 100
batch_size = 100
dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,
n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)
dnn.train()
test_predict = dnn.predict(X_test)
# accuracy score from scikit library
print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict))
# equivalent in numpy
def accuracy_score_numpy(Y_test, Y_pred):
return np.sum(Y_test == Y_pred) / len(Y_test)
#print("Accuracy score on test set: ", accuracy_score_numpy(Y_test, test_predict))
!ec
!split
===== Adjust hyperparameters =====
We now perform a grid search to find the optimal hyperparameters for the network.
Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98\%$ ($2\%$ error rate).
!bc pycod
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
# store the models for later use
DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
# grid search
for i, eta in enumerate(eta_vals):
for j, lmbd in enumerate(lmbd_vals):
dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,
n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)
dnn.train()
DNN_numpy[i][j] = dnn
test_predict = dnn.predict(X_test)
print("Learning rate = ", eta)
print("Lambda = ", lmbd)
print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict))
print()
!ec
!split
===== Visualization =====
!bc pycod
# visual representation of grid search
# uses seaborn heatmap, you can also do this with matplotlib imshow
import seaborn as sns
sns.set()
train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
for i in range(len(eta_vals)):
for j in range(len(lmbd_vals)):
dnn = DNN_numpy[i][j]
train_pred = dnn.predict(X_train)
test_pred = dnn.predict(X_test)
train_accuracy[i][j] = accuracy_score(Y_train, train_pred)
test_accuracy[i][j] = accuracy_score(Y_test, test_pred)
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Training Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Test Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
!ec
!split
===== scikit-learn implementation =====
_scikit-learn_ focuses more
on traditional machine learning methods, such as regression,
clustering, decision trees, etc. As such, it has only two types of
neural networks: Multi Layer Perceptron outputting continuous values,
*MPLRegressor*, and Multi Layer Perceptron outputting labels,
*MLPClassifier*. We will see how simple it is to use these classes.
_scikit-learn_ implements a few improvements from our neural network,
such as early stopping, a varying learning rate, different
optimization methods, etc. We would therefore expect a better
performance overall.
!bc pycod
from sklearn.neural_network import MLPClassifier
# store models for later use
DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
for i, eta in enumerate(eta_vals):
for j, lmbd in enumerate(lmbd_vals):
dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
dnn.fit(X_train, Y_train)
DNN_scikit[i][j] = dnn
print("Learning rate = ", eta)
print("Lambda = ", lmbd)
print("Accuracy score on test set: ", dnn.score(X_test, Y_test))
print()
!ec
!split
===== Visualization =====
!bc pycod
# optional
# visual representation of grid search
# uses seaborn heatmap, could probably do this in matplotlib
import seaborn as sns
sns.set()
train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
for i in range(len(eta_vals)):
for j in range(len(lmbd_vals)):
dnn = DNN_scikit[i][j]
train_pred = dnn.predict(X_train)
test_pred = dnn.predict(X_test)
train_accuracy[i][j] = accuracy_score(Y_train, train_pred)
test_accuracy[i][j] = accuracy_score(Y_test, test_pred)
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Training Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Test Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
!ec
"""
doc/src/week41/programs/ffnn.py~
+428
View File
@@ -0,0 +1,428 @@
# import necessary packages
import numpy as np
import matplotlib.pyplot as plt
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score
import seaborn as sns
# ensure the same random numbers appear every time
np.random.seed(0)
# Design matrix
X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)
# The XOR gate
yXOR = np.array( [ 0, 1 ,1, 0])
# The OR gate
yOR = np.array( [ 0, 1 ,1, 1])
# The AND gate
yAND = np.array( [ 0, 0 ,0, 1])
# Defining the neural network
n_inputs, n_features = X.shape
n_hidden_neurons = 2
n_categories = 1
n_features = 2
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
# store models for later use
DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
epochs = 100
batch_size = 100
for i, eta in enumerate(eta_vals):
for j, lmbd in enumerate(lmbd_vals):
dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
dnn.fit(X, yXOR)
DNN_scikit[i][j] = dnn
print("Learning rate = ", eta)
print("Lambda = ", lmbd)
print("Accuracy score on data set: ", dnn.score(X, yXOR))
print()
sns.set()
test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
for i in range(len(eta_vals)):
for j in range(len(lmbd_vals)):
dnn = DNN_scikit[i][j]
test_pred = dnn.predict(X)
test_accuracy[i][j] = accuracy_score(yXOR, test_pred)
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Test Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
"""
# one-hot in numpy
def to_categorical_numpy(integer_vector):
n_inputs = len(integer_vector)
n_categories = np.max(integer_vector) + 1
onehot_vector = np.zeros((n_inputs, n_categories))
onehot_vector[range(n_inputs), integer_vector] = 1
return onehot_vector
#Y_train_onehot, Y_test_onehot = to_categorical(Y_train), to_categorical(Y_test)
Y_train_onehot, Y_test_onehot = to_categorical_numpy(Y_train), to_categorical_numpy(Y_test)
def feed_forward_train(X):
# weighted sum of inputs to the hidden layer
z_h = np.matmul(X, hidden_weights) + hidden_bias
# activation in the hidden layer
a_h = sigmoid(z_h)
# weighted sum of inputs to the output layer
z_o = np.matmul(a_h, output_weights) + output_bias
# softmax output
# axis 0 holds each input and axis 1 the probabilities of each category
exp_term = np.exp(z_o)
probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
# for backpropagation need activations in hidden and output layers
return a_h, probabilities
def backpropagation(X, Y):
a_h, probabilities = feed_forward_train(X)
# error in the output layer
error_output = probabilities - Y
# error in the hidden layer
error_hidden = np.matmul(error_output, output_weights.T) * a_h * (1 - a_h)
# gradients for the output layer
output_weights_gradient = np.matmul(a_h.T, error_output)
output_bias_gradient = np.sum(error_output, axis=0)
# gradient for the hidden layer
hidden_weights_gradient = np.matmul(X.T, error_hidden)
hidden_bias_gradient = np.sum(error_hidden, axis=0)
return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient
print("Old accuracy on training data: " + str(accuracy_score(predict(X_train), Y_train)))
eta = 0.01
lmbd = 0.01
for i in range(1000):
# calculate gradients
dWo, dBo, dWh, dBh = backpropagation(X_train, Y_train_onehot)
# regularization term gradients
dWo += lmbd * output_weights
dWh += lmbd * hidden_weights
# update weights and biases
output_weights -= eta * dWo
output_bias -= eta * dBo
hidden_weights -= eta * dWh
hidden_bias -= eta * dBh
print("New accuracy on training data: " + str(accuracy_score(predict(X_train), Y_train)))
!ec
!split
===== Improving performance =====
As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image.
In order to obtain a network that does something useful, we will have to do a bit more work.
The choice of *hyperparameters* such as learning rate and regularization parameter is hugely influential for the performance of the network. Typically a *grid-search* is performed, wherein we test different hyperparameters separated by orders of magnitude. For example we could test the learning rates $\eta = 10^{-6}, 10^{-5},...,10^{-1}$ with different regularization parameters $\lambda = 10^{-6},...,10^{-0}$.
Next, we haven't implemented minibatching yet, which introduces stochasticity and is though to act as an important regularizer on the weights. We call a feed-forward + backward pass with a minibatch an *iteration*, and a full training period
going through the entire dataset ($n/M$ batches) an *epoch*.
If this does not improve network performance, you may want to consider altering the network architecture, adding more neurons or hidden layers.
Andrew Ng goes through some of these considerations in this "video":"https://youtu.be/F1ka6a13S9I". You can find a summary of the video "here":"https://kevinzakka.github.io/2016/09/26/applying-deep-learning/".
!split
===== Full object-oriented implementation =====
It is very natural to think of the network as an object, with specific instances of the network
being realizations of this object with different hyperparameters. An implementation using Python classes provides a clean structure and interface, and the full implementation of our neural network is given below.
!bc pycod
class NeuralNetwork:
def __init__(
self,
X_data,
Y_data,
n_hidden_neurons=50,
n_categories=10,
epochs=10,
batch_size=100,
eta=0.1,
lmbd=0.0):
self.X_data_full = X_data
self.Y_data_full = Y_data
self.n_inputs = X_data.shape[0]
self.n_features = X_data.shape[1]
self.n_hidden_neurons = n_hidden_neurons
self.n_categories = n_categories
self.epochs = epochs
self.batch_size = batch_size
self.iterations = self.n_inputs // self.batch_size
self.eta = eta
self.lmbd = lmbd
self.create_biases_and_weights()
def create_biases_and_weights(self):
self.hidden_weights = np.random.randn(self.n_features, self.n_hidden_neurons)
self.hidden_bias = np.zeros(self.n_hidden_neurons) + 0.01
self.output_weights = np.random.randn(self.n_hidden_neurons, self.n_categories)
self.output_bias = np.zeros(self.n_categories) + 0.01
def feed_forward(self):
# feed-forward for training
self.z_h = np.matmul(self.X_data, self.hidden_weights) + self.hidden_bias
self.a_h = sigmoid(self.z_h)
self.z_o = np.matmul(self.a_h, self.output_weights) + self.output_bias
exp_term = np.exp(self.z_o)
self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
def feed_forward_out(self, X):
# feed-forward for output
z_h = np.matmul(X, self.hidden_weights) + self.hidden_bias
a_h = sigmoid(z_h)
z_o = np.matmul(a_h, self.output_weights) + self.output_bias
exp_term = np.exp(z_o)
probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
return probabilities
def backpropagation(self):
error_output = self.probabilities - self.Y_data
error_hidden = np.matmul(error_output, self.output_weights.T) * self.a_h * (1 - self.a_h)
self.output_weights_gradient = np.matmul(self.a_h.T, error_output)
self.output_bias_gradient = np.sum(error_output, axis=0)
self.hidden_weights_gradient = np.matmul(self.X_data.T, error_hidden)
self.hidden_bias_gradient = np.sum(error_hidden, axis=0)
if self.lmbd > 0.0:
self.output_weights_gradient += self.lmbd * self.output_weights
self.hidden_weights_gradient += self.lmbd * self.hidden_weights
self.output_weights -= self.eta * self.output_weights_gradient
self.output_bias -= self.eta * self.output_bias_gradient
self.hidden_weights -= self.eta * self.hidden_weights_gradient
self.hidden_bias -= self.eta * self.hidden_bias_gradient
def predict(self, X):
probabilities = self.feed_forward_out(X)
return np.argmax(probabilities, axis=1)
def predict_probabilities(self, X):
probabilities = self.feed_forward_out(X)
return probabilities
def train(self):
data_indices = np.arange(self.n_inputs)
for i in range(self.epochs):
for j in range(self.iterations):
# pick datapoints with replacement
chosen_datapoints = np.random.choice(
data_indices, size=self.batch_size, replace=False
)
# minibatch training data
self.X_data = self.X_data_full[chosen_datapoints]
self.Y_data = self.Y_data_full[chosen_datapoints]
self.feed_forward()
self.backpropagation()
!ec
!split
===== Evaluate model performance on test data =====
To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data.
We measure the performance of the network using the *accuracy* score.
The accuracy is as you would expect just the number of images correctly labeled divided by the total number of images. A perfect classifier will have an accuracy score of $1$.
$$ \text{Accuracy} = \frac{\sum_{i=1}^n I(\hat{y}_i = y_i)}{n} ,$$
where $I$ is the indicator function, $1$ if $\hat{y}_i = y_i$ and $0$ otherwise.
!bc pycod
epochs = 100
batch_size = 100
dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,
n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)
dnn.train()
test_predict = dnn.predict(X_test)
# accuracy score from scikit library
print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict))
# equivalent in numpy
def accuracy_score_numpy(Y_test, Y_pred):
return np.sum(Y_test == Y_pred) / len(Y_test)
#print("Accuracy score on test set: ", accuracy_score_numpy(Y_test, test_predict))
!ec
!split
===== Adjust hyperparameters =====
We now perform a grid search to find the optimal hyperparameters for the network.
Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98\%$ ($2\%$ error rate).
!bc pycod
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
# store the models for later use
DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
# grid search
for i, eta in enumerate(eta_vals):
for j, lmbd in enumerate(lmbd_vals):
dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,
n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)
dnn.train()
DNN_numpy[i][j] = dnn
test_predict = dnn.predict(X_test)
print("Learning rate = ", eta)
print("Lambda = ", lmbd)
print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict))
print()
!ec
!split
===== Visualization =====
!bc pycod
# visual representation of grid search
# uses seaborn heatmap, you can also do this with matplotlib imshow
import seaborn as sns
sns.set()
train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
for i in range(len(eta_vals)):
for j in range(len(lmbd_vals)):
dnn = DNN_numpy[i][j]
train_pred = dnn.predict(X_train)
test_pred = dnn.predict(X_test)
train_accuracy[i][j] = accuracy_score(Y_train, train_pred)
test_accuracy[i][j] = accuracy_score(Y_test, test_pred)
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Training Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Test Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
!ec
!split
===== scikit-learn implementation =====
_scikit-learn_ focuses more
on traditional machine learning methods, such as regression,
clustering, decision trees, etc. As such, it has only two types of
neural networks: Multi Layer Perceptron outputting continuous values,
*MPLRegressor*, and Multi Layer Perceptron outputting labels,
*MLPClassifier*. We will see how simple it is to use these classes.
_scikit-learn_ implements a few improvements from our neural network,
such as early stopping, a varying learning rate, different
optimization methods, etc. We would therefore expect a better
performance overall.
!bc pycod
from sklearn.neural_network import MLPClassifier
# store models for later use
DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
for i, eta in enumerate(eta_vals):
for j, lmbd in enumerate(lmbd_vals):
dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
dnn.fit(X_train, Y_train)
DNN_scikit[i][j] = dnn
print("Learning rate = ", eta)
print("Lambda = ", lmbd)
print("Accuracy score on test set: ", dnn.score(X_test, Y_test))
print()
!ec
!split
===== Visualization =====
!bc pycod
# optional
# visual representation of grid search
# uses seaborn heatmap, could probably do this in matplotlib
import seaborn as sns
sns.set()
train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
for i in range(len(eta_vals)):
for j in range(len(lmbd_vals)):
dnn = DNN_scikit[i][j]
train_pred = dnn.predict(X_train)
test_pred = dnn.predict(X_test)
train_accuracy[i][j] = accuracy_score(Y_train, train_pred)
test_accuracy[i][j] = accuracy_score(Y_test, test_pred)
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Training Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Test Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
!ec
"""
doc/src/week41/week41.do.txt
+68 -3
View File
@@ -1125,7 +1125,7 @@ while the vector of outputs is $\bm{y}^T=[0,1,1,0]$ for the XOR gate, $\bm{y}^T=
!split
===== Setting up the Neural Network =====
We define first our design matrix and the various input vectors.
We define first our design matrix and the various output vectors for the different gates.
!bc pycod
"""
@@ -1184,7 +1184,7 @@ np.random.seed(0)
# Defining the neural network
n_inputs, n_features = X.shape
n_hidden_neurons = 2
n_categories = 1
n_categories = 2
n_features = 2
# we make the weights normally distributed using numpy.random.randn
@@ -1204,7 +1204,7 @@ print(probabilities)
predictions = predict(X)
print(predictions)
!ec
Not an impressive result. Let us now add the full network with the back-propagation algorithm discussed above.
Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above.
!split
===== The full Network for the Various Gates =====
@@ -1212,6 +1212,71 @@ Not an impressive result. Let us now add the full network with the back-propagat
!ec
!split
===== And the same using Scikit-Learn =====
!bc pycod
# import necessary packages
import numpy as np
import matplotlib.pyplot as plt
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score
import seaborn as sns
# ensure the same random numbers appear every time
np.random.seed(0)
# Design matrix
X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)
# The XOR gate
yXOR = np.array( [ 0, 1 ,1, 0])
# The OR gate
yOR = np.array( [ 0, 1 ,1, 1])
# The AND gate
yAND = np.array( [ 0, 0 ,0, 1])
# Defining the neural network
n_inputs, n_features = X.shape
n_hidden_neurons = 2
n_categories = 2
n_features = 2
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
# store models for later use
DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
epochs = 100
for i, eta in enumerate(eta_vals):
for j, lmbd in enumerate(lmbd_vals):
dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
dnn.fit(X, yXOR)
DNN_scikit[i][j] = dnn
print("Learning rate = ", eta)
print("Lambda = ", lmbd)
print("Accuracy score on data set: ", dnn.score(X, yXOR))
print()
sns.set()
test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
for i in range(len(eta_vals)):
for j in range(len(lmbd_vals)):
dnn = DNN_scikit[i][j]
test_pred = dnn.predict(X)
test_accuracy[i][j] = accuracy_score(yXOR, test_pred)
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Test Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
!ec
!split
===== Building neural networks in Tensorflow and Keras =====