lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-10-13 16:19:10 +02:00
parent 027fe32f90
commit a462464a08
8 changed files with 504 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week41/html/week41-bs.html
+51 -41
View File
@@ -120,6 +120,14 @@ Automatically generated HTML file from DocOnce source
2,
None,
'setting-up-the-neural-network'),
('Then the first Feed Forward pass',
2,
None,
'then-the-first-feed-forward-pass'),
('The full Network for the Various Gates',
2,
None,
'the-full-network-for-the-various-gates'),
('Building neural networks in Tensorflow and Keras',
2,
None,
@@ -265,7 +273,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week41-bs006.html#example-binary-classification-problem" style="font-size: 80%;">Example: binary classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs007.html#the-softmax-function" style="font-size: 80%;">The Softmax function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs008.html#developing-a-code-for-doing-neural-networks-with-back-propagation" style="font-size: 80%;">Developing a code for doing neural networks with back propagation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs034.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs036.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs010.html#train-and-test-datasets" style="font-size: 80%;">Train and test datasets</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs011.html#define-model-and-architecture" style="font-size: 80%;">Define model and architecture</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs012.html#layers" style="font-size: 80%;">Layers</a></li>
@@ -287,45 +295,47 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week41-bs028.html#the-and-and-xor-gates" style="font-size: 80%;">The AND and XOR Gates</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs029.html#representing-the-data-sets" style="font-size: 80%;">Representing the Data Sets</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs030.html#setting-up-the-neural-network" style="font-size: 80%;">Setting up the Neural Network</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs031.html#building-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">Building neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs032.html#tensorflow" style="font-size: 80%;">Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs033.html#using-keras" style="font-size: 80%;">Using Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs034.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs035.html#the-breast-cancer-data-now-with-keras" style="font-size: 80%;">The Breast Cancer Data, now with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs036.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;">Fine-tuning neural network hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs037.html#hidden-layers" style="font-size: 80%;">Hidden layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs038.html#which-activation-function-should-i-use" style="font-size: 80%;">Which activation function should I use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs039.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;">Is the Logistic activation function (Sigmoid) our choice?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs040.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;">The derivative of the Logistic funtion</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs041.html#the-relu-function-family" style="font-size: 80%;">The RELU function family</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs042.html#which-activation-function-should-we-use" style="font-size: 80%;">Which activation function should we use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs043.html#more-on-activation-functions-output-layers" style="font-size: 80%;">More on activation functions, output layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs044.html#batch-normalization" style="font-size: 80%;">Batch Normalization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs045.html#dropout" style="font-size: 80%;">Dropout</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs046.html#gradient-clipping" style="font-size: 80%;">Gradient Clipping</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs047.html#a-very-nice-website-on-neural-networks" style="font-size: 80%;">A very nice website on Neural Networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs048.html#a-top-down-perspective-on-neural-networks" style="font-size: 80%;">A top-down perspective on Neural networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs049.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;">Limitations of supervised learning with deep networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs050.html#overarching-views-a-personal-note" style="font-size: 80%;">Overarching Views, a personal note</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs051.html#from-a-spherical-cow-to-a-real-one" style="font-size: 80%;">From a Spherical Cow to a real one</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs052.html#convolutional-neural-networks-recognizing-images" style="font-size: 80%;">Convolutional Neural Networks (recognizing images)</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs053.html#regular-nns-don-t-scale-well-to-full-images" style="font-size: 80%;">Regular NNs don’t scale well to full images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs054.html#3d-volumes-of-neurons" style="font-size: 80%;">3D volumes of neurons</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs055.html#layers-used-to-build-cnns" style="font-size: 80%;">Layers used to build CNNs</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs056.html#transforming-images" style="font-size: 80%;">Transforming images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs057.html#cnns-in-brief" style="font-size: 80%;">CNNs in brief</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs058.html#cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs059.html#setting-it-up" style="font-size: 80%;">Setting it up</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs060.html#the-mnist-dataset-again" style="font-size: 80%;">The MNIST dataset again</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs061.html#strong-correlations" style="font-size: 80%;">Strong correlations</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs062.html#layers-of-a-cnn" style="font-size: 80%;">Layers of a CNN</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs063.html#systematic-reduction" style="font-size: 80%;">Systematic reduction</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs064.html#prerequisites-collect-and-pre-process-data" style="font-size: 80%;">Prerequisites: Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs065.html#importing-keras-and-tensorflow" style="font-size: 80%;">Importing Keras and Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs066.html#running-with-keras" style="font-size: 80%;">Running with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs067.html#final-part" style="font-size: 80%;">Final part</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs068.html#final-visualization" style="font-size: 80%;">Final visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs069.html#fun-links" style="font-size: 80%;">Fun links</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs031.html#then-the-first-feed-forward-pass" style="font-size: 80%;">Then the first Feed Forward pass</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs032.html#the-full-network-for-the-various-gates" style="font-size: 80%;">The full Network for the Various Gates</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs033.html#building-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">Building neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs034.html#tensorflow" style="font-size: 80%;">Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs035.html#using-keras" style="font-size: 80%;">Using Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs036.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs037.html#the-breast-cancer-data-now-with-keras" style="font-size: 80%;">The Breast Cancer Data, now with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs038.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;">Fine-tuning neural network hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs039.html#hidden-layers" style="font-size: 80%;">Hidden layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs040.html#which-activation-function-should-i-use" style="font-size: 80%;">Which activation function should I use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs041.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;">Is the Logistic activation function (Sigmoid) our choice?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs042.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;">The derivative of the Logistic funtion</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs043.html#the-relu-function-family" style="font-size: 80%;">The RELU function family</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs044.html#which-activation-function-should-we-use" style="font-size: 80%;">Which activation function should we use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs045.html#more-on-activation-functions-output-layers" style="font-size: 80%;">More on activation functions, output layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs046.html#batch-normalization" style="font-size: 80%;">Batch Normalization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs047.html#dropout" style="font-size: 80%;">Dropout</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs048.html#gradient-clipping" style="font-size: 80%;">Gradient Clipping</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs049.html#a-very-nice-website-on-neural-networks" style="font-size: 80%;">A very nice website on Neural Networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs050.html#a-top-down-perspective-on-neural-networks" style="font-size: 80%;">A top-down perspective on Neural networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs051.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;">Limitations of supervised learning with deep networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs052.html#overarching-views-a-personal-note" style="font-size: 80%;">Overarching Views, a personal note</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs053.html#from-a-spherical-cow-to-a-real-one" style="font-size: 80%;">From a Spherical Cow to a real one</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs054.html#convolutional-neural-networks-recognizing-images" style="font-size: 80%;">Convolutional Neural Networks (recognizing images)</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs055.html#regular-nns-don-t-scale-well-to-full-images" style="font-size: 80%;">Regular NNs don’t scale well to full images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs056.html#3d-volumes-of-neurons" style="font-size: 80%;">3D volumes of neurons</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs057.html#layers-used-to-build-cnns" style="font-size: 80%;">Layers used to build CNNs</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs058.html#transforming-images" style="font-size: 80%;">Transforming images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs059.html#cnns-in-brief" style="font-size: 80%;">CNNs in brief</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs060.html#cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs061.html#setting-it-up" style="font-size: 80%;">Setting it up</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs062.html#the-mnist-dataset-again" style="font-size: 80%;">The MNIST dataset again</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs063.html#strong-correlations" style="font-size: 80%;">Strong correlations</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs064.html#layers-of-a-cnn" style="font-size: 80%;">Layers of a CNN</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs065.html#systematic-reduction" style="font-size: 80%;">Systematic reduction</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs066.html#prerequisites-collect-and-pre-process-data" style="font-size: 80%;">Prerequisites: Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs067.html#importing-keras-and-tensorflow" style="font-size: 80%;">Importing Keras and Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs068.html#running-with-keras" style="font-size: 80%;">Running with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs069.html#final-part" style="font-size: 80%;">Final part</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs070.html#final-visualization" style="font-size: 80%;">Final visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs071.html#fun-links" style="font-size: 80%;">Fun links</a></li>
</ul>
</li>
@@ -384,7 +394,7 @@ MathJax.Hub.Config({
<li><a href="._week41-bs008.html">9</a></li>
<li><a href="._week41-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week41-bs069.html">70</a></li>
<li><a href="._week41-bs071.html">72</a></li>
<li><a href="._week41-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week41/html/week41-reveal.html
+74 -3
View File
@@ -1559,8 +1559,11 @@ We define first our design matrix and the various input vectors.
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #CD5555">Simple code that tests XOR, OR and AND gates with linear regression</span>
<span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #228B22"># import necessary packages</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn</span> <span style="color: #8B008B; font-weight: bold">import</span> datasets
<span style="color: #228B22"># Design matrix</span>
X = np.array([ [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>],[<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>]],dtype=np.float64)
@@ -1570,9 +1573,77 @@ yXOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B
yOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>])
<span style="color: #228B22"># The AND gate </span>
yAND = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span> ,<span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>])
</pre></div>
</section>
<span style="color: #228B22">#print(f&quot;The values of theta for the AND gate:{ThetaAND}&quot;)</span>
<span style="color: #228B22">#print(f&quot;The linear regression prediction for the AND gate:{X @ ThetaAND}&quot;)</span>
<section>
<h2 id="then-the-first-feed-forward-pass">Then the first Feed Forward pass </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">sigmoid</span>(x):
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">1</span>/(<span style="color: #B452CD">1</span> + np.exp(-x))
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">feed_forward</span>(X):
<span style="color: #228B22"># weighted sum of inputs to the hidden layer</span>
z_h = np.matmul(X, hidden_weights) + hidden_bias
<span style="color: #228B22"># activation in the hidden layer</span>
a_h = sigmoid(z_h)
<span style="color: #228B22"># weighted sum of inputs to the output layer</span>
z_o = np.matmul(a_h, output_weights) + output_bias
<span style="color: #228B22"># softmax output</span>
<span style="color: #228B22"># axis 0 holds each input and axis 1 the probabilities of each category</span>
probabilities = sigmoid(z_o)
<span style="color: #8B008B; font-weight: bold">return</span> probabilities
<span style="color: #228B22"># we obtain a prediction by taking the class with the highest likelihood</span>
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">predict</span>(X):
probabilities = feed_forward(X)
<span style="color: #8B008B; font-weight: bold">return</span> np.argmax(probabilities, axis=<span style="color: #B452CD">1</span>)
<span style="color: #228B22"># ensure the same random numbers appear every time</span>
np.random.seed(<span style="color: #B452CD">0</span>)
<span style="color: #228B22"># Defining the neural network</span>
n_inputs, n_features = X.shape
n_hidden_neurons = <span style="color: #B452CD">2</span>
n_categories = <span style="color: #B452CD">1</span>
n_features = <span style="color: #B452CD">2</span>
<span style="color: #228B22"># we make the weights normally distributed using numpy.random.randn</span>
<span style="color: #228B22"># weights and bias in the hidden layer</span>
hidden_weights = np.random.randn(n_features, n_hidden_neurons)
hidden_bias = np.zeros(n_hidden_neurons) + <span style="color: #B452CD">0.01</span>
<span style="color: #228B22"># weights and bias in the output layer</span>
output_weights = np.random.randn(n_hidden_neurons, n_categories)
output_bias = np.zeros(n_categories) + <span style="color: #B452CD">0.01</span>
probabilities = feed_forward(X)
<span style="color: #658b00">print</span>(probabilities)
predictions = predict(X)
<span style="color: #658b00">print</span>(predictions)
</pre></div>
<p>
Not an impressive result. Let us now add the full network with the back-propagation algorithm discussed above.
</section>
<section>
<h2 id="the-full-network-for-the-various-gates">The full Network for the Various Gates </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>
</pre></div>
</section>
doc/pub/week41/html/week41-solarized.html
+81 -3
View File
@@ -140,6 +140,14 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'setting-up-the-neural-network'),
('Then the first Feed Forward pass',
2,
None,
'then-the-first-feed-forward-pass'),
('The full Network for the Various Gates',
2,
None,
'the-full-network-for-the-various-gates'),
('Building neural networks in Tensorflow and Keras',
2,
None,
@@ -1564,8 +1572,11 @@ We define first our design matrix and the various input vectors.
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #CD5555">Simple code that tests XOR, OR and AND gates with linear regression</span>
<span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #228B22"># import necessary packages</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn</span> <span style="color: #8B008B; font-weight: bold">import</span> datasets
<span style="color: #228B22"># Design matrix</span>
X = np.array([ [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>],[<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>]],dtype=np.float64)
@@ -1575,9 +1586,76 @@ yXOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B
yOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>])
<span style="color: #228B22"># The AND gate </span>
yAND = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span> ,<span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>])
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<span style="color: #228B22">#print(f&quot;The values of theta for the AND gate:{ThetaAND}&quot;)</span>
<span style="color: #228B22">#print(f&quot;The linear regression prediction for the AND gate:{X @ ThetaAND}&quot;)</span>
<h2 id="then-the-first-feed-forward-pass">Then the first Feed Forward pass </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">sigmoid</span>(x):
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">1</span>/(<span style="color: #B452CD">1</span> + np.exp(-x))
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">feed_forward</span>(X):
<span style="color: #228B22"># weighted sum of inputs to the hidden layer</span>
z_h = np.matmul(X, hidden_weights) + hidden_bias
<span style="color: #228B22"># activation in the hidden layer</span>
a_h = sigmoid(z_h)
<span style="color: #228B22"># weighted sum of inputs to the output layer</span>
z_o = np.matmul(a_h, output_weights) + output_bias
<span style="color: #228B22"># softmax output</span>
<span style="color: #228B22"># axis 0 holds each input and axis 1 the probabilities of each category</span>
probabilities = sigmoid(z_o)
<span style="color: #8B008B; font-weight: bold">return</span> probabilities
<span style="color: #228B22"># we obtain a prediction by taking the class with the highest likelihood</span>
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">predict</span>(X):
probabilities = feed_forward(X)
<span style="color: #8B008B; font-weight: bold">return</span> np.argmax(probabilities, axis=<span style="color: #B452CD">1</span>)
<span style="color: #228B22"># ensure the same random numbers appear every time</span>
np.random.seed(<span style="color: #B452CD">0</span>)
<span style="color: #228B22"># Defining the neural network</span>
n_inputs, n_features = X.shape
n_hidden_neurons = <span style="color: #B452CD">2</span>
n_categories = <span style="color: #B452CD">1</span>
n_features = <span style="color: #B452CD">2</span>
<span style="color: #228B22"># we make the weights normally distributed using numpy.random.randn</span>
<span style="color: #228B22"># weights and bias in the hidden layer</span>
hidden_weights = np.random.randn(n_features, n_hidden_neurons)
hidden_bias = np.zeros(n_hidden_neurons) + <span style="color: #B452CD">0.01</span>
<span style="color: #228B22"># weights and bias in the output layer</span>
output_weights = np.random.randn(n_hidden_neurons, n_categories)
output_bias = np.zeros(n_categories) + <span style="color: #B452CD">0.01</span>
probabilities = feed_forward(X)
<span style="color: #658b00">print</span>(probabilities)
predictions = predict(X)
<span style="color: #658b00">print</span>(predictions)
</pre></div>
<p>
Not an impressive result. Let us now add the full network with the back-propagation algorithm discussed above.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-full-network-for-the-various-gates">The full Network for the Various Gates </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week41/html/week41.html
+81 -3
View File
@@ -145,6 +145,14 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'setting-up-the-neural-network'),
('Then the first Feed Forward pass',
2,
None,
'then-the-first-feed-forward-pass'),
('The full Network for the Various Gates',
2,
None,
'the-full-network-for-the-various-gates'),
('Building neural networks in Tensorflow and Keras',
2,
None,
@@ -1569,8 +1577,11 @@ We define first our design matrix and the various input vectors.
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic">Simple code that tests XOR, OR and AND gates with linear regression</span>
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #408080; font-style: italic"># import necessary packages</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
<span style="color: #408080; font-style: italic"># Design matrix</span>
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([ [<span style="color: #666666">1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">0</span>], [<span style="color: #666666">1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">1</span>], [<span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">0</span>],[<span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>]],dtype<span style="color: #666666">=</span>np<span style="color: #666666">.</span>float64)
@@ -1580,9 +1591,76 @@ yXOR <span style="color: #666666">=</span> np<span style="color: #666666">.</spa
yOR <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array( [ <span style="color: #666666">0</span>, <span style="color: #666666">1</span> ,<span style="color: #666666">1</span>, <span style="color: #666666">1</span>])
<span style="color: #408080; font-style: italic"># The AND gate </span>
yAND <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array( [ <span style="color: #666666">0</span>, <span style="color: #666666">0</span> ,<span style="color: #666666">0</span>, <span style="color: #666666">1</span>])
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<span style="color: #408080; font-style: italic">#print(f&quot;The values of theta for the AND gate:{ThetaAND}&quot;)</span>
<span style="color: #408080; font-style: italic">#print(f&quot;The linear regression prediction for the AND gate:{X @ ThetaAND}&quot;)</span>
<h2 id="then-the-first-feed-forward-pass">Then the first Feed Forward pass </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">sigmoid</span>(x):
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">1/</span>(<span style="color: #666666">1</span> <span style="color: #666666">+</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>x))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">feed_forward</span>(X):
<span style="color: #408080; font-style: italic"># weighted sum of inputs to the hidden layer</span>
z_h <span style="color: #666666">=</span> np<span style="color: #666666">.</span>matmul(X, hidden_weights) <span style="color: #666666">+</span> hidden_bias
<span style="color: #408080; font-style: italic"># activation in the hidden layer</span>
a_h <span style="color: #666666">=</span> sigmoid(z_h)
<span style="color: #408080; font-style: italic"># weighted sum of inputs to the output layer</span>
z_o <span style="color: #666666">=</span> np<span style="color: #666666">.</span>matmul(a_h, output_weights) <span style="color: #666666">+</span> output_bias
<span style="color: #408080; font-style: italic"># softmax output</span>
<span style="color: #408080; font-style: italic"># axis 0 holds each input and axis 1 the probabilities of each category</span>
probabilities <span style="color: #666666">=</span> sigmoid(z_o)
<span style="color: #008000; font-weight: bold">return</span> probabilities
<span style="color: #408080; font-style: italic"># we obtain a prediction by taking the class with the highest likelihood</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">predict</span>(X):
probabilities <span style="color: #666666">=</span> feed_forward(X)
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>argmax(probabilities, axis<span style="color: #666666">=1</span>)
<span style="color: #408080; font-style: italic"># ensure the same random numbers appear every time</span>
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">0</span>)
<span style="color: #408080; font-style: italic"># Defining the neural network</span>
n_inputs, n_features <span style="color: #666666">=</span> X<span style="color: #666666">.</span>shape
n_hidden_neurons <span style="color: #666666">=</span> <span style="color: #666666">2</span>
n_categories <span style="color: #666666">=</span> <span style="color: #666666">1</span>
n_features <span style="color: #666666">=</span> <span style="color: #666666">2</span>
<span style="color: #408080; font-style: italic"># we make the weights normally distributed using numpy.random.randn</span>
<span style="color: #408080; font-style: italic"># weights and bias in the hidden layer</span>
hidden_weights <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(n_features, n_hidden_neurons)
hidden_bias <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(n_hidden_neurons) <span style="color: #666666">+</span> <span style="color: #666666">0.01</span>
<span style="color: #408080; font-style: italic"># weights and bias in the output layer</span>
output_weights <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(n_hidden_neurons, n_categories)
output_bias <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(n_categories) <span style="color: #666666">+</span> <span style="color: #666666">0.01</span>
probabilities <span style="color: #666666">=</span> feed_forward(X)
<span style="color: #008000">print</span>(probabilities)
predictions <span style="color: #666666">=</span> predict(X)
<span style="color: #008000">print</span>(predictions)
</pre></div>
<p>
Not an impressive result. Let us now add the full network with the back-propagation algorithm discussed above.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-full-network-for-the-various-gates">The full Network for the Various Gates </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week41/ipynb/ipynb-week41-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week41/ipynb/week41.ipynb
+82 -4
View File
@@ -1473,8 +1473,11 @@
"\"\"\"\n",
"Simple code that tests XOR, OR and AND gates with linear regression\n",
"\"\"\"\n",
"\n",
"# import necessary packages\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from sklearn import datasets\n",
"\n",
"# Design matrix\n",
"X = np.array([ [1, 0, 0], [1, 0, 1], [1, 1, 0],[1, 1, 1]],dtype=np.float64)\n",
"\n",
@@ -1483,10 +1486,85 @@
"# The OR gate \n",
"yOR = np.array( [ 0, 1 ,1, 1])\n",
"# The AND gate \n",
"yAND = np.array( [ 0, 0 ,0, 1])\n",
"yAND = np.array( [ 0, 0 ,0, 1])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Then the first Feed Forward pass"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"\n",
"#print(f\"The values of theta for the AND gate:{ThetaAND}\")\n",
"#print(f\"The linear regression prediction for the AND gate:{X @ ThetaAND}\")"
"\n",
"def sigmoid(x):\n",
" return 1/(1 + np.exp(-x))\n",
"\n",
"def feed_forward(X):\n",
" # weighted sum of inputs to the hidden layer\n",
" z_h = np.matmul(X, hidden_weights) + hidden_bias\n",
" # activation in the hidden layer\n",
" a_h = sigmoid(z_h)\n",
" \n",
" # weighted sum of inputs to the output layer\n",
" z_o = np.matmul(a_h, output_weights) + output_bias\n",
" # softmax output\n",
" # axis 0 holds each input and axis 1 the probabilities of each category\n",
" probabilities = sigmoid(z_o)\n",
" return probabilities\n",
"\n",
"# we obtain a prediction by taking the class with the highest likelihood\n",
"def predict(X):\n",
" probabilities = feed_forward(X)\n",
" return np.argmax(probabilities, axis=1)\n",
"\n",
"\n",
"\n",
"# ensure the same random numbers appear every time\n",
"np.random.seed(0)\n",
"\n",
"\n",
"# Defining the neural network\n",
"n_inputs, n_features = X.shape\n",
"n_hidden_neurons = 2\n",
"n_categories = 1\n",
"n_features = 2\n",
"\n",
"# we make the weights normally distributed using numpy.random.randn\n",
"\n",
"# weights and bias in the hidden layer\n",
"hidden_weights = np.random.randn(n_features, n_hidden_neurons)\n",
"hidden_bias = np.zeros(n_hidden_neurons) + 0.01\n",
"\n",
"# weights and bias in the output layer\n",
"output_weights = np.random.randn(n_hidden_neurons, n_categories)\n",
"output_bias = np.zeros(n_categories) + 0.01\n",
"\n",
"probabilities = feed_forward(X)\n",
"print(probabilities)\n",
"\n",
"\n",
"predictions = predict(X)\n",
"print(predictions)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Not an impressive result. Let us now add the full network with the back-propagation algorithm discussed above.\n",
"\n",
"## The full Network for the Various Gates"
]
},
{
doc/src/week41/nn.py
+67
View File
@@ -0,0 +1,67 @@
"""
Simple code that tests XOR, OR and AND gates with linear regression
"""
# import necessary packages
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
def sigmoid(x):
return 1/(1 + np.exp(-x))
def feed_forward(X):
# weighted sum of inputs to the hidden layer
z_h = np.matmul(X, hidden_weights) + hidden_bias
# activation in the hidden layer
a_h = sigmoid(z_h)
# weighted sum of inputs to the output layer
z_o = np.matmul(a_h, output_weights) + output_bias
# softmax output
# axis 0 holds each input and axis 1 the probabilities of each category
probabilities = sigmoid(z_o)
return probabilities
# we obtain a prediction by taking the class with the highest likelihood
def predict(X):
probabilities = feed_forward(X)
return np.argmax(probabilities, axis=1)
# ensure the same random numbers appear every time
np.random.seed(0)
# Design matrix
X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)
# The XOR gate
yXOR = np.array( [ 0, 1 ,1, 0])
# The OR gate
yOR = np.array( [ 0, 1 ,1, 1])
# The AND gate
yAND = np.array( [ 0, 0 ,0, 1])
# Defining the neural network
n_inputs, n_features = X.shape
n_hidden_neurons = 2
n_categories = 1
n_features = 2
# we make the weights normally distributed using numpy.random.randn
# weights and bias in the hidden layer
hidden_weights = np.random.randn(n_features, n_hidden_neurons)
hidden_bias = np.zeros(n_hidden_neurons) + 0.01
# weights and bias in the output layer
output_weights = np.random.randn(n_hidden_neurons, n_categories)
output_bias = np.zeros(n_categories) + 0.01
probabilities = feed_forward(X)
print(probabilities)
predictions = predict(X)
print(predictions)
doc/src/week41/week41.do.txt
+68 -4
View File
@@ -1131,8 +1131,11 @@ We define first our design matrix and the various input vectors.
"""
Simple code that tests XOR, OR and AND gates with linear regression
"""
# import necessary packages
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
# Design matrix
X = np.array([ [1, 0, 0], [1, 0, 1], [1, 1, 0],[1, 1, 1]],dtype=np.float64)
@@ -1142,12 +1145,73 @@ yXOR = np.array( [ 0, 1 ,1, 0])
yOR = np.array( [ 0, 1 ,1, 1])
# The AND gate
yAND = np.array( [ 0, 0 ,0, 1])
#print(f"The values of theta for the AND gate:{ThetaAND}")
#print(f"The linear regression prediction for the AND gate:{X @ ThetaAND}")
!ec
!split
===== Then the first Feed Forward pass =====
!bc pycod
def sigmoid(x):
return 1/(1 + np.exp(-x))
def feed_forward(X):
# weighted sum of inputs to the hidden layer
z_h = np.matmul(X, hidden_weights) + hidden_bias
# activation in the hidden layer
a_h = sigmoid(z_h)
# weighted sum of inputs to the output layer
z_o = np.matmul(a_h, output_weights) + output_bias
# softmax output
# axis 0 holds each input and axis 1 the probabilities of each category
probabilities = sigmoid(z_o)
return probabilities
# we obtain a prediction by taking the class with the highest likelihood
def predict(X):
probabilities = feed_forward(X)
return np.argmax(probabilities, axis=1)
# ensure the same random numbers appear every time
np.random.seed(0)
# Defining the neural network
n_inputs, n_features = X.shape
n_hidden_neurons = 2
n_categories = 1
n_features = 2
# we make the weights normally distributed using numpy.random.randn
# weights and bias in the hidden layer
hidden_weights = np.random.randn(n_features, n_hidden_neurons)
hidden_bias = np.zeros(n_hidden_neurons) + 0.01
# weights and bias in the output layer
output_weights = np.random.randn(n_hidden_neurons, n_categories)
output_bias = np.zeros(n_categories) + 0.01
probabilities = feed_forward(X)
print(probabilities)
predictions = predict(X)
print(predictions)
!ec
Not an impressive result. Let us now add the full network with the back-propagation algorithm discussed above.
!split
===== The full Network for the Various Gates =====
!bc pycod
!ec
!split
===== Building neural networks in Tensorflow and Keras =====