lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week41

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-10-13 16:00:24 +02:00
parent b45f04756e
commit 027fe32f90
8 changed files with 798 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week41/html/week41-bs.html
+67 -40
View File
@@ -107,6 +107,19 @@ Automatically generated HTML file from DocOnce source
None,
'scikit-learn-implementation'),
('Visualization', 2, None, 'visualization'),
('Testing our code for the XOR, OR and AND gates',
2,
None,
'testing-our-code-for-the-xor-or-and-and-gates'),
('The AND and XOR Gates', 2, None, 'the-and-and-xor-gates'),
('Representing the Data Sets',
2,
None,
'representing-the-data-sets'),
('Setting up the Neural Network',
2,
None,
'setting-up-the-neural-network'),
('Building neural networks in Tensorflow and Keras',
2,
None,
@@ -162,6 +175,14 @@ Automatically generated HTML file from DocOnce source
2,
None,
'limitations-of-supervised-learning-with-deep-networks'),
('Overarching Views, a personal note',
2,
None,
'overarching-views-a-personal-note'),
('From a Spherical Cow to a real one',
2,
None,
'from-a-spherical-cow-to-a-real-one'),
('Convolutional Neural Networks (recognizing images)',
2,
None,
@@ -244,7 +265,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week41-bs006.html#example-binary-classification-problem" style="font-size: 80%;">Example: binary classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs007.html#the-softmax-function" style="font-size: 80%;">The Softmax function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs008.html#developing-a-code-for-doing-neural-networks-with-back-propagation" style="font-size: 80%;">Developing a code for doing neural networks with back propagation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs030.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs034.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs010.html#train-and-test-datasets" style="font-size: 80%;">Train and test datasets</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs011.html#define-model-and-architecture" style="font-size: 80%;">Define model and architecture</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs012.html#layers" style="font-size: 80%;">Layers</a></li>
@@ -262,43 +283,49 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week41-bs026.html#visualization" style="font-size: 80%;">Visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs025.html#scikit-learn-implementation" style="font-size: 80%;">scikit-learn implementation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs026.html#visualization" style="font-size: 80%;">Visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs027.html#building-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">Building neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs028.html#tensorflow" style="font-size: 80%;">Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs029.html#using-keras" style="font-size: 80%;">Using Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs030.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs031.html#the-breast-cancer-data-now-with-keras" style="font-size: 80%;">The Breast Cancer Data, now with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs032.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;">Fine-tuning neural network hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs033.html#hidden-layers" style="font-size: 80%;">Hidden layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs034.html#which-activation-function-should-i-use" style="font-size: 80%;">Which activation function should I use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs035.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;">Is the Logistic activation function (Sigmoid) our choice?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs036.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;">The derivative of the Logistic funtion</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs037.html#the-relu-function-family" style="font-size: 80%;">The RELU function family</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs038.html#which-activation-function-should-we-use" style="font-size: 80%;">Which activation function should we use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs039.html#more-on-activation-functions-output-layers" style="font-size: 80%;">More on activation functions, output layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs040.html#batch-normalization" style="font-size: 80%;">Batch Normalization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs041.html#dropout" style="font-size: 80%;">Dropout</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs042.html#gradient-clipping" style="font-size: 80%;">Gradient Clipping</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs043.html#a-very-nice-website-on-neural-networks" style="font-size: 80%;">A very nice website on Neural Networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs044.html#a-top-down-perspective-on-neural-networks" style="font-size: 80%;">A top-down perspective on Neural networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs045.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;">Limitations of supervised learning with deep networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs046.html#convolutional-neural-networks-recognizing-images" style="font-size: 80%;">Convolutional Neural Networks (recognizing images)</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs047.html#regular-nns-don-t-scale-well-to-full-images" style="font-size: 80%;">Regular NNs don’t scale well to full images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs048.html#3d-volumes-of-neurons" style="font-size: 80%;">3D volumes of neurons</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs049.html#layers-used-to-build-cnns" style="font-size: 80%;">Layers used to build CNNs</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs050.html#transforming-images" style="font-size: 80%;">Transforming images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs051.html#cnns-in-brief" style="font-size: 80%;">CNNs in brief</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs052.html#cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs053.html#setting-it-up" style="font-size: 80%;">Setting it up</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs054.html#the-mnist-dataset-again" style="font-size: 80%;">The MNIST dataset again</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs055.html#strong-correlations" style="font-size: 80%;">Strong correlations</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs056.html#layers-of-a-cnn" style="font-size: 80%;">Layers of a CNN</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs057.html#systematic-reduction" style="font-size: 80%;">Systematic reduction</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs058.html#prerequisites-collect-and-pre-process-data" style="font-size: 80%;">Prerequisites: Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs059.html#importing-keras-and-tensorflow" style="font-size: 80%;">Importing Keras and Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs060.html#running-with-keras" style="font-size: 80%;">Running with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs061.html#final-part" style="font-size: 80%;">Final part</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs062.html#final-visualization" style="font-size: 80%;">Final visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs063.html#fun-links" style="font-size: 80%;">Fun links</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs027.html#testing-our-code-for-the-xor-or-and-and-gates" style="font-size: 80%;">Testing our code for the XOR, OR and AND gates</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs028.html#the-and-and-xor-gates" style="font-size: 80%;">The AND and XOR Gates</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs029.html#representing-the-data-sets" style="font-size: 80%;">Representing the Data Sets</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs030.html#setting-up-the-neural-network" style="font-size: 80%;">Setting up the Neural Network</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs031.html#building-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">Building neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs032.html#tensorflow" style="font-size: 80%;">Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs033.html#using-keras" style="font-size: 80%;">Using Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs034.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs035.html#the-breast-cancer-data-now-with-keras" style="font-size: 80%;">The Breast Cancer Data, now with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs036.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;">Fine-tuning neural network hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs037.html#hidden-layers" style="font-size: 80%;">Hidden layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs038.html#which-activation-function-should-i-use" style="font-size: 80%;">Which activation function should I use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs039.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;">Is the Logistic activation function (Sigmoid) our choice?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs040.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;">The derivative of the Logistic funtion</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs041.html#the-relu-function-family" style="font-size: 80%;">The RELU function family</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs042.html#which-activation-function-should-we-use" style="font-size: 80%;">Which activation function should we use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs043.html#more-on-activation-functions-output-layers" style="font-size: 80%;">More on activation functions, output layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs044.html#batch-normalization" style="font-size: 80%;">Batch Normalization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs045.html#dropout" style="font-size: 80%;">Dropout</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs046.html#gradient-clipping" style="font-size: 80%;">Gradient Clipping</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs047.html#a-very-nice-website-on-neural-networks" style="font-size: 80%;">A very nice website on Neural Networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs048.html#a-top-down-perspective-on-neural-networks" style="font-size: 80%;">A top-down perspective on Neural networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs049.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;">Limitations of supervised learning with deep networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs050.html#overarching-views-a-personal-note" style="font-size: 80%;">Overarching Views, a personal note</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs051.html#from-a-spherical-cow-to-a-real-one" style="font-size: 80%;">From a Spherical Cow to a real one</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs052.html#convolutional-neural-networks-recognizing-images" style="font-size: 80%;">Convolutional Neural Networks (recognizing images)</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs053.html#regular-nns-don-t-scale-well-to-full-images" style="font-size: 80%;">Regular NNs don’t scale well to full images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs054.html#3d-volumes-of-neurons" style="font-size: 80%;">3D volumes of neurons</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs055.html#layers-used-to-build-cnns" style="font-size: 80%;">Layers used to build CNNs</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs056.html#transforming-images" style="font-size: 80%;">Transforming images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs057.html#cnns-in-brief" style="font-size: 80%;">CNNs in brief</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs058.html#cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs059.html#setting-it-up" style="font-size: 80%;">Setting it up</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs060.html#the-mnist-dataset-again" style="font-size: 80%;">The MNIST dataset again</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs061.html#strong-correlations" style="font-size: 80%;">Strong correlations</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs062.html#layers-of-a-cnn" style="font-size: 80%;">Layers of a CNN</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs063.html#systematic-reduction" style="font-size: 80%;">Systematic reduction</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs064.html#prerequisites-collect-and-pre-process-data" style="font-size: 80%;">Prerequisites: Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs065.html#importing-keras-and-tensorflow" style="font-size: 80%;">Importing Keras and Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs066.html#running-with-keras" style="font-size: 80%;">Running with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs067.html#final-part" style="font-size: 80%;">Final part</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs068.html#final-visualization" style="font-size: 80%;">Final visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs069.html#fun-links" style="font-size: 80%;">Fun links</a></li>
</ul>
</li>
@@ -333,7 +360,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 11, 2021</h4></center> <!-- date -->
<center><h4>Oct 13, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -357,7 +384,7 @@ MathJax.Hub.Config({
<li><a href="._week41-bs008.html">9</a></li>
<li><a href="._week41-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week41-bs063.html">64</a></li>
<li><a href="._week41-bs069.html">70</a></li>
<li><a href="._week41-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week41/html/week41-reveal.html
+149 -1
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Oct 11, 2021</h4></center> <!-- date -->
<center><h4>Oct 13, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -1468,6 +1468,115 @@ plt.show()
</section>
<section>
<h2 id="testing-our-code-for-the-xor-or-and-and-gates">Testing our code for the XOR, OR and AND gates </h2>
<p>
Last week we discussed three different types of gates, the so-called
XOR, the OR and the AND gates. Their inputs and outputs can be
summarized using the following tables, first for the OR gate with
inputs \( x_1 \) and \( x_2 \) and outputs \( y \):
<p>
<table border="1">
<thead>
<tr><td align="center">\( x_1 \)</td> <td align="center">\( x_2 \)</td> <td align="center">\( y \)</td> </tr>
</thead>
<tbody>
<tr><td align="center"> 0 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 0 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 0 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
</tbody>
</table>
</section>
<section>
<h2 id="the-and-and-xor-gates">The AND and XOR Gates </h2>
<p>
The <b>AND</b> gate is defined as
<p>
<table border="1">
<thead>
<tr><td align="center">\( x_1 \)</td> <td align="center">\( x_2 \)</td> <td align="center">\( y \)</td> </tr>
</thead>
<tbody>
<tr><td align="center"> 0 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 0 </td> <td align="center"> 1 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
</tbody>
</table>
<p>
And finally we have the <b>XOR</b> gate
<p>
<table border="1">
<thead>
<tr><td align="center">\( x_1 \)</td> <td align="center">\( x_2 \)</td> <td align="center">\( y \)</td> </tr>
</thead>
<tbody>
<tr><td align="center"> 0 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 0 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 0 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 1 </td> <td align="center"> 0 </td> </tr>
</tbody>
</table>
</section>
<section>
<h2 id="representing-the-data-sets">Representing the Data Sets </h2>
<p>
Our design matrix is defined by the input values \( x_1 \) and \( x_2 \). Since we have four possible outputs, our design matrix reads
<p>&nbsp;<br>
$$
\boldsymbol{X}=\begin{bmatrix} 0 & 0 \\
0 & 1 \\
1 & 0 \\
1 & 1 \end{bmatrix},
$$
<p>&nbsp;<br>
while the vector of outputs is \( \boldsymbol{y}^T=[0,1,1,0] \) for the XOR gate, \( \boldsymbol{y}^T=[0,0,0,1] \) for the AND gate and \( \boldsymbol{y}^T=[0,1,1,1] \) for the OR gate.
</section>
<section>
<h2 id="setting-up-the-neural-network">Setting up the Neural Network </h2>
<p>
We define first our design matrix and the various input vectors.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #CD5555">Simple code that tests XOR, OR and AND gates with linear regression</span>
<span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #228B22"># Design matrix</span>
X = np.array([ [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>],[<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>]],dtype=np.float64)
<span style="color: #228B22"># The XOR gate </span>
yXOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>])
<span style="color: #228B22"># The OR gate </span>
yOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>])
<span style="color: #228B22"># The AND gate </span>
yAND = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span> ,<span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>])
<span style="color: #228B22">#print(f&quot;The values of theta for the AND gate:{ThetaAND}&quot;)</span>
<span style="color: #228B22">#print(f&quot;The linear regression prediction for the AND gate:{X @ ThetaAND}&quot;)</span>
</pre></div>
</section>
<section>
<h2 id="building-neural-networks-in-tensorflow-and-keras">Building neural networks in Tensorflow and Keras </h2>
@@ -2226,6 +2335,45 @@ Some of these remarks are particular to DNNs, others are shared by all supervise
</section>
<section>
<h2 id="overarching-views-a-personal-note">Overarching Views, a personal note </h2>
<p>
The author of these lecture notes has an overarching take on many of
the machine learning algorithms we discuss here.
<p>
If we wish to understand complex systems, we need to find some
effective degrees of freedom or features that we find essential,
simply in order to reduce the complexity of the systems we are
studying. This leads, in one way or the other to dimensionality
reductions. Most of the Machine Learning methods we encounter deal
with this, whether we opt for a principal component analysis, or
clustering, or convolutional neural networks, or Ridge or Lasso
regression or random forest, yes, perhaps most machine learning
methods at large.
<p>
For neural networks and our previous discussion, we have seen that we
in essence end up with matrix-matrix and matrix-vector
multiplications. In all cases, our matrices are dense ones, and the
more data we deal with the larger the dimensionalities of the matrices
and vectors. How can we reduce such dimensionalities? One possible
answer is offered by <b>convolutional neural networks</b> (CNN), as
discussed below. The figure here shows a typical situation of the
reduction of information in an image and is typical of what CNNs
actually end up doing.
</section>
<section>
<h2 id="from-a-spherical-cow-to-a-real-one">From a Spherical Cow to a real one </h2>
<p>
<br /><br /><center><p><img src="figslides/ImageReduction.png" align="bottom" width=500></p></center><br /><br />
</section>
<section>
<h2 id="convolutional-neural-networks-recognizing-images">Convolutional Neural Networks (recognizing images) </h2>
doc/pub/week41/html/week41-solarized.html
+165 -1
View File
@@ -127,6 +127,19 @@ div { text-align: justify; text-justify: inter-word; }
None,
'scikit-learn-implementation'),
('Visualization', 2, None, 'visualization'),
('Testing our code for the XOR, OR and AND gates',
2,
None,
'testing-our-code-for-the-xor-or-and-and-gates'),
('The AND and XOR Gates', 2, None, 'the-and-and-xor-gates'),
('Representing the Data Sets',
2,
None,
'representing-the-data-sets'),
('Setting up the Neural Network',
2,
None,
'setting-up-the-neural-network'),
('Building neural networks in Tensorflow and Keras',
2,
None,
@@ -182,6 +195,14 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'limitations-of-supervised-learning-with-deep-networks'),
('Overarching Views, a personal note',
2,
None,
'overarching-views-a-personal-note'),
('From a Spherical Cow to a real one',
2,
None,
'from-a-spherical-cow-to-a-real-one'),
('Convolutional Neural Networks (recognizing images)',
2,
None,
@@ -260,7 +281,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 11, 2021</h4></center> <!-- date -->
<center><h4>Oct 13, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1457,6 +1478,110 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="testing-our-code-for-the-xor-or-and-and-gates">Testing our code for the XOR, OR and AND gates </h2>
<p>
Last week we discussed three different types of gates, the so-called
XOR, the OR and the AND gates. Their inputs and outputs can be
summarized using the following tables, first for the OR gate with
inputs \( x_1 \) and \( x_2 \) and outputs \( y \):
<p>
<table border="1">
<thead>
<tr><td align="center">\( x_1 \)</td> <td align="center">\( x_2 \)</td> <td align="center">\( y \)</td> </tr>
</thead>
<tbody>
<tr><td align="center"> 0 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 0 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 0 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
</tbody>
</table>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-and-and-xor-gates">The AND and XOR Gates </h2>
<p>
The <b>AND</b> gate is defined as
<p>
<table border="1">
<thead>
<tr><td align="center">\( x_1 \)</td> <td align="center">\( x_2 \)</td> <td align="center">\( y \)</td> </tr>
</thead>
<tbody>
<tr><td align="center"> 0 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 0 </td> <td align="center"> 1 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
</tbody>
</table>
<p>
And finally we have the <b>XOR</b> gate
<p>
<table border="1">
<thead>
<tr><td align="center">\( x_1 \)</td> <td align="center">\( x_2 \)</td> <td align="center">\( y \)</td> </tr>
</thead>
<tbody>
<tr><td align="center"> 0 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 0 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 0 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 1 </td> <td align="center"> 0 </td> </tr>
</tbody>
</table>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="representing-the-data-sets">Representing the Data Sets </h2>
<p>
Our design matrix is defined by the input values \( x_1 \) and \( x_2 \). Since we have four possible outputs, our design matrix reads
$$
\boldsymbol{X}=\begin{bmatrix} 0 & 0 \\
0 & 1 \\
1 & 0 \\
1 & 1 \end{bmatrix},
$$
while the vector of outputs is \( \boldsymbol{y}^T=[0,1,1,0] \) for the XOR gate, \( \boldsymbol{y}^T=[0,0,0,1] \) for the AND gate and \( \boldsymbol{y}^T=[0,1,1,1] \) for the OR gate.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-neural-network">Setting up the Neural Network </h2>
<p>
We define first our design matrix and the various input vectors.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #CD5555">Simple code that tests XOR, OR and AND gates with linear regression</span>
<span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #228B22"># Design matrix</span>
X = np.array([ [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>],[<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>]],dtype=np.float64)
<span style="color: #228B22"># The XOR gate </span>
yXOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>])
<span style="color: #228B22"># The OR gate </span>
yOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>])
<span style="color: #228B22"># The AND gate </span>
yAND = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span> ,<span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>])
<span style="color: #228B22">#print(f&quot;The values of theta for the AND gate:{ThetaAND}&quot;)</span>
<span style="color: #228B22">#print(f&quot;The linear regression prediction for the AND gate:{X @ ThetaAND}&quot;)</span>
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="building-neural-networks-in-tensorflow-and-keras">Building neural networks in Tensorflow and Keras </h2>
<p>
@@ -2205,6 +2330,45 @@ Some of these remarks are particular to DNNs, others are shared by all supervise
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="overarching-views-a-personal-note">Overarching Views, a personal note </h2>
<p>
The author of these lecture notes has an overarching take on many of
the machine learning algorithms we discuss here.
<p>
If we wish to understand complex systems, we need to find some
effective degrees of freedom or features that we find essential,
simply in order to reduce the complexity of the systems we are
studying. This leads, in one way or the other to dimensionality
reductions. Most of the Machine Learning methods we encounter deal
with this, whether we opt for a principal component analysis, or
clustering, or convolutional neural networks, or Ridge or Lasso
regression or random forest, yes, perhaps most machine learning
methods at large.
<p>
For neural networks and our previous discussion, we have seen that we
in essence end up with matrix-matrix and matrix-vector
multiplications. In all cases, our matrices are dense ones, and the
more data we deal with the larger the dimensionalities of the matrices
and vectors. How can we reduce such dimensionalities? One possible
answer is offered by <b>convolutional neural networks</b> (CNN), as
discussed below. The figure here shows a typical situation of the
reduction of information in an image and is typical of what CNNs
actually end up doing.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="from-a-spherical-cow-to-a-real-one">From a Spherical Cow to a real one </h2>
<p>
<br /><br /><center><p><img src="figslides/ImageReduction.png" align="bottom" width=500></p></center><br /><br />
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="convolutional-neural-networks-recognizing-images">Convolutional Neural Networks (recognizing images) </h2>
<p>
doc/pub/week41/html/week41.html
+165 -1
View File
@@ -132,6 +132,19 @@ div { text-align: justify; text-justify: inter-word; }
None,
'scikit-learn-implementation'),
('Visualization', 2, None, 'visualization'),
('Testing our code for the XOR, OR and AND gates',
2,
None,
'testing-our-code-for-the-xor-or-and-and-gates'),
('The AND and XOR Gates', 2, None, 'the-and-and-xor-gates'),
('Representing the Data Sets',
2,
None,
'representing-the-data-sets'),
('Setting up the Neural Network',
2,
None,
'setting-up-the-neural-network'),
('Building neural networks in Tensorflow and Keras',
2,
None,
@@ -187,6 +200,14 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'limitations-of-supervised-learning-with-deep-networks'),
('Overarching Views, a personal note',
2,
None,
'overarching-views-a-personal-note'),
('From a Spherical Cow to a real one',
2,
None,
'from-a-spherical-cow-to-a-real-one'),
('Convolutional Neural Networks (recognizing images)',
2,
None,
@@ -265,7 +286,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 11, 2021</h4></center> <!-- date -->
<center><h4>Oct 13, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1462,6 +1483,110 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="testing-our-code-for-the-xor-or-and-and-gates">Testing our code for the XOR, OR and AND gates </h2>
<p>
Last week we discussed three different types of gates, the so-called
XOR, the OR and the AND gates. Their inputs and outputs can be
summarized using the following tables, first for the OR gate with
inputs \( x_1 \) and \( x_2 \) and outputs \( y \):
<p>
<table border="1">
<thead>
<tr><td align="center">\( x_1 \)</td> <td align="center">\( x_2 \)</td> <td align="center">\( y \)</td> </tr>
</thead>
<tbody>
<tr><td align="center"> 0 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 0 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 0 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
</tbody>
</table>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-and-and-xor-gates">The AND and XOR Gates </h2>
<p>
The <b>AND</b> gate is defined as
<p>
<table border="1">
<thead>
<tr><td align="center">\( x_1 \)</td> <td align="center">\( x_2 \)</td> <td align="center">\( y \)</td> </tr>
</thead>
<tbody>
<tr><td align="center"> 0 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 0 </td> <td align="center"> 1 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
</tbody>
</table>
<p>
And finally we have the <b>XOR</b> gate
<p>
<table border="1">
<thead>
<tr><td align="center">\( x_1 \)</td> <td align="center">\( x_2 \)</td> <td align="center">\( y \)</td> </tr>
</thead>
<tbody>
<tr><td align="center"> 0 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 0 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 0 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 1 </td> <td align="center"> 0 </td> </tr>
</tbody>
</table>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="representing-the-data-sets">Representing the Data Sets </h2>
<p>
Our design matrix is defined by the input values \( x_1 \) and \( x_2 \). Since we have four possible outputs, our design matrix reads
$$
\boldsymbol{X}=\begin{bmatrix} 0 & 0 \\
0 & 1 \\
1 & 0 \\
1 & 1 \end{bmatrix},
$$
while the vector of outputs is \( \boldsymbol{y}^T=[0,1,1,0] \) for the XOR gate, \( \boldsymbol{y}^T=[0,0,0,1] \) for the AND gate and \( \boldsymbol{y}^T=[0,1,1,1] \) for the OR gate.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-neural-network">Setting up the Neural Network </h2>
<p>
We define first our design matrix and the various input vectors.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic">Simple code that tests XOR, OR and AND gates with linear regression</span>
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #408080; font-style: italic"># Design matrix</span>
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([ [<span style="color: #666666">1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">0</span>], [<span style="color: #666666">1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">1</span>], [<span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">0</span>],[<span style="color: #666666">1</span>, <span style="color: #666666">1</span>, <span style="color: #666666">1</span>]],dtype<span style="color: #666666">=</span>np<span style="color: #666666">.</span>float64)
<span style="color: #408080; font-style: italic"># The XOR gate </span>
yXOR <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array( [ <span style="color: #666666">0</span>, <span style="color: #666666">1</span> ,<span style="color: #666666">1</span>, <span style="color: #666666">0</span>])
<span style="color: #408080; font-style: italic"># The OR gate </span>
yOR <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array( [ <span style="color: #666666">0</span>, <span style="color: #666666">1</span> ,<span style="color: #666666">1</span>, <span style="color: #666666">1</span>])
<span style="color: #408080; font-style: italic"># The AND gate </span>
yAND <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array( [ <span style="color: #666666">0</span>, <span style="color: #666666">0</span> ,<span style="color: #666666">0</span>, <span style="color: #666666">1</span>])
<span style="color: #408080; font-style: italic">#print(f&quot;The values of theta for the AND gate:{ThetaAND}&quot;)</span>
<span style="color: #408080; font-style: italic">#print(f&quot;The linear regression prediction for the AND gate:{X @ ThetaAND}&quot;)</span>
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="building-neural-networks-in-tensorflow-and-keras">Building neural networks in Tensorflow and Keras </h2>
<p>
@@ -2210,6 +2335,45 @@ Some of these remarks are particular to DNNs, others are shared by all supervise
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="overarching-views-a-personal-note">Overarching Views, a personal note </h2>
<p>
The author of these lecture notes has an overarching take on many of
the machine learning algorithms we discuss here.
<p>
If we wish to understand complex systems, we need to find some
effective degrees of freedom or features that we find essential,
simply in order to reduce the complexity of the systems we are
studying. This leads, in one way or the other to dimensionality
reductions. Most of the Machine Learning methods we encounter deal
with this, whether we opt for a principal component analysis, or
clustering, or convolutional neural networks, or Ridge or Lasso
regression or random forest, yes, perhaps most machine learning
methods at large.
<p>
For neural networks and our previous discussion, we have seen that we
in essence end up with matrix-matrix and matrix-vector
multiplications. In all cases, our matrices are dense ones, and the
more data we deal with the larger the dimensionalities of the matrices
and vectors. How can we reduce such dimensionalities? One possible
answer is offered by <b>convolutional neural networks</b> (CNN), as
discussed below. The figure here shows a typical situation of the
reduction of information in an image and is typical of what CNNs
actually end up doing.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="from-a-spherical-cow-to-a-real-one">From a Spherical Cow to a real one </h2>
<p>
<br /><br /><center><p><img src="figslides/ImageReduction.png" align="bottom" width=500></p></center><br /><br />
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="convolutional-neural-networks-recognizing-images">Convolutional Neural Networks (recognizing images) </h2>
<p>
doc/pub/week41/ipynb/ipynb-week41-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week41/ipynb/week41.ipynb
+138 -1
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Oct 11, 2021**\n",
"Date: **Oct 13, 2021**\n",
"\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -1383,6 +1383,112 @@
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Testing our code for the XOR, OR and AND gates\n",
"\n",
"Last week we discussed three different types of gates, the so-called\n",
"XOR, the OR and the AND gates. Their inputs and outputs can be\n",
"summarized using the following tables, first for the OR gate with\n",
"inputs $x_1$ and $x_2$ and outputs $y$:\n",
"\n",
"<table border=\"1\">\n",
"<thead>\n",
"<tr><th align=\"center\">$x_1$</th> <th align=\"center\">$x_2$</th> <th align=\"center\">$y$</th> </tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> </tr>\n",
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
"</tbody>\n",
"</table>\n",
"## The AND and XOR Gates\n",
"\n",
"The **AND** gate is defined as\n",
"\n",
"<table border=\"1\">\n",
"<thead>\n",
"<tr><th align=\"center\">$x_1$</th> <th align=\"center\">$x_2$</th> <th align=\"center\">$y$</th> </tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> </tr>\n",
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
"</tbody>\n",
"</table>\n",
"And finally we have the **XOR** gate\n",
"\n",
"<table border=\"1\">\n",
"<thead>\n",
"<tr><th align=\"center\">$x_1$</th> <th align=\"center\">$x_2$</th> <th align=\"center\">$y$</th> </tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> </tr>\n",
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> </tr>\n",
"</tbody>\n",
"</table>\n",
"## Representing the Data Sets\n",
"\n",
"Our design matrix is defined by the input values $x_1$ and $x_2$. Since we have four possible outputs, our design matrix reads"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
" 0 & 1 \\\\\n",
"\t\t 1 & 0 \\\\\n",
"\t\t 1 & 1 \\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n",
"\n",
"## Setting up the Neural Network\n",
"\n",
"We define first our design matrix and the various input vectors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"\"\"\"\n",
"Simple code that tests XOR, OR and AND gates with linear regression\n",
"\"\"\"\n",
"\n",
"import numpy as np\n",
"# Design matrix\n",
"X = np.array([ [1, 0, 0], [1, 0, 1], [1, 1, 0],[1, 1, 1]],dtype=np.float64)\n",
"\n",
"# The XOR gate \n",
"yXOR = np.array( [ 0, 1 ,1, 0])\n",
"# The OR gate \n",
"yOR = np.array( [ 0, 1 ,1, 1])\n",
"# The AND gate \n",
"yAND = np.array( [ 0, 0 ,0, 1])\n",
"\n",
"#print(f\"The values of theta for the AND gate:{ThetaAND}\")\n",
"#print(f\"The linear regression prediction for the AND gate:{X @ ThetaAND}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
@@ -2171,6 +2277,37 @@
"\n",
"\n",
"\n",
"## Overarching Views, a personal note\n",
"\n",
"The author of these lecture notes has an overarching take on many of\n",
"the machine learning algorithms we discuss here. \n",
"\n",
"\n",
"If we wish to understand complex systems, we need to find some\n",
"effective degrees of freedom or features that we find essential,\n",
"simply in order to reduce the complexity of the systems we are\n",
"studying. This leads, in one way or the other to dimensionality\n",
"reductions. Most of the Machine Learning methods we encounter deal\n",
"with this, whether we opt for a principal component analysis, or\n",
"clustering, or convolutional neural networks, or Ridge or Lasso\n",
"regression or random forest, yes, perhaps most machine learning\n",
"methods at large.\n",
"\n",
"For neural networks and our previous discussion, we have seen that we\n",
"in essence end up with matrix-matrix and matrix-vector\n",
"multiplications. In all cases, our matrices are dense ones, and the\n",
"more data we deal with the larger the dimensionalities of the matrices\n",
"and vectors. How can we reduce such dimensionalities? One possible\n",
"answer is offered by **convolutional neural networks** (CNN), as\n",
"discussed below. The figure here shows a typical situation of the\n",
"reduction of information in an image and is typical of what CNNs\n",
"actually end up doing.\n",
"\n",
"## From a Spherical Cow to a real one\n",
"\n",
"<!-- dom:FIGURE: [figslides/ImageReduction.png, width=500 frac=0.6] -->\n",
"<!-- begin figure -->\n",
"<img src=\"figslides/ImageReduction.png\" width=500><p style=\"font-size: 0.9em\"><i>Figure 1: </i></p><!-- end figure -->\n",
"\n",
"\n",
"\n",
doc/src/week41/figslides/ImageReduction.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 439 KiB

doc/src/week41/week41.do.txt
+114
View File
@@ -1067,6 +1067,88 @@ plt.show()
!ec
!split
===== Testing our code for the XOR, OR and AND gates =====
Last week we discussed three different types of gates, the so-called
XOR, the OR and the AND gates. Their inputs and outputs can be
summarized using the following tables, first for the OR gate with
inputs $x_1$ and $x_2$ and outputs $y$:
|---------------------|
| $x_1$ | $x_2$ | $y$ |
|---------------------|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 1 |
|---------------------|
!split
===== The AND and XOR Gates =====
The _AND_ gate is defined as
|---------------------|
| $x_1$ | $x_2$ | $y$ |
|---------------------|
| 0 | 0 | 0 |
| 0 | 1 | 0 |
| 1 | 0 | 0 |
| 1 | 1 | 1 |
|---------------------|
And finally we have the _XOR_ gate
|---------------------|
| $x_1$ | $x_2$ | $y$ |
|---------------------|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
|---------------------|
!split
===== Representing the Data Sets =====
Our design matrix is defined by the input values $x_1$ and $x_2$. Since we have four possible outputs, our design matrix reads
!bt
\bm{X}=\begin{bmatrix} 0 & 0 \\
0 & 1 \\
1 & 0 \\
1 & 1 \end{bmatrix},
!et
while the vector of outputs is $\bm{y}^T=[0,1,1,0]$ for the XOR gate, $\bm{y}^T=[0,0,0,1]$ for the AND gate and $\bm{y}^T=[0,1,1,1]$ for the OR gate.
!split
===== Setting up the Neural Network =====
We define first our design matrix and the various input vectors.
!bc pycod
"""
Simple code that tests XOR, OR and AND gates with linear regression
"""
import numpy as np
# Design matrix
X = np.array([ [1, 0, 0], [1, 0, 1], [1, 1, 0],[1, 1, 1]],dtype=np.float64)
# The XOR gate
yXOR = np.array( [ 0, 1 ,1, 0])
# The OR gate
yOR = np.array( [ 0, 1 ,1, 1])
# The AND gate
yAND = np.array( [ 0, 0 ,0, 1])
#print(f"The values of theta for the AND gate:{ThetaAND}")
#print(f"The linear regression prediction for the AND gate:{X @ ThetaAND}")
!ec
!split
===== Building neural networks in Tensorflow and Keras =====
@@ -1749,7 +1831,37 @@ Some of these remarks are particular to DNNs, others are shared by all supervise
!split
===== Overarching Views, a personal note =====
The author of these lecture notes has an overarching take on many of
the machine learning algorithms we discuss here.
If we wish to understand complex systems, we need to find some
effective degrees of freedom or features that we find essential,
simply in order to reduce the complexity of the systems we are
studying. This leads, in one way or the other to dimensionality
reductions. Most of the Machine Learning methods we encounter deal
with this, whether we opt for a principal component analysis, or
clustering, or convolutional neural networks, or Ridge or Lasso
regression or random forest, yes, perhaps most machine learning
methods at large.
For neural networks and our previous discussion, we have seen that we
in essence end up with matrix-matrix and matrix-vector
multiplications. In all cases, our matrices are dense ones, and the
more data we deal with the larger the dimensionalities of the matrices
and vectors. How can we reduce such dimensionalities? One possible
answer is offered by _convolutional neural networks_ (CNN), as
discussed below. The figure here shows a typical situation of the
reduction of information in an image and is typical of what CNNs
actually end up doing.
!split
===== From a Spherical Cow to a real one =====
FIGURE: [figslides/ImageReduction.png, width=500 frac=0.6]
!split
@@ -2148,3 +2260,5 @@ o "Abstract art using convolutional neural networks":"https://deepdreamgenerator