update on cnn

2019-10-04 05:50:18 +02:00
parent 812fbe454e
commit c2be6efa93
162 changed files with 5235 additions and 2420 deletions
@@ -279,7 +279,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Oct 3, 2019</h4></center> <!-- date -->
+<center><h4>Oct 4, 2019</h4></center> <!-- date -->
 <br>
 <p>

@@ -272,7 +272,7 @@ The scikit-learn dataset we will use consists of a selection of 1797 images of s

 <p>
 To feed data into a feed-forward neural network we need to represent
-the inputs as a feature matrix \( X = (n_{inputs}, n_{features}) \).  Each
+the inputs as a design/feature matrix \( X = (n_{inputs}, n_{features}) \).  Each
 row represents an <em>input</em>, in this case a handwritten digit, and
 each column represents a <em>feature</em>, in this case a pixel.  The
 correct answers, also known as <em>labels</em> or <em>targets</em> are
@@ -282,7 +282,7 @@ represented as a 1D array of integers
 <p>
 As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from
 measurements of height (in m)  
-and weight (in kg). If we have measurements of 5 people the feature matrix could be for example:  
+and weight (in kg). If we have measurements of 5 people the design/feature matrix could be for example:  

 $$ X = \begin{bmatrix}
 1.85 &amp; 81\\
@@ -300,7 +300,7 @@ $$ Y = (23.7, 22.2, 27.1, 17.5, 21.1) $$
 <p>
 Since each input image is a 2D matrix, we need to flatten the image
 (i.e. "unravel" the 2D matrix into a 1D array) to turn the data into a
-feature matrix. This means we lose all spatial information in the
+design/feature matrix. This means we lose all spatial information in the
 image, such as locality and translational invariance. More complicated
 architectures such as Convolutional Neural Networks can take advantage
 of such information, and are most commonly applied when analyzing
@@ -273,9 +273,9 @@ reduces <em>overfitting</em>.
 <p>
 We will measure the size of the weights using the so called <em>L2-norm</em>, meaning our cost function becomes:  

-$$ \nabla \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) \quad \rightarrow \quad
-\frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) + \lambda \lvert \lvert \hat{w} \rvert \rvert_2^2 
-= \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}(\theta) + \lambda \sum_{ij} w_{ij}^2,$$
+$$  \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \mathcal{L}_i(\theta) \quad \rightarrow \quad
+\frac{1}{N} \sum_{i=1}^N  \mathcal{L}_i(\theta) + \lambda \lvert \lvert \hat{w} \rvert \rvert_2^2 
+= \frac{1}{N} \sum_{i=1}^N  \mathcal{L}(\theta) + \lambda \sum_{ij} w_{ij}^2,$$

 <p>
 i.e. we sum up all the weights squared. The factor \( \lambda \) is known as a regularization parameter.
@@ -279,7 +279,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Oct 3, 2019</h4></center> <!-- date -->
+<center><h4>Oct 4, 2019</h4></center> <!-- date -->
 <br>
 <p>

@@ -148,7 +148,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>&nbsp;<br>
-<center><h4>Oct 3, 2019</h4></center> <!-- date -->
+<center><h4>Oct 4, 2019</h4></center> <!-- date -->
 <br>
 <p>

@@ -1445,7 +1445,7 @@ The scikit-learn dataset we will use consists of a selection of 1797 images of s

 <p>
 To feed data into a feed-forward neural network we need to represent
-the inputs as a feature matrix \( X = (n_{inputs}, n_{features}) \).  Each
+the inputs as a design/feature matrix \( X = (n_{inputs}, n_{features}) \).  Each
 row represents an <em>input</em>, in this case a handwritten digit, and
 each column represents a <em>feature</em>, in this case a pixel.  The
 correct answers, also known as <em>labels</em> or <em>targets</em> are
@@ -1455,7 +1455,7 @@ represented as a 1D array of integers
 <p>
 As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from
 measurements of height (in m)  
-and weight (in kg). If we have measurements of 5 people the feature matrix could be for example:
+and weight (in kg). If we have measurements of 5 people the design/feature matrix could be for example:

 <p>&nbsp;<br>
 $$ X = \begin{bmatrix}
@@ -1477,7 +1477,7 @@ $$ Y = (23.7, 22.2, 27.1, 17.5, 21.1) $$
 <p>
 Since each input image is a 2D matrix, we need to flatten the image
 (i.e. "unravel" the 2D matrix into a 1D array) to turn the data into a
-feature matrix. This means we lose all spatial information in the
+design/feature matrix. This means we lose all spatial information in the
 image, such as locality and translational invariance. More complicated
 architectures such as Convolutional Neural Networks can take advantage
 of such information, and are most commonly applied when analyzing
@@ -1944,9 +1944,9 @@ reduces <em>overfitting</em>.
 We will measure the size of the weights using the so called <em>L2-norm</em>, meaning our cost function becomes:

 <p>&nbsp;<br>
-$$ \nabla \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) \quad \rightarrow \quad
-\frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) + \lambda \lvert \lvert \hat{w} \rvert \rvert_2^2 
-= \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}(\theta) + \lambda \sum_{ij} w_{ij}^2,$$
+$$  \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \mathcal{L}_i(\theta) \quad \rightarrow \quad
+\frac{1}{N} \sum_{i=1}^N  \mathcal{L}_i(\theta) + \lambda \lvert \lvert \hat{w} \rvert \rvert_2^2 
+= \frac{1}{N} \sum_{i=1}^N  \mathcal{L}(\theta) + \lambda \sum_{ij} w_{ij}^2,$$
 <p>&nbsp;<br>

 <p>
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Oct 3, 2019</h4></center> <!-- date -->
+<center><h4>Oct 4, 2019</h4></center> <!-- date -->
 <br>
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1368,7 +1368,7 @@ The scikit-learn dataset we will use consists of a selection of 1797 images of s

 <p>
 To feed data into a feed-forward neural network we need to represent
-the inputs as a feature matrix \( X = (n_{inputs}, n_{features}) \).  Each
+the inputs as a design/feature matrix \( X = (n_{inputs}, n_{features}) \).  Each
 row represents an <em>input</em>, in this case a handwritten digit, and
 each column represents a <em>feature</em>, in this case a pixel.  The
 correct answers, also known as <em>labels</em> or <em>targets</em> are
@@ -1378,7 +1378,7 @@ represented as a 1D array of integers
 <p>
 As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from
 measurements of height (in m)  
-and weight (in kg). If we have measurements of 5 people the feature matrix could be for example:  
+and weight (in kg). If we have measurements of 5 people the design/feature matrix could be for example:  

 $$ X = \begin{bmatrix}
 1.85 &amp; 81\\
@@ -1396,7 +1396,7 @@ $$ Y = (23.7, 22.2, 27.1, 17.5, 21.1) $$
 <p>
 Since each input image is a 2D matrix, we need to flatten the image
 (i.e. "unravel" the 2D matrix into a 1D array) to turn the data into a
-feature matrix. This means we lose all spatial information in the
+design/feature matrix. This means we lose all spatial information in the
 image, such as locality and translational invariance. More complicated
 architectures such as Convolutional Neural Networks can take advantage
 of such information, and are most commonly applied when analyzing
@@ -1813,9 +1813,9 @@ reduces <em>overfitting</em>.
 <p>
 We will measure the size of the weights using the so called <em>L2-norm</em>, meaning our cost function becomes:  

-$$ \nabla \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) \quad \rightarrow \quad
-\frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) + \lambda \lvert \lvert \hat{w} \rvert \rvert_2^2 
-= \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}(\theta) + \lambda \sum_{ij} w_{ij}^2,$$
+$$  \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \mathcal{L}_i(\theta) \quad \rightarrow \quad
+\frac{1}{N} \sum_{i=1}^N  \mathcal{L}_i(\theta) + \lambda \lvert \lvert \hat{w} \rvert \rvert_2^2 
+= \frac{1}{N} \sum_{i=1}^N  \mathcal{L}(\theta) + \lambda \sum_{ij} w_{ij}^2,$$

 <p>
 i.e. we sum up all the weights squared. The factor \( \lambda \) is known as a regularization parameter.
@@ -208,7 +208,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Oct 3, 2019</h4></center> <!-- date -->
+<center><h4>Oct 4, 2019</h4></center> <!-- date -->
 <br>
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1373,7 +1373,7 @@ The scikit-learn dataset we will use consists of a selection of 1797 images of s

 <p>
 To feed data into a feed-forward neural network we need to represent
-the inputs as a feature matrix \( X = (n_{inputs}, n_{features}) \).  Each
+the inputs as a design/feature matrix \( X = (n_{inputs}, n_{features}) \).  Each
 row represents an <em>input</em>, in this case a handwritten digit, and
 each column represents a <em>feature</em>, in this case a pixel.  The
 correct answers, also known as <em>labels</em> or <em>targets</em> are
@@ -1383,7 +1383,7 @@ represented as a 1D array of integers
 <p>
 As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from
 measurements of height (in m)  
-and weight (in kg). If we have measurements of 5 people the feature matrix could be for example:  
+and weight (in kg). If we have measurements of 5 people the design/feature matrix could be for example:  

 $$ X = \begin{bmatrix}
 1.85 &amp; 81\\
@@ -1401,7 +1401,7 @@ $$ Y = (23.7, 22.2, 27.1, 17.5, 21.1) $$
 <p>
 Since each input image is a 2D matrix, we need to flatten the image
 (i.e. "unravel" the 2D matrix into a 1D array) to turn the data into a
-feature matrix. This means we lose all spatial information in the
+design/feature matrix. This means we lose all spatial information in the
 image, such as locality and translational invariance. More complicated
 architectures such as Convolutional Neural Networks can take advantage
 of such information, and are most commonly applied when analyzing
@@ -1818,9 +1818,9 @@ reduces <em>overfitting</em>.
 <p>
 We will measure the size of the weights using the so called <em>L2-norm</em>, meaning our cost function becomes:  

-$$ \nabla \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) \quad \rightarrow \quad
-\frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) + \lambda \lvert \lvert \hat{w} \rvert \rvert_2^2 
-= \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}(\theta) + \lambda \sum_{ij} w_{ij}^2,$$
+$$  \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \mathcal{L}_i(\theta) \quad \rightarrow \quad
+\frac{1}{N} \sum_{i=1}^N  \mathcal{L}_i(\theta) + \lambda \lvert \lvert \hat{w} \rvert \rvert_2^2 
+= \frac{1}{N} \sum_{i=1}^N  \mathcal{L}(\theta) + \lambda \sum_{ij} w_{ij}^2,$$

 <p>
 i.e. we sum up all the weights squared. The factor \( \lambda \) is known as a regularization parameter.
@@ -10,7 +10,7 @@
    "<!-- Author: -->  \n",
    "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
    "\n",
-    "Date: **Oct 3, 2019**\n",
+    "Date: **Oct 4, 2019**\n",
    "\n",
    "Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
    "\n",
@@ -1656,7 +1656,7 @@
    "The scikit-learn dataset we will use consists of a selection of 1797 images of size $8\\times 8$ collected and processed from this database.  \n",
    "\n",
    "To feed data into a feed-forward neural network we need to represent\n",
-    "the inputs as a feature matrix $X = (n_{inputs}, n_{features})$.  Each\n",
+    "the inputs as a design/feature matrix $X = (n_{inputs}, n_{features})$.  Each\n",
    "row represents an *input*, in this case a handwritten digit, and\n",
    "each column represents a *feature*, in this case a pixel.  The\n",
    "correct answers, also known as *labels* or *targets* are\n",
@@ -1665,7 +1665,7 @@
    "\n",
    "As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from\n",
    "measurements of height (in m)  \n",
-    "and weight (in kg). If we have measurements of 5 people the feature matrix could be for example:  \n",
+    "and weight (in kg). If we have measurements of 5 people the design/feature matrix could be for example:  \n",
    "\n",
    "$$ X = \\begin{bmatrix}\n",
    "1.85 & 81\\\\\n",
@@ -1681,7 +1681,7 @@
    "\n",
    "Since each input image is a 2D matrix, we need to flatten the image\n",
    "(i.e. \"unravel\" the 2D matrix into a 1D array) to turn the data into a\n",
-    "feature matrix. This means we lose all spatial information in the\n",
+    "design/feature matrix. This means we lose all spatial information in the\n",
    "image, such as locality and translational invariance. More complicated\n",
    "architectures such as Convolutional Neural Networks can take advantage\n",
    "of such information, and are most commonly applied when analyzing\n",
@@ -2077,9 +2077,9 @@
    "\n",
    "We will measure the size of the weights using the so called *L2-norm*, meaning our cost function becomes:  \n",
    "\n",
-    "$$ \\nabla \\mathcal{C}(\\theta) = \\frac{1}{N} \\sum_{i=1}^N \\nabla \\mathcal{L}_i(\\theta) \\quad \\rightarrow \\quad\n",
-    "\\frac{1}{N} \\sum_{i=1}^N \\nabla \\mathcal{L}_i(\\theta) + \\lambda \\lvert \\lvert \\hat{w} \\rvert \\rvert_2^2 \n",
-    "= \\frac{1}{N} \\sum_{i=1}^N \\nabla \\mathcal{L}(\\theta) + \\lambda \\sum_{ij} w_{ij}^2,$$  \n",
+    "$$  \\mathcal{C}(\\theta) = \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}_i(\\theta) \\quad \\rightarrow \\quad\n",
+    "\\frac{1}{N} \\sum_{i=1}^N  \\mathcal{L}_i(\\theta) + \\lambda \\lvert \\lvert \\hat{w} \\rvert \\rvert_2^2 \n",
+    "= \\frac{1}{N} \\sum_{i=1}^N  \\mathcal{L}(\\theta) + \\lambda \\sum_{ij} w_{ij}^2,$$  \n",
    "\n",
    "i.e. we sum up all the weights squared. The factor $\\lambda$ is known as a regularization parameter.\n",
    "\n",