-Computing polynomial products can be implemented efficiently if we rewrite the the more brute force multiplications using convolution. +Computing polynomial products can be implemented efficiently if we rewrite the more brute force multiplications using convolution. We note first that the new coefficients are given as $$ \begin{split} \delta_0=&\alpha_0\beta_0\\ -\delta_1=&\alpha_1\beta_0+\beta_0\alpha_1\\ -\delta_2=&\alpha_0\beta_2+\beta_1\alpha_1+\alpha_2\beta_0\\ -\delta_3=&\alpha_1\beta_2+\beta_1\alpha_2+\alpha_0\beta_3\\ -\delta_4=&\alpha_2\beta_2+\beta_3\alpha_1\\ +\delta_1=&\alpha_1\beta_0+\alpha_1\beta_0\\ +\delta_2=&\alpha_0\beta_2+\alpha_1\beta_1+\alpha_2\beta_0\\ +\delta_3=&\alpha_1\beta_2+\alpha_2\beta_1+\alpha_0\beta_3\\ +\delta_4=&\alpha_2\beta_2+\alpha_1\beta_3\\ \delta_5=&\alpha_2\beta_3.\\ \end{split} $$ @@ -456,7 +463,7 @@ Do you see a potential drawback with these equations?
-The process is commutative and we can easily see that we can rewrite the multiplication in terms of a martrix holding \( \beta \) and a vector holding \( \alpha \). +The process is commutative and we can easily see that we can rewrite the multiplication in terms of a matrix holding \( \beta \) and a vector holding \( \alpha \). +In this case we have +$$ +\boldsymbol{\delta}=\begin{bmatrix}\beta_0 & 0 & 0 \\ + \beta_1 & \beta_0 & 0 \\ + \beta_2 & \beta_1 & \beta_0 \\ + \beta_3 & \beta_2 & \beta_1 \\ + 0 & \beta_3 & \beta_2 \\ + 0 & 0 & \beta_3 + \end{bmatrix}\begin{bmatrix} \alpha_0 \\ \alpha_1 \\ \alpha_2\end{bmatrix}. +$$ + +
+Note that the use of these matrices is for mathematical purposes only and not implementation purposes. +When implementing the above equation we do not encode (and allocate memory) the matrices explicitely. +We rather code the convolutions in the minimal memory footprint that they require.
@@ -442,7 +464,7 @@ The process is commutative and we can easily see that we can rewrite the multipl
-As discussed above, CNNs are neural networks built from the assumption that the inputs -to the network are 2D images. This is important because the number of features or pixels in images -grows very fast with the image size, and an enormous number of weights and biases are needed in order to build an accurate network. +In feilds like signal processing (and imaging as well), one designs +so-called filters. These filters are defined by the convolutions and +are often hand-crafted. One may specify filters for smoothing, edge +detection, frequency reshaping, and similar operations. However with +neural networks the idea is to automatically learn the filters and use +many of them in conjunction with non-linear operations (activation +functions).
-As before, we still have our input, a hidden layer and an output. What's novel about convolutional networks -are the convolutional and pooling layers stacked in pairs between the input and the hidden layer. -In addition, the data is no longer represented as a 2D feature matrix, instead each input is a number of 2D -matrices, typically 1 for each color dimension (Red, Green, Blue). +As an example consider a neural network operating on sound sequence +data. Assume that we an input vector \( \boldsymbol{x} \) of length \( d=10^6 \). We +construct then a neural network with onle hidden layer only with +\( 10^4 \) nodes. This means that we will have a weight matrix with +\( 10^4\times 10^6=10^{10} \) weights to be determined, together with \( 10^4 \) biases. + +
+Assume furthermore that we have an output layer which is meant to train whether the sound sequence represents a human voice (true) or something else (false). +It means that we have only one output node. But since this output node connects to \( 10^4 \) nodes in the hidden layer, there are in total \( 10^4 \) weights to be determined for the output layer, plus one bias. In total we have + +$$ +\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \approx 10^{10}, +$$ + +that is ten billion parameters to determine.
@@ -435,7 +457,7 @@ matrices, typically 1 for each color dimension (Red, Green, Blue).
-It means that to represent the entire -dataset of images, we require a 4D matrix or tensor. This tensor has the dimensions: -$$ -(n_{inputs},\, n_{pixels, width},\, n_{pixels, height},\, depth) . -$$ +In today’s architecture one can train such neural networks, however +this is a huge number of parameters for the task at hand. In general, +it is a very wasteful and inefficient use of dense matrices as +parameters. Just as importantly, such trained network parameters are +very specific for the type of input data on which they were trained +and the network is not likely to generalize easily to variations in +the input. + +
+The main principles that justify convolutions is locality of +information and repetion of patterns within the signal. Sound samples +of the input in adjacent spots are much more likely to affect each +other than those that are very far away. Similarly, sounds are +repeated in multiple times in the signal. While slightly simplistic, +reasoning about such a sound example demonstrates this. The same +principles then apply to images and other similar data.
@@ -431,7 +449,7 @@ $$
-The MNIST dataset consists of grayscale images with a pixel size of -\( 28\times 28 \), meaning we require \( 28 \times 28 = 724 \) weights to each -neuron in the first hidden layer. +As discussed above, CNNs are neural networks built from the assumption that the inputs +to the network are 2D images. This is important because the number of features or pixels in images +grows very fast with the image size, and an enormous number of weights and biases are needed in order to build an accurate network.
-If we were to analyze images of size \( 128\times 128 \) we would require -\( 128 \times 128 = 16384 \) weights to each neuron. Even worse if we were -dealing with color images, as most images are, we have an image matrix -of size \( 128\times 128 \) for each color dimension (Red, Green, Blue), -meaning 3 times the number of weights \( = 49152 \) are required for every -single neuron in the first hidden layer. +As before, we still have our input, a hidden layer and an output. What's novel about convolutional networks +are the convolutional and pooling layers stacked in pairs between the input and the hidden layer. +In addition, the data is no longer represented as a 2D feature matrix, instead each input is a number of 2D +matrices, typically 1 for each color dimension (Red, Green, Blue).
@@ -437,7 +442,7 @@ single neuron in the first hidden layer.
-Images typically have strong local correlations, meaning that a small -part of the image varies little from its neighboring regions. If for -example we have an image of a blue car, we can roughly assume that a -small blue part of the image is surrounded by other blue regions. - -
-Therefore, instead of connecting every single pixel to a neuron in the -first hidden layer, as we have previously done with deep neural -networks, we can instead connect each neuron to a small part of the -image (in all 3 RGB depth dimensions). The size of each small area is -fixed, and known as a receptive. +It means that to represent the entire +dataset of images, we require a 4D matrix or tensor. This tensor has the dimensions: +$$ +(n_{inputs},\, n_{pixels, width},\, n_{pixels, height},\, depth) . +$$
@@ -437,7 +438,7 @@ fixed, and known as a 82
- + -
-A convolution is performed on the image which outputs -a 3D volume of neurons. The weights to the input are arranged in a number of 2D matrices, known as filters. +The MNIST dataset consists of grayscale images with a pixel size of +\( 28\times 28 \), meaning we require \( 28 \times 28 = 724 \) weights to each +neuron in the first hidden layer.
-Each filter slides along the input image, taking the dot product -between each small part of the image and the filter, in all depth -dimensions. This is then passed through a non-linear function, -typically the Rectified Linear (ReLu) function, which serves as the -activation of the neurons in the first convolutional layer. This is -further passed through a pooling layer, which reduces the size of the -convolutional layer, e.g. by taking the maximum or average across some -small regions, and this serves as input to the next convolutional -layer. +If we were to analyze images of size \( 128\times 128 \) we would require +\( 128 \times 128 = 16384 \) weights to each neuron. Even worse if we were +dealing with color images, as most images are, we have an image matrix +of size \( 128\times 128 \) for each color dimension (Red, Green, Blue), +meaning 3 times the number of weights \( = 49152 \) are required for every +single neuron in the first hidden layer.
@@ -441,7 +444,7 @@ layer.
-By systematically reducing the size of the input volume, through -convolution and pooling, the network should create representations of -small parts of the input, and then from them assemble representations -of larger areas. The final pooling layer is flattened to serve as -input to a hidden layer, such that each neuron in the final pooling -layer is connected to every single neuron in the hidden layer. This -then serves as input to the output layer, e.g. a softmax output for -classification. +Images typically have strong local correlations, meaning that a small +part of the image varies little from its neighboring regions. If for +example we have an image of a blue car, we can roughly assume that a +small blue part of the image is surrounded by other blue regions. + +
+Therefore, instead of connecting every single pixel to a neuron in the +first hidden layer, as we have previously done with deep neural +networks, we can instead connect each neuron to a small part of the +image (in all 3 RGB depth dimensions). The size of each small area is +fixed, and known as a receptive.
@@ -434,7 +444,7 @@ classification.
-Computing polynomial products can be implemented efficiently if we rewrite the the more brute force multiplications using convolution. +Computing polynomial products can be implemented efficiently if we rewrite the more brute force multiplications using convolution. We note first that the new coefficients are given as
$$
\begin{split}
\delta_0=&\alpha_0\beta_0\\
-\delta_1=&\alpha_1\beta_0+\beta_0\alpha_1\\
-\delta_2=&\alpha_0\beta_2+\beta_1\alpha_1+\alpha_2\beta_0\\
-\delta_3=&\alpha_1\beta_2+\beta_1\alpha_2+\alpha_0\beta_3\\
-\delta_4=&\alpha_2\beta_2+\beta_3\alpha_1\\
+\delta_1=&\alpha_1\beta_0+\alpha_1\beta_0\\
+\delta_2=&\alpha_0\beta_2+\alpha_1\beta_1+\alpha_2\beta_0\\
+\delta_3=&\alpha_1\beta_2+\alpha_2\beta_1+\alpha_0\beta_3\\
+\delta_4=&\alpha_2\beta_2+\alpha_1\beta_3\\
\delta_5=&\alpha_2\beta_3.\\
\end{split}
$$
@@ -3312,12 +3312,29 @@ $$
0 & \alpha_2 & \alpha_1 & \alpha_0 \\
0 & 0 & \alpha_2 & \alpha_1 \\
0 & 0 & 0 & \alpha_2
- \end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3\end{bmatrix}
+ \end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3\end{bmatrix}.
$$
-The process is commutative and we can easily see that we can rewrite the multiplication in terms of a martrix holding \( \beta \) and a vector holding \( \alpha \). +The process is commutative and we can easily see that we can rewrite the multiplication in terms of a matrix holding \( \beta \) and a vector holding \( \alpha \). +In this case we have +
+$$
+\boldsymbol{\delta}=\begin{bmatrix}\beta_0 & 0 & 0 \\
+ \beta_1 & \beta_0 & 0 \\
+ \beta_2 & \beta_1 & \beta_0 \\
+ \beta_3 & \beta_2 & \beta_1 \\
+ 0 & \beta_3 & \beta_2 \\
+ 0 & 0 & \beta_3
+ \end{bmatrix}\begin{bmatrix} \alpha_0 \\ \alpha_1 \\ \alpha_2\end{bmatrix}.
+$$
+
+
+
+Note that the use of these matrices is for mathematical purposes only and not implementation purposes.
+When implementing the above equation we do not encode (and allocate memory) the matrices explicitely.
+We rather code the convolutions in the minimal memory footprint that they require.
@@ -3599,6 +3616,62 @@ More text will be added here
+
+In feilds like signal processing (and imaging as well), one designs
+so-called filters. These filters are defined by the convolutions and
+are often hand-crafted. One may specify filters for smoothing, edge
+detection, frequency reshaping, and similar operations. However with
+neural networks the idea is to automatically learn the filters and use
+many of them in conjunction with non-linear operations (activation
+functions).
+
+
+As an example consider a neural network operating on sound sequence
+data. Assume that we an input vector \( \boldsymbol{x} \) of length \( d=10^6 \). We
+construct then a neural network with onle hidden layer only with
+\( 10^4 \) nodes. This means that we will have a weight matrix with
+\( 10^4\times 10^6=10^{10} \) weights to be determined, together with \( 10^4 \) biases.
+
+
+Assume furthermore that we have an output layer which is meant to train whether the sound sequence represents a human voice (true) or something else (false).
+It means that we have only one output node. But since this output node connects to \( 10^4 \) nodes in the hidden layer, there are in total \( 10^4 \) weights to be determined for the output layer, plus one bias. In total we have
+
+
+In today’s architecture one can train such neural networks, however
+this is a huge number of parameters for the task at hand. In general,
+it is a very wasteful and inefficient use of dense matrices as
+parameters. Just as importantly, such trained network parameters are
+very specific for the type of input data on which they were trained
+and the network is not likely to generalize easily to variations in
+the input.
+
+
+The main principles that justify convolutions is locality of
+information and repetion of patterns within the signal. Sound samples
+of the input in adjacent spots are much more likely to affect each
+other than those that are very far away. Similarly, sounds are
+repeated in multiple times in the signal. While slightly simplistic,
+reasoning about such a sound example demonstrates this. The same
+principles then apply to images and other similar data.
+
-Computing polynomial products can be implemented efficiently if we rewrite the the more brute force multiplications using convolution.
+Computing polynomial products can be implemented efficiently if we rewrite the more brute force multiplications using convolution.
We note first that the new coefficients are given as
$$
\begin{split}
\delta_0=&\alpha_0\beta_0\\
-\delta_1=&\alpha_1\beta_0+\beta_0\alpha_1\\
-\delta_2=&\alpha_0\beta_2+\beta_1\alpha_1+\alpha_2\beta_0\\
-\delta_3=&\alpha_1\beta_2+\beta_1\alpha_2+\alpha_0\beta_3\\
-\delta_4=&\alpha_2\beta_2+\beta_3\alpha_1\\
+\delta_1=&\alpha_1\beta_0+\alpha_1\beta_0\\
+\delta_2=&\alpha_0\beta_2+\alpha_1\beta_1+\alpha_2\beta_0\\
+\delta_3=&\alpha_1\beta_2+\alpha_2\beta_1+\alpha_0\beta_3\\
+\delta_4=&\alpha_2\beta_2+\alpha_1\beta_3\\
\delta_5=&\alpha_2\beta_3.\\
\end{split}
$$
@@ -3324,11 +3329,26 @@ $$
0 & \alpha_2 & \alpha_1 & \alpha_0 \\
0 & 0 & \alpha_2 & \alpha_1 \\
0 & 0 & 0 & \alpha_2
- \end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3\end{bmatrix}
+ \end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3\end{bmatrix}.
$$
-The process is commutative and we can easily see that we can rewrite the multiplication in terms of a martrix holding \( \beta \) and a vector holding \( \alpha \).
+The process is commutative and we can easily see that we can rewrite the multiplication in terms of a matrix holding \( \beta \) and a vector holding \( \alpha \).
+In this case we have
+$$
+\boldsymbol{\delta}=\begin{bmatrix}\beta_0 & 0 & 0 \\
+ \beta_1 & \beta_0 & 0 \\
+ \beta_2 & \beta_1 & \beta_0 \\
+ \beta_3 & \beta_2 & \beta_1 \\
+ 0 & \beta_3 & \beta_2 \\
+ 0 & 0 & \beta_3
+ \end{bmatrix}\begin{bmatrix} \alpha_0 \\ \alpha_1 \\ \alpha_2\end{bmatrix}.
+$$
+
+
+Note that the use of these matrices is for mathematical purposes only and not implementation purposes.
+When implementing the above equation we do not encode (and allocate memory) the matrices explicitely.
+We rather code the convolutions in the minimal memory footprint that they require.
+In feilds like signal processing (and imaging as well), one designs
+so-called filters. These filters are defined by the convolutions and
+are often hand-crafted. One may specify filters for smoothing, edge
+detection, frequency reshaping, and similar operations. However with
+neural networks the idea is to automatically learn the filters and use
+many of them in conjunction with non-linear operations (activation
+functions).
+
+
+As an example consider a neural network operating on sound sequence
+data. Assume that we an input vector \( \boldsymbol{x} \) of length \( d=10^6 \). We
+construct then a neural network with onle hidden layer only with
+\( 10^4 \) nodes. This means that we will have a weight matrix with
+\( 10^4\times 10^6=10^{10} \) weights to be determined, together with \( 10^4 \) biases.
+
+
+Assume furthermore that we have an output layer which is meant to train whether the sound sequence represents a human voice (true) or something else (false).
+It means that we have only one output node. But since this output node connects to \( 10^4 \) nodes in the hidden layer, there are in total \( 10^4 \) weights to be determined for the output layer, plus one bias. In total we have
+
+$$
+\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \approx 10^{10},
+$$
+
+that is ten billion parameters to determine.
+
+
+
+In today’s architecture one can train such neural networks, however
+this is a huge number of parameters for the task at hand. In general,
+it is a very wasteful and inefficient use of dense matrices as
+parameters. Just as importantly, such trained network parameters are
+very specific for the type of input data on which they were trained
+and the network is not likely to generalize easily to variations in
+the input.
+
+
+The main principles that justify convolutions is locality of
+information and repetion of patterns within the signal. Sound samples
+of the input in adjacent spots are much more likely to affect each
+other than those that are very far away. Similarly, sounds are
+repeated in multiple times in the signal. While slightly simplistic,
+reasoning about such a sound example demonstrates this. The same
+principles then apply to images and other similar data.
+
+
+
diff --git a/doc/pub/week42/html/week42.html b/doc/pub/week42/html/week42.html
index cb0d00497..42373a3b2 100644
--- a/doc/pub/week42/html/week42.html
+++ b/doc/pub/week42/html/week42.html
@@ -250,6 +250,11 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'convolution-examples-probability-theory'),
+ ('More on Dimensionalities', 2, None, 'more-on-dimensionalities'),
+ ('Further Dimensionality Remarks',
+ 2,
+ None,
+ 'further-dimensionality-remarks'),
('CNNs in more detail, building convolutional neural networks in '
'Tensorflow and Keras',
2,
@@ -3281,16 +3286,16 @@ $$
-Computing polynomial products can be implemented efficiently if we rewrite the the more brute force multiplications using convolution.
+Computing polynomial products can be implemented efficiently if we rewrite the more brute force multiplications using convolution.
We note first that the new coefficients are given as
$$
\begin{split}
\delta_0=&\alpha_0\beta_0\\
-\delta_1=&\alpha_1\beta_0+\beta_0\alpha_1\\
-\delta_2=&\alpha_0\beta_2+\beta_1\alpha_1+\alpha_2\beta_0\\
-\delta_3=&\alpha_1\beta_2+\beta_1\alpha_2+\alpha_0\beta_3\\
-\delta_4=&\alpha_2\beta_2+\beta_3\alpha_1\\
+\delta_1=&\alpha_1\beta_0+\alpha_1\beta_0\\
+\delta_2=&\alpha_0\beta_2+\alpha_1\beta_1+\alpha_2\beta_0\\
+\delta_3=&\alpha_1\beta_2+\alpha_2\beta_1+\alpha_0\beta_3\\
+\delta_4=&\alpha_2\beta_2+\alpha_1\beta_3\\
\delta_5=&\alpha_2\beta_3.\\
\end{split}
$$
@@ -3329,11 +3334,26 @@ $$
0 & \alpha_2 & \alpha_1 & \alpha_0 \\
0 & 0 & \alpha_2 & \alpha_1 \\
0 & 0 & 0 & \alpha_2
- \end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3\end{bmatrix}
+ \end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3\end{bmatrix}.
$$
-The process is commutative and we can easily see that we can rewrite the multiplication in terms of a martrix holding \( \beta \) and a vector holding \( \alpha \).
+The process is commutative and we can easily see that we can rewrite the multiplication in terms of a matrix holding \( \beta \) and a vector holding \( \alpha \).
+In this case we have
+$$
+\boldsymbol{\delta}=\begin{bmatrix}\beta_0 & 0 & 0 \\
+ \beta_1 & \beta_0 & 0 \\
+ \beta_2 & \beta_1 & \beta_0 \\
+ \beta_3 & \beta_2 & \beta_1 \\
+ 0 & \beta_3 & \beta_2 \\
+ 0 & 0 & \beta_3
+ \end{bmatrix}\begin{bmatrix} \alpha_0 \\ \alpha_1 \\ \alpha_2\end{bmatrix}.
+$$
+
+
+Note that the use of these matrices is for mathematical purposes only and not implementation purposes.
+When implementing the above equation we do not encode (and allocate memory) the matrices explicitely.
+We rather code the convolutions in the minimal memory footprint that they require.
+In feilds like signal processing (and imaging as well), one designs
+so-called filters. These filters are defined by the convolutions and
+are often hand-crafted. One may specify filters for smoothing, edge
+detection, frequency reshaping, and similar operations. However with
+neural networks the idea is to automatically learn the filters and use
+many of them in conjunction with non-linear operations (activation
+functions).
+
+
+As an example consider a neural network operating on sound sequence
+data. Assume that we an input vector \( \boldsymbol{x} \) of length \( d=10^6 \). We
+construct then a neural network with onle hidden layer only with
+\( 10^4 \) nodes. This means that we will have a weight matrix with
+\( 10^4\times 10^6=10^{10} \) weights to be determined, together with \( 10^4 \) biases.
+
+
+Assume furthermore that we have an output layer which is meant to train whether the sound sequence represents a human voice (true) or something else (false).
+It means that we have only one output node. But since this output node connects to \( 10^4 \) nodes in the hidden layer, there are in total \( 10^4 \) weights to be determined for the output layer, plus one bias. In total we have
+
+$$
+\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \approx 10^{10},
+$$
+
+that is ten billion parameters to determine.
+
+
+
+In today’s architecture one can train such neural networks, however
+this is a huge number of parameters for the task at hand. In general,
+it is a very wasteful and inefficient use of dense matrices as
+parameters. Just as importantly, such trained network parameters are
+very specific for the type of input data on which they were trained
+and the network is not likely to generalize easily to variations in
+the input.
+
+
+The main principles that justify convolutions is locality of
+information and repetion of patterns within the signal. Sound samples
+of the input in adjacent spots are much more likely to affect each
+other than those that are very far away. Similarly, sounds are
+repeated in multiple times in the signal. While slightly simplistic,
+reasoning about such a sound example demonstrates this. The same
+principles then apply to images and other similar data.
+
+
+
diff --git a/doc/pub/week42/ipynb/ipynb-week42-src.tar.gz b/doc/pub/week42/ipynb/ipynb-week42-src.tar.gz
index 0d7e50f97..183e73ee0 100644
Binary files a/doc/pub/week42/ipynb/ipynb-week42-src.tar.gz and b/doc/pub/week42/ipynb/ipynb-week42-src.tar.gz differ
diff --git a/doc/pub/week42/ipynb/week42.ipynb b/doc/pub/week42/ipynb/week42.ipynb
index 510408be9..300382f65 100644
--- a/doc/pub/week42/ipynb/week42.ipynb
+++ b/doc/pub/week42/ipynb/week42.ipynb
@@ -3403,7 +3403,7 @@
"source": [
"## Efficient Polynomial Multiplication\n",
"\n",
- "Computing polynomial products can be implemented efficiently if we rewrite the the more brute force multiplications using convolution.\n",
+ "Computing polynomial products can be implemented efficiently if we rewrite the more brute force multiplications using convolution.\n",
"We note first that the new coefficients are given as"
]
},
@@ -3414,10 +3414,10 @@
"$$\n",
"\\begin{split}\n",
"\\delta_0=&\\alpha_0\\beta_0\\\\\n",
- "\\delta_1=&\\alpha_1\\beta_0+\\beta_0\\alpha_1\\\\\n",
- "\\delta_2=&\\alpha_0\\beta_2+\\beta_1\\alpha_1+\\alpha_2\\beta_0\\\\\n",
- "\\delta_3=&\\alpha_1\\beta_2+\\beta_1\\alpha_2+\\alpha_0\\beta_3\\\\\n",
- "\\delta_4=&\\alpha_2\\beta_2+\\beta_3\\alpha_1\\\\\n",
+ "\\delta_1=&\\alpha_1\\beta_0+\\alpha_1\\beta_0\\\\\n",
+ "\\delta_2=&\\alpha_0\\beta_2+\\alpha_1\\beta_1+\\alpha_2\\beta_0\\\\\n",
+ "\\delta_3=&\\alpha_1\\beta_2+\\alpha_2\\beta_1+\\alpha_0\\beta_3\\\\\n",
+ "\\delta_4=&\\alpha_2\\beta_2+\\alpha_1\\beta_3\\\\\n",
"\\delta_5=&\\alpha_2\\beta_3.\\\\\n",
"\\end{split}\n",
"$$"
@@ -3481,7 +3481,7 @@
"\t\t\t 0 & \\alpha_2 & \\alpha_1 & \\alpha_0 \\\\\n",
"\t\t\t 0 & 0 & \\alpha_2 & \\alpha_1 \\\\\n",
"\t\t\t 0 & 0 & 0 & \\alpha_2\n",
- "\t\t\t \\end{bmatrix}\\begin{bmatrix} \\beta_0 \\\\ \\beta_1 \\\\ \\beta_2 \\\\ \\beta_3\\end{bmatrix}\n",
+ "\t\t\t \\end{bmatrix}\\begin{bmatrix} \\beta_0 \\\\ \\beta_1 \\\\ \\beta_2 \\\\ \\beta_3\\end{bmatrix}.\n",
"$$"
]
},
@@ -3489,8 +3489,32 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "The process is commutative and we can easily see that we can rewrite the multiplication in terms of a martrix holding $\\beta$ and a vector holding $\\alpha$.\n",
- "\n",
+ "The process is commutative and we can easily see that we can rewrite the multiplication in terms of a matrix holding $\\beta$ and a vector holding $\\alpha$.\n",
+ "In this case we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{\\delta}=\\begin{bmatrix}\\beta_0 & 0 & 0 \\\\\n",
+ " \\beta_1 & \\beta_0 & 0 \\\\\n",
+ "\t\t\t \\beta_2 & \\beta_1 & \\beta_0 \\\\\n",
+ "\t\t\t \\beta_3 & \\beta_2 & \\beta_1 \\\\\n",
+ "\t\t\t 0 & \\beta_3 & \\beta_2 \\\\\n",
+ "\t\t\t 0 & 0 & \\beta_3\n",
+ "\t\t\t \\end{bmatrix}\\begin{bmatrix} \\alpha_0 \\\\ \\alpha_1 \\\\ \\alpha_2\\end{bmatrix}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note that the use of these matrices is for mathematical purposes only and not implementation purposes.\n",
+ "When implementing the above equation we do not encode (and allocate memory) the matrices explicitely.\n",
+ "We rather code the convolutions in the minimal memory footprint that they require.\n",
"\n",
"\n",
"\n",
@@ -3856,6 +3880,62 @@
"More text will be added here\n",
"\n",
"\n",
+ "## More on Dimensionalities\n",
+ "\n",
+ "In feilds like signal processing (and imaging as well), one designs\n",
+ "so-called filters. These filters are defined by the convolutions and\n",
+ "are often hand-crafted. One may specify filters for smoothing, edge\n",
+ "detection, frequency reshaping, and similar operations. However with\n",
+ "neural networks the idea is to automatically learn the filters and use\n",
+ "many of them in conjunction with non-linear operations (activation\n",
+ "functions).\n",
+ "\n",
+ "As an example consider a neural network operating on sound sequence\n",
+ "data. Assume that we an input vector $\\boldsymbol{x}$ of length $d=10^6$. We\n",
+ "construct then a neural network with onle hidden layer only with\n",
+ "$10^4$ nodes. This means that we will have a weight matrix with\n",
+ "$10^4\\times 10^6=10^{10}$ weights to be determined, together with $10^4$ biases.\n",
+ "\n",
+ "Assume furthermore that we have an output layer which is meant to train whether the sound sequence represents a human voice (true) or something else (false).\n",
+ "It means that we have only one output node. But since this output node connects to $10^4$ nodes in the hidden layer, there are in total $10^4$ weights to be determined for the output layer, plus one bias. In total we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \\approx 10^{10},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "that is ten billion parameters to determine. \n",
+ "\n",
+ "\n",
+ "## Further Dimensionality Remarks\n",
+ "\n",
+ "In today’s architecture one can train such neural networks, however\n",
+ "this is a huge number of parameters for the task at hand. In general,\n",
+ "it is a very wasteful and inefficient use of dense matrices as\n",
+ "parameters. Just as importantly, such trained network parameters are\n",
+ "very specific for the type of input data on which they were trained\n",
+ "and the network is not likely to generalize easily to variations in\n",
+ "the input.\n",
+ "\n",
+ "\n",
+ "The main principles that justify convolutions is locality of\n",
+ "information and repetion of patterns within the signal. Sound samples\n",
+ "of the input in adjacent spots are much more likely to affect each\n",
+ "other than those that are very far away. Similarly, sounds are\n",
+ "repeated in multiple times in the signal. While slightly simplistic,\n",
+ "reasoning about such a sound example demonstrates this. The same\n",
+ "principles then apply to images and other similar data.\n",
+ "\n",
+ "\n",
"\n",
"## CNNs in more detail, building convolutional neural networks in Tensorflow and Keras\n",
"\n",
diff --git a/doc/src/week42/week42.do.txt b/doc/src/week42/week42.do.txt
index c2d55cf94..b4c7f1d0f 100644
--- a/doc/src/week42/week42.do.txt
+++ b/doc/src/week42/week42.do.txt
@@ -2663,16 +2663,16 @@ z(t) = \delta_0+\delta_1 t+\delta_2 t^2+\delta_3 t^3+\delta_4 t^4+\delta_5 t^5.
!split
===== Efficient Polynomial Multiplication =====
-Computing polynomial products can be implemented efficiently if we rewrite the the more brute force multiplications using convolution.
+Computing polynomial products can be implemented efficiently if we rewrite the more brute force multiplications using convolution.
We note first that the new coefficients are given as
!bt
\begin{split}
\delta_0=&\alpha_0\beta_0\\
-\delta_1=&\alpha_1\beta_0+\beta_0\alpha_1\\
-\delta_2=&\alpha_0\beta_2+\beta_1\alpha_1+\alpha_2\beta_0\\
-\delta_3=&\alpha_1\beta_2+\beta_1\alpha_2+\alpha_0\beta_3\\
-\delta_4=&\alpha_2\beta_2+\beta_3\alpha_1\\
+\delta_1=&\alpha_1\beta_0+\alpha_1\beta_0\\
+\delta_2=&\alpha_0\beta_2+\alpha_1\beta_1+\alpha_2\beta_0\\
+\delta_3=&\alpha_1\beta_2+\alpha_2\beta_1+\alpha_0\beta_3\\
+\delta_4=&\alpha_2\beta_2+\alpha_1\beta_3\\
\delta_5=&\alpha_2\beta_3.\\
\end{split}
!et
@@ -2710,12 +2710,27 @@ as a matrix-vector multiplication
0 & \alpha_2 & \alpha_1 & \alpha_0 \\
0 & 0 & \alpha_2 & \alpha_1 \\
0 & 0 & 0 & \alpha_2
- \end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3\end{bmatrix}
+ \end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3\end{bmatrix}.
\]
!et
-The process is commutative and we can easily see that we can rewrite the multiplication in terms of a martrix holding $\beta$ and a vector holding $\alpha$.
+The process is commutative and we can easily see that we can rewrite the multiplication in terms of a matrix holding $\beta$ and a vector holding $\alpha$.
+In this case we have
+!bt
+\[
+\bm{\delta}=\begin{bmatrix}\beta_0 & 0 & 0 \\
+ \beta_1 & \beta_0 & 0 \\
+ \beta_2 & \beta_1 & \beta_0 \\
+ \beta_3 & \beta_2 & \beta_1 \\
+ 0 & \beta_3 & \beta_2 \\
+ 0 & 0 & \beta_3
+ \end{bmatrix}\begin{bmatrix} \alpha_0 \\ \alpha_1 \\ \alpha_2\end{bmatrix}.
+\]
+!et
+Note that the use of these matrices is for mathematical purposes only and not implementation purposes.
+When implementing the above equation we do not encode (and allocate memory) the matrices explicitely.
+We rather code the convolutions in the minimal memory footprint that they require.
@@ -2946,6 +2961,55 @@ plt.show()
More text will be added here
+!split
+===== More on Dimensionalities =====
+
+In feilds like signal processing (and imaging as well), one designs
+so-called filters. These filters are defined by the convolutions and
+are often hand-crafted. One may specify filters for smoothing, edge
+detection, frequency reshaping, and similar operations. However with
+neural networks the idea is to automatically learn the filters and use
+many of them in conjunction with non-linear operations (activation
+functions).
+
+As an example consider a neural network operating on sound sequence
+data. Assume that we an input vector $\bm{x}$ of length $d=10^6$. We
+construct then a neural network with onle hidden layer only with
+$10^4$ nodes. This means that we will have a weight matrix with
+$10^4\times 10^6=10^{10}$ weights to be determined, together with $10^4$ biases.
+
+Assume furthermore that we have an output layer which is meant to train whether the sound sequence represents a human voice (true) or something else (false).
+It means that we have only one output node. But since this output node connects to $10^4$ nodes in the hidden layer, there are in total $10^4$ weights to be determined for the output layer, plus one bias. In total we have
+
+!bt
+\[
+\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \approx 10^{10},
+\]
+!et
+that is ten billion parameters to determine.
+
+
+!split
+===== Further Dimensionality Remarks =====
+
+In today’s architecture one can train such neural networks, however
+this is a huge number of parameters for the task at hand. In general,
+it is a very wasteful and inefficient use of dense matrices as
+parameters. Just as importantly, such trained network parameters are
+very specific for the type of input data on which they were trained
+and the network is not likely to generalize easily to variations in
+the input.
+
+
+The main principles that justify convolutions is locality of
+information and repetion of patterns within the signal. Sound samples
+of the input in adjacent spots are much more likely to affect each
+other than those that are very far away. Similarly, sounds are
+repeated in multiple times in the signal. While slightly simplistic,
+reasoning about such a sound example demonstrates this. The same
+principles then apply to images and other similar data.
+
+
!split
===== CNNs in more detail, building convolutional neural networks in Tensorflow and Keras =====
More on Dimensionalities
+
+
+$$
+\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \approx 10^{10},
+$$
+
+
+that is ten billion parameters to determine.
+Further Dimensionality Remarks
+
+CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
diff --git a/doc/pub/week42/html/week42-solarized.html b/doc/pub/week42/html/week42-solarized.html
index 2dd74bff4..f295362f5 100644
--- a/doc/pub/week42/html/week42-solarized.html
+++ b/doc/pub/week42/html/week42-solarized.html
@@ -245,6 +245,11 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'convolution-examples-probability-theory'),
+ ('More on Dimensionalities', 2, None, 'more-on-dimensionalities'),
+ ('Further Dimensionality Remarks',
+ 2,
+ None,
+ 'further-dimensionality-remarks'),
('CNNs in more detail, building convolutional neural networks in '
'Tensorflow and Keras',
2,
@@ -3276,16 +3281,16 @@ $$
Efficient Polynomial Multiplication
@@ -3590,6 +3610,60 @@ More text will be added here
+More on Dimensionalities
+
+
+
+Further Dimensionality Remarks
+
+
+
CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
Efficient Polynomial Multiplication
@@ -3595,6 +3615,60 @@ More text will be added here
+More on Dimensionalities
+
+
+
+Further Dimensionality Remarks
+
+
+
CNNs in more detail, building convolutional neural networks in Tensorflow and Keras