diff --git a/doc/src/week44/week44.do.txt b/doc/src/week44/week44.do.txt index 76fa979fd..9641d7d3f 100644 --- a/doc/src/week44/week44.do.txt +++ b/doc/src/week44/week44.do.txt @@ -6,35 +6,36 @@ DATE: October 28-November 1 !split ===== Plan for week 44 ===== -!bblock Material for the active learning sessions on Tuesday and Wednesday - * Exercise on writing your own neural network code, application to the OR and XOR gates, see notes from last week - * The exercise this week is a continuation from last week - * Discussion of project 2 - * "Video of lab session from week 43":"https://youtu.be/Ia6wwDLxqtM" - * "Video of lab session from week 44":"https://youtu.be/EajWMW__k0I" - * "See also whiteboard notes from lab session week 44":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/Exercisesweek44.pdf" +!bblock Material for the lecture Monday October 28, 2024 +o Convolutional Neural Networks +o Readings and Videos: + * These lecture notes + * For a more in depth discussion on neural networks we recommend Goodfellow et al chapter 9. See also chapter 11 and 12 on practicalities and applications + * Reading suggestions for implementation of CNNs: "Rashcka et al.'s chapter 14":"https://github.com/rasbt/machine-learning-book/tree/main/ch14". T + * "Video on Deep Learning":"https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi" + * "Video on Convolutional Neural Networks from MIT":"https://www.youtube.com/watch?v=iaSUYvmCekI&ab_channel=AlexanderAmini" + * "Video on CNNs from Stanford":"https://www.youtube.com/watch?v=bNb2fEVKeEo&list=PLC1qU-LWwrF64f4QKQT-Vg5Wr4qEE1Zxk&index=6&ab_channel=StanfordUniversitySchoolofEngineering" +!eblock + + +!split +===== Lab sessions on Tuesday and Wednesday ===== + +!bblock +* Main focus is discussion of and work on project 2 +* If you did not get time to finish the exercises from week 43, you can also keep working on them and hand in this coming Friday +# * "Video of lab session from week 44":"https://youtu.be/EajWMW__k0I" +# * "See also whiteboard notes from lab session week 44":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/Exercisesweek44.pdf" !eblock -!bblock Material for the lecture Monday October 28, 2024 - * Convolutional Neural Networks - * Readings and Videos: - * These lecture notes - * For a more in depth discussion on neural networks we recommend Goodfellow et al chapter 9. See also chapter 11 and 12 on practicalities and applications - * Reading suggestions for implementation of CNNs: "Aurelien Geron's chapter 13":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf". - * "Video on Deep Learning":"https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi" - * "Video on Convolutional Neural Networks from MIT":"https://www.youtube.com/watch?v=iaSUYvmCekI&ab_channel=AlexanderAmini" - * "Video on CNNs from Stanford":"https://www.youtube.com/watch?v=bNb2fEVKeEo&list=PLC1qU-LWwrF64f4QKQT-Vg5Wr4qEE1Zxk&index=6&ab_channel=StanfordUniversitySchoolofEngineering" -!eblock - -!bblock And Lecture material on CNNs -* "See Michael Nielsen's Lectures":"http://neuralnetworksanddeeplearning.com/chap6.html" -!eblock !split ===== Material for Lecture Monday October 28 ===== + + !split ===== Convolutional Neural Networks (recognizing images) ===== @@ -152,6 +153,50 @@ dimension. FIGURE: [figslides/cnn.jpeg, width=500 frac=0.6] A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels). +!split +===== More on Dimensionalities ===== + +In fields like signal processing (and imaging as well), one designs +so-called filters. These filters are defined by the convolutions and +are often hand-crafted. One may specify filters for smoothing, edge +detection, frequency reshaping, and similar operations. However with +neural networks the idea is to automatically learn the filters and use +many of them in conjunction with non-linear operations (activation +functions). + +As an example consider a neural network operating on sound sequence +data. Assume that we an input vector $\bm{x}$ of length $d=10^6$. We +construct then a neural network with onle hidden layer only with +$10^4$ nodes. This means that we will have a weight matrix with +$10^4\times 10^6=10^{10}$ weights to be determined, together with $10^4$ biases. + +Assume furthermore that we have an output layer which is meant to train whether the sound sequence represents a human voice (true) or something else (false). +It means that we have only one output node. But since this output node connects to $10^4$ nodes in the hidden layer, there are in total $10^4$ weights to be determined for the output layer, plus one bias. In total we have + +!bt +\[ +\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \approx 10^{10}, +\] +!et +that is ten billion parameters to determine. + + +!split +===== Further remarks ===== + + + +The main principles that justify convolutions is locality of +information and repetion of patterns within the signal. Sound samples +of the input in adjacent spots are much more likely to affect each +other than those that are very far away. Similarly, sounds are +repeated in multiple times in the signal. While slightly simplistic, +reasoning about such a sound example demonstrates this. The same +principles then apply to images and other similar data. + + + + !split ===== Layers used to build CNNs ===== @@ -200,11 +245,10 @@ In summary: * Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don’t) * Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn’t) -For more material on convolutional networks, we strongly recommend -the course -"CS231":"http://cs231n.github.io/convolutional-networks/" which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). "Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs":"http://neuralnetworksanddeeplearning.com/chap6.html". +!split +===== A deep CNN model ("From Raschka et al":"https://github.com/rasbt/machine-learning-book") ===== -The textbook by Goodfellow et al, see chapter 9 contains an in depth discussion as well. +FIGURE: [figslides/deepcnn.png, width=500 frac=0.67] A deep CNN !split ===== Key Idea ===== @@ -217,15 +261,22 @@ only neighboring neurons in the input instead of connecting all with the first h We say we perform a filtering (convolution is the mathematical operation). + + + + !split ===== Mathematics of CNNs ===== The mathematics of CNNs is based on the mathematical operation of _convolution_. In mathematics (in particular in functional analysis), -convolution is represented by mathematical operation (integration, -summation etc) on two function in order to produce a third function +convolution is represented by mathematical operations (integration, +summation etc) on two functions in order to produce a third function that expresses how the shape of one gets modified by the other. -Convolution has a plethora of applications in a variety of disciplines, spanning from statistics to signal processing, computer vision, solutions of differential equations,linear algebra, engineering, and yes, machine learning. +Convolution has a plethora of applications in a variety of +disciplines, spanning from statistics to signal processing, computer +vision, solutions of differential equations,linear algebra, +engineering, and yes, machine learning. Mathematically, convolution is defined as follows (one-dimensional example): Let us define a continuous function $y(t)$ given by @@ -249,7 +300,7 @@ The discretized version reads y(t) = \sum_{a=-\infty}^{a=\infty}x(a)w(t-a). \] !et -Computing the inverse of the above convolution operations is known as deconvolution. +Computing the inverse of the above convolution operations is known as deconvolution and the process is commutative. How can we use this? And what does it mean? Let us study some familiar examples first. @@ -257,9 +308,13 @@ How can we use this? And what does it mean? Let us study some familiar examples !split ===== Convolution Examples: Polynomial multiplication ===== -We have already met such an example in project 1 when we tried to set -up the design matrix for a two-dimensional function. This was an -example of polynomial multiplication. Let us recast such a problem in terms of the convolution operation. +Our first example is that of a multiplication between two polynomials, +which we will rewrite in terms of the mathematics of convolution. In +the final stage, since the problem here is a discrete one, we will +recast the final expression in terms of a matrix-vector +multiplication, where the matrix is a so-called "Toeplitz matrix +":"https://link.springer.com/book/10.1007/978-93-86279-04-0". + Let us look a the following polynomials to second and third order, respectively: !bt \[ @@ -289,7 +344,7 @@ We note first that the new coefficients are given as !bt \begin{split} \delta_0=&\alpha_0\beta_0\\ -\delta_1=&\alpha_1\beta_0+\alpha_1\beta_0\\ +\delta_1=&\alpha_1\beta_0+\alpha_0\beta_1\\ \delta_2=&\alpha_0\beta_2+\alpha_1\beta_1+\alpha_2\beta_0\\ \delta_3=&\alpha_1\beta_2+\alpha_2\beta_1+\alpha_0\beta_3\\ \delta_4=&\alpha_2\beta_2+\alpha_1\beta_3\\ @@ -313,7 +368,17 @@ or as a double sum with restriction $l=i+j$ \] !et -Do you see a potential drawback with these equations? + +!split +===== Further simplification ===== + +Although we may have redundant operations with some few zeros for $\beta_i$, we can rewrite the above sum in a more compact way as +!bt +\[ +\delta_i = \sum_{k=0}^{k=m-1}\alpha_k\beta_{i-k}, +\] +!et +where $m=3$ in our case, the maximum length of the vector $\alpha$. Note that the vector $\bm{\beta}$ has length $n=4$. !split ===== A more efficient way of coding the above Convolution ===== @@ -348,11 +413,15 @@ In this case we have \] !et -Note that the use of these matrices is for mathematical purposes only and not implementation purposes. -When implementing the above equation we do not encode (and allocate memory) the matrices explicitely. -We rather code the convolutions in the minimal memory footprint that they require. +Note that the use of these matrices is for mathematical purposes only +and not implementation purposes. When implementing the above equation +we do not encode (and allocate memory) the matrices explicitely. We +rather code the convolutions in the minimal memory footprint that they +require. + + + -Does the number of floating point operations change here when we use the commutative property? The above matrices are examples of so-called "Toeplitz matrices":"https://link.springer.com/book/10.1007/978-93-86279-04-0". A @@ -372,289 +441,55 @@ rewrite as !et with elements $a_{ii}=a_{i+1,j+1}=a_{i-j}$ is an example of a Toeplitz matrix. Such a matrix does not need to be a square matrix. Toeplitz -matrices are also closely connected with Fourier series discussed -below, because the multiplication operator by a trigonometric +matrices are also closely connected with Fourier series, because the multiplication operator by a trigonometric polynomial, compressed to a finite-dimensional space, can be represented by such a matrix. The example above shows that we can represent linear convolution as multiplication of a Toeplitz matrix by a vector. -!split -===== Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms) ===== - -For problems with so-called harmonic oscillations, given by for example the following differential equation -!bt -\[ -m\frac{d^2x}{dt^2}+\eta\frac{dx}{dt}+x(t)=F(t), -\] -!et -where $F(t)$ is an applied external force acting on the system (often -called a driving force), one can use the theory of Fourier -transformations to find the solutions of this type of equations. - -If one has several driving forces, $F(t)=\sum_n F_n(t)$, one can find -the particular solution $x_{pn}(t)$ to the above differential equation for each $F_n$. The particular -solution for the entire driving force is then given by a series like - -!bt -\begin{equation} -x_p(t)=\sum_nx_{pn}(t). -\end{equation} -!et - -This is known as the principle of superposition. It only applies when -the homogenous equation is linear. -Superposition is especially useful when $F(t)$ can be written -as a sum of sinusoidal terms, because the solutions for each -sinusoidal (sine or cosine) term is analytic. - -Driving forces are often periodic, even when they are not -sinusoidal. Periodicity implies that for some time $t$ our function repeats itself periodically after a period $\tau$, that is - -!bt -\begin{eqnarray} -F(t+\tau)=F(t). -\end{eqnarray} -!et - -One example of a non-sinusoidal periodic force is a square wave. Many -components in electric circuits are non-linear, for example diodes. This -makes many wave forms non-sinusoidal even when the circuits are being -driven by purely sinusoidal sources. !split -===== Simple Code Example ===== +===== Fourier series and Toeplitz matrices ===== -The code here shows a typical example of such a square wave generated -using the functionality included in the _scipy_ Python package. We -have used a period of $\tau=0.2$. - -!bc pycod -import numpy as np -import math -from scipy import signal -import matplotlib.pyplot as plt - -# number of points -n = 500 -# start and final times -t0 = 0.0 -tn = 1.0 -# Period -t = np.linspace(t0, tn, n, endpoint=False) -SqrSignal = np.zeros(n) -SqrSignal = 1.0+signal.square(2*np.pi*5*t) -plt.plot(t, SqrSignal) -plt.ylim(-0.5, 2.5) -plt.show() -!ec - - -For the sinusoidal example the -period is $\tau=2\pi/\omega$. However, higher harmonics can also -satisfy the periodicity requirement. In general, any force that -satisfies the periodicity requirement can be expressed as a sum over -harmonics, - -!bt -\begin{equation} -F(t)=\frac{f_0}{2}+\sum_{n>0} f_n\cos(2n\pi t/\tau)+g_n\sin(2n\pi t/\tau). -\end{equation} -!et - -!split -===== Wrapping up Fourier transforms ===== - -We can write down the answer for -$x_{pn}(t)$, by substituting $f_n/m$ or $g_n/m$ for $F_0/m$. By -writing each factor $2n\pi t/\tau$ as $n\omega t$, with $\omega\equiv -2\pi/\tau$, - -!bt -\begin{equation} -label{eq:fourierdef1} -F(t)=\frac{f_0}{2}+\sum_{n>0}f_n\cos(n\omega t)+g_n\sin(n\omega t). -\end{equation} -!et - -The solutions for $x(t)$ then come from replacing $\omega$ with -$n\omega$ for each term in the particular solution, - -!bt -\begin{eqnarray} -x_p(t)&=&\frac{f_0}{2k}+\sum_{n>0} \alpha_n\cos(n\omega t-\delta_n)+\beta_n\sin(n\omega t-\delta_n),\\ -\nonumber -\alpha_n&=&\frac{f_n/m}{\sqrt{((n\omega)^2-\omega_0^2)+4\beta^2n^2\omega^2}},\\ -\nonumber -\beta_n&=&\frac{g_n/m}{\sqrt{((n\omega)^2-\omega_0^2)+4\beta^2n^2\omega^2}},\\ -\nonumber -\delta_n&=&\tan^{-1}\left(\frac{2\beta n\omega}{\omega_0^2-n^2\omega^2}\right). -\end{eqnarray} -!et - -!split -===== Finding the Coefficients ===== - -Because the forces have been applied for a long time, any non-zero -damping eliminates the homogenous parts of the solution. We need then -only consider the particular solution for each $n$. - -The problem is considered solved if one can find expressions for the -coefficients $f_n$ and $g_n$, even though the solutions are expressed -as an infinite sum. The coefficients can be extracted from the -function $F(t)$ by - -!bt -\begin{eqnarray} -label{eq:fourierdef2} -f_n&=&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~F(t)\cos(2n\pi t/\tau),\\ -\nonumber -g_n&=&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~F(t)\sin(2n\pi t/\tau). -\end{eqnarray} -!et - -To check the consistency of these expressions and to verify -Eq. (ref{eq:fourierdef2}), one can insert the expansion of $F(t)$ in -Eq. (ref{eq:fourierdef1}) into the expression for the coefficients in -Eq. (ref{eq:fourierdef2}) and see whether - -!bt -\[ -f_n=\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~\left\{\frac{f_0}{2}+\sum_{m>0}f_m\cos(m\omega t)+g_m\sin(m\omega t)\right\}\cos(n\omega t). -\] -!et - -Immediately, one can throw away all the terms with $g_m$ because they -convolute an even and an odd function. The term with $f_0/2$ -disappears because $\cos(n\omega t)$ is equally positive and negative -over the interval and will integrate to zero. For all the terms -$f_m\cos(m\omega t)$ appearing in the sum, one can use angle addition -formulas to see that $\cos(m\omega t)\cos(n\omega -t)=(1/2)(\cos[(m+n)\omega t]+\cos[(m-n)\omega t]$. This will integrate -to zero unless $m=n$. In that case the $m=n$ term gives - -!bt -\begin{equation} -\int_{-\tau/2}^{\tau/2}dt~\cos^2(m\omega t)=\frac{\tau}{2}, -\end{equation} -!et - -and - -!bt -\[ -f_n=\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~f_n/2=f_n. -\] -!et - -The same method can be used to check for the consistency of $g_n$. +This is an active and ogoing research area concerning CNNs. The following articles may be of interest +o "Read more about the convolution theorem and Fouriers series":"https://www.sciencedirect.com/topics/engineering/convolution-theorem#:~:text=The%20convolution%20theorem%20(together%20with,k%20)%20G%20(%20k%20)%20." +o "Fourier Transform Layer":"https://www.sciencedirect.com/science/article/pii/S1568494623006257" !split -===== Final words on Fourier Transforms ===== - -The code here uses the Fourier series applied to a -square wave signal. The code here -visualizes the various approximations given by Fourier series compared -with a square wave with period $T=0.2$ (dimensionless time), width $0.1$ and max value of the force $F=2$. We -see that when we increase the number of components in the Fourier -series, the Fourier series approximation gets closer and closer to the -square wave signal. - -!bc pycod -import numpy as np -import math -from scipy import signal -import matplotlib.pyplot as plt - -# number of points -n = 500 -# start and final times -t0 = 0.0 -tn = 1.0 -# Period -T =0.2 -# Max value of square signal -Fmax= 2.0 -# Width of signal -Width = 0.1 -t = np.linspace(t0, tn, n, endpoint=False) -SqrSignal = np.zeros(n) -FourierSeriesSignal = np.zeros(n) -SqrSignal = 1.0+signal.square(2*np.pi*5*t+np.pi*Width/T) -a0 = Fmax*Width/T -FourierSeriesSignal = a0 -Factor = 2.0*Fmax/np.pi -for i in range(1,500): - FourierSeriesSignal += Factor/(i)*np.sin(np.pi*i*Width/T)*np.cos(i*t*2*np.pi/T) -plt.plot(t, SqrSignal) -plt.plot(t, FourierSeriesSignal) -plt.ylim(-0.5, 2.5) -plt.show() -!ec - -=== Fourier transforms and convolution === - -We can use Fourier transforms in our studies of convolution as well. To see this, assume we have two functions $f$ and $g$ and their corresponding Fourier transforms $\hat{f}$ and $\hat{g}$. We remind the reader that the Fourier transform reads (say for the function $f$) +===== Generalizing the above one-dimensional case ===== +In order to align the above simple case with the more general convolution cases, we rename $\bm{\alpha}$, whose length is $m=3$, with $\bm{w}$. +We will interpret $\bm{w}$ as a weight/filter function with which we want to perform the convolution with an input varibale $\bm{x}$. +We replace thus $\bm{\beta}$ with $\bm{x}$ and $\bm{\delta}$ with $\bm{s}$ and have !bt \[ -\hat{f}(y)=\bm{F}[f(y)]=\frac{1}{2\pi}\int_{-\infty}^{\infty} d\omega \exp{-i\omega y} f(\omega), -\] -!et -and similarly we have -!bt -\[ -\hat{g}(y)=\bm{F}[g(y)]=\frac{1}{2\pi}\int_{-\infty}^{\infty} d\omega \exp{-i\omega y} g(\omega). -\] -!et -The inverse Fourier transform is given by -!bt -\[ -\bm{F}^{-1}[g(y)]=\frac{1}{2\pi}\int_{-\infty}^{\infty} d\omega \exp{i\omega y} g(\omega). +s(i) \left(x*w\right)(i)= \sum_{k=0}^{k=m-1}w(k)x(i-k), \] !et +where $m=3$ in our case, the maximum length of the vector $\bm{w}$. +Here the symbol $*$ represents the mathematical operation of convolution. -The inverse Fourier transform of the product of the two functions $\hat{f}\hat{g}$ can be written as -!bt -\[ -\bm{F}^{-1}[(\hat{f}\hat{g})(x)]=\frac{1}{2\pi}\int_{-\infty}^{\infty} d\omega \exp{i\omega x} \hat{f}(\omega)\hat{g}(\omega). -\] -!et -We can rewrite the latter as -!bt -\[ -\bm{F}^{-1}[(\hat{f}\hat{g})(x)]=\int_{-\infty}^{\infty} d\omega \exp{i\omega x} \hat{f}(\omega)\left[\frac{1}{2\pi}\int_{-\infty}^{\infty}g(y)dy \exp{-i\omega y}\right]=\frac{1}{2\pi}\int_{-\infty}^{\infty}dy g(y)\int_{-\infty}^{\infty} d\omega \hat{f}(\omega) \exp{i\omega(x- y)}, -\] -!et -which is simply -!bt -\[ -\bm{F}^{-1}[(\hat{f}\hat{g})(x)]=\int_{-\infty}^{\infty}dy g(y)f(x-y)=(f*g)(x), -\] -!et -the convolution of the functions $f$ and $g$. - ===== Two-dimensional Objects ===== We are now ready to start studying the discrete convolutions relevant for convolutional neural networks. We often use convolutions over more than one dimension at a time. If -we have a two-dimensional image $I$ as input, we can have a _filter_ -defined by a two-dimensional _kernel_ $K$. This leads to an output $S$ +we have a two-dimensional image $X$ as input, we can have a _filter_ +defined by a two-dimensional _kernel/weight/filter_ $W$. This leads to an output $Y$ !bt \[ -S_(i,j)=(I * K)(i,j) = \sum_m\sum_n I(m,n)K(i-m,j-n). +Y(i,j)=(X * W)(i,j) = \sum_m\sum_n X(m,n)W(i-m,j-n). \] !et -Convolution is a commutatitave process, which means we can rewrite this equation as +Convolution is a commutative process, which means we can rewrite this equation as !bt \[ -S_(i,j)=(I * K)(i,j) = \sum_m\sum_n I(i-m,j-n)K(m,n). +Y(i,j)=(X * W)(i,j) = \sum_m\sum_n X(i-m,j-n)W(m,n). \] !et @@ -663,73 +498,24 @@ Normally the latter is more straightforward to implement in a machine larning l Many deep learning libraries implement cross-correlation instead of convolution (although it is referred to s convolution) !bt -\[ -S_(i,j)=(I * K)(i,j) = \sum_m\sum_n I(i+m,j+n)K(m,n). +Y(i,j)=(X * W)(i,j) = \sum_m\sum_n X(i+m,j+n)W(m,n). \] !et -!split -===== More on Dimensionalities ===== - -In fields like signal processing (and imaging as well), one designs -so-called filters. These filters are defined by the convolutions and -are often hand-crafted. One may specify filters for smoothing, edge -detection, frequency reshaping, and similar operations. However with -neural networks the idea is to automatically learn the filters and use -many of them in conjunction with non-linear operations (activation -functions). - -As an example consider a neural network operating on sound sequence -data. Assume that we an input vector $\bm{x}$ of length $d=10^6$. We -construct then a neural network with onle hidden layer only with -$10^4$ nodes. This means that we will have a weight matrix with -$10^4\times 10^6=10^{10}$ weights to be determined, together with $10^4$ biases. - -Assume furthermore that we have an output layer which is meant to train whether the sound sequence represents a human voice (true) or something else (false). -It means that we have only one output node. But since this output node connects to $10^4$ nodes in the hidden layer, there are in total $10^4$ weights to be determined for the output layer, plus one bias. In total we have - -!bt -\[ -\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \approx 10^{10}, -\] -!et -that is ten billion parameters to determine. - - -!split -===== Further Dimensionality Remarks ===== - -In today’s architecture one can train such neural networks, however -this is a huge number of parameters for the task at hand. In general, -it is a very wasteful and inefficient use of dense matrices as -parameters. Just as importantly, such trained network parameters are -very specific for the type of input data on which they were trained -and the network is not likely to generalize easily to variations in -the input. - - -The main principles that justify convolutions is locality of -information and repetion of patterns within the signal. Sound samples -of the input in adjacent spots are much more likely to affect each -other than those that are very far away. Similarly, sounds are -repeated in multiple times in the signal. While slightly simplistic, -reasoning about such a sound example demonstrates this. The same -principles then apply to images and other similar data. - !split ===== CNNs in more detail ===== -Let assume we have an input matrix $I$ of dimensionality $3\times 3$ +Let assume we have an input matrix $X$ of dimensionality $3\times 3$ and a $2\times 2$ filter $W$ given by the following matrices !bt \[ -\bm{I}=\begin{bmatrix}i_{00} & i_{01} & i_{02} \\ - i_{10} & i_{11} & i_{12} \\ - i_{20} & i_{21} & i_{22} \end{bmatrix}, +\bm{X}=\begin{bmatrix}x_{00} & x_{01} & x_{02} \\ + x_{10} & x_{11} & x_{12} \\ + x_{20} & x_{21} & x_{22} \end{bmatrix}, \] !et and @@ -739,31 +525,31 @@ and w_{10} & w_{11}\end{bmatrix}. \] !et -We introduce now the hyperparameter $S$ _stride_. Stride represents how the filter $W$ moves the convolution process on the matrix $I$. +We introduce now the hyperparameter $S$ _stride_. Stride represents how the filter $W$ moves the convolution process on the matrix $X$. We strongly recommend the repository on "Arithmetic of deep learning by Dumoulin and Visin":"https://github.com/vdumoulin/conv_arithmetic" -Here we set the stride equal to $S=1$, which means that, starting with the element $i_{00}$, the filter will act on $2\times 2$ submatrices each time, starting with the upper corner and moving according to the stride value column by column. +Here we set the stride equal to $S=1$, which means that, starting with the element $x_{00}$, the filter will act on $2\times 2$ submatrices each time, starting with the upper corner and moving according to the stride value column by column. Here we perform the operation !bt \[ -S_(i,j)=(I * W)(i,j) = \sum_m\sum_n I(i-m,j-n)W(m,n), +Y_(i,j)=(X * W)(i,j) = \sum_m\sum_n X(i-m,j-n)W(m,n), \] !et and obtain !bt \[ -\bm{S}=\begin{bmatrix}i_{00}w_{00}+i_{01}w_{01}+i_{10}w_{10}+i_{11}w_{11} & i_{01}w_{00}+i_{02}w_{01}+i_{11}w_{10}+i_{12}w_{11} \\ - i_{10}w_{00}+i_{11}w_{01}+i_{20}w_{10}+i_{21}w_{11} & i_{11}w_{00}+i_{12}w_{01}+i_{21}w_{10}+i_{22}w_{11}\end{bmatrix}. +\bm{Y}=\begin{bmatrix}x_{00}w_{00}+x_{01}w_{01}+x_{10}w_{10}+x_{11}w_{11} & x_{01}w_{00}+x_{02}w_{01}+x_{11}w_{10}+x_{12}w_{11} \\ + x_{10}w_{00}+x_{11}w_{01}+x_{20}w_{10}+x_{21}w_{11} & x_{11}w_{00}+x_{12}w_{01}+x_{21}w_{10}+x_{22}w_{11}\end{bmatrix}. \] !et -We can rewrite this operation in terms of a matrix-vector multiplication by defining a new vector where we flatten out the inputs as a vector $\bm{I}'$ of length $9$ and +We can rewrite this operation in terms of a matrix-vector multiplication by defining a new vector where we flatten out the inputs as a vector $\bm{X}'$ of length $9$ and a matrix $\bm{W}'$ with dimension $4\times 9$ as !bt \[ -\bm{I}'=\begin{bmatrix}i_{00} \\ i_{01} \\ i_{02} \\ i_{10} \\ i_{11} \\ i_{12} \\ i_{20} \\ i_{21} \\ i_{22} \end{bmatrix}, +\bm{X}'=\begin{bmatrix}x_{00} \\ x_{01} \\ x_{02} \\ x_{10} \\ x_{11} \\ x_{12} \\ x_{20} \\ x_{21} \\ x_{22} \end{bmatrix}, \] !et @@ -777,16 +563,31 @@ and the new matrix \] !et -We see easily that performing the matrix-vector multiplication $\bm{W}'\bm{I}'$ is the same as the above convolution with stride $S=1$, that is +We see easily that performing the matrix-vector multiplication $\bm{W}'\bm{X}'$ is the same as the above convolution with stride $S=1$, that is !bt \[ -S=(\bm{W}*\bm{I}), +Y=(\bm{W}*\bm{X}), \] !et -is now given by $\bm{W}'\bm{I}'$ which is a vector of length $4$ instead of the originally resulting $2\times 2$ output matrix. +is now given by $\bm{W}'\bm{X}'$ which is a vector of length $4$ instead of the originally resulting $2\times 2$ output matrix. +!split +===== Performing a discrete convolution with padding ("From Raschka et al":"https://github.com/rasbt/machine-learning-book") ===== + +FIGURE: [figslides/discreteconv.png, width=500 frac=0.67] A deep CNN + + +!split +===== Performing a general discrete convolution ("From Raschka et al":"https://github.com/rasbt/machine-learning-book") ===== + +FIGURE: [figslides/discreteconv1.png, width=500 frac=0.67] A deep CNN + + +!split +===== Set of filters ===== + The collection of kernels/filters $W$ defining a discrete convolution has a shape corresponding to some permutation of $(n, m, k_1, \ldots, k_N)$, where @@ -950,387 +751,12 @@ For any $i$, $k$ and $s$, \end{equation*} !et - - - - - !split -===== CNNs in more detail, building convolutional neural networks in Tensorflow and Keras ===== +===== Pooling types ("From Raschka et al":"https://github.com/rasbt/machine-learning-book") ===== - -As discussed above, CNNs are neural networks built from the assumption that the inputs -to the network are 2D images. This is important because the number of features or pixels in images -grows very fast with the image size, and an enormous number of weights and biases are needed in order to build an accurate network. - -As before, we still have our input, a hidden layer and an output. What's novel about convolutional networks -are the _convolutional_ and _pooling_ layers stacked in pairs between the input and the hidden layer. -In addition, the data is no longer represented as a 2D feature matrix, instead each input is a number of 2D -matrices, typically 1 for each color dimension (Red, Green, Blue). +FIGURE: [figslides/maxpooling.png, width=500 frac=0.67] A deep CNN -!split -===== Setting it up ===== - -It means that to represent the entire -dataset of images, we require a 4D matrix or _tensor_. This tensor has the dimensions: -!bt -\[ -(n_{inputs},\, n_{pixels, width},\, n_{pixels, height},\, depth) . -\] -!et - -!split -===== The MNIST dataset again ===== - -The MNIST dataset consists of grayscale images with a pixel size of -$28\times 28$, meaning we require $28 \times 28 = 724$ weights to each -neuron in the first hidden layer. - -If we were to analyze images of size $128\times 128$ we would require -$128 \times 128 = 16384$ weights to each neuron. Even worse if we were -dealing with color images, as most images are, we have an image matrix -of size $128\times 128$ for each color dimension (Red, Green, Blue), -meaning 3 times the number of weights $= 49152$ are required for every -single neuron in the first hidden layer. - - -!split -===== Strong correlations ===== - -Images typically have strong local correlations, meaning that a small -part of the image varies little from its neighboring regions. If for -example we have an image of a blue car, we can roughly assume that a -small blue part of the image is surrounded by other blue regions. - -Therefore, instead of connecting every single pixel to a neuron in the -first hidden layer, as we have previously done with deep neural -networks, we can instead connect each neuron to a small part of the -image (in all 3 RGB depth dimensions). The size of each small area is -fixed, and known as a "receptive":"https://en.wikipedia.org/wiki/Receptive_field". - - -!split -===== Layers of a CNN ===== -The layers of a convolutional neural network arrange neurons in 3D: width, height and depth. -The input image is typically a square matrix of depth 3. - -A _convolution_ is performed on the image which outputs -a 3D volume of neurons. The weights to the input are arranged in a number of 2D matrices, known as _filters_. - - -Each filter slides along the input image, taking the dot product -between each small part of the image and the filter, in all depth -dimensions. This is then passed through a non-linear function, -typically the _Rectified Linear (ReLu)_ function, which serves as the -activation of the neurons in the first convolutional layer. This is -further passed through a _pooling layer_, which reduces the size of the -convolutional layer, e.g. by taking the maximum or average across some -small regions, and this serves as input to the next convolutional -layer. - - -!split -===== Systematic reduction ===== - -By systematically reducing the size of the input volume, through -convolution and pooling, the network should create representations of -small parts of the input, and then from them assemble representations -of larger areas. The final pooling layer is flattened to serve as -input to a hidden layer, such that each neuron in the final pooling -layer is connected to every single neuron in the hidden layer. This -then serves as input to the output layer, e.g. a softmax output for -classification. - - -!split -===== Prerequisites: Collect and pre-process data ===== -!bc pycod -# import necessary packages -import numpy as np -import matplotlib.pyplot as plt -from sklearn import datasets - - -# ensure the same random numbers appear every time -np.random.seed(0) - -# display images in notebook -%matplotlib inline -plt.rcParams['figure.figsize'] = (12,12) - - -# download MNIST dataset -digits = datasets.load_digits() - -# define inputs and labels -inputs = digits.images -labels = digits.target - -# RGB images have a depth of 3 -# our images are grayscale so they should have a depth of 1 -inputs = inputs[:,:,:,np.newaxis] - -print("inputs = (n_inputs, pixel_width, pixel_height, depth) = " + str(inputs.shape)) -print("labels = (n_inputs) = " + str(labels.shape)) - - -# choose some random images to display -n_inputs = len(inputs) -indices = np.arange(n_inputs) -random_indices = np.random.choice(indices, size=5) - -for i, image in enumerate(digits.images[random_indices]): - plt.subplot(1, 5, i+1) - plt.axis('off') - plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest') - plt.title("Label: %d" % digits.target[random_indices[i]]) -plt.show() -!ec - - -!split -===== Importing Keras and Tensorflow ===== -!bc pycod -from tensorflow.keras import datasets, layers, models -from tensorflow.keras.layers import Input -from tensorflow.keras.models import Sequential #This allows appending layers to existing models -from tensorflow.keras.layers import Dense #This allows defining the characteristics of a particular layer -from tensorflow.keras import optimizers #This allows using whichever optimiser we want (sgd,adam,RMSprop) -from tensorflow.keras import regularizers #This allows using whichever regularizer we want (l1,l2,l1_l2) -from tensorflow.keras.utils import to_categorical #This allows using categorical cross entropy as the cost function -#from tensorflow.keras import Conv2D -#from tensorflow.keras import MaxPooling2D -#from tensorflow.keras import Flatten - -from sklearn.model_selection import train_test_split - -# representation of labels -labels = to_categorical(labels) - -# split into train and test data -# one-liner from scikit-learn library -train_size = 0.8 -test_size = 1 - train_size -X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size, - test_size=test_size) -!ec - -!split -===== Running with Keras ===== - -!bc pycod -def create_convolutional_neural_network_keras(input_shape, receptive_field, - n_filters, n_neurons_connected, n_categories, - eta, lmbd): - model = Sequential() - model.add(layers.Conv2D(n_filters, (receptive_field, receptive_field), input_shape=input_shape, padding='same', - activation='relu', kernel_regularizer=regularizers.l2(lmbd))) - model.add(layers.MaxPooling2D(pool_size=(2, 2))) - model.add(layers.Flatten()) - model.add(layers.Dense(n_neurons_connected, activation='relu', kernel_regularizer=regularizers.l2(lmbd))) - model.add(layers.Dense(n_categories, activation='softmax', kernel_regularizer=regularizers.l2(lmbd))) - - sgd = optimizers.SGD(lr=eta) - model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy']) - - return model - -epochs = 100 -batch_size = 100 -input_shape = X_train.shape[1:4] -receptive_field = 3 -n_filters = 10 -n_neurons_connected = 50 -n_categories = 10 - -eta_vals = np.logspace(-5, 1, 7) -lmbd_vals = np.logspace(-5, 1, 7) -!ec - -!split -===== Final part ===== - -!bc pycod -CNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object) - -for i, eta in enumerate(eta_vals): - for j, lmbd in enumerate(lmbd_vals): - CNN = create_convolutional_neural_network_keras(input_shape, receptive_field, - n_filters, n_neurons_connected, n_categories, - eta, lmbd) - CNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0) - scores = CNN.evaluate(X_test, Y_test) - - CNN_keras[i][j] = CNN - - print("Learning rate = ", eta) - print("Lambda = ", lmbd) - print("Test accuracy: %.3f" % scores[1]) - print() -!ec - -!split -===== Final visualization ===== - -!bc pycod -# visual representation of grid search -# uses seaborn heatmap, could probably do this in matplotlib -import seaborn as sns - -sns.set() - -train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) -test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) - -for i in range(len(eta_vals)): - for j in range(len(lmbd_vals)): - CNN = CNN_keras[i][j] - - train_accuracy[i][j] = CNN.evaluate(X_train, Y_train)[1] - test_accuracy[i][j] = CNN.evaluate(X_test, Y_test)[1] - - -fig, ax = plt.subplots(figsize = (10, 10)) -sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis") -ax.set_title("Training Accuracy") -ax.set_ylabel("$\eta$") -ax.set_xlabel("$\lambda$") -plt.show() - -fig, ax = plt.subplots(figsize = (10, 10)) -sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis") -ax.set_title("Test Accuracy") -ax.set_ylabel("$\eta$") -ax.set_xlabel("$\lambda$") -plt.show() -!ec - - - -!split -===== The CIFAR01 data set ===== - -The CIFAR10 dataset contains 60,000 color images in 10 classes, with -6,000 images in each class. The dataset is divided into 50,000 -training images and 10,000 testing images. The classes are mutually -exclusive and there is no overlap between them. - -!bc pycod -import tensorflow as tf - -from tensorflow.keras import datasets, layers, models -import matplotlib.pyplot as plt - -# We import the data set -(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data() - -# Normalize pixel values to be between 0 and 1 by dividing by 255. -train_images, test_images = train_images / 255.0, test_images / 255.0 - -!ec - - - -!split -===== Verifying the data set ===== - -To verify that the dataset looks correct, let's plot the first 25 images from the training set and display the class name below each image. - -!bc pycod -class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', - 'dog', 'frog', 'horse', 'ship', 'truck'] -​ -plt.figure(figsize=(10,10)) -for i in range(25): - plt.subplot(5,5,i+1) - plt.xticks([]) - plt.yticks([]) - plt.grid(False) - plt.imshow(train_images[i], cmap=plt.cm.binary) - # The CIFAR labels happen to be arrays, - # which is why you need the extra index - plt.xlabel(class_names[train_labels[i][0]]) -plt.show() -!ec - -!split -===== Set up the model ===== - -The 6 lines of code below define the convolutional base using a common pattern: a stack of Conv2D and MaxPooling2D layers. - -As input, a CNN takes tensors of shape (image_height, image_width, color_channels), ignoring the batch size. If you are new to these dimensions, color_channels refers to (R,G,B). In this example, you will configure our CNN to process inputs of shape (32, 32, 3), which is the format of CIFAR images. You can do this by passing the argument input_shape to our first layer. - -!bc pycod -model = models.Sequential() -model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3))) -model.add(layers.MaxPooling2D((2, 2))) -model.add(layers.Conv2D(64, (3, 3), activation='relu')) -model.add(layers.MaxPooling2D((2, 2))) -model.add(layers.Conv2D(64, (3, 3), activation='relu')) - -# Let's display the architecture of our model so far. - -model.summary() -!ec - -You can see that the output of every Conv2D and MaxPooling2D layer is a 3D tensor of shape (height, width, channels). The width and height dimensions tend to shrink as you go deeper in the network. The number of output channels for each Conv2D layer is controlled by the first argument (e.g., 32 or 64). Typically, as the width and height shrink, you can afford (computationally) to add more output channels in each Conv2D layer. - - - - -!split -===== Add Dense layers on top ===== - -To complete our model, you will feed the last output tensor from the -convolutional base (of shape (4, 4, 64)) into one or more Dense layers -to perform classification. Dense layers take vectors as input (which -are 1D), while the current output is a 3D tensor. First, you will -flatten (or unroll) the 3D output to 1D, then add one or more Dense -layers on top. CIFAR has 10 output classes, so you use a final Dense -layer with 10 outputs and a softmax activation. - -!bc pycod -model.add(layers.Flatten()) -model.add(layers.Dense(64, activation='relu')) -model.add(layers.Dense(10)) -Here's the complete architecture of our model. - -model.summary() -!ec -As you can see, our (4, 4, 64) outputs were flattened into vectors of shape (1024) before going through two Dense layers. - -!split -===== Compile and train the model ===== - -!bc pycod -model.compile(optimizer='adam', - loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), - metrics=['accuracy']) -​ -history = model.fit(train_images, train_labels, epochs=10, - validation_data=(test_images, test_labels)) - -!ec - - -!split -===== Finally, evaluate the model ===== - -!bc pycod -plt.plot(history.history['accuracy'], label='accuracy') -plt.plot(history.history['val_accuracy'], label = 'val_accuracy') -plt.xlabel('Epoch') -plt.ylabel('Accuracy') -plt.ylim([0.5, 1]) -plt.legend(loc='lower right') - -test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2) - -print(test_acc) - -!ec - !split ===== Building our own CNN code ===== @@ -4103,3 +3529,258 @@ to remain cognizant of this fact when utilizing FFT as the primary optimization technique. + +!split +===== Building convolutional neural networks in Tensorflow and Keras ===== + + +As discussed above, CNNs are neural networks built from the assumption that the inputs +to the network are 2D images. This is important because the number of features or pixels in images +grows very fast with the image size, and an enormous number of weights and biases are needed in order to build an accurate network. + +As before, we still have our input, a hidden layer and an output. What's novel about convolutional networks +are the _convolutional_ and _pooling_ layers stacked in pairs between the input and the hidden layer. +In addition, the data is no longer represented as a 2D feature matrix, instead each input is a number of 2D +matrices, typically 1 for each color dimension (Red, Green, Blue). + + +!split +===== Setting it up ===== + +It means that to represent the entire +dataset of images, we require a 4D matrix or _tensor_. This tensor has the dimensions: +!bt +\[ +(n_{inputs},\, n_{pixels, width},\, n_{pixels, height},\, depth) . +\] +!et + +!split +===== The MNIST dataset again ===== + +The MNIST dataset consists of grayscale images with a pixel size of +$28\times 28$, meaning we require $28 \times 28 = 724$ weights to each +neuron in the first hidden layer. + +If we were to analyze images of size $128\times 128$ we would require +$128 \times 128 = 16384$ weights to each neuron. Even worse if we were +dealing with color images, as most images are, we have an image matrix +of size $128\times 128$ for each color dimension (Red, Green, Blue), +meaning 3 times the number of weights $= 49152$ are required for every +single neuron in the first hidden layer. + + +!split +===== Strong correlations ===== + +Images typically have strong local correlations, meaning that a small +part of the image varies little from its neighboring regions. If for +example we have an image of a blue car, we can roughly assume that a +small blue part of the image is surrounded by other blue regions. + +Therefore, instead of connecting every single pixel to a neuron in the +first hidden layer, as we have previously done with deep neural +networks, we can instead connect each neuron to a small part of the +image (in all 3 RGB depth dimensions). The size of each small area is +fixed, and known as a "receptive":"https://en.wikipedia.org/wiki/Receptive_field". + + +!split +===== Layers of a CNN ===== + +The layers of a convolutional neural network arrange neurons in 3D: width, height and depth. +The input image is typically a square matrix of depth 3. + +A _convolution_ is performed on the image which outputs +a 3D volume of neurons. The weights to the input are arranged in a number of 2D matrices, known as _filters_. + + +Each filter slides along the input image, taking the dot product +between each small part of the image and the filter, in all depth +dimensions. This is then passed through a non-linear function, +typically the _Rectified Linear (ReLu)_ function, which serves as the +activation of the neurons in the first convolutional layer. This is +further passed through a _pooling layer_, which reduces the size of the +convolutional layer, e.g. by taking the maximum or average across some +small regions, and this serves as input to the next convolutional +layer. + + +!split +===== Systematic reduction ===== + +By systematically reducing the size of the input volume, through +convolution and pooling, the network should create representations of +small parts of the input, and then from them assemble representations +of larger areas. The final pooling layer is flattened to serve as +input to a hidden layer, such that each neuron in the final pooling +layer is connected to every single neuron in the hidden layer. This +then serves as input to the output layer, e.g. a softmax output for +classification. + + +!split +===== Prerequisites: Collect and pre-process data ===== +!bc pycod +# import necessary packages +import numpy as np +import matplotlib.pyplot as plt +from sklearn import datasets + + +# ensure the same random numbers appear every time +np.random.seed(0) + +# display images in notebook +%matplotlib inline +plt.rcParams['figure.figsize'] = (12,12) + + +# download MNIST dataset +digits = datasets.load_digits() + +# define inputs and labels +inputs = digits.images +labels = digits.target + +# RGB images have a depth of 3 +# our images are grayscale so they should have a depth of 1 +inputs = inputs[:,:,:,np.newaxis] + +print("inputs = (n_inputs, pixel_width, pixel_height, depth) = " + str(inputs.shape)) +print("labels = (n_inputs) = " + str(labels.shape)) + + +# choose some random images to display +n_inputs = len(inputs) +indices = np.arange(n_inputs) +random_indices = np.random.choice(indices, size=5) + +for i, image in enumerate(digits.images[random_indices]): + plt.subplot(1, 5, i+1) + plt.axis('off') + plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest') + plt.title("Label: %d" % digits.target[random_indices[i]]) +plt.show() +!ec + + +!split +===== Importing Keras and Tensorflow ===== +!bc pycod +from tensorflow.keras import datasets, layers, models +from tensorflow.keras.layers import Input +from tensorflow.keras.models import Sequential #This allows appending layers to existing models +from tensorflow.keras.layers import Dense #This allows defining the characteristics of a particular layer +from tensorflow.keras import optimizers #This allows using whichever optimiser we want (sgd,adam,RMSprop) +from tensorflow.keras import regularizers #This allows using whichever regularizer we want (l1,l2,l1_l2) +from tensorflow.keras.utils import to_categorical #This allows using categorical cross entropy as the cost function +#from tensorflow.keras import Conv2D +#from tensorflow.keras import MaxPooling2D +#from tensorflow.keras import Flatten + +from sklearn.model_selection import train_test_split + +# representation of labels +labels = to_categorical(labels) + +# split into train and test data +# one-liner from scikit-learn library +train_size = 0.8 +test_size = 1 - train_size +X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size, + test_size=test_size) +!ec + +!split +===== Running with Keras ===== + +!bc pycod +def create_convolutional_neural_network_keras(input_shape, receptive_field, + n_filters, n_neurons_connected, n_categories, + eta, lmbd): + model = Sequential() + model.add(layers.Conv2D(n_filters, (receptive_field, receptive_field), input_shape=input_shape, padding='same', + activation='relu', kernel_regularizer=regularizers.l2(lmbd))) + model.add(layers.MaxPooling2D(pool_size=(2, 2))) + model.add(layers.Flatten()) + model.add(layers.Dense(n_neurons_connected, activation='relu', kernel_regularizer=regularizers.l2(lmbd))) + model.add(layers.Dense(n_categories, activation='softmax', kernel_regularizer=regularizers.l2(lmbd))) + + sgd = optimizers.SGD(learning_rate=eta) + model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy']) + + return model + +epochs = 100 +batch_size = 100 +input_shape = X_train.shape[1:4] +receptive_field = 3 +n_filters = 10 +n_neurons_connected = 50 +n_categories = 10 + +eta_vals = np.logspace(-5, 1, 7) +lmbd_vals = np.logspace(-5, 1, 7) +!ec + +!split +===== Final part ===== + +!bc pycod +CNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object) + +for i, eta in enumerate(eta_vals): + for j, lmbd in enumerate(lmbd_vals): + CNN = create_convolutional_neural_network_keras(input_shape, receptive_field, + n_filters, n_neurons_connected, n_categories, + eta, lmbd) + CNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0) + scores = CNN.evaluate(X_test, Y_test) + + CNN_keras[i][j] = CNN + + print("Learning rate = ", eta) + print("Lambda = ", lmbd) + print("Test accuracy: %.3f" % scores[1]) + print() +!ec + +!split +===== Final visualization ===== + +!bc pycod +# visual representation of grid search +# uses seaborn heatmap, could probably do this in matplotlib +import seaborn as sns + +sns.set() + +train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) +test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) + +for i in range(len(eta_vals)): + for j in range(len(lmbd_vals)): + CNN = CNN_keras[i][j] + + train_accuracy[i][j] = CNN.evaluate(X_train, Y_train)[1] + test_accuracy[i][j] = CNN.evaluate(X_test, Y_test)[1] + + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Training Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Test Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() +!ec + + +