+The textbook by Goodfellow et al, see chapter 9 contains an in depth discussion as well. +
@@ -448,7 +453,7 @@ and the slides of 68
-The mathematics of CNNs is based on the mathematical operation of -convolution. In mathematics (in particular in functional analysis), -convolution is represented by matheematical operation (integration, -summation etc) on two function in order to produce a third function -that expresses how the shape of one gets modified by the other. -Convolution has a plethora of applications in a variety of disciplines, spanning from statistics to signal processing, computer vision, solutions of differential equations,linear algebra, engineering, and yes, machine learning. +A dense neural network is representd by an affine operation (like matrix-matrix multiplication) where all parameters are included.
-Mathematically, convolution is defined as follows (one-dimensional example): -Let us define a continuous function \( y(t) \) given by -$$ -y(t) = \int x(a) w(t-a) da, -$$ - -where \( x(a) \) represents a so-called input and \( w(t-a) \) is normally called the weight function or kernel. +The key idea in CNNs for say imaging is that in images neighbor pixels tend to be related! So we connect +only neighboring neurons in the input instead of connecting all with the first hidden layer.
-The above integral is written in a more compact form as -$$ -y(t) = \left(x * w\right)(t). -$$ - -
-The discretized version reads -$$ -y(t) = \sum_{a=-\infty}^{a=\infty}x(a)w(t-a). -$$ - -Computing the inverse of the above convolution operations is known as deconvolution. - -
-How can we use this? And what does it mean? Let us study some familiar examples first. +We say we perform a filtering (convolution is the mathematical operation).
@@ -466,7 +444,7 @@ How can we use this? And what does it mean? Let us study some familiar examples
-We have already met such an example in project 1 when we tried to set -up the design matrix for a two-dimensional function. This was an -example of polynomial multiplication. Let us recast such a problem in terms of the convolution operation. -Let us look a the following polynomials to second and third order, respectively: +The mathematics of CNNs is based on the mathematical operation of +convolution. In mathematics (in particular in functional analysis), +convolution is represented by matheematical operation (integration, +summation etc) on two function in order to produce a third function +that expresses how the shape of one gets modified by the other. +Convolution has a plethora of applications in a variety of disciplines, spanning from statistics to signal processing, computer vision, solutions of differential equations,linear algebra, engineering, and yes, machine learning. + +
+Mathematically, convolution is defined as follows (one-dimensional example): +Let us define a continuous function \( y(t) \) given by $$ -p(t) = \alpha_0+\alpha_1 t+\alpha_2 t^2, +y(t) = \int x(a) w(t-a) da, $$ -and +where \( x(a) \) represents a so-called input and \( w(t-a) \) is normally called the weight function or kernel. + +
+The above integral is written in a more compact form as $$ -s(t) = \beta_0+\beta_1 t+\beta_2 t^2+\beta_3 t^3. +y(t) = \left(x * w\right)(t). $$
-The polynomial multiplication gives us a new polynomial of degree \( 5 \) +The discretized version reads $$ -z(t) = \delta_0+\delta_1 t+\delta_2 t^2+\delta_3 t^3+\delta_4 t^4+\delta_5 t^5. +y(t) = \sum_{a=-\infty}^{a=\infty}x(a)w(t-a). $$ +Computing the inverse of the above convolution operations is known as deconvolution. + +
+How can we use this? And what does it mean? Let us study some familiar examples first. +
@@ -452,7 +468,7 @@ $$
-Computing polynomial products can be implemented efficiently if we rewrite the more brute force multiplications using convolution. -We note first that the new coefficients are given as - +We have already met such an example in project 1 when we tried to set +up the design matrix for a two-dimensional function. This was an +example of polynomial multiplication. Let us recast such a problem in terms of the convolution operation. +Let us look a the following polynomials to second and third order, respectively: $$ -\begin{split} -\delta_0=&\alpha_0\beta_0\\ -\delta_1=&\alpha_1\beta_0+\alpha_1\beta_0\\ -\delta_2=&\alpha_0\beta_2+\alpha_1\beta_1+\alpha_2\beta_0\\ -\delta_3=&\alpha_1\beta_2+\alpha_2\beta_1+\alpha_0\beta_3\\ -\delta_4=&\alpha_2\beta_2+\alpha_1\beta_3\\ -\delta_5=&\alpha_2\beta_3.\\ -\end{split} +p(t) = \alpha_0+\alpha_1 t+\alpha_2 t^2, +$$ + +and +$$ +s(t) = \beta_0+\beta_1 t+\beta_2 t^2+\beta_3 t^3. $$
-We note that \( \alpha_i=0 \) except for \( i\in \left\{0,1,2\right\} \) and \( \beta_i=0 \) except for \( i\in\left\{0,1,2,3\right\} \). - -
-We can then rewrite the coefficients \( \delta_j \) using a discrete convolution as +The polynomial multiplication gives us a new polynomial of degree \( 5 \) $$ -\delta_j = \sum_{i=-\infty}^{i=\infty}\alpha_i\beta_{j-i}=(\alpha * \beta)_j, +z(t) = \delta_0+\delta_1 t+\delta_2 t^2+\delta_3 t^3+\delta_4 t^4+\delta_5 t^5. $$ -or as a double sum with restriction \( l=i+j \) -$$ -\delta_l = \sum_{ij}\alpha_i\beta_{j}. -$$ - -
-Do you see a potential drawback with these equations? -
@@ -464,7 +454,7 @@ Do you see a potential drawback with these equations?
-Since we only have a finite number of \( \alpha \) and \( \beta \) values -which are non-zero, we can rewrite the above convolution expressions -as a matrix-vector multiplication +Computing polynomial products can be implemented efficiently if we rewrite the more brute force multiplications using convolution. +We note first that the new coefficients are given as $$ -\boldsymbol{\delta}=\begin{bmatrix}\alpha_0 & 0 & 0 & 0 \\ - \alpha_1 & \alpha_0 & 0 & 0 \\ - \alpha_2 & \alpha_1 & \alpha_0 & 0 \\ - 0 & \alpha_2 & \alpha_1 & \alpha_0 \\ - 0 & 0 & \alpha_2 & \alpha_1 \\ - 0 & 0 & 0 & \alpha_2 - \end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3\end{bmatrix}. +\begin{split} +\delta_0=&\alpha_0\beta_0\\ +\delta_1=&\alpha_1\beta_0+\alpha_1\beta_0\\ +\delta_2=&\alpha_0\beta_2+\alpha_1\beta_1+\alpha_2\beta_0\\ +\delta_3=&\alpha_1\beta_2+\alpha_2\beta_1+\alpha_0\beta_3\\ +\delta_4=&\alpha_2\beta_2+\alpha_1\beta_3\\ +\delta_5=&\alpha_2\beta_3.\\ +\end{split} $$
-The process is commutative and we can easily see that we can rewrite the multiplication in terms of a matrix holding \( \beta \) and a vector holding \( \alpha \). -In this case we have +We note that \( \alpha_i=0 \) except for \( i\in \left\{0,1,2\right\} \) and \( \beta_i=0 \) except for \( i\in\left\{0,1,2,3\right\} \). + +
+We can then rewrite the coefficients \( \delta_j \) using a discrete convolution as $$ -\boldsymbol{\delta}=\begin{bmatrix}\beta_0 & 0 & 0 \\ - \beta_1 & \beta_0 & 0 \\ - \beta_2 & \beta_1 & \beta_0 \\ - \beta_3 & \beta_2 & \beta_1 \\ - 0 & \beta_3 & \beta_2 \\ - 0 & 0 & \beta_3 - \end{bmatrix}\begin{bmatrix} \alpha_0 \\ \alpha_1 \\ \alpha_2\end{bmatrix}. +\delta_j = \sum_{i=-\infty}^{i=\infty}\alpha_i\beta_{j-i}=(\alpha * \beta)_j, +$$ + +or as a double sum with restriction \( l=i+j \) +$$ +\delta_l = \sum_{ij}\alpha_i\beta_{j}. $$
-Note that the use of these matrices is for mathematical purposes only and not implementation purposes. -When implementing the above equation we do not encode (and allocate memory) the matrices explicitely. -We rather code the convolutions in the minimal memory footprint that they require. - -
-Does the number of floating point operations change here when we use the commutative property? +Do you see a potential drawback with these equations?
@@ -468,7 +466,7 @@ Does the number of floating point operations change here when we use the commuta
-For problems with so-called harmonic oscillations, given by for example the following differential equation -$$ -m\frac{d^2x}{dt^2}+\eta\frac{dx}{dt}+x(t)=F(t), -$$ +Since we only have a finite number of \( \alpha \) and \( \beta \) values +which are non-zero, we can rewrite the above convolution expressions +as a matrix-vector multiplication -where \( F(t) \) is an applied external force acting on the system (often called a driving force), one can use the theory of Fourier transformations to find the solutions of this type of equations. +$$ +\boldsymbol{\delta}=\begin{bmatrix}\alpha_0 & 0 & 0 & 0 \\ + \alpha_1 & \alpha_0 & 0 & 0 \\ + \alpha_2 & \alpha_1 & \alpha_0 & 0 \\ + 0 & \alpha_2 & \alpha_1 & \alpha_0 \\ + 0 & 0 & \alpha_2 & \alpha_1 \\ + 0 & 0 & 0 & \alpha_2 + \end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3\end{bmatrix}. +$$
-If one has several driving forces, \( F(t)=\sum_n F_n(t) \), one can find -the particular solution to each \( F_n \), \( x_{pn}(t) \), and the particular -solution for the entire driving force is then given by a series like +The process is commutative and we can easily see that we can rewrite the multiplication in terms of a matrix holding \( \beta \) and a vector holding \( \alpha \). +In this case we have +$$ +\boldsymbol{\delta}=\begin{bmatrix}\beta_0 & 0 & 0 \\ + \beta_1 & \beta_0 & 0 \\ + \beta_2 & \beta_1 & \beta_0 \\ + \beta_3 & \beta_2 & \beta_1 \\ + 0 & \beta_3 & \beta_2 \\ + 0 & 0 & \beta_3 + \end{bmatrix}\begin{bmatrix} \alpha_0 \\ \alpha_1 \\ \alpha_2\end{bmatrix}. +$$ -$$ -\begin{equation} -x_p(t)=\sum_nx_{pn}(t). -\tag{21} -\end{equation} -$$ +
+Note that the use of these matrices is for mathematical purposes only and not implementation purposes. +When implementing the above equation we do not encode (and allocate memory) the matrices explicitely. +We rather code the convolutions in the minimal memory footprint that they require. + +
+Does the number of floating point operations change here when we use the commutative property?
@@ -452,7 +470,7 @@ $$
-This is known as the principle of superposition. It only applies when -the homogenous equation is linear. If there were an anharmonic term -such as \( x^3 \) in the homogenous equation, then when one summed various -solutions, \( x=(\sum_n x_n)^2 \), one would get cross -terms. Superposition is especially useful when \( F(t) \) can be written -as a sum of sinusoidal terms, because the solutions for each -sinusoidal (sine or cosine) term is analytic. - -
-Driving forces are often periodic, even when they are not -sinusoidal. Periodicity implies that for some time \( \tau \) - +For problems with so-called harmonic oscillations, given by for example the following differential equation $$ -\begin{eqnarray} -F(t+\tau)=F(t). -\end{eqnarray} +m\frac{d^2x}{dt^2}+\eta\frac{dx}{dt}+x(t)=F(t), $$ +where \( F(t) \) is an applied external force acting on the system (often called a driving force), one can use the theory of Fourier transformations to find the solutions of this type of equations. +
-One example of a non-sinusoidal periodic force is a square wave. Many -components in electric circuits are non-linear, e.g. diodes, which -makes many wave forms non-sinusoidal even when the circuits are being -driven by purely sinusoidal sources. +If one has several driving forces, \( F(t)=\sum_n F_n(t) \), one can find +the particular solution to each \( F_n \), \( x_{pn}(t) \), and the particular +solution for the entire driving force is then given by a series like + +$$ +\begin{equation} +x_p(t)=\sum_nx_{pn}(t). +\tag{21} +\end{equation} +$$
@@ -457,7 +454,7 @@ driven by purely sinusoidal sources.
-The code here shows a typical example of such a square wave generated using the functionality included in the scipy Python package. We have used a period of \( \tau=0.2 \). +This is known as the principle of superposition. It only applies when +the homogenous equation is linear. If there were an anharmonic term +such as \( x^3 \) in the homogenous equation, then when one summed various +solutions, \( x=(\sum_n x_n)^2 \), one would get cross +terms. Superposition is especially useful when \( F(t) \) can be written +as a sum of sinusoidal terms, because the solutions for each +sinusoidal (sine or cosine) term is analytic.
- - -
import numpy as np
-import math
-from scipy import signal
-import matplotlib.pyplot as plt
-
-# number of points
-n = 500
-# start and final times
-t0 = 0.0
-tn = 1.0
-# Period
-t = np.linspace(t0, tn, n, endpoint=False)
-SqrSignal = np.zeros(n)
-SqrSignal = 1.0+signal.square(2*np.pi*5*t)
-plt.plot(t, SqrSignal)
-plt.ylim(-0.5, 2.5)
-plt.show()
--For the sinusoidal example the -period is \( \tau=2\pi/\omega \). However, higher harmonics can also -satisfy the periodicity requirement. In general, any force that -satisfies the periodicity requirement can be expressed as a sum over -harmonics, +Driving forces are often periodic, even when they are not +sinusoidal. Periodicity implies that for some time \( \tau \) $$ -\begin{equation} -F(t)=\frac{f_0}{2}+\sum_{n>0} f_n\cos(2n\pi t/\tau)+g_n\sin(2n\pi t/\tau). -\tag{22} -\end{equation} +\begin{eqnarray} +F(t+\tau)=F(t). +\end{eqnarray} $$ +
+One example of a non-sinusoidal periodic force is a square wave. Many +components in electric circuits are non-linear, e.g. diodes, which +makes many wave forms non-sinusoidal even when the circuits are being +driven by purely sinusoidal sources. +
@@ -470,7 +459,7 @@ $$
-We can write down the answer for -\( x_{pn}(t) \), by substituting \( f_n/m \) or \( g_n/m \) for \( F_0/m \). By -writing each factor \( 2n\pi t/\tau \) as \( n\omega t \), with \( \omega\equiv -2\pi/\tau \), +The code here shows a typical example of such a square wave generated using the functionality included in the scipy Python package. We have used a period of \( \tau=0.2 \). + +
+ + +
import numpy as np
+import math
+from scipy import signal
+import matplotlib.pyplot as plt
+
+# number of points
+n = 500
+# start and final times
+t0 = 0.0
+tn = 1.0
+# Period
+t = np.linspace(t0, tn, n, endpoint=False)
+SqrSignal = np.zeros(n)
+SqrSignal = 1.0+signal.square(2*np.pi*5*t)
+plt.plot(t, SqrSignal)
+plt.ylim(-0.5, 2.5)
+plt.show()
++For the sinusoidal example the +period is \( \tau=2\pi/\omega \). However, higher harmonics can also +satisfy the periodicity requirement. In general, any force that +satisfies the periodicity requirement can be expressed as a sum over +harmonics, $$ \begin{equation} -\tag{23} -F(t)=\frac{f_0}{2}+\sum_{n>0}f_n\cos(n\omega t)+g_n\sin(n\omega t). +F(t)=\frac{f_0}{2}+\sum_{n>0} f_n\cos(2n\pi t/\tau)+g_n\sin(2n\pi t/\tau). +\tag{22} \end{equation} $$ -
-The solutions for \( x(t) \) then come from replacing \( \omega \) with -\( n\omega \) for each term in the particular solution, - -$$ -\begin{eqnarray} -x_p(t)&=&\frac{f_0}{2k}+\sum_{n>0} \alpha_n\cos(n\omega t-\delta_n)+\beta_n\sin(n\omega t-\delta_n),\\ -\nonumber -\alpha_n&=&\frac{f_n/m}{\sqrt{((n\omega)^2-\omega_0^2)+4\beta^2n^2\omega^2}},\\ -\nonumber -\beta_n&=&\frac{g_n/m}{\sqrt{((n\omega)^2-\omega_0^2)+4\beta^2n^2\omega^2}},\\ -\nonumber -\delta_n&=&\tan^{-1}\left(\frac{2\beta n\omega}{\omega_0^2-n^2\omega^2}\right). -\end{eqnarray} -$$ -
@@ -461,7 +472,7 @@ $$
-Because the forces have been applied for a long time, any non-zero -damping eliminates the homogenous parts of the solution, so one need -only consider the particular solution for each \( n \). - -
-The problem is considered solved if one can find expressions for the -coefficients \( f_n \) and \( g_n \), even though the solutions are expressed -as an infinite sum. The coefficients can be extracted from the -function \( F(t) \) by - -$$ -\begin{eqnarray} -\tag{24} -f_n&=&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~F(t)\cos(2n\pi t/\tau),\\ -\nonumber -g_n&=&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~F(t)\sin(2n\pi t/\tau). -\end{eqnarray} -$$ - -
-To check the consistency of these expressions and to verify -Eq. (24), one can insert the expansion of \( F(t) \) in -Eq. (23) into the expression for the coefficients in -Eq. (24) and see whether - -$$ -\begin{eqnarray} -f_n&=?&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~\left\{ -\frac{f_0}{2}+\sum_{m>0}f_m\cos(m\omega t)+g_m\sin(m\omega t) -\right\}\cos(n\omega t). -\end{eqnarray} -$$ - -
-Immediately, one can throw away all the terms with \( g_m \) because they -convolute an even and an odd function. The term with \( f_0/2 \) -disappears because \( \cos(n\omega t) \) is equally positive and negative -over the interval and will integrate to zero. For all the terms -\( f_m\cos(m\omega t) \) appearing in the sum, one can use angle addition -formulas to see that \( \cos(m\omega t)\cos(n\omega -t)=(1/2)(\cos[(m+n)\omega t]+\cos[(m-n)\omega t] \). This will integrate -to zero unless \( m=n \). In that case the \( m=n \) term gives +We can write down the answer for +\( x_{pn}(t) \), by substituting \( f_n/m \) or \( g_n/m \) for \( F_0/m \). By +writing each factor \( 2n\pi t/\tau \) as \( n\omega t \), with \( \omega\equiv +2\pi/\tau \), $$ \begin{equation} -\int_{-\tau/2}^{\tau/2}dt~\cos^2(m\omega t)=\frac{\tau}{2}, -\tag{25} +\tag{23} +F(t)=\frac{f_0}{2}+\sum_{n>0}f_n\cos(n\omega t)+g_n\sin(n\omega t). \end{equation} $$
-and +The solutions for \( x(t) \) then come from replacing \( \omega \) with +\( n\omega \) for each term in the particular solution, $$ \begin{eqnarray} -f_n&=?&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~f_n/2\\ +x_p(t)&=&\frac{f_0}{2k}+\sum_{n>0} \alpha_n\cos(n\omega t-\delta_n)+\beta_n\sin(n\omega t-\delta_n),\\ \nonumber -&=&f_n~\checkmark. +\alpha_n&=&\frac{f_n/m}{\sqrt{((n\omega)^2-\omega_0^2)+4\beta^2n^2\omega^2}},\\ +\nonumber +\beta_n&=&\frac{g_n/m}{\sqrt{((n\omega)^2-\omega_0^2)+4\beta^2n^2\omega^2}},\\ +\nonumber +\delta_n&=&\tan^{-1}\left(\frac{2\beta n\omega}{\omega_0^2-n^2\omega^2}\right). \end{eqnarray} $$ -
-The same method can be used to check for the consistency of \( g_n \). -
@@ -497,7 +463,7 @@ The same method can be used to check for the consistency of \( g_n \).
-The code here uses the Fourier series applied to a -square wave signal. The code here -visualizes the various approximations given by Fourier series compared -with a square wave with period \( T=0.2 \) (dimensionless time), width \( 0.1 \) and max value of the force \( F=2 \). We -see that when we increase the number of components in the Fourier -series, the Fourier series approximation gets closer and closer to the -square wave signal. +Because the forces have been applied for a long time, any non-zero +damping eliminates the homogenous parts of the solution, so one need +only consider the particular solution for each \( n \).
+The problem is considered solved if one can find expressions for the +coefficients \( f_n \) and \( g_n \), even though the solutions are expressed +as an infinite sum. The coefficients can be extracted from the +function \( F(t) \) by - -
import numpy as np
-import math
-from scipy import signal
-import matplotlib.pyplot as plt
+$$
+\begin{eqnarray}
+\tag{24}
+f_n&=&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~F(t)\cos(2n\pi t/\tau),\\
+\nonumber
+g_n&=&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~F(t)\sin(2n\pi t/\tau).
+\end{eqnarray}
+$$
+
+
+To check the consistency of these expressions and to verify
+Eq. (24), one can insert the expansion of \( F(t) \) in
+Eq. (23) into the expression for the coefficients in
+Eq. (24) and see whether
+
+$$
+\begin{eqnarray}
+f_n&=?&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~\left\{
+\frac{f_0}{2}+\sum_{m>0}f_m\cos(m\omega t)+g_m\sin(m\omega t)
+\right\}\cos(n\omega t).
+\end{eqnarray}
+$$
+
+
+Immediately, one can throw away all the terms with \( g_m \) because they
+convolute an even and an odd function. The term with \( f_0/2 \)
+disappears because \( \cos(n\omega t) \) is equally positive and negative
+over the interval and will integrate to zero. For all the terms
+\( f_m\cos(m\omega t) \) appearing in the sum, one can use angle addition
+formulas to see that \( \cos(m\omega t)\cos(n\omega
+t)=(1/2)(\cos[(m+n)\omega t]+\cos[(m-n)\omega t] \). This will integrate
+to zero unless \( m=n \). In that case the \( m=n \) term gives
+
+$$
+\begin{equation}
+\int_{-\tau/2}^{\tau/2}dt~\cos^2(m\omega t)=\frac{\tau}{2},
+\tag{25}
+\end{equation}
+$$
+
+
+and
+
+$$
+\begin{eqnarray}
+f_n&=?&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~f_n/2\\
+\nonumber
+&=&f_n~\checkmark.
+\end{eqnarray}
+$$
+
+
+The same method can be used to check for the consistency of \( g_n \).
-# number of points
-n = 500
-# start and final times
-t0 = 0.0
-tn = 1.0
-# Period
-T =0.2
-# Max value of square signal
-Fmax= 2.0
-# Width of signal
-Width = 0.1
-t = np.linspace(t0, tn, n, endpoint=False)
-SqrSignal = np.zeros(n)
-FourierSeriesSignal = np.zeros(n)
-SqrSignal = 1.0+signal.square(2*np.pi*5*t+np.pi*Width/T)
-a0 = Fmax*Width/T
-FourierSeriesSignal = a0
-Factor = 2.0*Fmax/np.pi
-for i in range(1,500):
- FourierSeriesSignal += Factor/(i)*np.sin(np.pi*i*Width/T)*np.cos(i*t*2*np.pi/T)
-plt.plot(t, SqrSignal)
-plt.plot(t, FourierSeriesSignal)
-plt.ylim(-0.5, 2.5)
-plt.show()
-
@@ -474,7 +499,7 @@ plt.show()
-We often use convolutions over more than one dimension at a time. If -we have a two-dimensional image \( I \) as input, we can have a filter -defined by a two-dimensional kernel \( K \). This leads to an output \( S \) - -$$ -S_(i,j)=(I * K)(i,j) = \sum_m\sum_n I(m,n)K(i-m,j-n). -$$ +The code here uses the Fourier series applied to a +square wave signal. The code here +visualizes the various approximations given by Fourier series compared +with a square wave with period \( T=0.2 \) (dimensionless time), width \( 0.1 \) and max value of the force \( F=2 \). We +see that when we increase the number of components in the Fourier +series, the Fourier series approximation gets closer and closer to the +square wave signal.
-Convolution is a commutatitave process, which means we can rewrite this equation as -$$ -S_(i,j)=(I * K)(i,j) = \sum_m\sum_n I(i-m,j-n)K(m,n). -$$ -
-Normally the latter is more straightforward to implement in a machine elarning library since there is less variation in the range of values of \( m \) and \( n \). + +
import numpy as np
+import math
+from scipy import signal
+import matplotlib.pyplot as plt
+# number of points
+n = 500
+# start and final times
+t0 = 0.0
+tn = 1.0
+# Period
+T =0.2
+# Max value of square signal
+Fmax= 2.0
+# Width of signal
+Width = 0.1
+t = np.linspace(t0, tn, n, endpoint=False)
+SqrSignal = np.zeros(n)
+FourierSeriesSignal = np.zeros(n)
+SqrSignal = 1.0+signal.square(2*np.pi*5*t+np.pi*Width/T)
+a0 = Fmax*Width/T
+FourierSeriesSignal = a0
+Factor = 2.0*Fmax/np.pi
+for i in range(1,500):
+ FourierSeriesSignal += Factor/(i)*np.sin(np.pi*i*Width/T)*np.cos(i*t*2*np.pi/T)
+plt.plot(t, SqrSignal)
+plt.plot(t, FourierSeriesSignal)
+plt.ylim(-0.5, 2.5)
+plt.show()
+
@@ -450,7 +476,7 @@ Normally the latter is more straightforward to implement in a machine elarning
-Many deep learning libraries implement cross-correlation instead of convolution +We often use convolutions over more than one dimension at a time. If +we have a two-dimensional image \( I \) as input, we can have a filter +defined by a two-dimensional kernel \( K \). This leads to an output \( S \) + $$ -S_(i,j)=(I * K)(i,j) = \sum_m\sum_n I(i+m,j-+)K(m,n). +S_(i,j)=(I * K)(i,j) = \sum_m\sum_n I(m,n)K(i-m,j-n). $$ +
+Convolution is a commutatitave process, which means we can rewrite this equation as +$$ +S_(i,j)=(I * K)(i,j) = \sum_m\sum_n I(i-m,j-n)K(m,n). +$$ + +
+Normally the latter is more straightforward to implement in a machine elarning library since there is less variation in the range of values of \( m \) and \( n \). +
@@ -438,7 +452,7 @@ $$
-In feilds like signal processing (and imaging as well), one designs -so-called filters. These filters are defined by the convolutions and -are often hand-crafted. One may specify filters for smoothing, edge -detection, frequency reshaping, and similar operations. However with -neural networks the idea is to automatically learn the filters and use -many of them in conjunction with non-linear operations (activation -functions). - -
-As an example consider a neural network operating on sound sequence -data. Assume that we an input vector \( \boldsymbol{x} \) of length \( d=10^6 \). We -construct then a neural network with onle hidden layer only with -\( 10^4 \) nodes. This means that we will have a weight matrix with -\( 10^4\times 10^6=10^{10} \) weights to be determined, together with \( 10^4 \) biases. - -
-Assume furthermore that we have an output layer which is meant to train whether the sound sequence represents a human voice (true) or something else (false). -It means that we have only one output node. But since this output node connects to \( 10^4 \) nodes in the hidden layer, there are in total \( 10^4 \) weights to be determined for the output layer, plus one bias. In total we have - +Many deep learning libraries implement cross-correlation instead of convolution $$ -\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \approx 10^{10}, +S_(i,j)=(I * K)(i,j) = \sum_m\sum_n I(i+m,j-+)K(m,n). $$ -that is ten billion parameters to determine. -
@@ -458,7 +440,7 @@ that is ten billion parameters to determine.
-In today’s architecture one can train such neural networks, however -this is a huge number of parameters for the task at hand. In general, -it is a very wasteful and inefficient use of dense matrices as -parameters. Just as importantly, such trained network parameters are -very specific for the type of input data on which they were trained -and the network is not likely to generalize easily to variations in -the input. +In feilds like signal processing (and imaging as well), one designs +so-called filters. These filters are defined by the convolutions and +are often hand-crafted. One may specify filters for smoothing, edge +detection, frequency reshaping, and similar operations. However with +neural networks the idea is to automatically learn the filters and use +many of them in conjunction with non-linear operations (activation +functions).
-The main principles that justify convolutions is locality of -information and repetion of patterns within the signal. Sound samples -of the input in adjacent spots are much more likely to affect each -other than those that are very far away. Similarly, sounds are -repeated in multiple times in the signal. While slightly simplistic, -reasoning about such a sound example demonstrates this. The same -principles then apply to images and other similar data. +As an example consider a neural network operating on sound sequence +data. Assume that we an input vector \( \boldsymbol{x} \) of length \( d=10^6 \). We +construct then a neural network with onle hidden layer only with +\( 10^4 \) nodes. This means that we will have a weight matrix with +\( 10^4\times 10^6=10^{10} \) weights to be determined, together with \( 10^4 \) biases. + +
+Assume furthermore that we have an output layer which is meant to train whether the sound sequence represents a human voice (true) or something else (false). +It means that we have only one output node. But since this output node connects to \( 10^4 \) nodes in the hidden layer, there are in total \( 10^4 \) weights to be determined for the output layer, plus one bias. In total we have + +$$ +\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \approx 10^{10}, +$$ + +that is ten billion parameters to determine.
@@ -450,7 +460,7 @@ principles then apply to images and other similar data.
-As discussed above, CNNs are neural networks built from the assumption that the inputs -to the network are 2D images. This is important because the number of features or pixels in images -grows very fast with the image size, and an enormous number of weights and biases are needed in order to build an accurate network. +In today’s architecture one can train such neural networks, however +this is a huge number of parameters for the task at hand. In general, +it is a very wasteful and inefficient use of dense matrices as +parameters. Just as importantly, such trained network parameters are +very specific for the type of input data on which they were trained +and the network is not likely to generalize easily to variations in +the input.
-As before, we still have our input, a hidden layer and an output. What's novel about convolutional networks -are the convolutional and pooling layers stacked in pairs between the input and the hidden layer. -In addition, the data is no longer represented as a 2D feature matrix, instead each input is a number of 2D -matrices, typically 1 for each color dimension (Red, Green, Blue). +The main principles that justify convolutions is locality of +information and repetion of patterns within the signal. Sound samples +of the input in adjacent spots are much more likely to affect each +other than those that are very far away. Similarly, sounds are +repeated in multiple times in the signal. While slightly simplistic, +reasoning about such a sound example demonstrates this. The same +principles then apply to images and other similar data.
@@ -443,7 +452,7 @@ matrices, typically 1 for each color dimension (Red, Green, Blue).
-It means that to represent the entire -dataset of images, we require a 4D matrix or tensor. This tensor has the dimensions: -$$ -(n_{inputs},\, n_{pixels, width},\, n_{pixels, height},\, depth) . -$$ +As discussed above, CNNs are neural networks built from the assumption that the inputs +to the network are 2D images. This is important because the number of features or pixels in images +grows very fast with the image size, and an enormous number of weights and biases are needed in order to build an accurate network. + +
+As before, we still have our input, a hidden layer and an output. What's novel about convolutional networks +are the convolutional and pooling layers stacked in pairs between the input and the hidden layer. +In addition, the data is no longer represented as a 2D feature matrix, instead each input is a number of 2D +matrices, typically 1 for each color dimension (Red, Green, Blue).
@@ -439,7 +445,7 @@ $$
-The MNIST dataset consists of grayscale images with a pixel size of -\( 28\times 28 \), meaning we require \( 28 \times 28 = 724 \) weights to each -neuron in the first hidden layer. - -
-If we were to analyze images of size \( 128\times 128 \) we would require -\( 128 \times 128 = 16384 \) weights to each neuron. Even worse if we were -dealing with color images, as most images are, we have an image matrix -of size \( 128\times 128 \) for each color dimension (Red, Green, Blue), -meaning 3 times the number of weights \( = 49152 \) are required for every -single neuron in the first hidden layer. +It means that to represent the entire +dataset of images, we require a 4D matrix or tensor. This tensor has the dimensions: +$$ +(n_{inputs},\, n_{pixels, width},\, n_{pixels, height},\, depth) . +$$
@@ -445,7 +441,7 @@ single neuron in the first hidden layer.
-Images typically have strong local correlations, meaning that a small -part of the image varies little from its neighboring regions. If for -example we have an image of a blue car, we can roughly assume that a -small blue part of the image is surrounded by other blue regions. +The MNIST dataset consists of grayscale images with a pixel size of +\( 28\times 28 \), meaning we require \( 28 \times 28 = 724 \) weights to each +neuron in the first hidden layer.
-Therefore, instead of connecting every single pixel to a neuron in the -first hidden layer, as we have previously done with deep neural -networks, we can instead connect each neuron to a small part of the -image (in all 3 RGB depth dimensions). The size of each small area is -fixed, and known as a receptive. +If we were to analyze images of size \( 128\times 128 \) we would require +\( 128 \times 128 = 16384 \) weights to each neuron. Even worse if we were +dealing with color images, as most images are, we have an image matrix +of size \( 128\times 128 \) for each color dimension (Red, Green, Blue), +meaning 3 times the number of weights \( = 49152 \) are required for every +single neuron in the first hidden layer.
@@ -445,7 +447,7 @@ fixed, and known as a 86
- + -
-A convolution is performed on the image which outputs -a 3D volume of neurons. The weights to the input are arranged in a number of 2D matrices, known as filters. +Images typically have strong local correlations, meaning that a small +part of the image varies little from its neighboring regions. If for +example we have an image of a blue car, we can roughly assume that a +small blue part of the image is surrounded by other blue regions.
-Each filter slides along the input image, taking the dot product -between each small part of the image and the filter, in all depth -dimensions. This is then passed through a non-linear function, -typically the Rectified Linear (ReLu) function, which serves as the -activation of the neurons in the first convolutional layer. This is -further passed through a pooling layer, which reduces the size of the -convolutional layer, e.g. by taking the maximum or average across some -small regions, and this serves as input to the next convolutional -layer. +Therefore, instead of connecting every single pixel to a neuron in the +first hidden layer, as we have previously done with deep neural +networks, we can instead connect each neuron to a small part of the +image (in all 3 RGB depth dimensions). The size of each small area is +fixed, and known as a receptive.
@@ -449,7 +447,7 @@ layer.
- + -
-By systematically reducing the size of the input volume, through -convolution and pooling, the network should create representations of -small parts of the input, and then from them assemble representations -of larger areas. The final pooling layer is flattened to serve as -input to a hidden layer, such that each neuron in the final pooling -layer is connected to every single neuron in the hidden layer. This -then serves as input to the output layer, e.g. a softmax output for -classification. +A convolution is performed on the image which outputs +a 3D volume of neurons. The weights to the input are arranged in a number of 2D matrices, known as filters. + +
+Each filter slides along the input image, taking the dot product +between each small part of the image and the filter, in all depth +dimensions. This is then passed through a non-linear function, +typically the Rectified Linear (ReLu) function, which serves as the +activation of the neurons in the first convolutional layer. This is +further passed through a pooling layer, which reduces the size of the +convolutional layer, e.g. by taking the maximum or average across some +small regions, and this serves as input to the next convolutional +layer.
@@ -442,7 +451,7 @@ classification.
+By systematically reducing the size of the input volume, through +convolution and pooling, the network should create representations of +small parts of the input, and then from them assemble representations +of larger areas. The final pooling layer is flattened to serve as +input to a hidden layer, such that each neuron in the final pooling +layer is connected to every single neuron in the hidden layer. This +then serves as input to the output layer, e.g. a softmax output for +classification. - -
# import necessary packages
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn import datasets
-
-
-# ensure the same random numbers appear every time
-np.random.seed(0)
-
-# display images in notebook
-%matplotlib inline
-plt.rcParams['figure.figsize'] = (12,12)
-
-
-# download MNIST dataset
-digits = datasets.load_digits()
-
-# define inputs and labels
-inputs = digits.images
-labels = digits.target
-
-# RGB images have a depth of 3
-# our images are grayscale so they should have a depth of 1
-inputs = inputs[:,:,:,np.newaxis]
-
-print("inputs = (n_inputs, pixel_width, pixel_height, depth) = " + str(inputs.shape))
-print("labels = (n_inputs) = " + str(labels.shape))
-
-
-# choose some random images to display
-n_inputs = len(inputs)
-indices = np.arange(n_inputs)
-random_indices = np.random.choice(indices, size=5)
-
-for i, image in enumerate(digits.images[random_indices]):
- plt.subplot(1, 5, i+1)
- plt.axis('off')
- plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
- plt.title("Label: %d" % digits.target[random_indices[i]])
-plt.show()
-
@@ -475,7 +444,7 @@ plt.show()
-
from tensorflow.keras import datasets, layers, models
-from tensorflow.keras.layers import Input
-from tensorflow.keras.models import Sequential #This allows appending layers to existing models
-from tensorflow.keras.layers import Dense #This allows defining the characteristics of a particular layer
-from tensorflow.keras import optimizers #This allows using whichever optimiser we want (sgd,adam,RMSprop)
-from tensorflow.keras import regularizers #This allows using whichever regularizer we want (l1,l2,l1_l2)
-from tensorflow.keras.utils import to_categorical #This allows using categorical cross entropy as the cost function
-#from tensorflow.keras import Conv2D
-#from tensorflow.keras import MaxPooling2D
-#from tensorflow.keras import Flatten
+# import necessary packages
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn import datasets
-from sklearn.model_selection import train_test_split
-# representation of labels
-labels = to_categorical(labels)
+# ensure the same random numbers appear every time
+np.random.seed(0)
-# split into train and test data
-# one-liner from scikit-learn library
-train_size = 0.8
-test_size = 1 - train_size
-X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,
- test_size=test_size)
+# display images in notebook
+%matplotlib inline
+plt.rcParams['figure.figsize'] = (12,12)
+
+
+# download MNIST dataset
+digits = datasets.load_digits()
+
+# define inputs and labels
+inputs = digits.images
+labels = digits.target
+
+# RGB images have a depth of 3
+# our images are grayscale so they should have a depth of 1
+inputs = inputs[:,:,:,np.newaxis]
+
+print("inputs = (n_inputs, pixel_width, pixel_height, depth) = " + str(inputs.shape))
+print("labels = (n_inputs) = " + str(labels.shape))
+
+
+# choose some random images to display
+n_inputs = len(inputs)
+indices = np.arange(n_inputs)
+random_indices = np.random.choice(indices, size=5)
+
+for i, image in enumerate(digits.images[random_indices]):
+ plt.subplot(1, 5, i+1)
+ plt.axis('off')
+ plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
+ plt.title("Label: %d" % digits.target[random_indices[i]])
+plt.show()
@@ -456,6 +476,8 @@ X_train, X_test, Y_train, Y_test = train_tes
- - -
-
def create_convolutional_neural_network_keras(input_shape, receptive_field,
- n_filters, n_neurons_connected, n_categories,
- eta, lmbd):
- model = Sequential()
- model.add(layers.Conv2D(n_filters, (receptive_field, receptive_field), input_shape=input_shape, padding='same',
- activation='relu', kernel_regularizer=regularizers.l2(lmbd)))
- model.add(layers.MaxPooling2D(pool_size=(2, 2)))
- model.add(layers.Flatten())
- model.add(layers.Dense(n_neurons_connected, activation='relu', kernel_regularizer=regularizers.l2(lmbd)))
- model.add(layers.Dense(n_categories, activation='softmax', kernel_regularizer=regularizers.l2(lmbd)))
-
- sgd = optimizers.SGD(lr=eta)
- model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
-
- return model
+from tensorflow.keras import datasets, layers, models
+from tensorflow.keras.layers import Input
+from tensorflow.keras.models import Sequential #This allows appending layers to existing models
+from tensorflow.keras.layers import Dense #This allows defining the characteristics of a particular layer
+from tensorflow.keras import optimizers #This allows using whichever optimiser we want (sgd,adam,RMSprop)
+from tensorflow.keras import regularizers #This allows using whichever regularizer we want (l1,l2,l1_l2)
+from tensorflow.keras.utils import to_categorical #This allows using categorical cross entropy as the cost function
+#from tensorflow.keras import Conv2D
+#from tensorflow.keras import MaxPooling2D
+#from tensorflow.keras import Flatten
-epochs = 100
-batch_size = 100
-input_shape = X_train.shape[1:4]
-receptive_field = 3
-n_filters = 10
-n_neurons_connected = 50
-n_categories = 10
+from sklearn.model_selection import train_test_split
-eta_vals = np.logspace(-5, 1, 7)
-lmbd_vals = np.logspace(-5, 1, 7)
+# representation of labels
+labels = to_categorical(labels)
+
+# split into train and test data
+# one-liner from scikit-learn library
+train_size = 0.8
+test_size = 1 - train_size
+X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,
+ test_size=test_size)
@@ -460,6 +457,7 @@ lmbd_vals = np.
89
90
91
+ 92
»
diff --git a/doc/pub/week42/html/._week42-bs083.html b/doc/pub/week42/html/._week42-bs083.html
index 21e2ec214..40c6de957 100644
--- a/doc/pub/week42/html/._week42-bs083.html
+++ b/doc/pub/week42/html/._week42-bs083.html
@@ -190,6 +190,7 @@ Automatically generated HTML file from DocOnce source
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
+ ('Key Idea', 2, None, 'key-idea'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Convolution Examples: Polynomial multiplication',
2,
@@ -358,37 +359,38 @@ MathJax.Hub.Config({
Layers used to build CNNs
Transforming images
CNNs in brief
- Mathematics of CNNs
- Convolution Examples: Polynomial multiplication
- Efficient Polynomial Multiplication
- A more efficient way of coding the above Convolution
- Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
- Principle of Superposition
- Simple Code Example
- Wrapping up Fourier transforms
- Finding the Coefficients
- Final words on Fourier Transforms
- Two-dimensional Objects
- Cross-Correlation
- More on Dimensionalities
- Further Dimensionality Remarks
- CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
- Setting it up
- The MNIST dataset again
- Strong correlations
- Layers of a CNN
- Systematic reduction
- Prerequisites: Collect and pre-process data
- Importing Keras and Tensorflow
- Running with Keras
- Final part
- Final visualization
- The CIFAR01 data set
- Verifying the data set
- Set up the model
- Add Dense layers on top
- Compile and train the model
- Finally, evaluate the model
+ Key Idea
+ Mathematics of CNNs
+ Convolution Examples: Polynomial multiplication
+ Efficient Polynomial Multiplication
+ A more efficient way of coding the above Convolution
+ Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
+ Principle of Superposition
+ Simple Code Example
+ Wrapping up Fourier transforms
+ Finding the Coefficients
+ Final words on Fourier Transforms
+ Two-dimensional Objects
+ Cross-Correlation
+ More on Dimensionalities
+ Further Dimensionality Remarks
+ CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
+ Setting it up
+ The MNIST dataset again
+ Strong correlations
+ Layers of a CNN
+ Systematic reduction
+ Prerequisites: Collect and pre-process data
+ Importing Keras and Tensorflow
+ Running with Keras
+ Final part
+ Final visualization
+ The CIFAR01 data set
+ Verifying the data set
+ Set up the model
+ Add Dense layers on top
+ Compile and train the model
+ Finally, evaluate the model
@@ -402,29 +404,39 @@ MathJax.Hub.Config({
-
+
-Final part
+Running with Keras
-
CNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
-
-for i, eta in enumerate(eta_vals):
- for j, lmbd in enumerate(lmbd_vals):
- CNN = create_convolutional_neural_network_keras(input_shape, receptive_field,
+def create_convolutional_neural_network_keras(input_shape, receptive_field,
n_filters, n_neurons_connected, n_categories,
- eta, lmbd)
- CNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0)
- scores = CNN.evaluate(X_test, Y_test)
-
- CNN_keras[i][j] = CNN
-
- print("Learning rate = ", eta)
- print("Lambda = ", lmbd)
- print("Test accuracy: %.3f" % scores[1])
- print()
+ eta, lmbd):
+ model = Sequential()
+ model.add(layers.Conv2D(n_filters, (receptive_field, receptive_field), input_shape=input_shape, padding='same',
+ activation='relu', kernel_regularizer=regularizers.l2(lmbd)))
+ model.add(layers.MaxPooling2D(pool_size=(2, 2)))
+ model.add(layers.Flatten())
+ model.add(layers.Dense(n_neurons_connected, activation='relu', kernel_regularizer=regularizers.l2(lmbd)))
+ model.add(layers.Dense(n_categories, activation='softmax', kernel_regularizer=regularizers.l2(lmbd)))
+
+ sgd = optimizers.SGD(lr=eta)
+ model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
+
+ return model
+
+epochs = 100
+batch_size = 100
+input_shape = X_train.shape[1:4]
+receptive_field = 3
+n_filters = 10
+n_neurons_connected = 50
+n_categories = 10
+
+eta_vals = np.logspace(-5, 1, 7)
+lmbd_vals = np.logspace(-5, 1, 7)
@@ -449,6 +461,7 @@ MathJax.Hub.Config({
89
90
91
+ 92
»
diff --git a/doc/pub/week42/html/._week42-bs084.html b/doc/pub/week42/html/._week42-bs084.html
index 6ebb54a7b..70dbd7d04 100644
--- a/doc/pub/week42/html/._week42-bs084.html
+++ b/doc/pub/week42/html/._week42-bs084.html
@@ -190,6 +190,7 @@ Automatically generated HTML file from DocOnce source
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
+ ('Key Idea', 2, None, 'key-idea'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Convolution Examples: Polynomial multiplication',
2,
@@ -358,37 +359,38 @@ MathJax.Hub.Config({
Layers used to build CNNs
Transforming images
CNNs in brief
- Mathematics of CNNs
- Convolution Examples: Polynomial multiplication
- Efficient Polynomial Multiplication
- A more efficient way of coding the above Convolution
- Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
- Principle of Superposition
- Simple Code Example
- Wrapping up Fourier transforms
- Finding the Coefficients
- Final words on Fourier Transforms
- Two-dimensional Objects
- Cross-Correlation
- More on Dimensionalities
- Further Dimensionality Remarks
- CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
- Setting it up
- The MNIST dataset again
- Strong correlations
- Layers of a CNN
- Systematic reduction
- Prerequisites: Collect and pre-process data
- Importing Keras and Tensorflow
- Running with Keras
- Final part
- Final visualization
- The CIFAR01 data set
- Verifying the data set
- Set up the model
- Add Dense layers on top
- Compile and train the model
- Finally, evaluate the model
+ Key Idea
+ Mathematics of CNNs
+ Convolution Examples: Polynomial multiplication
+ Efficient Polynomial Multiplication
+ A more efficient way of coding the above Convolution
+ Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
+ Principle of Superposition
+ Simple Code Example
+ Wrapping up Fourier transforms
+ Finding the Coefficients
+ Final words on Fourier Transforms
+ Two-dimensional Objects
+ Cross-Correlation
+ More on Dimensionalities
+ Further Dimensionality Remarks
+ CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
+ Setting it up
+ The MNIST dataset again
+ Strong correlations
+ Layers of a CNN
+ Systematic reduction
+ Prerequisites: Collect and pre-process data
+ Importing Keras and Tensorflow
+ Running with Keras
+ Final part
+ Final visualization
+ The CIFAR01 data set
+ Verifying the data set
+ Set up the model
+ Add Dense layers on top
+ Compile and train the model
+ Finally, evaluate the model
@@ -404,41 +406,27 @@ MathJax.Hub.Config({
-Final visualization
+Final part
-
# visual representation of grid search
-# uses seaborn heatmap, could probably do this in matplotlib
-import seaborn as sns
-
-sns.set()
-
-train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
-test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
-
-for i in range(len(eta_vals)):
- for j in range(len(lmbd_vals)):
- CNN = CNN_keras[i][j]
-
- train_accuracy[i][j] = CNN.evaluate(X_train, Y_train)[1]
- test_accuracy[i][j] = CNN.evaluate(X_test, Y_test)[1]
-
+CNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
-fig, ax = plt.subplots(figsize = (10, 10))
-sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
-ax.set_title("Training Accuracy")
-ax.set_ylabel("$\eta$")
-ax.set_xlabel("$\lambda$")
-plt.show()
-
-fig, ax = plt.subplots(figsize = (10, 10))
-sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
-ax.set_title("Test Accuracy")
-ax.set_ylabel("$\eta$")
-ax.set_xlabel("$\lambda$")
-plt.show()
+for i, eta in enumerate(eta_vals):
+ for j, lmbd in enumerate(lmbd_vals):
+ CNN = create_convolutional_neural_network_keras(input_shape, receptive_field,
+ n_filters, n_neurons_connected, n_categories,
+ eta, lmbd)
+ CNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0)
+ scores = CNN.evaluate(X_test, Y_test)
+
+ CNN_keras[i][j] = CNN
+
+ print("Learning rate = ", eta)
+ print("Lambda = ", lmbd)
+ print("Test accuracy: %.3f" % scores[1])
+ print()
@@ -462,6 +450,7 @@ plt.show()
89
90
91
+ 92
»
diff --git a/doc/pub/week42/html/._week42-bs085.html b/doc/pub/week42/html/._week42-bs085.html
index de3a7aae0..8c7408e29 100644
--- a/doc/pub/week42/html/._week42-bs085.html
+++ b/doc/pub/week42/html/._week42-bs085.html
@@ -190,6 +190,7 @@ Automatically generated HTML file from DocOnce source
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
+ ('Key Idea', 2, None, 'key-idea'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Convolution Examples: Polynomial multiplication',
2,
@@ -358,37 +359,38 @@ MathJax.Hub.Config({
Layers used to build CNNs
Transforming images
CNNs in brief
- Mathematics of CNNs
- Convolution Examples: Polynomial multiplication
- Efficient Polynomial Multiplication
- A more efficient way of coding the above Convolution
- Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
- Principle of Superposition
- Simple Code Example
- Wrapping up Fourier transforms
- Finding the Coefficients
- Final words on Fourier Transforms
- Two-dimensional Objects
- Cross-Correlation
- More on Dimensionalities
- Further Dimensionality Remarks
- CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
- Setting it up
- The MNIST dataset again
- Strong correlations
- Layers of a CNN
- Systematic reduction
- Prerequisites: Collect and pre-process data
- Importing Keras and Tensorflow
- Running with Keras
- Final part
- Final visualization
- The CIFAR01 data set
- Verifying the data set
- Set up the model
- Add Dense layers on top
- Compile and train the model
- Finally, evaluate the model
+ Key Idea
+ Mathematics of CNNs
+ Convolution Examples: Polynomial multiplication
+ Efficient Polynomial Multiplication
+ A more efficient way of coding the above Convolution
+ Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
+ Principle of Superposition
+ Simple Code Example
+ Wrapping up Fourier transforms
+ Finding the Coefficients
+ Final words on Fourier Transforms
+ Two-dimensional Objects
+ Cross-Correlation
+ More on Dimensionalities
+ Further Dimensionality Remarks
+ CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
+ Setting it up
+ The MNIST dataset again
+ Strong correlations
+ Layers of a CNN
+ Systematic reduction
+ Prerequisites: Collect and pre-process data
+ Importing Keras and Tensorflow
+ Running with Keras
+ Final part
+ Final visualization
+ The CIFAR01 data set
+ Verifying the data set
+ Set up the model
+ Add Dense layers on top
+ Compile and train the model
+ Finally, evaluate the model
@@ -404,27 +406,41 @@ MathJax.Hub.Config({
-The CIFAR01 data set
-
-
-The CIFAR10 dataset contains 60,000 color images in 10 classes, with
-6,000 images in each class. The dataset is divided into 50,000
-training images and 10,000 testing images. The classes are mutually
-exclusive and there is no overlap between them.
+
Final visualization
-
import tensorflow as tf
+# visual representation of grid search
+# uses seaborn heatmap, could probably do this in matplotlib
+import seaborn as sns
-from tensorflow.keras import datasets, layers, models
-import matplotlib.pyplot as plt
+sns.set()
-# We import the data set
-(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
+train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
-# Normalize pixel values to be between 0 and 1 by dividing by 255.
-train_images, test_images = train_images / 255.0, test_images / 255.0
+for i in range(len(eta_vals)):
+ for j in range(len(lmbd_vals)):
+ CNN = CNN_keras[i][j]
+
+ train_accuracy[i][j] = CNN.evaluate(X_train, Y_train)[1]
+ test_accuracy[i][j] = CNN.evaluate(X_test, Y_test)[1]
+
+
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Training Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
+plt.show()
+
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Test Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
+plt.show()
@@ -447,6 +463,7 @@ train_images, test_images = train_images 89
90
91
+ 92
»
diff --git a/doc/pub/week42/html/._week42-bs086.html b/doc/pub/week42/html/._week42-bs086.html
index f40123642..122a8e8fb 100644
--- a/doc/pub/week42/html/._week42-bs086.html
+++ b/doc/pub/week42/html/._week42-bs086.html
@@ -190,6 +190,7 @@ Automatically generated HTML file from DocOnce source
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
+ ('Key Idea', 2, None, 'key-idea'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Convolution Examples: Polynomial multiplication',
2,
@@ -358,37 +359,38 @@ MathJax.Hub.Config({
Layers used to build CNNs
Transforming images
CNNs in brief
- Mathematics of CNNs
- Convolution Examples: Polynomial multiplication
- Efficient Polynomial Multiplication
- A more efficient way of coding the above Convolution
- Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
- Principle of Superposition
- Simple Code Example
- Wrapping up Fourier transforms
- Finding the Coefficients
- Final words on Fourier Transforms
- Two-dimensional Objects
- Cross-Correlation
- More on Dimensionalities
- Further Dimensionality Remarks
- CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
- Setting it up
- The MNIST dataset again
- Strong correlations
- Layers of a CNN
- Systematic reduction
- Prerequisites: Collect and pre-process data
- Importing Keras and Tensorflow
- Running with Keras
- Final part
- Final visualization
- The CIFAR01 data set
- Verifying the data set
- Set up the model
- Add Dense layers on top
- Compile and train the model
- Finally, evaluate the model
+ Key Idea
+ Mathematics of CNNs
+ Convolution Examples: Polynomial multiplication
+ Efficient Polynomial Multiplication
+ A more efficient way of coding the above Convolution
+ Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
+ Principle of Superposition
+ Simple Code Example
+ Wrapping up Fourier transforms
+ Finding the Coefficients
+ Final words on Fourier Transforms
+ Two-dimensional Objects
+ Cross-Correlation
+ More on Dimensionalities
+ Further Dimensionality Remarks
+ CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
+ Setting it up
+ The MNIST dataset again
+ Strong correlations
+ Layers of a CNN
+ Systematic reduction
+ Prerequisites: Collect and pre-process data
+ Importing Keras and Tensorflow
+ Running with Keras
+ Final part
+ Final visualization
+ The CIFAR01 data set
+ Verifying the data set
+ Set up the model
+ Add Dense layers on top
+ Compile and train the model
+ Finally, evaluate the model
@@ -404,28 +406,27 @@ MathJax.Hub.Config({
-Verifying the data set
+The CIFAR01 data set
-To verify that the dataset looks correct, let's plot the first 25 images from the training set and display the class name below each image.
+The CIFAR10 dataset contains 60,000 color images in 10 classes, with
+6,000 images in each class. The dataset is divided into 50,000
+training images and 10,000 testing images. The classes are mutually
+exclusive and there is no overlap between them.
-
class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
- 'dog', 'frog', 'horse', 'ship', 'truck']
-
-plt.figure(figsize=(10,10))
-for i in range(25):
- plt.subplot(5,5,i+1)
- plt.xticks([])
- plt.yticks([])
- plt.grid(False)
- plt.imshow(train_images[i], cmap=plt.cm.binary)
- # The CIFAR labels happen to be arrays,
- # which is why you need the extra index
- plt.xlabel(class_names[train_labels[i][0]])
-plt.show()
+import tensorflow as tf
+
+from tensorflow.keras import datasets, layers, models
+import matplotlib.pyplot as plt
+
+# We import the data set
+(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
+
+# Normalize pixel values to be between 0 and 1 by dividing by 255.
+train_images, test_images = train_images / 255.0, test_images / 255.0
@@ -447,6 +448,7 @@ plt.show()
89
90
91
+ 92
»
diff --git a/doc/pub/week42/html/._week42-bs087.html b/doc/pub/week42/html/._week42-bs087.html
index 15c11f9a5..a55fb61ea 100644
--- a/doc/pub/week42/html/._week42-bs087.html
+++ b/doc/pub/week42/html/._week42-bs087.html
@@ -190,6 +190,7 @@ Automatically generated HTML file from DocOnce source
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
+ ('Key Idea', 2, None, 'key-idea'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Convolution Examples: Polynomial multiplication',
2,
@@ -358,37 +359,38 @@ MathJax.Hub.Config({
Layers used to build CNNs
Transforming images
CNNs in brief
- Mathematics of CNNs
- Convolution Examples: Polynomial multiplication
- Efficient Polynomial Multiplication
- A more efficient way of coding the above Convolution
- Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
- Principle of Superposition
- Simple Code Example
- Wrapping up Fourier transforms
- Finding the Coefficients
- Final words on Fourier Transforms
- Two-dimensional Objects
- Cross-Correlation
- More on Dimensionalities
- Further Dimensionality Remarks
- CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
- Setting it up
- The MNIST dataset again
- Strong correlations
- Layers of a CNN
- Systematic reduction
- Prerequisites: Collect and pre-process data
- Importing Keras and Tensorflow
- Running with Keras
- Final part
- Final visualization
- The CIFAR01 data set
- Verifying the data set
- Set up the model
- Add Dense layers on top
- Compile and train the model
- Finally, evaluate the model
+ Key Idea
+ Mathematics of CNNs
+ Convolution Examples: Polynomial multiplication
+ Efficient Polynomial Multiplication
+ A more efficient way of coding the above Convolution
+ Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
+ Principle of Superposition
+ Simple Code Example
+ Wrapping up Fourier transforms
+ Finding the Coefficients
+ Final words on Fourier Transforms
+ Two-dimensional Objects
+ Cross-Correlation
+ More on Dimensionalities
+ Further Dimensionality Remarks
+ CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
+ Setting it up
+ The MNIST dataset again
+ Strong correlations
+ Layers of a CNN
+ Systematic reduction
+ Prerequisites: Collect and pre-process data
+ Importing Keras and Tensorflow
+ Running with Keras
+ Final part
+ Final visualization
+ The CIFAR01 data set
+ Verifying the data set
+ Set up the model
+ Add Dense layers on top
+ Compile and train the model
+ Finally, evaluate the model
@@ -404,31 +406,29 @@ MathJax.Hub.Config({
-Set up the model
+Verifying the data set
-The 6 lines of code below define the convolutional base using a common pattern: a stack of Conv2D and MaxPooling2D layers.
-
-
-As input, a CNN takes tensors of shape (image_height, image_width, color_channels), ignoring the batch size. If you are new to these dimensions, color_channels refers to (R,G,B). In this example, you will configure our CNN to process inputs of shape (32, 32, 3), which is the format of CIFAR images. You can do this by passing the argument input_shape to our first layer.
+To verify that the dataset looks correct, let's plot the first 25 images from the training set and display the class name below each image.
-
model = models.Sequential()
-model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
-model.add(layers.MaxPooling2D((2, 2)))
-model.add(layers.Conv2D(64, (3, 3), activation='relu'))
-model.add(layers.MaxPooling2D((2, 2)))
-model.add(layers.Conv2D(64, (3, 3), activation='relu'))
-
-# Let's display the architecture of our model so far.
-
-model.summary()
+class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
+ 'dog', 'frog', 'horse', 'ship', 'truck']
+
+plt.figure(figsize=(10,10))
+for i in range(25):
+ plt.subplot(5,5,i+1)
+ plt.xticks([])
+ plt.yticks([])
+ plt.grid(False)
+ plt.imshow(train_images[i], cmap=plt.cm.binary)
+ # The CIFAR labels happen to be arrays,
+ # which is why you need the extra index
+ plt.xlabel(class_names[train_labels[i][0]])
+plt.show()
-
-You can see that the output of every Conv2D and MaxPooling2D layer is a 3D tensor of shape (height, width, channels). The width and height dimensions tend to shrink as you go deeper in the network. The number of output channels for each Conv2D layer is controlled by the first argument (e.g., 32 or 64). Typically, as the width and height shrink, you can afford (computationally) to add more output channels in each Conv2D layer.
-
@@ -448,6 +448,7 @@ You can see that the output of every Conv2D and MaxPooling2D layer is a 3D tenso
89
90
91
+ 92
»
diff --git a/doc/pub/week42/html/._week42-bs088.html b/doc/pub/week42/html/._week42-bs088.html
index eb6e406ac..7d38e6705 100644
--- a/doc/pub/week42/html/._week42-bs088.html
+++ b/doc/pub/week42/html/._week42-bs088.html
@@ -190,6 +190,7 @@ Automatically generated HTML file from DocOnce source
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
+ ('Key Idea', 2, None, 'key-idea'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Convolution Examples: Polynomial multiplication',
2,
@@ -358,37 +359,38 @@ MathJax.Hub.Config({
Layers used to build CNNs
Transforming images
CNNs in brief
- Mathematics of CNNs
- Convolution Examples: Polynomial multiplication
- Efficient Polynomial Multiplication
- A more efficient way of coding the above Convolution
- Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
- Principle of Superposition
- Simple Code Example
- Wrapping up Fourier transforms
- Finding the Coefficients
- Final words on Fourier Transforms
- Two-dimensional Objects
- Cross-Correlation
- More on Dimensionalities
- Further Dimensionality Remarks
- CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
- Setting it up
- The MNIST dataset again
- Strong correlations
- Layers of a CNN
- Systematic reduction
- Prerequisites: Collect and pre-process data
- Importing Keras and Tensorflow
- Running with Keras
- Final part
- Final visualization
- The CIFAR01 data set
- Verifying the data set
- Set up the model
- Add Dense layers on top
- Compile and train the model
- Finally, evaluate the model
+ Key Idea
+ Mathematics of CNNs
+ Convolution Examples: Polynomial multiplication
+ Efficient Polynomial Multiplication
+ A more efficient way of coding the above Convolution
+ Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
+ Principle of Superposition
+ Simple Code Example
+ Wrapping up Fourier transforms
+ Finding the Coefficients
+ Final words on Fourier Transforms
+ Two-dimensional Objects
+ Cross-Correlation
+ More on Dimensionalities
+ Further Dimensionality Remarks
+ CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
+ Setting it up
+ The MNIST dataset again
+ Strong correlations
+ Layers of a CNN
+ Systematic reduction
+ Prerequisites: Collect and pre-process data
+ Importing Keras and Tensorflow
+ Running with Keras
+ Final part
+ Final visualization
+ The CIFAR01 data set
+ Verifying the data set
+ Set up the model
+ Add Dense layers on top
+ Compile and train the model
+ Finally, evaluate the model
@@ -404,29 +406,30 @@ MathJax.Hub.Config({
-Add Dense layers on top
+Set up the model
-To complete our model, you will feed the last output tensor from the
-convolutional base (of shape (4, 4, 64)) into one or more Dense layers
-to perform classification. Dense layers take vectors as input (which
-are 1D), while the current output is a 3D tensor. First, you will
-flatten (or unroll) the 3D output to 1D, then add one or more Dense
-layers on top. CIFAR has 10 output classes, so you use a final Dense
-layer with 10 outputs and a softmax activation.
+The 6 lines of code below define the convolutional base using a common pattern: a stack of Conv2D and MaxPooling2D layers.
+
+
+As input, a CNN takes tensors of shape (image_height, image_width, color_channels), ignoring the batch size. If you are new to these dimensions, color_channels refers to (R,G,B). In this example, you will configure our CNN to process inputs of shape (32, 32, 3), which is the format of CIFAR images. You can do this by passing the argument input_shape to our first layer.
-
model.add(layers.Flatten())
-model.add(layers.Dense(64, activation='relu'))
-model.add(layers.Dense(10))
-Here's the complete architecture of our model.
+model = models.Sequential()
+model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
+model.add(layers.MaxPooling2D((2, 2)))
+model.add(layers.Conv2D(64, (3, 3), activation='relu'))
+model.add(layers.MaxPooling2D((2, 2)))
+model.add(layers.Conv2D(64, (3, 3), activation='relu'))
+
+# Let's display the architecture of our model so far.
model.summary()
-As you can see, our (4, 4, 64) outputs were flattened into vectors of shape (1024) before going through two Dense layers.
+You can see that the output of every Conv2D and MaxPooling2D layer is a 3D tensor of shape (height, width, channels). The width and height dimensions tend to shrink as you go deeper in the network. The number of output channels for each Conv2D layer is controlled by the first argument (e.g., 32 or 64). Typically, as the width and height shrink, you can afford (computationally) to add more output channels in each Conv2D layer.
@@ -446,6 +449,7 @@ As you can see, our (4, 4, 64) outputs were flattened into vectors of shape (102
89
90
91
+ 92
»
diff --git a/doc/pub/week42/html/._week42-bs089.html b/doc/pub/week42/html/._week42-bs089.html
index 0a7f5776f..9325f8a66 100644
--- a/doc/pub/week42/html/._week42-bs089.html
+++ b/doc/pub/week42/html/._week42-bs089.html
@@ -190,6 +190,7 @@ Automatically generated HTML file from DocOnce source
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
+ ('Key Idea', 2, None, 'key-idea'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Convolution Examples: Polynomial multiplication',
2,
@@ -358,37 +359,38 @@ MathJax.Hub.Config({
Layers used to build CNNs
Transforming images
CNNs in brief
- Mathematics of CNNs
- Convolution Examples: Polynomial multiplication
- Efficient Polynomial Multiplication
- A more efficient way of coding the above Convolution
- Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
- Principle of Superposition
- Simple Code Example
- Wrapping up Fourier transforms
- Finding the Coefficients
- Final words on Fourier Transforms
- Two-dimensional Objects
- Cross-Correlation
- More on Dimensionalities
- Further Dimensionality Remarks
- CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
- Setting it up
- The MNIST dataset again
- Strong correlations
- Layers of a CNN
- Systematic reduction
- Prerequisites: Collect and pre-process data
- Importing Keras and Tensorflow
- Running with Keras
- Final part
- Final visualization
- The CIFAR01 data set
- Verifying the data set
- Set up the model
- Add Dense layers on top
- Compile and train the model
- Finally, evaluate the model
+ Key Idea
+ Mathematics of CNNs
+ Convolution Examples: Polynomial multiplication
+ Efficient Polynomial Multiplication
+ A more efficient way of coding the above Convolution
+ Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
+ Principle of Superposition
+ Simple Code Example
+ Wrapping up Fourier transforms
+ Finding the Coefficients
+ Final words on Fourier Transforms
+ Two-dimensional Objects
+ Cross-Correlation
+ More on Dimensionalities
+ Further Dimensionality Remarks
+ CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
+ Setting it up
+ The MNIST dataset again
+ Strong correlations
+ Layers of a CNN
+ Systematic reduction
+ Prerequisites: Collect and pre-process data
+ Importing Keras and Tensorflow
+ Running with Keras
+ Final part
+ Final visualization
+ The CIFAR01 data set
+ Verifying the data set
+ Set up the model
+ Add Dense layers on top
+ Compile and train the model
+ Finally, evaluate the model
@@ -404,18 +406,30 @@ MathJax.Hub.Config({
-Compile and train the model
+Add Dense layers on top
+
+
+To complete our model, you will feed the last output tensor from the
+convolutional base (of shape (4, 4, 64)) into one or more Dense layers
+to perform classification. Dense layers take vectors as input (which
+are 1D), while the current output is a 3D tensor. First, you will
+flatten (or unroll) the 3D output to 1D, then add one or more Dense
+layers on top. CIFAR has 10 output classes, so you use a final Dense
+layer with 10 outputs and a softmax activation.
-
model.compile(optimizer='adam',
- loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
- metrics=['accuracy'])
-
-history = model.fit(train_images, train_labels, epochs=10,
- validation_data=(test_images, test_labels))
+model.add(layers.Flatten())
+model.add(layers.Dense(64, activation='relu'))
+model.add(layers.Dense(10))
+Here's the complete architecture of our model.
+
+model.summary()
+
+As you can see, our (4, 4, 64) outputs were flattened into vectors of shape (1024) before going through two Dense layers.
+
@@ -433,6 +447,7 @@ history = model
89
90
91
+ 92
»
diff --git a/doc/pub/week42/html/._week42-bs090.html b/doc/pub/week42/html/._week42-bs090.html
index 9d4668ce0..2c42201c0 100644
--- a/doc/pub/week42/html/._week42-bs090.html
+++ b/doc/pub/week42/html/._week42-bs090.html
@@ -190,6 +190,7 @@ Automatically generated HTML file from DocOnce source
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
+ ('Key Idea', 2, None, 'key-idea'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Convolution Examples: Polynomial multiplication',
2,
@@ -358,37 +359,38 @@ MathJax.Hub.Config({
Layers used to build CNNs
Transforming images
CNNs in brief
- Mathematics of CNNs
- Convolution Examples: Polynomial multiplication
- Efficient Polynomial Multiplication
- A more efficient way of coding the above Convolution
- Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
- Principle of Superposition
- Simple Code Example
- Wrapping up Fourier transforms
- Finding the Coefficients
- Final words on Fourier Transforms
- Two-dimensional Objects
- Cross-Correlation
- More on Dimensionalities
- Further Dimensionality Remarks
- CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
- Setting it up
- The MNIST dataset again
- Strong correlations
- Layers of a CNN
- Systematic reduction
- Prerequisites: Collect and pre-process data
- Importing Keras and Tensorflow
- Running with Keras
- Final part
- Final visualization
- The CIFAR01 data set
- Verifying the data set
- Set up the model
- Add Dense layers on top
- Compile and train the model
- Finally, evaluate the model
+ Key Idea
+ Mathematics of CNNs
+ Convolution Examples: Polynomial multiplication
+ Efficient Polynomial Multiplication
+ A more efficient way of coding the above Convolution
+ Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
+ Principle of Superposition
+ Simple Code Example
+ Wrapping up Fourier transforms
+ Finding the Coefficients
+ Final words on Fourier Transforms
+ Two-dimensional Objects
+ Cross-Correlation
+ More on Dimensionalities
+ Further Dimensionality Remarks
+ CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
+ Setting it up
+ The MNIST dataset again
+ Strong correlations
+ Layers of a CNN
+ Systematic reduction
+ Prerequisites: Collect and pre-process data
+ Importing Keras and Tensorflow
+ Running with Keras
+ Final part
+ Final visualization
+ The CIFAR01 data set
+ Verifying the data set
+ Set up the model
+ Add Dense layers on top
+ Compile and train the model
+ Finally, evaluate the model
@@ -404,24 +406,19 @@ MathJax.Hub.Config({
-Finally, evaluate the model
+Compile and train the model
-
plt.plot(history.history['accuracy'], label='accuracy')
-plt.plot(history.history['val_accuracy'], label = 'val_accuracy')
-plt.xlabel('Epoch')
-plt.ylabel('Accuracy')
-plt.ylim([0.5, 1])
-plt.legend(loc='lower right')
-
-test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
-
-print(test_acc)
+model.compile(optimizer='adam',
+ loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
+ metrics=['accuracy'])
+
+history = model.fit(train_images, train_labels, epochs=10,
+ validation_data=(test_images, test_labels))
-
diff --git a/doc/pub/week42/html/week42-bs.html b/doc/pub/week42/html/week42-bs.html
index 5522dc854..eaf81fcd0 100644
--- a/doc/pub/week42/html/week42-bs.html
+++ b/doc/pub/week42/html/week42-bs.html
@@ -190,6 +190,7 @@ Automatically generated HTML file from DocOnce source
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
+ ('Key Idea', 2, None, 'key-idea'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Convolution Examples: Polynomial multiplication',
2,
@@ -358,37 +359,38 @@ MathJax.Hub.Config({
Layers used to build CNNs
Transforming images
CNNs in brief
- Mathematics of CNNs
- Convolution Examples: Polynomial multiplication
- Efficient Polynomial Multiplication
- A more efficient way of coding the above Convolution
- Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
- Principle of Superposition
- Simple Code Example
- Wrapping up Fourier transforms
- Finding the Coefficients
- Final words on Fourier Transforms
- Two-dimensional Objects
- Cross-Correlation
- More on Dimensionalities
- Further Dimensionality Remarks
- CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
- Setting it up
- The MNIST dataset again
- Strong correlations
- Layers of a CNN
- Systematic reduction
- Prerequisites: Collect and pre-process data
- Importing Keras and Tensorflow
- Running with Keras
- Final part
- Final visualization
- The CIFAR01 data set
- Verifying the data set
- Set up the model
- Add Dense layers on top
- Compile and train the model
- Finally, evaluate the model
+ Key Idea
+ Mathematics of CNNs
+ Convolution Examples: Polynomial multiplication
+ Efficient Polynomial Multiplication
+ A more efficient way of coding the above Convolution
+ Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)
+ Principle of Superposition
+ Simple Code Example
+ Wrapping up Fourier transforms
+ Finding the Coefficients
+ Final words on Fourier Transforms
+ Two-dimensional Objects
+ Cross-Correlation
+ More on Dimensionalities
+ Further Dimensionality Remarks
+ CNNs in more detail, building convolutional neural networks in Tensorflow and Keras
+ Setting it up
+ The MNIST dataset again
+ Strong correlations
+ Layers of a CNN
+ Systematic reduction
+ Prerequisites: Collect and pre-process data
+ Importing Keras and Tensorflow
+ Running with Keras
+ Final part
+ Final visualization
+ The CIFAR01 data set
+ Verifying the data set
+ Set up the model
+ Add Dense layers on top
+ Compile and train the model
+ Finally, evaluate the model
@@ -447,7 +449,7 @@ MathJax.Hub.Config({
9
10
...
- 91
+ 92
»
diff --git a/doc/pub/week42/html/week42-reveal.html b/doc/pub/week42/html/week42-reveal.html
index 5e3f382b4..959e2fb02 100644
--- a/doc/pub/week42/html/week42-reveal.html
+++ b/doc/pub/week42/html/week42-reveal.html
@@ -3191,6 +3191,24 @@ For more material on convolutional networks, we strongly recommend
the course
IN5400 – Machine Learning for Image Analysis
and the slides of CS231 which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs.
+
+
+The textbook by Goodfellow et al, see chapter 9 contains an in depth discussion as well.
+
+
+
+
+Key Idea
+
+
+A dense neural network is representd by an affine operation (like matrix-matrix multiplication) where all parameters are included.
+
+
+The key idea in CNNs for say imaging is that in images neighbor pixels tend to be related! So we connect
+only neighboring neurons in the input instead of connecting all with the first hidden layer.
+
+
+We say we perform a filtering (convolution is the mathematical operation).
diff --git a/doc/pub/week42/html/week42-solarized.html b/doc/pub/week42/html/week42-solarized.html
index 8a545de8c..da3cb0a00 100644
--- a/doc/pub/week42/html/week42-solarized.html
+++ b/doc/pub/week42/html/week42-solarized.html
@@ -210,6 +210,7 @@ div { text-align: justify; text-justify: inter-word; }
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
+ ('Key Idea', 2, None, 'key-idea'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Convolution Examples: Polynomial multiplication',
2,
@@ -3230,6 +3231,24 @@ the course
IN5400 – Machine Learning for Image Analysis
and the slides of CS231 which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs.
+
+The textbook by Goodfellow et al, see chapter 9 contains an in depth discussion as well.
+
+
+
+
+
Key Idea
+
+
+A dense neural network is representd by an affine operation (like matrix-matrix multiplication) where all parameters are included.
+
+
+The key idea in CNNs for say imaging is that in images neighbor pixels tend to be related! So we connect
+only neighboring neurons in the input instead of connecting all with the first hidden layer.
+
+
+We say we perform a filtering (convolution is the mathematical operation).
+
diff --git a/doc/pub/week42/html/week42.html b/doc/pub/week42/html/week42.html
index bdce9a0ac..b7c1d0fd9 100644
--- a/doc/pub/week42/html/week42.html
+++ b/doc/pub/week42/html/week42.html
@@ -215,6 +215,7 @@ div { text-align: justify; text-justify: inter-word; }
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
+ ('Key Idea', 2, None, 'key-idea'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Convolution Examples: Polynomial multiplication',
2,
@@ -3235,6 +3236,24 @@ the course
IN5400 – Machine Learning for Image Analysis
and the slides of CS231 which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs.
+
+The textbook by Goodfellow et al, see chapter 9 contains an in depth discussion as well.
+
+
+
+
+
Key Idea
+
+
+A dense neural network is representd by an affine operation (like matrix-matrix multiplication) where all parameters are included.
+
+
+The key idea in CNNs for say imaging is that in images neighbor pixels tend to be related! So we connect
+only neighboring neurons in the input instead of connecting all with the first hidden layer.
+
+
+We say we perform a filtering (convolution is the mathematical operation).
+
diff --git a/doc/pub/week42/ipynb/ipynb-week42-src.tar.gz b/doc/pub/week42/ipynb/ipynb-week42-src.tar.gz
index 750fb55cc..ee3d5823c 100644
Binary files a/doc/pub/week42/ipynb/ipynb-week42-src.tar.gz and b/doc/pub/week42/ipynb/ipynb-week42-src.tar.gz differ
diff --git a/doc/pub/week42/ipynb/week42.ipynb b/doc/pub/week42/ipynb/week42.ipynb
index 21d701d87..86309df61 100644
--- a/doc/pub/week42/ipynb/week42.ipynb
+++ b/doc/pub/week42/ipynb/week42.ipynb
@@ -3298,6 +3298,17 @@
"[IN5400 – Machine Learning for Image Analysis](https://www.uio.no/studier/emner/matnat/ifi/IN5400/index-eng.html)\n",
"and the slides of [CS231](http://cs231n.github.io/convolutional-networks/) which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). [Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs](http://neuralnetworksanddeeplearning.com/chap6.html).\n",
"\n",
+ "The textbook by Goodfellow et al, see chapter 9 contains an in depth discussion as well.\n",
+ "\n",
+ "## Key Idea\n",
+ "\n",
+ "A dense neural network is representd by an affine operation (like matrix-matrix multiplication) where all parameters are included.\n",
+ "\n",
+ "The key idea in CNNs for say imaging is that in images neighbor pixels tend to be related! So we connect\n",
+ "only neighboring neurons in the input instead of connecting all with the first hidden layer.\n",
+ "\n",
+ "We say we perform a filtering (convolution is the mathematical operation). \n",
+ "\n",
"\n",
"## Mathematics of CNNs\n",
"\n",
diff --git a/doc/src/week42/week42.do.txt b/doc/src/week42/week42.do.txt
index f6fc309d1..49d873545 100644
--- a/doc/src/week42/week42.do.txt
+++ b/doc/src/week42/week42.do.txt
@@ -2608,6 +2608,18 @@ the course
"IN5400 – Machine Learning for Image Analysis":"https://www.uio.no/studier/emner/matnat/ifi/IN5400/index-eng.html"
and the slides of "CS231":"http://cs231n.github.io/convolutional-networks/" which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). "Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs":"http://neuralnetworksanddeeplearning.com/chap6.html".
+The textbook by Goodfellow et al, see chapter 9 contains an in depth discussion as well.
+
+!split
+===== Key Idea =====
+
+A dense neural network is representd by an affine operation (like matrix-matrix multiplication) where all parameters are included.
+
+The key idea in CNNs for say imaging is that in images neighbor pixels tend to be related! So we connect
+only neighboring neurons in the input instead of connecting all with the first hidden layer.
+
+We say we perform a filtering (convolution is the mathematical operation).
+
!split
===== Mathematics of CNNs =====