update on week 42
This commit is contained in:
Morten Hjorth-Jensen
@@ -2597,6 +2597,214 @@ the course
|
||||
and the slides of "CS231":"http://cs231n.github.io/convolutional-networks/" which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). "Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs":"http://neuralnetworksanddeeplearning.com/chap6.html".
|
||||
|
||||
|
||||
!split
|
||||
===== Mathematics of CNNs =====
|
||||
|
||||
The mathematics of CNNs is based on the mathematical operation of
|
||||
_convolution_. In mathematics (in particular in functional analysis),
|
||||
convolution is represented by matheematical operation (integration,
|
||||
summation etc) on two function in order to produce a third function
|
||||
that expresses how the shape of one gets modified by the other.
|
||||
Convolution has a plethora of applications in a variety of disciplines, spanning from statistics to signal processing, computer vision, solutions of differential equations,linear algebra, engineering, and yes, machine learning.
|
||||
|
||||
Mathematically, convolution is defined as follows (one-dimensional example):
|
||||
Let us define a continuous function $y(t)$ given by
|
||||
!bt
|
||||
\[
|
||||
y(t) = \int x(a) w(t-a) da,
|
||||
\]
|
||||
!et
|
||||
where $x(a)$ represents a so-called input and $w(t-a)$ is normally called the weight function or kernel.
|
||||
|
||||
The above integral is written in a more compact form as
|
||||
!bt
|
||||
\[
|
||||
y(t) = \left(x * w\right)(t).
|
||||
\]
|
||||
!et
|
||||
|
||||
The discretized version reads
|
||||
!bt
|
||||
\[
|
||||
y(t) = \sum_{a=-\infty}^{a=\infty}x(a)w(t-a).
|
||||
\]
|
||||
!et
|
||||
Computing the inverse of the above convolution operations is known as deconvolution.
|
||||
|
||||
How can we use this? And what does it mean? Let us study some familiar examples first.
|
||||
|
||||
|
||||
!split
|
||||
===== Convolution Examples: Polynomial multiplication =====
|
||||
|
||||
We have already met such an example in project 1 when we tried to set up the design matrix for a two-dimensional function.
|
||||
Let us remind of this and recast it in terms of the mathematical operation of convolution.
|
||||
|
||||
!split
|
||||
===== Convolution Examples: Probability Theory =====
|
||||
|
||||
|
||||
!split
|
||||
===== Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms) =====
|
||||
|
||||
If one has several driving forces, $F(t)=\sum_n F_n(t)$, one can find
|
||||
the particular solution to each $F_n$, $x_{pn}(t)$, and the particular
|
||||
solution for the entire driving force is
|
||||
|
||||
!bt
|
||||
\begin{equation}
|
||||
x_p(t)=\sum_nx_{pn}(t).
|
||||
\end{equation}
|
||||
!et
|
||||
|
||||
This is known as the principal of superposition. It only applies when
|
||||
the homogenous equation is linear. If there were an anharmonic term
|
||||
such as $x^3$ in the homogenous equation, then when one summed various
|
||||
solutions, $x=(\sum_n x_n)^2$, one would get cross
|
||||
terms. Superposition is especially useful when $F(t)$ can be written
|
||||
as a sum of sinusoidal terms, because the solutions for each
|
||||
sinusoidal (sine or cosine) term is analytic, as we saw above.
|
||||
|
||||
Driving forces are often periodic, even when they are not
|
||||
sinusoidal. Periodicity implies that for some time $\tau$
|
||||
|
||||
!bt
|
||||
\begin{eqnarray}
|
||||
F(t+\tau)=F(t).
|
||||
\end{eqnarray}
|
||||
!et
|
||||
|
||||
One example of a non-sinusoidal periodic force is a square wave. Many
|
||||
components in electric circuits are non-linear, e.g. diodes, which
|
||||
makes many wave forms non-sinusoidal even when the circuits are being
|
||||
driven by purely sinusoidal sources.
|
||||
|
||||
The code here shows a typical example of such a square wave generated using the functionality included in the _scipy_ Python package. We have used a period of $\tau=0.2$.
|
||||
|
||||
!bc pycod
|
||||
import numpy as np
|
||||
import math
|
||||
from scipy import signal
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
# number of points
|
||||
n = 500
|
||||
# start and final times
|
||||
t0 = 0.0
|
||||
tn = 1.0
|
||||
# Period
|
||||
t = np.linspace(t0, tn, n, endpoint=False)
|
||||
SqrSignal = np.zeros(n)
|
||||
SqrSignal = 1.0+signal.square(2*np.pi*5*t)
|
||||
plt.plot(t, SqrSignal)
|
||||
plt.ylim(-0.5, 2.5)
|
||||
plt.show()
|
||||
!ec
|
||||
|
||||
|
||||
For the sinusoidal example the
|
||||
period is $\tau=2\pi/\omega$. However, higher harmonics can also
|
||||
satisfy the periodicity requirement. In general, any force that
|
||||
satisfies the periodicity requirement can be expressed as a sum over
|
||||
harmonics,
|
||||
|
||||
!bt
|
||||
\begin{equation}
|
||||
F(t)=\frac{f_0}{2}+\sum_{n>0} f_n\cos(2n\pi t/\tau)+g_n\sin(2n\pi t/\tau).
|
||||
\end{equation}
|
||||
!et
|
||||
|
||||
We can write down the answer for
|
||||
$x_{pn}(t)$, by substituting $f_n/m$ or $g_n/m$ for $F_0/m$. By
|
||||
writing each factor $2n\pi t/\tau$ as $n\omega t$, with $\omega\equiv
|
||||
2\pi/\tau$,
|
||||
|
||||
!bt
|
||||
\begin{equation}
|
||||
label{eq:fourierdef1}
|
||||
F(t)=\frac{f_0}{2}+\sum_{n>0}f_n\cos(n\omega t)+g_n\sin(n\omega t).
|
||||
\end{equation}
|
||||
!et
|
||||
|
||||
The solutions for $x(t)$ then come from replacing $\omega$ with
|
||||
$n\omega$ for each term in the particular solution,
|
||||
|
||||
!bt
|
||||
\begin{eqnarray}
|
||||
x_p(t)&=&\frac{f_0}{2k}+\sum_{n>0} \alpha_n\cos(n\omega t-\delta_n)+\beta_n\sin(n\omega t-\delta_n),\\
|
||||
\nonumber
|
||||
\alpha_n&=&\frac{f_n/m}{\sqrt{((n\omega)^2-\omega_0^2)+4\beta^2n^2\omega^2}},\\
|
||||
\nonumber
|
||||
\beta_n&=&\frac{g_n/m}{\sqrt{((n\omega)^2-\omega_0^2)+4\beta^2n^2\omega^2}},\\
|
||||
\nonumber
|
||||
\delta_n&=&\tan^{-1}\left(\frac{2\beta n\omega}{\omega_0^2-n^2\omega^2}\right).
|
||||
\end{eqnarray}
|
||||
!et
|
||||
|
||||
Because the forces have been applied for a long time, any non-zero
|
||||
damping eliminates the homogenous parts of the solution, so one need
|
||||
only consider the particular solution for each $n$.
|
||||
|
||||
The problem will considered solved if one can find expressions for the
|
||||
coefficients $f_n$ and $g_n$, even though the solutions are expressed
|
||||
as an infinite sum. The coefficients can be extracted from the
|
||||
function $F(t)$ by
|
||||
|
||||
!bt
|
||||
\begin{eqnarray}
|
||||
label{eq:fourierdef2}
|
||||
f_n&=&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~F(t)\cos(2n\pi t/\tau),\\
|
||||
\nonumber
|
||||
g_n&=&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~F(t)\sin(2n\pi t/\tau).
|
||||
\end{eqnarray}
|
||||
!et
|
||||
|
||||
To check the consistency of these expressions and to verify
|
||||
Eq. (ref{eq:fourierdef2}), one can insert the expansion of $F(t)$ in
|
||||
Eq. (ref{eq:fourierdef1}) into the expression for the coefficients in
|
||||
Eq. (ref{eq:fourierdef2}) and see whether
|
||||
|
||||
!bt
|
||||
\begin{eqnarray}
|
||||
f_n&=?&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~\left\{
|
||||
\frac{f_0}{2}+\sum_{m>0}f_m\cos(m\omega t)+g_m\sin(m\omega t)
|
||||
\right\}\cos(n\omega t).
|
||||
\end{eqnarray}
|
||||
!et
|
||||
|
||||
Immediately, one can throw away all the terms with $g_m$ because they
|
||||
convolute an even and an odd function. The term with $f_0/2$
|
||||
disappears because $\cos(n\omega t)$ is equally positive and negative
|
||||
over the interval and will integrate to zero. For all the terms
|
||||
$f_m\cos(m\omega t)$ appearing in the sum, one can use angle addition
|
||||
formulas to see that $\cos(m\omega t)\cos(n\omega
|
||||
t)=(1/2)(\cos[(m+n)\omega t]+\cos[(m-n)\omega t]$. This will integrate
|
||||
to zero unless $m=n$. In that case the $m=n$ term gives
|
||||
|
||||
!bt
|
||||
\begin{equation}
|
||||
\int_{-\tau/2}^{\tau/2}dt~\cos^2(m\omega t)=\frac{\tau}{2},
|
||||
\end{equation}
|
||||
!et
|
||||
|
||||
and
|
||||
|
||||
!bt
|
||||
\begin{eqnarray}
|
||||
f_n&=?&\frac{2}{\tau}\int_{-\tau/2}^{\tau/2} dt~f_n/2\\
|
||||
\nonumber
|
||||
&=&f_n~\checkmark.
|
||||
\end{eqnarray}
|
||||
!et
|
||||
|
||||
The same method can be used to check for the consistency of $g_n$.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
!split
|
||||
===== CNNs in more detail, building convolutional neural networks in Tensorflow and Keras =====
|
||||
@@ -2977,3 +3185,6 @@ print(test_acc)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user