diff --git a/doc/pub/week44/ipynb/week44.ipynb b/doc/pub/week44/ipynb/week44.ipynb index 1bf8133ee..7141a1a94 100644 --- a/doc/pub/week44/ipynb/week44.ipynb +++ b/doc/pub/week44/ipynb/week44.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "e819719c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "560c07a9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Week 44, Convolutional Neural Networks (CNN)\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", @@ -28,9 +24,7 @@ { "cell_type": "markdown", "id": "6cfe69eb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plan for week 44\n", "\n", @@ -76,9 +70,7 @@ { "cell_type": "markdown", "id": "969e3bfb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Material for Lecture Thursday November 2" ] @@ -86,9 +78,7 @@ { "cell_type": "markdown", "id": "4658c67a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convolutional Neural Networks (recognizing images)\n", "\n", @@ -113,9 +103,7 @@ { "cell_type": "markdown", "id": "99ef8af3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## What is the Difference\n", "\n", @@ -129,9 +117,7 @@ { "cell_type": "markdown", "id": "6c8fb76c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Neural Networks vs CNNs\n", "\n", @@ -147,9 +133,7 @@ { "cell_type": "markdown", "id": "0dd2198c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Why CNNS for images, sound files, medical images from CT scans etc?\n", "\n", @@ -177,9 +161,7 @@ { "cell_type": "markdown", "id": "70924749", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Regular NNs don’t scale well to full images\n", "\n", @@ -207,9 +189,7 @@ { "cell_type": "markdown", "id": "618b6e71", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## 3D volumes of neurons\n", "\n", @@ -247,9 +227,7 @@ { "cell_type": "markdown", "id": "8f87c704", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Layers used to build CNNs\n", "\n", @@ -276,9 +254,7 @@ { "cell_type": "markdown", "id": "15d8f220", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Transforming images\n", "\n", @@ -298,9 +274,7 @@ { "cell_type": "markdown", "id": "6b314939", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## CNNs in brief\n", "\n", @@ -326,9 +300,7 @@ { "cell_type": "markdown", "id": "08156a7a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Key Idea\n", "\n", @@ -343,9 +315,7 @@ { "cell_type": "markdown", "id": "bb53a363", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Mathematics of CNNs\n", "\n", @@ -363,9 +333,7 @@ { "cell_type": "markdown", "id": "8835386d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "y(t) = \\int x(a) w(t-a) da,\n", @@ -375,9 +343,7 @@ { "cell_type": "markdown", "id": "098e15c2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $x(a)$ represents a so-called input and $w(t-a)$ is normally called the weight function or kernel.\n", "\n", @@ -387,9 +353,7 @@ { "cell_type": "markdown", "id": "4dfb7ed3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "y(t) = \\left(x * w\\right)(t).\n", @@ -399,9 +363,7 @@ { "cell_type": "markdown", "id": "99b69cad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The discretized version reads" ] @@ -409,9 +371,7 @@ { "cell_type": "markdown", "id": "ab342220", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "y(t) = \\sum_{a=-\\infty}^{a=\\infty}x(a)w(t-a).\n", @@ -421,9 +381,7 @@ { "cell_type": "markdown", "id": "44d11bbe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Computing the inverse of the above convolution operations is known as deconvolution.\n", "\n", @@ -433,9 +391,7 @@ { "cell_type": "markdown", "id": "1331f0c5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convolution Examples: Polynomial multiplication\n", "\n", @@ -448,9 +404,7 @@ { "cell_type": "markdown", "id": "55402188", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(t) = \\alpha_0+\\alpha_1 t+\\alpha_2 t^2,\n", @@ -460,9 +414,7 @@ { "cell_type": "markdown", "id": "e98603af", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -470,9 +422,7 @@ { "cell_type": "markdown", "id": "0bf4d76c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "s(t) = \\beta_0+\\beta_1 t+\\beta_2 t^2+\\beta_3 t^3.\n", @@ -482,9 +432,7 @@ { "cell_type": "markdown", "id": "7a909516", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The polynomial multiplication gives us a new polynomial of degree $5$" ] @@ -492,9 +440,7 @@ { "cell_type": "markdown", "id": "40f9a726", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z(t) = \\delta_0+\\delta_1 t+\\delta_2 t^2+\\delta_3 t^3+\\delta_4 t^4+\\delta_5 t^5.\n", @@ -504,9 +450,7 @@ { "cell_type": "markdown", "id": "b6720a8f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Efficient Polynomial Multiplication\n", "\n", @@ -517,9 +461,7 @@ { "cell_type": "markdown", "id": "f515eff5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{split}\n", @@ -536,9 +478,7 @@ { "cell_type": "markdown", "id": "6dc0dfd7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We note that $\\alpha_i=0$ except for $i\\in \\left\\{0,1,2\\right\\}$ and $\\beta_i=0$ except for $i\\in\\left\\{0,1,2,3\\right\\}$.\n", "\n", @@ -548,9 +488,7 @@ { "cell_type": "markdown", "id": "619e31c1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j = \\sum_{i=-\\infty}^{i=\\infty}\\alpha_i\\beta_{j-i}=(\\alpha * \\beta)_j,\n", @@ -560,9 +498,7 @@ { "cell_type": "markdown", "id": "bd052428", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or as a double sum with restriction $l=i+j$" ] @@ -570,9 +506,7 @@ { "cell_type": "markdown", "id": "584f7ccb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_l = \\sum_{ij}\\alpha_i\\beta_{j}.\n", @@ -582,9 +516,7 @@ { "cell_type": "markdown", "id": "a7a12d8f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Do you see a potential drawback with these equations?" ] @@ -592,9 +524,7 @@ { "cell_type": "markdown", "id": "dedb3be2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A more efficient way of coding the above Convolution\n", "\n", @@ -606,9 +536,7 @@ { "cell_type": "markdown", "id": "524eab3c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\delta}=\\begin{bmatrix}\\alpha_0 & 0 & 0 & 0 \\\\\n", @@ -624,9 +552,7 @@ { "cell_type": "markdown", "id": "ad6d8e0e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The process is commutative and we can easily see that we can rewrite the multiplication in terms of a matrix holding $\\beta$ and a vector holding $\\alpha$.\n", "In this case we have" @@ -635,9 +561,7 @@ { "cell_type": "markdown", "id": "fc3e73a5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\delta}=\\begin{bmatrix}\\beta_0 & 0 & 0 \\\\\n", @@ -653,9 +577,7 @@ { "cell_type": "markdown", "id": "1766bfc3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that the use of these matrices is for mathematical purposes only and not implementation purposes.\n", "When implementing the above equation we do not encode (and allocate memory) the matrices explicitely.\n", @@ -673,9 +595,7 @@ { "cell_type": "markdown", "id": "d79f9173", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}=\\begin{bmatrix}a_0 & 0 & 0 \\\\\n", @@ -691,9 +611,7 @@ { "cell_type": "markdown", "id": "ad6ed7a5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with elements $a_{ii}=a_{i+1,j+1}=a_{i-j}$ is an example of a Toeplitz\n", "matrix. Such a matrix does not need to be a square matrix. Toeplitz\n", @@ -708,9 +626,7 @@ { "cell_type": "markdown", "id": "e792cb5f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)\n", "\n", @@ -720,9 +636,7 @@ { "cell_type": "markdown", "id": "d5aa44c1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "m\\frac{d^2x}{dt^2}+\\eta\\frac{dx}{dt}+x(t)=F(t),\n", @@ -732,9 +646,7 @@ { "cell_type": "markdown", "id": "2d9e1071", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $F(t)$ is an applied external force acting on the system (often\n", "called a driving force), one can use the theory of Fourier\n", @@ -748,9 +660,7 @@ { "cell_type": "markdown", "id": "74eb7fa4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -766,9 +676,7 @@ { "cell_type": "markdown", "id": "27373683", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This is known as the principle of superposition. It only applies when\n", "the homogenous equation is linear. \n", @@ -783,9 +691,7 @@ { "cell_type": "markdown", "id": "fd5e9b0f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{eqnarray}\n", @@ -797,9 +703,7 @@ { "cell_type": "markdown", "id": "b0651b77", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "One example of a non-sinusoidal periodic force is a square wave. Many\n", "components in electric circuits are non-linear, for example diodes. This \n", @@ -810,9 +714,7 @@ { "cell_type": "markdown", "id": "36246f87", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple Code Example\n", "\n", @@ -825,10 +727,7 @@ "cell_type": "code", "execution_count": 1, "id": "fa7c932a", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", @@ -855,9 +754,7 @@ { "cell_type": "markdown", "id": "aff076a3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For the sinusoidal example the\n", "period is $\\tau=2\\pi/\\omega$. However, higher harmonics can also\n", @@ -869,9 +766,7 @@ { "cell_type": "markdown", "id": "8879200e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -887,9 +782,7 @@ { "cell_type": "markdown", "id": "418c3a5e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Wrapping up Fourier transforms\n", "\n", @@ -902,9 +795,7 @@ { "cell_type": "markdown", "id": "2ef255bf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -920,9 +811,7 @@ { "cell_type": "markdown", "id": "256eacba", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The solutions for $x(t)$ then come from replacing $\\omega$ with\n", "$n\\omega$ for each term in the particular solution," @@ -931,9 +820,7 @@ { "cell_type": "markdown", "id": "ae377df9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{eqnarray}\n", @@ -951,9 +838,7 @@ { "cell_type": "markdown", "id": "c18496b9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Finding the Coefficients\n", "\n", @@ -970,9 +855,7 @@ { "cell_type": "markdown", "id": "98909a25", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -990,9 +873,7 @@ { "cell_type": "markdown", "id": "24ea20f8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "To check the consistency of these expressions and to verify\n", "Eq. ([4](#eq:fourierdef2)), one can insert the expansion of $F(t)$ in\n", @@ -1003,9 +884,7 @@ { "cell_type": "markdown", "id": "667eef29", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f_n=\\frac{2}{\\tau}\\int_{-\\tau/2}^{\\tau/2} dt~\\left\\{\\frac{f_0}{2}+\\sum_{m>0}f_m\\cos(m\\omega t)+g_m\\sin(m\\omega t)\\right\\}\\cos(n\\omega t).\n", @@ -1015,9 +894,7 @@ { "cell_type": "markdown", "id": "cc55e413", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Immediately, one can throw away all the terms with $g_m$ because they\n", "convolute an even and an odd function. The term with $f_0/2$\n", @@ -1032,9 +909,7 @@ { "cell_type": "markdown", "id": "a6736340", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -1050,9 +925,7 @@ { "cell_type": "markdown", "id": "f7fc5c92", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1060,9 +933,7 @@ { "cell_type": "markdown", "id": "145eb0ed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f_n=\\frac{2}{\\tau}\\int_{-\\tau/2}^{\\tau/2} dt~f_n/2=f_n.\n", @@ -1072,9 +943,7 @@ { "cell_type": "markdown", "id": "c55a6961", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The same method can be used to check for the consistency of $g_n$." ] @@ -1082,9 +951,7 @@ { "cell_type": "markdown", "id": "efe26d49", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final words on Fourier Transforms\n", "\n", @@ -1101,10 +968,7 @@ "cell_type": "code", "execution_count": 2, "id": "2d7c13f1", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -1141,9 +1005,7 @@ { "cell_type": "markdown", "id": "7eb888fa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Fourier transforms and convolution\n", "\n", @@ -1153,9 +1015,7 @@ { "cell_type": "markdown", "id": "ea8a36c0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{f}(y)=\\boldsymbol{F}[f(y)]=\\frac{1}{2\\pi}\\int_{-\\infty}^{\\infty} d\\omega \\exp{-i\\omega y} f(\\omega),\n", @@ -1165,9 +1025,7 @@ { "cell_type": "markdown", "id": "13ec89cc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and similarly we have" ] @@ -1175,9 +1033,7 @@ { "cell_type": "markdown", "id": "ec321c01", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{g}(y)=\\boldsymbol{F}[g(y)]=\\frac{1}{2\\pi}\\int_{-\\infty}^{\\infty} d\\omega \\exp{-i\\omega y} g(\\omega).\n", @@ -1187,9 +1043,7 @@ { "cell_type": "markdown", "id": "fb441853", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The inverse Fourier transform is given by" ] @@ -1197,9 +1051,7 @@ { "cell_type": "markdown", "id": "0e6c8c97", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{F}^{-1}[g(y)]=\\frac{1}{2\\pi}\\int_{-\\infty}^{\\infty} d\\omega \\exp{i\\omega y} g(\\omega).\n", @@ -1209,9 +1061,7 @@ { "cell_type": "markdown", "id": "926ad151", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The inverse Fourier transform of the product of the two functions $\\hat{f}\\hat{g}$ can be written as" ] @@ -1219,9 +1069,7 @@ { "cell_type": "markdown", "id": "6b94454c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{F}^{-1}[(\\hat{f}\\hat{g})(x)]=\\frac{1}{2\\pi}\\int_{-\\infty}^{\\infty} d\\omega \\exp{i\\omega x} \\hat{f}(\\omega)\\hat{g}(\\omega).\n", @@ -1231,9 +1079,7 @@ { "cell_type": "markdown", "id": "30482910", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can rewrite the latter as" ] @@ -1241,9 +1087,7 @@ { "cell_type": "markdown", "id": "de702fa7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{F}^{-1}[(\\hat{f}\\hat{g})(x)]=\\int_{-\\infty}^{\\infty} d\\omega \\exp{i\\omega x} \\hat{f}(\\omega)\\left[\\frac{1}{2\\pi}\\int_{-\\infty}^{\\infty}g(y)dy \\exp{-i\\omega y}\\right]=\\frac{1}{2\\pi}\\int_{-\\infty}^{\\infty}dy g(y)\\int_{-\\infty}^{\\infty} d\\omega \\hat{f}(\\omega) \\exp{i\\omega(x- y)},\n", @@ -1253,9 +1097,7 @@ { "cell_type": "markdown", "id": "8dc14e03", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is simply" ] @@ -1263,9 +1105,7 @@ { "cell_type": "markdown", "id": "58880666", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{F}^{-1}[(\\hat{f}\\hat{g})(x)]=\\int_{-\\infty}^{\\infty}dy g(y)f(x-y)=(f*g)(x),\n", @@ -1275,9 +1115,7 @@ { "cell_type": "markdown", "id": "8d5d93c6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "the convolution of the functions $f$ and $g$." ] @@ -1285,9 +1123,7 @@ { "cell_type": "markdown", "id": "b1b68406", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Two-dimensional Objects\n", "\n", @@ -1300,9 +1136,7 @@ { "cell_type": "markdown", "id": "f095e15f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "S_(i,j)=(I * K)(i,j) = \\sum_m\\sum_n I(m,n)K(i-m,j-n).\n", @@ -1312,9 +1146,7 @@ { "cell_type": "markdown", "id": "6d6c489a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Convolution is a commutatitave process, which means we can rewrite this equation as" ] @@ -1322,9 +1154,7 @@ { "cell_type": "markdown", "id": "76c943b7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "S_(i,j)=(I * K)(i,j) = \\sum_m\\sum_n I(i-m,j-n)K(m,n).\n", @@ -1334,9 +1164,7 @@ { "cell_type": "markdown", "id": "8f38d5d5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Normally the latter is more straightforward to implement in a machine larning library since there is less variation in the range of values of $m$ and $n$.\n", "\n", @@ -1346,9 +1174,7 @@ { "cell_type": "markdown", "id": "832c1e07", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "S_(i,j)=(I * K)(i,j) = \\sum_m\\sum_n I(i+m,j+n)K(m,n).\n", @@ -1358,9 +1184,7 @@ { "cell_type": "markdown", "id": "83a03b60", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on Dimensionalities\n", "\n", @@ -1385,9 +1209,7 @@ { "cell_type": "markdown", "id": "c495dae2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \\approx 10^{10},\n", @@ -1397,9 +1219,7 @@ { "cell_type": "markdown", "id": "13f7e07c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "that is ten billion parameters to determine." ] @@ -1407,9 +1227,7 @@ { "cell_type": "markdown", "id": "19ebaf8a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Further Dimensionality Remarks\n", "\n", @@ -1433,9 +1251,7 @@ { "cell_type": "markdown", "id": "598b540b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## CNNs in more detail\n", "\n", @@ -1446,9 +1262,7 @@ { "cell_type": "markdown", "id": "511322b6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{I}=\\begin{bmatrix}i_{00} & i_{01} & i_{02} \\\\\n", @@ -1460,9 +1274,7 @@ { "cell_type": "markdown", "id": "1c45deca", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1470,9 +1282,7 @@ { "cell_type": "markdown", "id": "9c41eaff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{W}=\\begin{bmatrix}w_{00} & w_{01} \\\\\n", @@ -1483,9 +1293,7 @@ { "cell_type": "markdown", "id": "64f84453", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We introduce now the hyperparameter $S$ **stride**. Stride represents how the filter $W$ moves the convolution process on the matrix $I$.\n", "We strongly recommend the repository on [Arithmetic of deep learning by Dumoulin and Visin](https://github.com/vdumoulin/conv_arithmetic) \n", @@ -1498,9 +1306,7 @@ { "cell_type": "markdown", "id": "e2cb4b08", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "S_(i,j)=(I * W)(i,j) = \\sum_m\\sum_n I(i-m,j-n)W(m,n),\n", @@ -1510,9 +1316,7 @@ { "cell_type": "markdown", "id": "37d20741", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and obtain" ] @@ -1520,9 +1324,7 @@ { "cell_type": "markdown", "id": "e864de9c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{S}=\\begin{bmatrix}i_{00}w_{00}+i_{01}w_{01}+i_{10}w_{10}+i_{11}w_{11} & i_{01}w_{00}+i_{02}w_{01}+i_{11}w_{10}+i_{12}w_{11} \\\\\n", @@ -1533,9 +1335,7 @@ { "cell_type": "markdown", "id": "7b6c2fcd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can rewrite this operation in terms of a matrix-vector multiplication by defining a new vector where we flatten out the inputs as a vector $\\boldsymbol{I}'$ of length $9$ and\n", "a matrix $\\boldsymbol{W}'$ with dimension $4\\times 9$ as" @@ -1544,9 +1344,7 @@ { "cell_type": "markdown", "id": "2ad6a202", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{I}'=\\begin{bmatrix}i_{00} \\\\ i_{01} \\\\ i_{02} \\\\ i_{10} \\\\ i_{11} \\\\ i_{12} \\\\ i_{20} \\\\ i_{21} \\\\ i_{22} \\end{bmatrix},\n", @@ -1556,9 +1354,7 @@ { "cell_type": "markdown", "id": "0d239cf4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and the new matrix" ] @@ -1566,9 +1362,7 @@ { "cell_type": "markdown", "id": "f653b7fe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{W}'=\\begin{bmatrix} w_{00} & w_{01} & 0 & w_{10} & w_{11} & 0 & 0 & 0 & 0 \\\\\n", @@ -1581,9 +1375,7 @@ { "cell_type": "markdown", "id": "f06adedd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see easily that performing the matrix-vector multiplication $\\boldsymbol{W}'\\boldsymbol{I}'$ is the same as the above convolution with stride $S=1$, that is" ] @@ -1591,9 +1383,7 @@ { "cell_type": "markdown", "id": "9313111f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "S=(\\boldsymbol{W}*\\boldsymbol{I}),\n", @@ -1603,9 +1393,7 @@ { "cell_type": "markdown", "id": "36340b3c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "is now given by $\\boldsymbol{W}'\\boldsymbol{I}'$ which is a vector of length $4$ instead of the originally resulting $2\\times 2$ output matrix.\n", "\n", @@ -1616,9 +1404,7 @@ { "cell_type": "markdown", "id": "7720c270", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{split}\n", @@ -1632,9 +1418,7 @@ { "cell_type": "markdown", "id": "7b10925c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The following properties affect the output size $o_j$ of a convolutional layer\n", "along axis $j$:\n", @@ -1660,9 +1444,7 @@ { "cell_type": "markdown", "id": "ec92fea8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Pooling\n", "\n", @@ -1710,9 +1492,7 @@ { "cell_type": "markdown", "id": "08aa5e12", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## No zero padding, unit strides\n", "\n", @@ -1725,9 +1505,7 @@ { "cell_type": "markdown", "id": "18e6266b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "o = (i - k) + 1.\n", @@ -1737,9 +1515,7 @@ { "cell_type": "markdown", "id": "588b8363", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Zero padding, unit strides\n", "\n", @@ -1754,9 +1530,7 @@ { "cell_type": "markdown", "id": "8f0af9ec", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "o = (i - k) + 2p + 1.\n", @@ -1766,9 +1540,7 @@ { "cell_type": "markdown", "id": "700387e0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Half (same) padding\n", "\n", @@ -1782,9 +1554,7 @@ { "cell_type": "markdown", "id": "c81ee97e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{split}\n", @@ -1798,9 +1568,7 @@ { "cell_type": "markdown", "id": "789a08f8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Full padding\n", "\n", @@ -1814,9 +1582,7 @@ { "cell_type": "markdown", "id": "f6ed2c6b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{split}\n", @@ -1829,9 +1595,7 @@ { "cell_type": "markdown", "id": "56b14905", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This is sometimes referred to as full padding, because in this\n", "setting every possible partial or complete superimposition of the kernel on the\n", @@ -1841,9 +1605,7 @@ { "cell_type": "markdown", "id": "e9efdbb2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Pooling arithmetic\n", "\n", @@ -1863,9 +1625,7 @@ { "cell_type": "markdown", "id": "26d22ac6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "o = \\left\\lfloor \\frac{i - k}{s} \\right\\rfloor + 1.\n", @@ -1875,9 +1635,7 @@ { "cell_type": "markdown", "id": "931da696", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## CNNs in more detail, building convolutional neural networks in Tensorflow and Keras\n", "\n", @@ -1894,9 +1652,7 @@ { "cell_type": "markdown", "id": "d217547f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting it up\n", "\n", @@ -1907,9 +1663,7 @@ { "cell_type": "markdown", "id": "27ecc7de", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(n_{inputs},\\, n_{pixels, width},\\, n_{pixels, height},\\, depth) .\n", @@ -1919,9 +1673,7 @@ { "cell_type": "markdown", "id": "8b06f6ba", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The MNIST dataset again\n", "\n", @@ -1940,9 +1692,7 @@ { "cell_type": "markdown", "id": "ad0a0a50", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Strong correlations\n", "\n", @@ -1961,9 +1711,7 @@ { "cell_type": "markdown", "id": "8583f2c0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Layers of a CNN\n", "The layers of a convolutional neural network arrange neurons in 3D: width, height and depth. \n", @@ -1986,9 +1734,7 @@ { "cell_type": "markdown", "id": "86e0bc8c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Systematic reduction\n", "\n", @@ -2005,22 +1751,36 @@ { "cell_type": "markdown", "id": "445a29e1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Prerequisites: Collect and pre-process data" ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 1, "id": "79884b14", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "inputs = (n_inputs, pixel_width, pixel_height, depth) = (1797, 8, 8, 1)\n", + "labels = (n_inputs) = (1797,)\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA7YAAADICAYAAADcOn20AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAQx0lEQVR4nO3dbWydZRkH8OuMjklYaEHMxla3ji46/KBdiItodB2oEUVXDNMYois6QgLCkKkhEGwHGDCaWIxODY510SUuJmSdZr4AdjUmxPBWEhaNjlBEzSYOOl+SbQwfPxhqN9Y68G7rdc7vl/QDZz3/c5+T6z7P899zOKtVVVUFAAAAJDVrphcAAAAA/wvFFgAAgNQUWwAAAFJTbAEAAEhNsQUAACA1xRYAAIDUFFsAAABSU2wBAABITbEFAAAgtYYptv39/VGr1eLhhx8ukler1eLTn/50kazxmb29va/6/nv37o2Pf/zjsWjRojjttNOivb09brjhhjhw4EC5RZJSvc9/b29v1Gq1CX++//3vF10r+dgD9kAjq/f5f+aZZ+LSSy+Nc889N04//fRobm6O5cuXx9e//vU4evRo0XWSj/lvHE0zvQDKePbZZ+Ntb3tbnHHGGXHbbbfFokWL4rHHHouenp4YHByMRx55JGbNapi/x6DBrFu3Lt73vve97PYrr7wynnzyyRP+GdQTe4BG9o9//CPOOOOMuOWWW2LRokVx5MiR2LVrV1x77bUxPDwc3/nOd2Z6iTBlzP9/KLZ1YmBgIA4cOBDbt2+Piy66KCIiVq1aFYcPH46bbropHn/88Vi+fPkMrxKmRmtra7S2th5z28jISOzZsycuv/zyaGlpmZmFwTSxB2hky5Yti61btx5z28UXXxx//vOfY+vWrfGNb3wj5syZM0Org6ll/v/DJbxxDh06FBs2bIiOjo5obm6Os846Ky644IIYGBiY8D7f/va34w1veEPMmTMn3vSmN53w41779u2Lq666KlpbW+PUU0+NJUuWxMaNG4t+PGD27NkREdHc3HzM7S+dzLzmNa8p9ljUp8zzfyL33HNPVFUV69atm9LHoX7YAzSyepv/iIjXve51MWvWrDjllFOm/LHIzfzXB1dsxzl8+HA899xz8dnPfjYWLlwYR44cifvvvz8+/OEPx5YtW+ITn/jEMb+/c+fOGBwcjFtvvTVOP/302LRpU3zsYx+LpqamuOyyyyLi3wO9YsWKmDVrVnzhC1+I9vb2ePDBB+P222+PkZGR2LJly6Rramtri4h//837ZLq6umLRokWxYcOG2LRpUyxevDgeffTRuPPOO+ODH/xgnHfeea/6daExZJ7/4/3zn/+M/v7+WLp0aaxcufIV3ZfGZQ/QyOph/quqihdffDH+9re/xc9+9rPo7++PDRs2RFOT010mZ/7rRNUgtmzZUkVE9dBDD530fY4ePVq98MIL1ac+9alq+fLlx/xZRFSnnXZatW/fvmN+f9myZdXSpUvHbrvqqququXPnVk8//fQx9//KV75SRUS1Z8+eYzJ7enqO+b329vaqvb39pNb7pz/9qbrggguqiBj7WbNmTXXo0KGTfcrUqUaY//F+/OMfVxFR3XHHHa/4vtQne4BG1ijzf8cdd4yd/9Rqtermm28+6ftSv8x/4/BR5OP84Ac/iHe84x0xd+7caGpqitmzZ8fmzZvj17/+9ct+96KLLop58+aN/fcpp5wSH/3oR2Pv3r3xhz/8ISIifvSjH8WqVatiwYIFcfTo0bGfiy++OCIihoaGJl3P3r17Y+/evf913c8//3ysXr06/vrXv8a2bdviF7/4RWzatCl++ctfxoc+9KGG+1Y0Xp2s83+8zZs3R1NTU3R3d7/i+9LY7AEaWfb57+7ujoceeih++tOfxuc///n48pe/HNdee+1J35/GZv7za6Br0//dvffeGx/5yEdizZo18bnPfS7mz58fTU1N8c1vfjPuueeel/3+/PnzJ7ztwIED0draGvv3748f/vCHY/8P7PH+8pe/FFn7l770pRgeHo6nn346zjnnnIiIeOc73xnLli2LCy+8MLZt2xZr164t8ljUp8zzf3zmzp074wMf+MAJ1wgTsQdoZPUw//Pnzx9bw3vf+94488wz48Ybb4xPfvKTvkCTSZn/+qDYjvO9730vlixZEtu3b49arTZ2++HDh0/4+/v27Zvwtte+9rUREXH22WfHm9/85vjiF794wowFCxb8r8uOiIjh4eFYuHDhWKl9yVvf+taIiHjiiSeKPA71K/P8j/fd7343jhw54gtzeMXsARpZvcz/eCtWrIiIiN/+9rcNc2LPq2P+64NiO06tVotTTz31mIHet2/fhN+I9sADD8T+/fvHPorw4osvxvbt26O9vX3sn1245JJLYteuXdHe3h5nnnnmlK19wYIF8cADD8Qf//jHWLhw4djtDz74YETEy/4ZCDhe5vkfb/PmzbFgwYKxj/rAybIHaGT1Mv/jDQ4ORkTE0qVLp/2xycX814eGK7Y///nPT/jtYu9///vjkksuiXvvvTeuvvrquOyyy+KZZ56J2267Lc4555z43e9+97L7nH322XHhhRfGLbfcMvaNaL/5zW+O+brvW2+9Ne677754+9vfHtddd1288Y1vjEOHDsXIyEjs2rUrvvWtb01aOl8axv/2Gftrrrkmtm3bFu95z3vixhtvjNe//vXxxBNPxO233x7z5s2Lyy+//CRfIepZvc7/S371q1/Fnj174qabbmqor7fn5NkDNLJ6nf+enp7Yv39/vOtd74qFCxfG6Oho/OQnP4m777471qxZE+eff/5JvkLUM/PfAGb626umy0vfiDbRz1NPPVVVVVXdeeedVVtbWzVnzpzqvPPOq+6+++6qp6enOv6liojqmmuuqTZt2lS1t7dXs2fPrpYtW1Zt27btZY/97LPPVtddd121ZMmSavbs2dVZZ51VnX/++dXNN99c/f3vfz8m8/hvRFu8eHG1ePHik3qOjz76aHXppZdWra2t1Zw5c6pzzz23WrduXfX73//+Fb1W1J9GmP+qqqorr7yyqtVq1ZNPPnnS96Ex2AM0snqf/507d1bvfve7q3nz5lVNTU3V3LlzqxUrVlRf+9rXqhdeeOEVv17UF/PfOGpVVVWlyzIAAABMF//cDwAAAKkptgAAAKSm2AIAAJCaYgsAAEBqii0AAACpKbYAAACkptgCAACQWtNML6CU/v7+onm9vb1F81paWormRUT09fUVzevs7Cyax/TZvXt30bzS+2nHjh1F8yIiDh48WDRvcHCwaJ79NL0GBgaK5q1fv75o3lQove/b2tqK5jGxkZGRonmlzwdKHwNKv19HRDQ3NxfNGx4eLppnP01udHS0aN7/+x4o/XwjzOyJuGILAABAaootAAAAqSm2AAAApKbYAgAAkJpiCwAAQGqKLQAAAKkptgAAAKSm2AIAAJCaYgsAAEBqii0AAACpKbYAAACkptgCAACQmmILAABAaootAAAAqSm2AAAApKbYAgAAkJpiCwAAQGqKLQAAAKk1zdQD7969u2jeFVdcUTRv9erVRfNaWlqK5kVEdHV1Fc0bHR0tmsf0uf7664vmlZ6F7u7uonkREXfddVfRvKnYo0xsZGSkaF7p98MMduzYUTSv9PsIE/t/f623bt1aNG9wcLBoXkT5Y4BzoOlV+vUu/X5Y+phSen0REf39/UXzent7i+bNBFdsAQAASE2xBQAAIDXFFgAAgNQUWwAAAFJTbAEAAEhNsQUAACA1xRYAAIDUFFsAAABSU2wBAABITbEFAAAgNcUWAACA1BRbAAAAUlNsAQAASE2xBQAAIDXFFgAAgNQUWwAAAFJTbAEAAEhNsQUAACA1xRYAAIDUalVVVTPxwNdff33RvJGRkaJ5O3bsKJrX2dlZNC8ioqWlpWhe6efM9Ck9/6Vna2hoqGheRMTatWuL5o2OjhbNY3r19fUVzevo6Ciat2rVqqJ5ERErV64smrd79+6iefCS0ud8ERHDw8NF88w/U2kqekDp41Tp4+hMcMUWAACA1BRbAAAAUlNsAQAASE2xBQAAIDXFFgAAgNQUWwAAAFJTbAEAAEhNsQUAACA1xRYAAIDUFFsAAABSU2wBAABITbEFAAAgNcUWAACA1BRbAAAAUlNsAQAASE2xBQAAIDXFFgAAgNQUWwAAAFJTbAEAAEitaaYeuK2trWjeyMhI0bze3t6ieUNDQ0XzIiIee+yx4pnkNDo6WjSv9H7q6ekpmhcR0dLSUjSv9HMu/R7H5Lq7u4vmlT4GTIXSx5XSzznDa8j06OjoKJ7Z399fNK/0cbT0MYrJlT6Gd3V1Fc2bCn19fTO9hP87rtgCAACQmmILAABAaootAAAAqSm2AAAApKbYAgAAkJpiCwAAQGqKLQAAAKkptgAAAKSm2AIAAJCaYgsAAEBqii0AAACpKbYAAACkptgCAACQmmILAABAaootAAAAqSm2AAAApKbYAgAAkJpiCwAAQGqKLQAAAKnVqqqqZnoRJXR0dBTNe/zxx4vmrV27tmheRER/f3/xTKbHwMBA0byurq6ieY2op6enaF5vb2/RvHozPDxcNK+zs7No3sGDB4vmTYXSx5XSM9vW1lY0D8YrPV+lj6N9fX1F85hcIx5TtmzZUjSvu7u7aN5McMUWAACA1BRbAAAAUlNsAQAASE2xBQAAIDXFFgAAgNQUWwAAAFJTbAEAAEhNsQUAACA1xRYAAIDUFFsAAABSU2wBAABITbEFAAAgNcUWAACA1BRbAAAAUlNsAQAASE2xBQAAIDXFFgAAgNQUWwAAAFJTbAEAAEhNsQUAACC1WlVV1UwvooSOjo6ZXsKkWlpaimeWfs59fX1F85jY7t27i+bt2LGjaN7w8HDRvJGRkaJ5EeXXOBV7lImV3gOrVq0qmlfa6tWri2eW3veQSWdn50wvYVKl3+Pqzejo6EwvYVKlzwmmYl5Ln1tNxbnadHPFFgAAgNQUWwAAAFJTbAEAAEhNsQUAACA1xRYAAIDUFFsAAABSU2wBAABITbEFAAAgNcUWAACA1BRbAAAAUlNsAQAASE2xBQAAIDXFFgAAgNQUWwAAAFJTbAEAAEhNsQUAACA1xRYAAIDUFFsAAABSU2wBAABIrWmmF1BKS0tL0bzOzs6ieb29vUXzIso/59JrLL2+elJ6vg4ePFg0r7+/v2heV1dX0bwI85Vd6T2wfv36onl33XVX0bwrrriiaB6MNzAwUDRv8eLFRfOGh4eL5k1F5lScpzGxoaGhonk9PT1F8zZu3Fg0r7u7u2heRPnjyujoaNG8mThPc8UWAACA1BRbAAAAUlNsAQAASE2xBQAAIDXFFgAAgNQUWwAAAFJTbAEAAEhNsQUAACA1xRYAAIDUFFsAAABSU2wBAABITbEFAAAgNcUWAACA1BRbAAAAUlNsAQAASE2xBQAAIDXFFgAAgNQUWwAAAFJTbAEAAEitaaYXUMpnPvOZonldXV1F8zZu3Fg0LyJi9erVRfNaWlqK5jF9nn/++aJ5Bw8eLJrX3d1dNA+m2lve8paieaXfr2G8r371q0XzhoaGiuY1NzcXzYsof1xxnJpeK1euLJrX2dlZNK/0nhodHS2aFxGxfv36onn10ANcsQUAACA1xRYAAIDUFFsAAABSU2wBAABITbEFAAAgNcUWAACA1BRbAAAAUlNsAQAASE2xBQAAIDXFFgAAgNQUWwAAAFJTbAEAAEhNsQUAACA1xRYAAIDUFFsAAABSU2wBAABITbEFAAAgNcUWAACA1BRbAAAAUqtVVVXN9CIAAADg1XLFFgAAgNQUWwAAAFJTbAEAAEhNsQUAACA1xRYAAIDUFFsAAABSU2wBAABITbEFAAAgNcUWAACA1P4F8SqxxH5w1EMAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ "# import necessary packages\n", "import numpy as np\n", @@ -2067,21 +1827,16 @@ { "cell_type": "markdown", "id": "6fafadc0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Importing Keras and Tensorflow" ] }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 2, "id": "d10aa7b4", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from tensorflow.keras import datasets, layers, models\n", @@ -2111,21 +1866,16 @@ { "cell_type": "markdown", "id": "4542534a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Running with Keras" ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 3, "id": "95dcc4fc", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "def create_convolutional_neural_network_keras(input_shape, receptive_field,\n", @@ -2159,22 +1909,285 @@ { "cell_type": "markdown", "id": "404c8750", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final part" ] }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 4, "id": "b26480e4", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Metal device set to: Apple M1\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/optimizer_v2/gradient_descent.py:102: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n", + " super(SGD, self).__init__(name, **kwargs)\n", + "2023-11-02 08:18:04.108752: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "12/12 [==============================] - 2s 83ms/step - loss: 3.4256 - accuracy: 0.0917\n", + "Learning rate = 1e-05\n", + "Lambda = 1e-05\n", + "Test accuracy: 0.092\n", + "\n", + "12/12 [==============================] - 2s 73ms/step - loss: 3.4338 - accuracy: 0.0944\n", + "Learning rate = 1e-05\n", + "Lambda = 0.0001\n", + "Test accuracy: 0.094\n", + "\n", + "12/12 [==============================] - 2s 73ms/step - loss: 3.5186 - accuracy: 0.0944\n", + "Learning rate = 1e-05\n", + "Lambda = 0.001\n", + "Test accuracy: 0.094\n", + "\n", + "12/12 [==============================] - 2s 74ms/step - loss: 4.3646 - accuracy: 0.0944\n", + "Learning rate = 1e-05\n", + "Lambda = 0.01\n", + "Test accuracy: 0.094\n", + "\n", + "12/12 [==============================] - 2s 79ms/step - loss: 12.7761 - accuracy: 0.0944\n", + "Learning rate = 1e-05\n", + "Lambda = 0.1\n", + "Test accuracy: 0.094\n", + "\n", + "12/12 [==============================] - 2s 76ms/step - loss: 92.0269 - accuracy: 0.0944\n", + "Learning rate = 1e-05\n", + "Lambda = 1.0\n", + "Test accuracy: 0.094\n", + "\n", + "12/12 [==============================] - 2s 77ms/step - loss: 519.9384 - accuracy: 0.0972\n", + "Learning rate = 1e-05\n", + "Lambda = 10.0\n", + "Test accuracy: 0.097\n", + "\n", + "12/12 [==============================] - 2s 81ms/step - loss: 1.5349 - accuracy: 0.4694\n", + "Learning rate = 0.0001\n", + "Lambda = 1e-05\n", + "Test accuracy: 0.469\n", + "\n", + "12/12 [==============================] - 2s 81ms/step - loss: 1.5442 - accuracy: 0.4667\n", + "Learning rate = 0.0001\n", + "Lambda = 0.0001\n", + "Test accuracy: 0.467\n", + "\n", + "12/12 [==============================] - 2s 81ms/step - loss: 1.6288 - accuracy: 0.4667\n", + "Learning rate = 0.0001\n", + "Lambda = 0.001\n", + "Test accuracy: 0.467\n", + "\n", + "12/12 [==============================] - 2s 77ms/step - loss: 2.4714 - accuracy: 0.4722\n", + "Learning rate = 0.0001\n", + "Lambda = 0.01\n", + "Test accuracy: 0.472\n", + "\n", + "12/12 [==============================] - 2s 80ms/step - loss: 10.4141 - accuracy: 0.4556\n", + "Learning rate = 0.0001\n", + "Lambda = 0.1\n", + "Test accuracy: 0.456\n", + "\n", + "12/12 [==============================] - 2s 79ms/step - loss: 53.5240 - accuracy: 0.4472\n", + "Learning rate = 0.0001\n", + "Lambda = 1.0\n", + "Test accuracy: 0.447\n", + "\n", + "12/12 [==============================] - 2s 82ms/step - loss: 4.6259 - accuracy: 0.0889\n", + "Learning rate = 0.0001\n", + "Lambda = 10.0\n", + "Test accuracy: 0.089\n", + "\n", + "12/12 [==============================] - 2s 79ms/step - loss: 0.2641 - accuracy: 0.9194\n", + "Learning rate = 0.001\n", + "Lambda = 1e-05\n", + "Test accuracy: 0.919\n", + "\n", + "12/12 [==============================] - 2s 79ms/step - loss: 0.2736 - accuracy: 0.9167\n", + "Learning rate = 0.001\n", + "Lambda = 0.0001\n", + "Test accuracy: 0.917\n", + "\n", + "12/12 [==============================] - 2s 82ms/step - loss: 0.3604 - accuracy: 0.9167\n", + "Learning rate = 0.001\n", + "Lambda = 0.001\n", + "Test accuracy: 0.917\n", + "\n", + "12/12 [==============================] - 2s 94ms/step - loss: 1.1848 - accuracy: 0.9167\n", + "Learning rate = 0.001\n", + "Lambda = 0.01\n", + "Test accuracy: 0.917\n", + "\n", + "12/12 [==============================] - 2s 94ms/step - loss: 5.7980 - accuracy: 0.9250\n", + "Learning rate = 0.001\n", + "Lambda = 0.1\n", + "Test accuracy: 0.925\n", + "\n", + "12/12 [==============================] - 2s 92ms/step - loss: 2.6003 - accuracy: 0.3472\n", + "Learning rate = 0.001\n", + "Lambda = 1.0\n", + "Test accuracy: 0.347\n", + "\n", + "12/12 [==============================] - 2s 95ms/step - loss: 2.3032 - accuracy: 0.0889\n", + "Learning rate = 0.001\n", + "Lambda = 10.0\n", + "Test accuracy: 0.089\n", + "\n", + "12/12 [==============================] - 2s 96ms/step - loss: 0.0958 - accuracy: 0.9694\n", + "Learning rate = 0.01\n", + "Lambda = 1e-05\n", + "Test accuracy: 0.969\n", + "\n", + "12/12 [==============================] - 2s 97ms/step - loss: 0.1060 - accuracy: 0.9750\n", + "Learning rate = 0.01\n", + "Lambda = 0.0001\n", + "Test accuracy: 0.975\n", + "\n", + "12/12 [==============================] - 2s 96ms/step - loss: 0.1862 - accuracy: 0.9778\n", + "Learning rate = 0.01\n", + "Lambda = 0.001\n", + "Test accuracy: 0.978\n", + "\n", + "12/12 [==============================] - 2s 96ms/step - loss: 0.6819 - accuracy: 0.9750\n", + "Learning rate = 0.01\n", + "Lambda = 0.01\n", + "Test accuracy: 0.975\n", + "\n", + "12/12 [==============================] - 2s 96ms/step - loss: 0.9978 - accuracy: 0.9028\n", + "Learning rate = 0.01\n", + "Lambda = 0.1\n", + "Test accuracy: 0.903\n", + "\n", + "12/12 [==============================] - 2s 98ms/step - loss: 2.3064 - accuracy: 0.0889\n", + "Learning rate = 0.01\n", + "Lambda = 1.0\n", + "Test accuracy: 0.089\n", + "\n", + "12/12 [==============================] - 2s 95ms/step - loss: 2.3065 - accuracy: 0.0889\n", + "Learning rate = 0.01\n", + "Lambda = 10.0\n", + "Test accuracy: 0.089\n", + "\n", + "12/12 [==============================] - 2s 93ms/step - loss: 0.2141 - accuracy: 0.9528\n", + "Learning rate = 0.1\n", + "Lambda = 1e-05\n", + "Test accuracy: 0.953\n", + "\n", + "12/12 [==============================] - 2s 95ms/step - loss: 0.2714 - accuracy: 0.9472\n", + "Learning rate = 0.1\n", + "Lambda = 0.0001\n", + "Test accuracy: 0.947\n", + "\n", + "12/12 [==============================] - 2s 89ms/step - loss: 0.2996 - accuracy: 0.9556\n", + "Learning rate = 0.1\n", + "Lambda = 0.001\n", + "Test accuracy: 0.956\n", + "\n", + "12/12 [==============================] - 2s 88ms/step - loss: 0.6088 - accuracy: 0.8611\n", + "Learning rate = 0.1\n", + "Lambda = 0.01\n", + "Test accuracy: 0.861\n", + "\n", + "12/12 [==============================] - 2s 93ms/step - loss: 1.7069 - accuracy: 0.6556\n", + "Learning rate = 0.1\n", + "Lambda = 0.1\n", + "Test accuracy: 0.656\n", + "\n", + "12/12 [==============================] - 2s 89ms/step - loss: 2.3077 - accuracy: 0.0778\n", + "Learning rate = 0.1\n", + "Lambda = 1.0\n", + "Test accuracy: 0.078\n", + "\n", + "12/12 [==============================] - 3s 117ms/step - loss: nan - accuracy: 0.0778\n", + "Learning rate = 0.1\n", + "Lambda = 10.0\n", + "Test accuracy: 0.078\n", + "\n", + "12/12 [==============================] - 2s 109ms/step - loss: 22.0022 - accuracy: 0.0778\n", + "Learning rate = 1.0\n", + "Lambda = 1e-05\n", + "Test accuracy: 0.078\n", + "\n", + "12/12 [==============================] - 3s 104ms/step - loss: 44.2070 - accuracy: 0.0778\n", + "Learning rate = 1.0\n", + "Lambda = 0.0001\n", + "Test accuracy: 0.078\n", + "\n", + "12/12 [==============================] - 3s 112ms/step - loss: 6.3536 - accuracy: 0.0778\n", + "Learning rate = 1.0\n", + "Lambda = 0.001\n", + "Test accuracy: 0.078\n", + "\n", + "12/12 [==============================] - 3s 125ms/step - loss: 2.3082 - accuracy: 0.0889\n", + "Learning rate = 1.0\n", + "Lambda = 0.01\n", + "Test accuracy: 0.089\n", + "\n", + "12/12 [==============================] - 3s 122ms/step - loss: 2.3126 - accuracy: 0.0778\n", + "Learning rate = 1.0\n", + "Lambda = 0.1\n", + "Test accuracy: 0.078\n", + "\n", + "12/12 [==============================] - 3s 123ms/step - loss: nan - accuracy: 0.0778\n", + "Learning rate = 1.0\n", + "Lambda = 1.0\n", + "Test accuracy: 0.078\n", + "\n", + "12/12 [==============================] - 3s 120ms/step - loss: nan - accuracy: 0.0778\n", + "Learning rate = 1.0\n", + "Lambda = 10.0\n", + "Test accuracy: 0.078\n", + "\n", + "12/12 [==============================] - 3s 116ms/step - loss: 6130353.0000 - accuracy: 0.1056\n", + "Learning rate = 10.0\n", + "Lambda = 1e-05\n", + "Test accuracy: 0.106\n", + "\n", + "12/12 [==============================] - 3s 120ms/step - loss: 388451.5000 - accuracy: 0.0917\n", + "Learning rate = 10.0\n", + "Lambda = 0.0001\n", + "Test accuracy: 0.092\n", + "\n", + "12/12 [==============================] - 3s 117ms/step - loss: 2.4314 - accuracy: 0.0889\n", + "Learning rate = 10.0\n", + "Lambda = 0.001\n", + "Test accuracy: 0.089\n", + "\n", + "12/12 [==============================] - 3s 120ms/step - loss: 2.5014 - accuracy: 0.0889\n", + "Learning rate = 10.0\n", + "Lambda = 0.01\n", + "Test accuracy: 0.089\n", + "\n", + "12/12 [==============================] - 3s 131ms/step - loss: nan - accuracy: 0.0778\n", + "Learning rate = 10.0\n", + "Lambda = 0.1\n", + "Test accuracy: 0.078\n", + "\n", + "12/12 [==============================] - 3s 124ms/step - loss: nan - accuracy: 0.0778\n", + "Learning rate = 10.0\n", + "Lambda = 1.0\n", + "Test accuracy: 0.078\n", + "\n", + "12/12 [==============================] - 3s 127ms/step - loss: nan - accuracy: 0.0778\n", + "Learning rate = 10.0\n", + "Lambda = 10.0\n", + "Test accuracy: 0.078\n", + "\n" + ] + } + ], "source": [ "CNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", " \n", @@ -2197,9 +2210,7 @@ { "cell_type": "markdown", "id": "67264745", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final visualization" ] @@ -2208,10 +2219,7 @@ "cell_type": "code", "execution_count": 7, "id": "db92d957", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# visual representation of grid search\n", @@ -2249,9 +2257,7 @@ { "cell_type": "markdown", "id": "8f104706", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The CIFAR01 data set\n", "\n", @@ -2263,12 +2269,9 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 5, "id": "665b4103", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import tensorflow as tf\n", @@ -2286,9 +2289,7 @@ { "cell_type": "markdown", "id": "34035c79", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Verifying the data set\n", "\n", @@ -2297,13 +2298,19 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 6, "id": "966ea558", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "ename": "SyntaxError", + "evalue": "invalid non-printable character U+200B (2758649921.py, line 3)", + "output_type": "error", + "traceback": [ + "\u001b[0;36m Input \u001b[0;32mIn [6]\u001b[0;36m\u001b[0m\n\u001b[0;31m ​\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid non-printable character U+200B\n" + ] + } + ], "source": [ "class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',\n", " 'dog', 'frog', 'horse', 'ship', 'truck']\n", @@ -2324,9 +2331,7 @@ { "cell_type": "markdown", "id": "8685c2f8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Set up the model\n", "\n", @@ -2339,10 +2344,7 @@ "cell_type": "code", "execution_count": 10, "id": "30f87291", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "model = models.Sequential()\n", @@ -2360,9 +2362,7 @@ { "cell_type": "markdown", "id": "b7c68c02", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "You can see that the output of every Conv2D and MaxPooling2D layer is a 3D tensor of shape (height, width, channels). The width and height dimensions tend to shrink as you go deeper in the network. The number of output channels for each Conv2D layer is controlled by the first argument (e.g., 32 or 64). Typically, as the width and height shrink, you can afford (computationally) to add more output channels in each Conv2D layer." ] @@ -2370,9 +2370,7 @@ { "cell_type": "markdown", "id": "a3eefda1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Add Dense layers on top\n", "\n", @@ -2389,10 +2387,7 @@ "cell_type": "code", "execution_count": 11, "id": "725ce3a1", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "model.add(layers.Flatten())\n", @@ -2406,9 +2401,7 @@ { "cell_type": "markdown", "id": "2f0e86ba", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "As you can see, our (4, 4, 64) outputs were flattened into vectors of shape (1024) before going through two Dense layers." ] @@ -2416,9 +2409,7 @@ { "cell_type": "markdown", "id": "9fe3e5a8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Compile and train the model" ] @@ -2427,10 +2418,7 @@ "cell_type": "code", "execution_count": 12, "id": "904da8d6", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "model.compile(optimizer='adam',\n", @@ -2444,9 +2432,7 @@ { "cell_type": "markdown", "id": "98cf0684", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Finally, evaluate the model" ] @@ -2455,10 +2441,7 @@ "cell_type": "code", "execution_count": 13, "id": "b040f2e9", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "plt.plot(history.history['accuracy'], label='accuracy')\n", @@ -2476,9 +2459,7 @@ { "cell_type": "markdown", "id": "837aed5a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Building our own CNN code\n", "\n", @@ -2507,9 +2488,7 @@ { "cell_type": "markdown", "id": "af874397", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### List of contents:\n", "\n", @@ -2531,9 +2510,7 @@ { "cell_type": "markdown", "id": "32d09325", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Schedulers\n", "\n", @@ -2554,10 +2531,7 @@ "cell_type": "code", "execution_count": 14, "id": "8b746f30", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2695,9 +2669,7 @@ { "cell_type": "markdown", "id": "d8f99677", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Usage of schedulers\n", "\n", @@ -2708,10 +2680,7 @@ "cell_type": "code", "execution_count": 15, "id": "4bd66798", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n", @@ -2721,9 +2690,7 @@ { "cell_type": "markdown", "id": "8fdac498", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here is a small example for how a segment of code using schedulers could look. Switching out the schedulers is simple." ] @@ -2732,10 +2699,7 @@ "cell_type": "code", "execution_count": 16, "id": "ffe6bf79", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "weights = np.ones((3,3))\n", @@ -2754,9 +2718,7 @@ { "cell_type": "markdown", "id": "da6184be", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Cost functions\n", "\n", @@ -2770,10 +2732,7 @@ "cell_type": "code", "execution_count": 17, "id": "d6f94f7a", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "def CostOLS(target):\n", @@ -2817,9 +2776,7 @@ { "cell_type": "markdown", "id": "2e31a305", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Usage of cost functions\n", "\n", @@ -2832,10 +2789,7 @@ "cell_type": "code", "execution_count": 18, "id": "86787efe", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from autograd import grad\n", @@ -2853,9 +2807,7 @@ { "cell_type": "markdown", "id": "4dce940e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Activation functions\n", "\n", @@ -2870,10 +2822,7 @@ "cell_type": "code", "execution_count": 19, "id": "9d2c299f", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\n", @@ -2929,9 +2878,7 @@ { "cell_type": "markdown", "id": "83b08081", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Usage of activation functions\n", "\n", @@ -2946,10 +2893,7 @@ "cell_type": "code", "execution_count": 20, "id": "25d64b84", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "z = np.array([[4, 5, 6]]).T\n", @@ -2967,9 +2911,7 @@ { "cell_type": "markdown", "id": "2ac7f432", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Convolution\n", "\n", @@ -2984,9 +2926,7 @@ { "cell_type": "markdown", "id": "9e40a4b8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(f \\ast g)(t):=\\int_{-\\infty}^{\\infty} f(\\tau) g(t-\\tau) d \\tau.\n", @@ -2996,9 +2936,7 @@ { "cell_type": "markdown", "id": "c6f8f49d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here, f and g are the two functions on which we want to perform an\n", "operation. The outcome of the convolution operation is represented by\n", @@ -3012,9 +2950,7 @@ { "cell_type": "markdown", "id": "990d7b4c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(f \\ast g)[n]=\\sum_{m=0}^{n-1} f[m] g[n-m].\n", @@ -3024,9 +2960,7 @@ { "cell_type": "markdown", "id": "cdcb5e34", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The key idea we utilize to extract the information contained in an\n", "image is to slide an $m \\times n$ matrix *g* over an $m \\times n$\n", @@ -3039,9 +2973,7 @@ { "cell_type": "markdown", "id": "1a31b881", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(f \\ast g)[i, j]\\sum_{m=0}^{M-1}\\sum_{n=0}^{N-1} f[m,n] g[i-m, j-n].\n", @@ -3051,9 +2983,7 @@ { "cell_type": "markdown", "id": "bc501ada", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It is imperative to note that the size of the kernel g is\n", "significantly smaller than the size of the input image f, thereby\n", @@ -3069,9 +2999,7 @@ { "cell_type": "markdown", "id": "826b08dc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f = \\begin{bmatrix}\n", @@ -3088,9 +3016,7 @@ { "cell_type": "markdown", "id": "005de3fb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and a $3 \\times 3$ kernel *g* called a low-pass filter. Note that the\n", "kernel is usually rotated by 180 degrees during convolution, however\n", @@ -3100,9 +3026,7 @@ { "cell_type": "markdown", "id": "05a330e4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "g = \\frac{1}{9}\n", @@ -3117,9 +3041,7 @@ { "cell_type": "markdown", "id": "2b2f22cd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In order to filter the image, we have to extract a $3 \\times 3$\n", "element from the upper left corner of *f*, and perform element-wise\n", @@ -3130,9 +3052,7 @@ { "cell_type": "markdown", "id": "96f92012", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{bmatrix}\n", @@ -3159,9 +3079,7 @@ { "cell_type": "markdown", "id": "9da8697b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Then, following the multiplication, we summarize all the elements of the resulting matrix A:" ] @@ -3169,9 +3087,7 @@ { "cell_type": "markdown", "id": "ca826ccb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(f \\ast g)[0, 0]= \\sum_{i=0}^{2} \\sum_{j=0}^{2} a_{i,j} = 5\n", @@ -3181,9 +3097,7 @@ { "cell_type": "markdown", "id": "bee3496b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Which corresponds to the first element of the filtered image $(f \\ast g)$.\n", "\n", @@ -3202,9 +3116,7 @@ { "cell_type": "markdown", "id": "56669311", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(f \\ast g) =\n", @@ -3220,9 +3132,7 @@ { "cell_type": "markdown", "id": "b012e0eb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The result is markedly smaller in shape than the original image. This occurs when using convolution without first padding the image with additional columns and rows, allowing us to keep the original image shape after sliding the kernel over the image.\n", "How many rows and columns we wish to pad the image with depends strictly on the shape of the kernel, as we wish to pad the image with *r* additional rows and *c* additional columns." @@ -3231,9 +3141,7 @@ { "cell_type": "markdown", "id": "638c9e09", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "r =\\lfloor \\frac{kernel\\ height}{2} \\rfloor \\cdot 2 \\\\\n", @@ -3244,9 +3152,7 @@ { "cell_type": "markdown", "id": "7774dd55", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note the notation $\\lfloor \\frac{kernel width}{2} \\rfloor$ means that\n", "we floor the result of the division, meaning we round down to a whole\n", @@ -3268,9 +3174,7 @@ { "cell_type": "markdown", "id": "eaecffed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{bmatrix}\n", @@ -3290,9 +3194,7 @@ { "cell_type": "markdown", "id": "9492033c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Below we have provided code that demonstrates padding and convolution. As you will see when we run the code, the size of the image will remain unchanged when using padding.~" ] @@ -3301,10 +3203,7 @@ "cell_type": "code", "execution_count": 21, "id": "0554e3ab", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -3383,9 +3282,7 @@ { "cell_type": "markdown", "id": "0eeb696e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Fun fact: When filtering images, you will see that convolution involves rotating the kernel by 180 degrees. \n", "However, this is not the case when applying convolution in a CNN, where the same operation not rotated by 180 degrees is called \n", @@ -3396,10 +3293,7 @@ "cell_type": "code", "execution_count": 22, "id": "85bbe906", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\n", @@ -3423,9 +3317,7 @@ { "cell_type": "markdown", "id": "c5a5e0db", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "As you can see, the resulting image is of the same size as the\n", "original image. To round of our demonstration of convolution, we will\n", @@ -3444,10 +3336,7 @@ "cell_type": "code", "execution_count": 23, "id": "eaa84aaa", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# Now an example using a real image and first a gaussian low-pass filter and then a sobel filter\n", @@ -3494,9 +3383,7 @@ { "cell_type": "markdown", "id": "96fa5861", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Layers\n", "\n", @@ -3509,10 +3396,7 @@ "cell_type": "code", "execution_count": 24, "id": "d8afa149", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import math\n", @@ -3550,9 +3434,7 @@ { "cell_type": "markdown", "id": "72355a22", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Convolution2DLayer: convolution in a hidden layer\n", "\n", @@ -3592,10 +3474,7 @@ "cell_type": "code", "execution_count": 25, "id": "32559517", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "class Convolution2DLayer(Layer):\n", @@ -3878,9 +3757,7 @@ { "cell_type": "markdown", "id": "fc174600", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Backpropagation in the convolutional layer\n", "\n", @@ -3901,9 +3778,7 @@ { "cell_type": "markdown", "id": "8eacacfe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Demonstration\n", "\n", @@ -3914,10 +3789,7 @@ "cell_type": "code", "execution_count": 26, "id": "46196b67", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -3962,9 +3834,7 @@ { "cell_type": "markdown", "id": "dc29eda8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We cobserve that the result has half the pixels on each axis due to\n", "the fact that we've used a horizontal and vertical stride of 2. The\n", @@ -3984,9 +3854,7 @@ { "cell_type": "markdown", "id": "f77ea3fb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Pooling Layer\n", "\n", @@ -4006,10 +3874,7 @@ "cell_type": "code", "execution_count": 27, "id": "8738e00a", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "class Pooling2DLayer(Layer):\n", @@ -4168,9 +4033,7 @@ { "cell_type": "markdown", "id": "eee176d8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Flattening Layer\n", "\n", @@ -4189,10 +4052,7 @@ "cell_type": "code", "execution_count": 28, "id": "8b9d8469", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "class FlattenLayer(Layer):\n", @@ -4252,9 +4112,7 @@ { "cell_type": "markdown", "id": "65bb0e6d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Fully Connected Layers\n", "\n", @@ -4276,10 +4134,7 @@ "cell_type": "code", "execution_count": 29, "id": "20a4a7d1", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "class FullyConnectedLayer(Layer):\n", @@ -4504,9 +4359,7 @@ { "cell_type": "markdown", "id": "f02f8704", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Optimized Convolution2DLayer\n", "\n", @@ -4525,10 +4378,7 @@ "cell_type": "code", "execution_count": 30, "id": "abe01496", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "class Convolution2DLayerOPT(Convolution2DLayer):\n", @@ -4829,9 +4679,7 @@ { "cell_type": "markdown", "id": "d9317467", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### The Convolutional Neural Network (CNN)\n", "\n", @@ -4842,10 +4690,7 @@ "cell_type": "code", "execution_count": 31, "id": "dea10f08", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import math\n", @@ -5377,9 +5222,7 @@ { "cell_type": "markdown", "id": "e7315a76", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Usage of CNN code\n", "\n", @@ -5396,10 +5239,7 @@ "cell_type": "code", "execution_count": 32, "id": "8fcf6203", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n", @@ -5409,9 +5249,7 @@ { "cell_type": "markdown", "id": "4a2a562a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Now that we have our CNN object, we can begin to add layers to it!\n", "Many of the add_layer functions have default values, for example\n", @@ -5425,10 +5263,7 @@ "cell_type": "code", "execution_count": 33, "id": "db3f1eb5", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "cnn.add_Convolution2DLayer(\n", @@ -5451,9 +5286,7 @@ { "cell_type": "markdown", "id": "b7303b63", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here we have created a CNN with the following architecture:\n", "\n", @@ -5475,10 +5308,7 @@ "cell_type": "code", "execution_count": 34, "id": "f1403d34", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import fetch_openml\n", @@ -5510,9 +5340,7 @@ { "cell_type": "markdown", "id": "0d565479", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Now we may train our model. Note that we can utilize regularization in\n", "the CNN by using the lam (lambda) parameter in fit(), and utilize\n", @@ -5529,10 +5357,7 @@ "cell_type": "code", "execution_count": 35, "id": "7a4f925e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "scores = cnn.fit(\n", @@ -5557,9 +5382,7 @@ { "cell_type": "markdown", "id": "f896c2cf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Considering we only trained the model for 100 epochs without any tuning of the hyperparameters, this result is pretty good.\n", "\n", @@ -5586,10 +5409,7 @@ "cell_type": "code", "execution_count": 36, "id": "1f785537", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n", @@ -5655,9 +5475,7 @@ { "cell_type": "markdown", "id": "b301a921", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here we see the use of asymmetrical 1D kernels such as the $7 \\times\n", "1$ kernel in the first convolutional layer, both max and average\n", @@ -5673,9 +5491,7 @@ { "cell_type": "markdown", "id": "2820a187", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Additional Remarks\n", "\n", @@ -5697,10 +5513,7 @@ "cell_type": "code", "execution_count": 37, "id": "03397138", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "def convolve(image, kernel, stride=1):\n", @@ -5728,9 +5541,7 @@ { "cell_type": "markdown", "id": "d880c6e5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Remarks on the speed\n", "\n", @@ -5758,9 +5569,7 @@ { "cell_type": "markdown", "id": "508b0369", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Convolution using separable kernels" ] @@ -5769,10 +5578,7 @@ "cell_type": "code", "execution_count": 38, "id": "50089042", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "def conv2DSep(image, kernel, coef, stride=1, pad=\"zero\"):\n", @@ -5819,9 +5625,7 @@ { "cell_type": "markdown", "id": "a17d2eea", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "By taking advantage of the capabilities of separable kernels, we can\n", "effectively cut the computational expense of filtering an image in\n", @@ -5840,9 +5644,7 @@ { "cell_type": "markdown", "id": "46da3baa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Convolution in the Fourier domain" ] @@ -5851,10 +5653,7 @@ "cell_type": "code", "execution_count": 39, "id": "5e72abeb", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "start_time = time.time()\n", @@ -5872,9 +5671,7 @@ { "cell_type": "markdown", "id": "eefbce6f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It is evident that executing convolution in the Fourier domain yields\n", "the quickest computation time. Nonetheless, one should exercise\n", @@ -5889,7 +5686,25 @@ ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.10" + } + }, "nbformat": 4, "nbformat_minor": 5 }