diff --git a/doc/pub/week44/html/._week44-bs001.html b/doc/pub/week44/html/._week44-bs001.html index 5555f64c1..2f2a580de 100644 --- a/doc/pub/week44/html/._week44-bs001.html +++ b/doc/pub/week44/html/._week44-bs001.html @@ -359,12 +359,12 @@ MathJax.Hub.Config({
  • Convolutional Neural Networks
  • Readings and Videos:
    • diff --git a/doc/pub/week44/html/._week44-bs038.html b/doc/pub/week44/html/._week44-bs038.html index ce55b0d63..e3dfaca01 100644 --- a/doc/pub/week44/html/._week44-bs038.html +++ b/doc/pub/week44/html/._week44-bs038.html @@ -356,7 +356,7 @@ MathJax.Hub.Config({

    In total we have

    $$ -\left(F\times F\times D_1)\right) \times K+(K\mathrm{--biases}), +\left(F\times F\times D_1\right) \times K+K_{\mathrm{biases}}, $$

    parameters to train by back propagation.

    diff --git a/doc/pub/week44/html/week44-reveal.html b/doc/pub/week44/html/week44-reveal.html index 6f2942bdf..5f68b7a7a 100644 --- a/doc/pub/week44/html/week44-reveal.html +++ b/doc/pub/week44/html/week44-reveal.html @@ -205,17 +205,17 @@ MathJax.Hub.Config({

  • Readings and Videos:

    • @@ -1190,7 +1190,7 @@ $$

    In total we have

     
    $$ -\left(F\times F\times D_1)\right) \times K+(K\mathrm{--biases}), +\left(F\times F\times D_1\right) \times K+K_{\mathrm{biases}}, $$

     
    diff --git a/doc/pub/week44/html/week44-solarized.html b/doc/pub/week44/html/week44-solarized.html index ee5abe447..0f294ff20 100644 --- a/doc/pub/week44/html/week44-solarized.html +++ b/doc/pub/week44/html/week44-solarized.html @@ -299,12 +299,12 @@ MathJax.Hub.Config({

  • Convolutional Neural Networks
  • Readings and Videos:
    • @@ -1169,7 +1169,7 @@ $$

    In total we have

    $$ -\left(F\times F\times D_1)\right) \times K+(K\mathrm{--biases}), +\left(F\times F\times D_1\right) \times K+K_{\mathrm{biases}}, $$

    parameters to train by back propagation.

    diff --git a/doc/pub/week44/html/week44.html b/doc/pub/week44/html/week44.html index 088207579..5592c0311 100644 --- a/doc/pub/week44/html/week44.html +++ b/doc/pub/week44/html/week44.html @@ -376,12 +376,12 @@ MathJax.Hub.Config({
  • Convolutional Neural Networks
  • Readings and Videos:
    • @@ -1246,7 +1246,7 @@ $$

    In total we have

    $$ -\left(F\times F\times D_1)\right) \times K+(K\mathrm{--biases}), +\left(F\times F\times D_1\right) \times K+K_{\mathrm{biases}}, $$

    parameters to train by back propagation.

    diff --git a/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz b/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz index 24586dcc3..74399d932 100644 Binary files a/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz and b/doc/pub/week44/ipynb/ipynb-week44-src.tar.gz differ diff --git a/doc/pub/week44/ipynb/week44.ipynb b/doc/pub/week44/ipynb/week44.ipynb index e2219cc41..6460a1fd0 100644 --- a/doc/pub/week44/ipynb/week44.ipynb +++ b/doc/pub/week44/ipynb/week44.ipynb @@ -2,8 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "afaf8390", - "metadata": {}, + "id": "eb5b9ba3", + "metadata": { + "editable": true + }, "source": [ "\n", @@ -12,8 +14,10 @@ }, { "cell_type": "markdown", - "id": "f17d25d5", - "metadata": {}, + "id": "26c3b839", + "metadata": { + "editable": true + }, "source": [ "# Week 44, Convolutional Neural Networks (CNN)\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", @@ -23,8 +27,10 @@ }, { "cell_type": "markdown", - "id": "736a723a", - "metadata": {}, + "id": "43b75648", + "metadata": { + "editable": true + }, "source": [ "## Plan for week 44\n", "\n", @@ -34,23 +40,25 @@ "\n", "2. Readings and Videos:\n", "\n", - " * These lecture notes\n", + " * These lecture notes at \n", "\n", " * For a more in depth discussion on neural networks we recommend Goodfellow et al chapter 9. See also chapter 11 and 12 on practicalities and applications\n", "\n", - " * Reading suggestions for implementation of CNNs: [Rashcka et al.'s chapter 14](https://github.com/rasbt/machine-learning-book/tree/main/ch14). T \n", + " * Reading suggestions for implementation of CNNs see :\"https://github.com/rasbt/machine-learning-book/tree/main/ch14\". \n", "\n", - " * [Video on Deep Learning](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi)\n", + " * Video on Deep Learning at \n", "\n", - " * [Video on Convolutional Neural Networks from MIT](https://www.youtube.com/watch?v=iaSUYvmCekI&ab_channel=AlexanderAmini)\n", + " * Video on Convolutional Neural Networks from MIT at \n", "\n", - " * [Video on CNNs from Stanford](https://www.youtube.com/watch?v=bNb2fEVKeEo&list=PLC1qU-LWwrF64f4QKQT-Vg5Wr4qEE1Zxk&index=6&ab_channel=StanfordUniversitySchoolofEngineering)" + " * Video on CNNs from Stanford at " ] }, { "cell_type": "markdown", - "id": "93caafca", - "metadata": {}, + "id": "e4b628e2", + "metadata": { + "editable": true + }, "source": [ "## Lab sessions on Tuesday and Wednesday\n", "\n", @@ -63,16 +71,20 @@ }, { "cell_type": "markdown", - "id": "c99e8ca9", - "metadata": {}, + "id": "bdbb2f02", + "metadata": { + "editable": true + }, "source": [ "## Material for Lecture Monday October 28" ] }, { "cell_type": "markdown", - "id": "5a52c2d6", - "metadata": {}, + "id": "6869e73d", + "metadata": { + "editable": true + }, "source": [ "## Convolutional Neural Networks (recognizing images)\n", "\n", @@ -96,8 +108,10 @@ }, { "cell_type": "markdown", - "id": "05ae37f5", - "metadata": {}, + "id": "ad008db0", + "metadata": { + "editable": true + }, "source": [ "## What is the Difference\n", "\n", @@ -110,8 +124,10 @@ }, { "cell_type": "markdown", - "id": "e174ca11", - "metadata": {}, + "id": "99968c7d", + "metadata": { + "editable": true + }, "source": [ "## Neural Networks vs CNNs\n", "\n", @@ -126,8 +142,10 @@ }, { "cell_type": "markdown", - "id": "a8f5c3af", - "metadata": {}, + "id": "ede81326", + "metadata": { + "editable": true + }, "source": [ "## Why CNNS for images, sound files, medical images from CT scans etc?\n", "\n", @@ -154,8 +172,10 @@ }, { "cell_type": "markdown", - "id": "e784f567", - "metadata": {}, + "id": "3eda879d", + "metadata": { + "editable": true + }, "source": [ "## Regular NNs don’t scale well to full images\n", "\n", @@ -182,8 +202,10 @@ }, { "cell_type": "markdown", - "id": "d28632ea", - "metadata": {}, + "id": "71201a7e", + "metadata": { + "editable": true + }, "source": [ "## 3D volumes of neurons\n", "\n", @@ -220,8 +242,10 @@ }, { "cell_type": "markdown", - "id": "8fdd2e1e", - "metadata": {}, + "id": "5c19c911", + "metadata": { + "editable": true + }, "source": [ "## More on Dimensionalities\n", "\n", @@ -245,8 +269,10 @@ }, { "cell_type": "markdown", - "id": "fe077130", - "metadata": {}, + "id": "4e36a19f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \\approx 10^{10},\n", @@ -255,16 +281,20 @@ }, { "cell_type": "markdown", - "id": "62200bea", - "metadata": {}, + "id": "15bdcf25", + "metadata": { + "editable": true + }, "source": [ "that is ten billion parameters to determine." ] }, { "cell_type": "markdown", - "id": "64a09aab", - "metadata": {}, + "id": "20e64a8f", + "metadata": { + "editable": true + }, "source": [ "## Further remarks\n", "\n", @@ -279,8 +309,10 @@ }, { "cell_type": "markdown", - "id": "ec7a27eb", - "metadata": {}, + "id": "6ac3273b", + "metadata": { + "editable": true + }, "source": [ "## Layers used to build CNNs\n", "\n", @@ -306,8 +338,10 @@ }, { "cell_type": "markdown", - "id": "891c4e18", - "metadata": {}, + "id": "63e60007", + "metadata": { + "editable": true + }, "source": [ "## Transforming images\n", "\n", @@ -326,8 +360,10 @@ }, { "cell_type": "markdown", - "id": "f9a224d5", - "metadata": {}, + "id": "d9d4ad18", + "metadata": { + "editable": true + }, "source": [ "## CNNs in brief\n", "\n", @@ -346,8 +382,10 @@ }, { "cell_type": "markdown", - "id": "74ebf7e1", - "metadata": {}, + "id": "010d56f0", + "metadata": { + "editable": true + }, "source": [ "## A deep CNN model ([From Raschka et al](https://github.com/rasbt/machine-learning-book))\n", "\n", @@ -360,8 +398,10 @@ }, { "cell_type": "markdown", - "id": "127a2805", - "metadata": {}, + "id": "4876c352", + "metadata": { + "editable": true + }, "source": [ "## Key Idea\n", "\n", @@ -375,8 +415,10 @@ }, { "cell_type": "markdown", - "id": "bdfeefdc", - "metadata": {}, + "id": "cb4c5e7b", + "metadata": { + "editable": true + }, "source": [ "## How to do image compression before the era of deep learning\n", "\n", @@ -397,8 +439,10 @@ }, { "cell_type": "markdown", - "id": "5b9c0de1", - "metadata": {}, + "id": "f4dc5cbd", + "metadata": { + "editable": true + }, "source": [ "## The SVD example" ] @@ -406,23 +450,12 @@ { "cell_type": "code", "execution_count": 1, - "id": "36668e14", - "metadata": {}, - "outputs": [ - { - "ename": "FileNotFoundError", - "evalue": "[Errno 2] No such file or directory: 'figslides/photo1.jpg'", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mFileNotFoundError\u001b[0m Traceback (most recent call last)", - "Cell \u001b[0;32mIn[1], line 12\u001b[0m\n\u001b[1;32m 10\u001b[0m plt\u001b[38;5;241m.\u001b[39mrcParams[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mfigure.figsize\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m [\u001b[38;5;241m16\u001b[39m, \u001b[38;5;241m8\u001b[39m]\n\u001b[1;32m 11\u001b[0m \u001b[38;5;66;03m# Import image\u001b[39;00m\n\u001b[0;32m---> 12\u001b[0m A \u001b[38;5;241m=\u001b[39m \u001b[43mimread\u001b[49m\u001b[43m(\u001b[49m\u001b[43mos\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpath\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mjoin\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mfigslides/photo1.jpg\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 13\u001b[0m X \u001b[38;5;241m=\u001b[39m A\u001b[38;5;241m.\u001b[39mdot([\u001b[38;5;241m0.299\u001b[39m, \u001b[38;5;241m0.5870\u001b[39m, \u001b[38;5;241m0.114\u001b[39m]) \u001b[38;5;66;03m# Convert RGB to grayscale\u001b[39;00m\n\u001b[1;32m 14\u001b[0m img \u001b[38;5;241m=\u001b[39m plt\u001b[38;5;241m.\u001b[39mimshow(X)\n", - "File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/matplotlib/image.py:1544\u001b[0m, in \u001b[0;36mimread\u001b[0;34m(fname, format)\u001b[0m\n\u001b[1;32m 1537\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(fname, \u001b[38;5;28mstr\u001b[39m) \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(parse\u001b[38;5;241m.\u001b[39murlparse(fname)\u001b[38;5;241m.\u001b[39mscheme) \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[1;32m 1538\u001b[0m \u001b[38;5;66;03m# Pillow doesn't handle URLs directly.\u001b[39;00m\n\u001b[1;32m 1539\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m 1540\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mPlease open the URL for reading and pass the \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 1541\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mresult to Pillow, e.g. with \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 1542\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m``np.array(PIL.Image.open(urllib.request.urlopen(url)))``.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 1543\u001b[0m )\n\u001b[0;32m-> 1544\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[43mimg_open\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfname\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m image:\n\u001b[1;32m 1545\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m (_pil_png_to_float_array(image)\n\u001b[1;32m 1546\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(image, PIL\u001b[38;5;241m.\u001b[39mPngImagePlugin\u001b[38;5;241m.\u001b[39mPngImageFile) \u001b[38;5;28;01melse\u001b[39;00m\n\u001b[1;32m 1547\u001b[0m pil_to_array(image))\n", - "File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/PIL/Image.py:3247\u001b[0m, in \u001b[0;36mopen\u001b[0;34m(fp, mode, formats)\u001b[0m\n\u001b[1;32m 3244\u001b[0m filename \u001b[38;5;241m=\u001b[39m fp\n\u001b[1;32m 3246\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m filename:\n\u001b[0;32m-> 3247\u001b[0m fp \u001b[38;5;241m=\u001b[39m \u001b[43mbuiltins\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mopen\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfilename\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mrb\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 3248\u001b[0m exclusive_fp \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[1;32m 3250\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n", - "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'figslides/photo1.jpg'" - ] - } - ], + "id": "82c8bc1e", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "%matplotlib inline\n", "\n", @@ -488,8 +521,10 @@ }, { "cell_type": "markdown", - "id": "68fd7bf4", - "metadata": {}, + "id": "6dba3ada", + "metadata": { + "editable": true + }, "source": [ "## Mathematics of CNNs\n", "\n", @@ -509,8 +544,10 @@ }, { "cell_type": "markdown", - "id": "af30ff6d", - "metadata": {}, + "id": "207a033a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(t) = \\int x(a) w(t-a) da,\n", @@ -519,8 +556,10 @@ }, { "cell_type": "markdown", - "id": "0b6a0a87", - "metadata": {}, + "id": "f07342b7", + "metadata": { + "editable": true + }, "source": [ "where $x(a)$ represents a so-called input and $w(t-a)$ is normally called the weight function or kernel.\n", "\n", @@ -529,8 +568,10 @@ }, { "cell_type": "markdown", - "id": "06204c0f", - "metadata": {}, + "id": "f65ae7e2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(t) = \\left(x * w\\right)(t).\n", @@ -539,16 +580,20 @@ }, { "cell_type": "markdown", - "id": "f793221b", - "metadata": {}, + "id": "26e8c43a", + "metadata": { + "editable": true + }, "source": [ "The discretized version reads" ] }, { "cell_type": "markdown", - "id": "754789c3", - "metadata": {}, + "id": "ff63939b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(t) = \\sum_{a=-\\infty}^{a=\\infty}x(a)w(t-a).\n", @@ -557,8 +602,10 @@ }, { "cell_type": "markdown", - "id": "edaf2d6b", - "metadata": {}, + "id": "6920d5c5", + "metadata": { + "editable": true + }, "source": [ "Computing the inverse of the above convolution operations is known as deconvolution and the process is commutative.\n", "\n", @@ -567,8 +614,10 @@ }, { "cell_type": "markdown", - "id": "c10e0e6e", - "metadata": {}, + "id": "5c81fc69", + "metadata": { + "editable": true + }, "source": [ "## Mathematics of CNNs\n", "\n", @@ -588,8 +637,10 @@ }, { "cell_type": "markdown", - "id": "3e21ff12", - "metadata": {}, + "id": "29145257", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(t) = \\int x(a) w(t-a) da,\n", @@ -598,8 +649,10 @@ }, { "cell_type": "markdown", - "id": "763cf144", - "metadata": {}, + "id": "d69392c2", + "metadata": { + "editable": true + }, "source": [ "where $x(a)$ represents a so-called input and $w(t-a)$ is normally called the weight function or kernel.\n", "\n", @@ -608,8 +661,10 @@ }, { "cell_type": "markdown", - "id": "d50d06cc", - "metadata": {}, + "id": "3bcc01e9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(t) = \\left(x * w\\right)(t).\n", @@ -618,16 +673,20 @@ }, { "cell_type": "markdown", - "id": "6924641b", - "metadata": {}, + "id": "17ac2b45", + "metadata": { + "editable": true + }, "source": [ "The discretized version reads" ] }, { "cell_type": "markdown", - "id": "187e8990", - "metadata": {}, + "id": "00ad9c0b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(t) = \\sum_{a=-\\infty}^{a=\\infty}x(a)w(t-a).\n", @@ -636,8 +695,10 @@ }, { "cell_type": "markdown", - "id": "6470335b", - "metadata": {}, + "id": "cca6de09", + "metadata": { + "editable": true + }, "source": [ "Computing the inverse of the above convolution operations is known as deconvolution and the process is commutative.\n", "\n", @@ -646,8 +707,10 @@ }, { "cell_type": "markdown", - "id": "ec95c776", - "metadata": {}, + "id": "f9c24acc", + "metadata": { + "editable": true + }, "source": [ "## Convolution Examples: Polynomial multiplication\n", "\n", @@ -663,8 +726,10 @@ }, { "cell_type": "markdown", - "id": "d4b3c227", - "metadata": {}, + "id": "c249ab57", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(t) = \\alpha_0+\\alpha_1 t+\\alpha_2 t^2,\n", @@ -673,16 +738,20 @@ }, { "cell_type": "markdown", - "id": "9d58c7b8", - "metadata": {}, + "id": "cc2e004b", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "122efa36", - "metadata": {}, + "id": "d5ceddc1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "s(t) = \\beta_0+\\beta_1 t+\\beta_2 t^2+\\beta_3 t^3.\n", @@ -691,16 +760,20 @@ }, { "cell_type": "markdown", - "id": "195156ea", - "metadata": {}, + "id": "4435c19d", + "metadata": { + "editable": true + }, "source": [ "The polynomial multiplication gives us a new polynomial of degree $5$" ] }, { "cell_type": "markdown", - "id": "19dc2f3a", - "metadata": {}, + "id": "55069e63", + "metadata": { + "editable": true + }, "source": [ "$$\n", "z(t) = \\delta_0+\\delta_1 t+\\delta_2 t^2+\\delta_3 t^3+\\delta_4 t^4+\\delta_5 t^5.\n", @@ -709,8 +782,10 @@ }, { "cell_type": "markdown", - "id": "4f9f3064", - "metadata": {}, + "id": "c99899c6", + "metadata": { + "editable": true + }, "source": [ "## Efficient Polynomial Multiplication\n", "\n", @@ -720,8 +795,10 @@ }, { "cell_type": "markdown", - "id": "478460ae", - "metadata": {}, + "id": "be8c662d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{split}\n", @@ -737,8 +814,10 @@ }, { "cell_type": "markdown", - "id": "e73dc29f", - "metadata": {}, + "id": "a04e7c0b", + "metadata": { + "editable": true + }, "source": [ "We note that $\\alpha_i=0$ except for $i\\in \\left\\{0,1,2\\right\\}$ and $\\beta_i=0$ except for $i\\in\\left\\{0,1,2,3\\right\\}$.\n", "\n", @@ -747,8 +826,10 @@ }, { "cell_type": "markdown", - "id": "5fc26071", - "metadata": {}, + "id": "338ac316", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\delta_j = \\sum_{i=-\\infty}^{i=\\infty}\\alpha_i\\beta_{j-i}=(\\alpha * \\beta)_j,\n", @@ -757,16 +838,20 @@ }, { "cell_type": "markdown", - "id": "555caa5e", - "metadata": {}, + "id": "b5d95275", + "metadata": { + "editable": true + }, "source": [ "or as a double sum with restriction $l=i+j$" ] }, { "cell_type": "markdown", - "id": "1e18c64b", - "metadata": {}, + "id": "252e9e0f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\delta_l = \\sum_{ij}\\alpha_i\\beta_{j}.\n", @@ -775,8 +860,10 @@ }, { "cell_type": "markdown", - "id": "8f490b21", - "metadata": {}, + "id": "035cdf4a", + "metadata": { + "editable": true + }, "source": [ "## Further simplification\n", "\n", @@ -785,8 +872,10 @@ }, { "cell_type": "markdown", - "id": "ed3baa45", - "metadata": {}, + "id": "d499561c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\delta_i = \\sum_{k=0}^{k=m-1}\\alpha_k\\beta_{i-k},\n", @@ -795,8 +884,10 @@ }, { "cell_type": "markdown", - "id": "2cb35d1f", - "metadata": {}, + "id": "b3184764", + "metadata": { + "editable": true + }, "source": [ "where $m=3$ in our case, the maximum length of\n", "the vector $\\alpha$. Note that the vector $\\boldsymbol{\\beta}$ has length $n=4$. Below we will find an even more efficient representation." @@ -804,8 +895,10 @@ }, { "cell_type": "markdown", - "id": "b3a4e62a", - "metadata": {}, + "id": "c1f2bb44", + "metadata": { + "editable": true + }, "source": [ "## A more efficient way of coding the above Convolution\n", "\n", @@ -816,8 +909,10 @@ }, { "cell_type": "markdown", - "id": "fbc69d73", - "metadata": {}, + "id": "724342ac", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\delta}=\\begin{bmatrix}\\alpha_0 & 0 & 0 & 0 \\\\\n", @@ -832,8 +927,10 @@ }, { "cell_type": "markdown", - "id": "c2692c2d", - "metadata": {}, + "id": "74231212", + "metadata": { + "editable": true + }, "source": [ "## Commutative process\n", "\n", @@ -843,8 +940,10 @@ }, { "cell_type": "markdown", - "id": "cdcce1ea", - "metadata": {}, + "id": "e49db96c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\delta}=\\begin{bmatrix}\\beta_0 & 0 & 0 \\\\\n", @@ -859,8 +958,10 @@ }, { "cell_type": "markdown", - "id": "5e82c416", - "metadata": {}, + "id": "33bfc290", + "metadata": { + "editable": true + }, "source": [ "Note that the use of these matrices is for mathematical purposes only\n", "and not implementation purposes. When implementing the above equation\n", @@ -871,8 +972,10 @@ }, { "cell_type": "markdown", - "id": "066fb9c0", - "metadata": {}, + "id": "292a27b8", + "metadata": { + "editable": true + }, "source": [ "## Toeplitz matrices\n", "\n", @@ -885,8 +988,10 @@ }, { "cell_type": "markdown", - "id": "7164a220", - "metadata": {}, + "id": "1aadd8f9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}=\\begin{bmatrix}a_0 & 0 & 0 \\\\\n", @@ -901,8 +1006,10 @@ }, { "cell_type": "markdown", - "id": "5b0b6645", - "metadata": {}, + "id": "2c04c419", + "metadata": { + "editable": true + }, "source": [ "with elements $a_{ii}=a_{i+1,j+1}=a_{i-j}$ is an example of a Toeplitz\n", "matrix. Such a matrix does not need to be a square matrix. Toeplitz\n", @@ -915,8 +1022,10 @@ }, { "cell_type": "markdown", - "id": "78c20ffc", - "metadata": {}, + "id": "a17e2ce7", + "metadata": { + "editable": true + }, "source": [ "## Fourier series and Toeplitz matrices\n", "\n", @@ -928,8 +1037,10 @@ }, { "cell_type": "markdown", - "id": "7d6555e5", - "metadata": {}, + "id": "2590ecf5", + "metadata": { + "editable": true + }, "source": [ "## Generalizing the above one-dimensional case\n", "\n", @@ -945,8 +1056,10 @@ }, { "cell_type": "markdown", - "id": "cb3ae3f4", - "metadata": {}, + "id": "97e18086", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(i)= \\left(x*w\\right)(i)= \\sum_{k=0}^{k=m-1}w(k)x(i-k),\n", @@ -955,8 +1068,10 @@ }, { "cell_type": "markdown", - "id": "ba84498b", - "metadata": {}, + "id": "49670cad", + "metadata": { + "editable": true + }, "source": [ "where $m=3$ in our case, the maximum length of the vector $\\boldsymbol{w}$.\n", "Here the symbol $*$ represents the mathematical operation of convolution." @@ -964,8 +1079,10 @@ }, { "cell_type": "markdown", - "id": "afdd9570", - "metadata": {}, + "id": "73e4091f", + "metadata": { + "editable": true + }, "source": [ "## Memory considerations\n", "\n", @@ -983,8 +1100,10 @@ }, { "cell_type": "markdown", - "id": "9d8a1259", - "metadata": {}, + "id": "b974944d", + "metadata": { + "editable": true + }, "source": [ "## Padding\n", "\n", @@ -997,8 +1116,10 @@ }, { "cell_type": "markdown", - "id": "93166a7d", - "metadata": {}, + "id": "15319ba5", + "metadata": { + "editable": true + }, "source": [ "## New vector\n", "\n", @@ -1013,8 +1134,10 @@ }, { "cell_type": "markdown", - "id": "217d00a9", - "metadata": {}, + "id": "aeb47546", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(i) = \\sum_{k=0}^{k=m-1}w(k)x(i+(m-1)-k).\n", @@ -1023,16 +1146,20 @@ }, { "cell_type": "markdown", - "id": "2dcc48bf", - "metadata": {}, + "id": "10d44759", + "metadata": { + "editable": true + }, "source": [ "As an example, we have" ] }, { "cell_type": "markdown", - "id": "28874f6c", - "metadata": {}, + "id": "bb1af8aa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(4)=x(6)w(0)+x(5)w(1)+x(4)w(2)=0\\times \\alpha_0+\\beta_3\\alpha_1+\\beta_2\\alpha_2,\n", @@ -1041,8 +1168,10 @@ }, { "cell_type": "markdown", - "id": "2ba7a9dc", - "metadata": {}, + "id": "0ddf7069", + "metadata": { + "editable": true + }, "source": [ "as before except that we have an additional term $x(6)w(0)$, which is zero.\n", "\n", @@ -1051,8 +1180,10 @@ }, { "cell_type": "markdown", - "id": "4daa6524", - "metadata": {}, + "id": "2a6cd4c6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(5)=x(7)w(0)+x(6)w(1)+x(5)w(2)=0\\times \\alpha_0+0\\times\\alpha_1+\\beta_3\\alpha_2.\n", @@ -1061,16 +1192,20 @@ }, { "cell_type": "markdown", - "id": "c5ef900a", - "metadata": {}, + "id": "f8e3ce61", + "metadata": { + "editable": true + }, "source": [ "The zeroth-order term is" ] }, { "cell_type": "markdown", - "id": "a8e76abb", - "metadata": {}, + "id": "08c8f403", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(0)=x(2)w(0)+x(1)w(1)+x(0)w(2)=\\beta_0 \\alpha_0+0\\times\\alpha_1+0\\times\\alpha_2=\\alpha_0\\beta_0.\n", @@ -1079,8 +1214,10 @@ }, { "cell_type": "markdown", - "id": "3c8e17ec", - "metadata": {}, + "id": "b64dcf7f", + "metadata": { + "editable": true + }, "source": [ "## Rewriting as dot products\n", "\n", @@ -1089,8 +1226,10 @@ }, { "cell_type": "markdown", - "id": "965e447e", - "metadata": {}, + "id": "56fb4c30", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(0)=x(2)w(0)+x(1)w(1)+x(0)w(2)=x(2)\\tilde{w}(2)+x(1)\\tilde{w}(1)+x(0)\\tilde{w}(0),\n", @@ -1099,8 +1238,10 @@ }, { "cell_type": "markdown", - "id": "53e47b06", - "metadata": {}, + "id": "e56843df", + "metadata": { + "editable": true + }, "source": [ "with $\\tilde{w}(0)=w(2)$, $\\tilde{w}(1)=w(1)$, and $\\tilde{w}(2)=w(0)$, we can then rewrite the above sum as a dot product of\n", "$x(i:i+(m-1))\\tilde{w}$ for element $y(i)$, where $x(i:i+(m-1))$ is simply a patch of $\\boldsymbol{x}$ of size $m-1$.\n", @@ -1113,8 +1254,10 @@ }, { "cell_type": "markdown", - "id": "b7a66ef0", - "metadata": {}, + "id": "fe166596", + "metadata": { + "editable": true + }, "source": [ "## Cross correlation\n", "\n", @@ -1124,8 +1267,10 @@ }, { "cell_type": "markdown", - "id": "bd4feb76", - "metadata": {}, + "id": "dc50439a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(i) = \\sum_{k=-\\infty}^{k=\\infty}w(k)x(i-k),\n", @@ -1134,16 +1279,20 @@ }, { "cell_type": "markdown", - "id": "1b68f20e", - "metadata": {}, + "id": "ce7f37bc", + "metadata": { + "editable": true + }, "source": [ "we have now" ] }, { "cell_type": "markdown", - "id": "76daf815", - "metadata": {}, + "id": "1b024c34", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y(i) = \\sum_{k=-\\infty}^{k=\\infty}w(k)x(i+k).\n", @@ -1152,8 +1301,10 @@ }, { "cell_type": "markdown", - "id": "79ff7b3c", - "metadata": {}, + "id": "1b53fb45", + "metadata": { + "editable": true + }, "source": [ "Both TensorFlow and PyTorch (as well as our own code example below),\n", "implement the last equation, although it is normally referred to as\n", @@ -1165,8 +1316,10 @@ }, { "cell_type": "markdown", - "id": "29705589", - "metadata": {}, + "id": "d019fba8", + "metadata": { + "editable": true + }, "source": [ "## Two-dimensional objects\n", "\n", @@ -1178,8 +1331,10 @@ }, { "cell_type": "markdown", - "id": "8af7b9af", - "metadata": {}, + "id": "fc1470d2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "Y(i,j)=(X * W)(i,j) = \\sum_m\\sum_n X(m,n)W(i-m,j-n).\n", @@ -1188,16 +1343,20 @@ }, { "cell_type": "markdown", - "id": "860db2b8", - "metadata": {}, + "id": "1adf88a4", + "metadata": { + "editable": true + }, "source": [ "Convolution is a commutative process, which means we can rewrite this equation as" ] }, { "cell_type": "markdown", - "id": "3eb3e476", - "metadata": {}, + "id": "081d258b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "Y(i,j)=(X * W)(i,j) = \\sum_m\\sum_n X(i-m,j-n)W(m,n).\n", @@ -1206,8 +1365,10 @@ }, { "cell_type": "markdown", - "id": "faf1c3be", - "metadata": {}, + "id": "232aa6bc", + "metadata": { + "editable": true + }, "source": [ "Normally the latter is more straightforward to implement in a machine\n", "larning library since there is less variation in the range of values\n", @@ -1220,8 +1381,10 @@ }, { "cell_type": "markdown", - "id": "33a2839a", - "metadata": {}, + "id": "ba023438", + "metadata": { + "editable": true + }, "source": [ "$$\n", "Y(i,j)=(X * W)(i,j) = \\sum_m\\sum_n X(i+m,j+n)W(m,n).\n", @@ -1230,8 +1393,10 @@ }, { "cell_type": "markdown", - "id": "36a0a4cd", - "metadata": {}, + "id": "9c825a5f", + "metadata": { + "editable": true + }, "source": [ "## CNNs in more detail, simple example\n", "\n", @@ -1241,8 +1406,10 @@ }, { "cell_type": "markdown", - "id": "c1d8190e", - "metadata": {}, + "id": "64bf9fbc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}x_{00} & x_{01} & x_{02} \\\\\n", @@ -1253,16 +1420,20 @@ }, { "cell_type": "markdown", - "id": "d90132aa", - "metadata": {}, + "id": "060bccca", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "1bc5f98e", - "metadata": {}, + "id": "ebf54553", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{W}=\\begin{bmatrix}w_{00} & w_{01} \\\\\n", @@ -1272,8 +1443,10 @@ }, { "cell_type": "markdown", - "id": "125bab1b", - "metadata": {}, + "id": "73d2b460", + "metadata": { + "editable": true + }, "source": [ "We introduce now the hyperparameter $S$ **stride**. Stride represents how the filter $W$ moves the convolution process on the matrix $X$.\n", "We strongly recommend the repository on [Arithmetic of deep learning by Dumoulin and Visin](https://github.com/vdumoulin/conv_arithmetic) \n", @@ -1285,8 +1458,10 @@ }, { "cell_type": "markdown", - "id": "a1cf4eab", - "metadata": {}, + "id": "d796a4fa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "Y_(i,j)=(X * W)(i,j) = \\sum_m\\sum_n X(i-m,j-n)W(m,n),\n", @@ -1295,16 +1470,20 @@ }, { "cell_type": "markdown", - "id": "7d134ea5", - "metadata": {}, + "id": "da9c9aa5", + "metadata": { + "editable": true + }, "source": [ "and obtain" ] }, { "cell_type": "markdown", - "id": "b8b5ba3b", - "metadata": {}, + "id": "fbd0f9e4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{Y}=\\begin{bmatrix}x_{00}w_{00}+x_{01}w_{01}+x_{10}w_{10}+x_{11}w_{11} & x_{01}w_{00}+x_{02}w_{01}+x_{11}w_{10}+x_{12}w_{11} \\\\\n", @@ -1314,8 +1493,10 @@ }, { "cell_type": "markdown", - "id": "067c8a61", - "metadata": {}, + "id": "f464698a", + "metadata": { + "editable": true + }, "source": [ "We can rewrite this operation in terms of a matrix-vector multiplication by defining a new vector where we flatten out the inputs as a vector $\\boldsymbol{X}'$ of length $9$ and\n", "a matrix $\\boldsymbol{W}'$ with dimension $4\\times 9$ as" @@ -1323,8 +1504,10 @@ }, { "cell_type": "markdown", - "id": "7eabf42d", - "metadata": {}, + "id": "b6dd9a91", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}'=\\begin{bmatrix}x_{00} \\\\ x_{01} \\\\ x_{02} \\\\ x_{10} \\\\ x_{11} \\\\ x_{12} \\\\ x_{20} \\\\ x_{21} \\\\ x_{22} \\end{bmatrix},\n", @@ -1333,16 +1516,20 @@ }, { "cell_type": "markdown", - "id": "a722d6ed", - "metadata": {}, + "id": "928c2b77", + "metadata": { + "editable": true + }, "source": [ "and the new matrix" ] }, { "cell_type": "markdown", - "id": "1719d3d2", - "metadata": {}, + "id": "9cea653c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{W}'=\\begin{bmatrix} w_{00} & w_{01} & 0 & w_{10} & w_{11} & 0 & 0 & 0 & 0 \\\\\n", @@ -1354,16 +1541,20 @@ }, { "cell_type": "markdown", - "id": "549a1cdd", - "metadata": {}, + "id": "56c72840", + "metadata": { + "editable": true + }, "source": [ "We see easily that performing the matrix-vector multiplication $\\boldsymbol{W}'\\boldsymbol{X}'$ is the same as the above convolution with stride $S=1$, that is" ] }, { "cell_type": "markdown", - "id": "e8ad9923", - "metadata": {}, + "id": "ffbf2c60", + "metadata": { + "editable": true + }, "source": [ "$$\n", "Y=(\\boldsymbol{W}*\\boldsymbol{X}),\n", @@ -1372,16 +1563,20 @@ }, { "cell_type": "markdown", - "id": "c5d4600d", - "metadata": {}, + "id": "34081f43", + "metadata": { + "editable": true + }, "source": [ "is now given by $\\boldsymbol{W}'\\boldsymbol{X}'$ which is a vector of length $4$ instead of the originally resulting $2\\times 2$ output matrix." ] }, { "cell_type": "markdown", - "id": "f10091e5", - "metadata": {}, + "id": "4a23919b", + "metadata": { + "editable": true + }, "source": [ "## The convolution stage\n", "\n", @@ -1394,8 +1589,10 @@ }, { "cell_type": "markdown", - "id": "8e54f0e7", - "metadata": {}, + "id": "d4f2e0d5", + "metadata": { + "editable": true + }, "source": [ "## Finding the number of parameters\n", "\n", @@ -1420,8 +1617,10 @@ }, { "cell_type": "markdown", - "id": "6822b435", - "metadata": {}, + "id": "2fe0bdbe", + "metadata": { + "editable": true + }, "source": [ "## New image (or volume)\n", "\n", @@ -1434,8 +1633,10 @@ }, { "cell_type": "markdown", - "id": "4e6fef6f", - "metadata": {}, + "id": "a82b0473", + "metadata": { + "editable": true + }, "source": [ "$$\n", "W_2 = \\frac{(W_1-F+2P)}{S}+1,\n", @@ -1444,8 +1645,10 @@ }, { "cell_type": "markdown", - "id": "e06aef00", - "metadata": {}, + "id": "dd64500e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "H_2 = \\frac{(H_1-F+2P)}{S}+1,\n", @@ -1454,16 +1657,20 @@ }, { "cell_type": "markdown", - "id": "1ab90aac", - "metadata": {}, + "id": "87016e5e", + "metadata": { + "editable": true + }, "source": [ "and $D_2=K$." ] }, { "cell_type": "markdown", - "id": "87ac7fd6", - "metadata": {}, + "id": "d7417ecf", + "metadata": { + "editable": true + }, "source": [ "## Parameters to train, common settings\n", "\n", @@ -1474,18 +1681,22 @@ }, { "cell_type": "markdown", - "id": "36b302f0", - "metadata": {}, + "id": "ae0220ff", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "\\left(F\\times F\\times D_1)\\right) \\times K+(K\\mathrm{--biases}),\n", + "\\left(F\\times F\\times D_1\\right) \\times K+K_{\\mathrm{biases}},\n", "$$" ] }, { "cell_type": "markdown", - "id": "81eb0abf", - "metadata": {}, + "id": "4e104aa8", + "metadata": { + "editable": true + }, "source": [ "parameters to train by back propagation.\n", "\n", @@ -1504,8 +1715,10 @@ }, { "cell_type": "markdown", - "id": "340ead69", - "metadata": {}, + "id": "e8647fdc", + "metadata": { + "editable": true + }, "source": [ "## Examples of CNN setups\n", "\n", @@ -1534,8 +1747,10 @@ }, { "cell_type": "markdown", - "id": "c09104a7", - "metadata": {}, + "id": "200fd8d5", + "metadata": { + "editable": true + }, "source": [ "## Summarizing: Performing a general discrete convolution ([From Raschka et al](https://github.com/rasbt/machine-learning-book))\n", "\n", @@ -1548,8 +1763,10 @@ }, { "cell_type": "markdown", - "id": "c3ee6c93", - "metadata": {}, + "id": "ab9cbc55", + "metadata": { + "editable": true + }, "source": [ "## Pooling\n", "\n", @@ -1566,8 +1783,10 @@ }, { "cell_type": "markdown", - "id": "33f5dc6c", - "metadata": {}, + "id": "200c198c", + "metadata": { + "editable": true + }, "source": [ "## Pooling arithmetic\n", "\n", @@ -1581,8 +1800,10 @@ }, { "cell_type": "markdown", - "id": "8a792d17", - "metadata": {}, + "id": "c6f9dfbd", + "metadata": { + "editable": true + }, "source": [ "## Pooling types ([From Raschka et al](https://github.com/rasbt/machine-learning-book))\n", "\n", @@ -1595,8 +1816,10 @@ }, { "cell_type": "markdown", - "id": "ccba7ad4", - "metadata": {}, + "id": "d8e9029f", + "metadata": { + "editable": true + }, "source": [ "## Building convolutional neural networks in Tensorflow and Keras\n", "\n", @@ -1612,8 +1835,10 @@ }, { "cell_type": "markdown", - "id": "13645c6d", - "metadata": {}, + "id": "b86c5d26", + "metadata": { + "editable": true + }, "source": [ "## Setting it up\n", "\n", @@ -1623,8 +1848,10 @@ }, { "cell_type": "markdown", - "id": "7dda54d3", - "metadata": {}, + "id": "d044bcf6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(n_{inputs},\\, n_{pixels, width},\\, n_{pixels, height},\\, depth) .\n", @@ -1633,8 +1860,10 @@ }, { "cell_type": "markdown", - "id": "1912abac", - "metadata": {}, + "id": "91b66cbf", + "metadata": { + "editable": true + }, "source": [ "## The MNIST dataset again\n", "\n", @@ -1652,8 +1881,10 @@ }, { "cell_type": "markdown", - "id": "92d33009", - "metadata": {}, + "id": "e78d269f", + "metadata": { + "editable": true + }, "source": [ "## Strong correlations\n", "\n", @@ -1671,8 +1902,10 @@ }, { "cell_type": "markdown", - "id": "afdc6174", - "metadata": {}, + "id": "45d416f0", + "metadata": { + "editable": true + }, "source": [ "## Layers of a CNN\n", "\n", @@ -1695,8 +1928,10 @@ }, { "cell_type": "markdown", - "id": "003f75b8", - "metadata": {}, + "id": "80264df6", + "metadata": { + "editable": true + }, "source": [ "## Systematic reduction\n", "\n", @@ -1712,8 +1947,10 @@ }, { "cell_type": "markdown", - "id": "6d323d3d", - "metadata": {}, + "id": "bb2120c9", + "metadata": { + "editable": true + }, "source": [ "## Prerequisites: Collect and pre-process data" ] @@ -1721,8 +1958,11 @@ { "cell_type": "code", "execution_count": 2, - "id": "f386213c", - "metadata": {}, + "id": "6fbfcb2f", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# import necessary packages\n", @@ -1769,8 +2009,10 @@ }, { "cell_type": "markdown", - "id": "7a38d39a", - "metadata": {}, + "id": "c7b8383b", + "metadata": { + "editable": true + }, "source": [ "## Importing Keras and Tensorflow" ] @@ -1778,8 +2020,11 @@ { "cell_type": "code", "execution_count": 3, - "id": "11ec9e2e", - "metadata": {}, + "id": "b8e59957", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from tensorflow.keras import datasets, layers, models\n", @@ -1808,8 +2053,10 @@ }, { "cell_type": "markdown", - "id": "d725fcbe", - "metadata": {}, + "id": "9270657f", + "metadata": { + "editable": true + }, "source": [ "## Running with Keras" ] @@ -1817,8 +2064,11 @@ { "cell_type": "code", "execution_count": 4, - "id": "8d265c27", - "metadata": {}, + "id": "55cb8e96", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def create_convolutional_neural_network_keras(input_shape, receptive_field,\n", @@ -1851,8 +2101,10 @@ }, { "cell_type": "markdown", - "id": "3f66294b", - "metadata": {}, + "id": "9a574946", + "metadata": { + "editable": true + }, "source": [ "## Final part" ] @@ -1860,8 +2112,11 @@ { "cell_type": "code", "execution_count": 5, - "id": "52313979", - "metadata": {}, + "id": "4ddd053f", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "CNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", @@ -1884,8 +2139,10 @@ }, { "cell_type": "markdown", - "id": "c0ae00f7", - "metadata": {}, + "id": "a0e2931c", + "metadata": { + "editable": true + }, "source": [ "## Final visualization" ] @@ -1893,8 +2150,11 @@ { "cell_type": "code", "execution_count": 6, - "id": "d28b398b", - "metadata": {}, + "id": "5fb4795c", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# visual representation of grid search\n", @@ -1931,8 +2191,10 @@ }, { "cell_type": "markdown", - "id": "957d569f", - "metadata": {}, + "id": "37ef01cf", + "metadata": { + "editable": true + }, "source": [ "## The CIFAR01 data set\n", "\n", @@ -1945,8 +2207,11 @@ { "cell_type": "code", "execution_count": 7, - "id": "875738c6", - "metadata": {}, + "id": "873a9388", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import tensorflow as tf\n", @@ -1963,8 +2228,10 @@ }, { "cell_type": "markdown", - "id": "3d02bafa", - "metadata": {}, + "id": "581a4299", + "metadata": { + "editable": true + }, "source": [ "## Verifying the data set\n", "\n", @@ -1974,8 +2241,11 @@ { "cell_type": "code", "execution_count": 8, - "id": "f6fa2efe", - "metadata": {}, + "id": "0390f74a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',\n", @@ -1995,8 +2265,10 @@ }, { "cell_type": "markdown", - "id": "7c5c4a79", - "metadata": {}, + "id": "f3067f2f", + "metadata": { + "editable": true + }, "source": [ "## Set up the model\n", "\n", @@ -2008,8 +2280,11 @@ { "cell_type": "code", "execution_count": 9, - "id": "0849002f", - "metadata": {}, + "id": "06da651a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "model = models.Sequential()\n", @@ -2026,16 +2301,20 @@ }, { "cell_type": "markdown", - "id": "42a62b52", - "metadata": {}, + "id": "9e8c82be", + "metadata": { + "editable": true + }, "source": [ "You can see that the output of every Conv2D and MaxPooling2D layer is a 3D tensor of shape (height, width, channels). The width and height dimensions tend to shrink as you go deeper in the network. The number of output channels for each Conv2D layer is controlled by the first argument (e.g., 32 or 64). Typically, as the width and height shrink, you can afford (computationally) to add more output channels in each Conv2D layer." ] }, { "cell_type": "markdown", - "id": "be17ccdf", - "metadata": {}, + "id": "c97fe381", + "metadata": { + "editable": true + }, "source": [ "## Add Dense layers on top\n", "\n", @@ -2051,8 +2330,11 @@ { "cell_type": "code", "execution_count": 10, - "id": "0a2841f0", - "metadata": {}, + "id": "8173bc50", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "model.add(layers.Flatten())\n", @@ -2065,16 +2347,20 @@ }, { "cell_type": "markdown", - "id": "f70f18a5", - "metadata": {}, + "id": "6125e588", + "metadata": { + "editable": true + }, "source": [ "As you can see, our (4, 4, 64) outputs were flattened into vectors of shape (1024) before going through two Dense layers." ] }, { "cell_type": "markdown", - "id": "8ab1a8ab", - "metadata": {}, + "id": "d7f88b84", + "metadata": { + "editable": true + }, "source": [ "## Compile and train the model" ] @@ -2082,8 +2368,11 @@ { "cell_type": "code", "execution_count": 11, - "id": "04d76fd5", - "metadata": {}, + "id": "abbbc4d6", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "model.compile(optimizer='adam',\n", @@ -2096,8 +2385,10 @@ }, { "cell_type": "markdown", - "id": "5c0392e2", - "metadata": {}, + "id": "9096d87b", + "metadata": { + "editable": true + }, "source": [ "## Finally, evaluate the model" ] @@ -2105,8 +2396,11 @@ { "cell_type": "code", "execution_count": 12, - "id": "2e869c20", - "metadata": {}, + "id": "23a9fb47", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "plt.plot(history.history['accuracy'], label='accuracy')\n", @@ -2123,8 +2417,10 @@ }, { "cell_type": "markdown", - "id": "66a5e54d", - "metadata": {}, + "id": "c25f501a", + "metadata": { + "editable": true + }, "source": [ "## Building our own CNN code\n", "\n", @@ -2154,8 +2450,10 @@ }, { "cell_type": "markdown", - "id": "af73a2bd", - "metadata": {}, + "id": "c526a678", + "metadata": { + "editable": true + }, "source": [ "### List of contents:\n", "\n", @@ -2176,8 +2474,10 @@ }, { "cell_type": "markdown", - "id": "9975f54f", - "metadata": {}, + "id": "4d2523f3", + "metadata": { + "editable": true + }, "source": [ "### Schedulers\n", "\n", @@ -2197,8 +2497,11 @@ { "cell_type": "code", "execution_count": 13, - "id": "660130ae", - "metadata": {}, + "id": "0c19073a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2335,8 +2638,10 @@ }, { "cell_type": "markdown", - "id": "9f8cf94f", - "metadata": {}, + "id": "04e5d590", + "metadata": { + "editable": true + }, "source": [ "### Usage of schedulers\n", "\n", @@ -2346,8 +2651,11 @@ { "cell_type": "code", "execution_count": 14, - "id": "273ba0b1", - "metadata": {}, + "id": "a3d38983", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n", @@ -2356,8 +2664,10 @@ }, { "cell_type": "markdown", - "id": "154e5891", - "metadata": {}, + "id": "eabe4856", + "metadata": { + "editable": true + }, "source": [ "Here is a small example for how a segment of code using schedulers could look. Switching out the schedulers is simple." ] @@ -2365,8 +2675,11 @@ { "cell_type": "code", "execution_count": 15, - "id": "8034d8f2", - "metadata": {}, + "id": "5f33282b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "weights = np.ones((3,3))\n", @@ -2384,8 +2697,10 @@ }, { "cell_type": "markdown", - "id": "f47f75c6", - "metadata": {}, + "id": "b7549596", + "metadata": { + "editable": true + }, "source": [ "### Cost functions\n", "\n", @@ -2398,8 +2713,11 @@ { "cell_type": "code", "execution_count": 16, - "id": "35db1625", - "metadata": {}, + "id": "e60c0f4e", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def CostOLS(target):\n", @@ -2442,8 +2760,10 @@ }, { "cell_type": "markdown", - "id": "446a3378", - "metadata": {}, + "id": "764c4963", + "metadata": { + "editable": true + }, "source": [ "### Usage of cost functions\n", "\n", @@ -2455,8 +2775,11 @@ { "cell_type": "code", "execution_count": 17, - "id": "2df9d962", - "metadata": {}, + "id": "ffd3244a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from autograd import grad\n", @@ -2473,8 +2796,10 @@ }, { "cell_type": "markdown", - "id": "bf534f8c", - "metadata": {}, + "id": "535daa3c", + "metadata": { + "editable": true + }, "source": [ "### Activation functions\n", "\n", @@ -2488,8 +2813,11 @@ { "cell_type": "code", "execution_count": 18, - "id": "a1faf180", - "metadata": {}, + "id": "8c660380", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\n", @@ -2544,8 +2872,10 @@ }, { "cell_type": "markdown", - "id": "ace037b0", - "metadata": {}, + "id": "d2e6619d", + "metadata": { + "editable": true + }, "source": [ "### Usage of activation functions\n", "\n", @@ -2559,8 +2889,11 @@ { "cell_type": "code", "execution_count": 19, - "id": "4ab94623", - "metadata": {}, + "id": "ea6a35d6", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "z = np.array([[4, 5, 6]]).T\n", @@ -2577,8 +2910,10 @@ }, { "cell_type": "markdown", - "id": "156dff4f", - "metadata": {}, + "id": "646e209b", + "metadata": { + "editable": true + }, "source": [ "### Convolution\n", "\n", @@ -2592,8 +2927,10 @@ }, { "cell_type": "markdown", - "id": "d4ec39fa", - "metadata": {}, + "id": "fedb7138", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(f \\ast g)(t):=\\int_{-\\infty}^{\\infty} f(\\tau) g(t-\\tau) d \\tau.\n", @@ -2602,8 +2939,10 @@ }, { "cell_type": "markdown", - "id": "f84a7f10", - "metadata": {}, + "id": "5b5ddeb4", + "metadata": { + "editable": true + }, "source": [ "Here, $f$ and $g$ are the two functions on which we want to perform an\n", "operation. The outcome of the convolution operation is represented by\n", @@ -2616,8 +2955,10 @@ }, { "cell_type": "markdown", - "id": "e4dec2b6", - "metadata": {}, + "id": "f50eb658", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(f \\ast g)[n]=\\sum_{m=0}^{n-1} f(m) g(n-m).\n", @@ -2626,8 +2967,10 @@ }, { "cell_type": "markdown", - "id": "2840d48f", - "metadata": {}, + "id": "d2ec25ae", + "metadata": { + "editable": true + }, "source": [ "The key idea we utilize to extract the information contained in an\n", "image is to slide an $m \\times n$ matrix $g$ over an $m \\times n$\n", @@ -2639,8 +2982,10 @@ }, { "cell_type": "markdown", - "id": "e8e58c03", - "metadata": {}, + "id": "21f4dd8f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(f \\ast g)(i, j)\\sum_{m=0}^{M-1}\\sum_{n=0}^{N-1} f(m,n) g(i-m, j-n).\n", @@ -2649,8 +2994,10 @@ }, { "cell_type": "markdown", - "id": "11ea612a", - "metadata": {}, + "id": "42be05f6", + "metadata": { + "editable": true + }, "source": [ "It is imperative to note that the size of the kernel g is\n", "significantly smaller than the size of the input image f, thereby\n", @@ -2665,8 +3012,10 @@ }, { "cell_type": "markdown", - "id": "836da562", - "metadata": {}, + "id": "37bf49e4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f = \\begin{bmatrix}\n", @@ -2682,8 +3031,10 @@ }, { "cell_type": "markdown", - "id": "2b3bb8f1", - "metadata": {}, + "id": "c5da49a2", + "metadata": { + "editable": true + }, "source": [ "and a $3 \\times 3$ kernel $g$ called a low-pass filter. Note that the\n", "kernel is usually rotated by 180 degrees during convolution, however\n", @@ -2692,8 +3043,10 @@ }, { "cell_type": "markdown", - "id": "3f7c208c", - "metadata": {}, + "id": "aa39df3b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "g = \\frac{1}{9}\n", @@ -2707,8 +3060,10 @@ }, { "cell_type": "markdown", - "id": "3d8b79f4", - "metadata": {}, + "id": "f42ea93d", + "metadata": { + "editable": true + }, "source": [ "In order to filter the image, we have to extract a $3 \\times 3$\n", "element from the upper left corner of $f$, and perform element-wise\n", @@ -2718,8 +3073,10 @@ }, { "cell_type": "markdown", - "id": "7e21a80b", - "metadata": {}, + "id": "f8ac79e2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{bmatrix}\n", @@ -2745,16 +3102,20 @@ }, { "cell_type": "markdown", - "id": "93aab016", - "metadata": {}, + "id": "8e18a9c1", + "metadata": { + "editable": true + }, "source": [ "Then, following the multiplication, we summarize all the elements of the resulting matrix $\\boldsymbol{A}$:" ] }, { "cell_type": "markdown", - "id": "4e0b557b", - "metadata": {}, + "id": "fccc7f61", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(f \\ast g)(0, 0)= \\sum_{i=0}^{2} \\sum_{j=0}^{2} a_{i,j} = 5,\n", @@ -2763,8 +3124,10 @@ }, { "cell_type": "markdown", - "id": "391389d2", - "metadata": {}, + "id": "c25c2cd3", + "metadata": { + "editable": true + }, "source": [ "which corresponds to the first element of the filtered image $(f \\ast g)$.\n", "\n", @@ -2782,8 +3145,10 @@ }, { "cell_type": "markdown", - "id": "2948c748", - "metadata": {}, + "id": "a9c9d3b4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(f \\ast g) =\n", @@ -2798,8 +3163,10 @@ }, { "cell_type": "markdown", - "id": "59ccebe4", - "metadata": {}, + "id": "1c299c32", + "metadata": { + "editable": true + }, "source": [ "The result is markedly smaller in shape than the original image. This\n", "occurs when using convolution without first padding the image with\n", @@ -2812,8 +3179,10 @@ }, { "cell_type": "markdown", - "id": "3ac3ba4b", - "metadata": {}, + "id": "710fbef3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "r =\\lfloor \\frac{\\mathrm{kernel height}}{2} \\rfloor \\cdot 2 \\\\\n", @@ -2823,8 +3192,10 @@ }, { "cell_type": "markdown", - "id": "60d7d729", - "metadata": {}, + "id": "96e1814c", + "metadata": { + "editable": true + }, "source": [ "Note the notation $\\lfloor \\frac{\\mathrm{kernel width}}{2} \\rfloor$ means that\n", "we floor the result of the division, meaning we round down to a whole\n", @@ -2845,8 +3216,10 @@ }, { "cell_type": "markdown", - "id": "0f223085", - "metadata": {}, + "id": "079e8928", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{bmatrix}\n", @@ -2864,8 +3237,10 @@ }, { "cell_type": "markdown", - "id": "d2a919ef", - "metadata": {}, + "id": "adfb6996", + "metadata": { + "editable": true + }, "source": [ "Below we have provided code that demonstrates padding and\n", "convolution. As you will see when we run the code, the size of the\n", @@ -2875,8 +3250,11 @@ { "cell_type": "code", "execution_count": 20, - "id": "98158090", - "metadata": {}, + "id": "281564b9", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -2954,8 +3332,10 @@ }, { "cell_type": "markdown", - "id": "e386b5f9", - "metadata": {}, + "id": "bab9972a", + "metadata": { + "editable": true + }, "source": [ "Fun fact: When filtering images, you will see that convolution\n", "involves rotating the kernel by 180 degrees. However, this is not the\n", @@ -2966,8 +3346,11 @@ { "cell_type": "code", "execution_count": 21, - "id": "b196835c", - "metadata": {}, + "id": "5640f2fd", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\n", @@ -2990,8 +3373,10 @@ }, { "cell_type": "markdown", - "id": "c598de0f", - "metadata": {}, + "id": "ce4eaaa6", + "metadata": { + "editable": true + }, "source": [ "As you can see, the resulting image is of the same size as the\n", "original image. To round of our demonstration of convolution, we will\n", @@ -3009,8 +3394,11 @@ { "cell_type": "code", "execution_count": 22, - "id": "1dc54fc6", - "metadata": {}, + "id": "29697bcf", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Now an example using a real image and first a gaussian low-pass filter and then a Sobel filter\n", @@ -3056,8 +3444,10 @@ }, { "cell_type": "markdown", - "id": "8fb0749c", - "metadata": {}, + "id": "7b489593", + "metadata": { + "editable": true + }, "source": [ "### Layers\n", "\n", @@ -3069,8 +3459,11 @@ { "cell_type": "code", "execution_count": 23, - "id": "fedc419c", - "metadata": {}, + "id": "861cf1b4", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import math\n", @@ -3107,8 +3500,10 @@ }, { "cell_type": "markdown", - "id": "c8370f0c", - "metadata": {}, + "id": "bcd8f4b2", + "metadata": { + "editable": true + }, "source": [ "### Convolution2DLayer: convolution in a hidden layer\n", "\n", @@ -3147,8 +3542,11 @@ { "cell_type": "code", "execution_count": 24, - "id": "d97d0055", - "metadata": {}, + "id": "88c07b19", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "class Convolution2DLayer(Layer):\n", @@ -3430,8 +3828,10 @@ }, { "cell_type": "markdown", - "id": "0dfbd0fc", - "metadata": {}, + "id": "63ef9aad", + "metadata": { + "editable": true + }, "source": [ "### Backpropagation in the convolutional layer\n", "\n", @@ -3451,8 +3851,10 @@ }, { "cell_type": "markdown", - "id": "739539f5", - "metadata": {}, + "id": "5eb4280a", + "metadata": { + "editable": true + }, "source": [ "### Demonstration\n", "\n", @@ -3462,8 +3864,11 @@ { "cell_type": "code", "execution_count": 25, - "id": "fb42cade", - "metadata": {}, + "id": "67e3c0dd", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -3507,8 +3912,10 @@ }, { "cell_type": "markdown", - "id": "2037fef5", - "metadata": {}, + "id": "0316f627", + "metadata": { + "editable": true + }, "source": [ "We cobserve that the result has half the pixels on each axis due to\n", "the fact that we've used a horizontal and vertical stride of 2. The\n", @@ -3527,8 +3934,10 @@ }, { "cell_type": "markdown", - "id": "ee0b7034", - "metadata": {}, + "id": "7c379ef3", + "metadata": { + "editable": true + }, "source": [ "### Pooling Layer\n", "\n", @@ -3547,8 +3956,11 @@ { "cell_type": "code", "execution_count": 26, - "id": "37fdedf3", - "metadata": {}, + "id": "02a9a4cf", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "class Pooling2DLayer(Layer):\n", @@ -3706,8 +4118,10 @@ }, { "cell_type": "markdown", - "id": "dbd0a7e6", - "metadata": {}, + "id": "845e3464", + "metadata": { + "editable": true + }, "source": [ "### Flattening Layer\n", "\n", @@ -3725,8 +4139,11 @@ { "cell_type": "code", "execution_count": 27, - "id": "3196697f", - "metadata": {}, + "id": "3b9abf2f", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "class FlattenLayer(Layer):\n", @@ -3785,8 +4202,10 @@ }, { "cell_type": "markdown", - "id": "7af01737", - "metadata": {}, + "id": "fb737a67", + "metadata": { + "editable": true + }, "source": [ "### Fully Connected Layers\n", "\n", @@ -3807,8 +4226,11 @@ { "cell_type": "code", "execution_count": 28, - "id": "652a07a7", - "metadata": {}, + "id": "74c92d5c", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "class FullyConnectedLayer(Layer):\n", @@ -4032,8 +4454,10 @@ }, { "cell_type": "markdown", - "id": "e7caf94b", - "metadata": {}, + "id": "2652f606", + "metadata": { + "editable": true + }, "source": [ "### Optimized Convolution2DLayer\n", "\n", @@ -4051,8 +4475,11 @@ { "cell_type": "code", "execution_count": 29, - "id": "fc2c6d43", - "metadata": {}, + "id": "29ad6f7c", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "class Convolution2DLayerOPT(Convolution2DLayer):\n", @@ -4352,8 +4779,10 @@ }, { "cell_type": "markdown", - "id": "20e3247d", - "metadata": {}, + "id": "395f9902", + "metadata": { + "editable": true + }, "source": [ "### The Convolutional Neural Network (CNN)\n", "\n", @@ -4363,8 +4792,11 @@ { "cell_type": "code", "execution_count": 30, - "id": "7841dc47", - "metadata": {}, + "id": "cdd943f8", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import math\n", @@ -4895,8 +5327,10 @@ }, { "cell_type": "markdown", - "id": "2cba848d", - "metadata": {}, + "id": "831ae54f", + "metadata": { + "editable": true + }, "source": [ "### Usage of CNN code\n", "\n", @@ -4912,8 +5346,11 @@ { "cell_type": "code", "execution_count": 31, - "id": "869f4ea1", - "metadata": {}, + "id": "6a97d78d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n", @@ -4922,8 +5359,10 @@ }, { "cell_type": "markdown", - "id": "b7d283b2", - "metadata": {}, + "id": "b2431975", + "metadata": { + "editable": true + }, "source": [ "Now that we have our CNN object, we can begin to add layers to it!\n", "Many of the add_layer functions have default values, for example\n", @@ -4936,8 +5375,11 @@ { "cell_type": "code", "execution_count": 32, - "id": "795b9ebb", - "metadata": {}, + "id": "3886f7ca", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "cnn.add_Convolution2DLayer(\n", @@ -4959,8 +5401,10 @@ }, { "cell_type": "markdown", - "id": "ba9a8be1", - "metadata": {}, + "id": "2cc2458f", + "metadata": { + "editable": true + }, "source": [ "Here we have created a CNN with the following architecture:\n", "\n", @@ -4981,8 +5425,11 @@ { "cell_type": "code", "execution_count": 33, - "id": "bb4c98fa", - "metadata": {}, + "id": "1a5923ed", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.datasets import fetch_openml\n", @@ -5013,8 +5460,10 @@ }, { "cell_type": "markdown", - "id": "97a9484f", - "metadata": {}, + "id": "9b0e2828", + "metadata": { + "editable": true + }, "source": [ "Now we may train our model. Note that we can utilize regularization in\n", "the CNN by using the lam (lambda) parameter in fit(), and utilize\n", @@ -5030,8 +5479,11 @@ { "cell_type": "code", "execution_count": 34, - "id": "539baa72", - "metadata": {}, + "id": "ba15b81d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "scores = cnn.fit(\n", @@ -5055,8 +5507,10 @@ }, { "cell_type": "markdown", - "id": "c7bb6ac0", - "metadata": {}, + "id": "133aa0a3", + "metadata": { + "editable": true + }, "source": [ "Considering we only trained the model for 100 epochs without any tuning of the hyperparameters, this result is pretty good.\n", "\n", @@ -5082,8 +5536,11 @@ { "cell_type": "code", "execution_count": 35, - "id": "93e706ae", - "metadata": {}, + "id": "5388d00d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n", @@ -5148,8 +5605,10 @@ }, { "cell_type": "markdown", - "id": "c72032ee", - "metadata": {}, + "id": "d90444ef", + "metadata": { + "editable": true + }, "source": [ "Here we see the use of asymmetrical 1D kernels such as the $7 \\times\n", "1$ kernel in the first convolutional layer, both max and average\n", @@ -5164,8 +5623,10 @@ }, { "cell_type": "markdown", - "id": "4ae81157", - "metadata": {}, + "id": "04853024", + "metadata": { + "editable": true + }, "source": [ "### Additional Remarks\n", "\n", @@ -5186,8 +5647,11 @@ { "cell_type": "code", "execution_count": 36, - "id": "4d2d2032", - "metadata": {}, + "id": "595540dd", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def convolve(image, kernel, stride=1):\n", @@ -5214,8 +5678,10 @@ }, { "cell_type": "markdown", - "id": "0b61f114", - "metadata": {}, + "id": "f8e2a91f", + "metadata": { + "editable": true + }, "source": [ "### Remarks on the speed\n", "\n", @@ -5242,8 +5708,10 @@ }, { "cell_type": "markdown", - "id": "953ca041", - "metadata": {}, + "id": "31a9149b", + "metadata": { + "editable": true + }, "source": [ "### Convolution using separable kernels" ] @@ -5251,8 +5719,11 @@ { "cell_type": "code", "execution_count": 37, - "id": "6f5e8589", - "metadata": {}, + "id": "6cfdb448", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def conv2DSep(image, kernel, coef, stride=1, pad=\"zero\"):\n", @@ -5298,8 +5769,10 @@ }, { "cell_type": "markdown", - "id": "ba379cac", - "metadata": {}, + "id": "7480a4ff", + "metadata": { + "editable": true + }, "source": [ "By taking advantage of the capabilities of separable kernels, we can\n", "effectively cut the computational expense of filtering an image in\n", @@ -5317,8 +5790,10 @@ }, { "cell_type": "markdown", - "id": "951e4f08", - "metadata": {}, + "id": "c702ba45", + "metadata": { + "editable": true + }, "source": [ "### Convolution in the Fourier domain" ] @@ -5326,8 +5801,11 @@ { "cell_type": "code", "execution_count": 38, - "id": "b23c5570", - "metadata": {}, + "id": "0f16fec2", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "start_time = time.time()\n", @@ -5344,8 +5822,10 @@ }, { "cell_type": "markdown", - "id": "ecbe9144", - "metadata": {}, + "id": "80344fd3", + "metadata": { + "editable": true + }, "source": [ "It is evident that executing convolution in the Fourier domain yields\n", "the quickest computation time. Nonetheless, one should exercise\n", @@ -5360,25 +5840,7 @@ ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.15" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week44/week44.do.txt b/doc/src/week44/week44.do.txt index 38fd6dac3..e55978a66 100644 --- a/doc/src/week44/week44.do.txt +++ b/doc/src/week44/week44.do.txt @@ -9,12 +9,12 @@ DATE: October 28-November 1 !bblock Material for the lecture Monday October 28, 2024 o Convolutional Neural Networks o Readings and Videos: - * These lecture notes + * These lecture notes at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/pub/week44/ipynb/week44.ipynb" * For a more in depth discussion on neural networks we recommend Goodfellow et al chapter 9. See also chapter 11 and 12 on practicalities and applications - * Reading suggestions for implementation of CNNs: "Rashcka et al.'s chapter 14":"https://github.com/rasbt/machine-learning-book/tree/main/ch14". T - * "Video on Deep Learning":"https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi" - * "Video on Convolutional Neural Networks from MIT":"https://www.youtube.com/watch?v=iaSUYvmCekI&ab_channel=AlexanderAmini" - * "Video on CNNs from Stanford":"https://www.youtube.com/watch?v=bNb2fEVKeEo&list=PLC1qU-LWwrF64f4QKQT-Vg5Wr4qEE1Zxk&index=6&ab_channel=StanfordUniversitySchoolofEngineering" + * Reading suggestions for implementation of CNNs see URL:"Rashcka et al.'s chapter 14":"https://github.com/rasbt/machine-learning-book/tree/main/ch14". + * Video on Deep Learning at URL:"https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi" + * Video on Convolutional Neural Networks from MIT at URL:"https://www.youtube.com/watch?v=iaSUYvmCekI&ab_channel=AlexanderAmini" + * Video on CNNs from Stanford at URL:"https://www.youtube.com/watch?v=bNb2fEVKeEo&list=PLC1qU-LWwrF64f4QKQT-Vg5Wr4qEE1Zxk&index=6&ab_channel=StanfordUniversitySchoolofEngineering" !eblock @@ -873,7 +873,7 @@ With parameter sharing, the convolution involves thus for each filter $F\times In total we have !bt \[ -\left(F\times F\times D_1)\right) \times K+(K\mathrm{--biases}), +\left(F\times F\times D_1\right) \times K+K_{\mathrm{biases}}, \] !et parameters to train by back propagation.