From 6aa49a742854f3a313ea5aaf50f32c49b7e9acb9 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Mon, 14 Oct 2024 06:09:08 +0200 Subject: [PATCH] additional change --- doc/LectureNotes/week42.ipynb | 1246 ++-- doc/pub/week42/html/._week42-bs016.html | 8 +- doc/pub/week42/html/week42-reveal.html | 8 +- doc/pub/week42/html/week42-solarized.html | 8 +- doc/pub/week42/html/week42.html | 8 +- doc/pub/week42/ipynb/ipynb-week42-src.tar.gz | Bin 487084 -> 487084 bytes doc/pub/week42/ipynb/week42.ipynb | 646 +- doc/src/week42/._week42-bs000.html | 529 -- doc/src/week42/._week42-bs001.html | 530 -- doc/src/week42/._week42-bs002.html | 517 -- doc/src/week42/._week42-bs003.html | 517 -- doc/src/week42/._week42-bs004.html | 518 -- doc/src/week42/._week42-bs005.html | 517 -- doc/src/week42/._week42-bs006.html | 513 -- doc/src/week42/._week42-bs007.html | 532 -- doc/src/week42/._week42-bs008.html | 517 -- doc/src/week42/._week42-bs009.html | 538 -- doc/src/week42/._week42-bs010.html | 535 -- doc/src/week42/._week42-bs011.html | 519 -- doc/src/week42/._week42-bs012.html | 533 -- doc/src/week42/._week42-bs013.html | 526 -- doc/src/week42/._week42-bs014.html | 530 -- doc/src/week42/._week42-bs015.html | 607 -- doc/src/week42/._week42-bs016.html | 531 -- doc/src/week42/._week42-bs017.html | 519 -- doc/src/week42/._week42-bs018.html | 526 -- doc/src/week42/._week42-bs019.html | 526 -- doc/src/week42/._week42-bs020.html | 524 -- doc/src/week42/._week42-bs021.html | 535 -- doc/src/week42/._week42-bs022.html | 530 -- doc/src/week42/._week42-bs023.html | 528 -- doc/src/week42/._week42-bs024.html | 529 -- doc/src/week42/._week42-bs025.html | 525 -- doc/src/week42/._week42-bs026.html | 537 -- doc/src/week42/._week42-bs027.html | 531 -- doc/src/week42/._week42-bs028.html | 519 -- doc/src/week42/._week42-bs029.html | 536 -- doc/src/week42/._week42-bs030.html | 525 -- doc/src/week42/._week42-bs031.html | 519 -- doc/src/week42/._week42-bs032.html | 529 -- doc/src/week42/._week42-bs033.html | 532 -- doc/src/week42/._week42-bs034.html | 529 -- doc/src/week42/._week42-bs035.html | 522 -- doc/src/week42/._week42-bs036.html | 532 -- doc/src/week42/._week42-bs037.html | 526 -- doc/src/week42/._week42-bs038.html | 540 -- doc/src/week42/._week42-bs039.html | 520 -- doc/src/week42/._week42-bs040.html | 532 -- doc/src/week42/._week42-bs041.html | 529 -- doc/src/week42/._week42-bs042.html | 528 -- doc/src/week42/._week42-bs043.html | 524 -- doc/src/week42/._week42-bs044.html | 529 -- doc/src/week42/._week42-bs045.html | 529 -- doc/src/week42/._week42-bs046.html | 525 -- doc/src/week42/._week42-bs047.html | 534 -- doc/src/week42/._week42-bs048.html | 614 -- doc/src/week42/._week42-bs049.html | 527 -- doc/src/week42/._week42-bs050.html | 522 -- doc/src/week42/._week42-bs051.html | 528 -- doc/src/week42/._week42-bs052.html | 523 -- doc/src/week42/._week42-bs053.html | 533 -- doc/src/week42/._week42-bs054.html | 524 -- doc/src/week42/._week42-bs055.html | 526 -- doc/src/week42/._week42-bs056.html | 523 -- doc/src/week42/._week42-bs057.html | 527 -- doc/src/week42/._week42-bs058.html | 527 -- doc/src/week42/._week42-bs059.html | 535 -- doc/src/week42/._week42-bs060.html | 531 -- doc/src/week42/._week42-bs061.html | 531 -- doc/src/week42/._week42-bs062.html | 525 -- doc/src/week42/._week42-bs063.html | 525 -- doc/src/week42/._week42-bs064.html | 529 -- doc/src/week42/._week42-bs065.html | 534 -- doc/src/week42/._week42-bs066.html | 522 -- doc/src/week42/._week42-bs067.html | 519 -- doc/src/week42/._week42-bs068.html | 516 -- doc/src/week42/._week42-bs069.html | 518 -- doc/src/week42/._week42-bs070.html | 547 -- doc/src/week42/._week42-bs071.html | 562 -- doc/src/week42/._week42-bs072.html | 556 -- doc/src/week42/._week42-bs073.html | 530 -- doc/src/week42/._week42-bs074.html | 523 -- doc/src/week42/._week42-bs075.html | 622 -- doc/src/week42/._week42-bs076.html | 574 -- doc/src/week42/._week42-bs077.html | 552 -- doc/src/week42/._week42-bs078.html | 551 -- doc/src/week42/._week42-bs079.html | 562 -- doc/src/week42/._week42-bs080.html | 533 -- doc/src/week42/._week42-bs081.html | 600 -- doc/src/week42/._week42-bs082.html | 538 -- doc/src/week42/._week42-bs083.html | 545 -- doc/src/week42/._week42-bs084.html | 540 -- doc/src/week42/._week42-bs085.html | 637 -- doc/src/week42/._week42-bs086.html | 527 -- doc/src/week42/._week42-bs087.html | 639 -- doc/src/week42/._week42-bs088.html | 561 -- doc/src/week42/._week42-bs089.html | 560 -- doc/src/week42/._week42-bs090.html | 567 -- doc/src/week42/._week42-bs091.html | 563 -- doc/src/week42/._week42-bs092.html | 565 -- doc/src/week42/._week42-bs093.html | 518 -- doc/src/week42/._week42-bs094.html | 610 -- doc/src/week42/._week42-bs095.html | 537 -- doc/src/week42/._week42-bs096.html | 729 -- doc/src/week42/._week42-bs097.html | 692 -- doc/src/week42/._week42-bs098.html | 1809 ----- doc/src/week42/._week42-bs099.html | 538 -- doc/src/week42/ipynb-week42-src.tar.gz | Bin 487084 -> 0 bytes doc/src/week42/reveal.js/.gitignore | 8 - doc/src/week42/reveal.js/.travis.yml | 5 - doc/src/week42/reveal.js/CONTRIBUTING.md | 23 - doc/src/week42/reveal.js/Gruntfile.js | 140 - doc/src/week42/reveal.js/LICENSE | 19 - doc/src/week42/reveal.js/README.md | 1052 --- doc/src/week42/reveal.js/bower.json | 27 - .../reveal.js/css/images/cbc_footer.png | Bin 10008 -> 0 bytes .../reveal.js/css/images/cbc_symbol.png | Bin 2946 -> 0 bytes .../reveal.js/css/images/simula_footer.png | Bin 2513 -> 0 bytes .../reveal.js/css/images/simula_logo.png | Bin 2138 -> 0 bytes .../reveal.js/css/images/simula_symbol.png | Bin 2138 -> 0 bytes .../reveal.js/css/images/uio_footer.png | Bin 18189 -> 0 bytes .../reveal.js/css/images/uio_symbol.png | Bin 11352 -> 0 bytes doc/src/week42/reveal.js/css/print/paper.css | 202 - doc/src/week42/reveal.js/css/print/pdf.css | 157 - doc/src/week42/reveal.js/css/reveal.css | 1886 ----- doc/src/week42/reveal.js/css/reveal.scss | 1319 ---- doc/src/week42/reveal.js/css/theme/README.md | 23 - doc/src/week42/reveal.js/css/theme/beige.css | 154 - .../week42/reveal.js/css/theme/beigesmall.css | 155 - doc/src/week42/reveal.js/css/theme/black.css | 273 - doc/src/week42/reveal.js/css/theme/blood.css | 180 - doc/src/week42/reveal.js/css/theme/cbc.css | 144 - .../week42/reveal.js/css/theme/darkgray.css | 153 - .../week42/reveal.js/css/theme/default.css | 153 - doc/src/week42/reveal.js/css/theme/league.css | 279 - doc/src/week42/reveal.js/css/theme/moon.css | 153 - doc/src/week42/reveal.js/css/theme/night.css | 141 - doc/src/week42/reveal.js/css/theme/serif.css | 143 - doc/src/week42/reveal.js/css/theme/simple.css | 144 - doc/src/week42/reveal.js/css/theme/simula.css | 144 - doc/src/week42/reveal.js/css/theme/sky.css | 150 - .../week42/reveal.js/css/theme/solarized.css | 153 - .../reveal.js/css/theme/source/beige.scss | 50 - .../css/theme/source/beigesmall.scss | 51 - .../reveal.js/css/theme/source/black.scss | 49 - .../reveal.js/css/theme/source/blood.scss | 91 - .../reveal.js/css/theme/source/cbc.scss | 39 - .../reveal.js/css/theme/source/darkgray.scss | 42 - .../reveal.js/css/theme/source/default.scss | 42 - .../reveal.js/css/theme/source/league.scss | 34 - .../reveal.js/css/theme/source/moon.scss | 68 - .../reveal.js/css/theme/source/night.scss | 35 - .../reveal.js/css/theme/source/serif.scss | 35 - .../reveal.js/css/theme/source/simple.scss | 38 - .../reveal.js/css/theme/source/simula.scss | 39 - .../reveal.js/css/theme/source/sky.scss | 46 - .../reveal.js/css/theme/source/solarized.scss | 74 - .../reveal.js/css/theme/source/white.scss | 49 - doc/src/week42/reveal.js/css/theme/white.css | 273 - doc/src/week42/reveal.js/index.html | 411 -- doc/src/week42/reveal.js/js/reveal.js | 4508 ------------ doc/src/week42/reveal.js/lib/css/zenburn.css | 117 - .../reveal.js/lib/font/league-gothic/LICENSE | 2 - .../lib/font/league-gothic/league-gothic.css | 10 - .../lib/font/league-gothic/league-gothic.eot | Bin 25696 -> 0 bytes .../lib/font/league-gothic/league-gothic.ttf | Bin 64256 -> 0 bytes .../lib/font/league-gothic/league-gothic.woff | Bin 30764 -> 0 bytes .../lib/font/source-sans-pro/LICENSE | 45 - .../source-sans-pro-italic.eot | Bin 75720 -> 0 bytes .../source-sans-pro-italic.ttf | Bin 238084 -> 0 bytes .../source-sans-pro-italic.woff | Bin 98556 -> 0 bytes .../source-sans-pro-regular.eot | Bin 88070 -> 0 bytes .../source-sans-pro-regular.ttf | Bin 288008 -> 0 bytes .../source-sans-pro-regular.woff | Bin 114324 -> 0 bytes .../source-sans-pro-semibold.eot | Bin 89897 -> 0 bytes .../source-sans-pro-semibold.ttf | Bin 284640 -> 0 bytes .../source-sans-pro-semibold.woff | Bin 115648 -> 0 bytes .../source-sans-pro-semibolditalic.eot | Bin 75706 -> 0 bytes .../source-sans-pro-semibolditalic.ttf | Bin 240944 -> 0 bytes .../source-sans-pro-semibolditalic.woff | Bin 98816 -> 0 bytes .../font/source-sans-pro/source-sans-pro.css | 39 - doc/src/week42/reveal.js/lib/js/classList.js | 2 - doc/src/week42/reveal.js/lib/js/head.min.js | 8 - doc/src/week42/reveal.js/lib/js/html5shiv.js | 7 - doc/src/week42/reveal.js/package.json | 45 - .../reveal.js/plugin/highlight/highlight.js | 30 - doc/src/week42/reveal.js/plugin/leap/leap.js | 159 - .../reveal.js/plugin/markdown/example.html | 129 - .../reveal.js/plugin/markdown/example.md | 31 - .../reveal.js/plugin/markdown/markdown.js | 393 -- .../reveal.js/plugin/markdown/marked.js | 6 - doc/src/week42/reveal.js/plugin/math/math.js | 64 - .../reveal.js/plugin/multiplex/client.js | 13 - .../reveal.js/plugin/multiplex/index.js | 56 - .../reveal.js/plugin/multiplex/master.js | 51 - .../reveal.js/plugin/notes-server/client.js | 60 - .../reveal.js/plugin/notes-server/index.js | 66 - .../reveal.js/plugin/notes-server/notes.html | 396 -- .../week42/reveal.js/plugin/notes/notes.html | 406 -- .../week42/reveal.js/plugin/notes/notes.js | 122 - .../reveal.js/plugin/print-pdf/print-pdf.js | 48 - .../reveal.js/plugin/remotes/remotes.js | 39 - .../week42/reveal.js/plugin/search/search.js | 196 - .../week42/reveal.js/plugin/zoom-js/zoom.js | 278 - .../reveal.js/test/examples/assets/image1.png | Bin 21991 -> 0 bytes .../reveal.js/test/examples/assets/image2.png | Bin 10237 -> 0 bytes .../reveal.js/test/examples/barebones.html | 41 - .../test/examples/embedded-media.html | 49 - .../week42/reveal.js/test/examples/math.html | 185 - .../test/examples/slide-backgrounds.html | 144 - .../test/examples/slide-transitions.html | 101 - .../week42/reveal.js/test/qunit-1.12.0.css | 244 - doc/src/week42/reveal.js/test/qunit-1.12.0.js | 2212 ------ .../test-markdown-element-attributes.html | 134 - .../test/test-markdown-element-attributes.js | 46 - .../test/test-markdown-slide-attributes.html | 128 - .../test/test-markdown-slide-attributes.js | 47 - .../week42/reveal.js/test/test-markdown.html | 52 - .../week42/reveal.js/test/test-markdown.js | 15 - doc/src/week42/reveal.js/test/test-pdf.html | 83 - doc/src/week42/reveal.js/test/test-pdf.js | 15 - doc/src/week42/reveal.js/test/test.html | 85 - doc/src/week42/reveal.js/test/test.js | 589 -- doc/src/week42/week42-bs.html | 529 -- doc/src/week42/week42-reveal.html | 5174 -------------- doc/src/week42/week42-solarized.html | 4901 ------------- doc/src/week42/week42.do.txt | 8 +- doc/src/week42/week42.html | 4978 ------------- doc/src/week42/week42.ipynb | 6133 ----------------- 229 files changed, 928 insertions(+), 98485 deletions(-) delete mode 100644 doc/src/week42/._week42-bs000.html delete mode 100644 doc/src/week42/._week42-bs001.html delete mode 100644 doc/src/week42/._week42-bs002.html delete mode 100644 doc/src/week42/._week42-bs003.html delete mode 100644 doc/src/week42/._week42-bs004.html delete mode 100644 doc/src/week42/._week42-bs005.html delete mode 100644 doc/src/week42/._week42-bs006.html delete mode 100644 doc/src/week42/._week42-bs007.html delete mode 100644 doc/src/week42/._week42-bs008.html delete mode 100644 doc/src/week42/._week42-bs009.html delete mode 100644 doc/src/week42/._week42-bs010.html delete mode 100644 doc/src/week42/._week42-bs011.html delete mode 100644 doc/src/week42/._week42-bs012.html delete mode 100644 doc/src/week42/._week42-bs013.html delete mode 100644 doc/src/week42/._week42-bs014.html delete mode 100644 doc/src/week42/._week42-bs015.html delete mode 100644 doc/src/week42/._week42-bs016.html delete mode 100644 doc/src/week42/._week42-bs017.html delete mode 100644 doc/src/week42/._week42-bs018.html delete mode 100644 doc/src/week42/._week42-bs019.html delete mode 100644 doc/src/week42/._week42-bs020.html delete mode 100644 doc/src/week42/._week42-bs021.html delete mode 100644 doc/src/week42/._week42-bs022.html delete mode 100644 doc/src/week42/._week42-bs023.html delete mode 100644 doc/src/week42/._week42-bs024.html delete mode 100644 doc/src/week42/._week42-bs025.html delete mode 100644 doc/src/week42/._week42-bs026.html delete mode 100644 doc/src/week42/._week42-bs027.html delete mode 100644 doc/src/week42/._week42-bs028.html delete mode 100644 doc/src/week42/._week42-bs029.html delete mode 100644 doc/src/week42/._week42-bs030.html delete mode 100644 doc/src/week42/._week42-bs031.html delete mode 100644 doc/src/week42/._week42-bs032.html delete mode 100644 doc/src/week42/._week42-bs033.html delete mode 100644 doc/src/week42/._week42-bs034.html delete mode 100644 doc/src/week42/._week42-bs035.html delete mode 100644 doc/src/week42/._week42-bs036.html delete mode 100644 doc/src/week42/._week42-bs037.html delete mode 100644 doc/src/week42/._week42-bs038.html delete mode 100644 doc/src/week42/._week42-bs039.html delete mode 100644 doc/src/week42/._week42-bs040.html delete mode 100644 doc/src/week42/._week42-bs041.html delete mode 100644 doc/src/week42/._week42-bs042.html delete mode 100644 doc/src/week42/._week42-bs043.html delete mode 100644 doc/src/week42/._week42-bs044.html delete mode 100644 doc/src/week42/._week42-bs045.html delete mode 100644 doc/src/week42/._week42-bs046.html delete mode 100644 doc/src/week42/._week42-bs047.html delete mode 100644 doc/src/week42/._week42-bs048.html delete mode 100644 doc/src/week42/._week42-bs049.html delete mode 100644 doc/src/week42/._week42-bs050.html delete mode 100644 doc/src/week42/._week42-bs051.html delete mode 100644 doc/src/week42/._week42-bs052.html delete mode 100644 doc/src/week42/._week42-bs053.html delete mode 100644 doc/src/week42/._week42-bs054.html delete mode 100644 doc/src/week42/._week42-bs055.html delete mode 100644 doc/src/week42/._week42-bs056.html delete mode 100644 doc/src/week42/._week42-bs057.html delete mode 100644 doc/src/week42/._week42-bs058.html delete mode 100644 doc/src/week42/._week42-bs059.html delete mode 100644 doc/src/week42/._week42-bs060.html delete mode 100644 doc/src/week42/._week42-bs061.html delete mode 100644 doc/src/week42/._week42-bs062.html delete mode 100644 doc/src/week42/._week42-bs063.html delete mode 100644 doc/src/week42/._week42-bs064.html delete mode 100644 doc/src/week42/._week42-bs065.html delete mode 100644 doc/src/week42/._week42-bs066.html delete mode 100644 doc/src/week42/._week42-bs067.html delete mode 100644 doc/src/week42/._week42-bs068.html delete mode 100644 doc/src/week42/._week42-bs069.html delete mode 100644 doc/src/week42/._week42-bs070.html delete mode 100644 doc/src/week42/._week42-bs071.html delete mode 100644 doc/src/week42/._week42-bs072.html delete mode 100644 doc/src/week42/._week42-bs073.html delete mode 100644 doc/src/week42/._week42-bs074.html delete mode 100644 doc/src/week42/._week42-bs075.html delete mode 100644 doc/src/week42/._week42-bs076.html delete mode 100644 doc/src/week42/._week42-bs077.html delete mode 100644 doc/src/week42/._week42-bs078.html delete mode 100644 doc/src/week42/._week42-bs079.html delete mode 100644 doc/src/week42/._week42-bs080.html delete mode 100644 doc/src/week42/._week42-bs081.html delete mode 100644 doc/src/week42/._week42-bs082.html delete mode 100644 doc/src/week42/._week42-bs083.html delete mode 100644 doc/src/week42/._week42-bs084.html delete mode 100644 doc/src/week42/._week42-bs085.html delete mode 100644 doc/src/week42/._week42-bs086.html delete mode 100644 doc/src/week42/._week42-bs087.html delete mode 100644 doc/src/week42/._week42-bs088.html delete mode 100644 doc/src/week42/._week42-bs089.html delete mode 100644 doc/src/week42/._week42-bs090.html delete mode 100644 doc/src/week42/._week42-bs091.html delete mode 100644 doc/src/week42/._week42-bs092.html delete mode 100644 doc/src/week42/._week42-bs093.html delete mode 100644 doc/src/week42/._week42-bs094.html delete mode 100644 doc/src/week42/._week42-bs095.html delete mode 100644 doc/src/week42/._week42-bs096.html delete mode 100644 doc/src/week42/._week42-bs097.html delete mode 100644 doc/src/week42/._week42-bs098.html delete mode 100644 doc/src/week42/._week42-bs099.html delete mode 100644 doc/src/week42/ipynb-week42-src.tar.gz delete mode 100644 doc/src/week42/reveal.js/.gitignore delete mode 100644 doc/src/week42/reveal.js/.travis.yml delete mode 100644 doc/src/week42/reveal.js/CONTRIBUTING.md delete mode 100644 doc/src/week42/reveal.js/Gruntfile.js delete mode 100644 doc/src/week42/reveal.js/LICENSE delete mode 100644 doc/src/week42/reveal.js/README.md delete mode 100644 doc/src/week42/reveal.js/bower.json delete mode 100644 doc/src/week42/reveal.js/css/images/cbc_footer.png delete mode 100644 doc/src/week42/reveal.js/css/images/cbc_symbol.png delete mode 100644 doc/src/week42/reveal.js/css/images/simula_footer.png delete mode 100644 doc/src/week42/reveal.js/css/images/simula_logo.png delete mode 100644 doc/src/week42/reveal.js/css/images/simula_symbol.png delete mode 100644 doc/src/week42/reveal.js/css/images/uio_footer.png delete mode 100644 doc/src/week42/reveal.js/css/images/uio_symbol.png delete mode 100644 doc/src/week42/reveal.js/css/print/paper.css delete mode 100644 doc/src/week42/reveal.js/css/print/pdf.css delete mode 100644 doc/src/week42/reveal.js/css/reveal.css delete mode 100644 doc/src/week42/reveal.js/css/reveal.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/README.md delete mode 100644 doc/src/week42/reveal.js/css/theme/beige.css delete mode 100644 doc/src/week42/reveal.js/css/theme/beigesmall.css delete mode 100644 doc/src/week42/reveal.js/css/theme/black.css delete mode 100644 doc/src/week42/reveal.js/css/theme/blood.css delete mode 100644 doc/src/week42/reveal.js/css/theme/cbc.css delete mode 100644 doc/src/week42/reveal.js/css/theme/darkgray.css delete mode 100644 doc/src/week42/reveal.js/css/theme/default.css delete mode 100644 doc/src/week42/reveal.js/css/theme/league.css delete mode 100644 doc/src/week42/reveal.js/css/theme/moon.css delete mode 100644 doc/src/week42/reveal.js/css/theme/night.css delete mode 100644 doc/src/week42/reveal.js/css/theme/serif.css delete mode 100644 doc/src/week42/reveal.js/css/theme/simple.css delete mode 100644 doc/src/week42/reveal.js/css/theme/simula.css delete mode 100644 doc/src/week42/reveal.js/css/theme/sky.css delete mode 100644 doc/src/week42/reveal.js/css/theme/solarized.css delete mode 100644 doc/src/week42/reveal.js/css/theme/source/beige.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/beigesmall.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/black.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/blood.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/cbc.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/darkgray.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/default.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/league.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/moon.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/night.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/serif.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/simple.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/simula.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/sky.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/solarized.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/source/white.scss delete mode 100644 doc/src/week42/reveal.js/css/theme/white.css delete mode 100644 doc/src/week42/reveal.js/index.html delete mode 100644 doc/src/week42/reveal.js/js/reveal.js delete mode 100644 doc/src/week42/reveal.js/lib/css/zenburn.css delete mode 100644 doc/src/week42/reveal.js/lib/font/league-gothic/LICENSE delete mode 100644 doc/src/week42/reveal.js/lib/font/league-gothic/league-gothic.css delete mode 100644 doc/src/week42/reveal.js/lib/font/league-gothic/league-gothic.eot delete mode 100644 doc/src/week42/reveal.js/lib/font/league-gothic/league-gothic.ttf delete mode 100644 doc/src/week42/reveal.js/lib/font/league-gothic/league-gothic.woff delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/LICENSE delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro-italic.eot delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro-italic.ttf delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro-italic.woff delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro-regular.eot delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro-regular.ttf delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro-regular.woff delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro-semibold.eot delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro-semibold.ttf delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro-semibold.woff delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro-semibolditalic.eot delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro-semibolditalic.ttf delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro-semibolditalic.woff delete mode 100644 doc/src/week42/reveal.js/lib/font/source-sans-pro/source-sans-pro.css delete mode 100644 doc/src/week42/reveal.js/lib/js/classList.js delete mode 100644 doc/src/week42/reveal.js/lib/js/head.min.js delete mode 100644 doc/src/week42/reveal.js/lib/js/html5shiv.js delete mode 100644 doc/src/week42/reveal.js/package.json delete mode 100644 doc/src/week42/reveal.js/plugin/highlight/highlight.js delete mode 100644 doc/src/week42/reveal.js/plugin/leap/leap.js delete mode 100644 doc/src/week42/reveal.js/plugin/markdown/example.html delete mode 100644 doc/src/week42/reveal.js/plugin/markdown/example.md delete mode 100644 doc/src/week42/reveal.js/plugin/markdown/markdown.js delete mode 100644 doc/src/week42/reveal.js/plugin/markdown/marked.js delete mode 100644 doc/src/week42/reveal.js/plugin/math/math.js delete mode 100644 doc/src/week42/reveal.js/plugin/multiplex/client.js delete mode 100644 doc/src/week42/reveal.js/plugin/multiplex/index.js delete mode 100644 doc/src/week42/reveal.js/plugin/multiplex/master.js delete mode 100644 doc/src/week42/reveal.js/plugin/notes-server/client.js delete mode 100644 doc/src/week42/reveal.js/plugin/notes-server/index.js delete mode 100644 doc/src/week42/reveal.js/plugin/notes-server/notes.html delete mode 100644 doc/src/week42/reveal.js/plugin/notes/notes.html delete mode 100644 doc/src/week42/reveal.js/plugin/notes/notes.js delete mode 100644 doc/src/week42/reveal.js/plugin/print-pdf/print-pdf.js delete mode 100644 doc/src/week42/reveal.js/plugin/remotes/remotes.js delete mode 100644 doc/src/week42/reveal.js/plugin/search/search.js delete mode 100644 doc/src/week42/reveal.js/plugin/zoom-js/zoom.js delete mode 100644 doc/src/week42/reveal.js/test/examples/assets/image1.png delete mode 100644 doc/src/week42/reveal.js/test/examples/assets/image2.png delete mode 100644 doc/src/week42/reveal.js/test/examples/barebones.html delete mode 100644 doc/src/week42/reveal.js/test/examples/embedded-media.html delete mode 100644 doc/src/week42/reveal.js/test/examples/math.html delete mode 100644 doc/src/week42/reveal.js/test/examples/slide-backgrounds.html delete mode 100644 doc/src/week42/reveal.js/test/examples/slide-transitions.html delete mode 100644 doc/src/week42/reveal.js/test/qunit-1.12.0.css delete mode 100644 doc/src/week42/reveal.js/test/qunit-1.12.0.js delete mode 100644 doc/src/week42/reveal.js/test/test-markdown-element-attributes.html delete mode 100644 doc/src/week42/reveal.js/test/test-markdown-element-attributes.js delete mode 100644 doc/src/week42/reveal.js/test/test-markdown-slide-attributes.html delete mode 100644 doc/src/week42/reveal.js/test/test-markdown-slide-attributes.js delete mode 100644 doc/src/week42/reveal.js/test/test-markdown.html delete mode 100644 doc/src/week42/reveal.js/test/test-markdown.js delete mode 100644 doc/src/week42/reveal.js/test/test-pdf.html delete mode 100644 doc/src/week42/reveal.js/test/test-pdf.js delete mode 100644 doc/src/week42/reveal.js/test/test.html delete mode 100644 doc/src/week42/reveal.js/test/test.js delete mode 100644 doc/src/week42/week42-bs.html delete mode 100644 doc/src/week42/week42-reveal.html delete mode 100644 doc/src/week42/week42-solarized.html delete mode 100644 doc/src/week42/week42.html delete mode 100644 doc/src/week42/week42.ipynb diff --git a/doc/LectureNotes/week42.ipynb b/doc/LectureNotes/week42.ipynb index 8699e7c79..fdfeb303f 100644 --- a/doc/LectureNotes/week42.ipynb +++ b/doc/LectureNotes/week42.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "474b773e", + "id": "71674611", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "a8ac8249", + "id": "65b3502e", "metadata": { "editable": true }, @@ -27,7 +27,7 @@ }, { "cell_type": "markdown", - "id": "c8760378", + "id": "1d840be4", "metadata": { "editable": true }, @@ -56,7 +56,7 @@ }, { "cell_type": "markdown", - "id": "76c6895d", + "id": "8c43f62c", "metadata": { "editable": true }, @@ -73,7 +73,7 @@ }, { "cell_type": "markdown", - "id": "ebb39354", + "id": "bb52c881", "metadata": { "editable": true }, @@ -92,7 +92,7 @@ }, { "cell_type": "markdown", - "id": "cfb26b1b", + "id": "e53a998a", "metadata": { "editable": true }, @@ -108,7 +108,7 @@ }, { "cell_type": "markdown", - "id": "28f60678", + "id": "2be1dbc1", "metadata": { "editable": true }, @@ -119,7 +119,7 @@ }, { "cell_type": "markdown", - "id": "3c7f8f61", + "id": "d81e5954", "metadata": { "editable": true }, @@ -133,7 +133,7 @@ }, { "cell_type": "markdown", - "id": "9ee5faa5", + "id": "bf67ca94", "metadata": { "editable": true }, @@ -148,7 +148,7 @@ }, { "cell_type": "markdown", - "id": "e82fbb01", + "id": "ea5ccdfd", "metadata": { "editable": true }, @@ -160,7 +160,7 @@ }, { "cell_type": "markdown", - "id": "e1b4862d", + "id": "a67526dc", "metadata": { "editable": true }, @@ -174,7 +174,7 @@ }, { "cell_type": "markdown", - "id": "f7b41bcb", + "id": "004f244e", "metadata": { "editable": true }, @@ -186,7 +186,7 @@ }, { "cell_type": "markdown", - "id": "2825a689", + "id": "d5019705", "metadata": { "editable": true }, @@ -202,7 +202,7 @@ }, { "cell_type": "markdown", - "id": "1853d23c", + "id": "b28a1451", "metadata": { "editable": true }, @@ -218,7 +218,7 @@ }, { "cell_type": "markdown", - "id": "1625a8c3", + "id": "bb9e817a", "metadata": { "editable": true }, @@ -230,7 +230,7 @@ }, { "cell_type": "markdown", - "id": "7427a24a", + "id": "cb070b5e", "metadata": { "editable": true }, @@ -240,7 +240,7 @@ }, { "cell_type": "markdown", - "id": "d4911a64", + "id": "f68d4801", "metadata": { "editable": true }, @@ -252,7 +252,7 @@ }, { "cell_type": "markdown", - "id": "0f290988", + "id": "fffa97bd", "metadata": { "editable": true }, @@ -262,7 +262,7 @@ }, { "cell_type": "markdown", - "id": "4cbbad95", + "id": "7f751c77", "metadata": { "editable": true }, @@ -274,7 +274,7 @@ }, { "cell_type": "markdown", - "id": "cc8122a6", + "id": "e8f8479b", "metadata": { "editable": true }, @@ -284,7 +284,7 @@ }, { "cell_type": "markdown", - "id": "39f03825", + "id": "9d52f786", "metadata": { "editable": true }, @@ -296,7 +296,7 @@ }, { "cell_type": "markdown", - "id": "3eeb7a29", + "id": "cdb55ad0", "metadata": { "editable": true }, @@ -312,7 +312,7 @@ }, { "cell_type": "markdown", - "id": "33e1d35f", + "id": "ece1a1cc", "metadata": { "editable": true }, @@ -324,7 +324,7 @@ }, { "cell_type": "markdown", - "id": "e846a441", + "id": "e2af50fb", "metadata": { "editable": true }, @@ -336,7 +336,7 @@ }, { "cell_type": "markdown", - "id": "fbb3f1ff", + "id": "c883f2ef", "metadata": { "editable": true }, @@ -346,7 +346,7 @@ }, { "cell_type": "markdown", - "id": "40200643", + "id": "f1129306", "metadata": { "editable": true }, @@ -358,7 +358,7 @@ }, { "cell_type": "markdown", - "id": "a2e96689", + "id": "1d14bedd", "metadata": { "editable": true }, @@ -368,7 +368,7 @@ }, { "cell_type": "markdown", - "id": "1a014316", + "id": "84378f69", "metadata": { "editable": true }, @@ -384,7 +384,7 @@ }, { "cell_type": "markdown", - "id": "06a0b720", + "id": "7f6f41e3", "metadata": { "editable": true }, @@ -396,7 +396,7 @@ }, { "cell_type": "markdown", - "id": "48581ce4", + "id": "f38cb151", "metadata": { "editable": true }, @@ -408,7 +408,7 @@ }, { "cell_type": "markdown", - "id": "58f46792", + "id": "d7f60566", "metadata": { "editable": true }, @@ -420,7 +420,7 @@ }, { "cell_type": "markdown", - "id": "dc5aa924", + "id": "81219134", "metadata": { "editable": true }, @@ -432,7 +432,7 @@ }, { "cell_type": "markdown", - "id": "407690af", + "id": "8f0f27a7", "metadata": { "editable": true }, @@ -444,7 +444,7 @@ }, { "cell_type": "markdown", - "id": "8f5cacad", + "id": "aa9974f2", "metadata": { "editable": true }, @@ -454,7 +454,7 @@ }, { "cell_type": "markdown", - "id": "79f592c1", + "id": "02021c85", "metadata": { "editable": true }, @@ -470,7 +470,7 @@ }, { "cell_type": "markdown", - "id": "412247f9", + "id": "d5b4c3d8", "metadata": { "editable": true }, @@ -482,7 +482,7 @@ }, { "cell_type": "markdown", - "id": "83c09361", + "id": "0c0f2d45", "metadata": { "editable": true }, @@ -494,7 +494,7 @@ }, { "cell_type": "markdown", - "id": "d341a3c7", + "id": "9f8b567b", "metadata": { "editable": true }, @@ -504,7 +504,7 @@ }, { "cell_type": "markdown", - "id": "2bb5b78f", + "id": "6afa5b0f", "metadata": { "editable": true }, @@ -516,7 +516,7 @@ }, { "cell_type": "markdown", - "id": "3d6558a8", + "id": "a8447e5f", "metadata": { "editable": true }, @@ -530,7 +530,7 @@ }, { "cell_type": "markdown", - "id": "f55c404e", + "id": "0262d1c4", "metadata": { "editable": true }, @@ -548,7 +548,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "076950d8", + "id": "91932727", "metadata": { "collapsed": false, "editable": true @@ -621,7 +621,7 @@ }, { "cell_type": "markdown", - "id": "b5a2150f", + "id": "6695945c", "metadata": { "editable": true }, @@ -631,7 +631,7 @@ }, { "cell_type": "markdown", - "id": "cb93f2d2", + "id": "30bd4411", "metadata": { "editable": true }, @@ -643,51 +643,51 @@ "layer with two hidden nodes and one output layer with one output node/neuron only (see graph)..\n", "\n", "We need to define the following parameters and variables with the input layer (layer $(0)$) \n", - "where we label the nodes $x_0$ and $x_1$" + "where we label the nodes $x_1$ and $x_2$" ] }, { "cell_type": "markdown", - "id": "893b1b33", + "id": "03303707", "metadata": { "editable": true }, "source": [ "$$\n", - "x_0 = a_0^{(0)} \\wedge x_1 = a_1^{(0)}.\n", + "x_1 = a_1^{(0)} \\wedge x_2 = a_2^{(0)}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "f1554902", + "id": "24dc6874", "metadata": { "editable": true }, "source": [ - "The hidden layer (layer $(1)$) has nodes which yield the outputs $a_0^{(1)}$ and $a_1^{(1)}$) with weight $\\boldsymbol{w}$ and bias $\\boldsymbol{b}$ parameters" + "The hidden layer (layer $(1)$) has nodes which yield the outputs $a_1^{(1)}$ and $a_2^{(1)}$) with weight $\\boldsymbol{w}$ and bias $\\boldsymbol{b}$ parameters" ] }, { "cell_type": "markdown", - "id": "e01cd011", + "id": "9229190d", "metadata": { "editable": true }, "source": [ "$$\n", - "w_{ij}^{(1)}=\\left\\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)}\\right\\} \\wedge b^{(1)}=\\left\\{b_0^{(1)},b_1^{(1)}\\right\\}.\n", + "w_{ij}^{(1)}=\\left\\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)}\\right\\} \\wedge b^{(1)}=\\left\\{b_1^{(1)},b_2^{(1)}\\right\\}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "691e42c5", + "id": "db21c4eb", "metadata": { "editable": true }, "source": [ - "## Layout of a simple neural network with two input nodes, one hidden layer and one output node\n", + "## Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node\n", "\n", "\n", "\n", @@ -698,31 +698,31 @@ }, { "cell_type": "markdown", - "id": "57573a8a", + "id": "cf9b69e8", "metadata": { "editable": true }, "source": [ "## The ouput layer\n", "\n", - "Finally, we have the ouput layer given by layer label $(2)$ with output $a^{(2)}$ and weights and biases to be determined given by the variables" + "We have the ouput layer given by layer label $(2)$ with output $a^{(2)}$ and weights and biases to be determined given by the variables" ] }, { "cell_type": "markdown", - "id": "91ff8dfb", + "id": "53130107", "metadata": { "editable": true }, "source": [ "$$\n", - "w_{i}^{(2)}=\\left\\{w_{0}^{(2)},w_{1}^{(2)}\\right\\} \\wedge b^{(2)}.\n", + "w_{i}^{(2)}=\\left\\{w_{1}^{(2)},w_{2}^{(2)}\\right\\} \\wedge b^{(2)}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "315f787f", + "id": "5835245c", "metadata": { "editable": true }, @@ -733,19 +733,19 @@ }, { "cell_type": "markdown", - "id": "612ebd67", + "id": "dd31d181", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\Theta}=\\left\\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)},w_{0}^{(2)},w_{1}^{(2)},b_0^{(1)},b_1^{(1)},b^{(2)}\\right\\}.\n", + "\\boldsymbol{\\Theta}=\\left\\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)},w_{1}^{(2)},w_{2}^{(2)},b_1^{(1)},b_2^{(1)},b^{(2)}\\right\\}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "c28762dc", + "id": "36a9d52a", "metadata": { "editable": true }, @@ -758,19 +758,19 @@ }, { "cell_type": "markdown", - "id": "b4d0309d", + "id": "3af3b240", "metadata": { "editable": true }, "source": [ "$$\n", - "\\begin{bmatrix}z_0^{(1)} \\\\ z_1^{(1)} \\end{bmatrix}=\\left(\\begin{bmatrix}w_{00}^{(1)} & w_{01}^{(1)}\\\\ w_{10}^{(1)} &w_{11}^{(1)} \\end{bmatrix}\\right)^{T}\\begin{bmatrix}a_0^{(0)} \\\\ a_1^{(0)} \\end{bmatrix}+\\begin{bmatrix}b_0^{(1)} \\\\ b_1^{(1)} \\end{bmatrix},\n", + "\\begin{bmatrix}z_1^{(1)} \\\\ z_2^{(1)} \\end{bmatrix}=\\left(\\begin{bmatrix}w_{11}^{(1)} & w_{12}^{(1)}\\\\ w_{21}^{(1)} &w_{22}^{(1)} \\end{bmatrix}\\right)^{T}\\begin{bmatrix}a_1^{(0)} \\\\ a_2^{(0)} \\end{bmatrix}+\\begin{bmatrix}b_1^{(1)} \\\\ b_2^{(1)} \\end{bmatrix},\n", "$$" ] }, { "cell_type": "markdown", - "id": "f3c715f1", + "id": "3ef7b15b", "metadata": { "editable": true }, @@ -780,19 +780,19 @@ }, { "cell_type": "markdown", - "id": "5db82289", + "id": "31e47e2c", "metadata": { "editable": true }, "source": [ "$$\n", - "\\begin{bmatrix}a_0^{(1)} \\\\ a_1^{(1)} \\end{bmatrix}=\\begin{bmatrix}\\sigma^{(1)}(z_0^{(1)}) \\\\ \\sigma^{(1)}(z_1^{(1)}) \\end{bmatrix}.\n", + "\\begin{bmatrix}a_1^{(1)} \\\\ a_2^{(1)} \\end{bmatrix}=\\begin{bmatrix}\\sigma^{(1)}(z_1^{(1)}) \\\\ \\sigma^{(1)}(z_2^{(1)}) \\end{bmatrix}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "b6b2ad78", + "id": "c5690e76", "metadata": { "editable": true }, @@ -804,19 +804,19 @@ }, { "cell_type": "markdown", - "id": "dcefdba1", + "id": "919ce153", "metadata": { "editable": true }, "source": [ "$$\n", - "z^{(2)} = w_{0}^{(2)}a_0^{(1)} +w_{1}^{(2)}a_1^{(1)}+b^{(2)},\n", + "z^{(2)} = w_{1}^{(2)}a_1^{(1)} +w_{2}^{(2)}a_2^{(1)}+b^{(2)},\n", "$$" ] }, { "cell_type": "markdown", - "id": "d554c6ce", + "id": "69df1f30", "metadata": { "editable": true }, @@ -826,7 +826,7 @@ }, { "cell_type": "markdown", - "id": "651ff447", + "id": "42ea9246", "metadata": { "editable": true }, @@ -838,7 +838,7 @@ }, { "cell_type": "markdown", - "id": "f171ceaa", + "id": "af203af1", "metadata": { "editable": true }, @@ -854,7 +854,7 @@ }, { "cell_type": "markdown", - "id": "609bb2dd", + "id": "fd36e08b", "metadata": { "editable": true }, @@ -866,7 +866,7 @@ }, { "cell_type": "markdown", - "id": "da1d696b", + "id": "997bddf7", "metadata": { "editable": true }, @@ -876,7 +876,7 @@ }, { "cell_type": "markdown", - "id": "d21d62cc", + "id": "ba4380bd", "metadata": { "editable": true }, @@ -888,7 +888,7 @@ }, { "cell_type": "markdown", - "id": "fb9dd90f", + "id": "13be072b", "metadata": { "editable": true }, @@ -898,7 +898,7 @@ }, { "cell_type": "markdown", - "id": "9026fb2d", + "id": "2c6bcc22", "metadata": { "editable": true }, @@ -910,7 +910,7 @@ }, { "cell_type": "markdown", - "id": "2d20d5d1", + "id": "a50dfdeb", "metadata": { "editable": true }, @@ -922,20 +922,20 @@ }, { "cell_type": "markdown", - "id": "e6fbb9b5", + "id": "0d50cd56", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial w_{00}^{(1)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n", - "\\frac{\\partial z^{(2)}}{\\partial z_0^{(1)}}\\frac{\\partial z_0^{(1)}}{\\partial w_{00}^{(1)}}= \\delta^{(2)}\\frac{\\partial z^{(2)}}{\\partial z_0^{(1)}}\\frac{\\partial z_0^{(1)}}{\\partial w_{00}^{(1)}},\n", + "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n", + "\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}}= \\delta^{(2)}\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}},\n", "$$" ] }, { "cell_type": "markdown", - "id": "781a7ac8", + "id": "4fc9436b", "metadata": { "editable": true }, @@ -945,19 +945,19 @@ }, { "cell_type": "markdown", - "id": "3ee09d8f", + "id": "7545f5c9", "metadata": { "editable": true }, "source": [ "$$\n", - "z^{(2)} =w_0^{(2)}a_0^{(1)}+w_1^{(2)}a_1^{(1)}+b^{(2)},\n", + "z^{(2)} =w_1^{(2)}a_1^{(1)}+w_2^{(2)}a_2^{(1)}+b^{(2)},\n", "$$" ] }, { "cell_type": "markdown", - "id": "65f44692", + "id": "bcbea03f", "metadata": { "editable": true }, @@ -967,19 +967,19 @@ }, { "cell_type": "markdown", - "id": "0b84b7af", + "id": "ee05da6d", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial z^{(2)}}{\\partial z_0^{(1)}}\\frac{\\partial z_0^{(1)}}{\\partial w_{00}^{(1)}}=w_0^{(2)}\\frac{\\partial a_0^{(1)}}{\\partial z_0^{(1)}}a_0^{(1)}.\n", + "\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}a_1^{(1)}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "53c138bd", + "id": "1f9491ce", "metadata": { "editable": true }, @@ -990,19 +990,19 @@ }, { "cell_type": "markdown", - "id": "3f81bb3f", + "id": "07772fef", "metadata": { "editable": true }, "source": [ "$$\n", - "\\delta_0^{(1)}=w_0^{(2)}\\frac{\\partial a_0^{(1)}}{\\partial z_0^{(1)}}\\delta^{(2)},\n", + "\\delta_1^{(1)}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}\\delta^{(2)},\n", "$$" ] }, { "cell_type": "markdown", - "id": "eb6783b2", + "id": "c432668f", "metadata": { "editable": true }, @@ -1012,19 +1012,19 @@ }, { "cell_type": "markdown", - "id": "1c754dbb", + "id": "4274417c", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial w_{00}^{(1)}}=\\delta_0^{(1)}a_0^{(1)}.\n", + "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\delta_1^{(1)}a_1^{(1)}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "e0b9760c", + "id": "b615718d", "metadata": { "editable": true }, @@ -1034,19 +1034,19 @@ }, { "cell_type": "markdown", - "id": "b1efb446", + "id": "c541b15f", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial w_{01}^{(1)}}=\\delta_0^{(1)}a_1^{(1)}.\n", + "\\frac{\\partial C}{\\partial w_{12}^{(1)}}=\\delta_1^{(1)}a_2^{(1)}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "2cf7b43f", + "id": "6b741552", "metadata": { "editable": true }, @@ -1058,19 +1058,19 @@ }, { "cell_type": "markdown", - "id": "262b517f", + "id": "d564e7a9", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial w_{10}^{(1)}}=\\delta_1^{(1)}a_0^{(1)},\n", + "\\frac{\\partial C}{\\partial w_{21}^{(1)}}=\\delta_2^{(1)}a_1^{(1)},\n", "$$" ] }, { "cell_type": "markdown", - "id": "59285e3f", + "id": "927894b5", "metadata": { "editable": true }, @@ -1080,19 +1080,19 @@ }, { "cell_type": "markdown", - "id": "2813790f", + "id": "75624550", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\delta_1^{(1)}a_1^{(1)},\n", + "\\frac{\\partial C}{\\partial w_{22}^{(1)}}=\\delta_2^{(1)}a_2^{(1)},\n", "$$" ] }, { "cell_type": "markdown", - "id": "a56b94b4", + "id": "9252c078", "metadata": { "editable": true }, @@ -1102,19 +1102,19 @@ }, { "cell_type": "markdown", - "id": "a3ed9010", + "id": "10c7da6e", "metadata": { "editable": true }, "source": [ "$$\n", - "\\delta_1^{(1)}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}\\delta^{(2)}.\n", + "\\delta_2^{(1)}=w_2^{(2)}\\frac{\\partial a_2^{(1)}}{\\partial z_2^{(1)}}\\delta^{(2)}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "03a5562c", + "id": "0fe640a9", "metadata": { "editable": true }, @@ -1126,19 +1126,19 @@ }, { "cell_type": "markdown", - "id": "025bd04f", + "id": "01ff9a38", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial b_{0}^{(1)}}=\\delta_0^{(1)},\n", + "\\frac{\\partial C}{\\partial b_{1}^{(1)}}=\\delta_1^{(1)},\n", "$$" ] }, { "cell_type": "markdown", - "id": "e1ef6c2f", + "id": "37fda9de", "metadata": { "editable": true }, @@ -1148,19 +1148,19 @@ }, { "cell_type": "markdown", - "id": "157d7580", + "id": "861af2b9", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial b_{1}^{(1)}}=\\delta_1^{(1)}.\n", + "\\frac{\\partial C}{\\partial b_{2}^{(1)}}=\\delta_2^{(1)}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "389a7e2d", + "id": "f9cea8b7", "metadata": { "editable": true }, @@ -1170,7 +1170,7 @@ }, { "cell_type": "markdown", - "id": "bccf9918", + "id": "12e3298b", "metadata": { "editable": true }, @@ -1183,7 +1183,7 @@ }, { "cell_type": "markdown", - "id": "ab05945c", + "id": "a104df98", "metadata": { "editable": true }, @@ -1195,7 +1195,7 @@ }, { "cell_type": "markdown", - "id": "cb2ef195", + "id": "9bc2f036", "metadata": { "editable": true }, @@ -1205,7 +1205,7 @@ }, { "cell_type": "markdown", - "id": "da672f61", + "id": "568ced5c", "metadata": { "editable": true }, @@ -1217,7 +1217,7 @@ }, { "cell_type": "markdown", - "id": "95dd3d2b", + "id": "906d2bd9", "metadata": { "editable": true }, @@ -1227,7 +1227,7 @@ }, { "cell_type": "markdown", - "id": "532b99a2", + "id": "79992e6f", "metadata": { "editable": true }, @@ -1239,7 +1239,7 @@ }, { "cell_type": "markdown", - "id": "5d5f45cc", + "id": "0745b6ba", "metadata": { "editable": true }, @@ -1249,7 +1249,7 @@ }, { "cell_type": "markdown", - "id": "2333cfb1", + "id": "4fb2781f", "metadata": { "editable": true }, @@ -1261,7 +1261,7 @@ }, { "cell_type": "markdown", - "id": "4b555f81", + "id": "57576b6e", "metadata": { "editable": true }, @@ -1271,7 +1271,7 @@ }, { "cell_type": "markdown", - "id": "20c0952f", + "id": "d1f38053", "metadata": { "editable": true }, @@ -1288,7 +1288,7 @@ }, { "cell_type": "markdown", - "id": "5a155f81", + "id": "6f6f31e8", "metadata": { "editable": true }, @@ -1300,7 +1300,7 @@ }, { "cell_type": "markdown", - "id": "ff1fc071", + "id": "f206ae2b", "metadata": { "editable": true }, @@ -1312,7 +1312,7 @@ }, { "cell_type": "markdown", - "id": "63576478", + "id": "5e7af877", "metadata": { "editable": true }, @@ -1328,7 +1328,7 @@ }, { "cell_type": "markdown", - "id": "3410ad89", + "id": "96c13dab", "metadata": { "editable": true }, @@ -1345,7 +1345,7 @@ }, { "cell_type": "markdown", - "id": "c71d887c", + "id": "a6781c7d", "metadata": { "editable": true }, @@ -1357,7 +1357,7 @@ }, { "cell_type": "markdown", - "id": "ffeae21c", + "id": "4db58da4", "metadata": { "editable": true }, @@ -1370,7 +1370,7 @@ }, { "cell_type": "markdown", - "id": "2fe917a4", + "id": "b4458c55", "metadata": { "editable": true }, @@ -1382,7 +1382,7 @@ }, { "cell_type": "markdown", - "id": "65404bec", + "id": "b32e0714", "metadata": { "editable": true }, @@ -1398,7 +1398,7 @@ }, { "cell_type": "markdown", - "id": "a6c0c8b6", + "id": "fffb7785", "metadata": { "editable": true }, @@ -1410,7 +1410,7 @@ }, { "cell_type": "markdown", - "id": "2454faec", + "id": "08bff16c", "metadata": { "editable": true }, @@ -1426,7 +1426,7 @@ }, { "cell_type": "markdown", - "id": "e5b67040", + "id": "fb907bb3", "metadata": { "editable": true }, @@ -1438,7 +1438,7 @@ }, { "cell_type": "markdown", - "id": "aaec49a4", + "id": "f97ed7ef", "metadata": { "editable": true }, @@ -1450,7 +1450,7 @@ }, { "cell_type": "markdown", - "id": "e8ad40d3", + "id": "136c2230", "metadata": { "editable": true }, @@ -1460,7 +1460,7 @@ }, { "cell_type": "markdown", - "id": "e04a4d28", + "id": "4b2344b6", "metadata": { "editable": true }, @@ -1472,7 +1472,7 @@ }, { "cell_type": "markdown", - "id": "9b121a33", + "id": "52a4e7a7", "metadata": { "editable": true }, @@ -1482,7 +1482,7 @@ }, { "cell_type": "markdown", - "id": "3bf7d884", + "id": "163ee2e5", "metadata": { "editable": true }, @@ -1494,7 +1494,7 @@ }, { "cell_type": "markdown", - "id": "7d58918c", + "id": "5aa607a5", "metadata": { "editable": true }, @@ -1508,7 +1508,7 @@ }, { "cell_type": "markdown", - "id": "d21dce97", + "id": "da13c77b", "metadata": { "editable": true }, @@ -1520,7 +1520,7 @@ }, { "cell_type": "markdown", - "id": "dc143bea", + "id": "7bf944d0", "metadata": { "editable": true }, @@ -1530,7 +1530,7 @@ }, { "cell_type": "markdown", - "id": "49415a06", + "id": "ea130e95", "metadata": { "editable": true }, @@ -1542,7 +1542,7 @@ }, { "cell_type": "markdown", - "id": "4398e11e", + "id": "2fba64b7", "metadata": { "editable": true }, @@ -1552,7 +1552,7 @@ }, { "cell_type": "markdown", - "id": "da3fca7e", + "id": "5904a528", "metadata": { "editable": true }, @@ -1564,7 +1564,7 @@ }, { "cell_type": "markdown", - "id": "9055bc57", + "id": "86f8199b", "metadata": { "editable": true }, @@ -1576,7 +1576,7 @@ }, { "cell_type": "markdown", - "id": "c7995e73", + "id": "e4370f9f", "metadata": { "editable": true }, @@ -1588,7 +1588,7 @@ }, { "cell_type": "markdown", - "id": "d142f966", + "id": "f60e1730", "metadata": { "editable": true }, @@ -1598,7 +1598,7 @@ }, { "cell_type": "markdown", - "id": "d5dc5f9f", + "id": "e282d002", "metadata": { "editable": true }, @@ -1610,7 +1610,7 @@ }, { "cell_type": "markdown", - "id": "5f598e3e", + "id": "5de6d59f", "metadata": { "editable": true }, @@ -1620,7 +1620,7 @@ }, { "cell_type": "markdown", - "id": "9769b50e", + "id": "97c35e7d", "metadata": { "editable": true }, @@ -1632,7 +1632,7 @@ }, { "cell_type": "markdown", - "id": "5a8851ae", + "id": "c4754c54", "metadata": { "editable": true }, @@ -1650,7 +1650,7 @@ }, { "cell_type": "markdown", - "id": "b455efff", + "id": "0b03f12b", "metadata": { "editable": true }, @@ -1668,7 +1668,7 @@ }, { "cell_type": "markdown", - "id": "8b430778", + "id": "ad079735", "metadata": { "editable": true }, @@ -1680,7 +1680,7 @@ }, { "cell_type": "markdown", - "id": "7fb2cea5", + "id": "0bf757b6", "metadata": { "editable": true }, @@ -1690,7 +1690,7 @@ }, { "cell_type": "markdown", - "id": "6c77ac74", + "id": "b6246783", "metadata": { "editable": true }, @@ -1702,7 +1702,7 @@ }, { "cell_type": "markdown", - "id": "708f42bb", + "id": "b7575d50", "metadata": { "editable": true }, @@ -1714,7 +1714,7 @@ }, { "cell_type": "markdown", - "id": "40f66bf5", + "id": "e6c93f95", "metadata": { "editable": true }, @@ -1726,7 +1726,7 @@ }, { "cell_type": "markdown", - "id": "21b9ebbe", + "id": "b52b78ac", "metadata": { "editable": true }, @@ -1736,7 +1736,7 @@ }, { "cell_type": "markdown", - "id": "665ad548", + "id": "a5fdfb9c", "metadata": { "editable": true }, @@ -1748,7 +1748,7 @@ }, { "cell_type": "markdown", - "id": "c182abd9", + "id": "8bd7f846", "metadata": { "editable": true }, @@ -1758,7 +1758,7 @@ }, { "cell_type": "markdown", - "id": "aa70c6cf", + "id": "550334c8", "metadata": { "editable": true }, @@ -1770,7 +1770,7 @@ }, { "cell_type": "markdown", - "id": "db460a07", + "id": "ac298c05", "metadata": { "editable": true }, @@ -1788,7 +1788,7 @@ }, { "cell_type": "markdown", - "id": "3af036d6", + "id": "6609cbf5", "metadata": { "editable": true }, @@ -1798,7 +1798,7 @@ }, { "cell_type": "markdown", - "id": "5662288a", + "id": "64741262", "metadata": { "editable": true }, @@ -1816,7 +1816,7 @@ }, { "cell_type": "markdown", - "id": "b6a8a1ac", + "id": "76c4610d", "metadata": { "editable": true }, @@ -1826,7 +1826,7 @@ }, { "cell_type": "markdown", - "id": "dda8fce0", + "id": "efd0ba93", "metadata": { "editable": true }, @@ -1844,7 +1844,7 @@ }, { "cell_type": "markdown", - "id": "f1957ab0", + "id": "98e88435", "metadata": { "editable": true }, @@ -1856,7 +1856,7 @@ }, { "cell_type": "markdown", - "id": "31c8f2c9", + "id": "08bdbdff", "metadata": { "editable": true }, @@ -1868,7 +1868,7 @@ }, { "cell_type": "markdown", - "id": "bba8d29e", + "id": "e234c8a6", "metadata": { "editable": true }, @@ -1878,7 +1878,7 @@ }, { "cell_type": "markdown", - "id": "af4922ec", + "id": "6a4118be", "metadata": { "editable": true }, @@ -1890,7 +1890,7 @@ }, { "cell_type": "markdown", - "id": "f43f0f87", + "id": "d211df4b", "metadata": { "editable": true }, @@ -1902,7 +1902,7 @@ }, { "cell_type": "markdown", - "id": "eba489e4", + "id": "bf0c2817", "metadata": { "editable": true }, @@ -1912,7 +1912,7 @@ }, { "cell_type": "markdown", - "id": "82610f92", + "id": "2de9333a", "metadata": { "editable": true }, @@ -1924,7 +1924,7 @@ }, { "cell_type": "markdown", - "id": "62513d62", + "id": "00d82ece", "metadata": { "editable": true }, @@ -1934,7 +1934,7 @@ }, { "cell_type": "markdown", - "id": "ca381ec8", + "id": "19f78643", "metadata": { "editable": true }, @@ -1946,7 +1946,7 @@ }, { "cell_type": "markdown", - "id": "bdf32ad4", + "id": "1424f687", "metadata": { "editable": true }, @@ -1958,14 +1958,40 @@ }, { "cell_type": "markdown", - "id": "764448ea", + "id": "9d7c23b2", "metadata": { "editable": true }, "source": [ - "## Setting up the back propagation algorithm\n", + "## Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations\n", "\n", - "The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.\n", + "**The architecture (our model).**\n", + "\n", + "1. Set up your inputs and outputs (scalars, vectors, matrices or higher-order arrays)\n", + "\n", + "2. Define the number of hidden layers and hidden nodes\n", + "\n", + "3. Define activation functions for hidden layers and output layers\n", + "\n", + "4. Define optimizer (plan learning rate, momentum, ADAgrad, RMSprop, ADAM etc) and array of initial learning rates\n", + "\n", + "5. Define cost function and possible regularization terms with hyperparameters\n", + "\n", + "6. Initialize weights and biases\n", + "\n", + "7. Fix number of iterations for the feed forward part and back propagation part" + ] + }, + { + "cell_type": "markdown", + "id": "1decfbef", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the back propagation algorithm, part 1\n", + "\n", + "The four equations provide us with a way of computing the gradients of the cost function. Let us write this out in the form of an algorithm.\n", "\n", "**First**, we set up the input data $\\boldsymbol{x}$ and the activations\n", "$\\boldsymbol{z}_1$ of the input layer and compute the activation function and\n", @@ -1981,7 +2007,7 @@ }, { "cell_type": "markdown", - "id": "989c3082", + "id": "7f237d52", "metadata": { "editable": true }, @@ -1993,7 +2019,7 @@ }, { "cell_type": "markdown", - "id": "9d14949a", + "id": "d37fa1b5", "metadata": { "editable": true }, @@ -2005,7 +2031,7 @@ }, { "cell_type": "markdown", - "id": "4ee118b1", + "id": "213b757d", "metadata": { "editable": true }, @@ -2015,7 +2041,7 @@ }, { "cell_type": "markdown", - "id": "ba89ad67", + "id": "3137751d", "metadata": { "editable": true }, @@ -2027,7 +2053,7 @@ }, { "cell_type": "markdown", - "id": "e1f840c5", + "id": "da1cf61b", "metadata": { "editable": true }, @@ -2041,7 +2067,7 @@ }, { "cell_type": "markdown", - "id": "9f06bcfc", + "id": "3b57ef97", "metadata": { "editable": true }, @@ -2053,7 +2079,7 @@ }, { "cell_type": "markdown", - "id": "7063f2be", + "id": "bb0c4d59", "metadata": { "editable": true }, @@ -2065,7 +2091,7 @@ }, { "cell_type": "markdown", - "id": "3e299445", + "id": "a9483fc9", "metadata": { "editable": true }, @@ -2075,7 +2101,7 @@ }, { "cell_type": "markdown", - "id": "28921878", + "id": "49c7c2f3", "metadata": { "editable": true }, @@ -2087,7 +2113,7 @@ }, { "cell_type": "markdown", - "id": "90f5404c", + "id": "734ef014", "metadata": { "editable": true }, @@ -2099,7 +2125,7 @@ }, { "cell_type": "markdown", - "id": "53c2dfbd", + "id": "ac393c38", "metadata": { "editable": true }, @@ -2109,7 +2135,7 @@ }, { "cell_type": "markdown", - "id": "03805799", + "id": "c38ed8eb", "metadata": { "editable": true }, @@ -2121,7 +2147,7 @@ }, { "cell_type": "markdown", - "id": "4f8f9c02", + "id": "d097bbf6", "metadata": { "editable": true }, @@ -2133,17 +2159,17 @@ }, { "cell_type": "markdown", - "id": "9bf86f9c", + "id": "56e28349", "metadata": { "editable": true }, "source": [ - "### Activation functions\n", + "## Activation functions\n", "\n", "A property that characterizes a neural network, other than its\n", - "connectivity, is the choice of activation function(s). As described\n", - "in, the following restrictions are imposed on an activation function\n", - "for a FFNN to fulfill the universal approximation theorem\n", + "connectivity, is the choice of activation function(s). The following\n", + "restrictions are imposed on an activation function for an FFNN to\n", + "fulfill the universal approximation theorem\n", "\n", " * Non-constant\n", "\n", @@ -2156,7 +2182,7 @@ }, { "cell_type": "markdown", - "id": "eeaed73c", + "id": "0f764f08", "metadata": { "editable": true }, @@ -2175,7 +2201,7 @@ }, { "cell_type": "markdown", - "id": "0786aaf0", + "id": "697fbd9c", "metadata": { "editable": true }, @@ -2187,7 +2213,7 @@ }, { "cell_type": "markdown", - "id": "aa76601b", + "id": "e9f79c9b", "metadata": { "editable": true }, @@ -2197,7 +2223,7 @@ }, { "cell_type": "markdown", - "id": "e1bca915", + "id": "8285a58e", "metadata": { "editable": true }, @@ -2209,12 +2235,12 @@ }, { "cell_type": "markdown", - "id": "1832a8c4", + "id": "eba13151", "metadata": { "editable": true }, "source": [ - "### Relevance\n", + "## Relevance\n", "\n", "The *sigmoid* function are more biologically plausible because the\n", "output of inactive neurons are zero. Such activation function are\n", @@ -2226,7 +2252,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "5c689736", + "id": "d5693cd0", "metadata": { "collapsed": false, "editable": true @@ -2310,7 +2336,228 @@ }, { "cell_type": "markdown", - "id": "0b3a4c95", + "id": "b0e28a6b", + "metadata": { + "editable": true + }, + "source": [ + "## Vanishing gradients\n", + "\n", + "The Back propagation algorithm we derived above works by going from\n", + "the output layer to the input layer, propagating the error gradient on\n", + "the way. Once the algorithm has computed the gradient of the cost\n", + "function with regards to each parameter in the network, it uses these\n", + "gradients to update each parameter with a Gradient Descent (GD) step.\n", + "\n", + "Unfortunately for us, the gradients often get smaller and smaller as\n", + "the algorithm progresses down to the first hidden layers. As a result,\n", + "the GD update leaves the lower layer connection weights virtually\n", + "unchanged, and training never converges to a good solution. This is\n", + "known in the literature as **the vanishing gradients problem**." + ] + }, + { + "cell_type": "markdown", + "id": "436fb27b", + "metadata": { + "editable": true + }, + "source": [ + "## Exploding gradients\n", + "\n", + "In other cases, the opposite can happen, namely the the gradients can\n", + "grow bigger and bigger. The result is that many of the layers get\n", + "large updates of the weights the algorithm diverges. This is the\n", + "**exploding gradients problem**, which is mostly encountered in\n", + "recurrent neural networks. More generally, deep neural networks suffer\n", + "from unstable gradients, different layers may learn at widely\n", + "different speeds" + ] + }, + { + "cell_type": "markdown", + "id": "9319de67", + "metadata": { + "editable": true + }, + "source": [ + "## Is the Logistic activation function (Sigmoid) our choice?\n", + "\n", + "Although this unfortunate behavior has been empirically observed for\n", + "quite a while (it was one of the reasons why deep neural networks were\n", + "mostly abandoned for a long time), it is only around 2010 that\n", + "significant progress was made in understanding it.\n", + "\n", + "A paper titled [Understanding the Difficulty of Training Deep\n", + "Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio](http://proceedings.mlr.press/v9/glorot10a.html) found that\n", + "the problems with the popular logistic\n", + "sigmoid activation function and the weight initialization technique\n", + "that was most popular at the time, namely random initialization using\n", + "a normal distribution with a mean of 0 and a standard deviation of\n", + "1." + ] + }, + { + "cell_type": "markdown", + "id": "e90fbb0a", + "metadata": { + "editable": true + }, + "source": [ + "## Logistic function as the root of problems\n", + "\n", + "They showed that with this activation function and this\n", + "initialization scheme, the variance of the outputs of each layer is\n", + "much greater than the variance of its inputs. Going forward in the\n", + "network, the variance keeps increasing after each layer until the\n", + "activation function saturates at the top layers. This is actually made\n", + "worse by the fact that the logistic function has a mean of 0.5, not 0\n", + "(the hyperbolic tangent function has a mean of 0 and behaves slightly\n", + "better than the logistic function in deep networks)." + ] + }, + { + "cell_type": "markdown", + "id": "7a18427f", + "metadata": { + "editable": true + }, + "source": [ + "## The derivative of the Logistic funtion\n", + "\n", + "Looking at the logistic activation function, when inputs become large\n", + "(negative or positive), the function saturates at 0 or 1, with a\n", + "derivative extremely close to 0. Thus when backpropagation kicks in,\n", + "it has virtually no gradient to propagate back through the network,\n", + "and what little gradient exists keeps getting diluted as\n", + "backpropagation progresses down through the top layers, so there is\n", + "really nothing left for the lower layers.\n", + "\n", + "In their paper, Glorot and Bengio propose a way to significantly\n", + "alleviate this problem. We need the signal to flow properly in both\n", + "directions: in the forward direction when making predictions, and in\n", + "the reverse direction when backpropagating gradients. We don’t want\n", + "the signal to die out, nor do we want it to explode and saturate. For\n", + "the signal to flow properly, the authors argue that we need the\n", + "variance of the outputs of each layer to be equal to the variance of\n", + "its inputs, and we also need the gradients to have equal variance\n", + "before and after flowing through a layer in the reverse direction." + ] + }, + { + "cell_type": "markdown", + "id": "ac352aa1", + "metadata": { + "editable": true + }, + "source": [ + "## Insights from the paper by Glorot and Bengio\n", + "\n", + "One of the insights in the 2010 paper by Glorot and Bengio was that\n", + "the vanishing/exploding gradients problems were in part due to a poor\n", + "choice of activation function. Until then most people had assumed that\n", + "if Nature had chosen to use roughly sigmoid activation functions in\n", + "biological neurons, they must be an excellent choice. But it turns out\n", + "that other activation functions behave much better in deep neural\n", + "networks, in particular the ReLU activation function, mostly because\n", + "it does not saturate for positive values (and also because it is quite\n", + "fast to compute)." + ] + }, + { + "cell_type": "markdown", + "id": "67a8bef0", + "metadata": { + "editable": true + }, + "source": [ + "## The RELU function family\n", + "\n", + "The ReLU activation function suffers from a problem known as the dying\n", + "ReLUs: during training, some neurons effectively die, meaning they\n", + "stop outputting anything other than 0.\n", + "\n", + "In some cases, you may find that half of your network’s neurons are\n", + "dead, especially if you used a large learning rate. During training,\n", + "if a neuron’s weights get updated such that the weighted sum of the\n", + "neuron’s inputs is negative, it will start outputting 0. When this\n", + "happen, the neuron is unlikely to come back to life since the gradient\n", + "of the ReLU function is 0 when its input is negative." + ] + }, + { + "cell_type": "markdown", + "id": "76de2016", + "metadata": { + "editable": true + }, + "source": [ + "## ELU function\n", + "\n", + "To solve this problem, nowadays practitioners use a variant of the\n", + "ReLU function, such as the leaky ReLU discussed above or the so-called\n", + "exponential linear unit (ELU) function" + ] + }, + { + "cell_type": "markdown", + "id": "e798fa5d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "6f33abb9", + "metadata": { + "editable": true + }, + "source": [ + "## Which activation function should we use?\n", + "\n", + "In general it seems that the ELU activation function is better than\n", + "the leaky ReLU function (and its variants), which is better than\n", + "ReLU. ReLU performs better than $\\tanh$ which in turn performs better\n", + "than the logistic function.\n", + "\n", + "If runtime performance is an issue, then you may opt for the leaky\n", + "ReLU function over the ELU function If you don’t want to tweak yet\n", + "another hyperparameter, you may just use the default $\\alpha$ of\n", + "$0.01$ for the leaky ReLU, and $1$ for ELU. If you have spare time and\n", + "computing power, you can use cross-validation or bootstrap to evaluate\n", + "other activation functions." + ] + }, + { + "cell_type": "markdown", + "id": "ad76ab9d", + "metadata": { + "editable": true + }, + "source": [ + "## More on activation functions, output layers\n", + "\n", + "In most cases you can use the ReLU activation function in the hidden\n", + "layers (or one of its variants).\n", + "\n", + "It is a bit faster to compute than other activation functions, and the\n", + "gradient descent optimization does in general not get stuck.\n", + "\n", + "**For the output layer:**\n", + "\n", + "* For classification the softmax activation function is generally a good choice for classification tasks (when the classes are mutually exclusive).\n", + "\n", + "* For regression tasks, you can simply use no activation function at all." + ] + }, + { + "cell_type": "markdown", + "id": "4d0588bb", "metadata": { "editable": true }, @@ -2339,7 +2586,7 @@ }, { "cell_type": "markdown", - "id": "9955a636", + "id": "cb5679f8", "metadata": { "editable": true }, @@ -2347,7 +2594,7 @@ "## Hidden layers\n", "\n", "For many problems you can start with just one or two hidden layers and\n", - "it will work just fine. For the MNIST data set you ca easily get a\n", + "it will work just fine. For the MNIST data set discussed below you can easily get a\n", "high accuracy using just one hidden layer with a few hundred neurons.\n", "You can reach for this data set above 98% accuracy using two hidden\n", "layers with the same total amount of neurons, in roughly the same\n", @@ -2365,228 +2612,7 @@ }, { "cell_type": "markdown", - "id": "f1aa77e9", - "metadata": { - "editable": true - }, - "source": [ - "## Vanishing gradients\n", - "\n", - "The Back propagation algorithm we derived above works by going from\n", - "the output layer to the input layer, propagating the error gradient on\n", - "the way. Once the algorithm has computed the gradient of the cost\n", - "function with regards to each parameter in the network, it uses these\n", - "gradients to update each parameter with a Gradient Descent (GD) step.\n", - "\n", - "Unfortunately for us, the gradients often get smaller and smaller as\n", - "the algorithm progresses down to the first hidden layers. As a result,\n", - "the GD update leaves the lower layer connection weights virtually\n", - "unchanged, and training never converges to a good solution. This is\n", - "known in the literature as **the vanishing gradients problem**." - ] - }, - { - "cell_type": "markdown", - "id": "3703bb33", - "metadata": { - "editable": true - }, - "source": [ - "## Exploding gradients\n", - "\n", - "In other cases, the opposite can happen, namely the the gradients can\n", - "grow bigger and bigger. The result is that many of the layers get\n", - "large updates of the weights the algorithm diverges. This is the\n", - "**exploding gradients problem**, which is mostly encountered in\n", - "recurrent neural networks. More generally, deep neural networks suffer\n", - "from unstable gradients, different layers may learn at widely\n", - "different speeds" - ] - }, - { - "cell_type": "markdown", - "id": "43b1aa26", - "metadata": { - "editable": true - }, - "source": [ - "## Is the Logistic activation function (Sigmoid) our choice?\n", - "\n", - "Although this unfortunate behavior has been empirically observed for\n", - "quite a while (it was one of the reasons why deep neural networks were\n", - "mostly abandoned for a long time), it is only around 2010 that\n", - "significant progress was made in understanding it.\n", - "\n", - "A paper titled [Understanding the Difficulty of Training Deep\n", - "Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio](http://proceedings.mlr.press/v9/glorot10a.html) found that\n", - "the problems with the popular logistic\n", - "sigmoid activation function and the weight initialization technique\n", - "that was most popular at the time, namely random initialization using\n", - "a normal distribution with a mean of 0 and a standard deviation of\n", - "1." - ] - }, - { - "cell_type": "markdown", - "id": "463f4f64", - "metadata": { - "editable": true - }, - "source": [ - "## Logistic function as the root of problems\n", - "\n", - "They showed that with this activation function and this\n", - "initialization scheme, the variance of the outputs of each layer is\n", - "much greater than the variance of its inputs. Going forward in the\n", - "network, the variance keeps increasing after each layer until the\n", - "activation function saturates at the top layers. This is actually made\n", - "worse by the fact that the logistic function has a mean of 0.5, not 0\n", - "(the hyperbolic tangent function has a mean of 0 and behaves slightly\n", - "better than the logistic function in deep networks)." - ] - }, - { - "cell_type": "markdown", - "id": "6c9ea582", - "metadata": { - "editable": true - }, - "source": [ - "## The derivative of the Logistic funtion\n", - "\n", - "Looking at the logistic activation function, when inputs become large\n", - "(negative or positive), the function saturates at 0 or 1, with a\n", - "derivative extremely close to 0. Thus when backpropagation kicks in,\n", - "it has virtually no gradient to propagate back through the network,\n", - "and what little gradient exists keeps getting diluted as\n", - "backpropagation progresses down through the top layers, so there is\n", - "really nothing left for the lower layers.\n", - "\n", - "In their paper, Glorot and Bengio propose a way to significantly\n", - "alleviate this problem. We need the signal to flow properly in both\n", - "directions: in the forward direction when making predictions, and in\n", - "the reverse direction when backpropagating gradients. We don’t want\n", - "the signal to die out, nor do we want it to explode and saturate. For\n", - "the signal to flow properly, the authors argue that we need the\n", - "variance of the outputs of each layer to be equal to the variance of\n", - "its inputs, and we also need the gradients to have equal variance\n", - "before and after flowing through a layer in the reverse direction." - ] - }, - { - "cell_type": "markdown", - "id": "80c83d2c", - "metadata": { - "editable": true - }, - "source": [ - "## Insights from the paper by Glorot and Bengio\n", - "\n", - "One of the insights in the 2010 paper by Glorot and Bengio was that\n", - "the vanishing/exploding gradients problems were in part due to a poor\n", - "choice of activation function. Until then most people had assumed that\n", - "if Nature had chosen to use roughly sigmoid activation functions in\n", - "biological neurons, they must be an excellent choice. But it turns out\n", - "that other activation functions behave much better in deep neural\n", - "networks, in particular the ReLU activation function, mostly because\n", - "it does not saturate for positive values (and also because it is quite\n", - "fast to compute)." - ] - }, - { - "cell_type": "markdown", - "id": "55a1ad5d", - "metadata": { - "editable": true - }, - "source": [ - "## The RELU function family\n", - "\n", - "The ReLU activation function suffers from a problem known as the dying\n", - "ReLUs: during training, some neurons effectively die, meaning they\n", - "stop outputting anything other than 0.\n", - "\n", - "In some cases, you may find that half of your network’s neurons are\n", - "dead, especially if you used a large learning rate. During training,\n", - "if a neuron’s weights get updated such that the weighted sum of the\n", - "neuron’s inputs is negative, it will start outputting 0. When this\n", - "happen, the neuron is unlikely to come back to life since the gradient\n", - "of the ReLU function is 0 when its input is negative." - ] - }, - { - "cell_type": "markdown", - "id": "b7fafe4a", - "metadata": { - "editable": true - }, - "source": [ - "## ELU function\n", - "\n", - "To solve this problem, nowadays practitioners use a variant of the\n", - "ReLU function, such as the leaky ReLU discussed above or the so-called\n", - "exponential linear unit (ELU) function" - ] - }, - { - "cell_type": "markdown", - "id": "46e10153", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "0930ba50", - "metadata": { - "editable": true - }, - "source": [ - "## Which activation function should we use?\n", - "\n", - "In general it seems that the ELU activation function is better than\n", - "the leaky ReLU function (and its variants), which is better than\n", - "ReLU. ReLU performs better than $\\tanh$ which in turn performs better\n", - "than the logistic function.\n", - "\n", - "If runtime performance is an issue, then you may opt for the leaky\n", - "ReLU function over the ELU function If you don’t want to tweak yet\n", - "another hyperparameter, you may just use the default $\\alpha$ of\n", - "$0.01$ for the leaky ReLU, and $1$ for ELU. If you have spare time and\n", - "computing power, you can use cross-validation or bootstrap to evaluate\n", - "other activation functions." - ] - }, - { - "cell_type": "markdown", - "id": "5c0c59af", - "metadata": { - "editable": true - }, - "source": [ - "## More on activation functions, output layers\n", - "\n", - "In most cases you can use the ReLU activation function in the hidden\n", - "layers (or one of its variants).\n", - "\n", - "It is a bit faster to compute than other activation functions, and the\n", - "gradient descent optimization does in general not get stuck.\n", - "\n", - "**For the output layer:**\n", - "\n", - "* For classification the softmax activation function is generally a good choice for classification tasks (when the classes are mutually exclusive).\n", - "\n", - "* For regression tasks, you can simply use no activation function at all." - ] - }, - { - "cell_type": "markdown", - "id": "582e4df4", + "id": "fa928c9b", "metadata": { "editable": true }, @@ -2612,7 +2638,7 @@ }, { "cell_type": "markdown", - "id": "ea360013", + "id": "4a9462ce", "metadata": { "editable": true }, @@ -2632,7 +2658,7 @@ }, { "cell_type": "markdown", - "id": "fee98b12", + "id": "e523997a", "metadata": { "editable": true }, @@ -2652,7 +2678,7 @@ }, { "cell_type": "markdown", - "id": "c9f175b0", + "id": "19bba5e1", "metadata": { "editable": true }, @@ -2678,7 +2704,7 @@ }, { "cell_type": "markdown", - "id": "f6f04fbb", + "id": "ff6b5f13", "metadata": { "editable": true }, @@ -2707,7 +2733,7 @@ }, { "cell_type": "markdown", - "id": "d4184eee", + "id": "df905d9f", "metadata": { "editable": true }, @@ -2725,7 +2751,7 @@ }, { "cell_type": "markdown", - "id": "82d314da", + "id": "233e93d6", "metadata": { "editable": true }, @@ -2741,7 +2767,7 @@ }, { "cell_type": "markdown", - "id": "abc6640b", + "id": "0034168c", "metadata": { "editable": true }, @@ -2753,7 +2779,7 @@ }, { "cell_type": "markdown", - "id": "bb7d1f89", + "id": "cd701f7d", "metadata": { "editable": true }, @@ -2767,109 +2793,7 @@ }, { "cell_type": "markdown", - "id": "442b1b5e", - "metadata": { - "editable": true - }, - "source": [ - "## Setting up the back-propagation algorithm\n", - "\n", - "Let us write this out in the form of an algorithm.\n", - "\n", - "First, we set up the input data $\\boldsymbol{x}$ and the activations\n", - "$\\boldsymbol{z}_1$ of the input layer and compute the activation function and\n", - "the pertinent outputs $\\boldsymbol{a}^1$.\n", - "\n", - "Secondly, we perform then the feed forward till we reach the output\n", - "layer and compute all $\\boldsymbol{z}_l$ of the input layer and compute the\n", - "activation function and the pertinent outputs $\\boldsymbol{a}^l$ for\n", - "$l=2,3,\\dots,L$.\n", - "\n", - "Thereafter we compute the ouput error $\\boldsymbol{\\delta}^L$ by computing all" - ] - }, - { - "cell_type": "markdown", - "id": "0005e340", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\delta_j^L = f'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "2b27751b", - "metadata": { - "editable": true - }, - "source": [ - "Then we compute the back propagate error for each $l=L-1,L-2,\\dots,2$ as" - ] - }, - { - "cell_type": "markdown", - "id": "dd544b4b", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l).\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "57a7d213", - "metadata": { - "editable": true - }, - "source": [ - "Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,2$ and update the weights and biases according to the rules" - ] - }, - { - "cell_type": "markdown", - "id": "932510f4", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "w_{ij}^l\\leftarrow = w_{ij}^l- \\eta \\delta_j^la_i^{l-1},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "bf80ed99", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "f682bf76", - "metadata": { - "editable": true - }, - "source": [ - "The parameter $\\eta$ is the learning parameter discussed in connection with the gradient descent methods.\n", - "Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training." - ] - }, - { - "cell_type": "markdown", - "id": "ad2b26a4", + "id": "84048fb0", "metadata": { "editable": true }, @@ -2897,7 +2821,7 @@ }, { "cell_type": "markdown", - "id": "a08331b2", + "id": "61ea03ed", "metadata": { "editable": true }, @@ -2909,7 +2833,7 @@ }, { "cell_type": "markdown", - "id": "9ebfde52", + "id": "abd0c817", "metadata": { "editable": true }, @@ -2919,7 +2843,7 @@ }, { "cell_type": "markdown", - "id": "531dfa25", + "id": "2e0379cc", "metadata": { "editable": true }, @@ -2931,7 +2855,7 @@ }, { "cell_type": "markdown", - "id": "8941d38c", + "id": "59290cf7", "metadata": { "editable": true }, @@ -2942,7 +2866,7 @@ }, { "cell_type": "markdown", - "id": "7ae7e222", + "id": "6bc256eb", "metadata": { "editable": true }, @@ -2954,7 +2878,7 @@ }, { "cell_type": "markdown", - "id": "cbb157cc", + "id": "831eee08", "metadata": { "editable": true }, @@ -2967,7 +2891,7 @@ }, { "cell_type": "markdown", - "id": "0178770c", + "id": "ba46bcc0", "metadata": { "editable": true }, @@ -2992,7 +2916,7 @@ }, { "cell_type": "markdown", - "id": "cb69f3df", + "id": "a475ed33", "metadata": { "editable": true }, @@ -3005,7 +2929,7 @@ }, { "cell_type": "markdown", - "id": "11aa1335", + "id": "378895ce", "metadata": { "editable": true }, @@ -3017,7 +2941,7 @@ }, { "cell_type": "markdown", - "id": "a225e57a", + "id": "268989a6", "metadata": { "editable": true }, @@ -3029,7 +2953,7 @@ }, { "cell_type": "markdown", - "id": "569dd3f2", + "id": "1ac5dfe0", "metadata": { "editable": true }, @@ -3039,7 +2963,7 @@ }, { "cell_type": "markdown", - "id": "f8e84cae", + "id": "4d3d6bb7", "metadata": { "editable": true }, @@ -3051,7 +2975,7 @@ }, { "cell_type": "markdown", - "id": "1dd303e8", + "id": "bc4e4a53", "metadata": { "editable": true }, @@ -3063,7 +2987,7 @@ }, { "cell_type": "markdown", - "id": "33494270", + "id": "4cc000a1", "metadata": { "editable": true }, @@ -3075,7 +2999,7 @@ }, { "cell_type": "markdown", - "id": "01bd6a05", + "id": "7f5ea691", "metadata": { "editable": true }, @@ -3087,7 +3011,7 @@ }, { "cell_type": "markdown", - "id": "60f16a47", + "id": "a7f4fc8e", "metadata": { "editable": true }, @@ -3097,7 +3021,7 @@ }, { "cell_type": "markdown", - "id": "ab0c42b3", + "id": "acdecb39", "metadata": { "editable": true }, @@ -3109,7 +3033,7 @@ }, { "cell_type": "markdown", - "id": "dcfb2e28", + "id": "b0c3e8c7", "metadata": { "editable": true }, @@ -3119,7 +3043,7 @@ }, { "cell_type": "markdown", - "id": "5076b392", + "id": "6e130deb", "metadata": { "editable": true }, @@ -3131,7 +3055,7 @@ }, { "cell_type": "markdown", - "id": "9d6001f4", + "id": "23e83ce8", "metadata": { "editable": true }, @@ -3144,7 +3068,7 @@ }, { "cell_type": "markdown", - "id": "d0245c18", + "id": "67c77893", "metadata": { "editable": true }, @@ -3156,7 +3080,7 @@ }, { "cell_type": "markdown", - "id": "030ac014", + "id": "36ede636", "metadata": { "editable": true }, @@ -3166,7 +3090,7 @@ }, { "cell_type": "markdown", - "id": "bdc01c83", + "id": "4008b26c", "metadata": { "editable": true }, @@ -3178,7 +3102,7 @@ }, { "cell_type": "markdown", - "id": "1a77b25c", + "id": "258ae1d4", "metadata": { "editable": true }, @@ -3189,7 +3113,7 @@ }, { "cell_type": "markdown", - "id": "1f169ed1", + "id": "b9c648be", "metadata": { "editable": true }, @@ -3201,7 +3125,7 @@ }, { "cell_type": "markdown", - "id": "6bc460e2", + "id": "615f1fce", "metadata": { "editable": true }, @@ -3211,7 +3135,7 @@ }, { "cell_type": "markdown", - "id": "fe1f92c0", + "id": "620b34cb", "metadata": { "editable": true }, @@ -3223,7 +3147,7 @@ }, { "cell_type": "markdown", - "id": "78cf6843", + "id": "7d09dc1a", "metadata": { "editable": true }, @@ -3233,7 +3157,7 @@ }, { "cell_type": "markdown", - "id": "81ed1ccc", + "id": "8589b8ee", "metadata": { "editable": true }, @@ -3244,7 +3168,7 @@ }, { "cell_type": "markdown", - "id": "2a16cf53", + "id": "63160327", "metadata": { "editable": true }, @@ -3257,7 +3181,7 @@ }, { "cell_type": "markdown", - "id": "21988141", + "id": "98614055", "metadata": { "editable": true }, @@ -3267,7 +3191,7 @@ }, { "cell_type": "markdown", - "id": "b37cc0cd", + "id": "37435d7c", "metadata": { "editable": true }, @@ -3279,7 +3203,7 @@ }, { "cell_type": "markdown", - "id": "259618c5", + "id": "64a4d79f", "metadata": { "editable": true }, @@ -3289,7 +3213,7 @@ }, { "cell_type": "markdown", - "id": "af5a331e", + "id": "7e3891af", "metadata": { "editable": true }, @@ -3301,7 +3225,7 @@ }, { "cell_type": "markdown", - "id": "208ab286", + "id": "7205e781", "metadata": { "editable": true }, @@ -3311,7 +3235,7 @@ }, { "cell_type": "markdown", - "id": "8f30fbc3", + "id": "a5645092", "metadata": { "editable": true }, @@ -3335,7 +3259,7 @@ }, { "cell_type": "markdown", - "id": "09538ef9", + "id": "09e9b12d", "metadata": { "editable": true }, @@ -3385,7 +3309,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "ced37e42", + "id": "84fd1480", "metadata": { "collapsed": false, "editable": true @@ -3438,7 +3362,7 @@ }, { "cell_type": "markdown", - "id": "ebabb274", + "id": "e671e2a8", "metadata": { "editable": true }, @@ -3459,7 +3383,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "2c87b9f9", + "id": "5d1c4624", "metadata": { "collapsed": false, "editable": true @@ -3497,7 +3421,7 @@ }, { "cell_type": "markdown", - "id": "0c186fec", + "id": "b397d4ee", "metadata": { "editable": true }, @@ -3541,7 +3465,7 @@ }, { "cell_type": "markdown", - "id": "2cbece98", + "id": "1fce534f", "metadata": { "editable": true }, @@ -3581,7 +3505,7 @@ }, { "cell_type": "markdown", - "id": "442935af", + "id": "9c50a158", "metadata": { "editable": true }, @@ -3602,7 +3526,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "07fbb18c", + "id": "4daeba9a", "metadata": { "collapsed": false, "editable": true @@ -3628,7 +3552,7 @@ }, { "cell_type": "markdown", - "id": "6a3ea326", + "id": "f26a835c", "metadata": { "editable": true }, @@ -3656,7 +3580,7 @@ }, { "cell_type": "markdown", - "id": "525624c4", + "id": "09f1187b", "metadata": { "editable": true }, @@ -3693,7 +3617,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "32d0a43f", + "id": "35d71f0e", "metadata": { "collapsed": false, "editable": true @@ -3739,7 +3663,7 @@ }, { "cell_type": "markdown", - "id": "da2ede88", + "id": "e318575f", "metadata": { "editable": true }, @@ -3770,7 +3694,7 @@ }, { "cell_type": "markdown", - "id": "069a250d", + "id": "788f45d2", "metadata": { "editable": true }, @@ -3808,7 +3732,7 @@ }, { "cell_type": "markdown", - "id": "29ab6ee7", + "id": "599a7b8f", "metadata": { "editable": true }, @@ -3842,7 +3766,7 @@ }, { "cell_type": "markdown", - "id": "b7c4a178", + "id": "cc694849", "metadata": { "editable": true }, @@ -3883,7 +3807,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "4142969e", + "id": "e3607c66", "metadata": { "collapsed": false, "editable": true @@ -3962,7 +3886,7 @@ }, { "cell_type": "markdown", - "id": "4d79c3b3", + "id": "0a70cdcf", "metadata": { "editable": true }, @@ -3983,7 +3907,7 @@ }, { "cell_type": "markdown", - "id": "58012d93", + "id": "b0600212", "metadata": { "editable": true }, @@ -3997,7 +3921,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "ea4b7741", + "id": "4e0c1326", "metadata": { "collapsed": false, "editable": true @@ -4107,7 +4031,7 @@ }, { "cell_type": "markdown", - "id": "04860b2c", + "id": "125f8eca", "metadata": { "editable": true }, @@ -4126,7 +4050,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "b52d4df3", + "id": "cef0e788", "metadata": { "collapsed": false, "editable": true @@ -4153,7 +4077,7 @@ }, { "cell_type": "markdown", - "id": "6dce68ff", + "id": "b12d6f86", "metadata": { "editable": true }, @@ -4167,7 +4091,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "65caeb78", + "id": "974faa4e", "metadata": { "collapsed": false, "editable": true @@ -4198,7 +4122,7 @@ }, { "cell_type": "markdown", - "id": "0c44c38e", + "id": "3bb19122", "metadata": { "editable": true }, @@ -4209,7 +4133,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "fbdc64f1", + "id": "baeb1ea3", "metadata": { "collapsed": false, "editable": true @@ -4253,7 +4177,7 @@ }, { "cell_type": "markdown", - "id": "e07e69e3", + "id": "d75dad8e", "metadata": { "editable": true }, @@ -4276,7 +4200,7 @@ { "cell_type": "code", "execution_count": 12, - "id": "f48eaa77", + "id": "d55fae28", "metadata": { "collapsed": false, "editable": true @@ -4303,7 +4227,7 @@ }, { "cell_type": "markdown", - "id": "ac3c7bcb", + "id": "276badf8", "metadata": { "editable": true }, @@ -4314,7 +4238,7 @@ { "cell_type": "code", "execution_count": 13, - "id": "2d81262e", + "id": "7d6c714e", "metadata": { "collapsed": false, "editable": true @@ -4359,7 +4283,7 @@ }, { "cell_type": "markdown", - "id": "8f5f8f4b", + "id": "0d45b429", "metadata": { "editable": true }, @@ -4377,7 +4301,7 @@ }, { "cell_type": "markdown", - "id": "6ec638b9", + "id": "67aec670", "metadata": { "editable": true }, @@ -4412,7 +4336,7 @@ { "cell_type": "code", "execution_count": 14, - "id": "6443ca72", + "id": "0ab38a83", "metadata": { "collapsed": false, "editable": true @@ -4424,7 +4348,7 @@ }, { "cell_type": "markdown", - "id": "a8918d58", + "id": "8f53f5b9", "metadata": { "editable": true }, @@ -4436,7 +4360,7 @@ { "cell_type": "code", "execution_count": 15, - "id": "20e9a0f5", + "id": "5190c7ff", "metadata": { "collapsed": false, "editable": true @@ -4449,7 +4373,7 @@ }, { "cell_type": "markdown", - "id": "1381a34b", + "id": "676ff9d7", "metadata": { "editable": true }, @@ -4460,7 +4384,7 @@ { "cell_type": "code", "execution_count": 16, - "id": "80e92af9", + "id": "479149f7", "metadata": { "collapsed": false, "editable": true @@ -4473,7 +4397,7 @@ }, { "cell_type": "markdown", - "id": "074ac69f", + "id": "62d1b789", "metadata": { "editable": true }, @@ -4488,7 +4412,7 @@ { "cell_type": "code", "execution_count": 17, - "id": "0d710bd3", + "id": "4a11035a", "metadata": { "collapsed": false, "editable": true @@ -4500,7 +4424,7 @@ }, { "cell_type": "markdown", - "id": "7adbcab8", + "id": "abf44b70", "metadata": { "editable": true }, @@ -4512,7 +4436,7 @@ }, { "cell_type": "markdown", - "id": "e17c7253", + "id": "3f163559", "metadata": { "editable": true }, @@ -4525,7 +4449,7 @@ { "cell_type": "code", "execution_count": 18, - "id": "5ae741f5", + "id": "f7418c1e", "metadata": { "collapsed": false, "editable": true @@ -4580,7 +4504,7 @@ { "cell_type": "code", "execution_count": 19, - "id": "4f7ef6bb", + "id": "49ec0156", "metadata": { "collapsed": false, "editable": true @@ -4609,7 +4533,7 @@ { "cell_type": "code", "execution_count": 20, - "id": "7b74e049", + "id": "302ad127", "metadata": { "collapsed": false, "editable": true @@ -4639,7 +4563,7 @@ { "cell_type": "code", "execution_count": 21, - "id": "eaa25983", + "id": "436a3e0a", "metadata": { "collapsed": false, "editable": true @@ -4666,7 +4590,7 @@ { "cell_type": "code", "execution_count": 22, - "id": "949eca1f", + "id": "a26e83e0", "metadata": { "collapsed": false, "editable": true @@ -4708,7 +4632,7 @@ }, { "cell_type": "markdown", - "id": "bb8b8ac1", + "id": "8d69b494", "metadata": { "editable": true }, @@ -4719,7 +4643,7 @@ { "cell_type": "code", "execution_count": 23, - "id": "9d5ebbb2", + "id": "cad16bbe", "metadata": { "collapsed": false, "editable": true @@ -4896,7 +4820,7 @@ }, { "cell_type": "markdown", - "id": "f1baeb0b", + "id": "61624838", "metadata": { "editable": true }, @@ -4915,7 +4839,7 @@ }, { "cell_type": "markdown", - "id": "6549fa13", + "id": "f825f2da", "metadata": { "editable": true }, @@ -4937,7 +4861,7 @@ { "cell_type": "code", "execution_count": 24, - "id": "21062ebc", + "id": "409b4250", "metadata": { "collapsed": false, "editable": true @@ -5078,7 +5002,7 @@ }, { "cell_type": "markdown", - "id": "f965b277", + "id": "c6830d86", "metadata": { "editable": true }, @@ -5094,7 +5018,7 @@ { "cell_type": "code", "execution_count": 25, - "id": "35ab935b", + "id": "041fc0bf", "metadata": { "collapsed": false, "editable": true @@ -5107,7 +5031,7 @@ }, { "cell_type": "markdown", - "id": "b26972b4", + "id": "0e18ef84", "metadata": { "editable": true }, @@ -5119,7 +5043,7 @@ { "cell_type": "code", "execution_count": 26, - "id": "3df9caac", + "id": "8be0e7fd", "metadata": { "collapsed": false, "editable": true @@ -5141,7 +5065,7 @@ }, { "cell_type": "markdown", - "id": "1c07f6da", + "id": "2d2bc7a5", "metadata": { "editable": true }, @@ -5157,7 +5081,7 @@ { "cell_type": "code", "execution_count": 27, - "id": "743aa0bd", + "id": "f5cb107e", "metadata": { "collapsed": false, "editable": true @@ -5195,7 +5119,7 @@ }, { "cell_type": "markdown", - "id": "c78bdc16", + "id": "beb4f622", "metadata": { "editable": true }, @@ -5208,7 +5132,7 @@ { "cell_type": "code", "execution_count": 28, - "id": "4ec3a28c", + "id": "1b508839", "metadata": { "collapsed": false, "editable": true @@ -5229,7 +5153,7 @@ }, { "cell_type": "markdown", - "id": "5c924e11", + "id": "afb2e0af", "metadata": { "editable": true }, @@ -5245,7 +5169,7 @@ { "cell_type": "code", "execution_count": 29, - "id": "87402ee9", + "id": "b96d6c89", "metadata": { "collapsed": false, "editable": true @@ -5303,7 +5227,7 @@ }, { "cell_type": "markdown", - "id": "45ddd0bc", + "id": "0be588f8", "metadata": { "editable": true }, @@ -5318,7 +5242,7 @@ { "cell_type": "code", "execution_count": 30, - "id": "77034679", + "id": "49b8531e", "metadata": { "collapsed": false, "editable": true @@ -5339,7 +5263,7 @@ }, { "cell_type": "markdown", - "id": "9fd8140f", + "id": "874d306a", "metadata": { "editable": true }, @@ -5363,7 +5287,7 @@ { "cell_type": "code", "execution_count": 31, - "id": "7540d755", + "id": "c25e9955", "metadata": { "collapsed": false, "editable": true @@ -5835,7 +5759,7 @@ }, { "cell_type": "markdown", - "id": "5ee996ef", + "id": "86746540", "metadata": { "editable": true }, @@ -5847,7 +5771,7 @@ { "cell_type": "code", "execution_count": 32, - "id": "183b180b", + "id": "6f232f0a", "metadata": { "collapsed": false, "editable": true @@ -5891,7 +5815,7 @@ }, { "cell_type": "markdown", - "id": "21a48508", + "id": "7227f21d", "metadata": { "editable": true }, @@ -5907,7 +5831,7 @@ { "cell_type": "code", "execution_count": 33, - "id": "37eba90c", + "id": "944ba89b", "metadata": { "collapsed": false, "editable": true @@ -5922,7 +5846,7 @@ }, { "cell_type": "markdown", - "id": "58a6b726", + "id": "3cef110a", "metadata": { "editable": true }, @@ -5933,7 +5857,7 @@ { "cell_type": "code", "execution_count": 34, - "id": "046d7076", + "id": "8eb39d5e", "metadata": { "collapsed": false, "editable": true @@ -5948,7 +5872,7 @@ }, { "cell_type": "markdown", - "id": "981e95ce", + "id": "4c0ab092", "metadata": { "editable": true }, @@ -5964,7 +5888,7 @@ { "cell_type": "code", "execution_count": 35, - "id": "6da23296", + "id": "af77a32b", "metadata": { "collapsed": false, "editable": true @@ -5978,7 +5902,7 @@ }, { "cell_type": "markdown", - "id": "ba0fea41", + "id": "dfdb722f", "metadata": { "editable": true }, @@ -5993,7 +5917,7 @@ { "cell_type": "code", "execution_count": 36, - "id": "1bd5c3e5", + "id": "d740dbad", "metadata": { "collapsed": false, "editable": true @@ -6019,7 +5943,7 @@ { "cell_type": "code", "execution_count": 37, - "id": "9923ffbf", + "id": "bfb7c28b", "metadata": { "collapsed": false, "editable": true @@ -6034,7 +5958,7 @@ }, { "cell_type": "markdown", - "id": "95431510", + "id": "17a8dc93", "metadata": { "editable": true }, @@ -6045,7 +5969,7 @@ { "cell_type": "code", "execution_count": 38, - "id": "9d71cef9", + "id": "7efc0180", "metadata": { "collapsed": false, "editable": true @@ -6060,7 +5984,7 @@ }, { "cell_type": "markdown", - "id": "acfee190", + "id": "8255133c", "metadata": { "editable": true }, @@ -6071,7 +5995,7 @@ { "cell_type": "code", "execution_count": 39, - "id": "c834fb60", + "id": "4fa47196", "metadata": { "collapsed": false, "editable": true @@ -6091,7 +6015,7 @@ { "cell_type": "code", "execution_count": 40, - "id": "0c557301", + "id": "b7b5ed9f", "metadata": { "collapsed": false, "editable": true @@ -6106,7 +6030,7 @@ }, { "cell_type": "markdown", - "id": "750490a3", + "id": "bc0fc41e", "metadata": { "editable": true }, @@ -6121,7 +6045,7 @@ { "cell_type": "code", "execution_count": 41, - "id": "5318d06b", + "id": "4ccf32f6", "metadata": { "collapsed": false, "editable": true @@ -6158,7 +6082,7 @@ }, { "cell_type": "markdown", - "id": "ff519831", + "id": "382301fa", "metadata": { "editable": true }, @@ -6171,7 +6095,7 @@ { "cell_type": "code", "execution_count": 42, - "id": "57e3fb33", + "id": "0feb8f2a", "metadata": { "collapsed": false, "editable": true @@ -6194,7 +6118,7 @@ }, { "cell_type": "markdown", - "id": "e4c08bad", + "id": "3f48285b", "metadata": { "editable": true }, diff --git a/doc/pub/week42/html/._week42-bs016.html b/doc/pub/week42/html/._week42-bs016.html index 4a9ec98ed..21cee57db 100644 --- a/doc/pub/week42/html/._week42-bs016.html +++ b/doc/pub/week42/html/._week42-bs016.html @@ -473,15 +473,15 @@ layer with two hidden nodes and one output layer with one output node/neuron onl

We need to define the following parameters and variables with the input layer (layer \( (0) \)) -where we label the nodes \( x_0 \) and \( x_1 \) +where we label the nodes \( x_1 \) and \( x_2 \)

$$ -x_0 = a_0^{(0)} \wedge x_1 = a_1^{(0)}. +x_1 = a_1^{(0)} \wedge x_2 = a_2^{(0)}. $$ -

The hidden layer (layer \( (1) \)) has nodes which yield the outputs \( a_0^{(1)} \) and \( a_1^{(1)} \)) with weight \( \boldsymbol{w} \) and bias \( \boldsymbol{b} \) parameters

+

The hidden layer (layer \( (1) \)) has nodes which yield the outputs \( a_1^{(1)} \) and \( a_2^{(1)} \)) with weight \( \boldsymbol{w} \) and bias \( \boldsymbol{b} \) parameters

$$ -w_{ij}^{(1)}=\left\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)}\right\} \wedge b^{(1)}=\left\{b_0^{(1)},b_1^{(1)}\right\}. +w_{ij}^{(1)}=\left\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)}\right\} \wedge b^{(1)}=\left\{b_1^{(1)},b_2^{(1)}\right\}. $$ diff --git a/doc/pub/week42/html/week42-reveal.html b/doc/pub/week42/html/week42-reveal.html index 88db3fbfa..0e2e44853 100644 --- a/doc/pub/week42/html/week42-reveal.html +++ b/doc/pub/week42/html/week42-reveal.html @@ -584,18 +584,18 @@ layer with two hidden nodes and one output layer with one output node/neuron onl

We need to define the following parameters and variables with the input layer (layer \( (0) \)) -where we label the nodes \( x_0 \) and \( x_1 \) +where we label the nodes \( x_1 \) and \( x_2 \)

 
$$ -x_0 = a_0^{(0)} \wedge x_1 = a_1^{(0)}. +x_1 = a_1^{(0)} \wedge x_2 = a_2^{(0)}. $$

 
-

The hidden layer (layer \( (1) \)) has nodes which yield the outputs \( a_0^{(1)} \) and \( a_1^{(1)} \)) with weight \( \boldsymbol{w} \) and bias \( \boldsymbol{b} \) parameters

+

The hidden layer (layer \( (1) \)) has nodes which yield the outputs \( a_1^{(1)} \) and \( a_2^{(1)} \)) with weight \( \boldsymbol{w} \) and bias \( \boldsymbol{b} \) parameters

 
$$ -w_{ij}^{(1)}=\left\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)}\right\} \wedge b^{(1)}=\left\{b_0^{(1)},b_1^{(1)}\right\}. +w_{ij}^{(1)}=\left\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)}\right\} \wedge b^{(1)}=\left\{b_1^{(1)},b_2^{(1)}\right\}. $$

 
diff --git a/doc/pub/week42/html/week42-solarized.html b/doc/pub/week42/html/week42-solarized.html index 09a6aa095..ee3e75686 100644 --- a/doc/pub/week42/html/week42-solarized.html +++ b/doc/pub/week42/html/week42-solarized.html @@ -730,15 +730,15 @@ layer with two hidden nodes and one output layer with one output node/neuron onl

We need to define the following parameters and variables with the input layer (layer \( (0) \)) -where we label the nodes \( x_0 \) and \( x_1 \) +where we label the nodes \( x_1 \) and \( x_2 \)

$$ -x_0 = a_0^{(0)} \wedge x_1 = a_1^{(0)}. +x_1 = a_1^{(0)} \wedge x_2 = a_2^{(0)}. $$ -

The hidden layer (layer \( (1) \)) has nodes which yield the outputs \( a_0^{(1)} \) and \( a_1^{(1)} \)) with weight \( \boldsymbol{w} \) and bias \( \boldsymbol{b} \) parameters

+

The hidden layer (layer \( (1) \)) has nodes which yield the outputs \( a_1^{(1)} \) and \( a_2^{(1)} \)) with weight \( \boldsymbol{w} \) and bias \( \boldsymbol{b} \) parameters

$$ -w_{ij}^{(1)}=\left\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)}\right\} \wedge b^{(1)}=\left\{b_0^{(1)},b_1^{(1)}\right\}. +w_{ij}^{(1)}=\left\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)}\right\} \wedge b^{(1)}=\left\{b_1^{(1)},b_2^{(1)}\right\}. $$ diff --git a/doc/pub/week42/html/week42.html b/doc/pub/week42/html/week42.html index 10fd5c21b..53d75e1db 100644 --- a/doc/pub/week42/html/week42.html +++ b/doc/pub/week42/html/week42.html @@ -807,15 +807,15 @@ layer with two hidden nodes and one output layer with one output node/neuron onl

We need to define the following parameters and variables with the input layer (layer \( (0) \)) -where we label the nodes \( x_0 \) and \( x_1 \) +where we label the nodes \( x_1 \) and \( x_2 \)

$$ -x_0 = a_0^{(0)} \wedge x_1 = a_1^{(0)}. +x_1 = a_1^{(0)} \wedge x_2 = a_2^{(0)}. $$ -

The hidden layer (layer \( (1) \)) has nodes which yield the outputs \( a_0^{(1)} \) and \( a_1^{(1)} \)) with weight \( \boldsymbol{w} \) and bias \( \boldsymbol{b} \) parameters

+

The hidden layer (layer \( (1) \)) has nodes which yield the outputs \( a_1^{(1)} \) and \( a_2^{(1)} \)) with weight \( \boldsymbol{w} \) and bias \( \boldsymbol{b} \) parameters

$$ -w_{ij}^{(1)}=\left\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)}\right\} \wedge b^{(1)}=\left\{b_0^{(1)},b_1^{(1)}\right\}. +w_{ij}^{(1)}=\left\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)}\right\} \wedge b^{(1)}=\left\{b_1^{(1)},b_2^{(1)}\right\}. $$ diff --git a/doc/pub/week42/ipynb/ipynb-week42-src.tar.gz b/doc/pub/week42/ipynb/ipynb-week42-src.tar.gz index 126c044289633198d9443b1c449bf129eedb130f..3c9edfc0b65949844d069689387d1184136c0b37 100644 GIT binary patch delta 38 ucmZ2;S9Z-^SvL7@4hF}Wjcl!KjIC@;t!&J#Y%Hy8tgUQpTiMu`