lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2024-10-13 17:18:30 +02:00
parent 01d438f071
commit 33477fd296
41 changed files with 30024 additions and 243 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/exercisesweek42.html
+825
View File
@@ -0,0 +1,825 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Exercises week 42 &#8212; Applied Data Analysis and Machine Learning</title>
<link href="_static/css/theme.css" rel="stylesheet">
<link href="_static/css/index.ff1ffe594081f20da1ef19478df9384b.css" rel="stylesheet">
<link rel="stylesheet"
href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
<link rel="stylesheet" type="text/css" href="_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-book-theme.css?digest=c3fdc42140077d1ad13ad2f1588a4309" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-main.c949a650a448cc0ae9fd3441c0e17fb0.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-variables.06eb56fa6e07937060861dad626602ad.css" />
<link rel="preload" as="script" href="_static/js/index.be7d3bbb2ef33a8344ce.js">
<script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/clipboard.min.js"></script>
<script src="_static/copybutton.js"></script>
<script>let toggleHintShow = 'Click to show';</script>
<script>let toggleHintHide = 'Click to hide';</script>
<script>let toggleOpenOnPrint = 'true';</script>
<script src="_static/togglebutton.js"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown, .tag_hide_input div.cell_input, .tag_hide-input div.cell_input, .tag_hide_output div.cell_output, .tag_hide-output div.cell_output, .tag_hide_cell.cell, .tag_hide-cell.cell';</script>
<script src="_static/sphinx-book-theme.d59cb220de22ca1c485ebbdc042f0030.js"></script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"
const thebe_selector = ".thebe,.cell"
const thebe_selector_input = "pre"
const thebe_selector_output = ".output, .cell_output"
</script>
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Week 42 Constructing a Neural Network code with examples" href="week42.html" />
<link rel="prev" title="Week 41 Neural networks and constructing a neural network code" href="week41.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="None">
<!-- Google Analytics -->
</head>
<body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
<div class="container-fluid" id="banner"></div>
<div class="container-xl">
<div class="row">
<div class="col-12 col-md-3 bd-sidebar site-navigation show" id="site-navigation">
<div class="navbar-brand-box">
<a class="navbar-brand text-wrap" href="index.html">
<!-- `logo` is deprecated in Sphinx 4.0, so remove this when we stop supporting 3 -->
<img src="_static/logo.png" class="logo" alt="logo">
<h1 class="site-logo" id="site-title">Applied Data Analysis and Machine Learning</h1>
</a>
</div><form class="bd-search d-flex align-items-center" action="search.html" method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search this book..." aria-label="Search this book..." autocomplete="off" >
</form><nav class="bd-links" id="bd-docs-nav" aria-label="Main">
<div class="bd-toc-item active">
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="intro.html">
Applied Data Analysis and Machine Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
About the course
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="schedule.html">
Teaching schedule with links to material
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="teachers.html">
Teachers and Grading
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="textbooks.html">
Textbooks
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Review of Statistics with Resampling Techniques and Linear Algebra
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="statistics.html">
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="linalg.html">
2. Linear Algebra, Handling of Arrays and more Python Features
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
From Regression to Support Vector Machines
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter1.html">
3. Linear Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html">
4. Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
5. Resampling Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
6. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapteroptimization.html">
7. Optimization, the central part of any Machine Learning algortithm
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
8. Support Vector Machines, overarching aims
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Decision Trees, Ensemble Methods and Boosting
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
9. Decision trees, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Dimensionality Reduction
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="clustering.html">
12. Clustering and Unsupervised Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Deep Learning Methods
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter9.html">
13. Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter10.html">
14. Building a Feed Forward Neural Network
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter11.html">
15. Solving Differential Equations with Deep Learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter12.html">
16. Convolutional Neural Networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter13.html">
17. Recurrent neural networks: Overarching view
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Weekly material, notes and exercises
</span>
</p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek34.html">
Exercises week 34
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week34.html">
Week 34: Introduction to the course, Logistics and Practicalities
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek35.html">
Exercises week 35
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week35.html">
Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek36.html">
Exercises week 36
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week36.html">
Week 36: Linear Regression and Statistical interpretations
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek37.html">
Exercises week 37
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week37.html">
Week 37: Statistical interpretations and Resampling Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek38.html">
Exercises week 38
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week38.html">
Week 38: Logistic Regression and Optimization
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek39.html">
Exercises week 39
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week39.html">
Week 39: Optimization and Gradient Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week40.html">
Week 40: Gradient descent methods (continued) and start Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek41.html">
Exercises week 41
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week41.html">
Week 41 Neural networks and constructing a neural network code
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
Exercises week 42
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week42.html">
Week 42 Constructing a Neural Network code with examples
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Projects
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="project1.html">
Project 1 on Machine Learning, deadline October 7 (midnight), 2024
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="project2.html">
Project 2 on Machine Learning, deadline November 4 (Midnight)
</a>
</li>
</ul>
</div>
</nav> <!-- To handle the deprecated key -->
<div class="navbar_extra_footer">
Powered by <a href="https://jupyterbook.org">Jupyter Book</a>
</div>
</div>
<main class="col py-md-3 pl-md-4 bd-content overflow-auto" role="main">
<div class="topbar container-xl fixed-top">
<div class="topbar-contents row">
<div class="col-12 col-md-3 bd-topbar-whitespace site-navigation show"></div>
<div class="col pl-md-4 topbar-main">
<button id="navbar-toggler" class="navbar-toggler ml-0" type="button" data-toggle="collapse"
data-toggle="tooltip" data-placement="bottom" data-target=".site-navigation" aria-controls="navbar-menu"
aria-expanded="true" aria-label="Toggle navigation" aria-controls="site-navigation"
title="Toggle navigation" data-toggle="tooltip" data-placement="left">
<i class="fas fa-bars"></i>
<i class="fas fa-arrow-left"></i>
<i class="fas fa-arrow-up"></i>
</button>
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn" aria-label="Download this page"><i
class="fas fa-download"></i></button>
<div class="dropdown-buttons">
<!-- ipynb file if we had a myst markdown file -->
<!-- Download raw file -->
<a class="dropdown-buttons" href="_sources/exercisesweek42.ipynb"><button type="button"
class="btn btn-secondary topbarbtn" title="Download source file" data-toggle="tooltip"
data-placement="left">.ipynb</button></a>
<!-- Download PDF via print -->
<button type="button" id="download-print" class="btn btn-secondary topbarbtn" title="Print to PDF"
onclick="printPdf(this)" data-toggle="tooltip" data-placement="left">.pdf</button>
</div>
</div>
<!-- Source interaction buttons -->
<!-- Full screen (wrap in <a> to have style consistency -->
<a class="full-screen-button"><button type="button" class="btn btn-secondary topbarbtn" data-toggle="tooltip"
data-placement="bottom" onclick="toggleFullScreen()" aria-label="Fullscreen mode"
title="Fullscreen mode"><i
class="fas fa-expand"></i></button></a>
<!-- Launch buttons -->
</div>
<!-- Table of contents -->
<div class="d-none d-md-block col-md-2 bd-toc show noprint">
<div class="tocsection onthispage pt-5 pb-3">
<i class="fas fa-list"></i> Contents
</div>
<nav id="bd-toc-nav" aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#">
Exercises week 42
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#overarching-aims-of-the-exercises-this-week">
Overarching aims of the exercises this week
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#exercise-1">
Exercise 1
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#exercise-2">
Exercise 2
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#exercise-3">
Exercise 3
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#exercise-4">
Exercise 4
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#exercise-5-very-optional-and-very-hard">
Exercise 5 (Very optional and very hard :)
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div id="main-content" class="row">
<div class="col-12 col-md-9 pl-md-3 pr-md-0">
<!-- Table of contents that is only displayed when printing the page -->
<div id="jb-print-docs-body" class="onlyprint">
<h1>Exercises week 42</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
<div>
<h2> Contents </h2>
</div>
<nav aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#">
Exercises week 42
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#overarching-aims-of-the-exercises-this-week">
Overarching aims of the exercises this week
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#exercise-1">
Exercise 1
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#exercise-2">
Exercise 2
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#exercise-3">
Exercise 3
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#exercise-4">
Exercise 4
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#exercise-5-very-optional-and-very-hard">
Exercise 5 (Very optional and very hard :)
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div>
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
doconce format html exercisesweek41.do.txt -->
<!-- dom:TITLE: Exercises week 41 -->
<div class="tex2jax_ignore mathjax_ignore section" id="exercises-week-42">
<h1>Exercises week 42<a class="headerlink" href="#exercises-week-42" title="Permalink to this headline"></a></h1>
<p><strong>October 14-18, 2024</strong></p>
<p>Date: <strong>Deadline is Friday October 18 at midnight</strong></p>
</div>
<div class="tex2jax_ignore mathjax_ignore section" id="overarching-aims-of-the-exercises-this-week">
<h1>Overarching aims of the exercises this week<a class="headerlink" href="#overarching-aims-of-the-exercises-this-week" title="Permalink to this headline"></a></h1>
<p>The aim of the exercises this week is to get started with implementing a neural network. There are a lot of technical and finicky parts of implementing a neutal network, so take your time.</p>
<p>This week, you will implement only the feed-forward pass. Next week, you will implement backpropagation. We recommend that you do the exercises this week by editing and running this notebook file, as it includes several checks along the way that you have implemented the pieces of the feed-forward pass correctly. If you have trouble running a notebook, or importing pytorch, you can run this notebook in google colab instead: (LINK TO COLAB), though we recommend that you set up VSCode and your python environment to run code like this locally.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="tex2jax_ignore mathjax_ignore section" id="exercise-1">
<h1>Exercise 1<a class="headerlink" href="#exercise-1" title="Permalink to this headline"></a></h1>
<p>Complete the following parts to compute the activation of the first layer.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">2024</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">ReLU</span><span class="p">(</span><span class="n">z</span><span class="p">):</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">where</span><span class="p">(</span><span class="n">z</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">,</span> <span class="n">z</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="c1"># network input</span>
<span class="n">W1</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="c1"># first layer weights</span>
</pre></div>
</div>
</div>
</div>
<p><strong>a)</strong> Define the bias of the first layer, <code class="docutils literal notranslate"><span class="pre">b1</span></code>with the correct shape</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">b1</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p><strong>b)</strong> Compute the intermediary <code class="docutils literal notranslate"><span class="pre">z1</span></code> for the first layer</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">z1</span> <span class="o">=</span> <span class="n">W1</span> <span class="o">@</span> <span class="n">x</span> <span class="o">+</span> <span class="n">b1</span>
</pre></div>
</div>
</div>
</div>
<p><strong>c)</strong> Compute the activation <code class="docutils literal notranslate"><span class="pre">a1</span></code> for the first layer using the ReLU activation function defined earlier.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">a1</span> <span class="o">=</span> <span class="n">ReLU</span><span class="p">(</span><span class="n">z1</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>Confirm that you got the correct activation with the test below.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">sol1</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">0.60610368</span><span class="p">,</span> <span class="mf">4.0076268</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.56469864</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">allclose</span><span class="p">(</span><span class="n">a1</span><span class="p">,</span> <span class="n">sol1</span><span class="p">))</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>True
</pre></div>
</div>
</div>
</div>
</div>
<div class="tex2jax_ignore mathjax_ignore section" id="exercise-2">
<h1>Exercise 2<a class="headerlink" href="#exercise-2" title="Permalink to this headline"></a></h1>
<p>Compute the activation of the second layer with an output of length 8 and ReLU activation.</p>
<p><strong>a)</strong> Define the weight and bias of the second layer with the right shapes.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">W2</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">8</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span>
<span class="n">b2</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">8</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p><strong>b)</strong> Compute intermediary <code class="docutils literal notranslate"><span class="pre">z2</span></code> and activation <code class="docutils literal notranslate"><span class="pre">a2</span></code> for the second layer.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">z2</span> <span class="o">=</span> <span class="n">W2</span> <span class="o">@</span> <span class="n">a1</span>
<span class="n">a2</span> <span class="o">=</span> <span class="n">ReLU</span><span class="p">(</span><span class="n">z2</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>Confirm that you got the correct activation shape with the test below.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">a2</span><span class="o">.</span><span class="n">shape</span> <span class="o">==</span> <span class="p">(</span><span class="mi">8</span><span class="p">,))</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>True
</pre></div>
</div>
</div>
</div>
</div>
<div class="tex2jax_ignore mathjax_ignore section" id="exercise-3">
<h1>Exercise 3<a class="headerlink" href="#exercise-3" title="Permalink to this headline"></a></h1>
<p>We often want our neural networks to have many layers of varying sizes. To avoid writing very long and error-prone code where we explicitly define and evaluate each layer we should keep all our layers in a single variable which is easy to create and use.</p>
<p><strong>a)</strong> Complete the function below so that it returns a list <code class="docutils literal notranslate"><span class="pre">layers</span></code> of weight and bias tuples <code class="docutils literal notranslate"><span class="pre">(W,</span> <span class="pre">b)</span></code> for each layer, in order, with the correct shapes that we can use later as our network parameters.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">create_layers</span><span class="p">(</span><span class="n">network_input_size</span><span class="p">,</span> <span class="n">output_sizes</span><span class="p">):</span>
<span class="n">layers</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">i_size</span> <span class="o">=</span> <span class="n">network_input_size</span>
<span class="k">for</span> <span class="n">output_size</span> <span class="ow">in</span> <span class="n">output_sizes</span><span class="p">:</span>
<span class="n">W</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">output_size</span><span class="p">,</span> <span class="n">i_size</span><span class="p">)</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">output_size</span><span class="p">)</span>
<span class="n">layers</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">W</span><span class="p">,</span> <span class="n">b</span><span class="p">))</span>
<span class="n">i_size</span> <span class="o">=</span> <span class="n">output_size</span>
<span class="k">return</span> <span class="n">layers</span>
</pre></div>
</div>
</div>
</div>
<p><strong>b)</strong> Comple the function below so that it evaluates the intermediate <code class="docutils literal notranslate"><span class="pre">z</span></code> and activation <code class="docutils literal notranslate"><span class="pre">a</span></code> for each layer, and returns the final activation <code class="docutils literal notranslate"><span class="pre">a</span></code>. This is the complete feed-forward pass, a full neural network!</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">feed_forward</span><span class="p">(</span><span class="n">layers</span><span class="p">,</span> <span class="nb">input</span><span class="p">):</span>
<span class="n">a</span> <span class="o">=</span> <span class="nb">input</span>
<span class="k">for</span> <span class="n">W</span><span class="p">,</span> <span class="n">b</span> <span class="ow">in</span> <span class="n">layers</span><span class="p">:</span>
<span class="n">z</span> <span class="o">=</span> <span class="n">W</span> <span class="o">@</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
<span class="n">a</span> <span class="o">=</span> <span class="n">ReLU</span><span class="p">(</span><span class="n">z</span><span class="p">)</span>
<span class="k">return</span> <span class="n">a</span>
</pre></div>
</div>
</div>
</div>
<p><strong>c)</strong> Create a network with input size 8 and layers with output sizes 10, 16, 6, 2. Evaluate it and make sure that you get the correct size vectors along the way.</p>
</div>
<div class="tex2jax_ignore mathjax_ignore section" id="exercise-4">
<h1>Exercise 4<a class="headerlink" href="#exercise-4" title="Permalink to this headline"></a></h1>
<p>So far, every layer has used the same activation, ReLU. We often want to use other types of activation however, so we need to update our code to support multiple types of activation. Make sure that you have completed every previous exercise before trying this one.</p>
<p><strong>a)</strong> Make the <code class="docutils literal notranslate"><span class="pre">create_layers</span></code> function also accept a list of activation functions, which is used to add activation functions to each of the tuples in <code class="docutils literal notranslate"><span class="pre">layers</span></code>. Make new functions to not mess with the old ones.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">create_layers_4</span><span class="p">(</span><span class="n">network_input_size</span><span class="p">,</span> <span class="n">output_sizes</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">):</span>
<span class="n">layers</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">i_size</span> <span class="o">=</span> <span class="n">network_input_size</span>
<span class="k">for</span> <span class="n">output_size</span><span class="p">,</span> <span class="n">activation</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">output_sizes</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">):</span>
<span class="n">W</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">output_size</span><span class="p">,</span> <span class="n">i_size</span><span class="p">)</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">output_size</span><span class="p">)</span>
<span class="n">layers</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">W</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">activation</span><span class="p">))</span>
<span class="n">i_size</span> <span class="o">=</span> <span class="n">output_size</span>
<span class="k">return</span> <span class="n">layers</span>
</pre></div>
</div>
</div>
</div>
<p><strong>b)</strong> Update the <code class="docutils literal notranslate"><span class="pre">feed_forward</span></code> function to support this change.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">feed_forward_4</span><span class="p">(</span><span class="n">layers</span><span class="p">,</span> <span class="nb">input</span><span class="p">):</span>
<span class="n">a</span> <span class="o">=</span> <span class="nb">input</span>
<span class="k">for</span> <span class="n">W</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">activation</span> <span class="ow">in</span> <span class="n">layers</span><span class="p">:</span>
<span class="n">z</span> <span class="o">=</span> <span class="n">W</span> <span class="o">@</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
<span class="n">a</span> <span class="o">=</span> <span class="n">activation</span><span class="p">(</span><span class="n">z</span><span class="p">)</span>
<span class="k">return</span> <span class="n">a</span>
</pre></div>
</div>
</div>
</div>
<p><strong>c)</strong> Create and evaluate a neural network with 4 inputs and layers with output sizes 12, 10, 3 and activations ReLU, ReLU, softmax.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">scipy.special</span> <span class="kn">import</span> <span class="n">softmax</span>
<span class="n">network_input_size</span> <span class="o">=</span> <span class="mi">4</span>
<span class="n">output_sizes</span> <span class="o">=</span> <span class="p">[</span><span class="mi">12</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
<span class="n">activation_funcs</span> <span class="o">=</span> <span class="p">[</span><span class="n">ReLU</span><span class="p">,</span> <span class="n">ReLU</span><span class="p">,</span> <span class="n">softmax</span><span class="p">]</span>
<span class="n">layers</span> <span class="o">=</span> <span class="n">create_layers_4</span><span class="p">(</span><span class="n">network_input_size</span><span class="p">,</span> <span class="n">output_sizes</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">network_input_size</span><span class="p">)</span>
<span class="n">predict</span> <span class="o">=</span> <span class="n">feed_forward_4</span><span class="p">(</span><span class="n">layers</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>The final exercise will hopefully be very simple if everything has worked so far. You will evaluate your neural network on the iris data set (<a class="reference external" href="https://scikit-learn.org/1.5/auto_examples/datasets/plot_iris_dataset.html">https://scikit-learn.org/1.5/auto_examples/datasets/plot_iris_dataset.html</a>).</p>
<p>This dataset contains data on 150 flowers of 3 different types which can be separated pretty well using the four features given for each flower, which includes the width and length of their leaves. You are not expected to do any training of the network or actual classification, unless you feel like it, in that case you can do exercise 5.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Loading and plotting iris dataset</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">datasets</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="n">iris</span> <span class="o">=</span> <span class="n">datasets</span><span class="o">.</span><span class="n">load_iris</span><span class="p">()</span>
<span class="n">_</span><span class="p">,</span> <span class="n">ax</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">subplots</span><span class="p">()</span>
<span class="n">scatter</span> <span class="o">=</span> <span class="n">ax</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">iris</span><span class="o">.</span><span class="n">data</span><span class="p">[:,</span> <span class="mi">0</span><span class="p">],</span> <span class="n">iris</span><span class="o">.</span><span class="n">data</span><span class="p">[:,</span> <span class="mi">1</span><span class="p">],</span> <span class="n">c</span><span class="o">=</span><span class="n">iris</span><span class="o">.</span><span class="n">target</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">xlabel</span><span class="o">=</span><span class="n">iris</span><span class="o">.</span><span class="n">feature_names</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">ylabel</span><span class="o">=</span><span class="n">iris</span><span class="o">.</span><span class="n">feature_names</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="n">_</span> <span class="o">=</span> <span class="n">ax</span><span class="o">.</span><span class="n">legend</span><span class="p">(</span>
<span class="n">scatter</span><span class="o">.</span><span class="n">legend_elements</span><span class="p">()[</span><span class="mi">0</span><span class="p">],</span> <span class="n">iris</span><span class="o">.</span><span class="n">target_names</span><span class="p">,</span> <span class="n">loc</span><span class="o">=</span><span class="s2">&quot;lower right&quot;</span><span class="p">,</span> <span class="n">title</span><span class="o">=</span><span class="s2">&quot;Classes&quot;</span>
<span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/exercisesweek42_34_0.png" src="_images/exercisesweek42_34_0.png" />
</div>
</div>
<p><strong>c)</strong> Loop over the iris dataset(<code class="docutils literal notranslate"><span class="pre">iris.data</span></code>) and evaluate the network for each data point.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># No need to change this cell! Just make sure it works!</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">iris</span><span class="o">.</span><span class="n">data</span><span class="p">:</span>
<span class="n">prediction</span> <span class="o">=</span> <span class="n">feed_forward_4</span><span class="p">(</span><span class="n">layers</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="tex2jax_ignore mathjax_ignore section" id="exercise-5-very-optional-and-very-hard">
<h1>Exercise 5 (Very optional and very hard :)<a class="headerlink" href="#exercise-5-very-optional-and-very-hard" title="Permalink to this headline"></a></h1>
<p><strong>a)</strong> Make the iris target values into one-hot vectors.</p>
<p><strong>b)</strong> Define the cross-entropy loss function to evaluate the performance of your network on the data set.</p>
<p><strong>c)</strong> Use the autograd package to take the gradient of the cross entropy wrt. the weights and biases of the network.</p>
<p><strong>d)</strong> Use gradient descent of some sort to optimize the parameters.</p>
<p><strong>e)</strong> Evaluate the accuracy of the network.</p>
<p><strong>e)</strong> Show off how you did in a group session!</p>
</div>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
kernelName: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</div>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
<a class='left-prev' id="prev-link" href="week41.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
<p class="prev-next-title">Week 41 Neural networks and constructing a neural network code</p>
</div>
</a>
<a class='right-next' id="next-link" href="week42.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">Week 42 Constructing a Neural Network code with examples</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
</div>
</div>
</div>
<footer class="footer">
<p>
By Morten Hjorth-Jensen<br/>
&copy; Copyright 2021.<br/>
</p>
</footer>
</main>
</div>
</div>
<script src="_static/js/index.be7d3bbb2ef33a8344ce.js"></script>
</body>
</html>