lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-10-23 07:29:30 +02:00
parent 36c4cd352b
commit 5ea024b9cd
32 changed files with 31684 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/exercisesweek43.html
+654
View File
@@ -0,0 +1,654 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Exercises weeks 43 and 44 &#8212; Applied Data Analysis and Machine Learning</title>
<link href="_static/css/theme.css" rel="stylesheet">
<link href="_static/css/index.ff1ffe594081f20da1ef19478df9384b.css" rel="stylesheet">
<link rel="stylesheet"
href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
<link rel="stylesheet" type="text/css" href="_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-book-theme.css?digest=c3fdc42140077d1ad13ad2f1588a4309" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-main.c949a650a448cc0ae9fd3441c0e17fb0.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-variables.06eb56fa6e07937060861dad626602ad.css" />
<link rel="preload" as="script" href="_static/js/index.be7d3bbb2ef33a8344ce.js">
<script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/clipboard.min.js"></script>
<script src="_static/copybutton.js"></script>
<script>let toggleHintShow = 'Click to show';</script>
<script>let toggleHintHide = 'Click to hide';</script>
<script>let toggleOpenOnPrint = 'true';</script>
<script src="_static/togglebutton.js"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown, .tag_hide_input div.cell_input, .tag_hide-input div.cell_input, .tag_hide_output div.cell_output, .tag_hide-output div.cell_output, .tag_hide_cell.cell, .tag_hide-cell.cell';</script>
<script src="_static/sphinx-book-theme.d59cb220de22ca1c485ebbdc042f0030.js"></script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"
const thebe_selector = ".thebe,.cell"
const thebe_selector_input = "pre"
const thebe_selector_output = ".output, .cell_output"
</script>
<script async="async" src="_static/sphinx-thebe.js"></script>
<script>window.MathJax = {"options": {"processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Week 43: Deep Learning: Constructing a Neural Network code and solving differential equations" href="week43.html" />
<link rel="prev" title="Week 42 Constructing a Neural Network code with introduction to Tensor flow" href="week42.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="None">
<!-- Google Analytics -->
</head>
<body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
<div class="container-fluid" id="banner"></div>
<div class="container-xl">
<div class="row">
<div class="col-12 col-md-3 bd-sidebar site-navigation show" id="site-navigation">
<div class="navbar-brand-box">
<a class="navbar-brand text-wrap" href="index.html">
<!-- `logo` is deprecated in Sphinx 4.0, so remove this when we stop supporting 3 -->
<img src="_static/logo.png" class="logo" alt="logo">
<h1 class="site-logo" id="site-title">Applied Data Analysis and Machine Learning</h1>
</a>
</div><form class="bd-search d-flex align-items-center" action="search.html" method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search this book..." aria-label="Search this book..." autocomplete="off" >
</form><nav class="bd-links" id="bd-docs-nav" aria-label="Main">
<div class="bd-toc-item active">
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="intro.html">
Applied Data Analysis and Machine Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
About the course
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="schedule.html">
Teaching schedule with links to material
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="teachers.html">
Teachers and Grading
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="textbooks.html">
Textbooks
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Review of Statistics with Resampling Techniques and Linear Algebra
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="statistics.html">
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="linalg.html">
2. Linear Algebra, Handling of Arrays and more Python Features
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
From Regression to Support Vector Machines
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter1.html">
3. Linear Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html">
4. Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
5. Resampling Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
6. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapteroptimization.html">
7. Optimization, the central part of any Machine Learning algortithm
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
8. Support Vector Machines, overarching aims
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Decision Trees, Ensemble Methods and Boosting
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
9. Decision trees, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Dimensionality Reduction
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="clustering.html">
12. Clustering and Unsupervised Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Deep Learning Methods
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter9.html">
13. Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter10.html">
14. Building a Feed Forward Neural Network
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter11.html">
15. Solving Differential Equations with Deep Learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter12.html">
16. Convolutional Neural Networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter13.html">
17. Recurrent neural networks: Overarching view
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Weekly material, notes and exercises
</span>
</p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek34.html">
Exercises week 34
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week34.html">
Week 34: Introduction to the course, Logistics and Practicalities
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek35.html">
Exercises week 35
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week35.html">
Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek36.html">
Exercises week 36
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week36.html">
Week 36: Statistical interpretation of Linear Regression and Resampling techniques
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek37.html">
Exercises week 37
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week37.html">
Week 37: Statistical interpretations and Resampling Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek38.html">
Exercises week 38
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week38.html">
Week 38: Logistic Regression and Optimization
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek39.html">
Exercises week 39
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week39.html">
Week 39: Optimization and Gradient Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week40.html">
Week 40: Gradient descent methods (continued) and start Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek41.html">
Exercises week 41
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week41.html">
Week 41 Neural networks and constructing a neural network code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek42.html">
Exercises week 42
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week42.html">
Week 42 Constructing a Neural Network code with introduction to Tensor flow
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
Exercises weeks 43 and 44
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week43.html">
Week 43: Deep Learning: Constructing a Neural Network code and solving differential equations
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Projects
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="project1.html">
Project 1 on Machine Learning, deadline October 9 (midnight), 2023
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="project2.html">
Project 2 on Machine Learning, deadline November 13 (Midnight)
</a>
</li>
</ul>
</div>
</nav> <!-- To handle the deprecated key -->
<div class="navbar_extra_footer">
Powered by <a href="https://jupyterbook.org">Jupyter Book</a>
</div>
</div>
<main class="col py-md-3 pl-md-4 bd-content overflow-auto" role="main">
<div class="topbar container-xl fixed-top">
<div class="topbar-contents row">
<div class="col-12 col-md-3 bd-topbar-whitespace site-navigation show"></div>
<div class="col pl-md-4 topbar-main">
<button id="navbar-toggler" class="navbar-toggler ml-0" type="button" data-toggle="collapse"
data-toggle="tooltip" data-placement="bottom" data-target=".site-navigation" aria-controls="navbar-menu"
aria-expanded="true" aria-label="Toggle navigation" aria-controls="site-navigation"
title="Toggle navigation" data-toggle="tooltip" data-placement="left">
<i class="fas fa-bars"></i>
<i class="fas fa-arrow-left"></i>
<i class="fas fa-arrow-up"></i>
</button>
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn" aria-label="Download this page"><i
class="fas fa-download"></i></button>
<div class="dropdown-buttons">
<!-- ipynb file if we had a myst markdown file -->
<!-- Download raw file -->
<a class="dropdown-buttons" href="_sources/exercisesweek43.ipynb"><button type="button"
class="btn btn-secondary topbarbtn" title="Download source file" data-toggle="tooltip"
data-placement="left">.ipynb</button></a>
<!-- Download PDF via print -->
<button type="button" id="download-print" class="btn btn-secondary topbarbtn" title="Print to PDF"
onclick="printPdf(this)" data-toggle="tooltip" data-placement="left">.pdf</button>
</div>
</div>
<!-- Source interaction buttons -->
<!-- Full screen (wrap in <a> to have style consistency -->
<a class="full-screen-button"><button type="button" class="btn btn-secondary topbarbtn" data-toggle="tooltip"
data-placement="bottom" onclick="toggleFullScreen()" aria-label="Fullscreen mode"
title="Fullscreen mode"><i
class="fas fa-expand"></i></button></a>
<!-- Launch buttons -->
</div>
<!-- Table of contents -->
<div class="d-none d-md-block col-md-2 bd-toc show noprint">
<div class="tocsection onthispage pt-5 pb-3">
<i class="fas fa-list"></i> Contents
</div>
<nav id="bd-toc-nav" aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#">
Exercises weeks 43 and 44
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#overarching-aims-of-the-exercises-weeks-43-and-44">
Overarching aims of the exercises weeks 43 and 44
</a>
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-and-and-xor-gates">
The AND and XOR Gates
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#representing-the-data-sets">
Representing the Data Sets
</a>
</li>
</ul>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div id="main-content" class="row">
<div class="col-12 col-md-9 pl-md-3 pr-md-0">
<!-- Table of contents that is only displayed when printing the page -->
<div id="jb-print-docs-body" class="onlyprint">
<h1>Exercises weeks 43 and 44</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
<div>
<h2> Contents </h2>
</div>
<nav aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#">
Exercises weeks 43 and 44
</a>
</li>
<li class="toc-h1 nav-item toc-entry">
<a class="reference internal nav-link" href="#overarching-aims-of-the-exercises-weeks-43-and-44">
Overarching aims of the exercises weeks 43 and 44
</a>
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-and-and-xor-gates">
The AND and XOR Gates
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#representing-the-data-sets">
Representing the Data Sets
</a>
</li>
</ul>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div>
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
doconce format html exercisesweek43.do.txt -->
<!-- dom:TITLE: Exercises weeks 43 and 44 --><div class="tex2jax_ignore mathjax_ignore section" id="exercises-weeks-43-and-44">
<h1>Exercises weeks 43 and 44<a class="headerlink" href="#exercises-weeks-43-and-44" title="Permalink to this headline"></a></h1>
<p><strong>October 9-13, 2023</strong></p>
<p>Date: <strong>Deadline is Sunday November 5 at midnight</strong></p>
<p>You can hand in the exercises from week 43 and week 44 as one exercise and get a total score of two additional points.</p>
</div>
<div class="tex2jax_ignore mathjax_ignore section" id="overarching-aims-of-the-exercises-weeks-43-and-44">
<h1>Overarching aims of the exercises weeks 43 and 44<a class="headerlink" href="#overarching-aims-of-the-exercises-weeks-43-and-44" title="Permalink to this headline"></a></h1>
<p>The aim of the exercises this week and next week is to get started with writing a neural network code
of relevance for project 2.</p>
<p>During week 41 we discussed three different types of gates, the
so-called XOR, the OR and the AND gates. In order to develop a code
for neural networks, it can be useful to set up a simpler system with
only two inputs and one output. This can make it easier to debug and
study the feed forward pass and the back propagation part. In the
exercise this and next week, we propose to study this system with just
one hidden layer and two hidden nodes. There is only one output node
and we can choose to use either a simple regression case (fitting a
line) or just a binary classification case with the cross-entropy as
cost function.</p>
<p>Their inputs and outputs can be
summarized using the following tables, first for the OR gate with
inputs <span class="math notranslate nohighlight">\(x_1\)</span> and <span class="math notranslate nohighlight">\(x_2\)</span> and outputs <span class="math notranslate nohighlight">\(y\)</span>:</p>
<table class="dotable" border="1">
<thead>
<tr><th align="center">$x_1$</th> <th align="center">$x_2$</th> <th align="center">$y$</th> </tr>
</thead>
<tbody>
<tr><td align="center"> 0 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 0 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 0 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
</tbody>
</table><div class="section" id="the-and-and-xor-gates">
<h2>The AND and XOR Gates<a class="headerlink" href="#the-and-and-xor-gates" title="Permalink to this headline"></a></h2>
<p>The AND gate is defined as</p>
<table class="dotable" border="1">
<thead>
<tr><th align="center">$x_1$</th> <th align="center">$x_2$</th> <th align="center">$y$</th> </tr>
</thead>
<tbody>
<tr><td align="center"> 0 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 0 </td> <td align="center"> 1 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
</tbody>
</table>
<p>And finally we have the XOR gate</p>
<table class="dotable" border="1">
<thead>
<tr><th align="center">$x_1$</th> <th align="center">$x_2$</th> <th align="center">$y$</th> </tr>
</thead>
<tbody>
<tr><td align="center"> 0 </td> <td align="center"> 0 </td> <td align="center"> 0 </td> </tr>
<tr><td align="center"> 0 </td> <td align="center"> 1 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 0 </td> <td align="center"> 1 </td> </tr>
<tr><td align="center"> 1 </td> <td align="center"> 1 </td> <td align="center"> 0 </td> </tr>
</tbody>
</table></div>
<div class="section" id="representing-the-data-sets">
<h2>Representing the Data Sets<a class="headerlink" href="#representing-the-data-sets" title="Permalink to this headline"></a></h2>
<p>Our design matrix is defined by the input values <span class="math notranslate nohighlight">\(x_1\)</span> and <span class="math notranslate nohighlight">\(x_2\)</span>. Since we have four possible outputs, our design matrix reads</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\boldsymbol{X}=\begin{bmatrix} 0 &amp; 0 \\
0 &amp; 1 \\
1 &amp; 0 \\
1 &amp; 1 \end{bmatrix},
\end{split}\]</div>
<p>while the vector of outputs is <span class="math notranslate nohighlight">\(\boldsymbol{y}^T=[0,1,1,0]\)</span> for the XOR gate, <span class="math notranslate nohighlight">\(\boldsymbol{y}^T=[0,0,0,1]\)</span> for the AND gate and <span class="math notranslate nohighlight">\(\boldsymbol{y}^T=[0,1,1,1]\)</span> for the OR gate.</p>
<p>Your tasks here are</p>
<ol class="simple">
<li><p>Set up the design matrix with the inputs as discussed above and a vector containing the output, the so-called targets. Note that the design matrix is the same for all gates. You need just to define different outputs.</p></li>
<li><p>Construct a neural network with only one hidden layer and two hidden nodes using the Sigmoid function as activation function.</p></li>
<li><p>Set up the output layer with only one output node and use again the Sigmoid function as activation function for the output.</p></li>
<li><p>Initialize the weights and biases and perform a feed forward pass and compare the outputs with the targets.</p></li>
<li><p>Set up the cost function (cross entropy for classification of binary cases).</p></li>
<li><p>Calculate the gradients needed for the back propagation part.</p></li>
<li><p>Use the gradients to train the network in the back propagation part. Think of using automatic differentiation.</p></li>
<li><p>Train the network and study your results and compare with results obtained either with <strong>scikit-learn</strong> or <strong>TensorFlow</strong>.</p></li>
</ol>
<p>Everything you develop here can be used directly into the code for the project.</p>
</div>
</div>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
kernelName: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</div>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
<a class='left-prev' id="prev-link" href="week42.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
<p class="prev-next-title">Week 42 Constructing a Neural Network code with introduction to Tensor flow</p>
</div>
</a>
<a class='right-next' id="next-link" href="week43.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">Week 43: Deep Learning: Constructing a Neural Network code and solving differential equations</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
</div>
</div>
</div>
<footer class="footer">
<p>
By Morten Hjorth-Jensen<br/>
&copy; Copyright 2021.<br/>
</p>
</footer>
</main>
</div>
</div>
<script src="_static/js/index.be7d3bbb2ef33a8344ce.js"></script>
</body>
</html>