lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

minor typos

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2024-08-25 21:36:53 +02:00
parent 053f351848
commit 757cad43cd
16 changed files with 1952 additions and 3003 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/.doctrees/environment.pickle
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/.doctrees/exercisesweek34.doctree
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/.doctrees/week35.doctree
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/html/_sources/exercisesweek34.ipynb
+20 -19
View File
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "72dc0997",
"id": "a50ea987",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
"id": "8c591125",
"id": "291e6015",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "1ad21876",
"id": "141ff111",
"metadata": {
"editable": true
},
@@ -39,7 +39,7 @@
},
{
"cell_type": "markdown",
"id": "4eaa91e2",
"id": "32a1fc9e",
"metadata": {
"editable": true
},
@@ -108,7 +108,7 @@
},
{
"cell_type": "markdown",
"id": "360adb56",
"id": "f103f4c1",
"metadata": {
"editable": true
},
@@ -122,20 +122,21 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "278aa34f",
"id": "81c11edc",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"x = np.random.rand(100,1)\n",
"y = 2.0+5*x*x+0.1*np.random.randn(100,1)"
]
},
{
"cell_type": "markdown",
"id": "1ff20e45",
"id": "970bd604",
"metadata": {
"editable": true
},
@@ -149,7 +150,7 @@
},
{
"cell_type": "markdown",
"id": "e8809de1",
"id": "bda3ed04",
"metadata": {
"editable": true
},
@@ -162,7 +163,7 @@
},
{
"cell_type": "markdown",
"id": "8c9e5277",
"id": "ca995691",
"metadata": {
"editable": true
},
@@ -173,7 +174,7 @@
},
{
"cell_type": "markdown",
"id": "85a5881c",
"id": "27cbfeed",
"metadata": {
"editable": true
},
@@ -185,7 +186,7 @@
},
{
"cell_type": "markdown",
"id": "4fb35d0d",
"id": "9e063f6d",
"metadata": {
"editable": true
},
@@ -195,7 +196,7 @@
},
{
"cell_type": "markdown",
"id": "608f9780",
"id": "e582c2bb",
"metadata": {
"editable": true
},
@@ -207,7 +208,7 @@
},
{
"cell_type": "markdown",
"id": "014bbc09",
"id": "59e0ed7a",
"metadata": {
"editable": true
},
@@ -218,7 +219,7 @@
},
{
"cell_type": "markdown",
"id": "f28863ad",
"id": "051e71aa",
"metadata": {
"editable": true
},
@@ -237,7 +238,7 @@
{
"cell_type": "code",
"execution_count": 2,
"id": "85f75a99",
"id": "c82d8b64",
"metadata": {
"collapsed": false,
"editable": true
@@ -253,7 +254,7 @@
},
{
"cell_type": "markdown",
"id": "bee8d814",
"id": "3e810bba",
"metadata": {
"editable": true
},
@@ -263,7 +264,7 @@
},
{
"cell_type": "markdown",
"id": "7c94f8bd",
"id": "c1198fce",
"metadata": {
"editable": true
},
@@ -274,7 +275,7 @@
},
{
"cell_type": "markdown",
"id": "c0e789dd",
"id": "1b0cea0a",
"metadata": {
"editable": true
},
@@ -286,7 +287,7 @@
},
{
"cell_type": "markdown",
"id": "5ee1d70e",
"id": "5194ba08",
"metadata": {
"editable": true
},
doc/LectureNotes/_build/html/_sources/week35.ipynb
+422 -782
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/html/exercisesweek34.html
+2 -12
View File
@@ -480,22 +480,12 @@ license.</p>
The following simple Python instructions define our <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> values (with 100 data points).</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">5</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mf">0.1</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
<span class="ne">NameError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
<span class="n">Cell</span> <span class="n">In</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">line</span> <span class="mi">1</span>
<span class="ne">----&gt; </span><span class="mi">1</span> <span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">2</span> <span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">5</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mf">0.1</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="ne">NameError</span>: name &#39;np&#39; is not defined
</pre></div>
</div>
</div>
</div>
<ol class="simple">
<li><p>Write your own code (following the examples under the <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter1.html">regression notes</a>) for computing the parametrization of the data set fitting a second-order polynomial.</p></li>
doc/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
doc/LectureNotes/_build/html/week35.html
+150 -322
View File
@@ -351,8 +351,8 @@ const thebe_selector_output = ".output, .cell_output"
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week">
Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week
<a class="reference internal nav-link" href="#for-exercise-sessions-why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week">
For exercise sessions: Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -536,18 +536,8 @@ const thebe_selector_output = ".output, .cell_output"
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-boston-housing-data-example">
The Boston housing data example
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#housing-data-the-code">
Housing data, the code
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#material-for-lecture-thursday-august-31">
Material for lecture Thursday, August 31
<a class="reference internal nav-link" href="#material-for-lecture-monday-august-26">
Material for lecture Monday, August 26
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -746,8 +736,8 @@ const thebe_selector_output = ".output, .cell_output"
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week">
Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week
<a class="reference internal nav-link" href="#for-exercise-sessions-why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week">
For exercise sessions: Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -931,18 +921,8 @@ const thebe_selector_output = ".output, .cell_output"
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-boston-housing-data-example">
The Boston housing data example
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#housing-data-the-code">
Housing data, the code
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#material-for-lecture-thursday-august-31">
Material for lecture Thursday, August 31
<a class="reference internal nav-link" href="#material-for-lecture-monday-august-26">
Material for lecture Monday, August 26
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -1121,14 +1101,14 @@ const thebe_selector_output = ".output, .cell_output"
doconce format html week35.do.txt --no_mako -->
<!-- dom:TITLE: Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression --><div class="tex2jax_ignore mathjax_ignore section" id="week-35-from-ordinary-linear-regression-to-ridge-and-lasso-regression">
<h1>Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression<a class="headerlink" href="#week-35-from-ordinary-linear-regression-to-ridge-and-lasso-regression" title="Permalink to this headline"></a></h1>
<p><strong>Morten Hjorth-Jensen</strong>, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</p>
<p>Date: <strong>August 26-30</strong></p>
<p><strong>Morten Hjorth-Jensen</strong>, Department of Physics, University of Oslo</p>
<p>Date: <strong>August 26-30, 2024</strong></p>
<div class="section" id="plans-for-week-35">
<h2>Plans for week 35<a class="headerlink" href="#plans-for-week-35" title="Permalink to this headline"></a></h2>
<p>The main topics are:</p>
<ol class="simple">
<li><p>Brief repetition from last week</p></li>
<li><p>Derivation of the equations for ordinary least squares</p></li>
<li><p>Discussions of the equations for ordinary least squares</p></li>
<li><p>Discussion on how to prepare data and examples of applications of linear regression</p></li>
<li><p>Material for the lecture on Monday: Mathematical interpretations of linear regression</p></li>
<li><p>Monday: Ridge and Lasso regression and Singular Value Decomposition</p></li>
@@ -1136,13 +1116,15 @@ doconce format html week35.do.txt --no_mako -->
<div class="section" id="reading-recommendations">
<h3>Reading recommendations:<a class="headerlink" href="#reading-recommendations" title="Permalink to this headline"></a></h3>
<ol class="simple">
<li><p>See lecture notes for week 35 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a></p></li>
<li><p>These lecture notes</p></li>
<li><p>Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)</p></li>
<li><p>Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4.</p></li>
<li><p>For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Marc Peter Deisenroth on the Mathematics of Machine Learning, may be very relevant. In particular chapter 5 at URL”<a class="reference external" href="https://mml-book.github.io/">https://mml-book.github.io/</a>” (section 5.5 on derivatives) is very useful for exercise 1 this coming week.</p></li>
</ol>
</div>
</div>
<div class="section" id="why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week">
<h2>Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week<a class="headerlink" href="#why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week" title="Permalink to this headline"></a></h2>
<div class="section" id="for-exercise-sessions-why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week">
<h2>For exercise sessions: Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week<a class="headerlink" href="#for-exercise-sessions-why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week" title="Permalink to this headline"></a></h2>
<p>We need first a reminder from last week about linear regression.</p>
<p>Fitting a continuous function with linear parameterization in terms of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
<ul class="simple">
@@ -1610,7 +1592,7 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.9952638231265687
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.9966579875073338
</pre></div>
</div>
</div>
@@ -1627,7 +1609,7 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.011761161707539526
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.007652435292578527
</pre></div>
</div>
</div>
@@ -1642,23 +1624,31 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.00035753 0.04937621 0.02268114 0.03112297 0.01856502 0.05322899
0.01397927 0.03999935 0.02508836 0.01834042 0.0652633 0.00887114
0.01956921 0.01987248 0.06294313 0.01152476 0.00328431 0.03564082
0.02337045 0.01743586 0.01278035 0.02400968 0.08388138 0.03270341
0.00335956 0.01031903 0.09575469 0.00956774 0.00900879 0.01867985
0.01191227 0.02090296 0.04045671 0.03582958 0.06692165 0.06615863
0.06594872 0.03756493 0.00488992 0.01405237 0.00117071 0.0017567
0.05006306 0.02545639 0.03456485 0.00373984 0.02576476 0.03530741
0.00092902 0.0275742 0.05948007 0.01321652 0.18500705 0.00382166
0.00327313 0.01853877 0.01771317 0.05293662 0.07199977 0.00836148
0.01541649 0.00343257 0.00626797 0.05350297 0.01548272 0.05235058
0.04310698 0.00225189 0.02356396 0.01690512 0.03467756 0.00064364
0.02593596 0.00019607 0.0029508 0.0180194 0.06825695 0.01659559
0.01971341 0.02012338 0.02241311 0.00135736 0.0095653 0.05695438
0.00395659 0.07068033 0.02699873 0.00919237 0.02493299 0.00803115
0.0293055 0.02178063 0.00594353 0.04081883 0.01325225 0.0386443
0.01889695 0.02810253 0.0181166 0.01135924]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[3.25042813e-02 2.38298328e-02 2.64093905e-02 1.31145723e-02
7.85640706e-02 2.08298871e-02 2.08187418e-02 6.69029803e-04
2.18905573e-02 2.14957387e-02 2.32863661e-02 4.66181587e-03
8.93710544e-03 2.89564974e-03 2.51647799e-03 3.61336683e-02
1.11766707e-02 6.39623570e-02 7.54190070e-03 2.85193598e-03
4.90018622e-02 1.46957262e-02 4.86592590e-02 9.69161166e-05
1.11309850e-02 1.47605918e-02 1.80419748e-03 3.18023984e-02
2.92063704e-02 3.53149684e-02 1.18573920e-02 3.32775737e-03
1.71196799e-02 1.96079426e-02 1.69232893e-02 1.89273223e-02
3.13021533e-02 2.96574402e-03 4.06270382e-02 4.92293994e-02
1.07282462e-02 1.29064873e-02 8.37618253e-03 4.02472584e-02
2.44637348e-02 1.00149403e-02 2.21129518e-02 2.30674822e-03
4.87391702e-02 2.53371547e-02 8.26280707e-03 1.80203363e-02
2.38115471e-02 7.62038116e-02 4.34816464e-02 3.92213792e-02
6.68818255e-03 8.88623007e-03 8.28473779e-02 4.53859920e-02
8.26121775e-02 4.67869899e-03 1.62034461e-02 2.78672237e-02
8.98401894e-03 3.17303133e-02 3.51729923e-02 2.90612093e-03
5.70221196e-03 3.49065324e-02 3.22091393e-02 2.15706394e-02
1.40920261e-02 3.82602592e-03 4.59101492e-02 3.65313891e-02
1.52725477e-02 3.05989176e-02 1.28347954e-03 3.00674454e-02
8.92998014e-02 1.63727258e-02 3.77117804e-02 1.17886788e-02
8.76071777e-02 4.11958697e-02 4.99322932e-03 7.57875488e-02
1.16954396e-04 1.90884438e-02 1.41441328e-02 5.38923192e-02
9.21237677e-03 1.36654107e-02 9.03904224e-04 9.90027459e-03
3.01828101e-02 4.71038982e-02 3.87200322e-02 1.62397973e-02]
</pre></div>
</div>
</div>
@@ -1727,15 +1717,15 @@ but now splitting the data into a training set and a test set.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 2.00507876 0.39026883 3.20764972 2.73806921 -1.39609089]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 1.86544433 1.28573357 0.76075617 5.54611024 -2.45878472]
Training R2
0.9972493421341901
0.9958543328528107
Training MSE
0.007021475481484069
0.011031108072207247
Test R2
0.9976639055733267
0.9958824304846606
Test MSE
0.00575158411598985
0.006546368424400406
</pre></div>
</div>
</div>
@@ -2729,270 +2719,8 @@ back the intercept, we will get an MSE which then contains the
intercept. This becomes more important when we discuss Ridge and Lasso
regression next week.</p>
</div>
<div class="section" id="the-boston-housing-data-example">
<h2>The Boston housing data example<a class="headerlink" href="#the-boston-housing-data-example" title="Permalink to this headline"></a></h2>
<p>The Boston housing<br />
data set was originally a part of UCI Machine Learning Repository
and has been removed now. The data set is now included in <strong>Scikit-Learn</strong>’s
library. There are 506 samples and 13 feature (predictor) variables
in this data set. The objective is to predict the value of prices of
the house using the features (predictors) listed here.</p>
<p>The features/predictors are</p>
<ol class="simple">
<li><p>CRIM: Per capita crime rate by town</p></li>
<li><p>ZN: Proportion of residential land zoned for lots over 25000 square feet</p></li>
<li><p>INDUS: Proportion of non-retail business acres per town</p></li>
<li><p>CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)</p></li>
<li><p>NOX: Nitric oxide concentration (parts per 10 million)</p></li>
<li><p>RM: Average number of rooms per dwelling</p></li>
<li><p>AGE: Proportion of owner-occupied units built prior to 1940</p></li>
<li><p>DIS: Weighted distances to five Boston employment centers</p></li>
<li><p>RAD: Index of accessibility to radial highways</p></li>
<li><p>TAX: Full-value property tax rate per USD10000</p></li>
<li><p>B: <span class="math notranslate nohighlight">\(1000(Bk - 0.63)^2\)</span>, where <span class="math notranslate nohighlight">\(Bk\)</span> is the proportion of [people of African American descent] by town</p></li>
<li><p>LSTAT: Percentage of lower status of the population</p></li>
<li><p>MEDV: Median value of owner-occupied homes in USD 1000s</p></li>
</ol>
</div>
<div class="section" id="housing-data-the-code">
<h2>Housing data, the code<a class="headerlink" href="#housing-data-the-code" title="Permalink to this headline"></a></h2>
<p>We start by importing the libraries</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">seaborn</span> <span class="k">as</span> <span class="nn">sns</span>
</pre></div>
</div>
</div>
</div>
<p>and load the Boston Housing DataSet from <strong>Scikit-Learn</strong></p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="kn">import</span> <span class="n">load_boston</span>
<span class="n">boston_dataset</span> <span class="o">=</span> <span class="n">load_boston</span><span class="p">()</span>
<span class="c1"># boston_dataset is a dictionary</span>
<span class="c1"># let&#39;s check what it contains</span>
<span class="n">boston_dataset</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span>---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
Cell In[16], line 1
----&gt; 1 from sklearn.datasets import load_boston
3 boston_dataset = load_boston()
5 # boston_dataset is a dictionary
6 # let&#39;s check what it contains
File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/datasets/__init__.py:157, in __getattr__(name)
108 if name == &quot;load_boston&quot;:
109 msg = textwrap.dedent(&quot;&quot;&quot;
110 `load_boston` has been removed from scikit-learn since version 1.2.
111
(...)
155 &lt;https://www.researchgate.net/publication/4974606_Hedonic_housing_prices_and_the_demand_for_clean_air&gt;
156 &quot;&quot;&quot;)
--&gt; 157 raise ImportError(msg)
158 try:
159 return globals()[name]
ImportError:
`load_boston` has been removed from scikit-learn since version 1.2.
The Boston housing prices dataset has an ethical problem: as
investigated in [1], the authors of this dataset engineered a
non-invertible variable &quot;B&quot; assuming that racial self-segregation had a
positive impact on house prices [2]. Furthermore the goal of the
research that led to the creation of this dataset was to study the
impact of air quality but it did not give adequate demonstration of the
validity of this assumption.
The scikit-learn maintainers therefore strongly discourage the use of
this dataset unless the purpose of the code is to study and educate
about ethical issues in data science and machine learning.
In this special case, you can fetch the dataset from the original
source::
import pandas as pd
import numpy as np
data_url = &quot;http://lib.stat.cmu.edu/datasets/boston&quot;
raw_df = pd.read_csv(data_url, sep=&quot;\s+&quot;, skiprows=22, header=None)
data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])
target = raw_df.values[1::2, 2]
Alternative datasets include the California housing dataset and the
Ames housing dataset. You can load the datasets as follows::
from sklearn.datasets import fetch_california_housing
housing = fetch_california_housing()
for the California housing dataset and::
from sklearn.datasets import fetch_openml
housing = fetch_openml(name=&quot;house_prices&quot;, as_frame=True)
for the Ames housing dataset.
[1] M Carlisle.
&quot;Racist data destruction?&quot;
&lt;https://medium.com/@docintangible/racist-data-destruction-113e3eff54a8&gt;
[2] Harrison Jr, David, and Daniel L. Rubinfeld.
&quot;Hedonic housing prices and the demand for clean air.&quot;
Journal of environmental economics and management 5.1 (1978): 81-102.
&lt;https://www.researchgate.net/publication/4974606_Hedonic_housing_prices_and_the_demand_for_clean_air&gt;
</pre></div>
</div>
</div>
</div>
<p>Then we invoke Pandas</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">boston</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">boston_dataset</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="n">boston_dataset</span><span class="o">.</span><span class="n">feature_names</span><span class="p">)</span>
<span class="n">boston</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="n">boston</span><span class="p">[</span><span class="s1">&#39;MEDV&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">boston_dataset</span><span class="o">.</span><span class="n">target</span>
</pre></div>
</div>
</div>
</div>
<p>and preprocess the data</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># check for missing values in all the columns</span>
<span class="n">boston</span><span class="o">.</span><span class="n">isnull</span><span class="p">()</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
<p>We can then visualize the data</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># set the size of the figure</span>
<span class="n">sns</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">rc</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;figure.figsize&#39;</span><span class="p">:(</span><span class="mf">11.7</span><span class="p">,</span><span class="mf">8.27</span><span class="p">)})</span>
<span class="c1"># plot a histogram showing the distribution of the target values</span>
<span class="n">sns</span><span class="o">.</span><span class="n">distplot</span><span class="p">(</span><span class="n">boston</span><span class="p">[</span><span class="s1">&#39;MEDV&#39;</span><span class="p">],</span> <span class="n">bins</span><span class="o">=</span><span class="mi">30</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
<p>It is now useful to look at the correlation matrix</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># compute the pair wise correlation for all columns </span>
<span class="n">correlation_matrix</span> <span class="o">=</span> <span class="n">boston</span><span class="o">.</span><span class="n">corr</span><span class="p">()</span><span class="o">.</span><span class="n">round</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="c1"># use the heatmap function from seaborn to plot the correlation matrix</span>
<span class="c1"># annot = True to print the values inside the square</span>
<span class="n">sns</span><span class="o">.</span><span class="n">heatmap</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">correlation_matrix</span><span class="p">,</span> <span class="n">annot</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>From the above coorelation plot we can see that <strong>MEDV</strong> is strongly correlated to <strong>LSTAT</strong> and <strong>RM</strong>. We see also that <strong>RAD</strong> and <strong>TAX</strong> are stronly correlated, but we don’t include this in our features together to avoid multi-colinearity</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span> <span class="mi">5</span><span class="p">))</span>
<span class="n">features</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;LSTAT&#39;</span><span class="p">,</span> <span class="s1">&#39;RM&#39;</span><span class="p">]</span>
<span class="n">target</span> <span class="o">=</span> <span class="n">boston</span><span class="p">[</span><span class="s1">&#39;MEDV&#39;</span><span class="p">]</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">col</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">features</span><span class="p">):</span>
<span class="n">plt</span><span class="o">.</span><span class="n">subplot</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">features</span><span class="p">)</span> <span class="p">,</span> <span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">boston</span><span class="p">[</span><span class="n">col</span><span class="p">]</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">target</span>
<span class="n">plt</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">marker</span><span class="o">=</span><span class="s1">&#39;o&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="n">col</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="n">col</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">&#39;MEDV&#39;</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>Now we start training our model</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">X</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">boston</span><span class="p">[</span><span class="s1">&#39;LSTAT&#39;</span><span class="p">],</span> <span class="n">boston</span><span class="p">[</span><span class="s1">&#39;RM&#39;</span><span class="p">]],</span> <span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;LSTAT&#39;</span><span class="p">,</span><span class="s1">&#39;RM&#39;</span><span class="p">])</span>
<span class="n">Y</span> <span class="o">=</span> <span class="n">boston</span><span class="p">[</span><span class="s1">&#39;MEDV&#39;</span><span class="p">]</span>
</pre></div>
</div>
</div>
</div>
<p>We split the data into training and test sets</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="c1"># splits the training and test data set in 80% : 20%</span>
<span class="c1"># assign random_state to any value.This ensures consistency.</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Y</span><span class="p">,</span> <span class="n">test_size</span> <span class="o">=</span> <span class="mf">0.2</span><span class="p">,</span> <span class="n">random_state</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X_test</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">Y_train</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">Y_test</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>Then we use the linear regression functionality from <strong>Scikit-Learn</strong></p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LinearRegression</span>
<span class="kn">from</span> <span class="nn">sklearn.metrics</span> <span class="kn">import</span> <span class="n">mean_squared_error</span><span class="p">,</span> <span class="n">r2_score</span>
<span class="n">lin_model</span> <span class="o">=</span> <span class="n">LinearRegression</span><span class="p">()</span>
<span class="n">lin_model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">)</span>
<span class="c1"># model evaluation for training set</span>
<span class="n">y_train_predict</span> <span class="o">=</span> <span class="n">lin_model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
<span class="n">rmse</span> <span class="o">=</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">mean_squared_error</span><span class="p">(</span><span class="n">Y_train</span><span class="p">,</span> <span class="n">y_train_predict</span><span class="p">)))</span>
<span class="n">r2</span> <span class="o">=</span> <span class="n">r2_score</span><span class="p">(</span><span class="n">Y_train</span><span class="p">,</span> <span class="n">y_train_predict</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;The model performance for training set&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;--------------------------------------&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;RMSE is </span><span class="si">{}</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">rmse</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;R2 score is </span><span class="si">{}</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">r2</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;</span><span class="se">\n</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="c1"># model evaluation for testing set</span>
<span class="n">y_test_predict</span> <span class="o">=</span> <span class="n">lin_model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="c1"># root mean square error of the model</span>
<span class="n">rmse</span> <span class="o">=</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">mean_squared_error</span><span class="p">(</span><span class="n">Y_test</span><span class="p">,</span> <span class="n">y_test_predict</span><span class="p">)))</span>
<span class="c1"># r-squared score of the model</span>
<span class="n">r2</span> <span class="o">=</span> <span class="n">r2_score</span><span class="p">(</span><span class="n">Y_test</span><span class="p">,</span> <span class="n">y_test_predict</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;The model performance for testing set&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;--------------------------------------&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;RMSE is </span><span class="si">{}</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">rmse</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;R2 score is </span><span class="si">{}</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">r2</span><span class="p">))</span>
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># plotting the y_test vs y_pred</span>
<span class="c1"># ideally should have been a straight line</span>
<span class="n">plt</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">Y_test</span><span class="p">,</span> <span class="n">y_test_predict</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="material-for-lecture-thursday-august-31">
<h2>Material for lecture Thursday, August 31<a class="headerlink" href="#material-for-lecture-thursday-august-31" title="Permalink to this headline"></a></h2>
<div class="section" id="material-for-lecture-monday-august-26">
<h2>Material for lecture Monday, August 26<a class="headerlink" href="#material-for-lecture-monday-august-26" title="Permalink to this headline"></a></h2>
</div>
<div class="section" id="mathematical-interpretation-of-ordinary-least-squares">
<h2>Mathematical Interpretation of Ordinary Least Squares<a class="headerlink" href="#mathematical-interpretation-of-ordinary-least-squares" title="Permalink to this headline"></a></h2>
@@ -3268,6 +2996,25 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ 1. -1.]
[ 1. -1.]]
test U
[[0. 0.]
[0. 0.]]
test VT
[[0. 0.]
[0. 0.]]
[[-0.70710678 -0.70710678]
[-0.70710678 0.70710678]]
[2.00000000e+00 3.35470445e-17]
[[-0.70710678 0.70710678]
[ 0.70710678 0.70710678]]
[[-3.33066907e-16 4.44089210e-16]
[ 0.00000000e+00 2.22044605e-16]]
</pre></div>
</div>
</div>
</div>
<p>The matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> has columns that are linearly dependent. The first
column is the row-wise sum of the other two columns. The rank of a
@@ -3623,6 +3370,14 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.10790125813226321
4.340071371496255
[[ 1.04193203 3.08165104]
[ 3.08165104 10.18383522]]
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="correlation-matrix">
@@ -3658,6 +3413,14 @@ a more brute force way. Here we scale the mean values for each column of the des
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.08881497884574564
1.7086067479626619
[[1. 0.66080313]
[0.66080313 1. ]]
</pre></div>
</div>
</div>
</div>
<p>We see that the matrix elements along the diagonal are one as they
should be and that the matrix is symmetric. Furthermore, diagonalizing
@@ -3686,6 +3449,34 @@ this matrix we easily see that it is a positive definite matrix.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-0.40620066 -2.01265755]
[ 0.01458611 0.37737221]
[-1.0895387 -3.65442354]
[ 0.2338675 1.12044974]
[ 0.4676059 1.54393936]
[-0.65891389 -3.16304863]
[-0.1715252 0.39197698]
[ 0.71142161 2.95511792]
[ 0.39214397 0.13069442]
[ 0.50655336 2.3105791 ]]
0 1
0 -0.406201 -2.012658
1 0.014586 0.377372
2 -1.089539 -3.654424
3 0.233868 1.120450
4 0.467606 1.543939
5 -0.658914 -3.163049
6 -0.171525 0.391977
7 0.711422 2.955118
8 0.392144 0.130694
9 0.506553 2.310579
0 1
0 1.000000 0.952387
1 0.952387 1.000000
</pre></div>
</div>
</div>
</div>
<p>We expand this model to the Franke function discussed above.</p>
</div>
@@ -3739,6 +3530,43 @@ this matrix we easily see that it is a positive definite matrix.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0 1 2 3 4 5 6 7 \
0 0.0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.0 0.078974 0.081276 0.075889 0.077168 0.078540 0.065410 0.066474
2 0.0 0.081276 0.084076 0.078336 0.079946 0.081655 0.067707 0.069028
3 0.0 0.075889 0.078336 0.077460 0.078986 0.080616 0.069452 0.070737
4 0.0 0.077168 0.079946 0.078986 0.080764 0.082653 0.070986 0.072476
5 0.0 0.078540 0.081655 0.080616 0.082653 0.084809 0.072621 0.074323
6 0.0 0.065410 0.067707 0.069452 0.070986 0.072621 0.064074 0.065378
7 0.0 0.066474 0.069028 0.070737 0.072476 0.074323 0.065378 0.066854
8 0.0 0.067637 0.070457 0.072132 0.074084 0.076150 0.066787 0.068441
9 0.0 0.068906 0.072000 0.073644 0.075816 0.078110 0.068307 0.070146
10 0.0 0.055734 0.057835 0.060872 0.062337 0.063894 0.057393 0.058645
11 0.0 0.056683 0.058996 0.062016 0.063653 0.065390 0.058552 0.059951
12 0.0 0.057722 0.060254 0.063260 0.065077 0.066999 0.059807 0.061359
13 0.0 0.058854 0.061614 0.064609 0.066612 0.068727 0.061163 0.062874
14 0.0 0.060083 0.063080 0.066066 0.068264 0.070582 0.062624 0.064501
8 9 10 11 12 13 14
0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
1 0.067637 0.068906 0.055734 0.056683 0.057722 0.058854 0.060083
2 0.070457 0.072000 0.057835 0.058996 0.060254 0.061614 0.063080
3 0.072132 0.073644 0.060872 0.062016 0.063260 0.064609 0.066066
4 0.074084 0.075816 0.062337 0.063653 0.065077 0.066612 0.068264
5 0.076150 0.078110 0.063894 0.065390 0.066999 0.068727 0.070582
6 0.066787 0.068307 0.057393 0.058552 0.059807 0.061163 0.062624
7 0.068441 0.070146 0.058645 0.059951 0.061359 0.062874 0.064501
8 0.070213 0.072111 0.059993 0.061452 0.063019 0.064699 0.066500
9 0.072111 0.074210 0.061443 0.063061 0.064793 0.066647 0.068629
10 0.059993 0.061443 0.052305 0.053417 0.054617 0.055910 0.057300
11 0.061452 0.063061 0.053417 0.054655 0.055987 0.057418 0.058952
12 0.063019 0.064793 0.054617 0.055987 0.057457 0.059031 0.060716
13 0.064699 0.066647 0.055910 0.057418 0.059031 0.060756 0.062599
14 0.066500 0.068629 0.057300 0.058952 0.060716 0.062599 0.064606
</pre></div>
</div>
</div>
</div>
<p>We note here that the covariance is zero for the first rows and
columns since all matrix elements in the design matrix were set to one
doc/LectureNotes/_build/jupyter_execute/exercisesweek34.ipynb
+21 -32
View File
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "72dc0997",
"id": "a50ea987",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
"id": "8c591125",
"id": "291e6015",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "1ad21876",
"id": "141ff111",
"metadata": {
"editable": true
},
@@ -39,7 +39,7 @@
},
{
"cell_type": "markdown",
"id": "4eaa91e2",
"id": "32a1fc9e",
"metadata": {
"editable": true
},
@@ -108,7 +108,7 @@
},
{
"cell_type": "markdown",
"id": "360adb56",
"id": "f103f4c1",
"metadata": {
"editable": true
},
@@ -122,32 +122,21 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "278aa34f",
"id": "81c11edc",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [
{
"ename": "NameError",
"evalue": "name 'np' is not defined",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[1], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m x \u001b[38;5;241m=\u001b[39m \u001b[43mnp\u001b[49m\u001b[38;5;241m.\u001b[39mrandom\u001b[38;5;241m.\u001b[39mrand(\u001b[38;5;241m100\u001b[39m,\u001b[38;5;241m1\u001b[39m)\n\u001b[1;32m 2\u001b[0m y \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m2.0\u001b[39m\u001b[38;5;241m+\u001b[39m\u001b[38;5;241m5\u001b[39m\u001b[38;5;241m*\u001b[39mx\u001b[38;5;241m*\u001b[39mx\u001b[38;5;241m+\u001b[39m\u001b[38;5;241m0.1\u001b[39m\u001b[38;5;241m*\u001b[39mnp\u001b[38;5;241m.\u001b[39mrandom\u001b[38;5;241m.\u001b[39mrandn(\u001b[38;5;241m100\u001b[39m,\u001b[38;5;241m1\u001b[39m)\n",
"\u001b[0;31mNameError\u001b[0m: name 'np' is not defined"
]
}
],
"outputs": [],
"source": [
"import numpy as np\n",
"x = np.random.rand(100,1)\n",
"y = 2.0+5*x*x+0.1*np.random.randn(100,1)"
]
},
{
"cell_type": "markdown",
"id": "1ff20e45",
"id": "970bd604",
"metadata": {
"editable": true
},
@@ -161,7 +150,7 @@
},
{
"cell_type": "markdown",
"id": "e8809de1",
"id": "bda3ed04",
"metadata": {
"editable": true
},
@@ -174,7 +163,7 @@
},
{
"cell_type": "markdown",
"id": "8c9e5277",
"id": "ca995691",
"metadata": {
"editable": true
},
@@ -185,7 +174,7 @@
},
{
"cell_type": "markdown",
"id": "85a5881c",
"id": "27cbfeed",
"metadata": {
"editable": true
},
@@ -197,7 +186,7 @@
},
{
"cell_type": "markdown",
"id": "4fb35d0d",
"id": "9e063f6d",
"metadata": {
"editable": true
},
@@ -207,7 +196,7 @@
},
{
"cell_type": "markdown",
"id": "608f9780",
"id": "e582c2bb",
"metadata": {
"editable": true
},
@@ -219,7 +208,7 @@
},
{
"cell_type": "markdown",
"id": "014bbc09",
"id": "59e0ed7a",
"metadata": {
"editable": true
},
@@ -230,7 +219,7 @@
},
{
"cell_type": "markdown",
"id": "f28863ad",
"id": "051e71aa",
"metadata": {
"editable": true
},
@@ -249,7 +238,7 @@
{
"cell_type": "code",
"execution_count": 2,
"id": "85f75a99",
"id": "c82d8b64",
"metadata": {
"collapsed": false,
"editable": true
@@ -265,7 +254,7 @@
},
{
"cell_type": "markdown",
"id": "bee8d814",
"id": "3e810bba",
"metadata": {
"editable": true
},
@@ -275,7 +264,7 @@
},
{
"cell_type": "markdown",
"id": "7c94f8bd",
"id": "c1198fce",
"metadata": {
"editable": true
},
@@ -286,7 +275,7 @@
},
{
"cell_type": "markdown",
"id": "c0e789dd",
"id": "1b0cea0a",
"metadata": {
"editable": true
},
@@ -298,7 +287,7 @@
},
{
"cell_type": "markdown",
"id": "5ee1d70e",
"id": "5194ba08",
"metadata": {
"editable": true
},
doc/LectureNotes/_build/jupyter_execute/exercisesweek34.py
+1
View File
@@ -83,6 +83,7 @@
# In[1]:
import numpy as np
x = np.random.rand(100,1)
y = 2.0+5*x*x+0.1*np.random.randn(100,1)
doc/LectureNotes/_build/jupyter_execute/week35.ipynb
+574 -824
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/jupyter_execute/week35.py
+15 -210
View File
@@ -6,9 +6,9 @@
# <!-- dom:TITLE: Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression -->
# # Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
# **Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
# **Morten Hjorth-Jensen**, Department of Physics, University of Oslo
#
# Date: **August 26-30**
# Date: **August 26-30, 2024**
# ## Plans for week 35
#
@@ -16,7 +16,7 @@
#
# 1. Brief repetition from last week
#
# 2. Derivation of the equations for ordinary least squares
# 2. Discussions of the equations for ordinary least squares
#
# 3. Discussion on how to prepare data and examples of applications of linear regression
#
@@ -26,11 +26,15 @@
# ### Reading recommendations:
#
# 1. See lecture notes for week 35 at <https://compphysics.github.io/MachineLearning/doc/web/course.html>
# 1. These lecture notes
#
# 2. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)
#
# 3. Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4.
#
# 4. For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Marc Peter Deisenroth on the Mathematics of Machine Learning, may be very relevant. In particular chapter 5 at URL"https://mml-book.github.io/" (section 5.5 on derivatives) is very useful for exercise 1 this coming week.
# ## Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week
# ## For exercise sessions: Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week
#
# We need first a reminder from last week about linear regression.
#
@@ -1264,206 +1268,7 @@ plt.show()
# intercept. This becomes more important when we discuss Ridge and Lasso
# regression next week.
# ## The Boston housing data example
#
# The Boston housing
# data set was originally a part of UCI Machine Learning Repository
# and has been removed now. The data set is now included in **Scikit-Learn**'s
# library. There are 506 samples and 13 feature (predictor) variables
# in this data set. The objective is to predict the value of prices of
# the house using the features (predictors) listed here.
#
# The features/predictors are
# 1. CRIM: Per capita crime rate by town
#
# 2. ZN: Proportion of residential land zoned for lots over 25000 square feet
#
# 3. INDUS: Proportion of non-retail business acres per town
#
# 4. CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
#
# 5. NOX: Nitric oxide concentration (parts per 10 million)
#
# 6. RM: Average number of rooms per dwelling
#
# 7. AGE: Proportion of owner-occupied units built prior to 1940
#
# 8. DIS: Weighted distances to five Boston employment centers
#
# 9. RAD: Index of accessibility to radial highways
#
# 10. TAX: Full-value property tax rate per USD10000
#
# 11. B: $1000(Bk - 0.63)^2$, where $Bk$ is the proportion of [people of African American descent] by town
#
# 12. LSTAT: Percentage of lower status of the population
#
# 13. MEDV: Median value of owner-occupied homes in USD 1000s
# ## Housing data, the code
# We start by importing the libraries
# In[15]:
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
# and load the Boston Housing DataSet from **Scikit-Learn**
# In[16]:
from sklearn.datasets import load_boston
boston_dataset = load_boston()
# boston_dataset is a dictionary
# let's check what it contains
boston_dataset.keys()
# Then we invoke Pandas
# In[17]:
boston = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)
boston.head()
boston['MEDV'] = boston_dataset.target
# and preprocess the data
# In[18]:
# check for missing values in all the columns
boston.isnull().sum()
# We can then visualize the data
# In[19]:
# set the size of the figure
sns.set(rc={'figure.figsize':(11.7,8.27)})
# plot a histogram showing the distribution of the target values
sns.distplot(boston['MEDV'], bins=30)
plt.show()
# It is now useful to look at the correlation matrix
# In[20]:
# compute the pair wise correlation for all columns
correlation_matrix = boston.corr().round(2)
# use the heatmap function from seaborn to plot the correlation matrix
# annot = True to print the values inside the square
sns.heatmap(data=correlation_matrix, annot=True)
# From the above coorelation plot we can see that **MEDV** is strongly correlated to **LSTAT** and **RM**. We see also that **RAD** and **TAX** are stronly correlated, but we don't include this in our features together to avoid multi-colinearity
# In[21]:
plt.figure(figsize=(20, 5))
features = ['LSTAT', 'RM']
target = boston['MEDV']
for i, col in enumerate(features):
plt.subplot(1, len(features) , i+1)
x = boston[col]
y = target
plt.scatter(x, y, marker='o')
plt.title(col)
plt.xlabel(col)
plt.ylabel('MEDV')
# Now we start training our model
# In[22]:
X = pd.DataFrame(np.c_[boston['LSTAT'], boston['RM']], columns = ['LSTAT','RM'])
Y = boston['MEDV']
# We split the data into training and test sets
# In[23]:
from sklearn.model_selection import train_test_split
# splits the training and test data set in 80% : 20%
# assign random_state to any value.This ensures consistency.
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2, random_state=5)
print(X_train.shape)
print(X_test.shape)
print(Y_train.shape)
print(Y_test.shape)
# Then we use the linear regression functionality from **Scikit-Learn**
# In[24]:
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
lin_model = LinearRegression()
lin_model.fit(X_train, Y_train)
# model evaluation for training set
y_train_predict = lin_model.predict(X_train)
rmse = (np.sqrt(mean_squared_error(Y_train, y_train_predict)))
r2 = r2_score(Y_train, y_train_predict)
print("The model performance for training set")
print("--------------------------------------")
print('RMSE is {}'.format(rmse))
print('R2 score is {}'.format(r2))
print("\n")
# model evaluation for testing set
y_test_predict = lin_model.predict(X_test)
# root mean square error of the model
rmse = (np.sqrt(mean_squared_error(Y_test, y_test_predict)))
# r-squared score of the model
r2 = r2_score(Y_test, y_test_predict)
print("The model performance for testing set")
print("--------------------------------------")
print('RMSE is {}'.format(rmse))
print('R2 score is {}'.format(r2))
# In[25]:
# plotting the y_test vs y_pred
# ideally should have been a straight line
plt.scatter(Y_test, y_test_predict)
plt.show()
# ## Material for lecture Thursday, August 31
# ## Material for lecture Monday, August 26
# ## Mathematical Interpretation of Ordinary Least Squares
#
@@ -1738,7 +1543,7 @@ plt.show()
# ## Codes for the SVD
# In[26]:
# In[15]:
import numpy as np
@@ -2155,7 +1960,7 @@ print(C-X)
# function **np.mean(x)**. We can also extract the eigenvalues of the
# covariance matrix through the **np.linalg.eig()** function.
# In[27]:
# In[16]:
# Importing various packages
@@ -2178,7 +1983,7 @@ print(C)
# code which sets up the correlations matrix for the previous example in
# a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the $2\times 2$ correlation matrix (since we have only two vectors).
# In[28]:
# In[17]:
import numpy as np
@@ -2214,7 +2019,7 @@ print(C)
#
# We whow here how we can set up the correlation matrix using **pandas**, as done in this simple code
# In[29]:
# In[18]:
import numpy as np
@@ -2237,7 +2042,7 @@ print(correlation_matrix)
# ## Correlation Matrix with Pandas and the Franke function
# In[30]:
# In[19]:
# Common imports
doc/LectureNotes/exercisesweek34.ipynb
+20 -19
View File
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "72dc0997",
"id": "a50ea987",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
"id": "8c591125",
"id": "291e6015",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "1ad21876",
"id": "141ff111",
"metadata": {
"editable": true
},
@@ -39,7 +39,7 @@
},
{
"cell_type": "markdown",
"id": "4eaa91e2",
"id": "32a1fc9e",
"metadata": {
"editable": true
},
@@ -108,7 +108,7 @@
},
{
"cell_type": "markdown",
"id": "360adb56",
"id": "f103f4c1",
"metadata": {
"editable": true
},
@@ -122,20 +122,21 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "278aa34f",
"id": "81c11edc",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"x = np.random.rand(100,1)\n",
"y = 2.0+5*x*x+0.1*np.random.randn(100,1)"
]
},
{
"cell_type": "markdown",
"id": "1ff20e45",
"id": "970bd604",
"metadata": {
"editable": true
},
@@ -149,7 +150,7 @@
},
{
"cell_type": "markdown",
"id": "e8809de1",
"id": "bda3ed04",
"metadata": {
"editable": true
},
@@ -162,7 +163,7 @@
},
{
"cell_type": "markdown",
"id": "8c9e5277",
"id": "ca995691",
"metadata": {
"editable": true
},
@@ -173,7 +174,7 @@
},
{
"cell_type": "markdown",
"id": "85a5881c",
"id": "27cbfeed",
"metadata": {
"editable": true
},
@@ -185,7 +186,7 @@
},
{
"cell_type": "markdown",
"id": "4fb35d0d",
"id": "9e063f6d",
"metadata": {
"editable": true
},
@@ -195,7 +196,7 @@
},
{
"cell_type": "markdown",
"id": "608f9780",
"id": "e582c2bb",
"metadata": {
"editable": true
},
@@ -207,7 +208,7 @@
},
{
"cell_type": "markdown",
"id": "014bbc09",
"id": "59e0ed7a",
"metadata": {
"editable": true
},
@@ -218,7 +219,7 @@
},
{
"cell_type": "markdown",
"id": "f28863ad",
"id": "051e71aa",
"metadata": {
"editable": true
},
@@ -237,7 +238,7 @@
{
"cell_type": "code",
"execution_count": 2,
"id": "85f75a99",
"id": "c82d8b64",
"metadata": {
"collapsed": false,
"editable": true
@@ -253,7 +254,7 @@
},
{
"cell_type": "markdown",
"id": "bee8d814",
"id": "3e810bba",
"metadata": {
"editable": true
},
@@ -263,7 +264,7 @@
},
{
"cell_type": "markdown",
"id": "7c94f8bd",
"id": "c1198fce",
"metadata": {
"editable": true
},
@@ -274,7 +275,7 @@
},
{
"cell_type": "markdown",
"id": "c0e789dd",
"id": "1b0cea0a",
"metadata": {
"editable": true
},
@@ -286,7 +287,7 @@
},
{
"cell_type": "markdown",
"id": "5ee1d70e",
"id": "5194ba08",
"metadata": {
"editable": true
},
doc/LectureNotes/week35.ipynb
+422 -782
View File
File diff suppressed because it is too large Load Diff