new update
This commit is contained in:
Morten Hjorth-Jensen
@@ -159,7 +159,7 @@
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter5.html">
|
||||
7. Support Vector Machines, overarching aims
|
||||
8. Support Vector Machines, overarching aims
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
@@ -171,12 +171,12 @@
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter6.html">
|
||||
8. Decision trees, overarching aims
|
||||
9. Decision trees, overarching aims
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter7.html">
|
||||
9. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
|
||||
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
@@ -188,12 +188,12 @@
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter8.html">
|
||||
10. Basic ideas of the Principal Component Analysis (PCA)
|
||||
11. Basic ideas of the Principal Component Analysis (PCA)
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="Clustering.html">
|
||||
11. Clustering Analysis
|
||||
12. Clustering Analysis
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
@@ -205,12 +205,12 @@
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter9.html">
|
||||
12. Neural networks
|
||||
13. Neural networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
13. Building a Feed Forward Neural Network
|
||||
14. Building a Feed Forward Neural Network
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
@@ -357,39 +357,34 @@
|
||||
3.7. Housing data, the code
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#id1">
|
||||
3.8. Splitting our Data in Training and Test data
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#reducing-the-number-of-degrees-of-freedom-overarching-view">
|
||||
3.9. Reducing the number of degrees of freedom, overarching view
|
||||
3.8. Reducing the number of degrees of freedom, overarching view
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#testing-the-means-squared-error-as-function-of-complexity">
|
||||
3.10. Testing the Means Squared Error as function of Complexity
|
||||
3.9. Testing the Means Squared Error as function of Complexity
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercises">
|
||||
3.11. Exercises
|
||||
3.10. Exercises
|
||||
</a>
|
||||
<ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-setting-up-various-python-environments">
|
||||
3.11.1. Exercise: Setting up various Python environments
|
||||
3.10.1. Exercise: Setting up various Python environments
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-making-your-own-data-and-exploring-scikit-learn">
|
||||
3.11.2. Exercise: making your own data and exploring scikit-learn
|
||||
3.10.2. Exercise: making your own data and exploring scikit-learn
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-normalizing-our-data">
|
||||
3.11.3. Exercise: Normalizing our data
|
||||
3.10.3. Exercise: Normalizing our data
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
@@ -409,35 +404,34 @@
|
||||
<h1><span class="section-number">3. </span>Linear Regression<a class="headerlink" href="#linear-regression" title="Permalink to this headline">¶</a></h1>
|
||||
<div class="section" id="introduction">
|
||||
<h2><span class="section-number">3.1. </span>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Our emphasis throughout this series of lectures<br />
|
||||
is on understanding the mathematical aspects of
|
||||
different algorithms used in the fields of data analysis and machine learning.</p>
|
||||
<p>However, where possible we will emphasize the
|
||||
importance of using available software. We start thus with a hands-on
|
||||
and top-down approach to machine learning. The aim is thus to start with
|
||||
relevant data or data we have produced
|
||||
and use these to introduce statistical data analysis
|
||||
concepts and machine learning algorithms before we delve into the
|
||||
algorithms themselves. The examples we will use in the beginning, start with simple
|
||||
polynomials with random noise added. We will use the Python
|
||||
software package <a class="reference external" href="http://scikit-learn.org/stable/">Scikit-Learn</a> and
|
||||
introduce various machine learning algorithms to make fits of
|
||||
the data and predictions. We move thereafter to more interesting
|
||||
cases such as data from say experiments (below we will look at experimental nuclear binding energies as an example).
|
||||
These are examples where we can easily set up the data and
|
||||
then use machine learning algorithms included in for example
|
||||
<strong>Scikit-Learn</strong>.</p>
|
||||
<p>Our emphasis throughout this series of lectures is on understanding
|
||||
the mathematical aspects of different algorithms used in the fields of
|
||||
data analysis and machine learning.</p>
|
||||
<p>However, where possible we will emphasize the importance of using
|
||||
available software. We start thus with a hands-on and top-down
|
||||
approach to machine learning. The aim is thus to start with relevant
|
||||
data or data we have produced and use these to introduce statistical
|
||||
data analysis concepts and machine learning algorithms before we delve
|
||||
into the algorithms themselves. The examples we will use in the
|
||||
beginning, start with simple polynomials with random noise added. We
|
||||
will use the Python software package
|
||||
<a class="reference external" href="http://scikit-learn.org/stable/">Scikit-Learn</a> and introduce various
|
||||
machine learning algorithms to make fits of the data and
|
||||
predictions. We move thereafter to more interesting cases such as data
|
||||
from say experiments (below we will look at experimental nuclear
|
||||
binding energies as an example). These are examples where we can
|
||||
easily set up the data and then use machine learning algorithms
|
||||
included in for example <strong>Scikit-Learn</strong>.</p>
|
||||
<p>These examples will serve us the purpose of getting
|
||||
started. Furthermore, they allow us to catch more than two birds with
|
||||
a stone. They will allow us to bring in some programming specific
|
||||
topics and tools as well as showing the power of various Python
|
||||
libraries for machine learning and statistical data analysis.</p>
|
||||
<p>Here, we will mainly focus on two
|
||||
specific Python packages for Machine Learning, Scikit-Learn and
|
||||
Tensorflow (see below for links etc). Moreover, the examples we
|
||||
introduce will serve as inputs to many of our discussions later, as
|
||||
well as allowing you to set up models and produce your own data and
|
||||
get started with programming.</p>
|
||||
<p>Here, we will mainly focus on two specific Python packages for Machine
|
||||
Learning, Scikit-Learn and Tensorflow (see below for links etc).
|
||||
Moreover, the examples we introduce will serve as inputs to many of
|
||||
our discussions later, as well as allowing you to set up models and
|
||||
produce your own data and get started with programming.</p>
|
||||
</div>
|
||||
<div class="section" id="what-is-machine-learning">
|
||||
<h2><span class="section-number">3.2. </span>What is Machine Learning?<a class="headerlink" href="#what-is-machine-learning" title="Permalink to this headline">¶</a></h2>
|
||||
@@ -625,7 +619,15 @@ quality figures. Feel free to explore the extensive
|
||||
<a class="reference external" href="https://matplotlib.org/gallery/index.html">gallery</a> of examples. In
|
||||
this example we plot our original values of <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> as well as the
|
||||
prediction <strong>ypredict</strong> (<span class="math notranslate nohighlight">\(\tilde{y}\)</span>), which attempts at fitting our
|
||||
data with a straight line.</p>
|
||||
data with a straight line. Note also that <strong>Scikit-Learn</strong> requires a
|
||||
matrix as input for the input values <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span>. In the above code we
|
||||
have solved this by declaring <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> as arrays of dimension
|
||||
<span class="math notranslate nohighlight">\(n\times 1\)</span>.</p>
|
||||
<p>In the code here we have also made a new array for <span class="math notranslate nohighlight">\(x\in [0,1]\)</span>. Our
|
||||
prediction is computed for these values, meaning that they were not
|
||||
included in the data set used to <em>train</em> (or fit) the model.
|
||||
This is a recurrring theme in machine learning and data analysis. We would like to train a model on a specific given data set.
|
||||
Thereafter we wish to apply it to data which were not included in the training. Below we will encounter this again in the so-called <em>train-validate-test</em> spliting. We will typically split our data into different sets, oen for training, one for validation and finally, our data from the untouched test vault!</p>
|
||||
<p>The Python code follows here.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
@@ -640,6 +642,7 @@ data with a straight line.</p>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mi">2</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">linreg</span> <span class="o">=</span> <span class="n">LinearRegression</span><span class="p">()</span>
|
||||
<span class="n">linreg</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">)</span>
|
||||
<span class="c1"># This is our new x-array to which we test our model</span>
|
||||
<span class="n">xnew</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="mi">0</span><span class="p">],[</span><span class="mi">1</span><span class="p">]])</span>
|
||||
<span class="n">ypredict</span> <span class="o">=</span> <span class="n">linreg</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">xnew</span><span class="p">)</span>
|
||||
|
||||
@@ -745,8 +748,7 @@ to be dominated by outliers.</p>
|
||||
<p>Depending on the parameter in front of the normal distribution, we may
|
||||
have a small or larger relative error. Try to play around with
|
||||
different training data sets and study (graphically) the value of the
|
||||
relative error. Note also that <strong>Scikit-Learn</strong> requires a matrix as input for the input values <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span>. In the above code we have
|
||||
solved this by declaring <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> as arrays of dimension <span class="math notranslate nohighlight">\(n\times 1\)</span>.</p>
|
||||
relative error.</p>
|
||||
<p>As mentioned above, <strong>Scikit-Learn</strong> has an impressive functionality.
|
||||
We can for example extract the values of <span class="math notranslate nohighlight">\(\alpha\)</span> and <span class="math notranslate nohighlight">\(\beta\)</span> and
|
||||
their error estimates, or the variance and standard deviation and many
|
||||
@@ -787,13 +789,13 @@ example of the functionality of <strong>Scikit-Learn</strong>.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The intercept alpha:
|
||||
[1.84860939]
|
||||
[1.89992521]
|
||||
Coefficient beta :
|
||||
[[5.2571699]]
|
||||
Mean squared error: 0.25
|
||||
Variance score: 0.90
|
||||
[[5.12908963]]
|
||||
Mean squared error: 0.20
|
||||
Variance score: 0.92
|
||||
Mean squared log error: 0.01
|
||||
Mean absolute error: 0.40
|
||||
Mean absolute error: 0.36
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_13_1.png" src="_images/chapter1_13_1.png" />
|
||||
@@ -893,7 +895,7 @@ a linear <span class="math notranslate nohighlight">\(x\)</span>-dependence we s
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter1_27_0.png" src="_images/chapter1_27_0.png" />
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.0050000000000000044
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.005000000000000007
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1117,7 +1119,7 @@ A
|
||||
</div>
|
||||
</div>
|
||||
<p>The next step, and we will define this mathematically later, is to set up the so-called <strong>design matrix</strong>. We will throughout call this matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.
|
||||
It has dimensionality <span class="math notranslate nohighlight">\(p\times n\)</span>, where <span class="math notranslate nohighlight">\(n\)</span> is the number of data points and <span class="math notranslate nohighlight">\(p\)</span> are the so-called predictors. In our case here they are given by the number of polynomials in <span class="math notranslate nohighlight">\(A\)</span> we wish to include in the fit.</p>
|
||||
It has dimensionality <span class="math notranslate nohighlight">\(n\times p\)</span>, where <span class="math notranslate nohighlight">\(n\)</span> is the number of data points and <span class="math notranslate nohighlight">\(p\)</span> are the so-called predictors. In our case here they are given by the number of polynomials in <span class="math notranslate nohighlight">\(A\)</span> we wish to include in the fit.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Now we set up the design matrix X</span>
|
||||
@@ -1131,8 +1133,11 @@ It has dimensionality <span class="math notranslate nohighlight">\(p\times n\)</
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Note well that we have made life simple here. We perform a fit in terms of the number of nucleons only. A more sophisticated fit can be done by including an explicit dependence on the number of protons and neutrons in the asymmetry and Coulomb terms.</p>
|
||||
<p>With <strong>scikitlearn</strong> we are now ready to use linear regression and fit our data.</p>
|
||||
<p>Note well that we have made life simple here. We perform a fit in
|
||||
terms of the number of nucleons only. A more sophisticated fit can be
|
||||
done by including an explicit dependence on the number of protons and
|
||||
neutrons in the asymmetry and Coulomb terms. We leave this as an exercise to you the reader.</p>
|
||||
<p>With <strong>Scikit-Learn</strong> we are now ready to use linear regression and fit our data.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">clf</span> <span class="o">=</span> <span class="n">skl</span><span class="o">.</span><span class="n">LinearRegression</span><span class="p">()</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Energies</span><span class="p">)</span>
|
||||
@@ -1176,7 +1181,7 @@ Mean absolute error: 0.05
|
||||
<img alt="_images/chapter1_57_1.png" src="_images/chapter1_57_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>As a teaser, let us now see how we can do this with decision trees using <strong>scikit-learn</strong>. Later we will switch to so-called <strong>random forests</strong>!</p>
|
||||
<p>As a teaser, let us now see how we can do this with decision trees using <strong>Scikit-Learn</strong>. Later we will switch to so-called <strong>random forests</strong>!</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1">#Decision Tree Regression</span>
|
||||
@@ -1228,11 +1233,20 @@ A
|
||||
270 3344 160 110 270 Ds 7.253775 7.253775
|
||||
|
||||
[267 rows x 6 columns]
|
||||
0.009883615646716186
|
||||
0.009883615646716188
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>With a deeper and deeper tree level, we can almost reproduce every
|
||||
single data point by increasing the max depth of the tree.
|
||||
We can actually decide to make a decision tree which fits every single point.
|
||||
As we will
|
||||
see later, this has the benefit that we can really train a model which
|
||||
traverses every single data point. However, the price we pay is that
|
||||
we will easily overfit. That is, if we apply our model to unseen data,
|
||||
we will most likely fail miserably in our attempt at making
|
||||
predictions. As an exercise, try to make the tree level larger by adjusting the maximum depth variable. When printing out the predicition, you will note that the binding energy of every nucleus is accurately reproduced.</p>
|
||||
<p>The <strong>seaborn</strong> package allows us to visualize data in an efficient way. Note that we use <strong>scikit-learn</strong>’s multi-layer perceptron (or feed forward neural network)
|
||||
functionality.</p>
|
||||
<div class="cell docutils container">
|
||||
@@ -1270,14 +1284,6 @@ functionality.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
@@ -1334,16 +1340,12 @@ functionality.</p>
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
@@ -1360,7 +1362,11 @@ functionality.</p>
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_61_11.png" src="_images/chapter1_61_11.png" />
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_61_10.png" src="_images/chapter1_61_10.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1878,6 +1884,22 @@ K</p>
|
||||
\[
|
||||
\frac{\partial\log{\vert\boldsymbol{A}\vert}}{\partial \boldsymbol{A}}=(\boldsymbol{A}^{-1})^T.
|
||||
\]</div>
|
||||
<p>We can then compute the second derivative of the cost function, which in our case is the second derivative
|
||||
of the means squared error. This leads to</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\]</div>
|
||||
<p>This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).</p>
|
||||
<p>The Hessian matrix plays an important role and is defined for the mean squared error as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\]</div>
|
||||
<p>The Hessian matrix for ordinary least squares is also proportional to
|
||||
the covariance matrix. As we will see in the chapter on Ridge and Lasso regression, This means that we can use the Singular Value Decomposition of a matrix to find
|
||||
the eigenvalues of the covariance matrix and the Hessian matrix in
|
||||
terms of the singular values.</p>
|
||||
<p>The residuals <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> are in turn given by</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
@@ -1935,7 +1957,7 @@ write</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter1_132_0.png" src="_images/chapter1_132_0.png" />
|
||||
<img alt="_images/chapter1_136_0.png" src="_images/chapter1_136_0.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>We can easily test our fit by computing the <span class="math notranslate nohighlight">\(R2\)</span> score that we discussed in connection with the functionality of <strong>Scikit-Learn</strong> in the introductory slides.
|
||||
@@ -2138,9 +2160,7 @@ time the file is presented as a standard <strong>csv</strong> file.</p>
|
||||
before, with the same initializations and declarations. We use also
|
||||
<strong>pandas</strong> again, rather extensively in order to organize our data.</p>
|
||||
<p>The difference now is that we use <strong>Scikit-Learn’s</strong> regression tools
|
||||
instead of our own matrix inversion implementation. Furthermore, we
|
||||
sneak in <strong>Ridge</strong> regression (to be discussed below) which includes a
|
||||
hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span>, also to be explained below.</p>
|
||||
instead of our own matrix inversion implementation.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Common imports</span>
|
||||
@@ -2203,18 +2223,6 @@ hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span>, al
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Mean absolute error: </span><span class="si">%.2f</span><span class="s1">'</span> <span class="o">%</span> <span class="n">mean_absolute_error</span><span class="p">(</span><span class="n">Energies</span><span class="p">,</span> <span class="n">ytilde</span><span class="p">))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="n">coef_</span><span class="p">,</span> <span class="n">clf</span><span class="o">.</span><span class="n">intercept_</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># The Ridge regression with a hyperparameter lambda = 0.1</span>
|
||||
<span class="n">_lambda</span> <span class="o">=</span> <span class="mf">0.1</span>
|
||||
<span class="n">clf_ridge</span> <span class="o">=</span> <span class="n">skl</span><span class="o">.</span><span class="n">Ridge</span><span class="p">(</span><span class="n">alpha</span><span class="o">=</span><span class="n">_lambda</span><span class="p">)</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Energies</span><span class="p">)</span>
|
||||
<span class="n">yridge</span> <span class="o">=</span> <span class="n">clf_ridge</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
|
||||
<span class="n">EoS</span><span class="p">[</span><span class="s1">'Eridge'</span><span class="p">]</span> <span class="o">=</span> <span class="n">yridge</span>
|
||||
<span class="c1"># The mean squared error </span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Mean squared error: </span><span class="si">%.2f</span><span class="s2">"</span> <span class="o">%</span> <span class="n">mean_squared_error</span><span class="p">(</span><span class="n">Energies</span><span class="p">,</span> <span class="n">yridge</span><span class="p">))</span>
|
||||
<span class="c1"># Explained variance score: 1 is perfect prediction </span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Variance score: </span><span class="si">%.2f</span><span class="s1">'</span> <span class="o">%</span> <span class="n">r2_score</span><span class="p">(</span><span class="n">Energies</span><span class="p">,</span> <span class="n">yridge</span><span class="p">))</span>
|
||||
<span class="c1"># Mean absolute error </span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Mean absolute error: </span><span class="si">%.2f</span><span class="s1">'</span> <span class="o">%</span> <span class="n">mean_absolute_error</span><span class="p">(</span><span class="n">Energies</span><span class="p">,</span> <span class="n">yridge</span><span class="p">))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">clf_ridge</span><span class="o">.</span><span class="n">coef_</span><span class="p">,</span> <span class="n">clf_ridge</span><span class="o">.</span><span class="n">intercept_</span><span class="p">)</span>
|
||||
|
||||
<span class="n">fig</span><span class="p">,</span> <span class="n">ax</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">subplots</span><span class="p">()</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">set_xlabel</span><span class="p">(</span><span class="sa">r</span><span class="s1">'$\rho[\mathrm</span><span class="si">{fm}</span><span class="s1">^{-3}]$'</span><span class="p">)</span>
|
||||
@@ -2223,8 +2231,6 @@ hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span>, al
|
||||
<span class="n">label</span><span class="o">=</span><span class="s1">'Theoretical data'</span><span class="p">)</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">EoS</span><span class="p">[</span><span class="s1">'Density'</span><span class="p">],</span> <span class="n">EoS</span><span class="p">[</span><span class="s1">'Eols'</span><span class="p">],</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.7</span><span class="p">,</span> <span class="n">lw</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">c</span><span class="o">=</span><span class="s1">'m'</span><span class="p">,</span>
|
||||
<span class="n">label</span><span class="o">=</span><span class="s1">'OLS'</span><span class="p">)</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">EoS</span><span class="p">[</span><span class="s1">'Density'</span><span class="p">],</span> <span class="n">EoS</span><span class="p">[</span><span class="s1">'Eridge'</span><span class="p">],</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.7</span><span class="p">,</span> <span class="n">lw</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">c</span><span class="o">=</span><span class="s1">'g'</span><span class="p">,</span>
|
||||
<span class="n">label</span><span class="o">=</span><span class="s1">'Ridge $\lambda = 0.1$'</span><span class="p">)</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
|
||||
<span class="n">save_fig</span><span class="p">(</span><span class="s2">"EoSfitting"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
|
||||
@@ -2236,20 +2242,13 @@ hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span>, al
|
||||
Variance score: 1.00
|
||||
Mean absolute error: 2.83
|
||||
[ 0. 618.32047562 -861.13519106 1404.91549644] -11.057088709963637
|
||||
Mean squared error: 197.93
|
||||
Variance score: 1.00
|
||||
Mean absolute error: 11.69
|
||||
[ 0. 28.18220995 282.79902342 842.30879705] 12.946893955211749
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_179_1.png" src="_images/chapter1_179_1.png" />
|
||||
<img alt="_images/chapter1_183_1.png" src="_images/chapter1_183_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>The above simple polynomial in density <span class="math notranslate nohighlight">\(\rho\)</span> gives an excellent fit
|
||||
to the data.</p>
|
||||
<p>We note also that there is a small deviation between the
|
||||
standard OLS and the Ridge regression at higher densities. We discuss this in more detail
|
||||
below.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="splitting-our-data-in-training-and-test-data">
|
||||
@@ -2264,6 +2263,95 @@ postpone a discussion of this splitting to the end of these notes and
|
||||
our discussion of the so-called <strong>bias-variance</strong> tradeoff. Here we
|
||||
limit ourselves to repeat the above equation of state fitting example
|
||||
but now splitting the data into a training set and a test set.</p>
|
||||
<p>Let us study some examples. The first code here takes a simple
|
||||
one-dimensional second-order polynomial and we fit it to a
|
||||
second-order polynomial. Depending on the strength of the added noise,
|
||||
the various measures like the <span class="math notranslate nohighlight">\(R2\)</span> score or the mean-squared error,
|
||||
the fit becomes better or worse.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">R2</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span> <span class="n">y_model</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">y_model</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y_data</span><span class="p">))</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
|
||||
<span class="k">def</span> <span class="nf">MSE</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span><span class="n">y_model</span><span class="p">):</span>
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">y_model</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span><span class="o">-</span><span class="n">y_model</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
|
||||
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">5</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mf">0.1</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="c1"># The design matrix now as function of a given polynomial</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">),</span><span class="mi">3</span><span class="p">))</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mf">1.0</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="mi">2</span>
|
||||
<span class="c1"># We split the data in test and training data</span>
|
||||
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
|
||||
<span class="c1"># matrix inversion to find beta</span>
|
||||
<span class="n">beta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X_train</span><span class="p">)</span> <span class="o">@</span> <span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y_train</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">beta</span><span class="p">)</span>
|
||||
<span class="c1"># and then make the prediction</span>
|
||||
<span class="n">ytilde</span> <span class="o">=</span> <span class="n">X_train</span> <span class="o">@</span> <span class="n">beta</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Training R2"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">R2</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytilde</span><span class="p">))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Training MSE"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytilde</span><span class="p">))</span>
|
||||
<span class="n">ypredict</span> <span class="o">=</span> <span class="n">X_test</span> <span class="o">@</span> <span class="n">beta</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Test R2"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">R2</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredict</span><span class="p">))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Test MSE"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredict</span><span class="p">))</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 2.00247083e+00 -3.54710530e-03 5.02096350e+00]
|
||||
Training R2
|
||||
0.99620088364924
|
||||
Training MSE
|
||||
0.008171076530362356
|
||||
Test R2
|
||||
0.9968966158779216
|
||||
Test MSE
|
||||
0.006530396683779109
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Alternatively, you could write your own test-train splitting function as shown here.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># equivalently in numpy</span>
|
||||
<span class="k">def</span> <span class="nf">train_test_split_numpy</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">labels</span><span class="p">,</span> <span class="n">train_size</span><span class="p">,</span> <span class="n">test_size</span><span class="p">):</span>
|
||||
<span class="n">n_inputs</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">)</span>
|
||||
<span class="n">inputs_shuffled</span> <span class="o">=</span> <span class="n">inputs</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
|
||||
<span class="n">labels_shuffled</span> <span class="o">=</span> <span class="n">labels</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
|
||||
|
||||
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">shuffle</span><span class="p">(</span><span class="n">inputs_shuffled</span><span class="p">)</span>
|
||||
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">shuffle</span><span class="p">(</span><span class="n">labels_shuffled</span><span class="p">)</span>
|
||||
|
||||
<span class="n">train_end</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n_inputs</span><span class="o">*</span><span class="n">train_size</span><span class="p">)</span>
|
||||
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span> <span class="o">=</span> <span class="n">inputs_shuffled</span><span class="p">[:</span><span class="n">train_end</span><span class="p">],</span> <span class="n">inputs_shuffled</span><span class="p">[</span><span class="n">train_end</span><span class="p">:]</span>
|
||||
<span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_test</span> <span class="o">=</span> <span class="n">labels_shuffled</span><span class="p">[:</span><span class="n">train_end</span><span class="p">],</span> <span class="n">labels_shuffled</span><span class="p">[</span><span class="n">train_end</span><span class="p">:]</span>
|
||||
|
||||
<span class="k">return</span> <span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_test</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>But since <strong>scikit-learn</strong> has its own function for doing this and since
|
||||
it interfaces easily with <strong>tensorflow</strong> and other libraries, we
|
||||
normally recommend using the latter functionality.</p>
|
||||
<p>As another example, we apply the training and testing split to
|
||||
to the above equation of state fitting example
|
||||
but now splitting the data into a training set and a test set.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
@@ -2335,13 +2423,13 @@ but now splitting the data into a training set and a test set.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Training R2
|
||||
0.9999869956119286
|
||||
0.9999850282256434
|
||||
Training MSE
|
||||
5.745136489050356
|
||||
6.653725417896576
|
||||
Test R2
|
||||
0.9999787681537219
|
||||
0.9999887274726137
|
||||
Test MSE
|
||||
9.712199063818309
|
||||
5.071447584032141
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2460,7 +2548,7 @@ dtype: int64
|
||||
warnings.warn(msg, FutureWarning)
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_191_1.png" src="_images/chapter1_191_1.png" />
|
||||
<img alt="_images/chapter1_199_1.png" src="_images/chapter1_199_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>It is now useful to look at the correlation matrix</p>
|
||||
@@ -2478,7 +2566,7 @@ dtype: int64
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><AxesSubplot:>
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_193_1.png" src="_images/chapter1_193_1.png" />
|
||||
<img alt="_images/chapter1_201_1.png" src="_images/chapter1_201_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>From the above coorelation plot we can see that <strong>MEDV</strong> is strongly correlated to <strong>LSTAT</strong> and <strong>RM</strong>. We see also that <strong>RAD</strong> and <strong>TAX</strong> are stronly correlated, but we don’t include this in our features together to avoid multi-colinearity</p>
|
||||
@@ -2501,7 +2589,7 @@ dtype: int64
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter1_195_0.png" src="_images/chapter1_195_0.png" />
|
||||
<img alt="_images/chapter1_203_0.png" src="_images/chapter1_203_0.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>Now we start training our model</p>
|
||||
@@ -2599,111 +2687,12 @@ R2 score is 0.6628996975186952
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter1_202_0.png" src="_images/chapter1_202_0.png" />
|
||||
<img alt="_images/chapter1_210_0.png" src="_images/chapter1_210_0.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="id1">
|
||||
<h2><span class="section-number">3.8. </span>Splitting our Data in Training and Test data<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h2>
|
||||
<p>It is normal in essentially all Machine Learning studies to split the
|
||||
data in a training set and a test set (sometimes also an additional
|
||||
validation set). <strong>Scikit-Learn</strong> has an own function for this. There
|
||||
is no explicit recipe for how much data should be included as training
|
||||
data and say test data. An accepted rule of thumb is to use
|
||||
approximately <span class="math notranslate nohighlight">\(2/3\)</span> to <span class="math notranslate nohighlight">\(4/5\)</span> of the data as training data. We will
|
||||
postpone a discussion of this splitting to the end of these notes and
|
||||
our discussion of the so-called <strong>bias-variance</strong> tradeoff. Here we
|
||||
limit ourselves to repeat the above equation of state fitting example
|
||||
but now splitting the data into a training set and a test set.</p>
|
||||
<p>Let us study some examples. The first code here takes a simple
|
||||
one-dimensional second-order polynomial and we fit it to a
|
||||
second-order polynomial. Depending on the strength of the added noise,
|
||||
the various measures like the <span class="math notranslate nohighlight">\(R2\)</span> score or the mean-squared error,
|
||||
the fit becomes better or worse.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">R2</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span> <span class="n">y_model</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">y_model</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y_data</span><span class="p">))</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
|
||||
<span class="k">def</span> <span class="nf">MSE</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span><span class="n">y_model</span><span class="p">):</span>
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">y_model</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span><span class="o">-</span><span class="n">y_model</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
|
||||
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">5</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mf">0.1</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="c1"># The design matrix now as function of a given polynomial</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">),</span><span class="mi">3</span><span class="p">))</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mf">1.0</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="mi">2</span>
|
||||
<span class="c1"># We split the data in test and training data</span>
|
||||
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
|
||||
<span class="c1"># matrix inversion to find beta</span>
|
||||
<span class="n">beta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X_train</span><span class="p">)</span> <span class="o">@</span> <span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y_train</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">beta</span><span class="p">)</span>
|
||||
<span class="c1"># and then make the prediction</span>
|
||||
<span class="n">ytilde</span> <span class="o">=</span> <span class="n">X_train</span> <span class="o">@</span> <span class="n">beta</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Training R2"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">R2</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytilde</span><span class="p">))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Training MSE"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytilde</span><span class="p">))</span>
|
||||
<span class="n">ypredict</span> <span class="o">=</span> <span class="n">X_test</span> <span class="o">@</span> <span class="n">beta</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Test R2"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">R2</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredict</span><span class="p">))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Test MSE"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredict</span><span class="p">))</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 2.05802766 -0.30626706 5.27262964]
|
||||
Training R2
|
||||
0.9967292151090247
|
||||
Training MSE
|
||||
0.007931713723267314
|
||||
Test R2
|
||||
0.9966744029509663
|
||||
Test MSE
|
||||
0.006968283076248407
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Alternatively, you could write your own test-train splitting function as shown here.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># equivalently in numpy</span>
|
||||
<span class="k">def</span> <span class="nf">train_test_split_numpy</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">labels</span><span class="p">,</span> <span class="n">train_size</span><span class="p">,</span> <span class="n">test_size</span><span class="p">):</span>
|
||||
<span class="n">n_inputs</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">)</span>
|
||||
<span class="n">inputs_shuffled</span> <span class="o">=</span> <span class="n">inputs</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
|
||||
<span class="n">labels_shuffled</span> <span class="o">=</span> <span class="n">labels</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
|
||||
|
||||
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">shuffle</span><span class="p">(</span><span class="n">inputs_shuffled</span><span class="p">)</span>
|
||||
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">shuffle</span><span class="p">(</span><span class="n">labels_shuffled</span><span class="p">)</span>
|
||||
|
||||
<span class="n">train_end</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n_inputs</span><span class="o">*</span><span class="n">train_size</span><span class="p">)</span>
|
||||
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span> <span class="o">=</span> <span class="n">inputs_shuffled</span><span class="p">[:</span><span class="n">train_end</span><span class="p">],</span> <span class="n">inputs_shuffled</span><span class="p">[</span><span class="n">train_end</span><span class="p">:]</span>
|
||||
<span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_test</span> <span class="o">=</span> <span class="n">labels_shuffled</span><span class="p">[:</span><span class="n">train_end</span><span class="p">],</span> <span class="n">labels_shuffled</span><span class="p">[</span><span class="n">train_end</span><span class="p">:]</span>
|
||||
|
||||
<span class="k">return</span> <span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_test</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>But since <strong>scikit-learn</strong> has its own function for doing this and since
|
||||
it interfaces easily with <strong>tensorflow</strong> and other libraries, we
|
||||
normally recommend using the latter functionality.</p>
|
||||
</div>
|
||||
<div class="section" id="reducing-the-number-of-degrees-of-freedom-overarching-view">
|
||||
<h2><span class="section-number">3.9. </span>Reducing the number of degrees of freedom, overarching view<a class="headerlink" href="#reducing-the-number-of-degrees-of-freedom-overarching-view" title="Permalink to this headline">¶</a></h2>
|
||||
<h2><span class="section-number">3.8. </span>Reducing the number of degrees of freedom, overarching view<a class="headerlink" href="#reducing-the-number-of-degrees-of-freedom-overarching-view" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Many Machine Learning problems involve thousands or even millions of
|
||||
features for each training instance. Not only does this make training
|
||||
extremely slow, it can also make it much harder to find a good
|
||||
@@ -3156,7 +3145,7 @@ x_j^{(i)} \rightarrow (b-a)\frac{x_j^{(i)} - \min(x_j)}{\max(x_j) - \min(x_j)} -
|
||||
<p>where <span class="math notranslate nohighlight">\(\min(x_j)\)</span> and <span class="math notranslate nohighlight">\(\max(x_j)\)</span> return the minimum and maximum value of <span class="math notranslate nohighlight">\(x_j\)</span> over the data set, respectively.</p>
|
||||
</div>
|
||||
<div class="section" id="testing-the-means-squared-error-as-function-of-complexity">
|
||||
<h2><span class="section-number">3.10. </span>Testing the Means Squared Error as function of Complexity<a class="headerlink" href="#testing-the-means-squared-error-as-function-of-complexity" title="Permalink to this headline">¶</a></h2>
|
||||
<h2><span class="section-number">3.9. </span>Testing the Means Squared Error as function of Complexity<a class="headerlink" href="#testing-the-means-squared-error-as-function-of-complexity" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Before we proceed with a more detailed analysis of the so-called
|
||||
Bias-Variance tradeoff, we present here an example of the relation
|
||||
between model complexity and the mean squared error for the triaining
|
||||
@@ -3208,14 +3197,14 @@ ourmodel (here in terms of the polynomial degree of the model).</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter1_214_0.png" src="_images/chapter1_214_0.png" />
|
||||
<img alt="_images/chapter1_218_0.png" src="_images/chapter1_218_0.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="exercises">
|
||||
<h2><span class="section-number">3.11. </span>Exercises<a class="headerlink" href="#exercises" title="Permalink to this headline">¶</a></h2>
|
||||
<h2><span class="section-number">3.10. </span>Exercises<a class="headerlink" href="#exercises" title="Permalink to this headline">¶</a></h2>
|
||||
<div class="section" id="exercise-setting-up-various-python-environments">
|
||||
<h3><span class="section-number">3.11.1. </span>Exercise: Setting up various Python environments<a class="headerlink" href="#exercise-setting-up-various-python-environments" title="Permalink to this headline">¶</a></h3>
|
||||
<h3><span class="section-number">3.10.1. </span>Exercise: Setting up various Python environments<a class="headerlink" href="#exercise-setting-up-various-python-environments" title="Permalink to this headline">¶</a></h3>
|
||||
<p>The first exercise here is of a mere technical art. We want you to have</p>
|
||||
<ul class="simple">
|
||||
<li><p>git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo <a class="reference external" href="https://www.uio.no/tjenester/it/maskin/filer/versjonskontroll/github.html">GitHub facilities</a>.</p></li>
|
||||
@@ -3271,7 +3260,7 @@ license.</p>
|
||||
<p>We recommend using <strong>Anaconda</strong> if you are not too familiar with setting paths in a terminal environment.</p>
|
||||
</div>
|
||||
<div class="section" id="exercise-making-your-own-data-and-exploring-scikit-learn">
|
||||
<h3><span class="section-number">3.11.2. </span>Exercise: making your own data and exploring scikit-learn<a class="headerlink" href="#exercise-making-your-own-data-and-exploring-scikit-learn" title="Permalink to this headline">¶</a></h3>
|
||||
<h3><span class="section-number">3.10.2. </span>Exercise: making your own data and exploring scikit-learn<a class="headerlink" href="#exercise-making-your-own-data-and-exploring-scikit-learn" title="Permalink to this headline">¶</a></h3>
|
||||
<p>We will generate our own dataset for a function <span class="math notranslate nohighlight">\(y(x)\)</span> where <span class="math notranslate nohighlight">\(x \in [0,1]\)</span> and defined by random numbers computed with the uniform distribution. The function <span class="math notranslate nohighlight">\(y\)</span> is a quadratic polynomial in <span class="math notranslate nohighlight">\(x\)</span> with added stochastic noise according to the normal distribution <span class="math notranslate nohighlight">\(\cal {N}(0,1)\)</span>.
|
||||
The following simple Python instructions define our <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> values (with 100 data points).</p>
|
||||
<div class="cell docutils container">
|
||||
@@ -3307,7 +3296,7 @@ R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i
|
||||
Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.</p>
|
||||
</div>
|
||||
<div class="section" id="exercise-normalizing-our-data">
|
||||
<h3><span class="section-number">3.11.3. </span>Exercise: Normalizing our data<a class="headerlink" href="#exercise-normalizing-our-data" title="Permalink to this headline">¶</a></h3>
|
||||
<h3><span class="section-number">3.10.3. </span>Exercise: Normalizing our data<a class="headerlink" href="#exercise-normalizing-our-data" title="Permalink to this headline">¶</a></h3>
|
||||
<p>A much used approach before starting to train the data is to preprocess our
|
||||
data. Normally the data may need a rescaling and/or may be sensitive
|
||||
to extreme values. Scaling the data renders our inputs much more
|
||||
|
||||
Reference in New Issue
Block a user