updating figures
This commit is contained in:
Morten Hjorth-Jensen
@@ -54,7 +54,7 @@
|
||||
<script async="async" src="_static/sphinx-thebe.js"></script>
|
||||
<link rel="index" title="Index" href="genindex.html" />
|
||||
<link rel="search" title="Search" href="search.html" />
|
||||
<link rel="next" title="4. Resampling Methods" href="chapter2.html" />
|
||||
<link rel="next" title="4. Ridge and Lasso Regression" href="chapter2.html" />
|
||||
<link rel="prev" title="2. Linear Algebra, Handling of Arrays and more Python Features" href="linalg.html" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<meta name="docsearch:language" content="en" />
|
||||
@@ -144,12 +144,12 @@
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter2.html">
|
||||
4. Resampling Methods
|
||||
4. Ridge and Lasso Regression
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter3.html">
|
||||
5. Ridge and Lasso Regression
|
||||
5. Resampling Methods
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
@@ -358,35 +358,38 @@
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#reducing-the-number-of-degrees-of-freedom-overarching-view">
|
||||
3.8. Reducing the number of degrees of freedom, overarching view
|
||||
<a class="reference internal nav-link" href="#id1">
|
||||
3.8. Splitting our Data in Training and Test data
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#reducing-the-number-of-degrees-of-freedom-overarching-view">
|
||||
3.9. Reducing the number of degrees of freedom, overarching view
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#testing-the-means-squared-error-as-function-of-complexity">
|
||||
3.10. Testing the Means Squared Error as function of Complexity
|
||||
</a>
|
||||
<ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#simple-preprocessing-examples-franke-function-and-regression">
|
||||
3.8.1. Simple preprocessing examples, Franke function and regression
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercises">
|
||||
3.9. Exercises
|
||||
3.11. Exercises
|
||||
</a>
|
||||
<ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-setting-up-various-python-environments">
|
||||
3.9.1. Exercise: Setting up various Python environments
|
||||
3.11.1. Exercise: Setting up various Python environments
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-making-your-own-data-and-exploring-scikit-learn">
|
||||
3.9.2. Exercise: making your own data and exploring scikit-learn
|
||||
3.11.2. Exercise: making your own data and exploring scikit-learn
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-normalizing-our-data">
|
||||
3.9.3. Exercise: Normalizing our data
|
||||
3.11.3. Exercise: Normalizing our data
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
@@ -404,10 +407,9 @@
|
||||
|
||||
<div class="section" id="linear-regression">
|
||||
<h1><span class="section-number">3. </span>Linear Regression<a class="headerlink" href="#linear-regression" title="Permalink to this headline">¶</a></h1>
|
||||
<p><a class="reference external" href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/LectureAug21.mp4?vrtx=view-as-webpage">Video of Lecture</a></p>
|
||||
<div class="section" id="introduction">
|
||||
<h2><span class="section-number">3.1. </span>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Our emphasis throughout this series of lectures (small change)<br />
|
||||
<p>Our emphasis throughout this series of lectures<br />
|
||||
is on understanding the mathematical aspects of
|
||||
different algorithms used in the fields of data analysis and machine learning.</p>
|
||||
<p>However, where possible we will emphasize the
|
||||
@@ -585,7 +587,9 @@ may first try the simplest class of models, namely linear models, followed obvio
|
||||
</div>
|
||||
<div class="section" id="simple-linear-regression-model-using-scikit-learn">
|
||||
<h2><span class="section-number">3.3. </span>Simple linear regression model using <strong>scikit-learn</strong><a class="headerlink" href="#simple-linear-regression-model-using-scikit-learn" title="Permalink to this headline">¶</a></h2>
|
||||
<p>We start with perhaps our simplest possible example, using <strong>Scikit-Learn</strong> to perform linear regression analysis on a data set produced by us.</p>
|
||||
<p>We start with perhaps our simplest possible example, using
|
||||
<strong>Scikit-Learn</strong> to perform linear regression analysis on a data set
|
||||
produced by us.</p>
|
||||
<p>What follows is a simple Python code where we have defined a function
|
||||
<span class="math notranslate nohighlight">\(y\)</span> in terms of the variable <span class="math notranslate nohighlight">\(x\)</span>. Both are defined as vectors with <span class="math notranslate nohighlight">\(100\)</span> entries.
|
||||
The numbers in the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> are given
|
||||
@@ -717,7 +721,8 @@ to be dominated by outliers.</p>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LinearRegression</span>
|
||||
|
||||
<span class="c1"># Number of data points</span>
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mi">5</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mf">0.01</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">linreg</span> <span class="o">=</span> <span class="n">LinearRegression</span><span class="p">()</span>
|
||||
@@ -740,7 +745,8 @@ to be dominated by outliers.</p>
|
||||
<p>Depending on the parameter in front of the normal distribution, we may
|
||||
have a small or larger relative error. Try to play around with
|
||||
different training data sets and study (graphically) the value of the
|
||||
relative error.</p>
|
||||
relative error. Note also that <strong>Scikit-Learn</strong> requires a matrix as input for the input values <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span>. In the above code we have
|
||||
solved this by declaring <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> as arrays of dimension <span class="math notranslate nohighlight">\(n\times 1\)</span>.</p>
|
||||
<p>As mentioned above, <strong>Scikit-Learn</strong> has an impressive functionality.
|
||||
We can for example extract the values of <span class="math notranslate nohighlight">\(\alpha\)</span> and <span class="math notranslate nohighlight">\(\beta\)</span> and
|
||||
their error estimates, or the variance and standard deviation and many
|
||||
@@ -781,13 +787,13 @@ example of the functionality of <strong>Scikit-Learn</strong>.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The intercept alpha:
|
||||
[1.99723611]
|
||||
[1.84860939]
|
||||
Coefficient beta :
|
||||
[[5.1007125]]
|
||||
Mean squared error: 0.22
|
||||
[[5.2571699]]
|
||||
Mean squared error: 0.25
|
||||
Variance score: 0.90
|
||||
Mean squared log error: 0.01
|
||||
Mean absolute error: 0.36
|
||||
Mean absolute error: 0.40
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_13_1.png" src="_images/chapter1_13_1.png" />
|
||||
@@ -844,8 +850,7 @@ ways of dealing with outliers.</p>
|
||||
<p>The Huber cost function is defined as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
H_{\delta}(\boldsymbol{a})=\left\{\begin{array}{cc}\frac{1}{2} \boldsymbol{a}^{2}& \text{for }|\boldsymbol{a}|\leq \delta\\ \delta (|\b\
|
||||
m{a}|-\frac{1}{2}\delta ),&\text{otherwise}.\end{array}\right.
|
||||
H_{\delta}(\boldsymbol{a})=\left\{\begin{array}{cc}\frac{1}{2} \boldsymbol{a}^{2}& \text{for }|\boldsymbol{a}|\leq \delta\\ \delta (|\boldsymbol{a}|-\frac{1}{2}\delta ),&\text{otherwise}.\end{array}\right.
|
||||
\end{split}\]</div>
|
||||
<p>Here <span class="math notranslate nohighlight">\(\boldsymbol{a}=\boldsymbol{y} - \boldsymbol{\tilde{y}}\)</span>.</p>
|
||||
<p>We will discuss in more
|
||||
@@ -888,7 +893,7 @@ a linear <span class="math notranslate nohighlight">\(x\)</span>-dependence we s
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter1_27_0.png" src="_images/chapter1_27_0.png" />
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.005000000000000007
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.0050000000000000044
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1126,6 +1131,7 @@ It has dimensionality <span class="math notranslate nohighlight">\(p\times n\)</
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Note well that we have made life simple here. We perform a fit in terms of the number of nucleons only. A more sophisticated fit can be done by including an explicit dependence on the number of protons and neutrons in the asymmetry and Coulomb terms.</p>
|
||||
<p>With <strong>scikitlearn</strong> we are now ready to use linear regression and fit our data.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
@@ -1145,7 +1151,6 @@ Now we can print measures of how our fit is doing, the coefficients from the fit
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Variance score: </span><span class="si">%.2f</span><span class="s1">'</span> <span class="o">%</span> <span class="n">r2_score</span><span class="p">(</span><span class="n">Energies</span><span class="p">,</span> <span class="n">fity</span><span class="p">))</span>
|
||||
<span class="c1"># Mean absolute error </span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Mean absolute error: </span><span class="si">%.2f</span><span class="s1">'</span> <span class="o">%</span> <span class="n">mean_absolute_error</span><span class="p">(</span><span class="n">Energies</span><span class="p">,</span> <span class="n">fity</span><span class="p">))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="n">coef_</span><span class="p">,</span> <span class="n">clf</span><span class="o">.</span><span class="n">intercept_</span><span class="p">)</span>
|
||||
|
||||
<span class="n">Masses</span><span class="p">[</span><span class="s1">'Eapprox'</span><span class="p">]</span> <span class="o">=</span> <span class="n">fity</span>
|
||||
<span class="c1"># Generate a plot comparing the experimental with the fitted values values.</span>
|
||||
@@ -1166,8 +1171,6 @@ Now we can print measures of how our fit is doing, the coefficients from the fit
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Mean squared error: 0.04
|
||||
Variance score: 0.95
|
||||
Mean absolute error: 0.05
|
||||
[ 0.00000000e+00 7.06492086e-03 -1.73091052e-01 -1.66020213e+01
|
||||
1.17385778e+00] 15.212327334149492
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_57_1.png" src="_images/chapter1_57_1.png" />
|
||||
@@ -1225,7 +1228,7 @@ A
|
||||
270 3344 160 110 270 Ds 7.253775 7.253775
|
||||
|
||||
[267 rows x 6 columns]
|
||||
0.009883615646716182
|
||||
0.009883615646716186
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1273,8 +1276,6 @@ functionality.</p>
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
@@ -1339,7 +1340,9 @@ functionality.</p>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
@@ -1357,7 +1360,7 @@ functionality.</p>
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_61_10.png" src="_images/chapter1_61_10.png" />
|
||||
<img alt="_images/chapter1_61_11.png" src="_images/chapter1_61_11.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2332,13 +2335,13 @@ but now splitting the data into a training set and a test set.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Training R2
|
||||
0.9999878260589065
|
||||
0.9999869956119286
|
||||
Training MSE
|
||||
5.415634483874318
|
||||
5.745136489050356
|
||||
Test R2
|
||||
0.999974281880048
|
||||
0.9999787681537219
|
||||
Test MSE
|
||||
11.653095390463358
|
||||
9.712199063818309
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2600,8 +2603,107 @@ R2 score is 0.6628996975186952
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="id1">
|
||||
<h2><span class="section-number">3.8. </span>Splitting our Data in Training and Test data<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h2>
|
||||
<p>It is normal in essentially all Machine Learning studies to split the
|
||||
data in a training set and a test set (sometimes also an additional
|
||||
validation set). <strong>Scikit-Learn</strong> has an own function for this. There
|
||||
is no explicit recipe for how much data should be included as training
|
||||
data and say test data. An accepted rule of thumb is to use
|
||||
approximately <span class="math notranslate nohighlight">\(2/3\)</span> to <span class="math notranslate nohighlight">\(4/5\)</span> of the data as training data. We will
|
||||
postpone a discussion of this splitting to the end of these notes and
|
||||
our discussion of the so-called <strong>bias-variance</strong> tradeoff. Here we
|
||||
limit ourselves to repeat the above equation of state fitting example
|
||||
but now splitting the data into a training set and a test set.</p>
|
||||
<p>Let us study some examples. The first code here takes a simple
|
||||
one-dimensional second-order polynomial and we fit it to a
|
||||
second-order polynomial. Depending on the strength of the added noise,
|
||||
the various measures like the <span class="math notranslate nohighlight">\(R2\)</span> score or the mean-squared error,
|
||||
the fit becomes better or worse.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">R2</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span> <span class="n">y_model</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">y_model</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y_data</span><span class="p">))</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
|
||||
<span class="k">def</span> <span class="nf">MSE</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span><span class="n">y_model</span><span class="p">):</span>
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">y_model</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span><span class="o">-</span><span class="n">y_model</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
|
||||
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">5</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mf">0.1</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="c1"># The design matrix now as function of a given polynomial</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">),</span><span class="mi">3</span><span class="p">))</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mf">1.0</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="mi">2</span>
|
||||
<span class="c1"># We split the data in test and training data</span>
|
||||
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
|
||||
<span class="c1"># matrix inversion to find beta</span>
|
||||
<span class="n">beta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X_train</span><span class="p">)</span> <span class="o">@</span> <span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y_train</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">beta</span><span class="p">)</span>
|
||||
<span class="c1"># and then make the prediction</span>
|
||||
<span class="n">ytilde</span> <span class="o">=</span> <span class="n">X_train</span> <span class="o">@</span> <span class="n">beta</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Training R2"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">R2</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytilde</span><span class="p">))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Training MSE"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytilde</span><span class="p">))</span>
|
||||
<span class="n">ypredict</span> <span class="o">=</span> <span class="n">X_test</span> <span class="o">@</span> <span class="n">beta</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Test R2"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">R2</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredict</span><span class="p">))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Test MSE"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredict</span><span class="p">))</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 2.05802766 -0.30626706 5.27262964]
|
||||
Training R2
|
||||
0.9967292151090247
|
||||
Training MSE
|
||||
0.007931713723267314
|
||||
Test R2
|
||||
0.9966744029509663
|
||||
Test MSE
|
||||
0.006968283076248407
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Alternatively, you could write your own test-train splitting function as shown here.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># equivalently in numpy</span>
|
||||
<span class="k">def</span> <span class="nf">train_test_split_numpy</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">labels</span><span class="p">,</span> <span class="n">train_size</span><span class="p">,</span> <span class="n">test_size</span><span class="p">):</span>
|
||||
<span class="n">n_inputs</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">)</span>
|
||||
<span class="n">inputs_shuffled</span> <span class="o">=</span> <span class="n">inputs</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
|
||||
<span class="n">labels_shuffled</span> <span class="o">=</span> <span class="n">labels</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
|
||||
|
||||
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">shuffle</span><span class="p">(</span><span class="n">inputs_shuffled</span><span class="p">)</span>
|
||||
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">shuffle</span><span class="p">(</span><span class="n">labels_shuffled</span><span class="p">)</span>
|
||||
|
||||
<span class="n">train_end</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n_inputs</span><span class="o">*</span><span class="n">train_size</span><span class="p">)</span>
|
||||
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span> <span class="o">=</span> <span class="n">inputs_shuffled</span><span class="p">[:</span><span class="n">train_end</span><span class="p">],</span> <span class="n">inputs_shuffled</span><span class="p">[</span><span class="n">train_end</span><span class="p">:]</span>
|
||||
<span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_test</span> <span class="o">=</span> <span class="n">labels_shuffled</span><span class="p">[:</span><span class="n">train_end</span><span class="p">],</span> <span class="n">labels_shuffled</span><span class="p">[</span><span class="n">train_end</span><span class="p">:]</span>
|
||||
|
||||
<span class="k">return</span> <span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_test</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>But since <strong>scikit-learn</strong> has its own function for doing this and since
|
||||
it interfaces easily with <strong>tensorflow</strong> and other libraries, we
|
||||
normally recommend using the latter functionality.</p>
|
||||
</div>
|
||||
<div class="section" id="reducing-the-number-of-degrees-of-freedom-overarching-view">
|
||||
<h2><span class="section-number">3.8. </span>Reducing the number of degrees of freedom, overarching view<a class="headerlink" href="#reducing-the-number-of-degrees-of-freedom-overarching-view" title="Permalink to this headline">¶</a></h2>
|
||||
<h2><span class="section-number">3.9. </span>Reducing the number of degrees of freedom, overarching view<a class="headerlink" href="#reducing-the-number-of-degrees-of-freedom-overarching-view" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Many Machine Learning problems involve thousands or even millions of
|
||||
features for each training instance. Not only does this make training
|
||||
extremely slow, it can also make it much harder to find a good
|
||||
@@ -2624,6 +2726,13 @@ not met so many cases where we are too sensitive to the scaling of our
|
||||
data. Normally the data may need a rescaling and/or may be sensitive
|
||||
to extreme values. Scaling the data renders our inputs much more
|
||||
suitable for the algorithms we want to employ.</p>
|
||||
<p>For data sets gathered for real world applications, it is rather normal that
|
||||
different features have very different units and
|
||||
numerical scales. For example, a data set detailing health habits may include
|
||||
features such as <strong>age</strong> in the range <span class="math notranslate nohighlight">\(0-80\)</span>, and <strong>caloric intake</strong> of order <span class="math notranslate nohighlight">\(2000\)</span>.
|
||||
Many machine learning methods sensitive to the scales of the features and may perform poorly if they
|
||||
are very different scales. Therefore, it is typical to scale
|
||||
the features in a way to avoid such outlier values.</p>
|
||||
<p><strong>Scikit-Learn</strong> has several functions which allow us to rescale the
|
||||
data, normally resulting in much better results in terms of various
|
||||
accuracy scores. The <strong>StandardScaler</strong> function in <strong>Scikit-Learn</strong>
|
||||
@@ -2648,142 +2757,465 @@ RobustScaler ignore data points that are very different from the rest
|
||||
(like measurement errors). These odd data points are also called
|
||||
outliers, and might often lead to trouble for other scaling
|
||||
techniques.</p>
|
||||
<div class="section" id="simple-preprocessing-examples-franke-function-and-regression">
|
||||
<h3><span class="section-number">3.8.1. </span>Simple preprocessing examples, Franke function and regression<a class="headerlink" href="#simple-preprocessing-examples-franke-function-and-regression" title="Permalink to this headline">¶</a></h3>
|
||||
<p>Many features are often scaled using standardization to improve
|
||||
performance. In <strong>Scikit-Learn</strong> this is given by the <strong>StandardScaler</strong>
|
||||
function as discussed above. It is easy however to write your own.
|
||||
Mathematically, this involves subtracting the mean and divide by the
|
||||
standard deviation over the data set, for each feature:</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
x_j^{(i)} \rightarrow \frac{x_j^{(i)} - \overline{x}_j}{\sigma(x_j)},
|
||||
\]</div>
|
||||
<p>where <span class="math notranslate nohighlight">\(\overline{x}_j\)</span> and <span class="math notranslate nohighlight">\(\sigma(x_j)\)</span> are the mean and standard
|
||||
deviation, respectively, of the feature <span class="math notranslate nohighlight">\(x_j\)</span>. This ensures that each
|
||||
feature has zero mean and unit standard deviation. For data sets
|
||||
where we do not have the standard deviation or don’t wish to calculate
|
||||
it, it is then common to simply set it to one.</p>
|
||||
<p>Let us consider the following vanilla example where we use both
|
||||
<strong>Scikit-Learn</strong> and write our own function as well. We produce a
|
||||
simple test design matrix with random numbers. Each column could then
|
||||
represent a specific feature whose mean value is subracted.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Common imports</span>
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">import</span> <span class="nn">sklearn.linear_model</span> <span class="k">as</span> <span class="nn">skl</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sklearn.linear_model</span> <span class="k">as</span> <span class="nn">skl</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.metrics</span> <span class="kn">import</span> <span class="n">mean_squared_error</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">MinMaxScaler</span><span class="p">,</span> <span class="n">StandardScaler</span><span class="p">,</span> <span class="n">Normalizer</span>
|
||||
|
||||
<span class="c1"># Where to save the figures and data files</span>
|
||||
<span class="n">PROJECT_ROOT_DIR</span> <span class="o">=</span> <span class="s2">"Results"</span>
|
||||
<span class="n">FIGURE_ID</span> <span class="o">=</span> <span class="s2">"Results/FigureFiles"</span>
|
||||
<span class="n">DATA_ID</span> <span class="o">=</span> <span class="s2">"DataFiles/"</span>
|
||||
|
||||
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">PROJECT_ROOT_DIR</span><span class="p">):</span>
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">PROJECT_ROOT_DIR</span><span class="p">)</span>
|
||||
|
||||
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">):</span>
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">)</span>
|
||||
|
||||
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">):</span>
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">)</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">,</span> <span class="n">fig_id</span><span class="p">)</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">data_path</span><span class="p">(</span><span class="n">dat_id</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">,</span> <span class="n">dat_id</span><span class="p">)</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">save_fig</span><span class="p">(</span><span class="n">fig_id</span><span class="p">):</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="n">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">)</span> <span class="o">+</span> <span class="s2">".png"</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="s1">'png'</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">FrankeFunction</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">):</span>
|
||||
<span class="n">term1</span> <span class="o">=</span> <span class="mf">0.75</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="mf">0.25</span><span class="o">*</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">-</span> <span class="mf">0.25</span><span class="o">*</span><span class="p">((</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">))</span>
|
||||
<span class="n">term2</span> <span class="o">=</span> <span class="mf">0.75</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">((</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="mf">49.0</span> <span class="o">-</span> <span class="mf">0.1</span><span class="o">*</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">+</span><span class="mi">1</span><span class="p">))</span>
|
||||
<span class="n">term3</span> <span class="o">=</span> <span class="mf">0.5</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">-</span><span class="mi">7</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="o">/</span><span class="mf">4.0</span> <span class="o">-</span> <span class="mf">0.25</span><span class="o">*</span><span class="p">((</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">-</span><span class="mi">3</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">))</span>
|
||||
<span class="n">term4</span> <span class="o">=</span> <span class="o">-</span><span class="mf">0.2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">-</span><span class="mi">4</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span> <span class="o">-</span> <span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">-</span><span class="mi">7</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="n">term1</span> <span class="o">+</span> <span class="n">term2</span> <span class="o">+</span> <span class="n">term3</span> <span class="o">+</span> <span class="n">term4</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">create_X</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">n</span> <span class="p">):</span>
|
||||
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span> <span class="o">></span> <span class="mi">1</span><span class="p">:</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ravel</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ravel</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
|
||||
|
||||
<span class="n">N</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
|
||||
<span class="n">l</span> <span class="o">=</span> <span class="nb">int</span><span class="p">((</span><span class="n">n</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="p">(</span><span class="n">n</span><span class="o">+</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="mi">2</span><span class="p">)</span> <span class="c1"># Number of elements in beta</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">N</span><span class="p">,</span><span class="n">l</span><span class="p">))</span>
|
||||
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="n">n</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
|
||||
<span class="n">q</span> <span class="o">=</span> <span class="nb">int</span><span class="p">((</span><span class="n">i</span><span class="p">)</span><span class="o">*</span><span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">/</span><span class="mi">2</span><span class="p">)</span>
|
||||
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="n">q</span><span class="o">+</span><span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="n">x</span><span class="o">**</span><span class="p">(</span><span class="n">i</span><span class="o">-</span><span class="n">k</span><span class="p">))</span><span class="o">*</span><span class="p">(</span><span class="n">y</span><span class="o">**</span><span class="n">k</span><span class="p">)</span>
|
||||
|
||||
<span class="k">return</span> <span class="n">X</span>
|
||||
|
||||
|
||||
<span class="c1"># Making meshgrid of datapoints and compute Franke's function</span>
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="mi">5</span>
|
||||
<span class="n">N</span> <span class="o">=</span> <span class="mi">1000</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">N</span><span class="p">))</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">N</span><span class="p">))</span>
|
||||
<span class="n">z</span> <span class="o">=</span> <span class="n">FrankeFunction</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">create_X</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">n</span><span class="o">=</span><span class="n">n</span><span class="p">)</span>
|
||||
<span class="c1"># split in training and test data</span>
|
||||
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span><span class="n">z</span><span class="p">,</span><span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="n">clf</span> <span class="o">=</span> <span class="n">skl</span><span class="o">.</span><span class="n">LinearRegression</span><span class="p">()</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># The mean squared error and R2 score</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"MSE before scaling: </span><span class="si">{:.2f}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">mean_squared_error</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">),</span> <span class="n">y_test</span><span class="p">)))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"R2 score before scaling </span><span class="si">{:.2f}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X_test</span><span class="p">,</span><span class="n">y_test</span><span class="p">)))</span>
|
||||
|
||||
<span class="n">scaler</span> <span class="o">=</span> <span class="n">StandardScaler</span><span class="p">()</span>
|
||||
<span class="n">scaler</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
|
||||
<span class="n">X_train_scaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
|
||||
<span class="n">X_test_scaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
|
||||
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Feature min values before scaling:</span><span class="se">\n</span><span class="s2"> </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">min</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Feature max values before scaling:</span><span class="se">\n</span><span class="s2"> </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)))</span>
|
||||
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Feature min values after scaling:</span><span class="se">\n</span><span class="s2"> </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="o">.</span><span class="n">min</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Feature max values after scaling:</span><span class="se">\n</span><span class="s2"> </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)))</span>
|
||||
|
||||
<span class="n">clf</span> <span class="o">=</span> <span class="n">skl</span><span class="o">.</span><span class="n">LinearRegression</span><span class="p">()</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"MSE after scaling: </span><span class="si">{:.2f}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">mean_squared_error</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">),</span> <span class="n">y_test</span><span class="p">)))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"R2 score for scaled data: </span><span class="si">{:.2f}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">,</span><span class="n">y_test</span><span class="p">)))</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
|
||||
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">display</span>
|
||||
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
|
||||
<span class="c1"># setting up a 10 x 5 matrix</span>
|
||||
<span class="n">rows</span> <span class="o">=</span> <span class="mi">10</span>
|
||||
<span class="n">cols</span> <span class="o">=</span> <span class="mi">5</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">rows</span><span class="p">,</span><span class="n">cols</span><span class="p">)</span>
|
||||
<span class="n">XPandas</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
|
||||
<span class="n">display</span><span class="p">(</span><span class="n">XPandas</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">XPandas</span><span class="o">.</span><span class="n">mean</span><span class="p">())</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">XPandas</span><span class="o">.</span><span class="n">std</span><span class="p">())</span>
|
||||
<span class="n">XPandas</span> <span class="o">=</span> <span class="p">(</span><span class="n">XPandas</span> <span class="o">-</span><span class="n">XPandas</span><span class="o">.</span><span class="n">mean</span><span class="p">())</span>
|
||||
<span class="n">display</span><span class="p">(</span><span class="n">XPandas</span><span class="p">)</span>
|
||||
<span class="c1"># This option does not include the standard deviation</span>
|
||||
<span class="n">scaler</span> <span class="o">=</span> <span class="n">StandardScaler</span><span class="p">(</span><span class="n">with_std</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
|
||||
<span class="n">scaler</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
|
||||
<span class="n">Xscaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
|
||||
<span class="n">display</span><span class="p">(</span><span class="n">XPandas</span><span class="o">-</span><span class="n">Xscaled</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>MSE before scaling: 0.00
|
||||
R2 score before scaling 0.99
|
||||
Feature min values before scaling:
|
||||
[1.00000000e+00 2.31109204e-04 5.13266452e-03 5.34114641e-08
|
||||
1.18620601e-06 2.63442451e-05 1.23438810e-11 2.74143127e-10
|
||||
6.08839750e-09 1.35216172e-07 2.85278450e-15 6.33569998e-14
|
||||
1.40708470e-12 3.12497018e-11 6.94019247e-10 6.59304755e-19
|
||||
1.46423858e-17 3.25190225e-16 7.22209371e-15 1.60394236e-13
|
||||
3.56216797e-12]
|
||||
Feature max values before scaling:
|
||||
[1. 0.99997737 0.9950799 0.99995475 0.99505739 0.99018401
|
||||
0.99993212 0.99503487 0.99016161 0.98531221 0.9999095 0.99501236
|
||||
0.99013921 0.98528992 0.98046438 0.99988687 0.99498985 0.9901168
|
||||
0.98526763 0.9804422 0.9756404 ]
|
||||
Feature min values after scaling:
|
||||
[ 0. -1.7621419 -1.74382593 -1.13803928 -1.14662074 -1.15475232
|
||||
-0.89942598 -0.90825063 -0.9171356 -0.92605247 -0.76570177 -0.77317984
|
||||
-0.78080633 -0.78857629 -0.79648291 -0.67708423 -0.68316185 -0.68937695
|
||||
-0.69573183 -0.7022283 -0.70886748]
|
||||
Feature max values after scaling:
|
||||
[0. 1.71669651 1.63315151 2.21623344 2.15324631 2.088871
|
||||
2.62856593 2.5755035 2.52158335 2.46675058 2.987902 2.94034531
|
||||
2.8923 2.84372853 2.79459035 3.30968181 3.2653678 3.22076362
|
||||
3.1758506 3.13060772 3.08501147]
|
||||
MSE after scaling: 0.00
|
||||
R2 score for scaled data: 0.99
|
||||
<div class="output text_html"><div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.dataframe tbody tr th {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.dataframe thead th {
|
||||
text-align: right;
|
||||
}
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;">
|
||||
<th></th>
|
||||
<th>0</th>
|
||||
<th>1</th>
|
||||
<th>2</th>
|
||||
<th>3</th>
|
||||
<th>4</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>0</th>
|
||||
<td>-1.749765</td>
|
||||
<td>0.342680</td>
|
||||
<td>1.153036</td>
|
||||
<td>-0.252436</td>
|
||||
<td>0.981321</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>1</th>
|
||||
<td>0.514219</td>
|
||||
<td>0.221180</td>
|
||||
<td>-1.070043</td>
|
||||
<td>-0.189496</td>
|
||||
<td>0.255001</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>2</th>
|
||||
<td>-0.458027</td>
|
||||
<td>0.435163</td>
|
||||
<td>-0.583595</td>
|
||||
<td>0.816847</td>
|
||||
<td>0.672721</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>3</th>
|
||||
<td>-0.104411</td>
|
||||
<td>-0.531280</td>
|
||||
<td>1.029733</td>
|
||||
<td>-0.438136</td>
|
||||
<td>-1.118318</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>4</th>
|
||||
<td>1.618982</td>
|
||||
<td>1.541605</td>
|
||||
<td>-0.251879</td>
|
||||
<td>-0.842436</td>
|
||||
<td>0.184519</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>5</th>
|
||||
<td>0.937082</td>
|
||||
<td>0.731000</td>
|
||||
<td>1.361556</td>
|
||||
<td>-0.326238</td>
|
||||
<td>0.055676</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>6</th>
|
||||
<td>0.222400</td>
|
||||
<td>-1.443217</td>
|
||||
<td>-0.756352</td>
|
||||
<td>0.816454</td>
|
||||
<td>0.750445</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>7</th>
|
||||
<td>-0.455947</td>
|
||||
<td>1.189622</td>
|
||||
<td>-1.690617</td>
|
||||
<td>-1.356399</td>
|
||||
<td>-1.232435</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>8</th>
|
||||
<td>-0.544439</td>
|
||||
<td>-0.668172</td>
|
||||
<td>0.007315</td>
|
||||
<td>-0.612939</td>
|
||||
<td>1.299748</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>9</th>
|
||||
<td>-1.733096</td>
|
||||
<td>-0.983310</td>
|
||||
<td>0.357508</td>
|
||||
<td>-1.613579</td>
|
||||
<td>1.470714</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div></div><div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0 -0.175300
|
||||
1 0.083527
|
||||
2 -0.044334
|
||||
3 -0.399836
|
||||
4 0.331939
|
||||
dtype: float64
|
||||
0 1.069584
|
||||
1 0.965548
|
||||
2 1.018232
|
||||
3 0.793167
|
||||
4 0.918992
|
||||
dtype: float64
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output text_html"><div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.dataframe tbody tr th {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.dataframe thead th {
|
||||
text-align: right;
|
||||
}
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;">
|
||||
<th></th>
|
||||
<th>0</th>
|
||||
<th>1</th>
|
||||
<th>2</th>
|
||||
<th>3</th>
|
||||
<th>4</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>0</th>
|
||||
<td>-1.574465</td>
|
||||
<td>0.259153</td>
|
||||
<td>1.197370</td>
|
||||
<td>0.147400</td>
|
||||
<td>0.649382</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>1</th>
|
||||
<td>0.689519</td>
|
||||
<td>0.137652</td>
|
||||
<td>-1.025709</td>
|
||||
<td>0.210340</td>
|
||||
<td>-0.076938</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>2</th>
|
||||
<td>-0.282727</td>
|
||||
<td>0.351636</td>
|
||||
<td>-0.539261</td>
|
||||
<td>1.216683</td>
|
||||
<td>0.340782</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>3</th>
|
||||
<td>0.070889</td>
|
||||
<td>-0.614808</td>
|
||||
<td>1.074067</td>
|
||||
<td>-0.038300</td>
|
||||
<td>-1.450257</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>4</th>
|
||||
<td>1.794282</td>
|
||||
<td>1.458078</td>
|
||||
<td>-0.207545</td>
|
||||
<td>-0.442600</td>
|
||||
<td>-0.147420</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>5</th>
|
||||
<td>1.112383</td>
|
||||
<td>0.647473</td>
|
||||
<td>1.405890</td>
|
||||
<td>0.073598</td>
|
||||
<td>-0.276263</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>6</th>
|
||||
<td>0.397700</td>
|
||||
<td>-1.526744</td>
|
||||
<td>-0.712018</td>
|
||||
<td>1.216290</td>
|
||||
<td>0.418506</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>7</th>
|
||||
<td>-0.280647</td>
|
||||
<td>1.106095</td>
|
||||
<td>-1.646283</td>
|
||||
<td>-0.956563</td>
|
||||
<td>-1.564374</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>8</th>
|
||||
<td>-0.369139</td>
|
||||
<td>-0.751699</td>
|
||||
<td>0.051649</td>
|
||||
<td>-0.213103</td>
|
||||
<td>0.967809</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>9</th>
|
||||
<td>-1.557795</td>
|
||||
<td>-1.066837</td>
|
||||
<td>0.401842</td>
|
||||
<td>-1.213743</td>
|
||||
<td>1.138775</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div></div><div class="output text_html"><div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.dataframe tbody tr th {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.dataframe thead th {
|
||||
text-align: right;
|
||||
}
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;">
|
||||
<th></th>
|
||||
<th>0</th>
|
||||
<th>1</th>
|
||||
<th>2</th>
|
||||
<th>3</th>
|
||||
<th>4</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>0</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>1</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>2</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>3</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>4</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>5</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>6</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>7</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>8</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>9</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div></div></div>
|
||||
</div>
|
||||
<p>Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.</p>
|
||||
<p>Another commonly used scaling method is min-max scaling. This is very
|
||||
useful for when we want the features to lie in a certain interval. To
|
||||
scale the feature <span class="math notranslate nohighlight">\(x_j\)</span> to the interval <span class="math notranslate nohighlight">\([a, b]\)</span>, we can apply the
|
||||
transformation</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
x_j^{(i)} \rightarrow (b-a)\frac{x_j^{(i)} - \min(x_j)}{\max(x_j) - \min(x_j)} - a
|
||||
\]</div>
|
||||
<p>where <span class="math notranslate nohighlight">\(\min(x_j)\)</span> and <span class="math notranslate nohighlight">\(\max(x_j)\)</span> return the minimum and maximum value of <span class="math notranslate nohighlight">\(x_j\)</span> over the data set, respectively.</p>
|
||||
</div>
|
||||
<div class="section" id="testing-the-means-squared-error-as-function-of-complexity">
|
||||
<h2><span class="section-number">3.10. </span>Testing the Means Squared Error as function of Complexity<a class="headerlink" href="#testing-the-means-squared-error-as-function-of-complexity" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Before we proceed with a more detailed analysis of the so-called
|
||||
Bias-Variance tradeoff, we present here an example of the relation
|
||||
between model complexity and the mean squared error for the triaining
|
||||
data and the test data.</p>
|
||||
<p>The results here tell us clearly that for the data not included in the
|
||||
training, there is an optimal model as function of the complexity of
|
||||
ourmodel (here in terms of the polynomial degree of the model).</p>
|
||||
<p>The results here will vary as function of model complexity and the amount od data used for training.</p>
|
||||
<p>Our data is defined by <span class="math notranslate nohighlight">\(x\in [-3,3]\)</span> with a total of for example <span class="math notranslate nohighlight">\(100\)</span> data points.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LinearRegression</span><span class="p">,</span> <span class="n">Ridge</span><span class="p">,</span> <span class="n">Lasso</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">PolynomialFeatures</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.pipeline</span> <span class="kn">import</span> <span class="n">make_pipeline</span>
|
||||
|
||||
|
||||
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">2018</span><span class="p">)</span>
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
|
||||
<span class="n">maxdegree</span> <span class="o">=</span> <span class="mi">14</span>
|
||||
<span class="c1"># Make data set.</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linspace</span><span class="p">(</span><span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mf">1.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
|
||||
<span class="n">TestError</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">maxdegree</span><span class="p">)</span>
|
||||
<span class="n">TrainError</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">maxdegree</span><span class="p">)</span>
|
||||
<span class="n">polydegree</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">maxdegree</span><span class="p">)</span>
|
||||
<span class="n">x_train</span><span class="p">,</span> <span class="n">x_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
|
||||
<span class="n">scaler</span> <span class="o">=</span> <span class="n">StandardScaler</span><span class="p">()</span>
|
||||
<span class="n">scaler</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">x_train</span><span class="p">)</span>
|
||||
<span class="n">x_train_scaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">x_train</span><span class="p">)</span>
|
||||
<span class="n">x_test_scaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">x_test</span><span class="p">)</span>
|
||||
|
||||
<span class="k">for</span> <span class="n">degree</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">maxdegree</span><span class="p">):</span>
|
||||
<span class="n">model</span> <span class="o">=</span> <span class="n">make_pipeline</span><span class="p">(</span><span class="n">PolynomialFeatures</span><span class="p">(</span><span class="n">degree</span><span class="o">=</span><span class="n">degree</span><span class="p">),</span> <span class="n">LinearRegression</span><span class="p">(</span><span class="n">fit_intercept</span><span class="o">=</span><span class="kc">False</span><span class="p">))</span>
|
||||
<span class="n">clf</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">x_train_scaled</span><span class="p">,</span><span class="n">y_train</span><span class="p">)</span>
|
||||
<span class="n">y_fit</span> <span class="o">=</span> <span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">x_train_scaled</span><span class="p">)</span>
|
||||
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">x_test_scaled</span><span class="p">)</span>
|
||||
<span class="n">polydegree</span><span class="p">[</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">degree</span>
|
||||
<span class="n">TestError</span><span class="p">[</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">((</span><span class="n">y_test</span> <span class="o">-</span> <span class="n">y_pred</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="p">)</span>
|
||||
<span class="n">TrainError</span><span class="p">[</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">((</span><span class="n">y_train</span> <span class="o">-</span> <span class="n">y_fit</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="p">)</span>
|
||||
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">polydegree</span><span class="p">,</span> <span class="n">TestError</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s1">'Test Error'</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">polydegree</span><span class="p">,</span> <span class="n">TrainError</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s1">'Train Error'</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter1_214_0.png" src="_images/chapter1_214_0.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="exercises">
|
||||
<h2><span class="section-number">3.9. </span>Exercises<a class="headerlink" href="#exercises" title="Permalink to this headline">¶</a></h2>
|
||||
<h2><span class="section-number">3.11. </span>Exercises<a class="headerlink" href="#exercises" title="Permalink to this headline">¶</a></h2>
|
||||
<div class="section" id="exercise-setting-up-various-python-environments">
|
||||
<h3><span class="section-number">3.9.1. </span>Exercise: Setting up various Python environments<a class="headerlink" href="#exercise-setting-up-various-python-environments" title="Permalink to this headline">¶</a></h3>
|
||||
<h3><span class="section-number">3.11.1. </span>Exercise: Setting up various Python environments<a class="headerlink" href="#exercise-setting-up-various-python-environments" title="Permalink to this headline">¶</a></h3>
|
||||
<p>The first exercise here is of a mere technical art. We want you to have</p>
|
||||
<ul class="simple">
|
||||
<li><p>git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo <a class="reference external" href="https://www.uio.no/tjenester/it/maskin/filer/versjonskontroll/github.html">GitHub facilities</a>.</p></li>
|
||||
@@ -2839,7 +3271,7 @@ license.</p>
|
||||
<p>We recommend using <strong>Anaconda</strong> if you are not too familiar with setting paths in a terminal environment.</p>
|
||||
</div>
|
||||
<div class="section" id="exercise-making-your-own-data-and-exploring-scikit-learn">
|
||||
<h3><span class="section-number">3.9.2. </span>Exercise: making your own data and exploring scikit-learn<a class="headerlink" href="#exercise-making-your-own-data-and-exploring-scikit-learn" title="Permalink to this headline">¶</a></h3>
|
||||
<h3><span class="section-number">3.11.2. </span>Exercise: making your own data and exploring scikit-learn<a class="headerlink" href="#exercise-making-your-own-data-and-exploring-scikit-learn" title="Permalink to this headline">¶</a></h3>
|
||||
<p>We will generate our own dataset for a function <span class="math notranslate nohighlight">\(y(x)\)</span> where <span class="math notranslate nohighlight">\(x \in [0,1]\)</span> and defined by random numbers computed with the uniform distribution. The function <span class="math notranslate nohighlight">\(y\)</span> is a quadratic polynomial in <span class="math notranslate nohighlight">\(x\)</span> with added stochastic noise according to the normal distribution <span class="math notranslate nohighlight">\(\cal {N}(0,1)\)</span>.
|
||||
The following simple Python instructions define our <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> values (with 100 data points).</p>
|
||||
<div class="cell docutils container">
|
||||
@@ -2875,7 +3307,7 @@ R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i
|
||||
Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.</p>
|
||||
</div>
|
||||
<div class="section" id="exercise-normalizing-our-data">
|
||||
<h3><span class="section-number">3.9.3. </span>Exercise: Normalizing our data<a class="headerlink" href="#exercise-normalizing-our-data" title="Permalink to this headline">¶</a></h3>
|
||||
<h3><span class="section-number">3.11.3. </span>Exercise: Normalizing our data<a class="headerlink" href="#exercise-normalizing-our-data" title="Permalink to this headline">¶</a></h3>
|
||||
<p>A much used approach before starting to train the data is to preprocess our
|
||||
data. Normally the data may need a rescaling and/or may be sensitive
|
||||
to extreme values. Scaling the data renders our inputs much more
|
||||
@@ -2987,7 +3419,7 @@ your results. For which polynomial degree do you find an optimal MSE
|
||||
<div class='prev-next-bottom'>
|
||||
|
||||
<a class='left-prev' id="prev-link" href="linalg.html" title="previous page"><span class="section-number">2. </span>Linear Algebra, Handling of Arrays and more Python Features</a>
|
||||
<a class='right-next' id="next-link" href="chapter2.html" title="next page"><span class="section-number">4. </span>Resampling Methods</a>
|
||||
<a class='right-next' id="next-link" href="chapter2.html" title="next page"><span class="section-number">4. </span>Ridge and Lasso Regression</a>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user