update on book
This commit is contained in:
Morten Hjorth-Jensen
@@ -588,7 +588,7 @@ may first try the simplest class of models, namely linear models, followed obvio
|
||||
<p>We start with perhaps our simplest possible example, using <strong>Scikit-Learn</strong> to perform linear regression analysis on a data set produced by us.</p>
|
||||
<p>What follows is a simple Python code where we have defined a function
|
||||
<span class="math notranslate nohighlight">\(y\)</span> in terms of the variable <span class="math notranslate nohighlight">\(x\)</span>. Both are defined as vectors with <span class="math notranslate nohighlight">\(100\)</span> entries.
|
||||
The numbers in the vector <span class="math notranslate nohighlight">\(\hat{x}\)</span> are given
|
||||
The numbers in the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> are given
|
||||
by random numbers generated with a uniform distribution with entries
|
||||
<span class="math notranslate nohighlight">\(x_i \in [0,1]\)</span> (more about probability distribution functions
|
||||
later). These values are then used to define a function <span class="math notranslate nohighlight">\(y(x)\)</span>
|
||||
@@ -611,7 +611,7 @@ y = 2x+N(0,1),
|
||||
distribution. From <strong>Scikit-Learn</strong> we import then the
|
||||
<strong>LinearRegression</strong> functionality and make a prediction <span class="math notranslate nohighlight">\(\tilde{y} =
|
||||
\alpha + \beta x\)</span> using the function <strong>fit(x,y)</strong>. We call the set of
|
||||
data <span class="math notranslate nohighlight">\((\hat{x},\hat{y})\)</span> for our training data. The Python package
|
||||
data <span class="math notranslate nohighlight">\((\boldsymbol{x},\boldsymbol{y})\)</span> for our training data. The Python package
|
||||
<strong>scikit-learn</strong> has also a functionality which extracts the above
|
||||
fitting parameters <span class="math notranslate nohighlight">\(\alpha\)</span> and <span class="math notranslate nohighlight">\(\beta\)</span> (see below). Later we will
|
||||
distinguish between training data and test data.</p>
|
||||
@@ -703,7 +703,7 @@ the <span class="math notranslate nohighlight">\(\chi^2\)</span> function becom
|
||||
the relative error (why would we prefer the MSE instead of the relative error?) as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}.
|
||||
\epsilon_{\mathrm{relative}}= \frac{\vert \boldsymbol{y} -\boldsymbol{\tilde{y}}\vert}{\vert \boldsymbol{y}\vert}.
|
||||
\]</div>
|
||||
<p>The squared cost function results in an arithmetic mean-unbiased
|
||||
estimator, and the absolute-value cost function results in a
|
||||
@@ -781,13 +781,13 @@ example of the functionality of <strong>Scikit-Learn</strong>.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The intercept alpha:
|
||||
[1.98452685]
|
||||
[2.15024669]
|
||||
Coefficient beta :
|
||||
[[5.00109273]]
|
||||
Mean squared error: 0.19
|
||||
Variance score: 0.92
|
||||
[[4.89975818]]
|
||||
Mean squared error: 0.26
|
||||
Variance score: 0.90
|
||||
Mean squared log error: 0.01
|
||||
Mean absolute error: 0.35
|
||||
Mean absolute error: 0.42
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_13_1.png" src="_images/chapter1_13_1.png" />
|
||||
@@ -797,7 +797,7 @@ Mean absolute error: 0.35
|
||||
<span class="math notranslate nohighlight">\(\alpha\)</span>. Depending on the constant in front of the normal distribution, we get values near or far from <span class="math notranslate nohighlight">\(alpha =2\)</span> and <span class="math notranslate nohighlight">\(\beta =5\)</span>. Try to play around with different parameters in front of the normal distribution. The function <strong>meansquarederror</strong> gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
MSE(\boldsymbol{y},\boldsymbol{\tilde{y}}) = \frac{1}{n}
|
||||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||||
\]</div>
|
||||
<p>The smaller the value, the better the fit. Ideally we would like to
|
||||
@@ -807,14 +807,14 @@ this function as being similar to the <span class="math notranslate nohighlight"
|
||||
determination. It provides a measure of how well future samples are
|
||||
likely to be predicted by the model. Best possible score is 1.0 and it
|
||||
can be negative (because the model can be arbitrarily worse). A
|
||||
constant model that always predicts the expected value of <span class="math notranslate nohighlight">\(\hat{y}\)</span>,
|
||||
constant model that always predicts the expected value of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>,
|
||||
disregarding the input features, would get a <span class="math notranslate nohighlight">\(R^2\)</span> score of <span class="math notranslate nohighlight">\(0.0\)</span>.</p>
|
||||
<p>If <span class="math notranslate nohighlight">\(\tilde{\hat{y}}_i\)</span> is the predicted value of the <span class="math notranslate nohighlight">\(i-th\)</span> sample and <span class="math notranslate nohighlight">\(y_i\)</span> is the corresponding true value, then the score <span class="math notranslate nohighlight">\(R^2\)</span> is defined as</p>
|
||||
<p>If <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}_i\)</span> is the predicted value of the <span class="math notranslate nohighlight">\(i-th\)</span> sample and <span class="math notranslate nohighlight">\(y_i\)</span> is the corresponding true value, then the score <span class="math notranslate nohighlight">\(R^2\)</span> is defined as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
\]</div>
|
||||
<p>where we have defined the mean value of <span class="math notranslate nohighlight">\(\hat{y}\)</span> as</p>
|
||||
<p>where we have defined the mean value of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
||||
@@ -824,13 +824,13 @@ the mean absolute error (MAE), a risk metric corresponding to the expected value
|
||||
The MAE is defined as follows</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
|
||||
\text{MAE}(\boldsymbol{y}, \boldsymbol{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
|
||||
\]</div>
|
||||
<p>We present the
|
||||
squared logarithmic (quadratic) error</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2,
|
||||
\text{MSLE}(\boldsymbol{y}, \boldsymbol{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2,
|
||||
\]</div>
|
||||
<p>where <span class="math notranslate nohighlight">\(\log_e (x)\)</span> stands for the natural logarithm of <span class="math notranslate nohighlight">\(x\)</span>. This error
|
||||
estimate is best to use when targets having exponential growth, such
|
||||
@@ -888,7 +888,7 @@ a linear <span class="math notranslate nohighlight">\(x\)</span>-dependence we s
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter1_27_0.png" src="_images/chapter1_27_0.png" />
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.004999999999999996
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.0050000000000000044
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1225,7 +1225,7 @@ A
|
||||
270 3344 160 110 270 Ds 7.253775 7.253775
|
||||
|
||||
[267 rows x 6 columns]
|
||||
0.009883615646716184
|
||||
0.009883615646716182
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1283,8 +1283,6 @@ functionality.</p>
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
@@ -1307,6 +1305,14 @@ functionality.</p>
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
@@ -1331,14 +1337,14 @@ functionality.</p>
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
@@ -1349,7 +1355,7 @@ functionality.</p>
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_61_9.png" src="_images/chapter1_61_9.png" />
|
||||
<img alt="_images/chapter1_61_10.png" src="_images/chapter1_61_10.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2324,13 +2330,13 @@ but now splitting the data into a training set and a test set.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Training R2
|
||||
0.9999864543345858
|
||||
0.9999868619217517
|
||||
Training MSE
|
||||
6.180092462880674
|
||||
5.965885569080809
|
||||
Test R2
|
||||
0.9999822527140678
|
||||
0.9999794306626945
|
||||
Test MSE
|
||||
7.205466494327873
|
||||
8.300162456113691
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2741,31 +2747,31 @@ techniques.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>MSE before scaling: 0.00
|
||||
R2 score before scaling 0.99
|
||||
R2 score before scaling 1.00
|
||||
Feature min values before scaling:
|
||||
[1.00000000e+00 1.10094646e-03 9.51523276e-04 1.21208310e-06
|
||||
1.04757618e-06 9.05396545e-07 1.33443859e-09 1.15332528e-09
|
||||
9.96793117e-10 8.61505887e-10 1.46914544e-12 1.26974938e-12
|
||||
1.09741585e-12 9.48471852e-13 8.19742904e-13 1.61745046e-15
|
||||
1.39792608e-15 1.20819609e-15 1.04421672e-15 9.02493044e-16
|
||||
7.80004454e-16]
|
||||
[1.00000000e+00 2.54152940e-03 1.38207279e-03 6.45937170e-06
|
||||
3.51257863e-06 1.91012519e-06 1.64166831e-08 8.92732185e-09
|
||||
4.85463934e-09 2.63993205e-09 4.17234827e-11 2.26890510e-11
|
||||
1.23382086e-11 6.70946493e-12 3.64857826e-12 1.06041458e-13
|
||||
5.76648901e-14 3.13579199e-14 1.70523024e-14 9.27296891e-15
|
||||
5.04260073e-15]
|
||||
Feature max values before scaling:
|
||||
[1. 0.99825997 0.99883879 0.99652296 0.99710078 0.99767893
|
||||
0.99478898 0.9953658 0.99594294 0.99652042 0.99305802 0.99363383
|
||||
0.99420997 0.99478645 0.99536326 0.99133007 0.99190487 0.99248001
|
||||
0.99305549 0.99363129 0.99420743]
|
||||
[1. 0.99817842 0.99945628 0.99636015 0.99763569 0.99891285
|
||||
0.9945452 0.99581841 0.99709325 0.99836973 0.99273355 0.99400444
|
||||
0.99527696 0.99655111 0.99782689 0.9909252 0.99219378 0.99346398
|
||||
0.99473581 0.99600927 0.99728435]
|
||||
Feature min values after scaling:
|
||||
[ 0. -1.61869821 -1.66880047 -1.06209126 -1.08170444 -1.10131725
|
||||
-0.84087101 -0.85396354 -0.86692943 -0.87972591 -0.71351486 -0.7240496
|
||||
-0.73453972 -0.74495014 -0.75524378 -0.62783293 -0.63680118 -0.64580686
|
||||
-0.65482578 -0.66383151 -0.67279536]
|
||||
[ 0. -1.63437572 -1.76504618 -1.03871832 -1.08415761 -1.13457922
|
||||
-0.81620806 -0.83935285 -0.86436607 -0.8914984 -0.69347005 -0.70790937
|
||||
-0.72312577 -0.73921714 -0.75629493 -0.61234223 -0.6226921 -0.63339159
|
||||
-0.64447921 -0.65599927 -0.66800261]
|
||||
Feature max values after scaling:
|
||||
[0. 1.78944806 1.68342382 2.34172919 2.25457052 2.16553696
|
||||
2.7899453 2.71281409 2.63374631 2.55280484 3.17117385 3.10307631
|
||||
3.03303359 2.96104648 2.88712946 3.50394742 3.44395541 3.38216436
|
||||
3.31853484 3.25303483 3.1856411 ]
|
||||
[0. 1.86574276 1.72218808 2.41770932 2.31730641 2.21347282
|
||||
2.88297395 2.79328828 2.70234019 2.60999846 3.291614 3.20772452
|
||||
3.12283463 3.03697069 2.95014575 3.65766387 3.57871326 3.49865673
|
||||
3.41754964 3.33544681 3.25240108]
|
||||
MSE after scaling: 0.00
|
||||
R2 score for scaled data: 0.99
|
||||
R2 score for scaled data: 1.00
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2849,16 +2855,16 @@ The following simple Python instructions define our <span class="math notranslat
|
||||
</ol>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
MSE(\boldsymbol{y},\boldsymbol{\tilde{y}}) = \frac{1}{n}
|
||||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||||
\]</div>
|
||||
<p>and the <span class="math notranslate nohighlight">\(R^2\)</span> score function.
|
||||
If <span class="math notranslate nohighlight">\(\tilde{\hat{y}}_i\)</span> is the predicted value of the <span class="math notranslate nohighlight">\(i-th\)</span> sample and <span class="math notranslate nohighlight">\(y_i\)</span> is the corresponding true value, then the score <span class="math notranslate nohighlight">\(R^2\)</span> is defined as</p>
|
||||
If <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}_i\)</span> is the predicted value of the <span class="math notranslate nohighlight">\(i-th\)</span> sample and <span class="math notranslate nohighlight">\(y_i\)</span> is the corresponding true value, then the score <span class="math notranslate nohighlight">\(R^2\)</span> is defined as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
\]</div>
|
||||
<p>where we have defined the mean value of <span class="math notranslate nohighlight">\(\hat{y}\)</span> as</p>
|
||||
<p>where we have defined the mean value of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
||||
|
||||
Reference in New Issue
Block a user