lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added more scikit-learn functionality

Browse Source
This commit is contained in:
mhjensen
2018-05-28 08:41:15 -04:00
parent b4246a0ebd
commit eb8d0529b0
10 changed files with 247 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/How2ReadData/html/How2ReadData-bs.html
+38 -11
View File
@@ -191,7 +191,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>May 27, 2018</h4></center> <!-- date -->
<center><h4>May 28, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- potential-jumbotron-button -->
@@ -291,9 +291,9 @@ into the hassle of exploring how to set up dependencies and paths, we
recommend two widely used distrubutions which set up all relevant
dependencies for Python, namely
<ol>
<ul>
<li> <a href="https://docs.anaconda.com/" target="_self">Anaconda</a>,</li>
</ol>
</ul>
which is an open source
distribution of the Python and R programming languages for large-scale
@@ -301,9 +301,9 @@ data processing, predictive analytics, and scientific computing, that
aims to simplify package management and deployment. Package versions
are managed by the package management system <b>conda</b>.
<ol>
<ul>
<li> <a href="https://www.enthought.com/product/canopy/" target="_self">Enthought canopy</a></li>
</ol>
</ul>
is a Python
distribution for scientific and analytic computing distribution and
@@ -530,16 +530,18 @@ relative error.
As mentioned above, <b>scikit-learn</b> has an impressive functionality.
We can for example extract the values of \( \alpha \) and \( \beta \) and
their error estimates, or the variance and standard deviation and many
other properties from the statistical data analysis. Here we show an
example of the functionality of scikit-learn.
other properties from the statistical data analysis.
<p>
Here we show an
example of the functionality of scikit-learn.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LinearRegression
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> mean_squared_error, r2_score
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> mean_squared_error, r2_score, mean_squared_log_error, mean_absolute_error
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> <span style="color: #666666">2.0+</span> <span style="color: #666666">5*</span>x<span style="color: #666666">+0.5</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
@@ -552,6 +554,10 @@ ypredict <span style="color: #666666">=</span> linreg<span style="color: #666666
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Mean squared error: </span><span style="color: #BB6688; font-weight: bold">%.2f</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> mean_squared_error(y, ypredict))
<span style="color: #408080; font-style: italic"># Explained variance score: 1 is perfect prediction </span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Variance score: </span><span style="color: #BB6688; font-weight: bold">%.2f</span><span style="color: #BA2121">&#39;</span> <span style="color: #666666">%</span> r2_score(y, ypredict))
<span style="color: #408080; font-style: italic"># Mean squared log error </span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Mean squared log error: </span><span style="color: #BB6688; font-weight: bold">%.2f</span><span style="color: #BA2121">&#39;</span> <span style="color: #666666">%</span> mean_squared_log_error(y, ypredict) )
<span style="color: #408080; font-style: italic"># Mean absolute error </span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Mean absolute error: </span><span style="color: #BB6688; font-weight: bold">%.2f</span><span style="color: #BA2121">&#39;</span> <span style="color: #666666">%</span> mean_absolute_error(y, ypredict))
plt<span style="color: #666666">.</span>plot(x, ypredict, <span style="color: #BA2121">&quot;r-&quot;</span>)
plt<span style="color: #666666">.</span>plot(x, y ,<span style="color: #BA2121">&#39;ro&#39;</span>)
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">0.0</span>,<span style="color: #666666">1.0</span>,<span style="color: #666666">1.5</span>, <span style="color: #666666">7.0</span>])
@@ -581,17 +587,38 @@ constant model that always predicts the expected value of \( \hat{y} \),
disregarding the input features, would get a \( R^2 \) score of \( 0.0 \).
<p>
If \( \tilde{\hat{y}}_i \) is the predicted value of the i-th sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
$$
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
$$
where the mean value
where we have defined the mean value of \( \hat{y} \) as
$$
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
$$
We will discuss in more detail these and more function in the various lectures.
Another quantity will meet again in our discussions of regression analysis is
mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error.
The MAE is defined as follows
$$
\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
$$
Finally we present the
squared logarithmic (quadratic) error
$$
\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2,
$$
<p>
where \( \log_e (x) \) stands for the natural logarithm of \( x \). This error
estimate is best to use when targets having exponential growth, such
as population counts, average sales of a commodity over a span of
years etc.
<p>
We will discuss in more
detail these and more function in the various lectures.
<p>
Another useful Python package is
doc/pub/How2ReadData/html/How2ReadData-reveal.html
+42 -12
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>May 27, 2018</h4></center> <!-- date -->
<center><h4>May 28, 2018</h4></center> <!-- date -->
<br>
<p>
@@ -254,9 +254,9 @@ into the hassle of exploring how to set up dependencies and paths, we
recommend two widely used distrubutions which set up all relevant
dependencies for Python, namely
<ol>
<ul>
<p><li> <a href="https://docs.anaconda.com/" target="_blank">Anaconda</a>,</li>
</ol>
</ul>
<p>
which is an open source
@@ -265,9 +265,9 @@ data processing, predictive analytics, and scientific computing, that
aims to simplify package management and deployment. Package versions
are managed by the package management system <b>conda</b>.
<ol>
<ul>
<p><li> <a href="https://www.enthought.com/product/canopy/" target="_blank">Enthought canopy</a></li>
</ol>
</ul>
<p>
is a Python
@@ -503,16 +503,18 @@ relative error.
As mentioned above, <b>scikit-learn</b> has an impressive functionality.
We can for example extract the values of \( \alpha \) and \( \beta \) and
their error estimates, or the variance and standard deviation and many
other properties from the statistical data analysis. Here we show an
example of the functionality of scikit-learn.
other properties from the statistical data analysis.
<p>
Here we show an
example of the functionality of scikit-learn.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.metrics</span> <span style="color: #8B008B; font-weight: bold">import</span> mean_squared_error, r2_score
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.metrics</span> <span style="color: #8B008B; font-weight: bold">import</span> mean_squared_error, r2_score, mean_squared_log_error, mean_absolute_error
x = np.random.rand(<span style="color: #B452CD">100</span>,<span style="color: #B452CD">1</span>)
y = <span style="color: #B452CD">2.0</span>+ <span style="color: #B452CD">5</span>*x+<span style="color: #B452CD">0.5</span>np.random.randn(<span style="color: #B452CD">100</span>,<span style="color: #B452CD">1</span>)
@@ -525,6 +527,10 @@ ypredict = linreg.predict(x)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Mean squared error: %.2f&quot;</span> % mean_squared_error(y, ypredict))
<span style="color: #228B22"># Explained variance score: 1 is perfect prediction </span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Variance score: %.2f&#39;</span> % r2_score(y, ypredict))
<span style="color: #228B22"># Mean squared log error </span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Mean squared log error: %.2f&#39;</span> % mean_squared_log_error(y, ypredict) )
<span style="color: #228B22"># Mean absolute error </span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Mean absolute error: %.2f&#39;</span> % mean_absolute_error(y, ypredict))
plt.plot(x, ypredict, <span style="color: #CD5555">&quot;r-&quot;</span>)
plt.plot(x, y ,<span style="color: #CD5555">&#39;ro&#39;</span>)
plt.axis([<span style="color: #B452CD">0.0</span>,<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">1.5</span>, <span style="color: #B452CD">7.0</span>])
@@ -556,22 +562,46 @@ constant model that always predicts the expected value of \( \hat{y} \),
disregarding the input features, would get a \( R^2 \) score of \( 0.0 \).
<p>
If \( \tilde{\hat{y}}_i \) is the predicted value of the i-th sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
<p>&nbsp;<br>
$$
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
$$
<p>&nbsp;<br>
where the mean value
where we have defined the mean value of \( \hat{y} \) as
<p>&nbsp;<br>
$$
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
$$
<p>&nbsp;<br>
We will discuss in more detail these and more function in the various lectures.
Another quantity will meet again in our discussions of regression analysis is
mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error.
The MAE is defined as follows
<p>&nbsp;<br>
$$
\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
$$
<p>&nbsp;<br>
Finally we present the
squared logarithmic (quadratic) error
<p>&nbsp;<br>
$$
\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2,
$$
<p>&nbsp;<br>
<p>
where \( \log_e (x) \) stands for the natural logarithm of \( x \). This error
estimate is best to use when targets having exponential growth, such
as population counts, average sales of a commodity over a span of
years etc.
<p>
We will discuss in more
detail these and more function in the various lectures.
<p>
Another useful Python package is
doc/pub/How2ReadData/html/How2ReadData-solarized.html
+38 -11
View File
@@ -151,7 +151,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>May 27, 2018</h4></center> <!-- date -->
<center><h4>May 28, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -248,9 +248,9 @@ into the hassle of exploring how to set up dependencies and paths, we
recommend two widely used distrubutions which set up all relevant
dependencies for Python, namely
<ol>
<ul>
<li> <a href="https://docs.anaconda.com/" target="_blank">Anaconda</a>,</li>
</ol>
</ul>
which is an open source
distribution of the Python and R programming languages for large-scale
@@ -258,9 +258,9 @@ data processing, predictive analytics, and scientific computing, that
aims to simplify package management and deployment. Package versions
are managed by the package management system <b>conda</b>.
<ol>
<ul>
<li> <a href="https://www.enthought.com/product/canopy/" target="_blank">Enthought canopy</a></li>
</ol>
</ul>
is a Python
distribution for scientific and analytic computing distribution and
@@ -487,16 +487,18 @@ relative error.
As mentioned above, <b>scikit-learn</b> has an impressive functionality.
We can for example extract the values of \( \alpha \) and \( \beta \) and
their error estimates, or the variance and standard deviation and many
other properties from the statistical data analysis. Here we show an
example of the functionality of scikit-learn.
other properties from the statistical data analysis.
<p>
Here we show an
example of the functionality of scikit-learn.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.metrics</span> <span style="color: #8B008B; font-weight: bold">import</span> mean_squared_error, r2_score
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.metrics</span> <span style="color: #8B008B; font-weight: bold">import</span> mean_squared_error, r2_score, mean_squared_log_error, mean_absolute_error
x = np.random.rand(<span style="color: #B452CD">100</span>,<span style="color: #B452CD">1</span>)
y = <span style="color: #B452CD">2.0</span>+ <span style="color: #B452CD">5</span>*x+<span style="color: #B452CD">0.5</span>np.random.randn(<span style="color: #B452CD">100</span>,<span style="color: #B452CD">1</span>)
@@ -509,6 +511,10 @@ ypredict = linreg.predict(x)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Mean squared error: %.2f&quot;</span> % mean_squared_error(y, ypredict))
<span style="color: #228B22"># Explained variance score: 1 is perfect prediction </span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Variance score: %.2f&#39;</span> % r2_score(y, ypredict))
<span style="color: #228B22"># Mean squared log error </span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Mean squared log error: %.2f&#39;</span> % mean_squared_log_error(y, ypredict) )
<span style="color: #228B22"># Mean absolute error </span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Mean absolute error: %.2f&#39;</span> % mean_absolute_error(y, ypredict))
plt.plot(x, ypredict, <span style="color: #CD5555">&quot;r-&quot;</span>)
plt.plot(x, y ,<span style="color: #CD5555">&#39;ro&#39;</span>)
plt.axis([<span style="color: #B452CD">0.0</span>,<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">1.5</span>, <span style="color: #B452CD">7.0</span>])
@@ -538,17 +544,38 @@ constant model that always predicts the expected value of \( \hat{y} \),
disregarding the input features, would get a \( R^2 \) score of \( 0.0 \).
<p>
If \( \tilde{\hat{y}}_i \) is the predicted value of the i-th sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
$$
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
$$
where the mean value
where we have defined the mean value of \( \hat{y} \) as
$$
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
$$
We will discuss in more detail these and more function in the various lectures.
Another quantity will meet again in our discussions of regression analysis is
mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error.
The MAE is defined as follows
$$
\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
$$
Finally we present the
squared logarithmic (quadratic) error
$$
\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2,
$$
<p>
where \( \log_e (x) \) stands for the natural logarithm of \( x \). This error
estimate is best to use when targets having exponential growth, such
as population counts, average sales of a commodity over a span of
years etc.
<p>
We will discuss in more
detail these and more function in the various lectures.
<p>
Another useful Python package is
doc/pub/How2ReadData/html/How2ReadData.html
+38 -11
View File
@@ -156,7 +156,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>May 27, 2018</h4></center> <!-- date -->
<center><h4>May 28, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -253,9 +253,9 @@ into the hassle of exploring how to set up dependencies and paths, we
recommend two widely used distrubutions which set up all relevant
dependencies for Python, namely
<ol>
<ul>
<li> <a href="https://docs.anaconda.com/" target="_blank">Anaconda</a>,</li>
</ol>
</ul>
which is an open source
distribution of the Python and R programming languages for large-scale
@@ -263,9 +263,9 @@ data processing, predictive analytics, and scientific computing, that
aims to simplify package management and deployment. Package versions
are managed by the package management system <b>conda</b>.
<ol>
<ul>
<li> <a href="https://www.enthought.com/product/canopy/" target="_blank">Enthought canopy</a></li>
</ol>
</ul>
is a Python
distribution for scientific and analytic computing distribution and
@@ -492,16 +492,18 @@ relative error.
As mentioned above, <b>scikit-learn</b> has an impressive functionality.
We can for example extract the values of \( \alpha \) and \( \beta \) and
their error estimates, or the variance and standard deviation and many
other properties from the statistical data analysis. Here we show an
example of the functionality of scikit-learn.
other properties from the statistical data analysis.
<p>
Here we show an
example of the functionality of scikit-learn.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LinearRegression
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> mean_squared_error, r2_score
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> mean_squared_error, r2_score, mean_squared_log_error, mean_absolute_error
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> <span style="color: #666666">2.0+</span> <span style="color: #666666">5*</span>x<span style="color: #666666">+0.5</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
@@ -514,6 +516,10 @@ ypredict <span style="color: #666666">=</span> linreg<span style="color: #666666
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Mean squared error: </span><span style="color: #BB6688; font-weight: bold">%.2f</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> mean_squared_error(y, ypredict))
<span style="color: #408080; font-style: italic"># Explained variance score: 1 is perfect prediction </span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Variance score: </span><span style="color: #BB6688; font-weight: bold">%.2f</span><span style="color: #BA2121">&#39;</span> <span style="color: #666666">%</span> r2_score(y, ypredict))
<span style="color: #408080; font-style: italic"># Mean squared log error </span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Mean squared log error: </span><span style="color: #BB6688; font-weight: bold">%.2f</span><span style="color: #BA2121">&#39;</span> <span style="color: #666666">%</span> mean_squared_log_error(y, ypredict) )
<span style="color: #408080; font-style: italic"># Mean absolute error </span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Mean absolute error: </span><span style="color: #BB6688; font-weight: bold">%.2f</span><span style="color: #BA2121">&#39;</span> <span style="color: #666666">%</span> mean_absolute_error(y, ypredict))
plt<span style="color: #666666">.</span>plot(x, ypredict, <span style="color: #BA2121">&quot;r-&quot;</span>)
plt<span style="color: #666666">.</span>plot(x, y ,<span style="color: #BA2121">&#39;ro&#39;</span>)
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">0.0</span>,<span style="color: #666666">1.0</span>,<span style="color: #666666">1.5</span>, <span style="color: #666666">7.0</span>])
@@ -543,17 +549,38 @@ constant model that always predicts the expected value of \( \hat{y} \),
disregarding the input features, would get a \( R^2 \) score of \( 0.0 \).
<p>
If \( \tilde{\hat{y}}_i \) is the predicted value of the i-th sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
$$
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
$$
where the mean value
where we have defined the mean value of \( \hat{y} \) as
$$
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
$$
We will discuss in more detail these and more function in the various lectures.
Another quantity will meet again in our discussions of regression analysis is
mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error.
The MAE is defined as follows
$$
\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
$$
Finally we present the
squared logarithmic (quadratic) error
$$
\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2,
$$
<p>
where \( \log_e (x) \) stands for the natural logarithm of \( x \). This error
estimate is best to use when targets having exponential growth, such
as population counts, average sales of a commodity over a span of
years etc.
<p>
We will discuss in more
detail these and more function in the various lectures.
<p>
Another useful Python package is
doc/pub/How2ReadData/ipynb/How2ReadData.ipynb
+55 -8
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **May 27, 2018**\n",
"Date: **May 28, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -96,7 +96,7 @@
"recommend two widely used distrubutions which set up all relevant\n",
"dependencies for Python, namely \n",
"\n",
"1. [Anaconda](https://docs.anaconda.com/), \n",
"* [Anaconda](https://docs.anaconda.com/), \n",
"\n",
"which is an open source\n",
"distribution of the Python and R programming languages for large-scale\n",
@@ -104,7 +104,7 @@
"aims to simplify package management and deployment. Package versions\n",
"are managed by the package management system **conda**. \n",
"\n",
"1. [Enthought canopy](https://www.enthought.com/product/canopy/) \n",
"* [Enthought canopy](https://www.enthought.com/product/canopy/) \n",
"\n",
"is a Python\n",
"distribution for scientific and analytic computing distribution and\n",
@@ -382,7 +382,9 @@
"As mentioned above, **scikit-learn** has an impressive functionality.\n",
"We can for example extract the values of $\\alpha$ and $\\beta$ and\n",
"their error estimates, or the variance and standard deviation and many\n",
"other properties from the statistical data analysis. Here we show an\n",
"other properties from the statistical data analysis. \n",
"\n",
"Here we show an\n",
"example of the functionality of scikit-learn."
]
},
@@ -397,7 +399,7 @@
"import numpy as np \n",
"import matplotlib.pyplot as plt \n",
"from sklearn.linear_model import LinearRegression \n",
"from sklearn.metrics import mean_squared_error, r2_score\n",
"from sklearn.metrics import mean_squared_error, r2_score, mean_squared_log_error, mean_absolute_error\n",
"\n",
"x = np.random.rand(100,1)\n",
"y = 2.0+ 5*x+0.5np.random.randn(100,1)\n",
@@ -410,6 +412,10 @@
"print(\"Mean squared error: %.2f\" % mean_squared_error(y, ypredict))\n",
"# Explained variance score: 1 is perfect prediction \n",
"print('Variance score: %.2f' % r2_score(y, ypredict))\n",
"# Mean squared log error \n",
"print('Mean squared log error: %.2f' % mean_squared_log_error(y, ypredict) )\n",
"# Mean absolute error \n",
"print('Mean absolute error: %.2f' % mean_absolute_error(y, ypredict))\n",
"plt.plot(x, ypredict, \"r-\")\n",
"plt.plot(x, y ,'ro')\n",
"plt.axis([0.0,1.0,1.5, 7.0])\n",
@@ -452,7 +458,7 @@
"constant model that always predicts the expected value of $\\hat{y}$,\n",
"disregarding the input features, would get a $R^2$ score of $0.0$.\n",
"\n",
"If $\\tilde{\\hat{y}}_i$ is the predicted value of the i-th sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as"
"If $\\tilde{\\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as"
]
},
{
@@ -468,7 +474,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"where the mean value"
"where we have defined the mean value of $\\hat{y}$ as"
]
},
{
@@ -484,7 +490,48 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We will discuss in more detail these and more function in the various lectures.\n",
"Another quantity will meet again in our discussions of regression analysis is \n",
" mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the $l1$-norm loss. In our discussion above we presented the relative error.\n",
"The MAE is defined as follows"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\text{MAE}(\\hat{y}, \\hat{\\tilde{y}}) = \\frac{1}{n} \\sum_{i=0}^{n-1} \\left| y_i - \\tilde{y}_i \\right|.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally we present the \n",
"squared logarithmic (quadratic) error"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\text{MSLE}(\\hat{y}, \\hat{\\tilde{y}}) = \\frac{1}{n} \\sum_{i=0}^{n - 1} (\\log_e (1 + y_i) - \\log_e (1 + \\tilde{y}_i) )^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where $\\log_e (x)$ stands for the natural logarithm of $x$. This error\n",
"estimate is best to use when targets having exponential growth, such\n",
"as population counts, average sales of a commodity over a span of\n",
"years etc. \n",
"\n",
"We will discuss in more\n",
"detail these and more function in the various lectures.\n",
"\n",
"Another useful Python package is\n",
"[pandas](https://pandas.pydata.org/), which is an open source library\n",
doc/pub/How2ReadData/ipynb/ipynb-How2ReadData-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/How2ReadData/pdf/How2ReadData-beamer-handouts2x3.pdf
BIN
View File
Binary file not shown.
doc/pub/How2ReadData/pdf/How2ReadData-beamer.pdf
BIN
View File
Binary file not shown.
doc/pub/How2ReadData/pdf/How2ReadData-minted.pdf
BIN
View File
Binary file not shown.
doc/src/How2ReadData/How2ReadData.do.txt
+36 -8
View File
@@ -85,7 +85,7 @@ into the hassle of exploring how to set up dependencies and paths, we
recommend two widely used distrubutions which set up all relevant
dependencies for Python, namely
o "Anaconda":"https://docs.anaconda.com/",
* "Anaconda":"https://docs.anaconda.com/",
which is an open source
distribution of the Python and R programming languages for large-scale
@@ -93,7 +93,7 @@ data processing, predictive analytics, and scientific computing, that
aims to simplify package management and deployment. Package versions
are managed by the package management system _conda_.
o "Enthought canopy":"https://www.enthought.com/product/canopy/"
* "Enthought canopy":"https://www.enthought.com/product/canopy/"
is a Python
distribution for scientific and analytic computing distribution and
@@ -304,13 +304,15 @@ relative error.
As mentioned above, _scikit-learn_ has an impressive functionality.
We can for example extract the values of $\alpha$ and $\beta$ and
their error estimates, or the variance and standard deviation and many
other properties from the statistical data analysis. Here we show an
example of the functionality of scikit-learn.
other properties from the statistical data analysis.
Here we show an
example of the functionality of scikit-learn.
!bc pycod
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.metrics import mean_squared_error, r2_score, mean_squared_log_error, mean_absolute_error
x = np.random.rand(100,1)
y = 2.0+ 5*x+0.5np.random.randn(100,1)
@@ -323,6 +325,10 @@ print('Coefficient beta : \n', linreg.coef_)
print("Mean squared error: %.2f" % mean_squared_error(y, ypredict))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % r2_score(y, ypredict))
# Mean squared log error
print('Mean squared log error: %.2f' % mean_squared_log_error(y, ypredict) )
# Mean absolute error
print('Mean absolute error: %.2f' % mean_absolute_error(y, ypredict))
plt.plot(x, ypredict, "r-")
plt.plot(x, y ,'ro')
plt.axis([0.0,1.0,1.5, 7.0])
@@ -351,19 +357,41 @@ can be negative (because the model can be arbitrarily worse). A
constant model that always predicts the expected value of $\hat{y}$,
disregarding the input features, would get a $R^2$ score of $0.0$.
If $\tilde{\hat{y}}_i$ is the predicted value of the i-th sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
!bt
\[
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
\]
!et
where the mean value
where we have defined the mean value of $\hat{y}$ as
!bt
\[
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
\]
!et
We will discuss in more detail these and more function in the various lectures.
Another quantity will meet again in our discussions of regression analysis is
mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the $l1$-norm loss. In our discussion above we presented the relative error.
The MAE is defined as follows
!bt
\[
\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
\]
!et
Finally we present the
squared logarithmic (quadratic) error
!bt
\[
\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2,
\]
!et
where $\log_e (x)$ stands for the natural logarithm of $x$. This error
estimate is best to use when targets having exponential growth, such
as population counts, average sales of a commodity over a span of
years etc.
We will discuss in more
detail these and more function in the various lectures.
Another useful Python package is
"pandas":"https://pandas.pydata.org/", which is an open source library