Added more scikit-learn functions
This commit is contained in:
mhjensen
@@ -191,7 +191,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>May 26, 2018</h4></center> <!-- date -->
|
||||
<center><h4>May 27, 2018</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- potential-jumbotron-button -->
|
||||
@@ -204,27 +204,27 @@ MathJax.Hub.Config({
|
||||
<p>
|
||||
Our emphasis throughout this series of lectures
|
||||
is on understanding the mathematical aspects of
|
||||
different algorithms used in the fields of data analysis and machine learning.
|
||||
different algorithms used in the fields of data analysis and machine learning.
|
||||
|
||||
<p>
|
||||
However, where possible we will emphasize the
|
||||
importance of using available software. We start thus with a hands-on
|
||||
and top-down approach to machine learning. The aim is thus to start with
|
||||
relevant data and use these to introduce statistical data analysis
|
||||
relevant data or data we have produced
|
||||
and use these to introduce statistical data analysis
|
||||
concepts and machine learning algorithms before we delve into the
|
||||
algorithms themselves. The examples we will use in the beginning, start with simple
|
||||
polynomials with random noise added, and using the Python
|
||||
software package <a href="http://scikit-learn.org/stable/" target="_self">Scikit-learn</a> we
|
||||
will introduce various machine learning algorithms to make fits of
|
||||
polynomials with random noise added. We will use the Python
|
||||
software package <a href="http://scikit-learn.org/stable/" target="_self">Scikit-learn</a> and
|
||||
introduce various machine learning algorithms to make fits of
|
||||
the data and predictions. We move thereafter to more interesting
|
||||
cases such as the simulation of financial transactions or disease
|
||||
models. These are examples where we can easily set up the data and
|
||||
then use machine learning algorithms included in for example
|
||||
<b>scikit-learn</b>. Another model we will consider is the so-called Ising
|
||||
model. Here we will use this model to produce data for selected spin
|
||||
configurations and attempt to classify the data. Finally, our last
|
||||
example consists of economic data from the OECD.
|
||||
<b>scikit-learn</b>.
|
||||
|
||||
<p>
|
||||
All these examples will serve us the purpose of getting us started, furthermore, they
|
||||
These examples will serve us the purpose of getting started. Furthermore, they
|
||||
allow us to catch more than two birds with a stone. They will allow us
|
||||
to bring in some programming specific topics and tools as well as
|
||||
showing the power of various Python (and R) packages for machine
|
||||
@@ -242,7 +242,7 @@ Moreover, the examples we introduce will serve as inputs to many of our discussi
|
||||
|
||||
<p>
|
||||
We will make extensive use of Python as programming language and its
|
||||
myriad of available libraries. Furthermore, you will find
|
||||
myriad of available libraries. You will find
|
||||
IPython/Jupyter notebooks invaluable in your work. You can run <b>R</b>
|
||||
codes in the Jupyter/IPython notebooks, with the immediate benefit of
|
||||
visualizing your data. You can also use compiled languages like C++,
|
||||
@@ -254,16 +254,23 @@ a Jupyter notebook.
|
||||
<p>
|
||||
If you have Python installed (we recommend Python3) and you feel
|
||||
pretty familiar with installing different packages, we recommend that
|
||||
you install the following Python packages via <b>pip</b> as o pip install
|
||||
numpy scipy matplotlib ipython scikit-learn mglearn sympy pandas
|
||||
pillow For Python3, replace <b>pip</b> with <b>pip3</b>.
|
||||
you install the following Python packages via <b>pip</b> as
|
||||
|
||||
<ol>
|
||||
<li> pip install numpy scipy matplotlib ipython scikit-learn mglearn sympy pandas pillow</li>
|
||||
</ol>
|
||||
|
||||
For Python3, replace <b>pip</b> with <b>pip3</b>.
|
||||
|
||||
<p>
|
||||
For OSX users we recommend also, after having installed Xcode, to
|
||||
For OSX users we recommend, after having installed Xcode, to
|
||||
install <b>brew</b>. Brew allows for a seamless installation of additional
|
||||
software via for example o brew install python3
|
||||
software via for example
|
||||
|
||||
<ol>
|
||||
<li> brew install python3</li>
|
||||
</ol>
|
||||
|
||||
<p>
|
||||
For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,
|
||||
you can use <b>pip</b> as well and simply install Python as
|
||||
|
||||
@@ -282,13 +289,23 @@ etc etc.
|
||||
If you don't want to perform these operations separately and venture
|
||||
into the hassle of exploring how to set up dependencies and paths, we
|
||||
recommend two widely used distrubutions which set up all relevant
|
||||
dependencies for Python, namely o
|
||||
<a href="https://docs.anaconda.com/" target="_self">Anaconda</a>, which is an open source
|
||||
dependencies for Python, namely
|
||||
|
||||
<ol>
|
||||
<li> <a href="https://docs.anaconda.com/" target="_self">Anaconda</a>,</li>
|
||||
</ol>
|
||||
|
||||
which is an open source
|
||||
distribution of the Python and R programming languages for large-scale
|
||||
data processing, predictive analytics, and scientific computing, that
|
||||
aims to simplify package management and deployment. Package versions
|
||||
are managed by the package management system <b>conda</b> o <a href="https://www.enthought.com/product/canopy/" target="_self">Enthought
|
||||
canopy</a> is a Python
|
||||
are managed by the package management system <b>conda</b>.
|
||||
|
||||
<ol>
|
||||
<li> <a href="https://www.enthought.com/product/canopy/" target="_self">Enthought canopy</a></li>
|
||||
</ol>
|
||||
|
||||
is a Python
|
||||
distribution for scientific and analytic computing distribution and
|
||||
analysis environment, available for free and under a commercial
|
||||
license.
|
||||
@@ -311,8 +328,8 @@ and allows for an easy usage of the tools we will discuss in these
|
||||
texts.
|
||||
|
||||
<p>
|
||||
To install <b>R</b> with Jupyter notebook <a href="https://mpacer.org/maths/r-kernel-for-ipython-notebook" target="_self">following the link
|
||||
here</a>
|
||||
To install <b>R</b> with Jupyter notebook
|
||||
<a href="https://mpacer.org/maths/r-kernel-for-ipython-notebook" target="_self">follow the link here</a>
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
@@ -336,23 +353,27 @@ capabilities with minimal rewrites of your codes. With its
|
||||
versatility, including symbolic operations, Python offers a unique
|
||||
computational environment. Your Jupyter/IPython notebook can easily be
|
||||
converted into a nicely rendered <b>PDF</b> file or a Latex file for
|
||||
further processing. For example, convert to latex as
|
||||
further processing. For example, convert to latex as
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=text typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pycod jupyter nbconvert filename.ipynb --to latex
|
||||
</pre></div>
|
||||
<p>
|
||||
And to add more versatility, symbolic Python package <a href="http://www.sympy.org/en/index.html" target="_self">SymPy</a> is Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) and is entirely written in Python.
|
||||
And to add more versatility, the Python package <a href="http://www.sympy.org/en/index.html" target="_self">SymPy</a> is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) and is entirely written in Python.
|
||||
|
||||
<p>
|
||||
Finally, if you wish to use the light mark-up language <a href="https://github.com/hplgit/doconce" target="_self">doconce</a> you can convert a standard ascii text file into various HTML
|
||||
Finally, if you wish to use the light mark-up language
|
||||
<a href="https://github.com/hplgit/doconce" target="_self">doconce</a> you can convert a standard ascii text file into various HTML
|
||||
formats, ipython notebooks, latex files, pdf files etc with minimal edits.
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec5" class="anchor">Simple linear regression model using <b>scikit-learn</b> </h2>
|
||||
|
||||
<p>
|
||||
We start with perhaps our simplest possible example, using <b>scikit-learn</b> to perform linear regression analysis on a data set produced by us.
|
||||
What follows is a simple Python code where we have defined function \( y \) in terms of the variable \( x \). Both are defined as vectors of dimension \( 1\times 100 \). The entries to the vector \( \hat{x} \) are given by random numbers generated with a uniform distribution with entries \( x_i \in [0,1] \) (more about probability distribution functions later). These values are then used to define a function \( y(x) \) (tabulated again as a vector) with a linear dependence on \( x \) plus a random noise added via the normal distribution.
|
||||
|
||||
@@ -391,7 +412,7 @@ prediction <b>ypredict</b> (\( \tilde{y} \)), which attempts at fitting our
|
||||
data with a straight line.
|
||||
|
||||
<p>
|
||||
The Python follows here.
|
||||
The Python code follows here.
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
@@ -468,11 +489,12 @@ employed are various variants of <b>gradient</b> methods. These will be
|
||||
discussed in more detail later. Again, you'll be surprised to hear that
|
||||
many practitioners minimize the above function ''by the eye', popularly dubbed as
|
||||
'chi by the eye'. That is, change a parameter and see (visually and numerically) that
|
||||
the your \( \chi^2 \) function becomes smaller.
|
||||
the \( \chi^2 \) function becomes smaller.
|
||||
|
||||
<p>
|
||||
There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define
|
||||
the relative error as
|
||||
|
||||
$$
|
||||
\epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}.
|
||||
$$
|
||||
@@ -499,76 +521,93 @@ plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">r'
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
</pre></div>
|
||||
<p>
|
||||
Depending on the parameter in front of the normal distribution, we may have a small or larger relative error. Try to play around with different training data sets and study (graphically) the value of the relative error.
|
||||
Depending on the parameter in front of the normal distribution, we may
|
||||
have a small or larger relative error. Try to play around with
|
||||
different training data sets and study (graphically) the value of the
|
||||
relative error.
|
||||
|
||||
<p>
|
||||
As mentioned above, <b>scikit-learn</b> has an impressive functionality.
|
||||
We can for example extract the values of \( \alpha \) and \( \beta \) and their error estimates,
|
||||
or the variance and standard deviation and many other properties from the statistical data analysis. Here we show an example of the functionality of scikit-learn.
|
||||
As mentioned above, <b>scikit-learn</b> has an impressive functionality.
|
||||
We can for example extract the values of \( \alpha \) and \( \beta \) and
|
||||
their error estimates, or the variance and standard deviation and many
|
||||
other properties from the statistical data analysis. Here we show an
|
||||
example of the functionality of scikit-learn.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LinearRegression
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LinearRegression
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> mean_squared_error, r2_score
|
||||
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
|
||||
y <span style="color: #666666">=</span> <span style="color: #666666">2*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
|
||||
y <span style="color: #666666">=</span> <span style="color: #666666">2.0+</span> <span style="color: #666666">5*</span>x<span style="color: #666666">+0.5</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
|
||||
linreg <span style="color: #666666">=</span> LinearRegression()
|
||||
linreg<span style="color: #666666">.</span>fit(x,y)
|
||||
ypredict <span style="color: #666666">=</span> linreg<span style="color: #666666">.</span>predict(x)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">'Coefficients: </span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121">'</span>, linreg<span style="color: #666666">.</span>coef_)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">'The intercept alpha: </span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121">'</span>, linreg<span style="color: #666666">.</span>intercept_)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">'Coefficient beta : </span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121">'</span>, linreg<span style="color: #666666">.</span>coef_)
|
||||
<span style="color: #408080; font-style: italic"># The mean squared error </span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"Mean squared error: </span><span style="color: #BB6688; font-weight: bold">%.2f</span><span style="color: #BA2121">"</span> <span style="color: #666666">%</span> mean_squared_error(y, ypredict))
|
||||
<span style="color: #408080; font-style: italic"># Explained variance score: 1 is perfect prediction </span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">'Variance score: </span><span style="color: #BB6688; font-weight: bold">%.2f</span><span style="color: #BA2121">'</span> <span style="color: #666666">%</span> r2_score(y, ypredict))
|
||||
plt<span style="color: #666666">.</span>plot(x, ypredict, <span style="color: #BA2121">"r-"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(x, y ,<span style="color: #BA2121">'ro'</span>)
|
||||
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">0</span>,<span style="color: #666666">1.0</span>,<span style="color: #666666">0</span>, <span style="color: #666666">5.0</span>])
|
||||
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">0.0</span>,<span style="color: #666666">1.0</span>,<span style="color: #666666">1.5</span>, <span style="color: #666666">7.0</span>])
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r'$x$'</span>)
|
||||
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r'$y$'</span>)
|
||||
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">r'Linear Regression fit '</span>)
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
</pre></div>
|
||||
<p>
|
||||
We will come to the definition of these outputs later.
|
||||
The function <b>coef</b> gives us the parameter \( \beta \) of our fit while <b>intercept</b> yields
|
||||
\( \alpha \). Depending on the constant in front of the normal distribution, we get values near or far from \( alpha =2 \) and \( \beta =5 \). Try to play around with different parameters in front of the normal distribution. The function <b>meansquarederror</b> gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as
|
||||
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||||
$$
|
||||
|
||||
<p>
|
||||
The smaller the value, the better the fit. Ideally we would like to
|
||||
have an MSE equal zero. The attentive reader has probably recognized
|
||||
this function as being similar to the \( \chi^2 \) function defined above.
|
||||
|
||||
<p>
|
||||
The <b>r2score</b> function computes \( R^2 \), the coefficient of
|
||||
determination. It provides a measure of how well future samples are
|
||||
likely to be predicted by the model. Best possible score is 1.0 and it
|
||||
can be negative (because the model can be arbitrarily worse). A
|
||||
constant model that always predicts the expected value of \( \hat{y} \),
|
||||
disregarding the input features, would get a \( R^2 \) score of \( 0.0 \).
|
||||
|
||||
<p>
|
||||
If \( \tilde{\hat{y}}_i \) is the predicted value of the i-th sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
|
||||
$$
|
||||
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
$$
|
||||
|
||||
where the mean value
|
||||
$$
|
||||
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
||||
$$
|
||||
|
||||
We will discuss in more detail these and more function in the various lectures.
|
||||
|
||||
<p>
|
||||
Another useful Python package is
|
||||
<a href="https://pandas.pydata.org/" target="_self">pandas</a>, which is an open source library
|
||||
providing high-performance, easy-to-use data structures and data
|
||||
analysis tools for Python. The following simple example shows an example on how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of <b>pandas</b>, in particular in connection with classification of data.
|
||||
analysis tools for Python. The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of <b>pandas</b>, in particular in connection with classification of data.
|
||||
|
||||
<p>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">scipy</span> <span style="color: #008000; font-weight: bold">import</span> sparse
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> display
|
||||
eye <span style="color: #666666">=</span> np<span style="color: #666666">.</span>eye(<span style="color: #666666">4</span>)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(eye)
|
||||
sparse_mtx <span style="color: #666666">=</span> sparse<span style="color: #666666">.</span>csr_matrix(eye)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(sparse_mtx)
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-10</span>,<span style="color: #666666">10</span>,<span style="color: #666666">100</span>)
|
||||
y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sin(x)
|
||||
plt<span style="color: #666666">.</span>plot(x,y,marker<span style="color: #666666">=</span><span style="color: #BA2121">'x'</span>)
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
data <span style="color: #666666">=</span> {<span style="color: #BA2121">'Name'</span>: [<span style="color: #BA2121">"John"</span>, <span style="color: #BA2121">"Anna"</span>, <span style="color: #BA2121">"Peter"</span>, <span style="color: #BA2121">"Linda"</span>], <span style="color: #BA2121">'Location'</span>: [<span style="color: #BA2121">"Nairobi"</span>, <span style="color: #BA2121">"Napoli"</span>, <span style="color: #BA2121">"London"</span>, <span style="color: #BA2121">"Buenos Aires"</span>], <span style="color: #BA2121">'Age'</span>:[<span style="color: #666666">51</span>, <span style="color: #666666">21</span>, <span style="color: #666666">34</span>, <span style="color: #666666">45</span>]}
|
||||
data_pandas <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(data)
|
||||
display(data_pandas)
|
||||
</pre></div>
|
||||
<p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
|
||||
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p> <br>
|
||||
<center><h4>May 26, 2018</h4></center> <!-- date -->
|
||||
<center><h4>May 27, 2018</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
@@ -164,27 +164,27 @@ MathJax.Hub.Config({
|
||||
<p>
|
||||
Our emphasis throughout this series of lectures
|
||||
is on understanding the mathematical aspects of
|
||||
different algorithms used in the fields of data analysis and machine learning.
|
||||
different algorithms used in the fields of data analysis and machine learning.
|
||||
|
||||
<p>
|
||||
However, where possible we will emphasize the
|
||||
importance of using available software. We start thus with a hands-on
|
||||
and top-down approach to machine learning. The aim is thus to start with
|
||||
relevant data and use these to introduce statistical data analysis
|
||||
relevant data or data we have produced
|
||||
and use these to introduce statistical data analysis
|
||||
concepts and machine learning algorithms before we delve into the
|
||||
algorithms themselves. The examples we will use in the beginning, start with simple
|
||||
polynomials with random noise added, and using the Python
|
||||
software package <a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a> we
|
||||
will introduce various machine learning algorithms to make fits of
|
||||
polynomials with random noise added. We will use the Python
|
||||
software package <a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a> and
|
||||
introduce various machine learning algorithms to make fits of
|
||||
the data and predictions. We move thereafter to more interesting
|
||||
cases such as the simulation of financial transactions or disease
|
||||
models. These are examples where we can easily set up the data and
|
||||
then use machine learning algorithms included in for example
|
||||
<b>scikit-learn</b>. Another model we will consider is the so-called Ising
|
||||
model. Here we will use this model to produce data for selected spin
|
||||
configurations and attempt to classify the data. Finally, our last
|
||||
example consists of economic data from the OECD.
|
||||
<b>scikit-learn</b>.
|
||||
|
||||
<p>
|
||||
All these examples will serve us the purpose of getting us started, furthermore, they
|
||||
These examples will serve us the purpose of getting started. Furthermore, they
|
||||
allow us to catch more than two birds with a stone. They will allow us
|
||||
to bring in some programming specific topics and tools as well as
|
||||
showing the power of various Python (and R) packages for machine
|
||||
@@ -202,7 +202,7 @@ Moreover, the examples we introduce will serve as inputs to many of our discussi
|
||||
|
||||
<p>
|
||||
We will make extensive use of Python as programming language and its
|
||||
myriad of available libraries. Furthermore, you will find
|
||||
myriad of available libraries. You will find
|
||||
IPython/Jupyter notebooks invaluable in your work. You can run <b>R</b>
|
||||
codes in the Jupyter/IPython notebooks, with the immediate benefit of
|
||||
visualizing your data. You can also use compiled languages like C++,
|
||||
@@ -214,16 +214,25 @@ a Jupyter notebook.
|
||||
<p>
|
||||
If you have Python installed (we recommend Python3) and you feel
|
||||
pretty familiar with installing different packages, we recommend that
|
||||
you install the following Python packages via <b>pip</b> as o pip install
|
||||
numpy scipy matplotlib ipython scikit-learn mglearn sympy pandas
|
||||
pillow For Python3, replace <b>pip</b> with <b>pip3</b>.
|
||||
you install the following Python packages via <b>pip</b> as
|
||||
|
||||
<ol>
|
||||
<p><li> pip install numpy scipy matplotlib ipython scikit-learn mglearn sympy pandas pillow</li>
|
||||
</ol>
|
||||
<p>
|
||||
|
||||
For Python3, replace <b>pip</b> with <b>pip3</b>.
|
||||
|
||||
<p>
|
||||
For OSX users we recommend also, after having installed Xcode, to
|
||||
For OSX users we recommend, after having installed Xcode, to
|
||||
install <b>brew</b>. Brew allows for a seamless installation of additional
|
||||
software via for example o brew install python3
|
||||
software via for example
|
||||
|
||||
<ol>
|
||||
<p><li> brew install python3</li>
|
||||
</ol>
|
||||
<p>
|
||||
|
||||
For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,
|
||||
you can use <b>pip</b> as well and simply install Python as
|
||||
|
||||
@@ -243,13 +252,25 @@ etc etc.
|
||||
If you don't want to perform these operations separately and venture
|
||||
into the hassle of exploring how to set up dependencies and paths, we
|
||||
recommend two widely used distrubutions which set up all relevant
|
||||
dependencies for Python, namely o
|
||||
<a href="https://docs.anaconda.com/" target="_blank">Anaconda</a>, which is an open source
|
||||
dependencies for Python, namely
|
||||
|
||||
<ol>
|
||||
<p><li> <a href="https://docs.anaconda.com/" target="_blank">Anaconda</a>,</li>
|
||||
</ol>
|
||||
<p>
|
||||
|
||||
which is an open source
|
||||
distribution of the Python and R programming languages for large-scale
|
||||
data processing, predictive analytics, and scientific computing, that
|
||||
aims to simplify package management and deployment. Package versions
|
||||
are managed by the package management system <b>conda</b> o <a href="https://www.enthought.com/product/canopy/" target="_blank">Enthought
|
||||
canopy</a> is a Python
|
||||
are managed by the package management system <b>conda</b>.
|
||||
|
||||
<ol>
|
||||
<p><li> <a href="https://www.enthought.com/product/canopy/" target="_blank">Enthought canopy</a></li>
|
||||
</ol>
|
||||
<p>
|
||||
|
||||
is a Python
|
||||
distribution for scientific and analytic computing distribution and
|
||||
analysis environment, available for free and under a commercial
|
||||
license.
|
||||
@@ -272,8 +293,8 @@ and allows for an easy usage of the tools we will discuss in these
|
||||
texts.
|
||||
|
||||
<p>
|
||||
To install <b>R</b> with Jupyter notebook <a href="https://mpacer.org/maths/r-kernel-for-ipython-notebook" target="_blank">following the link
|
||||
here</a>
|
||||
To install <b>R</b> with Jupyter notebook
|
||||
<a href="https://mpacer.org/maths/r-kernel-for-ipython-notebook" target="_blank">follow the link here</a>
|
||||
</section>
|
||||
|
||||
|
||||
@@ -297,23 +318,27 @@ capabilities with minimal rewrites of your codes. With its
|
||||
versatility, including symbolic operations, Python offers a unique
|
||||
computational environment. Your Jupyter/IPython notebook can easily be
|
||||
converted into a nicely rendered <b>PDF</b> file or a Latex file for
|
||||
further processing. For example, convert to latex as
|
||||
further processing. For example, convert to latex as
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=text typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>pycod jupyter nbconvert filename.ipynb --to latex
|
||||
</pre></div>
|
||||
<p>
|
||||
And to add more versatility, symbolic Python package <a href="http://www.sympy.org/en/index.html" target="_blank">SymPy</a> is Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) and is entirely written in Python.
|
||||
And to add more versatility, the Python package <a href="http://www.sympy.org/en/index.html" target="_blank">SymPy</a> is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) and is entirely written in Python.
|
||||
|
||||
<p>
|
||||
Finally, if you wish to use the light mark-up language <a href="https://github.com/hplgit/doconce" target="_blank">doconce</a> you can convert a standard ascii text file into various HTML
|
||||
Finally, if you wish to use the light mark-up language
|
||||
<a href="https://github.com/hplgit/doconce" target="_blank">doconce</a> you can convert a standard ascii text file into various HTML
|
||||
formats, ipython notebooks, latex files, pdf files etc with minimal edits.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec5">Simple linear regression model using <b>scikit-learn</b> </h2>
|
||||
|
||||
<p>
|
||||
We start with perhaps our simplest possible example, using <b>scikit-learn</b> to perform linear regression analysis on a data set produced by us.
|
||||
What follows is a simple Python code where we have defined function \( y \) in terms of the variable \( x \). Both are defined as vectors of dimension \( 1\times 100 \). The entries to the vector \( \hat{x} \) are given by random numbers generated with a uniform distribution with entries \( x_i \in [0,1] \) (more about probability distribution functions later). These values are then used to define a function \( y(x) \) (tabulated again as a vector) with a linear dependence on \( x \) plus a random noise added via the normal distribution.
|
||||
|
||||
@@ -354,7 +379,7 @@ prediction <b>ypredict</b> (\( \tilde{y} \)), which attempts at fitting our
|
||||
data with a straight line.
|
||||
|
||||
<p>
|
||||
The Python follows here.
|
||||
The Python code follows here.
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
@@ -435,11 +460,12 @@ employed are various variants of <b>gradient</b> methods. These will be
|
||||
discussed in more detail later. Again, you'll be surprised to hear that
|
||||
many practitioners minimize the above function ''by the eye', popularly dubbed as
|
||||
'chi by the eye'. That is, change a parameter and see (visually and numerically) that
|
||||
the your \( \chi^2 \) function becomes smaller.
|
||||
the \( \chi^2 \) function becomes smaller.
|
||||
|
||||
<p>
|
||||
There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define
|
||||
the relative error as
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}.
|
||||
@@ -468,72 +494,100 @@ plt.title(<span style="color: #CD5555">r'Relative error'</span>)
|
||||
plt.show()
|
||||
</pre></div>
|
||||
<p>
|
||||
Depending on the parameter in front of the normal distribution, we may have a small or larger relative error. Try to play around with different training data sets and study (graphically) the value of the relative error.
|
||||
Depending on the parameter in front of the normal distribution, we may
|
||||
have a small or larger relative error. Try to play around with
|
||||
different training data sets and study (graphically) the value of the
|
||||
relative error.
|
||||
|
||||
<p>
|
||||
As mentioned above, <b>scikit-learn</b> has an impressive functionality.
|
||||
We can for example extract the values of \( \alpha \) and \( \beta \) and their error estimates,
|
||||
or the variance and standard deviation and many other properties from the statistical data analysis. Here we show an example of the functionality of scikit-learn.
|
||||
As mentioned above, <b>scikit-learn</b> has an impressive functionality.
|
||||
We can for example extract the values of \( \alpha \) and \( \beta \) and
|
||||
their error estimates, or the variance and standard deviation and many
|
||||
other properties from the statistical data analysis. Here we show an
|
||||
example of the functionality of scikit-learn.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.metrics</span> <span style="color: #8B008B; font-weight: bold">import</span> mean_squared_error, r2_score
|
||||
|
||||
x = np.random.rand(<span style="color: #B452CD">100</span>,<span style="color: #B452CD">1</span>)
|
||||
y = <span style="color: #B452CD">2</span>*x+np.random.randn(<span style="color: #B452CD">100</span>,<span style="color: #B452CD">1</span>)
|
||||
y = <span style="color: #B452CD">2.0</span>+ <span style="color: #B452CD">5</span>*x+<span style="color: #B452CD">0.5</span>np.random.randn(<span style="color: #B452CD">100</span>,<span style="color: #B452CD">1</span>)
|
||||
linreg = LinearRegression()
|
||||
linreg.fit(x,y)
|
||||
ypredict = linreg.predict(x)
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">'Coefficients: \n'</span>, linreg.coef_)
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">'The intercept alpha: \n'</span>, linreg.intercept_)
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">'Coefficient beta : \n'</span>, linreg.coef_)
|
||||
<span style="color: #228B22"># The mean squared error </span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"Mean squared error: %.2f"</span> % mean_squared_error(y, ypredict))
|
||||
<span style="color: #228B22"># Explained variance score: 1 is perfect prediction </span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">'Variance score: %.2f'</span> % r2_score(y, ypredict))
|
||||
plt.plot(x, ypredict, <span style="color: #CD5555">"r-"</span>)
|
||||
plt.plot(x, y ,<span style="color: #CD5555">'ro'</span>)
|
||||
plt.axis([<span style="color: #B452CD">0</span>,<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">0</span>, <span style="color: #B452CD">5.0</span>])
|
||||
plt.axis([<span style="color: #B452CD">0.0</span>,<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">1.5</span>, <span style="color: #B452CD">7.0</span>])
|
||||
plt.xlabel(<span style="color: #CD5555">r'$x$'</span>)
|
||||
plt.ylabel(<span style="color: #CD5555">r'$y$'</span>)
|
||||
plt.title(<span style="color: #CD5555">r'Linear Regression fit '</span>)
|
||||
plt.show()
|
||||
</pre></div>
|
||||
<p>
|
||||
We will come to the definition of these outputs later.
|
||||
The function <b>coef</b> gives us the parameter \( \beta \) of our fit while <b>intercept</b> yields
|
||||
\( \alpha \). Depending on the constant in front of the normal distribution, we get values near or far from \( alpha =2 \) and \( \beta =5 \). Try to play around with different parameters in front of the normal distribution. The function <b>meansquarederror</b> gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as
|
||||
<p> <br>
|
||||
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
The smaller the value, the better the fit. Ideally we would like to
|
||||
have an MSE equal zero. The attentive reader has probably recognized
|
||||
this function as being similar to the \( \chi^2 \) function defined above.
|
||||
|
||||
<p>
|
||||
The <b>r2score</b> function computes \( R^2 \), the coefficient of
|
||||
determination. It provides a measure of how well future samples are
|
||||
likely to be predicted by the model. Best possible score is 1.0 and it
|
||||
can be negative (because the model can be arbitrarily worse). A
|
||||
constant model that always predicts the expected value of \( \hat{y} \),
|
||||
disregarding the input features, would get a \( R^2 \) score of \( 0.0 \).
|
||||
|
||||
<p>
|
||||
If \( \tilde{\hat{y}}_i \) is the predicted value of the i-th sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
|
||||
<p> <br>
|
||||
$$
|
||||
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
where the mean value
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
We will discuss in more detail these and more function in the various lectures.
|
||||
|
||||
<p>
|
||||
Another useful Python package is
|
||||
<a href="https://pandas.pydata.org/" target="_blank">pandas</a>, which is an open source library
|
||||
providing high-performance, easy-to-use data structures and data
|
||||
analysis tools for Python. The following simple example shows an example on how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of <b>pandas</b>, in particular in connection with classification of data.
|
||||
analysis tools for Python. The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of <b>pandas</b>, in particular in connection with classification of data.
|
||||
|
||||
<p>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">scipy</span> <span style="color: #8B008B; font-weight: bold">import</span> sparse
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">IPython.display</span> <span style="color: #8B008B; font-weight: bold">import</span> display
|
||||
eye = np.eye(<span style="color: #B452CD">4</span>)
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(eye)
|
||||
sparse_mtx = sparse.csr_matrix(eye)
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(sparse_mtx)
|
||||
x = np.linspace(-<span style="color: #B452CD">10</span>,<span style="color: #B452CD">10</span>,<span style="color: #B452CD">100</span>)
|
||||
y = np.sin(x)
|
||||
plt.plot(x,y,marker=<span style="color: #CD5555">'x'</span>)
|
||||
plt.show()
|
||||
data = {<span style="color: #CD5555">'Name'</span>: [<span style="color: #CD5555">"John"</span>, <span style="color: #CD5555">"Anna"</span>, <span style="color: #CD5555">"Peter"</span>, <span style="color: #CD5555">"Linda"</span>], <span style="color: #CD5555">'Location'</span>: [<span style="color: #CD5555">"Nairobi"</span>, <span style="color: #CD5555">"Napoli"</span>, <span style="color: #CD5555">"London"</span>, <span style="color: #CD5555">"Buenos Aires"</span>], <span style="color: #CD5555">'Age'</span>:[<span style="color: #B452CD">51</span>, <span style="color: #B452CD">21</span>, <span style="color: #B452CD">34</span>, <span style="color: #B452CD">45</span>]}
|
||||
data_pandas = pd.DataFrame(data)
|
||||
display(data_pandas)
|
||||
</pre></div>
|
||||
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
|
||||
@@ -151,7 +151,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>May 26, 2018</h4></center> <!-- date -->
|
||||
<center><h4>May 27, 2018</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -161,27 +161,27 @@ MathJax.Hub.Config({
|
||||
<p>
|
||||
Our emphasis throughout this series of lectures
|
||||
is on understanding the mathematical aspects of
|
||||
different algorithms used in the fields of data analysis and machine learning.
|
||||
different algorithms used in the fields of data analysis and machine learning.
|
||||
|
||||
<p>
|
||||
However, where possible we will emphasize the
|
||||
importance of using available software. We start thus with a hands-on
|
||||
and top-down approach to machine learning. The aim is thus to start with
|
||||
relevant data and use these to introduce statistical data analysis
|
||||
relevant data or data we have produced
|
||||
and use these to introduce statistical data analysis
|
||||
concepts and machine learning algorithms before we delve into the
|
||||
algorithms themselves. The examples we will use in the beginning, start with simple
|
||||
polynomials with random noise added, and using the Python
|
||||
software package <a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a> we
|
||||
will introduce various machine learning algorithms to make fits of
|
||||
polynomials with random noise added. We will use the Python
|
||||
software package <a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a> and
|
||||
introduce various machine learning algorithms to make fits of
|
||||
the data and predictions. We move thereafter to more interesting
|
||||
cases such as the simulation of financial transactions or disease
|
||||
models. These are examples where we can easily set up the data and
|
||||
then use machine learning algorithms included in for example
|
||||
<b>scikit-learn</b>. Another model we will consider is the so-called Ising
|
||||
model. Here we will use this model to produce data for selected spin
|
||||
configurations and attempt to classify the data. Finally, our last
|
||||
example consists of economic data from the OECD.
|
||||
<b>scikit-learn</b>.
|
||||
|
||||
<p>
|
||||
All these examples will serve us the purpose of getting us started, furthermore, they
|
||||
These examples will serve us the purpose of getting started. Furthermore, they
|
||||
allow us to catch more than two birds with a stone. They will allow us
|
||||
to bring in some programming specific topics and tools as well as
|
||||
showing the power of various Python (and R) packages for machine
|
||||
@@ -199,7 +199,7 @@ Moreover, the examples we introduce will serve as inputs to many of our discussi
|
||||
|
||||
<p>
|
||||
We will make extensive use of Python as programming language and its
|
||||
myriad of available libraries. Furthermore, you will find
|
||||
myriad of available libraries. You will find
|
||||
IPython/Jupyter notebooks invaluable in your work. You can run <b>R</b>
|
||||
codes in the Jupyter/IPython notebooks, with the immediate benefit of
|
||||
visualizing your data. You can also use compiled languages like C++,
|
||||
@@ -211,16 +211,23 @@ a Jupyter notebook.
|
||||
<p>
|
||||
If you have Python installed (we recommend Python3) and you feel
|
||||
pretty familiar with installing different packages, we recommend that
|
||||
you install the following Python packages via <b>pip</b> as o pip install
|
||||
numpy scipy matplotlib ipython scikit-learn mglearn sympy pandas
|
||||
pillow For Python3, replace <b>pip</b> with <b>pip3</b>.
|
||||
you install the following Python packages via <b>pip</b> as
|
||||
|
||||
<ol>
|
||||
<li> pip install numpy scipy matplotlib ipython scikit-learn mglearn sympy pandas pillow</li>
|
||||
</ol>
|
||||
|
||||
For Python3, replace <b>pip</b> with <b>pip3</b>.
|
||||
|
||||
<p>
|
||||
For OSX users we recommend also, after having installed Xcode, to
|
||||
For OSX users we recommend, after having installed Xcode, to
|
||||
install <b>brew</b>. Brew allows for a seamless installation of additional
|
||||
software via for example o brew install python3
|
||||
software via for example
|
||||
|
||||
<ol>
|
||||
<li> brew install python3</li>
|
||||
</ol>
|
||||
|
||||
<p>
|
||||
For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,
|
||||
you can use <b>pip</b> as well and simply install Python as
|
||||
|
||||
@@ -239,13 +246,23 @@ etc etc.
|
||||
If you don't want to perform these operations separately and venture
|
||||
into the hassle of exploring how to set up dependencies and paths, we
|
||||
recommend two widely used distrubutions which set up all relevant
|
||||
dependencies for Python, namely o
|
||||
<a href="https://docs.anaconda.com/" target="_blank">Anaconda</a>, which is an open source
|
||||
dependencies for Python, namely
|
||||
|
||||
<ol>
|
||||
<li> <a href="https://docs.anaconda.com/" target="_blank">Anaconda</a>,</li>
|
||||
</ol>
|
||||
|
||||
which is an open source
|
||||
distribution of the Python and R programming languages for large-scale
|
||||
data processing, predictive analytics, and scientific computing, that
|
||||
aims to simplify package management and deployment. Package versions
|
||||
are managed by the package management system <b>conda</b> o <a href="https://www.enthought.com/product/canopy/" target="_blank">Enthought
|
||||
canopy</a> is a Python
|
||||
are managed by the package management system <b>conda</b>.
|
||||
|
||||
<ol>
|
||||
<li> <a href="https://www.enthought.com/product/canopy/" target="_blank">Enthought canopy</a></li>
|
||||
</ol>
|
||||
|
||||
is a Python
|
||||
distribution for scientific and analytic computing distribution and
|
||||
analysis environment, available for free and under a commercial
|
||||
license.
|
||||
@@ -268,8 +285,8 @@ and allows for an easy usage of the tools we will discuss in these
|
||||
texts.
|
||||
|
||||
<p>
|
||||
To install <b>R</b> with Jupyter notebook <a href="https://mpacer.org/maths/r-kernel-for-ipython-notebook" target="_blank">following the link
|
||||
here</a>
|
||||
To install <b>R</b> with Jupyter notebook
|
||||
<a href="https://mpacer.org/maths/r-kernel-for-ipython-notebook" target="_blank">follow the link here</a>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -293,23 +310,27 @@ capabilities with minimal rewrites of your codes. With its
|
||||
versatility, including symbolic operations, Python offers a unique
|
||||
computational environment. Your Jupyter/IPython notebook can easily be
|
||||
converted into a nicely rendered <b>PDF</b> file or a Latex file for
|
||||
further processing. For example, convert to latex as
|
||||
further processing. For example, convert to latex as
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=text typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>pycod jupyter nbconvert filename.ipynb --to latex
|
||||
</pre></div>
|
||||
<p>
|
||||
And to add more versatility, symbolic Python package <a href="http://www.sympy.org/en/index.html" target="_blank">SymPy</a> is Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) and is entirely written in Python.
|
||||
And to add more versatility, the Python package <a href="http://www.sympy.org/en/index.html" target="_blank">SymPy</a> is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) and is entirely written in Python.
|
||||
|
||||
<p>
|
||||
Finally, if you wish to use the light mark-up language <a href="https://github.com/hplgit/doconce" target="_blank">doconce</a> you can convert a standard ascii text file into various HTML
|
||||
Finally, if you wish to use the light mark-up language
|
||||
<a href="https://github.com/hplgit/doconce" target="_blank">doconce</a> you can convert a standard ascii text file into various HTML
|
||||
formats, ipython notebooks, latex files, pdf files etc with minimal edits.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec5">Simple linear regression model using <b>scikit-learn</b> </h2>
|
||||
|
||||
<p>
|
||||
We start with perhaps our simplest possible example, using <b>scikit-learn</b> to perform linear regression analysis on a data set produced by us.
|
||||
What follows is a simple Python code where we have defined function \( y \) in terms of the variable \( x \). Both are defined as vectors of dimension \( 1\times 100 \). The entries to the vector \( \hat{x} \) are given by random numbers generated with a uniform distribution with entries \( x_i \in [0,1] \) (more about probability distribution functions later). These values are then used to define a function \( y(x) \) (tabulated again as a vector) with a linear dependence on \( x \) plus a random noise added via the normal distribution.
|
||||
|
||||
@@ -348,7 +369,7 @@ prediction <b>ypredict</b> (\( \tilde{y} \)), which attempts at fitting our
|
||||
data with a straight line.
|
||||
|
||||
<p>
|
||||
The Python follows here.
|
||||
The Python code follows here.
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
@@ -425,11 +446,12 @@ employed are various variants of <b>gradient</b> methods. These will be
|
||||
discussed in more detail later. Again, you'll be surprised to hear that
|
||||
many practitioners minimize the above function ''by the eye', popularly dubbed as
|
||||
'chi by the eye'. That is, change a parameter and see (visually and numerically) that
|
||||
the your \( \chi^2 \) function becomes smaller.
|
||||
the \( \chi^2 \) function becomes smaller.
|
||||
|
||||
<p>
|
||||
There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define
|
||||
the relative error as
|
||||
|
||||
$$
|
||||
\epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}.
|
||||
$$
|
||||
@@ -456,75 +478,93 @@ plt.title(<span style="color: #CD5555">r'Relative error'</span>)
|
||||
plt.show()
|
||||
</pre></div>
|
||||
<p>
|
||||
Depending on the parameter in front of the normal distribution, we may have a small or larger relative error. Try to play around with different training data sets and study (graphically) the value of the relative error.
|
||||
Depending on the parameter in front of the normal distribution, we may
|
||||
have a small or larger relative error. Try to play around with
|
||||
different training data sets and study (graphically) the value of the
|
||||
relative error.
|
||||
|
||||
<p>
|
||||
As mentioned above, <b>scikit-learn</b> has an impressive functionality.
|
||||
We can for example extract the values of \( \alpha \) and \( \beta \) and their error estimates,
|
||||
or the variance and standard deviation and many other properties from the statistical data analysis. Here we show an example of the functionality of scikit-learn.
|
||||
As mentioned above, <b>scikit-learn</b> has an impressive functionality.
|
||||
We can for example extract the values of \( \alpha \) and \( \beta \) and
|
||||
their error estimates, or the variance and standard deviation and many
|
||||
other properties from the statistical data analysis. Here we show an
|
||||
example of the functionality of scikit-learn.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.metrics</span> <span style="color: #8B008B; font-weight: bold">import</span> mean_squared_error, r2_score
|
||||
|
||||
x = np.random.rand(<span style="color: #B452CD">100</span>,<span style="color: #B452CD">1</span>)
|
||||
y = <span style="color: #B452CD">2</span>*x+np.random.randn(<span style="color: #B452CD">100</span>,<span style="color: #B452CD">1</span>)
|
||||
y = <span style="color: #B452CD">2.0</span>+ <span style="color: #B452CD">5</span>*x+<span style="color: #B452CD">0.5</span>np.random.randn(<span style="color: #B452CD">100</span>,<span style="color: #B452CD">1</span>)
|
||||
linreg = LinearRegression()
|
||||
linreg.fit(x,y)
|
||||
ypredict = linreg.predict(x)
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">'Coefficients: \n'</span>, linreg.coef_)
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">'The intercept alpha: \n'</span>, linreg.intercept_)
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">'Coefficient beta : \n'</span>, linreg.coef_)
|
||||
<span style="color: #228B22"># The mean squared error </span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">"Mean squared error: %.2f"</span> % mean_squared_error(y, ypredict))
|
||||
<span style="color: #228B22"># Explained variance score: 1 is perfect prediction </span>
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">'Variance score: %.2f'</span> % r2_score(y, ypredict))
|
||||
plt.plot(x, ypredict, <span style="color: #CD5555">"r-"</span>)
|
||||
plt.plot(x, y ,<span style="color: #CD5555">'ro'</span>)
|
||||
plt.axis([<span style="color: #B452CD">0</span>,<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">0</span>, <span style="color: #B452CD">5.0</span>])
|
||||
plt.axis([<span style="color: #B452CD">0.0</span>,<span style="color: #B452CD">1.0</span>,<span style="color: #B452CD">1.5</span>, <span style="color: #B452CD">7.0</span>])
|
||||
plt.xlabel(<span style="color: #CD5555">r'$x$'</span>)
|
||||
plt.ylabel(<span style="color: #CD5555">r'$y$'</span>)
|
||||
plt.title(<span style="color: #CD5555">r'Linear Regression fit '</span>)
|
||||
plt.show()
|
||||
</pre></div>
|
||||
<p>
|
||||
We will come to the definition of these outputs later.
|
||||
The function <b>coef</b> gives us the parameter \( \beta \) of our fit while <b>intercept</b> yields
|
||||
\( \alpha \). Depending on the constant in front of the normal distribution, we get values near or far from \( alpha =2 \) and \( \beta =5 \). Try to play around with different parameters in front of the normal distribution. The function <b>meansquarederror</b> gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as
|
||||
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||||
$$
|
||||
|
||||
<p>
|
||||
The smaller the value, the better the fit. Ideally we would like to
|
||||
have an MSE equal zero. The attentive reader has probably recognized
|
||||
this function as being similar to the \( \chi^2 \) function defined above.
|
||||
|
||||
<p>
|
||||
The <b>r2score</b> function computes \( R^2 \), the coefficient of
|
||||
determination. It provides a measure of how well future samples are
|
||||
likely to be predicted by the model. Best possible score is 1.0 and it
|
||||
can be negative (because the model can be arbitrarily worse). A
|
||||
constant model that always predicts the expected value of \( \hat{y} \),
|
||||
disregarding the input features, would get a \( R^2 \) score of \( 0.0 \).
|
||||
|
||||
<p>
|
||||
If \( \tilde{\hat{y}}_i \) is the predicted value of the i-th sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
|
||||
$$
|
||||
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
$$
|
||||
|
||||
where the mean value
|
||||
$$
|
||||
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
||||
$$
|
||||
|
||||
We will discuss in more detail these and more function in the various lectures.
|
||||
|
||||
<p>
|
||||
Another useful Python package is
|
||||
<a href="https://pandas.pydata.org/" target="_blank">pandas</a>, which is an open source library
|
||||
providing high-performance, easy-to-use data structures and data
|
||||
analysis tools for Python. The following simple example shows an example on how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of <b>pandas</b>, in particular in connection with classification of data.
|
||||
analysis tools for Python. The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of <b>pandas</b>, in particular in connection with classification of data.
|
||||
|
||||
<p>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="highlight" style="background: #eee8d5"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">scipy</span> <span style="color: #8B008B; font-weight: bold">import</span> sparse
|
||||
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
|
||||
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
|
||||
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">IPython.display</span> <span style="color: #8B008B; font-weight: bold">import</span> display
|
||||
eye = np.eye(<span style="color: #B452CD">4</span>)
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(eye)
|
||||
sparse_mtx = sparse.csr_matrix(eye)
|
||||
<span style="color: #8B008B; font-weight: bold">print</span>(sparse_mtx)
|
||||
x = np.linspace(-<span style="color: #B452CD">10</span>,<span style="color: #B452CD">10</span>,<span style="color: #B452CD">100</span>)
|
||||
y = np.sin(x)
|
||||
plt.plot(x,y,marker=<span style="color: #CD5555">'x'</span>)
|
||||
plt.show()
|
||||
data = {<span style="color: #CD5555">'Name'</span>: [<span style="color: #CD5555">"John"</span>, <span style="color: #CD5555">"Anna"</span>, <span style="color: #CD5555">"Peter"</span>, <span style="color: #CD5555">"Linda"</span>], <span style="color: #CD5555">'Location'</span>: [<span style="color: #CD5555">"Nairobi"</span>, <span style="color: #CD5555">"Napoli"</span>, <span style="color: #CD5555">"London"</span>, <span style="color: #CD5555">"Buenos Aires"</span>], <span style="color: #CD5555">'Age'</span>:[<span style="color: #B452CD">51</span>, <span style="color: #B452CD">21</span>, <span style="color: #B452CD">34</span>, <span style="color: #B452CD">45</span>]}
|
||||
data_pandas = pd.DataFrame(data)
|
||||
display(data_pandas)
|
||||
</pre></div>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
|
||||
@@ -156,7 +156,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>May 26, 2018</h4></center> <!-- date -->
|
||||
<center><h4>May 27, 2018</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -166,27 +166,27 @@ MathJax.Hub.Config({
|
||||
<p>
|
||||
Our emphasis throughout this series of lectures
|
||||
is on understanding the mathematical aspects of
|
||||
different algorithms used in the fields of data analysis and machine learning.
|
||||
different algorithms used in the fields of data analysis and machine learning.
|
||||
|
||||
<p>
|
||||
However, where possible we will emphasize the
|
||||
importance of using available software. We start thus with a hands-on
|
||||
and top-down approach to machine learning. The aim is thus to start with
|
||||
relevant data and use these to introduce statistical data analysis
|
||||
relevant data or data we have produced
|
||||
and use these to introduce statistical data analysis
|
||||
concepts and machine learning algorithms before we delve into the
|
||||
algorithms themselves. The examples we will use in the beginning, start with simple
|
||||
polynomials with random noise added, and using the Python
|
||||
software package <a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a> we
|
||||
will introduce various machine learning algorithms to make fits of
|
||||
polynomials with random noise added. We will use the Python
|
||||
software package <a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a> and
|
||||
introduce various machine learning algorithms to make fits of
|
||||
the data and predictions. We move thereafter to more interesting
|
||||
cases such as the simulation of financial transactions or disease
|
||||
models. These are examples where we can easily set up the data and
|
||||
then use machine learning algorithms included in for example
|
||||
<b>scikit-learn</b>. Another model we will consider is the so-called Ising
|
||||
model. Here we will use this model to produce data for selected spin
|
||||
configurations and attempt to classify the data. Finally, our last
|
||||
example consists of economic data from the OECD.
|
||||
<b>scikit-learn</b>.
|
||||
|
||||
<p>
|
||||
All these examples will serve us the purpose of getting us started, furthermore, they
|
||||
These examples will serve us the purpose of getting started. Furthermore, they
|
||||
allow us to catch more than two birds with a stone. They will allow us
|
||||
to bring in some programming specific topics and tools as well as
|
||||
showing the power of various Python (and R) packages for machine
|
||||
@@ -204,7 +204,7 @@ Moreover, the examples we introduce will serve as inputs to many of our discussi
|
||||
|
||||
<p>
|
||||
We will make extensive use of Python as programming language and its
|
||||
myriad of available libraries. Furthermore, you will find
|
||||
myriad of available libraries. You will find
|
||||
IPython/Jupyter notebooks invaluable in your work. You can run <b>R</b>
|
||||
codes in the Jupyter/IPython notebooks, with the immediate benefit of
|
||||
visualizing your data. You can also use compiled languages like C++,
|
||||
@@ -216,16 +216,23 @@ a Jupyter notebook.
|
||||
<p>
|
||||
If you have Python installed (we recommend Python3) and you feel
|
||||
pretty familiar with installing different packages, we recommend that
|
||||
you install the following Python packages via <b>pip</b> as o pip install
|
||||
numpy scipy matplotlib ipython scikit-learn mglearn sympy pandas
|
||||
pillow For Python3, replace <b>pip</b> with <b>pip3</b>.
|
||||
you install the following Python packages via <b>pip</b> as
|
||||
|
||||
<ol>
|
||||
<li> pip install numpy scipy matplotlib ipython scikit-learn mglearn sympy pandas pillow</li>
|
||||
</ol>
|
||||
|
||||
For Python3, replace <b>pip</b> with <b>pip3</b>.
|
||||
|
||||
<p>
|
||||
For OSX users we recommend also, after having installed Xcode, to
|
||||
For OSX users we recommend, after having installed Xcode, to
|
||||
install <b>brew</b>. Brew allows for a seamless installation of additional
|
||||
software via for example o brew install python3
|
||||
software via for example
|
||||
|
||||
<ol>
|
||||
<li> brew install python3</li>
|
||||
</ol>
|
||||
|
||||
<p>
|
||||
For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,
|
||||
you can use <b>pip</b> as well and simply install Python as
|
||||
|
||||
@@ -244,13 +251,23 @@ etc etc.
|
||||
If you don't want to perform these operations separately and venture
|
||||
into the hassle of exploring how to set up dependencies and paths, we
|
||||
recommend two widely used distrubutions which set up all relevant
|
||||
dependencies for Python, namely o
|
||||
<a href="https://docs.anaconda.com/" target="_blank">Anaconda</a>, which is an open source
|
||||
dependencies for Python, namely
|
||||
|
||||
<ol>
|
||||
<li> <a href="https://docs.anaconda.com/" target="_blank">Anaconda</a>,</li>
|
||||
</ol>
|
||||
|
||||
which is an open source
|
||||
distribution of the Python and R programming languages for large-scale
|
||||
data processing, predictive analytics, and scientific computing, that
|
||||
aims to simplify package management and deployment. Package versions
|
||||
are managed by the package management system <b>conda</b> o <a href="https://www.enthought.com/product/canopy/" target="_blank">Enthought
|
||||
canopy</a> is a Python
|
||||
are managed by the package management system <b>conda</b>.
|
||||
|
||||
<ol>
|
||||
<li> <a href="https://www.enthought.com/product/canopy/" target="_blank">Enthought canopy</a></li>
|
||||
</ol>
|
||||
|
||||
is a Python
|
||||
distribution for scientific and analytic computing distribution and
|
||||
analysis environment, available for free and under a commercial
|
||||
license.
|
||||
@@ -273,8 +290,8 @@ and allows for an easy usage of the tools we will discuss in these
|
||||
texts.
|
||||
|
||||
<p>
|
||||
To install <b>R</b> with Jupyter notebook <a href="https://mpacer.org/maths/r-kernel-for-ipython-notebook" target="_blank">following the link
|
||||
here</a>
|
||||
To install <b>R</b> with Jupyter notebook
|
||||
<a href="https://mpacer.org/maths/r-kernel-for-ipython-notebook" target="_blank">follow the link here</a>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -298,23 +315,27 @@ capabilities with minimal rewrites of your codes. With its
|
||||
versatility, including symbolic operations, Python offers a unique
|
||||
computational environment. Your Jupyter/IPython notebook can easily be
|
||||
converted into a nicely rendered <b>PDF</b> file or a Latex file for
|
||||
further processing. For example, convert to latex as
|
||||
further processing. For example, convert to latex as
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=text typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pycod jupyter nbconvert filename.ipynb --to latex
|
||||
</pre></div>
|
||||
<p>
|
||||
And to add more versatility, symbolic Python package <a href="http://www.sympy.org/en/index.html" target="_blank">SymPy</a> is Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) and is entirely written in Python.
|
||||
And to add more versatility, the Python package <a href="http://www.sympy.org/en/index.html" target="_blank">SymPy</a> is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) and is entirely written in Python.
|
||||
|
||||
<p>
|
||||
Finally, if you wish to use the light mark-up language <a href="https://github.com/hplgit/doconce" target="_blank">doconce</a> you can convert a standard ascii text file into various HTML
|
||||
Finally, if you wish to use the light mark-up language
|
||||
<a href="https://github.com/hplgit/doconce" target="_blank">doconce</a> you can convert a standard ascii text file into various HTML
|
||||
formats, ipython notebooks, latex files, pdf files etc with minimal edits.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec5">Simple linear regression model using <b>scikit-learn</b> </h2>
|
||||
|
||||
<p>
|
||||
We start with perhaps our simplest possible example, using <b>scikit-learn</b> to perform linear regression analysis on a data set produced by us.
|
||||
What follows is a simple Python code where we have defined function \( y \) in terms of the variable \( x \). Both are defined as vectors of dimension \( 1\times 100 \). The entries to the vector \( \hat{x} \) are given by random numbers generated with a uniform distribution with entries \( x_i \in [0,1] \) (more about probability distribution functions later). These values are then used to define a function \( y(x) \) (tabulated again as a vector) with a linear dependence on \( x \) plus a random noise added via the normal distribution.
|
||||
|
||||
@@ -353,7 +374,7 @@ prediction <b>ypredict</b> (\( \tilde{y} \)), which attempts at fitting our
|
||||
data with a straight line.
|
||||
|
||||
<p>
|
||||
The Python follows here.
|
||||
The Python code follows here.
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
@@ -430,11 +451,12 @@ employed are various variants of <b>gradient</b> methods. These will be
|
||||
discussed in more detail later. Again, you'll be surprised to hear that
|
||||
many practitioners minimize the above function ''by the eye', popularly dubbed as
|
||||
'chi by the eye'. That is, change a parameter and see (visually and numerically) that
|
||||
the your \( \chi^2 \) function becomes smaller.
|
||||
the \( \chi^2 \) function becomes smaller.
|
||||
|
||||
<p>
|
||||
There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define
|
||||
the relative error as
|
||||
|
||||
$$
|
||||
\epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}.
|
||||
$$
|
||||
@@ -461,75 +483,93 @@ plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">r'
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
</pre></div>
|
||||
<p>
|
||||
Depending on the parameter in front of the normal distribution, we may have a small or larger relative error. Try to play around with different training data sets and study (graphically) the value of the relative error.
|
||||
Depending on the parameter in front of the normal distribution, we may
|
||||
have a small or larger relative error. Try to play around with
|
||||
different training data sets and study (graphically) the value of the
|
||||
relative error.
|
||||
|
||||
<p>
|
||||
As mentioned above, <b>scikit-learn</b> has an impressive functionality.
|
||||
We can for example extract the values of \( \alpha \) and \( \beta \) and their error estimates,
|
||||
or the variance and standard deviation and many other properties from the statistical data analysis. Here we show an example of the functionality of scikit-learn.
|
||||
As mentioned above, <b>scikit-learn</b> has an impressive functionality.
|
||||
We can for example extract the values of \( \alpha \) and \( \beta \) and
|
||||
their error estimates, or the variance and standard deviation and many
|
||||
other properties from the statistical data analysis. Here we show an
|
||||
example of the functionality of scikit-learn.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LinearRegression
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LinearRegression
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> mean_squared_error, r2_score
|
||||
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
|
||||
y <span style="color: #666666">=</span> <span style="color: #666666">2*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
|
||||
y <span style="color: #666666">=</span> <span style="color: #666666">2.0+</span> <span style="color: #666666">5*</span>x<span style="color: #666666">+0.5</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">100</span>,<span style="color: #666666">1</span>)
|
||||
linreg <span style="color: #666666">=</span> LinearRegression()
|
||||
linreg<span style="color: #666666">.</span>fit(x,y)
|
||||
ypredict <span style="color: #666666">=</span> linreg<span style="color: #666666">.</span>predict(x)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">'Coefficients: </span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121">'</span>, linreg<span style="color: #666666">.</span>coef_)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">'The intercept alpha: </span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121">'</span>, linreg<span style="color: #666666">.</span>intercept_)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">'Coefficient beta : </span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121">'</span>, linreg<span style="color: #666666">.</span>coef_)
|
||||
<span style="color: #408080; font-style: italic"># The mean squared error </span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"Mean squared error: </span><span style="color: #BB6688; font-weight: bold">%.2f</span><span style="color: #BA2121">"</span> <span style="color: #666666">%</span> mean_squared_error(y, ypredict))
|
||||
<span style="color: #408080; font-style: italic"># Explained variance score: 1 is perfect prediction </span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">'Variance score: </span><span style="color: #BB6688; font-weight: bold">%.2f</span><span style="color: #BA2121">'</span> <span style="color: #666666">%</span> r2_score(y, ypredict))
|
||||
plt<span style="color: #666666">.</span>plot(x, ypredict, <span style="color: #BA2121">"r-"</span>)
|
||||
plt<span style="color: #666666">.</span>plot(x, y ,<span style="color: #BA2121">'ro'</span>)
|
||||
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">0</span>,<span style="color: #666666">1.0</span>,<span style="color: #666666">0</span>, <span style="color: #666666">5.0</span>])
|
||||
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">0.0</span>,<span style="color: #666666">1.0</span>,<span style="color: #666666">1.5</span>, <span style="color: #666666">7.0</span>])
|
||||
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">r'$x$'</span>)
|
||||
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">r'$y$'</span>)
|
||||
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">r'Linear Regression fit '</span>)
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
</pre></div>
|
||||
<p>
|
||||
We will come to the definition of these outputs later.
|
||||
The function <b>coef</b> gives us the parameter \( \beta \) of our fit while <b>intercept</b> yields
|
||||
\( \alpha \). Depending on the constant in front of the normal distribution, we get values near or far from \( alpha =2 \) and \( \beta =5 \). Try to play around with different parameters in front of the normal distribution. The function <b>meansquarederror</b> gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as
|
||||
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||||
$$
|
||||
|
||||
<p>
|
||||
The smaller the value, the better the fit. Ideally we would like to
|
||||
have an MSE equal zero. The attentive reader has probably recognized
|
||||
this function as being similar to the \( \chi^2 \) function defined above.
|
||||
|
||||
<p>
|
||||
The <b>r2score</b> function computes \( R^2 \), the coefficient of
|
||||
determination. It provides a measure of how well future samples are
|
||||
likely to be predicted by the model. Best possible score is 1.0 and it
|
||||
can be negative (because the model can be arbitrarily worse). A
|
||||
constant model that always predicts the expected value of \( \hat{y} \),
|
||||
disregarding the input features, would get a \( R^2 \) score of \( 0.0 \).
|
||||
|
||||
<p>
|
||||
If \( \tilde{\hat{y}}_i \) is the predicted value of the i-th sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
|
||||
$$
|
||||
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
$$
|
||||
|
||||
where the mean value
|
||||
$$
|
||||
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
||||
$$
|
||||
|
||||
We will discuss in more detail these and more function in the various lectures.
|
||||
|
||||
<p>
|
||||
Another useful Python package is
|
||||
<a href="https://pandas.pydata.org/" target="_blank">pandas</a>, which is an open source library
|
||||
providing high-performance, easy-to-use data structures and data
|
||||
analysis tools for Python. The following simple example shows an example on how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of <b>pandas</b>, in particular in connection with classification of data.
|
||||
analysis tools for Python. The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of <b>pandas</b>, in particular in connection with classification of data.
|
||||
|
||||
<p>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">scipy</span> <span style="color: #008000; font-weight: bold">import</span> sparse
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> display
|
||||
eye <span style="color: #666666">=</span> np<span style="color: #666666">.</span>eye(<span style="color: #666666">4</span>)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(eye)
|
||||
sparse_mtx <span style="color: #666666">=</span> sparse<span style="color: #666666">.</span>csr_matrix(eye)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(sparse_mtx)
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-10</span>,<span style="color: #666666">10</span>,<span style="color: #666666">100</span>)
|
||||
y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sin(x)
|
||||
plt<span style="color: #666666">.</span>plot(x,y,marker<span style="color: #666666">=</span><span style="color: #BA2121">'x'</span>)
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
data <span style="color: #666666">=</span> {<span style="color: #BA2121">'Name'</span>: [<span style="color: #BA2121">"John"</span>, <span style="color: #BA2121">"Anna"</span>, <span style="color: #BA2121">"Peter"</span>, <span style="color: #BA2121">"Linda"</span>], <span style="color: #BA2121">'Location'</span>: [<span style="color: #BA2121">"Nairobi"</span>, <span style="color: #BA2121">"Napoli"</span>, <span style="color: #BA2121">"London"</span>, <span style="color: #BA2121">"Buenos Aires"</span>], <span style="color: #BA2121">'Age'</span>:[<span style="color: #666666">51</span>, <span style="color: #666666">21</span>, <span style="color: #666666">34</span>, <span style="color: #666666">45</span>]}
|
||||
data_pandas <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(data)
|
||||
display(data_pandas)
|
||||
</pre></div>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
"<!-- Author: --> \n",
|
||||
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
||||
"\n",
|
||||
"Date: **May 26, 2018**\n",
|
||||
"Date: **May 27, 2018**\n",
|
||||
"\n",
|
||||
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
|
||||
"\n",
|
||||
@@ -24,25 +24,24 @@
|
||||
"Our emphasis throughout this series of lectures \n",
|
||||
"is on understanding the mathematical aspects of\n",
|
||||
"different algorithms used in the fields of data analysis and machine learning. \n",
|
||||
"\n",
|
||||
"However, where possible we will emphasize the\n",
|
||||
"importance of using available software. We start thus with a hands-on\n",
|
||||
"and top-down approach to machine learning. The aim is thus to start with\n",
|
||||
"relevant data and use these to introduce statistical data analysis\n",
|
||||
"relevant data or data we have produced \n",
|
||||
"and use these to introduce statistical data analysis\n",
|
||||
"concepts and machine learning algorithms before we delve into the\n",
|
||||
"algorithms themselves. The examples we will use in the beginning, start with simple\n",
|
||||
"polynomials with random noise added, and using the Python\n",
|
||||
"software package [Scikit-learn](http://scikit-learn.org/stable/) we\n",
|
||||
"will introduce various machine learning algorithms to make fits of\n",
|
||||
"polynomials with random noise added. We will use the Python\n",
|
||||
"software package [Scikit-learn](http://scikit-learn.org/stable/) and\n",
|
||||
"introduce various machine learning algorithms to make fits of\n",
|
||||
"the data and predictions. We move thereafter to more interesting\n",
|
||||
"cases such as the simulation of financial transactions or disease\n",
|
||||
"models. These are examples where we can easily set up the data and\n",
|
||||
"then use machine learning algorithms included in for example\n",
|
||||
"**scikit-learn**. Another model we will consider is the so-called Ising\n",
|
||||
"model. Here we will use this model to produce data for selected spin\n",
|
||||
"configurations and attempt to classify the data. Finally, our last\n",
|
||||
"example consists of economic data from the OECD. \n",
|
||||
"**scikit-learn**. \n",
|
||||
"\n",
|
||||
"All these examples will serve us the purpose of getting us started, furthermore, they\n",
|
||||
"These examples will serve us the purpose of getting started. Furthermore, they\n",
|
||||
"allow us to catch more than two birds with a stone. They will allow us\n",
|
||||
"to bring in some programming specific topics and tools as well as\n",
|
||||
"showing the power of various Python (and R) packages for machine\n",
|
||||
@@ -59,7 +58,7 @@
|
||||
"## Software and needed installations\n",
|
||||
"\n",
|
||||
"We will make extensive use of Python as programming language and its\n",
|
||||
"myriad of available libraries. Furthermore, you will find\n",
|
||||
"myriad of available libraries. You will find\n",
|
||||
"IPython/Jupyter notebooks invaluable in your work. You can run **R**\n",
|
||||
"codes in the Jupyter/IPython notebooks, with the immediate benefit of\n",
|
||||
"visualizing your data. You can also use compiled languages like C++,\n",
|
||||
@@ -71,16 +70,21 @@
|
||||
"\n",
|
||||
"If you have Python installed (we recommend Python3) and you feel\n",
|
||||
"pretty familiar with installing different packages, we recommend that\n",
|
||||
"you install the following Python packages via **pip** as o pip install\n",
|
||||
"numpy scipy matplotlib ipython scikit-learn mglearn sympy pandas\n",
|
||||
"pillow For Python3, replace **pip** with **pip3**.\n",
|
||||
"you install the following Python packages via **pip** as \n",
|
||||
"\n",
|
||||
"For OSX users we recommend also, after having installed Xcode, to\n",
|
||||
"1. pip install numpy scipy matplotlib ipython scikit-learn mglearn sympy pandas pillow \n",
|
||||
"\n",
|
||||
"For Python3, replace **pip** with **pip3**.\n",
|
||||
"\n",
|
||||
"For OSX users we recommend, after having installed Xcode, to\n",
|
||||
"install **brew**. Brew allows for a seamless installation of additional\n",
|
||||
"software via for example o brew install python3\n",
|
||||
"software via for example \n",
|
||||
"\n",
|
||||
"1. brew install python3\n",
|
||||
"\n",
|
||||
"For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,\n",
|
||||
"you can use **pip** as well and simply install Python as \n",
|
||||
"\n",
|
||||
"1. sudo apt-get install python3 (or python for pyhton2.7)\n",
|
||||
"\n",
|
||||
"etc etc. \n",
|
||||
@@ -90,13 +94,19 @@
|
||||
"If you don't want to perform these operations separately and venture\n",
|
||||
"into the hassle of exploring how to set up dependencies and paths, we\n",
|
||||
"recommend two widely used distrubutions which set up all relevant\n",
|
||||
"dependencies for Python, namely o\n",
|
||||
"[Anaconda](https://docs.anaconda.com/), which is an open source\n",
|
||||
"dependencies for Python, namely \n",
|
||||
"\n",
|
||||
"1. [Anaconda](https://docs.anaconda.com/), \n",
|
||||
"\n",
|
||||
"which is an open source\n",
|
||||
"distribution of the Python and R programming languages for large-scale\n",
|
||||
"data processing, predictive analytics, and scientific computing, that\n",
|
||||
"aims to simplify package management and deployment. Package versions\n",
|
||||
"are managed by the package management system **conda** o [Enthought\n",
|
||||
"canopy](https://www.enthought.com/product/canopy/) is a Python\n",
|
||||
"are managed by the package management system **conda**. \n",
|
||||
"\n",
|
||||
"1. [Enthought canopy](https://www.enthought.com/product/canopy/) \n",
|
||||
"\n",
|
||||
"is a Python\n",
|
||||
"distribution for scientific and analytic computing distribution and\n",
|
||||
"analysis environment, available for free and under a commercial\n",
|
||||
"license.\n",
|
||||
@@ -115,8 +125,8 @@
|
||||
"and allows for an easy usage of the tools we will discuss in these\n",
|
||||
"texts.\n",
|
||||
"\n",
|
||||
"To install **R** with Jupyter notebook [following the link\n",
|
||||
"here](https://mpacer.org/maths/r-kernel-for-ipython-notebook)\n",
|
||||
"To install **R** with Jupyter notebook \n",
|
||||
"[follow the link here](https://mpacer.org/maths/r-kernel-for-ipython-notebook)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -152,12 +162,14 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"And to add more versatility, symbolic Python package [SymPy](http://www.sympy.org/en/index.html) is Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) and is entirely written in Python. \n",
|
||||
"And to add more versatility, the Python package [SymPy](http://www.sympy.org/en/index.html) is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) and is entirely written in Python. \n",
|
||||
"\n",
|
||||
"Finally, if you wish to use the light mark-up language [doconce](https://github.com/hplgit/doconce) you can convert a standard ascii text file into various HTML \n",
|
||||
"Finally, if you wish to use the light mark-up language \n",
|
||||
"[doconce](https://github.com/hplgit/doconce) you can convert a standard ascii text file into various HTML \n",
|
||||
"formats, ipython notebooks, latex files, pdf files etc with minimal edits.\n",
|
||||
"\n",
|
||||
"## Simple linear regression model using **scikit-learn**\n",
|
||||
"\n",
|
||||
"We start with perhaps our simplest possible example, using **scikit-learn** to perform linear regression analysis on a data set produced by us. \n",
|
||||
"What follows is a simple Python code where we have defined function $y$ in terms of the variable $x$. Both are defined as vectors of dimension $1\\times 100$. The entries to the vector $\\hat{x}$ are given by random numbers generated with a uniform distribution with entries $x_i \\in [0,1]$ (more about probability distribution functions later). These values are then used to define a function $y(x)$ (tabulated again as a vector) with a linear dependence on $x$ plus a random noise added via the normal distribution.\n",
|
||||
"\n",
|
||||
@@ -203,7 +215,7 @@
|
||||
"prediction **ypredict** ($\\tilde{y}$), which attempts at fitting our\n",
|
||||
"data with a straight line.\n",
|
||||
"\n",
|
||||
"The Python follows here."
|
||||
"The Python code follows here."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -310,7 +322,7 @@
|
||||
"discussed in more detail later. Again, you'll be surprised to hear that\n",
|
||||
"many practitioners minimize the above function ''by the eye', popularly dubbed as \n",
|
||||
"'chi by the eye'. That is, change a parameter and see (visually and numerically) that \n",
|
||||
"the your $\\chi^2$ function becomes smaller. \n",
|
||||
"the $\\chi^2$ function becomes smaller. \n",
|
||||
"\n",
|
||||
"There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define \n",
|
||||
"the relative error as"
|
||||
@@ -362,11 +374,16 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Depending on the parameter in front of the normal distribution, we may have a small or larger relative error. Try to play around with different training data sets and study (graphically) the value of the relative error. \n",
|
||||
"Depending on the parameter in front of the normal distribution, we may\n",
|
||||
"have a small or larger relative error. Try to play around with\n",
|
||||
"different training data sets and study (graphically) the value of the\n",
|
||||
"relative error.\n",
|
||||
"\n",
|
||||
"As mentioned above, **scikit-learn** has an impressive functionality. \n",
|
||||
"We can for example extract the values of $\\alpha$ and $\\beta$ and their error estimates,\n",
|
||||
"or the variance and standard deviation and many other properties from the statistical data analysis. Here we show an example of the functionality of scikit-learn."
|
||||
"As mentioned above, **scikit-learn** has an impressive functionality.\n",
|
||||
"We can for example extract the values of $\\alpha$ and $\\beta$ and\n",
|
||||
"their error estimates, or the variance and standard deviation and many\n",
|
||||
"other properties from the statistical data analysis. Here we show an\n",
|
||||
"example of the functionality of scikit-learn."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -377,24 +394,25 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from sklearn.linear_model import LinearRegression\n",
|
||||
"import numpy as np \n",
|
||||
"import matplotlib.pyplot as plt \n",
|
||||
"from sklearn.linear_model import LinearRegression \n",
|
||||
"from sklearn.metrics import mean_squared_error, r2_score\n",
|
||||
"\n",
|
||||
"x = np.random.rand(100,1)\n",
|
||||
"y = 2*x+np.random.randn(100,1)\n",
|
||||
"y = 2.0+ 5*x+0.5np.random.randn(100,1)\n",
|
||||
"linreg = LinearRegression()\n",
|
||||
"linreg.fit(x,y)\n",
|
||||
"ypredict = linreg.predict(x)\n",
|
||||
"print('Coefficients: \\n', linreg.coef_)\n",
|
||||
"print('The intercept alpha: \\n', linreg.intercept_)\n",
|
||||
"print('Coefficient beta : \\n', linreg.coef_)\n",
|
||||
"# The mean squared error \n",
|
||||
"print(\"Mean squared error: %.2f\" % mean_squared_error(y, ypredict))\n",
|
||||
"# Explained variance score: 1 is perfect prediction \n",
|
||||
"print('Variance score: %.2f' % r2_score(y, ypredict))\n",
|
||||
"plt.plot(x, ypredict, \"r-\")\n",
|
||||
"plt.plot(x, y ,'ro')\n",
|
||||
"plt.axis([0,1.0,0, 5.0])\n",
|
||||
"plt.axis([0.0,1.0,1.5, 7.0])\n",
|
||||
"plt.xlabel(r'$x$')\n",
|
||||
"plt.ylabel(r'$y$')\n",
|
||||
"plt.title(r'Linear Regression fit ')\n",
|
||||
@@ -405,12 +423,73 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We will come to the definition of these outputs later. \n",
|
||||
"The function **coef** gives us the parameter $\\beta$ of our fit while **intercept** yields \n",
|
||||
"$\\alpha$. Depending on the constant in front of the normal distribution, we get values near or far from $alpha =2$ and $\\beta =5$. Try to play around with different parameters in front of the normal distribution. The function **meansquarederror** gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n",
|
||||
"\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The smaller the value, the better the fit. Ideally we would like to\n",
|
||||
"have an MSE equal zero. The attentive reader has probably recognized\n",
|
||||
"this function as being similar to the $\\chi^2$ function defined above.\n",
|
||||
"\n",
|
||||
"The **r2score** function computes $R^2$, the coefficient of\n",
|
||||
"determination. It provides a measure of how well future samples are\n",
|
||||
"likely to be predicted by the model. Best possible score is 1.0 and it\n",
|
||||
"can be negative (because the model can be arbitrarily worse). A\n",
|
||||
"constant model that always predicts the expected value of $\\hat{y}$,\n",
|
||||
"disregarding the input features, would get a $R^2$ score of $0.0$.\n",
|
||||
"\n",
|
||||
"If $\\tilde{\\hat{y}}_i$ is the predicted value of the i-th sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"where the mean value"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\bar{y} = \\frac{1}{n} \\sum_{i=0}^{n - 1} y_i.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We will discuss in more detail these and more function in the various lectures.\n",
|
||||
"\n",
|
||||
"Another useful Python package is\n",
|
||||
"[pandas](https://pandas.pydata.org/), which is an open source library\n",
|
||||
"providing high-performance, easy-to-use data structures and data\n",
|
||||
"analysis tools for Python. The following simple example shows an example on how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of **pandas**, in particular in connection with classification of data."
|
||||
"analysis tools for Python. The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of **pandas**, in particular in connection with classification of data."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -421,19 +500,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from scipy import sparse\n",
|
||||
"import pandas as pd\n",
|
||||
"from IPython.display import display\n",
|
||||
"eye = np.eye(4)\n",
|
||||
"print(eye)\n",
|
||||
"sparse_mtx = sparse.csr_matrix(eye)\n",
|
||||
"print(sparse_mtx)\n",
|
||||
"x = np.linspace(-10,10,100)\n",
|
||||
"y = np.sin(x)\n",
|
||||
"plt.plot(x,y,marker='x')\n",
|
||||
"plt.show()\n",
|
||||
"data = {'Name': [\"John\", \"Anna\", \"Peter\", \"Linda\"], 'Location': [\"Nairobi\", \"Napoli\", \"London\", \"Buenos Aires\"], 'Age':[51, 21, 34, 45]}\n",
|
||||
"data_pandas = pd.DataFrame(data)\n",
|
||||
"display(data_pandas)"
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -93,7 +93,7 @@ data processing, predictive analytics, and scientific computing, that
|
||||
aims to simplify package management and deployment. Package versions
|
||||
are managed by the package management system _conda_.
|
||||
|
||||
o "Enthoughtmcanopy":"https://www.enthought.com/product/canopy/"
|
||||
o "Enthought canopy":"https://www.enthought.com/product/canopy/"
|
||||
|
||||
is a Python
|
||||
distribution for scientific and analytic computing distribution and
|
||||
@@ -296,37 +296,74 @@ plt.title(r'Relative error')
|
||||
plt.show()
|
||||
!ec
|
||||
|
||||
Depending on the parameter in front of the normal distribution, we may have a small or larger relative error. Try to play around with different training data sets and study (graphically) the value of the relative error.
|
||||
Depending on the parameter in front of the normal distribution, we may
|
||||
have a small or larger relative error. Try to play around with
|
||||
different training data sets and study (graphically) the value of the
|
||||
relative error.
|
||||
|
||||
As mentioned above, _scikit-learn_ has an impressive functionality.
|
||||
We can for example extract the values of $\alpha$ and $\beta$ and their error estimates,
|
||||
or the variance and standard deviation and many other properties from the statistical data analysis. Here we show an example of the functionality of scikit-learn.
|
||||
!bc pycod
|
||||
import numpy as np
|
||||
import matplotlib.pyplot as plt
|
||||
from sklearn.linear_model import LinearRegression
|
||||
As mentioned above, _scikit-learn_ has an impressive functionality.
|
||||
We can for example extract the values of $\alpha$ and $\beta$ and
|
||||
their error estimates, or the variance and standard deviation and many
|
||||
other properties from the statistical data analysis. Here we show an
|
||||
example of the functionality of scikit-learn.
|
||||
!bc pycod
|
||||
import numpy as np
|
||||
import matplotlib.pyplot as plt
|
||||
from sklearn.linear_model import LinearRegression
|
||||
from sklearn.metrics import mean_squared_error, r2_score
|
||||
|
||||
x = np.random.rand(100,1)
|
||||
y = 2*x+np.random.randn(100,1)
|
||||
y = 2.0+ 5*x+0.5np.random.randn(100,1)
|
||||
linreg = LinearRegression()
|
||||
linreg.fit(x,y)
|
||||
ypredict = linreg.predict(x)
|
||||
print('Coefficients: \n', linreg.coef_)
|
||||
print('The intercept alpha: \n', linreg.intercept_)
|
||||
print('Coefficient beta : \n', linreg.coef_)
|
||||
# The mean squared error
|
||||
print("Mean squared error: %.2f" % mean_squared_error(y, ypredict))
|
||||
# Explained variance score: 1 is perfect prediction
|
||||
print('Variance score: %.2f' % r2_score(y, ypredict))
|
||||
plt.plot(x, ypredict, "r-")
|
||||
plt.plot(x, y ,'ro')
|
||||
plt.axis([0,1.0,0, 5.0])
|
||||
plt.axis([0.0,1.0,1.5, 7.0])
|
||||
plt.xlabel(r'$x$')
|
||||
plt.ylabel(r'$y$')
|
||||
plt.title(r'Linear Regression fit ')
|
||||
plt.show()
|
||||
|
||||
!ec
|
||||
We will come to the definition of these outputs later.
|
||||
The function _coef_ gives us the parameter $\beta$ of our fit while _intercept_ yields
|
||||
$\alpha$. Depending on the constant in front of the normal distribution, we get values near or far from $alpha =2$ and $\beta =5$. Try to play around with different parameters in front of the normal distribution. The function _meansquarederror_ gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as
|
||||
!bt
|
||||
\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||||
\]
|
||||
!et
|
||||
|
||||
The smaller the value, the better the fit. Ideally we would like to
|
||||
have an MSE equal zero. The attentive reader has probably recognized
|
||||
this function as being similar to the $\chi^2$ function defined above.
|
||||
|
||||
The _r2score_ function computes $R^2$, the coefficient of
|
||||
determination. It provides a measure of how well future samples are
|
||||
likely to be predicted by the model. Best possible score is 1.0 and it
|
||||
can be negative (because the model can be arbitrarily worse). A
|
||||
constant model that always predicts the expected value of $\hat{y}$,
|
||||
disregarding the input features, would get a $R^2$ score of $0.0$.
|
||||
|
||||
If $\tilde{\hat{y}}_i$ is the predicted value of the i-th sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
|
||||
!bt
|
||||
\[
|
||||
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
\]
|
||||
!et
|
||||
where the mean value
|
||||
!bt
|
||||
\[
|
||||
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
||||
\]
|
||||
!et
|
||||
We will discuss in more detail these and more function in the various lectures.
|
||||
|
||||
Another useful Python package is
|
||||
"pandas":"https://pandas.pydata.org/", which is an open source library
|
||||
|
||||
Reference in New Issue
Block a user