lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

revised intro slides

Browse Source
This commit is contained in:
mhjensen
2020-08-20 05:47:13 +02:00
parent 2722adf924
commit d6eabe1d2d
7 changed files with 1086 additions and 322 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/Introduction/html/Introduction-bs.html
+247 -67
View File
@@ -43,16 +43,49 @@ Automatically generated HTML file from DocOnce source
{'highest level': 2,
'sections': [('Introduction', 2, None, '___sec0'),
('Learning outcomes', 2, None, '___sec1'),
('Types of Machine Learning', 2, None, '___sec2'),
('Choice of programming language', 2, None, '___sec3'),
('Machine Learning, short overview', 2, None, '___sec2'),
('Machine Learning, a small (and probably biased) introduction',
2,
None,
'___sec3'),
('Machine Learning, an extremely rich field', 2, None, '___sec4'),
('A multidisciplinary approach', 2, None, '___sec5'),
('Types of Machine Learning', 2, None, '___sec6'),
('Essential elements of ML', 2, None, '___sec7'),
('An optimization/minimization problem', 2, None, '___sec8'),
('A Frequentist approach to data analysis', 2, None, '___sec9'),
('What is a good model?', 2, None, '___sec10'),
('What is a good model? Can we define it?', 2, None, '___sec11'),
('Practicalities, choice of programming language and other '
'computational issues',
2,
None,
'___sec12'),
('Choice of Programming Language', 2, None, '___sec13'),
('Data handling, machine learning and ethical aspects',
2,
None,
'___sec4')]}
'___sec14')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
@@ -72,9 +105,19 @@ end of tocinfo -->
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Introduction</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Learning outcomes</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Types of Machine Learning</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Choice of programming language</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Data handling, machine learning and ethical aspects</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Machine Learning, short overview</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Machine Learning, a small (and probably biased) introduction</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Machine Learning, an extremely rich field</a></li>
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">A multidisciplinary approach</a></li>
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">Types of Machine Learning</a></li>
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;">Essential elements of ML</a></li>
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;">An optimization/minimization problem</a></li>
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;">A Frequentist approach to data analysis</a></li>
<!-- navigation toc: --> <li><a href="#___sec10" style="font-size: 80%;">What is a good model?</a></li>
<!-- navigation toc: --> <li><a href="#___sec11" style="font-size: 80%;">What is a good model? Can we define it?</a></li>
<!-- navigation toc: --> <li><a href="#___sec12" style="font-size: 80%;">Practicalities, choice of programming language and other computational issues</a></li>
<!-- navigation toc: --> <li><a href="#___sec13" style="font-size: 80%;">Choice of Programming Language</a></li>
<!-- navigation toc: --> <li><a href="#___sec14" style="font-size: 80%;">Data handling, machine learning and ethical aspects</a></li>
</ul>
</li>
@@ -108,11 +151,14 @@ end of tocinfo -->
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Aug 19, 2020</h4></center> <!-- date -->
<center><h4>Aug 20, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- potential-jumbotron-button -->
</div> <!-- end jumbotron -->
<!-- !split -->
<h2 id="___sec0" class="anchor">Introduction </h2>
<p>
@@ -251,7 +297,69 @@ and the jackknife methods and the infamous bias-variance tradeoff.
The second part of the material covers several algorithms used in
machine learning.
<h2 id="___sec2" class="anchor">Types of Machine Learning </h2>
<p>
<!-- !split -->
<p>
<!-- !split -->
<h2 id="___sec2" class="anchor">Machine Learning, short overview </h2>
<p>
<!-- !split -->
<h2 id="___sec3" class="anchor">Machine Learning, a small (and probably biased) introduction </h2>
<p>
Ideally, machine learning represents the science of giving computers
the ability to learn without being explicitly programmed. The idea is
that there exist generic algorithms which can be used to find patterns
in a broad class of data sets without having to write code
specifically for each problem. The algorithm will build its own logic
based on the data. You should however always keep in mind that
machines and algorithms are to a large extent developed by humans. The
insights and knowledge we have about a specific system, play a central
role when we develop a specific machine learning algorithm.
<p>
<!-- !split -->
<h2 id="___sec4" class="anchor">Machine Learning, an extremely rich field </h2>
<p>
Machine learning is an extremely rich field, in spite of its young
age. The increases we have seen during the last decades in
computational capabilities have been followed by developments of
methods and techniques for analyzing and handling large date sets,
relying heavily on statistics, computer science and mathematics. The
field is rather new and developing rapidly. Popular software libraries
written in Python for machine learning like
<a href="http://scikit-learn.org/stable/" target="_self">Scikit-learn</a>,
<a href="https://www.tensorflow.org/" target="_self">Tensorflow</a>,
<a href="http://pytorch.org/" target="_self">PyTorch</a> and <a href="https://keras.io/" target="_self">Keras</a>, all
freely available at their respective GitHub sites, encompass
communities of developers in the thousands or more. And the number of
code developers and contributors keeps increasing.
<p>
<!-- !split -->
<h2 id="___sec5" class="anchor">A multidisciplinary approach </h2>
<p>
Not all the
algorithms and methods can be given a rigorous mathematical
justification (for example decision trees and random forests), opening up thereby large rooms for experimenting and
trial and error and thereby exciting new developments. However, a
solid command of linear algebra, multivariate theory, probability
theory, statistical data analysis, understanding errors and Monte
Carlo methods are central elements in a proper understanding of many
of the algorithms and methods we will discuss.
<p>
<!-- !split -->
<h2 id="___sec6" class="anchor">Types of Machine Learning </h2>
<p>
The approaches to machine learning are many, but are often split into
@@ -268,43 +376,144 @@ solely from rewards and punishment.
Another way to categorize machine learning tasks is to consider the
desired output of a system. Some of the most common tasks are:
<p>
<!-- !bpop -->
<ul>
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
<li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is often supervised learning.</li>
<li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
</ul>
<!-- !epop -->
<p>
<!-- !split -->
<h2 id="___sec7" class="anchor">Essential elements of ML </h2>
<p>
The methods we cover have three main topics in common, irrespective of
whether we deal with supervised or unsupervised learning. The first
ingredient is normally our data set (which can be subdivided into
training and test data), the second item is a model which is normally
a function of some parameters. The model reflects our knowledge of
the system (or lack thereof). As an example, if we know that our data
show a behavior similar to what would be predicted by a polynomial,
fitting our data to a polynomial of some degree would then determin
our model.
whether we deal with supervised or unsupervised learning.
<!-- !bpop -->
<ul>
<li> The first ingredient is normally our data set (which can be subdivided into training, validation and test data). Many find the most difficult part of using Machine Learning to be the set up of your data in a meaningful way.</li>
<li> The second item is a model which is normally a function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.</li>
<li> The last ingredient is a so-called <b>cost/loss</b> function (or error function) which allows us to present an estimate on how good our model is in reproducing the data it is supposed to train.</li>
</ul>
<!-- !epop -->
<p>
The last ingredient is a so-called <b>cost</b>
function which allows us to present an estimate on how good our model
is in reproducing the data it is supposed to train.
<!-- !split -->
<h2 id="___sec8" class="anchor">An optimization/minimization problem </h2>
<p>
Here we will build our machine learning approach on elements of the
statistical foundation discussed above, with elements from data
analysis, stochastic processes etc. We will discuss the following
machine learning algorithms
At the heart of basically all Machine Learning algorithms we will encounter so-called minimization or optimization algorithms. A large family of such methods are so-called <b>gradient methods</b>.
<ol>
<li> Linear regression and its variants</li>
<li> Decision tree algorithms, from single trees to random forests</li>
<li> Bayesian statistics and regression</li>
<li> Support vector machines and finally various variants of</li>
<li> Artifical neural networks and deep learning, including convolutional neural networks and Bayesian neural networks</li>
<li> Networks for unsupervised learning using for example reduced Boltzmann machines.</li>
</ol>
<p>
<!-- !split -->
<h2 id="___sec3" class="anchor">Choice of programming language </h2>
<h2 id="___sec9" class="anchor">A Frequentist approach to data analysis </h2>
<p>
When you hear phrases like <b>predictions and estimations</b> and
<b>correlations and causations</b>, what do you think of? May be you think
of the difference between classifying new data points and generating
new data points.
Or perhaps you consider that correlations represent some kind of symmetric statements like
if \( A \) is correlated with \( B \), then \( B \) is correlated with
\( A \). Causation on the other hand is directional, that is if \( A \) causes \( B \), \( B \) does not
necessarily cause \( A \).
<p>
These concepts are in some sense the difference between machine
learning and statistics. In machine learning and prediction based
tasks, we are often interested in developing algorithms that are
capable of learning patterns from given data in an automated fashion,
and then using these learned patterns to make predictions or
assessments of newly given data. In many cases, our primary concern
is the quality of the predictions or assessments, and we are less
concerned about the underlying patterns that were learned in order
to make these predictions.
<p>
In machine learning we normally use <a href="https://en.wikipedia.org/wiki/Frequentist_inference" target="_self">a so-called frequentist approach</a>,
where the aim is to make predictions and find correlations. We focus
less on for example extracting a probability distribution function (PDF). The PDF can be
used in turn to make estimations and find causations such as given \( A \)
what is the likelihood of finding \( B \).
<p>
<!-- !split -->
<h2 id="___sec10" class="anchor">What is a good model? </h2>
<p>
In science and engineering we often end up in situations where we want to infer (or learn) a
quantitative model \( M \) for a given set of sample points \( \boldsymbol{X} \in [x_1, x_2,\dots x_N] \).
<p>
As we will see repeatedely in these lectures, we could try to fit these data points to a model given by a
straight line, or if we wish to be more sophisticated to a more complex
function.
<p>
The reason for inferring such a model is that it
serves many useful purposes. On the one hand, the model can reveal information
encoded in the data or underlying mechanisms from which the data were generated. For instance, we could discover important
corelations that relate interesting physics interpretations.
<p>
In addition, it can simplify the representation of the given data set and help
us in making predictions about future data samples.
<p>
A first important consideration to keep in mind is that inferring the <em>correct</em> model
for a given data set is an elusive, if not impossible, task. The fundamental difficulty
is that if we are not specific about what we mean by a <em>correct</em> model, there
could easily be many different models that fit the given data set <em>equally well</em>.
<p>
<!-- !split -->
<h2 id="___sec11" class="anchor">What is a good model? Can we define it? </h2>
<p>
The central question is this: what leads us to say that a model is correct or
optimal for a given data set? To make the model inference problem well posed, i.e.,
to guarantee that there is a unique optimal model for the given data, we need to
impose additional assumptions or restrictions on the class of models considered. To
this end, we should not be looking for just any model that can describe the data.
Instead, we should look for a <b>model</b> \( M \) that is the best among a restricted class
of models. In addition, to make the model inference problem computationally
tractable, we need to specify how restricted the class of models needs to be. A
common strategy is to start
with the simplest possible class of models that is just necessary to describe the data
or solve the problem at hand. More precisely, the model class should be rich enough
to contain at least one model that can fit the data to a desired accuracy and yet be
restricted enough that it is relatively simple to find the best model for the given data.
<p>
Thus, the most popular strategy is to start from the
simplest class of models and increase the complexity of the models only when the
simpler models become inadequate. For instance, if we work with a regression problem to fit a set of sample points, one
may first try the simplest class of models, namely linear models, followed obviously by more complex models.
<p>
How to evaluate which model fits best the data is something we will come back to over and over again in these set of lectures.
<p>
<!-- !split -->
<h2 id="___sec12" class="anchor">Practicalities, choice of programming language and other computational issues </h2>
<p>
<!-- !split -->
<h2 id="___sec13" class="anchor">Choice of Programming Language </h2>
<p>
Python plays nowadays a central role in the development of machine
@@ -316,43 +525,14 @@ Jupyter notebook framework with the possibility to run <b>R</b> codes or
compiled programs written in C++, and much more made our choice of
programming language for this series of lectures easy. However,
since the focus here is not only on using existing Python libraries such
as <b>Scikit-Learn</b> or <b>Tensorflow</b>, but also on developing your own
as <b>Scikit-Learn</b>, <b>Tensorflow</b> and <b>Pytorch</b>, but also on developing your own
algorithms and codes, we will as far as possible present many of these
algorithms either as a Python codes or C++ or Fortran (or other languages) codes.
<p>
The reason we also focus on compiled languages like C++ (or
Fortran), is that Python is still notoriously slow when we do not
utilize highly streamlined computational libraries like
<a href="http://www.netlib.org/lapack/" target="_self">Lapack</a> or other numerical libraries
written in compiled languages (many of these libraries are written in
Fortran). Although a project like <a href="https://numba.pydata.org/" target="_self">Numba</a>
holds great promise for speeding up the unrolling of lengthy loops, C++
and Fortran are presently still the performance winners. Numba gives
you potentially the power to speed up your applications with high
performance functions written directly in Python. In particular,
array-oriented and math-heavy Python code can achieve similar
performance to C, C++ and Fortran. However, even with these speed-ups,
for codes involving heavy Markov Chain Monte Carlo analyses and
optimizations of cost functions, C++/C or Fortran codes tend to
outperform Python codes.
<!-- !split -->
<p>
Presently thus, the community tends to let
code written in C++/C or Fortran do the heavy duty numerical
number crunching and leave the post-analysis of the data to the above
mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen
the changes during the last decade, the above situation may change swiftly in the not too distant future.
<p>
Many of the examples we discuss in this series of lectures come with
existing data files or provide code examples which produce the data to
be analyzed. Most of the applications we will discuss deal with
small data sets (less than a terabyte of information) and can easily
be analyzed and tested on standard off the shelf laptops you find in general
stores.
<h2 id="___sec4" class="anchor">Data handling, machine learning and ethical aspects </h2>
<h2 id="___sec14" class="anchor">Data handling, machine learning and ethical aspects </h2>
<p>
In most of the cases we will study, we will either generate the data
doc/pub/Introduction/html/Introduction-reveal.html
+214 -72
View File
@@ -107,6 +107,22 @@ td.padding {
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
@@ -132,9 +148,17 @@ td.padding {
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Aug 19, 2020</h4></center> <!-- date -->
<center><h4>Aug 20, 2020</h4></center> <!-- date -->
<br>
<p>
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</section>
<section>
<h2 id="___sec0">Introduction </h2>
<p>
@@ -215,12 +239,6 @@ solid command of linear algebra, multivariate theory, probability
theory, statistical data analysis, understanding errors and Monte
Carlo methods are central elements in a proper understanding of many
of algorithms and methods we will discuss.
<p>
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</section>
@@ -279,8 +297,71 @@ and the jackknife methods and the infamous bias-variance tradeoff.
<p>
The second part of the material covers several algorithms used in
machine learning.
</section>
<h2 id="___sec2">Types of Machine Learning </h2>
<section>
</section>
<section>
<h2 id="___sec2">Machine Learning, short overview </h2>
</section>
<section>
<h2 id="___sec3">Machine Learning, a small (and probably biased) introduction </h2>
<p>
Ideally, machine learning represents the science of giving computers
the ability to learn without being explicitly programmed. The idea is
that there exist generic algorithms which can be used to find patterns
in a broad class of data sets without having to write code
specifically for each problem. The algorithm will build its own logic
based on the data. You should however always keep in mind that
machines and algorithms are to a large extent developed by humans. The
insights and knowledge we have about a specific system, play a central
role when we develop a specific machine learning algorithm.
</section>
<section>
<h2 id="___sec4">Machine Learning, an extremely rich field </h2>
<p>
Machine learning is an extremely rich field, in spite of its young
age. The increases we have seen during the last decades in
computational capabilities have been followed by developments of
methods and techniques for analyzing and handling large date sets,
relying heavily on statistics, computer science and mathematics. The
field is rather new and developing rapidly. Popular software libraries
written in Python for machine learning like
<a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a>,
<a href="https://www.tensorflow.org/" target="_blank">Tensorflow</a>,
<a href="http://pytorch.org/" target="_blank">PyTorch</a> and <a href="https://keras.io/" target="_blank">Keras</a>, all
freely available at their respective GitHub sites, encompass
communities of developers in the thousands or more. And the number of
code developers and contributors keeps increasing.
</section>
<section>
<h2 id="___sec5">A multidisciplinary approach </h2>
<p>
Not all the
algorithms and methods can be given a rigorous mathematical
justification (for example decision trees and random forests), opening up thereby large rooms for experimenting and
trial and error and thereby exciting new developments. However, a
solid command of linear algebra, multivariate theory, probability
theory, statistical data analysis, understanding errors and Monte
Carlo methods are central elements in a proper understanding of many
of the algorithms and methods we will discuss.
</section>
<section>
<h2 id="___sec6">Types of Machine Learning </h2>
<p>
The approaches to machine learning are many, but are often split into
@@ -298,46 +379,136 @@ Another way to categorize machine learning tasks is to consider the
desired output of a system. Some of the most common tasks are:
<ul>
<p><li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
<p><li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<p><li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
<p><li class="fragment"> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is often supervised learning.</li>
<p><li class="fragment"> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<p><li class="fragment"> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
</ul>
<p>
</section>
<section>
<h2 id="___sec7">Essential elements of ML </h2>
<p>
The methods we cover have three main topics in common, irrespective of
whether we deal with supervised or unsupervised learning. The first
ingredient is normally our data set (which can be subdivided into
training and test data), the second item is a model which is normally
a function of some parameters. The model reflects our knowledge of
the system (or lack thereof). As an example, if we know that our data
show a behavior similar to what would be predicted by a polynomial,
fitting our data to a polynomial of some degree would then determin
our model.
whether we deal with supervised or unsupervised learning.
<ul>
<p><li class="fragment"> The first ingredient is normally our data set (which can be subdivided into training, validation and test data). Many find the most difficult part of using Machine Learning to be the set up of your data in a meaningful way.</li>
<p><li class="fragment"> The second item is a model which is normally a function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.</li>
<p><li class="fragment"> The last ingredient is a so-called <b>cost/loss</b> function (or error function) which allows us to present an estimate on how good our model is in reproducing the data it is supposed to train.</li>
</ul>
</section>
<section>
<h2 id="___sec8">An optimization/minimization problem </h2>
<p>
The last ingredient is a so-called <b>cost</b>
function which allows us to present an estimate on how good our model
is in reproducing the data it is supposed to train.
At the heart of basically all Machine Learning algorithms we will encounter so-called minimization or optimization algorithms. A large family of such methods are so-called <b>gradient methods</b>.
</section>
<section>
<h2 id="___sec9">A Frequentist approach to data analysis </h2>
<p>
Here we will build our machine learning approach on elements of the
statistical foundation discussed above, with elements from data
analysis, stochastic processes etc. We will discuss the following
machine learning algorithms
When you hear phrases like <b>predictions and estimations</b> and
<b>correlations and causations</b>, what do you think of? May be you think
of the difference between classifying new data points and generating
new data points.
Or perhaps you consider that correlations represent some kind of symmetric statements like
if \( A \) is correlated with \( B \), then \( B \) is correlated with
\( A \). Causation on the other hand is directional, that is if \( A \) causes \( B \), \( B \) does not
necessarily cause \( A \).
<ol>
<p><li> Linear regression and its variants</li>
<p><li> Decision tree algorithms, from single trees to random forests</li>
<p><li> Bayesian statistics and regression</li>
<p><li> Support vector machines and finally various variants of</li>
<p><li> Artifical neural networks and deep learning, including convolutional neural networks and Bayesian neural networks</li>
<p><li> Networks for unsupervised learning using for example reduced Boltzmann machines.</li>
</ol>
<p>
These concepts are in some sense the difference between machine
learning and statistics. In machine learning and prediction based
tasks, we are often interested in developing algorithms that are
capable of learning patterns from given data in an automated fashion,
and then using these learned patterns to make predictions or
assessments of newly given data. In many cases, our primary concern
is the quality of the predictions or assessments, and we are less
concerned about the underlying patterns that were learned in order
to make these predictions.
<h2 id="___sec3">Choice of programming language </h2>
<p>
In machine learning we normally use <a href="https://en.wikipedia.org/wiki/Frequentist_inference" target="_blank">a so-called frequentist approach</a>,
where the aim is to make predictions and find correlations. We focus
less on for example extracting a probability distribution function (PDF). The PDF can be
used in turn to make estimations and find causations such as given \( A \)
what is the likelihood of finding \( B \).
</section>
<section>
<h2 id="___sec10">What is a good model? </h2>
<p>
In science and engineering we often end up in situations where we want to infer (or learn) a
quantitative model \( M \) for a given set of sample points \( \boldsymbol{X} \in [x_1, x_2,\dots x_N] \).
<p>
As we will see repeatedely in these lectures, we could try to fit these data points to a model given by a
straight line, or if we wish to be more sophisticated to a more complex
function.
<p>
The reason for inferring such a model is that it
serves many useful purposes. On the one hand, the model can reveal information
encoded in the data or underlying mechanisms from which the data were generated. For instance, we could discover important
corelations that relate interesting physics interpretations.
<p>
In addition, it can simplify the representation of the given data set and help
us in making predictions about future data samples.
<p>
A first important consideration to keep in mind is that inferring the <em>correct</em> model
for a given data set is an elusive, if not impossible, task. The fundamental difficulty
is that if we are not specific about what we mean by a <em>correct</em> model, there
could easily be many different models that fit the given data set <em>equally well</em>.
</section>
<section>
<h2 id="___sec11">What is a good model? Can we define it? </h2>
<p>
The central question is this: what leads us to say that a model is correct or
optimal for a given data set? To make the model inference problem well posed, i.e.,
to guarantee that there is a unique optimal model for the given data, we need to
impose additional assumptions or restrictions on the class of models considered. To
this end, we should not be looking for just any model that can describe the data.
Instead, we should look for a <b>model</b> \( M \) that is the best among a restricted class
of models. In addition, to make the model inference problem computationally
tractable, we need to specify how restricted the class of models needs to be. A
common strategy is to start
with the simplest possible class of models that is just necessary to describe the data
or solve the problem at hand. More precisely, the model class should be rich enough
to contain at least one model that can fit the data to a desired accuracy and yet be
restricted enough that it is relatively simple to find the best model for the given data.
<p>
Thus, the most popular strategy is to start from the
simplest class of models and increase the complexity of the models only when the
simpler models become inadequate. For instance, if we work with a regression problem to fit a set of sample points, one
may first try the simplest class of models, namely linear models, followed obviously by more complex models.
<p>
How to evaluate which model fits best the data is something we will come back to over and over again in these set of lectures.
</section>
<section>
<h2 id="___sec12">Practicalities, choice of programming language and other computational issues </h2>
</section>
<section>
<h2 id="___sec13">Choice of Programming Language </h2>
<p>
Python plays nowadays a central role in the development of machine
@@ -349,43 +520,14 @@ Jupyter notebook framework with the possibility to run <b>R</b> codes or
compiled programs written in C++, and much more made our choice of
programming language for this series of lectures easy. However,
since the focus here is not only on using existing Python libraries such
as <b>Scikit-Learn</b> or <b>Tensorflow</b>, but also on developing your own
as <b>Scikit-Learn</b>, <b>Tensorflow</b> and <b>Pytorch</b>, but also on developing your own
algorithms and codes, we will as far as possible present many of these
algorithms either as a Python codes or C++ or Fortran (or other languages) codes.
</section>
<p>
The reason we also focus on compiled languages like C++ (or
Fortran), is that Python is still notoriously slow when we do not
utilize highly streamlined computational libraries like
<a href="http://www.netlib.org/lapack/" target="_blank">Lapack</a> or other numerical libraries
written in compiled languages (many of these libraries are written in
Fortran). Although a project like <a href="https://numba.pydata.org/" target="_blank">Numba</a>
holds great promise for speeding up the unrolling of lengthy loops, C++
and Fortran are presently still the performance winners. Numba gives
you potentially the power to speed up your applications with high
performance functions written directly in Python. In particular,
array-oriented and math-heavy Python code can achieve similar
performance to C, C++ and Fortran. However, even with these speed-ups,
for codes involving heavy Markov Chain Monte Carlo analyses and
optimizations of cost functions, C++/C or Fortran codes tend to
outperform Python codes.
<p>
Presently thus, the community tends to let
code written in C++/C or Fortran do the heavy duty numerical
number crunching and leave the post-analysis of the data to the above
mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen
the changes during the last decade, the above situation may change swiftly in the not too distant future.
<p>
Many of the examples we discuss in this series of lectures come with
existing data files or provide code examples which produce the data to
be analyzed. Most of the applications we will discuss deal with
small data sets (less than a terabyte of information) and can easily
be analyzed and tested on standard off the shelf laptops you find in general
stores.
<h2 id="___sec4">Data handling, machine learning and ethical aspects </h2>
<section>
<h2 id="___sec14">Data handling, machine learning and ethical aspects </h2>
<p>
In most of the cases we will study, we will either generate the data
doc/pub/Introduction/html/Introduction-solarized.html
+233 -64
View File
@@ -37,16 +37,49 @@ div { text-align: justify; text-justify: inter-word; }
{'highest level': 2,
'sections': [('Introduction', 2, None, '___sec0'),
('Learning outcomes', 2, None, '___sec1'),
('Types of Machine Learning', 2, None, '___sec2'),
('Choice of programming language', 2, None, '___sec3'),
('Machine Learning, short overview', 2, None, '___sec2'),
('Machine Learning, a small (and probably biased) introduction',
2,
None,
'___sec3'),
('Machine Learning, an extremely rich field', 2, None, '___sec4'),
('A multidisciplinary approach', 2, None, '___sec5'),
('Types of Machine Learning', 2, None, '___sec6'),
('Essential elements of ML', 2, None, '___sec7'),
('An optimization/minimization problem', 2, None, '___sec8'),
('A Frequentist approach to data analysis', 2, None, '___sec9'),
('What is a good model?', 2, None, '___sec10'),
('What is a good model? Can we define it?', 2, None, '___sec11'),
('Practicalities, choice of programming language and other '
'computational issues',
2,
None,
'___sec12'),
('Choice of Programming Language', 2, None, '___sec13'),
('Data handling, machine learning and ethical aspects',
2,
None,
'___sec4')]}
'___sec14')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- ------------------- main content ---------------------- -->
@@ -68,8 +101,10 @@ end of tocinfo -->
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Aug 19, 2020</h4></center> <!-- date -->
<center><h4>Aug 20, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Introduction </h2>
@@ -209,7 +244,69 @@ and the jackknife methods and the infamous bias-variance tradeoff.
The second part of the material covers several algorithms used in
machine learning.
<h2 id="___sec2">Types of Machine Learning </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Machine Learning, short overview </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Machine Learning, a small (and probably biased) introduction </h2>
<p>
Ideally, machine learning represents the science of giving computers
the ability to learn without being explicitly programmed. The idea is
that there exist generic algorithms which can be used to find patterns
in a broad class of data sets without having to write code
specifically for each problem. The algorithm will build its own logic
based on the data. You should however always keep in mind that
machines and algorithms are to a large extent developed by humans. The
insights and knowledge we have about a specific system, play a central
role when we develop a specific machine learning algorithm.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Machine Learning, an extremely rich field </h2>
<p>
Machine learning is an extremely rich field, in spite of its young
age. The increases we have seen during the last decades in
computational capabilities have been followed by developments of
methods and techniques for analyzing and handling large date sets,
relying heavily on statistics, computer science and mathematics. The
field is rather new and developing rapidly. Popular software libraries
written in Python for machine learning like
<a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a>,
<a href="https://www.tensorflow.org/" target="_blank">Tensorflow</a>,
<a href="http://pytorch.org/" target="_blank">PyTorch</a> and <a href="https://keras.io/" target="_blank">Keras</a>, all
freely available at their respective GitHub sites, encompass
communities of developers in the thousands or more. And the number of
code developers and contributors keeps increasing.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">A multidisciplinary approach </h2>
<p>
Not all the
algorithms and methods can be given a rigorous mathematical
justification (for example decision trees and random forests), opening up thereby large rooms for experimenting and
trial and error and thereby exciting new developments. However, a
solid command of linear algebra, multivariate theory, probability
theory, statistical data analysis, understanding errors and Monte
Carlo methods are central elements in a proper understanding of many
of the algorithms and methods we will discuss.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Types of Machine Learning </h2>
<p>
The approaches to machine learning are many, but are often split into
@@ -226,43 +323,144 @@ solely from rewards and punishment.
Another way to categorize machine learning tasks is to consider the
desired output of a system. Some of the most common tasks are:
<p>
<!-- !bpop -->
<ul>
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
<li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is often supervised learning.</li>
<li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
</ul>
<!-- !epop -->
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Essential elements of ML </h2>
<p>
The methods we cover have three main topics in common, irrespective of
whether we deal with supervised or unsupervised learning. The first
ingredient is normally our data set (which can be subdivided into
training and test data), the second item is a model which is normally
a function of some parameters. The model reflects our knowledge of
the system (or lack thereof). As an example, if we know that our data
show a behavior similar to what would be predicted by a polynomial,
fitting our data to a polynomial of some degree would then determin
our model.
whether we deal with supervised or unsupervised learning.
<!-- !bpop -->
<ul>
<li> The first ingredient is normally our data set (which can be subdivided into training, validation and test data). Many find the most difficult part of using Machine Learning to be the set up of your data in a meaningful way.</li>
<li> The second item is a model which is normally a function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.</li>
<li> The last ingredient is a so-called <b>cost/loss</b> function (or error function) which allows us to present an estimate on how good our model is in reproducing the data it is supposed to train.</li>
</ul>
<!-- !epop -->
<p>
The last ingredient is a so-called <b>cost</b>
function which allows us to present an estimate on how good our model
is in reproducing the data it is supposed to train.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">An optimization/minimization problem </h2>
<p>
Here we will build our machine learning approach on elements of the
statistical foundation discussed above, with elements from data
analysis, stochastic processes etc. We will discuss the following
machine learning algorithms
At the heart of basically all Machine Learning algorithms we will encounter so-called minimization or optimization algorithms. A large family of such methods are so-called <b>gradient methods</b>.
<ol>
<li> Linear regression and its variants</li>
<li> Decision tree algorithms, from single trees to random forests</li>
<li> Bayesian statistics and regression</li>
<li> Support vector machines and finally various variants of</li>
<li> Artifical neural networks and deep learning, including convolutional neural networks and Bayesian neural networks</li>
<li> Networks for unsupervised learning using for example reduced Boltzmann machines.</li>
</ol>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Choice of programming language </h2>
<h2 id="___sec9">A Frequentist approach to data analysis </h2>
<p>
When you hear phrases like <b>predictions and estimations</b> and
<b>correlations and causations</b>, what do you think of? May be you think
of the difference between classifying new data points and generating
new data points.
Or perhaps you consider that correlations represent some kind of symmetric statements like
if \( A \) is correlated with \( B \), then \( B \) is correlated with
\( A \). Causation on the other hand is directional, that is if \( A \) causes \( B \), \( B \) does not
necessarily cause \( A \).
<p>
These concepts are in some sense the difference between machine
learning and statistics. In machine learning and prediction based
tasks, we are often interested in developing algorithms that are
capable of learning patterns from given data in an automated fashion,
and then using these learned patterns to make predictions or
assessments of newly given data. In many cases, our primary concern
is the quality of the predictions or assessments, and we are less
concerned about the underlying patterns that were learned in order
to make these predictions.
<p>
In machine learning we normally use <a href="https://en.wikipedia.org/wiki/Frequentist_inference" target="_blank">a so-called frequentist approach</a>,
where the aim is to make predictions and find correlations. We focus
less on for example extracting a probability distribution function (PDF). The PDF can be
used in turn to make estimations and find causations such as given \( A \)
what is the likelihood of finding \( B \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">What is a good model? </h2>
<p>
In science and engineering we often end up in situations where we want to infer (or learn) a
quantitative model \( M \) for a given set of sample points \( \boldsymbol{X} \in [x_1, x_2,\dots x_N] \).
<p>
As we will see repeatedely in these lectures, we could try to fit these data points to a model given by a
straight line, or if we wish to be more sophisticated to a more complex
function.
<p>
The reason for inferring such a model is that it
serves many useful purposes. On the one hand, the model can reveal information
encoded in the data or underlying mechanisms from which the data were generated. For instance, we could discover important
corelations that relate interesting physics interpretations.
<p>
In addition, it can simplify the representation of the given data set and help
us in making predictions about future data samples.
<p>
A first important consideration to keep in mind is that inferring the <em>correct</em> model
for a given data set is an elusive, if not impossible, task. The fundamental difficulty
is that if we are not specific about what we mean by a <em>correct</em> model, there
could easily be many different models that fit the given data set <em>equally well</em>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">What is a good model? Can we define it? </h2>
<p>
The central question is this: what leads us to say that a model is correct or
optimal for a given data set? To make the model inference problem well posed, i.e.,
to guarantee that there is a unique optimal model for the given data, we need to
impose additional assumptions or restrictions on the class of models considered. To
this end, we should not be looking for just any model that can describe the data.
Instead, we should look for a <b>model</b> \( M \) that is the best among a restricted class
of models. In addition, to make the model inference problem computationally
tractable, we need to specify how restricted the class of models needs to be. A
common strategy is to start
with the simplest possible class of models that is just necessary to describe the data
or solve the problem at hand. More precisely, the model class should be rich enough
to contain at least one model that can fit the data to a desired accuracy and yet be
restricted enough that it is relatively simple to find the best model for the given data.
<p>
Thus, the most popular strategy is to start from the
simplest class of models and increase the complexity of the models only when the
simpler models become inadequate. For instance, if we work with a regression problem to fit a set of sample points, one
may first try the simplest class of models, namely linear models, followed obviously by more complex models.
<p>
How to evaluate which model fits best the data is something we will come back to over and over again in these set of lectures.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Practicalities, choice of programming language and other computational issues </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Choice of Programming Language </h2>
<p>
Python plays nowadays a central role in the development of machine
@@ -274,43 +472,14 @@ Jupyter notebook framework with the possibility to run <b>R</b> codes or
compiled programs written in C++, and much more made our choice of
programming language for this series of lectures easy. However,
since the focus here is not only on using existing Python libraries such
as <b>Scikit-Learn</b> or <b>Tensorflow</b>, but also on developing your own
as <b>Scikit-Learn</b>, <b>Tensorflow</b> and <b>Pytorch</b>, but also on developing your own
algorithms and codes, we will as far as possible present many of these
algorithms either as a Python codes or C++ or Fortran (or other languages) codes.
<p>
The reason we also focus on compiled languages like C++ (or
Fortran), is that Python is still notoriously slow when we do not
utilize highly streamlined computational libraries like
<a href="http://www.netlib.org/lapack/" target="_blank">Lapack</a> or other numerical libraries
written in compiled languages (many of these libraries are written in
Fortran). Although a project like <a href="https://numba.pydata.org/" target="_blank">Numba</a>
holds great promise for speeding up the unrolling of lengthy loops, C++
and Fortran are presently still the performance winners. Numba gives
you potentially the power to speed up your applications with high
performance functions written directly in Python. In particular,
array-oriented and math-heavy Python code can achieve similar
performance to C, C++ and Fortran. However, even with these speed-ups,
for codes involving heavy Markov Chain Monte Carlo analyses and
optimizations of cost functions, C++/C or Fortran codes tend to
outperform Python codes.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<p>
Presently thus, the community tends to let
code written in C++/C or Fortran do the heavy duty numerical
number crunching and leave the post-analysis of the data to the above
mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen
the changes during the last decade, the above situation may change swiftly in the not too distant future.
<p>
Many of the examples we discuss in this series of lectures come with
existing data files or provide code examples which produce the data to
be analyzed. Most of the applications we will discuss deal with
small data sets (less than a terabyte of information) and can easily
be analyzed and tested on standard off the shelf laptops you find in general
stores.
<h2 id="___sec4">Data handling, machine learning and ethical aspects </h2>
<h2 id="___sec14">Data handling, machine learning and ethical aspects </h2>
<p>
In most of the cases we will study, we will either generate the data
doc/pub/Introduction/html/Introduction.html
+233 -64
View File
@@ -42,16 +42,49 @@ div { text-align: justify; text-justify: inter-word; }
{'highest level': 2,
'sections': [('Introduction', 2, None, '___sec0'),
('Learning outcomes', 2, None, '___sec1'),
('Types of Machine Learning', 2, None, '___sec2'),
('Choice of programming language', 2, None, '___sec3'),
('Machine Learning, short overview', 2, None, '___sec2'),
('Machine Learning, a small (and probably biased) introduction',
2,
None,
'___sec3'),
('Machine Learning, an extremely rich field', 2, None, '___sec4'),
('A multidisciplinary approach', 2, None, '___sec5'),
('Types of Machine Learning', 2, None, '___sec6'),
('Essential elements of ML', 2, None, '___sec7'),
('An optimization/minimization problem', 2, None, '___sec8'),
('A Frequentist approach to data analysis', 2, None, '___sec9'),
('What is a good model?', 2, None, '___sec10'),
('What is a good model? Can we define it?', 2, None, '___sec11'),
('Practicalities, choice of programming language and other '
'computational issues',
2,
None,
'___sec12'),
('Choice of Programming Language', 2, None, '___sec13'),
('Data handling, machine learning and ethical aspects',
2,
None,
'___sec4')]}
'___sec14')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- ------------------- main content ---------------------- -->
@@ -73,8 +106,10 @@ end of tocinfo -->
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Aug 19, 2020</h4></center> <!-- date -->
<center><h4>Aug 20, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Introduction </h2>
@@ -214,7 +249,69 @@ and the jackknife methods and the infamous bias-variance tradeoff.
The second part of the material covers several algorithms used in
machine learning.
<h2 id="___sec2">Types of Machine Learning </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Machine Learning, short overview </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Machine Learning, a small (and probably biased) introduction </h2>
<p>
Ideally, machine learning represents the science of giving computers
the ability to learn without being explicitly programmed. The idea is
that there exist generic algorithms which can be used to find patterns
in a broad class of data sets without having to write code
specifically for each problem. The algorithm will build its own logic
based on the data. You should however always keep in mind that
machines and algorithms are to a large extent developed by humans. The
insights and knowledge we have about a specific system, play a central
role when we develop a specific machine learning algorithm.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Machine Learning, an extremely rich field </h2>
<p>
Machine learning is an extremely rich field, in spite of its young
age. The increases we have seen during the last decades in
computational capabilities have been followed by developments of
methods and techniques for analyzing and handling large date sets,
relying heavily on statistics, computer science and mathematics. The
field is rather new and developing rapidly. Popular software libraries
written in Python for machine learning like
<a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a>,
<a href="https://www.tensorflow.org/" target="_blank">Tensorflow</a>,
<a href="http://pytorch.org/" target="_blank">PyTorch</a> and <a href="https://keras.io/" target="_blank">Keras</a>, all
freely available at their respective GitHub sites, encompass
communities of developers in the thousands or more. And the number of
code developers and contributors keeps increasing.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">A multidisciplinary approach </h2>
<p>
Not all the
algorithms and methods can be given a rigorous mathematical
justification (for example decision trees and random forests), opening up thereby large rooms for experimenting and
trial and error and thereby exciting new developments. However, a
solid command of linear algebra, multivariate theory, probability
theory, statistical data analysis, understanding errors and Monte
Carlo methods are central elements in a proper understanding of many
of the algorithms and methods we will discuss.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Types of Machine Learning </h2>
<p>
The approaches to machine learning are many, but are often split into
@@ -231,43 +328,144 @@ solely from rewards and punishment.
Another way to categorize machine learning tasks is to consider the
desired output of a system. Some of the most common tasks are:
<p>
<!-- !bpop -->
<ul>
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
<li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is often supervised learning.</li>
<li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
<li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
</ul>
<!-- !epop -->
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Essential elements of ML </h2>
<p>
The methods we cover have three main topics in common, irrespective of
whether we deal with supervised or unsupervised learning. The first
ingredient is normally our data set (which can be subdivided into
training and test data), the second item is a model which is normally
a function of some parameters. The model reflects our knowledge of
the system (or lack thereof). As an example, if we know that our data
show a behavior similar to what would be predicted by a polynomial,
fitting our data to a polynomial of some degree would then determin
our model.
whether we deal with supervised or unsupervised learning.
<!-- !bpop -->
<ul>
<li> The first ingredient is normally our data set (which can be subdivided into training, validation and test data). Many find the most difficult part of using Machine Learning to be the set up of your data in a meaningful way.</li>
<li> The second item is a model which is normally a function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.</li>
<li> The last ingredient is a so-called <b>cost/loss</b> function (or error function) which allows us to present an estimate on how good our model is in reproducing the data it is supposed to train.</li>
</ul>
<!-- !epop -->
<p>
The last ingredient is a so-called <b>cost</b>
function which allows us to present an estimate on how good our model
is in reproducing the data it is supposed to train.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">An optimization/minimization problem </h2>
<p>
Here we will build our machine learning approach on elements of the
statistical foundation discussed above, with elements from data
analysis, stochastic processes etc. We will discuss the following
machine learning algorithms
At the heart of basically all Machine Learning algorithms we will encounter so-called minimization or optimization algorithms. A large family of such methods are so-called <b>gradient methods</b>.
<ol>
<li> Linear regression and its variants</li>
<li> Decision tree algorithms, from single trees to random forests</li>
<li> Bayesian statistics and regression</li>
<li> Support vector machines and finally various variants of</li>
<li> Artifical neural networks and deep learning, including convolutional neural networks and Bayesian neural networks</li>
<li> Networks for unsupervised learning using for example reduced Boltzmann machines.</li>
</ol>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Choice of programming language </h2>
<h2 id="___sec9">A Frequentist approach to data analysis </h2>
<p>
When you hear phrases like <b>predictions and estimations</b> and
<b>correlations and causations</b>, what do you think of? May be you think
of the difference between classifying new data points and generating
new data points.
Or perhaps you consider that correlations represent some kind of symmetric statements like
if \( A \) is correlated with \( B \), then \( B \) is correlated with
\( A \). Causation on the other hand is directional, that is if \( A \) causes \( B \), \( B \) does not
necessarily cause \( A \).
<p>
These concepts are in some sense the difference between machine
learning and statistics. In machine learning and prediction based
tasks, we are often interested in developing algorithms that are
capable of learning patterns from given data in an automated fashion,
and then using these learned patterns to make predictions or
assessments of newly given data. In many cases, our primary concern
is the quality of the predictions or assessments, and we are less
concerned about the underlying patterns that were learned in order
to make these predictions.
<p>
In machine learning we normally use <a href="https://en.wikipedia.org/wiki/Frequentist_inference" target="_blank">a so-called frequentist approach</a>,
where the aim is to make predictions and find correlations. We focus
less on for example extracting a probability distribution function (PDF). The PDF can be
used in turn to make estimations and find causations such as given \( A \)
what is the likelihood of finding \( B \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">What is a good model? </h2>
<p>
In science and engineering we often end up in situations where we want to infer (or learn) a
quantitative model \( M \) for a given set of sample points \( \boldsymbol{X} \in [x_1, x_2,\dots x_N] \).
<p>
As we will see repeatedely in these lectures, we could try to fit these data points to a model given by a
straight line, or if we wish to be more sophisticated to a more complex
function.
<p>
The reason for inferring such a model is that it
serves many useful purposes. On the one hand, the model can reveal information
encoded in the data or underlying mechanisms from which the data were generated. For instance, we could discover important
corelations that relate interesting physics interpretations.
<p>
In addition, it can simplify the representation of the given data set and help
us in making predictions about future data samples.
<p>
A first important consideration to keep in mind is that inferring the <em>correct</em> model
for a given data set is an elusive, if not impossible, task. The fundamental difficulty
is that if we are not specific about what we mean by a <em>correct</em> model, there
could easily be many different models that fit the given data set <em>equally well</em>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">What is a good model? Can we define it? </h2>
<p>
The central question is this: what leads us to say that a model is correct or
optimal for a given data set? To make the model inference problem well posed, i.e.,
to guarantee that there is a unique optimal model for the given data, we need to
impose additional assumptions or restrictions on the class of models considered. To
this end, we should not be looking for just any model that can describe the data.
Instead, we should look for a <b>model</b> \( M \) that is the best among a restricted class
of models. In addition, to make the model inference problem computationally
tractable, we need to specify how restricted the class of models needs to be. A
common strategy is to start
with the simplest possible class of models that is just necessary to describe the data
or solve the problem at hand. More precisely, the model class should be rich enough
to contain at least one model that can fit the data to a desired accuracy and yet be
restricted enough that it is relatively simple to find the best model for the given data.
<p>
Thus, the most popular strategy is to start from the
simplest class of models and increase the complexity of the models only when the
simpler models become inadequate. For instance, if we work with a regression problem to fit a set of sample points, one
may first try the simplest class of models, namely linear models, followed obviously by more complex models.
<p>
How to evaluate which model fits best the data is something we will come back to over and over again in these set of lectures.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Practicalities, choice of programming language and other computational issues </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Choice of Programming Language </h2>
<p>
Python plays nowadays a central role in the development of machine
@@ -279,43 +477,14 @@ Jupyter notebook framework with the possibility to run <b>R</b> codes or
compiled programs written in C++, and much more made our choice of
programming language for this series of lectures easy. However,
since the focus here is not only on using existing Python libraries such
as <b>Scikit-Learn</b> or <b>Tensorflow</b>, but also on developing your own
as <b>Scikit-Learn</b>, <b>Tensorflow</b> and <b>Pytorch</b>, but also on developing your own
algorithms and codes, we will as far as possible present many of these
algorithms either as a Python codes or C++ or Fortran (or other languages) codes.
<p>
The reason we also focus on compiled languages like C++ (or
Fortran), is that Python is still notoriously slow when we do not
utilize highly streamlined computational libraries like
<a href="http://www.netlib.org/lapack/" target="_blank">Lapack</a> or other numerical libraries
written in compiled languages (many of these libraries are written in
Fortran). Although a project like <a href="https://numba.pydata.org/" target="_blank">Numba</a>
holds great promise for speeding up the unrolling of lengthy loops, C++
and Fortran are presently still the performance winners. Numba gives
you potentially the power to speed up your applications with high
performance functions written directly in Python. In particular,
array-oriented and math-heavy Python code can achieve similar
performance to C, C++ and Fortran. However, even with these speed-ups,
for codes involving heavy Markov Chain Monte Carlo analyses and
optimizations of cost functions, C++/C or Fortran codes tend to
outperform Python codes.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<p>
Presently thus, the community tends to let
code written in C++/C or Fortran do the heavy duty numerical
number crunching and leave the post-analysis of the data to the above
mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen
the changes during the last decade, the above situation may change swiftly in the not too distant future.
<p>
Many of the examples we discuss in this series of lectures come with
existing data files or provide code examples which produce the data to
be analyzed. Most of the applications we will discuss deal with
small data sets (less than a terabyte of information) and can easily
be analyzed and tested on standard off the shelf laptops you find in general
stores.
<h2 id="___sec4">Data handling, machine learning and ethical aspects </h2>
<h2 id="___sec14">Data handling, machine learning and ethical aspects </h2>
<p>
In most of the cases we will study, we will either generate the data
doc/pub/Introduction/ipynb/ipynb-Introduction-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/Introduction/pdf/Introduction-minted.pdf
BIN
View File
Binary file not shown.
doc/src/Introduction/Introduction.do.txt
+159 -55
View File
@@ -3,7 +3,7 @@ AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of
DATE: today
!split
===== Introduction =====
During the last two decades there has been a swift and amazing
@@ -131,9 +131,59 @@ machine learning.
!split
!split
===== Machine Learning, short overview =====
!split
===== Machine Learning, a small (and probably biased) introduction =====
Ideally, machine learning represents the science of giving computers
the ability to learn without being explicitly programmed. The idea is
that there exist generic algorithms which can be used to find patterns
in a broad class of data sets without having to write code
specifically for each problem. The algorithm will build its own logic
based on the data. You should however always keep in mind that
machines and algorithms are to a large extent developed by humans. The
insights and knowledge we have about a specific system, play a central
role when we develop a specific machine learning algorithm.
!split
===== Machine Learning, an extremely rich field =====
Machine learning is an extremely rich field, in spite of its young
age. The increases we have seen during the last decades in
computational capabilities have been followed by developments of
methods and techniques for analyzing and handling large date sets,
relying heavily on statistics, computer science and mathematics. The
field is rather new and developing rapidly. Popular software libraries
written in Python for machine learning like
"Scikit-learn":"http://scikit-learn.org/stable/",
"Tensorflow":"https://www.tensorflow.org/",
"PyTorch":"http://pytorch.org/" and "Keras":"https://keras.io/", all
freely available at their respective GitHub sites, encompass
communities of developers in the thousands or more. And the number of
code developers and contributors keeps increasing.
!split
===== A multidisciplinary approach =====
Not all the
algorithms and methods can be given a rigorous mathematical
justification (for example decision trees and random forests), opening up thereby large rooms for experimenting and
trial and error and thereby exciting new developments. However, a
solid command of linear algebra, multivariate theory, probability
theory, statistical data analysis, understanding errors and Monte
Carlo methods are central elements in a proper understanding of many
of the algorithms and methods we will discuss.
!split
===== Types of Machine Learning =====
@@ -150,42 +200,120 @@ solely from rewards and punishment.
Another way to categorize machine learning tasks is to consider the
desired output of a system. Some of the most common tasks are:
* Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.
!bpop
* Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is often supervised learning.
* Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.
* Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.
* Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.
* Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.
!epop
!split
===== Essential elements of ML =====
The methods we cover have three main topics in common, irrespective of
whether we deal with supervised or unsupervised learning. The first
ingredient is normally our data set (which can be subdivided into
training and test data), the second item is a model which is normally
a function of some parameters. The model reflects our knowledge of
the system (or lack thereof). As an example, if we know that our data
show a behavior similar to what would be predicted by a polynomial,
fitting our data to a polynomial of some degree would then determin
our model.
whether we deal with supervised or unsupervised learning.
!bpop
* The first ingredient is normally our data set (which can be subdivided into training, validation and test data). Many find the most difficult part of using Machine Learning to be the set up of your data in a meaningful way.
The last ingredient is a so-called _cost_
function which allows us to present an estimate on how good our model
is in reproducing the data it is supposed to train.
* The second item is a model which is normally a function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.
Here we will build our machine learning approach on elements of the
statistical foundation discussed above, with elements from data
analysis, stochastic processes etc. We will discuss the following
machine learning algorithms
o Linear regression and its variants
o Decision tree algorithms, from single trees to random forests
o Bayesian statistics and regression
o Support vector machines and finally various variants of
o Artifical neural networks and deep learning, including convolutional neural networks and Bayesian neural networks
o Networks for unsupervised learning using for example reduced Boltzmann machines.
* The last ingredient is a so-called _cost/loss_ function (or error function) which allows us to present an estimate on how good our model is in reproducing the data it is supposed to train.
!epop
===== Choice of programming language =====
!split
===== An optimization/minimization problem =====
At the heart of basically all Machine Learning algorithms we will encounter so-called minimization or optimization algorithms. A large family of such methods are so-called _gradient methods_.
!split
===== A Frequentist approach to data analysis =====
When you hear phrases like _predictions and estimations_ and
_correlations and causations_, what do you think of? May be you think
of the difference between classifying new data points and generating
new data points.
Or perhaps you consider that correlations represent some kind of symmetric statements like
if $A$ is correlated with $B$, then $B$ is correlated with
$A$. Causation on the other hand is directional, that is if $A$ causes $B$, $B$ does not
necessarily cause $A$.
These concepts are in some sense the difference between machine
learning and statistics. In machine learning and prediction based
tasks, we are often interested in developing algorithms that are
capable of learning patterns from given data in an automated fashion,
and then using these learned patterns to make predictions or
assessments of newly given data. In many cases, our primary concern
is the quality of the predictions or assessments, and we are less
concerned about the underlying patterns that were learned in order
to make these predictions.
In machine learning we normally use "a so-called frequentist approach":"https://en.wikipedia.org/wiki/Frequentist_inference",
where the aim is to make predictions and find correlations. We focus
less on for example extracting a probability distribution function (PDF). The PDF can be
used in turn to make estimations and find causations such as given $A$
what is the likelihood of finding $B$.
!split
===== What is a good model? =====
In science and engineering we often end up in situations where we want to infer (or learn) a
quantitative model $M$ for a given set of sample points $\bm{X} \in [x_1, x_2,\dots x_N]$.
As we will see repeatedely in these lectures, we could try to fit these data points to a model given by a
straight line, or if we wish to be more sophisticated to a more complex
function.
The reason for inferring such a model is that it
serves many useful purposes. On the one hand, the model can reveal information
encoded in the data or underlying mechanisms from which the data were generated. For instance, we could discover important
corelations that relate interesting physics interpretations.
In addition, it can simplify the representation of the given data set and help
us in making predictions about future data samples.
A first important consideration to keep in mind is that inferring the *correct* model
for a given data set is an elusive, if not impossible, task. The fundamental difficulty
is that if we are not specific about what we mean by a *correct* model, there
could easily be many different models that fit the given data set *equally well*.
!split
===== What is a good model? Can we define it? =====
The central question is this: what leads us to say that a model is correct or
optimal for a given data set? To make the model inference problem well posed, i.e.,
to guarantee that there is a unique optimal model for the given data, we need to
impose additional assumptions or restrictions on the class of models considered. To
this end, we should not be looking for just any model that can describe the data.
Instead, we should look for a _model_ $M$ that is the best among a restricted class
of models. In addition, to make the model inference problem computationally
tractable, we need to specify how restricted the class of models needs to be. A
common strategy is to start
with the simplest possible class of models that is just necessary to describe the data
or solve the problem at hand. More precisely, the model class should be rich enough
to contain at least one model that can fit the data to a desired accuracy and yet be
restricted enough that it is relatively simple to find the best model for the given data.
Thus, the most popular strategy is to start from the
simplest class of models and increase the complexity of the models only when the
simpler models become inadequate. For instance, if we work with a regression problem to fit a set of sample points, one
may first try the simplest class of models, namely linear models, followed obviously by more complex models.
How to evaluate which model fits best the data is something we will come back to over and over again in these set of lectures.
!split
===== Practicalities, choice of programming language and other computational issues =====
!split
===== Choice of Programming Language =====
Python plays nowadays a central role in the development of machine
learning techniques and tools for data analysis. In particular, seen
@@ -196,39 +324,13 @@ Jupyter notebook framework with the possibility to run _R_ codes or
compiled programs written in C++, and much more made our choice of
programming language for this series of lectures easy. However,
since the focus here is not only on using existing Python libraries such
as _Scikit-Learn_ or _Tensorflow_, but also on developing your own
as _Scikit-Learn_, _Tensorflow_ and _Pytorch_, but also on developing your own
algorithms and codes, we will as far as possible present many of these
algorithms either as a Python codes or C++ or Fortran (or other languages) codes.
The reason we also focus on compiled languages like C++ (or
Fortran), is that Python is still notoriously slow when we do not
utilize highly streamlined computational libraries like
"Lapack":"http://www.netlib.org/lapack/" or other numerical libraries
written in compiled languages (many of these libraries are written in
Fortran). Although a project like "Numba":"https://numba.pydata.org/"
holds great promise for speeding up the unrolling of lengthy loops, C++
and Fortran are presently still the performance winners. Numba gives
you potentially the power to speed up your applications with high
performance functions written directly in Python. In particular,
array-oriented and math-heavy Python code can achieve similar
performance to C, C++ and Fortran. However, even with these speed-ups,
for codes involving heavy Markov Chain Monte Carlo analyses and
optimizations of cost functions, C++/C or Fortran codes tend to
outperform Python codes.
Presently thus, the community tends to let
code written in C++/C or Fortran do the heavy duty numerical
number crunching and leave the post-analysis of the data to the above
mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen
the changes during the last decade, the above situation may change swiftly in the not too distant future.
Many of the examples we discuss in this series of lectures come with
existing data files or provide code examples which produce the data to
be analyzed. Most of the applications we will discuss deal with
small data sets (less than a terabyte of information) and can easily
be analyzed and tested on standard off the shelf laptops you find in general
stores.
!split
===== Data handling, machine learning and ethical aspects =====
In most of the cases we will study, we will either generate the data
@@ -324,3 +426,5 @@ society.