mhjensen
479 lines
23 KiB
HTML
479 lines
23 KiB
HTML
<!--
|
|
Automatically generated HTML file from DocOnce source
|
|
(https://github.com/hplgit/doconce/)
|
|
-->
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
|
<meta name="description" content="Introduction to Applied Data Analysis and Machine Learning">
|
|
|
|
<title>Introduction to Applied Data Analysis and Machine Learning</title>
|
|
|
|
<!-- Bootstrap style: bootstrap -->
|
|
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
|
<!-- not necessary
|
|
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
|
-->
|
|
|
|
<style type="text/css">
|
|
|
|
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
|
.dropdown-menu {
|
|
height: auto;
|
|
max-height: 400px;
|
|
overflow-x: hidden;
|
|
}
|
|
|
|
/* Adds an invisible element before each target to offset for the navigation
|
|
bar */
|
|
.anchor::before {
|
|
content:"";
|
|
display:block;
|
|
height:50px; /* fixed header height for style bootstrap */
|
|
margin:-50px 0 0; /* negative fixed header height */
|
|
}
|
|
</style>
|
|
|
|
|
|
</head>
|
|
|
|
<!-- tocinfo
|
|
{'highest level': 2,
|
|
'sections': [('Introduction', 2, None, '___sec0'),
|
|
('Learning outcomes', 2, None, '___sec1'),
|
|
('Types of Machine Learning', 2, None, '___sec2'),
|
|
('Choice of programming language', 2, None, '___sec3'),
|
|
('Data handling, machine learning and ethical aspects',
|
|
2,
|
|
None,
|
|
'___sec4')]}
|
|
end of tocinfo -->
|
|
|
|
<body>
|
|
|
|
|
|
<!-- Bootstrap navigation bar -->
|
|
<div class="navbar navbar-default navbar-fixed-top">
|
|
<div class="navbar-header">
|
|
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
</button>
|
|
<a class="navbar-brand" href="Introduction-bs.html">Introduction to Applied Data Analysis and Machine Learning</a>
|
|
</div>
|
|
|
|
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
|
<ul class="nav navbar-nav navbar-right">
|
|
<li class="dropdown">
|
|
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
|
<ul class="dropdown-menu">
|
|
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Introduction</a></li>
|
|
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Learning outcomes</a></li>
|
|
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Types of Machine Learning</a></li>
|
|
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Choice of programming language</a></li>
|
|
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Data handling, machine learning and ethical aspects</a></li>
|
|
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
</div> <!-- end of navigation bar -->
|
|
|
|
<div class="container">
|
|
|
|
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
|
|
|
<!-- ------------------- main content ---------------------- -->
|
|
|
|
|
|
|
|
<div class="jumbotron">
|
|
<center><h1>Introduction to Applied Data Analysis and Machine Learning</h1></center> <!-- document title -->
|
|
|
|
<p>
|
|
<!-- author(s): Morten Hjorth-Jensen -->
|
|
|
|
<center>
|
|
<b>Morten Hjorth-Jensen</b> [1, 2]
|
|
</center>
|
|
|
|
<p>
|
|
<!-- institution(s) -->
|
|
|
|
<center>[1] <b>Department of Physics, University of Oslo</b></center>
|
|
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
|
<br>
|
|
<p>
|
|
<center><h4>Jun 10, 2019</h4></center> <!-- date -->
|
|
<br>
|
|
<p>
|
|
</div> <!-- end jumbotron -->
|
|
|
|
<h2 id="___sec0" class="anchor">Introduction </h2>
|
|
|
|
<p>
|
|
During the last two decades there has been a swift and amazing
|
|
development of Machine Learning techniques and algorithms that impact
|
|
many areas in not only Science and Technology but also the Humanities,
|
|
Social Sciences, Medicine, Law, indeed, almost all possible
|
|
disciplines. The applications are incredibly many, from self-driving
|
|
cars to solving high-dimensional differential equations or complicated
|
|
quantum mechanical many-body problems. Machine Learning is perceived
|
|
by many as one of the main disruptive techniques nowadays.
|
|
|
|
<p>
|
|
Statistics, Data science and Machine Learning form important
|
|
fields of research in modern science. They describe how to learn and
|
|
make predictions from data, as well as allowing us to extract
|
|
important correlations about physical process and the underlying laws
|
|
of motion in large data sets. The latter, big data sets, appear
|
|
frequently in essentially all disciplines, from the traditional
|
|
Science, Technology, Mathematics and Engineering fields to Life
|
|
Science, Law, education research, the Humanities and the Social
|
|
Sciences.
|
|
|
|
<p>
|
|
It has become more
|
|
and more common to see research projects on big data in for example
|
|
the Social Sciences where extracting patterns from complicated survey
|
|
data is one of many research directions. Having a solid grasp of data
|
|
analysis and machine learning is thus becoming central to scientific
|
|
computing in many fields, and competences and skills within the fields
|
|
of machine learning and scientific computing are nowadays strongly
|
|
requested by many potential employers. The latter cannot be
|
|
overstated, familiarity with machine learning has almost become a
|
|
prerequisite for many of the most exciting employment opportunities,
|
|
whether they are in bioinformatics, life science, physics or finance,
|
|
in the private or the public sector. This author has had several
|
|
students or met students who have been hired recently based on their
|
|
skills and competences in scientific computing and data science, often
|
|
with marginal knowledge of machine learning.
|
|
|
|
<p>
|
|
Machine learning is a subfield of computer science, and is closely
|
|
related to computational statistics. It evolved from the study of
|
|
pattern recognition in artificial intelligence (AI) research, and has
|
|
made contributions to AI tasks like computer vision, natural language
|
|
processing and speech recognition. Many of the methods we will study are also
|
|
strongly rooted in basic mathematics and physics research.
|
|
|
|
<p>
|
|
Ideally, machine learning represents the science of giving computers
|
|
the ability to learn without being explicitly programmed. The idea is
|
|
that there exist generic algorithms which can be used to find patterns
|
|
in a broad class of data sets without having to write code
|
|
specifically for each problem. The algorithm will build its own logic
|
|
based on the data. You should however always keep in mind that
|
|
machines and algorithms are to a large extent developed by humans. The
|
|
insights and knowledge we have about a specific system, play a central
|
|
role when we develop a specific machine learning algorithm.
|
|
|
|
<p>
|
|
Machine learning is an extremely rich field, in spite of its young
|
|
age. The increases we have seen during the last three decades in
|
|
computational capabilities have been followed by developments of
|
|
methods and techniques for analyzing and handling large date sets,
|
|
relying heavily on statistics, computer science and mathematics. The
|
|
field is rather new and developing rapidly. Popular software packages
|
|
written in Python for machine learning like
|
|
<a href="http://scikit-learn.org/stable/" target="_self">Scikit-learn</a>,
|
|
<a href="https://www.tensorflow.org/" target="_self">Tensorflow</a>,
|
|
<a href="http://pytorch.org/" target="_self">PyTorch</a> and <a href="https://keras.io/" target="_self">Keras</a>, all
|
|
freely available at their respective GitHub sites, encompass
|
|
communities of developers in the thousands or more. And the number of
|
|
code developers and contributors keeps increasing. Not all the
|
|
algorithms and methods can be given a rigorous mathematical
|
|
justification, opening up thereby large rooms for experimenting and
|
|
trial and error and thereby exciting new developments. However, a
|
|
solid command of linear algebra, multivariate theory, probability
|
|
theory, statistical data analysis, understanding errors and Monte
|
|
Carlo methods are central elements in a proper understanding of many
|
|
of algorithms and methods we will discuss.
|
|
|
|
<p>
|
|
<!-- !split -->
|
|
|
|
<h2 id="___sec1" class="anchor">Learning outcomes </h2>
|
|
|
|
<p>
|
|
These sets of lectures aim at giving you an overview of central aspects of
|
|
statistical data analysis as well as some of the central algorithms
|
|
used in machine learning. We will introduce a variety of central
|
|
algorithms and methods essential for studies of data analysis and
|
|
machine learning.
|
|
|
|
<p>
|
|
Hands-on projects and experimenting with data and algorithms plays a central role in
|
|
these lectures, and our hope is, through the various
|
|
projects and exercises, to expose you to fundamental
|
|
research problems in these fields, with the aim to reproduce state of
|
|
the art scientific results. You will learn to develop and
|
|
structure codes for studying these systems, get acquainted with
|
|
computing facilities and learn to handle large scientific projects. A
|
|
good scientific and ethical conduct is emphasized throughout the
|
|
course. More specifically, you will
|
|
|
|
<ol>
|
|
<li> Learn about basic data analysis, Bayesian statistics, Monte Carlo methods, data optimization and machine learning;</li>
|
|
<li> Be capable of extending the acquired knowledge to other systems and cases;</li>
|
|
<li> Have an understanding of central algorithms used in data analysis and machine learning;</li>
|
|
<li> Gain knowledge of central aspects of Monte Carlo methods, Markov chains, Gibbs samplers and their possible applications, from numerical integration to simulation of stock markets;</li>
|
|
<li> Understand methods for regression and classification;</li>
|
|
<li> Learn about neural network, genetic algorithms and Boltzmann machines;</li>
|
|
<li> Work on numerical projects to illustrate the theory. The projects play a central role and you are expected to know modern programming languages like Python or C++, in addition to a basic knowledge of linear algebra (typically taught during the first one or two years of undergraduate studies).</li>
|
|
</ol>
|
|
|
|
There are several topics we will cover here, spanning from
|
|
statistical data analysis and its basic concepts such as expectation
|
|
values, variance, covariance, correlation functions and errors, via
|
|
well-known probability distribution functions like the uniform
|
|
distribution, the binomial distribution, the Poisson distribution and
|
|
simple and multivariate normal distributions to central elements of
|
|
Bayesian statistics and modeling. We will also remind the reader about
|
|
central elements from linear algebra and standard methods based on
|
|
linear algebra used to optimize (minimize) functions (the family of gradient descent methods)
|
|
and the Singular-value decomposition and
|
|
least square methods for parameterizing data.
|
|
|
|
<p>
|
|
We will also cover Monte Carlo methods, Markov chains, well-known
|
|
algorithms for sampling stochastic events like the Metropolis-Hastings
|
|
and Gibbs sampling methods. An important aspect of all our
|
|
calculations is a proper estimation of errors. Here we will also
|
|
discuss famous resampling techniques like the blocking, the bootstrapping
|
|
and the jackknife methods and the infamous bias-variance tradeoff.
|
|
|
|
<p>
|
|
The second part of the material covers several algorithms used in
|
|
machine learning.
|
|
|
|
<h2 id="___sec2" class="anchor">Types of Machine Learning </h2>
|
|
|
|
<p>
|
|
The approaches to machine learning are many, but are often split into
|
|
two main categories. In <em>supervised learning</em> we know the answer to a
|
|
problem, and let the computer deduce the logic behind it. On the other
|
|
hand, <em>unsupervised learning</em> is a method for finding patterns and
|
|
relationship in data sets without any prior knowledge of the system.
|
|
Some authours also operate with a third category, namely
|
|
<em>reinforcement learning</em>. This is a paradigm of learning inspired by
|
|
behavioral psychology, where learning is achieved by trial-and-error,
|
|
solely from rewards and punishment.
|
|
|
|
<p>
|
|
Another way to categorize machine learning tasks is to consider the
|
|
desired output of a system. Some of the most common tasks are:
|
|
|
|
<ul>
|
|
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
|
|
<li> Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.</li>
|
|
<li> Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.</li>
|
|
</ul>
|
|
|
|
The methods we cover have three main topics in common, irrespective of
|
|
whether we deal with supervised or unsupervised learning. The first
|
|
ingredient is normally our data set (which can be subdivided into
|
|
training and test data), the second item is a model which is normally
|
|
a function of some parameters. The model reflects our knowledge of
|
|
the system (or lack thereof). As an example, if we know that our data
|
|
show a behavior similar to what would be predicted by a polynomial,
|
|
fitting our data to a polynomial of some degree would then determin
|
|
our model.
|
|
|
|
<p>
|
|
The last ingredient is a so-called <b>cost</b>
|
|
function which allows us to present an estimate on how good our model
|
|
is in reproducing the data it is supposed to train.
|
|
|
|
<p>
|
|
Here we will build our machine learning approach on elements of the
|
|
statistical foundation discussed above, with elements from data
|
|
analysis, stochastic processes etc. We will discuss the following
|
|
machine learning algorithms
|
|
|
|
<ol>
|
|
<li> Linear regression and its variants</li>
|
|
<li> Decision tree algorithms, from single trees to random forests</li>
|
|
<li> Bayesian statistics and regression</li>
|
|
<li> Support vector machines and finally various variants of</li>
|
|
<li> Artifical neural networks and deep learning, including convolutional neural networks and Bayesian neural networks</li>
|
|
<li> Networks for unsupervised learning using for example reduced Boltzmann machines.</li>
|
|
</ol>
|
|
|
|
<h2 id="___sec3" class="anchor">Choice of programming language </h2>
|
|
|
|
<p>
|
|
Python plays nowadays a central role in the development of machine
|
|
learning techniques and tools for data analysis. In particular, seen
|
|
the wealth of machine learning and data analysis libraries written in
|
|
Python, easy to use libraries with immediate visualization(and not the
|
|
least impressive galleries of existing examples), the popularity of the
|
|
Jupyter notebook framework with the possibility to run <b>R</b> codes or
|
|
compiled programs written in C++, and much more made our choice of
|
|
programming language for this series of lectures easy. However,
|
|
since the focus here is not only on using existing Python libraries such
|
|
as <b>Scikit-Learn</b> or <b>Tensorflow</b>, but also on developing your own
|
|
algorithms and codes, we will as far as possible present many of these
|
|
algorithms either as a Python codes or C++ or Fortran (or other languages) codes.
|
|
|
|
<p>
|
|
The reason we also focus on compiled languages like C++ (or
|
|
Fortran), is that Python is still notoriously slow when we do not
|
|
utilize highly streamlined computational libraries like
|
|
<a href="http://www.netlib.org/lapack/" target="_self">Lapack</a> or other numerical libraries
|
|
written in compiled languages (many of these libraries are written in
|
|
Fortran). Although a project like <a href="https://numba.pydata.org/" target="_self">Numba</a>
|
|
holds great promise for speeding up the unrolling of lengthy loops, C++
|
|
and Fortran are presently still the performance winners. Numba gives
|
|
you potentially the power to speed up your applications with high
|
|
performance functions written directly in Python. In particular,
|
|
array-oriented and math-heavy Python code can achieve similar
|
|
performance to C, C++ and Fortran. However, even with these speed-ups,
|
|
for codes involving heavy Markov Chain Monte Carlo analyses and
|
|
optimizations of cost functions, C++/C or Fortran codes tend to
|
|
outperform Python codes.
|
|
|
|
<p>
|
|
Presently thus, the community tends to let
|
|
code written in C++/C or Fortran do the heavy duty numerical
|
|
number crunching and leave the post-analysis of the data to the above
|
|
mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen
|
|
the changes during the last decade, the above situation may change swiftly in the not too distant future.
|
|
|
|
<p>
|
|
Many of the examples we discuss in this series of lectures come with
|
|
existing data files or provide code examples which produce the data to
|
|
be analyzed. Most of the applications we will discuss deal with
|
|
small data sets (less than a terabyte of information) and can easily
|
|
be analyzed and tested on standard off the shelf laptops you find in general
|
|
stores.
|
|
|
|
<h2 id="___sec4" class="anchor">Data handling, machine learning and ethical aspects </h2>
|
|
|
|
<p>
|
|
In most of the cases we will study, we will either generate the data
|
|
to analyze ourselves (both for supervised learning and unsupervised
|
|
learning) or we will recur again and again to data present in say
|
|
<b>Scikit-Learn</b> or <b>Tensorflow</b>. Many of the examples we end up
|
|
dealing with are from a privacy and data protection point of view,
|
|
rather inoccuous and boring results of numerical
|
|
calculations. However, this does not hinder us from developing a sound
|
|
ethical attitude to the data we use, how we analyze the data and how
|
|
we handle the data.
|
|
|
|
<p>
|
|
The most immediate and simplest possible ethical aspects deal with our
|
|
approach to the scientific process. Nowadays, with version control
|
|
software like <a href="https://git-scm.com/" target="_self">Git</a> and various online
|
|
repositories like <a href="https://github.com/" target="_self">Github</a>,
|
|
<a href="https://about.gitlab.com/" target="_self">Gitlab</a> etc, we can easily make our codes
|
|
and data sets we have used, freely and easily accessible to a wider
|
|
community. This helps us almost automagically in making our science
|
|
reproducible. The large open-source development communities involved
|
|
in say <a href="http://scikit-learn.org/stable/" target="_self">Scikit-Learn</a>,
|
|
<a href="https://www.tensorflow.org/" target="_self">Tensorflow</a>,
|
|
<a href="http://pytorch.org/" target="_self">PyTorch</a> and <a href="https://keras.io/" target="_self">Keras</a>, are
|
|
all excellent examples of this. The codes can be tested and improved
|
|
upon continuosly, helping thereby our scientific community at large in
|
|
developing data analysis and machine learning tools. It is much
|
|
easier today to gain traction and acceptance for making your science
|
|
reproducible. From a societal stand, this is an important element
|
|
since many of the developers are employees of large public institutions like
|
|
universities and research labs. Our fellow taxpayers do deserve to get
|
|
something back for their bucks.
|
|
|
|
<p>
|
|
However, this more mechanical aspect of the ethics of science (in
|
|
particular the reproducibility of scientific results) is something
|
|
which is obvious and everybody should do so as part of the dialectics of
|
|
science. The fact that many scientists are not willing to share their codes or
|
|
data is detrimental to the scientific discourse.
|
|
|
|
<p>
|
|
Before we proceed, we should add a disclaimer. Even though
|
|
we may dream of computers developing some kind of higher learning
|
|
capabilities, at the end (even if the artificial intelligence
|
|
community keeps touting our ears full of fancy futuristic avenues), it is we, yes you reading these lines,
|
|
who end up constructing and instructing, via various algorithms, the
|
|
machine learning approaches. Self-driving cars for example, rely on sofisticated
|
|
programs which take into account all possible situations a car can
|
|
encounter. In addition, extensive usage of training data from GPS
|
|
information, maps etc, are typically fed into the software for
|
|
self-driving cars. Adding to this various sensors and cameras that
|
|
feed information to the programs, there are zillions of ethical issues
|
|
which arise from this.
|
|
|
|
<p>
|
|
For self-driving cars, where basically many of the standard machine
|
|
learning algorithms discussed here enter into the codes, at a certain
|
|
stage we have to make choices. Yes, we , the lads and lasses who wrote
|
|
a program for a specific brand of a self-driving car. As an example,
|
|
all carmakers have as their utmost priority the security of the
|
|
driver and the accompanying passengers. A famous European carmaker, which is
|
|
one of the leaders in the market of self-driving cars, had <b>if</b>
|
|
statements of the following type: suppose there are two obstacles in
|
|
front of you and you cannot avoid to collide with one of them. One of
|
|
the obstacles is a monstertruck while the other one is a kindergarten
|
|
class trying to cross the road. The self-driving car algo would then
|
|
opt for the hitting the small folks instead of the monstertruck, since
|
|
the likelihood of surving a collision with our future citizens, is
|
|
much higher.
|
|
|
|
<p>
|
|
This leads to serious ethical aspects. Why should we
|
|
opt for such an option? Who decides and who is entitled to make such
|
|
choices? Keep in mind that many of the algorithms you will encounter in
|
|
this series of lectures or hear about later, are indeed based on
|
|
simple programming instructions. And you are very likely to be one of
|
|
the people who may end up writing such a code. Thus, developing a
|
|
sound ethical attitude to what we do, an approach well beyond the
|
|
simple mechanistic one of making our science available and
|
|
reproducible, is much needed. The example of the self-driving cars is
|
|
just one of infinitely many cases where we have to make choices. When
|
|
you analyze data on economic inequalities, who guarantees that you are
|
|
not weighting some data in a particular way, perhaps because you dearly want a
|
|
specific conclusion which may support your political views?
|
|
|
|
<p>
|
|
We do not have the answers here, nor will we venture into a deeper
|
|
discussions of these aspects, but we want you think over these topics
|
|
in a more overarching way. A statistical data analysis with its dry
|
|
numbers and graphs meant to guide the eye, does not necessarily
|
|
reflect the truth, whatever that is. As a scientist, and after a
|
|
university education, you are supposedly a better citizen, with an
|
|
improved critical view and understanding of the scientific method, and
|
|
perhaps some deeper understanding of the ethics of science at
|
|
large. Use these insights. Be a critical citizen. You owe it to our
|
|
society.
|
|
|
|
<p>
|
|
To do: Add references and acknowledgements
|
|
|
|
<!-- ------------------- end of main content --------------- -->
|
|
|
|
</div> <!-- end container -->
|
|
<!-- include javascript, jQuery *first* -->
|
|
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
|
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
|
|
|
<!-- Bootstrap footer
|
|
<footer>
|
|
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
|
</footer>
|
|
-->
|
|
|
|
|
|
<center style="font-size:80%">
|
|
<!-- copyright --> © 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
|
|
</center>
|
|
|
|
|
|
</body>
|
|
</html>
|
|
|
|
|