Updated the introduction
This commit is contained in:
mhjensen
@@ -447,7 +447,7 @@ factor in front of \( x \) and the normal distribution. Try to change the
|
||||
function \( y \) to
|
||||
|
||||
$$
|
||||
y = 10x+0.01N(0,1),
|
||||
y = 10x+0.01 \times N(0,1),
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -459,7 +459,7 @@ long run. Here we have only defined the training data and our model, and
|
||||
have not discussed a more rigorous approach to the <b>cost</b> function.
|
||||
|
||||
<p>
|
||||
We need more rigorous criteria in defining whether we have succeed or
|
||||
We need more rigorous criteria in defining whether we have succeeded or
|
||||
not in modeling our training data. You will be surprised to see that
|
||||
many scientists seldomly venture beyond this 'by the eye' approach. A
|
||||
standard approach for the <em>cost</em> function is the so-called \( \chi^2 \)
|
||||
|
||||
@@ -415,7 +415,7 @@ function \( y \) to
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
y = 10x+0.01N(0,1),
|
||||
y = 10x+0.01 \times N(0,1),
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -428,7 +428,7 @@ long run. Here we have only defined the training data and our model, and
|
||||
have not discussed a more rigorous approach to the <b>cost</b> function.
|
||||
|
||||
<p>
|
||||
We need more rigorous criteria in defining whether we have succeed or
|
||||
We need more rigorous criteria in defining whether we have succeeded or
|
||||
not in modeling our training data. You will be surprised to see that
|
||||
many scientists seldomly venture beyond this 'by the eye' approach. A
|
||||
standard approach for the <em>cost</em> function is the so-called \( \chi^2 \)
|
||||
|
||||
@@ -404,7 +404,7 @@ factor in front of \( x \) and the normal distribution. Try to change the
|
||||
function \( y \) to
|
||||
|
||||
$$
|
||||
y = 10x+0.01N(0,1),
|
||||
y = 10x+0.01 \times N(0,1),
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -416,7 +416,7 @@ long run. Here we have only defined the training data and our model, and
|
||||
have not discussed a more rigorous approach to the <b>cost</b> function.
|
||||
|
||||
<p>
|
||||
We need more rigorous criteria in defining whether we have succeed or
|
||||
We need more rigorous criteria in defining whether we have succeeded or
|
||||
not in modeling our training data. You will be surprised to see that
|
||||
many scientists seldomly venture beyond this 'by the eye' approach. A
|
||||
standard approach for the <em>cost</em> function is the so-called \( \chi^2 \)
|
||||
|
||||
@@ -409,7 +409,7 @@ factor in front of \( x \) and the normal distribution. Try to change the
|
||||
function \( y \) to
|
||||
|
||||
$$
|
||||
y = 10x+0.01N(0,1),
|
||||
y = 10x+0.01 \times N(0,1),
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -421,7 +421,7 @@ long run. Here we have only defined the training data and our model, and
|
||||
have not discussed a more rigorous approach to the <b>cost</b> function.
|
||||
|
||||
<p>
|
||||
We need more rigorous criteria in defining whether we have succeed or
|
||||
We need more rigorous criteria in defining whether we have succeeded or
|
||||
not in modeling our training data. You will be surprised to see that
|
||||
many scientists seldomly venture beyond this 'by the eye' approach. A
|
||||
standard approach for the <em>cost</em> function is the so-called \( \chi^2 \)
|
||||
|
||||
@@ -268,7 +268,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"y = 10x+0.01N(0,1),\n",
|
||||
"y = 10x+0.01 \\times N(0,1),\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -283,7 +283,7 @@
|
||||
"long run. Here we have only defined the training data and our model, and \n",
|
||||
"have not discussed a more rigorous approach to the **cost** function.\n",
|
||||
"\n",
|
||||
"We need more rigorous criteria in defining whether we have succeed or\n",
|
||||
"We need more rigorous criteria in defining whether we have succeeded or\n",
|
||||
"not in modeling our training data. You will be surprised to see that\n",
|
||||
"many scientists seldomly venture beyond this 'by the eye' approach. A\n",
|
||||
"standard approach for the *cost* function is the so-called $\\chi^2$\n",
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -6,9 +6,9 @@ Automatically generated HTML file from DocOnce source
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Representing data">
|
||||
<meta name="description" content="Introduction to Applied Data Analysis and Machine Learning">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Representing data</title>
|
||||
<title>Introduction to Applied Data Analysis and Machine Learning</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
@@ -43,13 +43,11 @@ Automatically generated HTML file from DocOnce source
|
||||
'sections': [('Introduction', 2, None, '___sec0'),
|
||||
('Learning outcomes', 2, None, '___sec1'),
|
||||
('Types of Machine Learning', 2, None, '___sec2'),
|
||||
('Why this text?', 2, None, '___sec3'),
|
||||
('Choice of programming language', 2, None, '___sec4'),
|
||||
('Choice of programming language', 2, None, '___sec3'),
|
||||
('Data handling, machine learning and ethical aspects',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Acknowledgements', 2, None, '___sec6')]}
|
||||
'___sec4')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -63,7 +61,7 @@ end of tocinfo -->
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="Introduction-bs.html">Data Analysis and Machine Learning: Representing data</a>
|
||||
<a class="navbar-brand" href="Introduction-bs.html">Introduction to Applied Data Analysis and Machine Learning</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
@@ -74,10 +72,8 @@ end of tocinfo -->
|
||||
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Introduction</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Learning outcomes</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Types of Machine Learning</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Why this text?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Choice of programming language</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">Data handling, machine learning and ethical aspects</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">Acknowledgements</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Choice of programming language</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Data handling, machine learning and ethical aspects</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -95,7 +91,7 @@ end of tocinfo -->
|
||||
|
||||
|
||||
<div class="jumbotron">
|
||||
<center><h1>Data Analysis and Machine Learning: Representing data</h1></center> <!-- document title -->
|
||||
<center><h1>Introduction to Applied Data Analysis and Machine Learning</h1></center> <!-- document title -->
|
||||
|
||||
<p>
|
||||
<!-- author(s): Morten Hjorth-Jensen -->
|
||||
@@ -111,14 +107,11 @@ end of tocinfo -->
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>May 22, 2018</h4></center> <!-- date -->
|
||||
<center><h4>May 28, 2018</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- potential-jumbotron-button -->
|
||||
</div> <!-- end jumbotron -->
|
||||
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec0" class="anchor">Introduction </h2>
|
||||
|
||||
<p>
|
||||
@@ -126,57 +119,68 @@ Statistics, data science and machine learning form important fields of
|
||||
research in modern science. They describe how to learn and make
|
||||
predictions from data, as well as allowing us to extract important
|
||||
correlations about physical process and the underlying laws of motion
|
||||
in large data sets. The latter, big data sets, appear
|
||||
frequently in essentially all disciplines, from the traditional Science,
|
||||
Technology, Mathematics and Engineering fields to Life Science, Law, education research,
|
||||
the Humanities and
|
||||
the Social Sciences. It has become more and more common to see
|
||||
research projects on big data in for example the Social
|
||||
Sciences where extracting patterns from complicated survey data is one of many research directions.
|
||||
Having a solid grasp of data analysis and machine learning
|
||||
is thus becoming central to scientific computing in many
|
||||
fields, and competences and skills within the fields of machine learning
|
||||
and scientific computing are nowadays strongly requested by many
|
||||
potential employers. The latter cannot be overstated, familiarity with
|
||||
machine learning has almost become a prerequisite for many of the most
|
||||
exciting employment opportunities, whether they are in bioinformatics,
|
||||
life science, physics or finance, in the private or the public
|
||||
sector. This author has had several students or met students who have
|
||||
been hired recently based on their skills and competences in
|
||||
scientific computing and data science, often with marginal knowledge
|
||||
of machine learning.
|
||||
in large data sets. The latter, big data sets, appear frequently in
|
||||
essentially all disciplines, from the traditional Science, Technology,
|
||||
Mathematics and Engineering fields to Life Science, Law, education
|
||||
research, the Humanities and the Social Sciences.
|
||||
|
||||
<p>
|
||||
It has become more
|
||||
and more common to see research projects on big data in for example
|
||||
the Social Sciences where extracting patterns from complicated survey
|
||||
data is one of many research directions. Having a solid grasp of data
|
||||
analysis and machine learning is thus becoming central to scientific
|
||||
computing in many fields, and competences and skills within the fields
|
||||
of machine learning and scientific computing are nowadays strongly
|
||||
requested by many potential employers. The latter cannot be
|
||||
overstated, familiarity with machine learning has almost become a
|
||||
prerequisite for many of the most exciting employment opportunities,
|
||||
whether they are in bioinformatics, life science, physics or finance,
|
||||
in the private or the public sector. This author has had several
|
||||
students or met students who have been hired recently based on their
|
||||
skills and competences in scientific computing and data science, often
|
||||
with marginal knowledge of machine learning.
|
||||
|
||||
<p>
|
||||
Machine learning is a subfield of computer science, and is closely
|
||||
related to computational statistics. It evolved from the study of
|
||||
pattern recognition in artificial intelligence (AI) research, and has
|
||||
made contributions to AI tasks like computer vision, natural language
|
||||
processing and speech recognition.
|
||||
Machine learning represents the
|
||||
science of giving computers the ability to learn without being
|
||||
explicitly programmed. The idea is that there exist generic
|
||||
algorithms which can be used to find patterns in a broad class of data
|
||||
sets without having to write code specifically for each problem. The
|
||||
algorithm will build its own logic based on the data.
|
||||
processing and speech recognition. Many of the methods we will study are also
|
||||
strongly rooted in basic mathematics and physics research.
|
||||
|
||||
<p>
|
||||
Machine learning is an extremely rich field, in spite of its young age. The
|
||||
increases we have seen during the last three decades in computational
|
||||
capabilities have been followed by developments of methods and
|
||||
techniques for analyzing and handling large date sets, relying heavily
|
||||
on statistics, computer science and mathematics. The field is rather
|
||||
new and developing rapidly. Popular software packages written in
|
||||
Python for machine learning like <a href="http://scikit-learn.org/stable/" target="_self">Scikit-learn</a>, <a href="https://www.tensorflow.org/" target="_self">Tensorflow</a>,
|
||||
<a href="http://pytorch.org/" target="_self">PyTorch</a> and <a href="https://keras.io/" target="_self">Keras</a>, all freely available at their respective GitHub sites,
|
||||
encompass communities of developers in the thousands or more. And the number
|
||||
of code developers and contributors keeps increasing. Not all the
|
||||
Ideally, machine learning represents the science of giving computers
|
||||
the ability to learn without being explicitly programmed. The idea is
|
||||
that there exist generic algorithms which can be used to find patterns
|
||||
in a broad class of data sets without having to write code
|
||||
specifically for each problem. The algorithm will build its own logic
|
||||
based on the data. You should however always keep in mind that
|
||||
machines and algorithms are to a large extent developed by humans. The
|
||||
insights and knowledge we have about a specific system, play a central
|
||||
role when we develop a specific machine learning algorithm.
|
||||
|
||||
<p>
|
||||
Machine learning is an extremely rich field, in spite of its young
|
||||
age. The increases we have seen during the last three decades in
|
||||
computational capabilities have been followed by developments of
|
||||
methods and techniques for analyzing and handling large date sets,
|
||||
relying heavily on statistics, computer science and mathematics. The
|
||||
field is rather new and developing rapidly. Popular software packages
|
||||
written in Python for machine learning like
|
||||
<a href="http://scikit-learn.org/stable/" target="_self">Scikit-learn</a>,
|
||||
<a href="https://www.tensorflow.org/" target="_self">Tensorflow</a>,
|
||||
<a href="http://pytorch.org/" target="_self">PyTorch</a> and <a href="https://keras.io/" target="_self">Keras</a>, all
|
||||
freely available at their respective GitHub sites, encompass
|
||||
communities of developers in the thousands or more. And the number of
|
||||
code developers and contributors keeps increasing. Not all the
|
||||
algorithms and methods can be given a rigorous mathematical
|
||||
justification, opening up thereby large rooms for experimenting
|
||||
and trial and error and thereby exciting new developments.
|
||||
However, a solid command of linear algebra, multivariate theory,
|
||||
probability theory, statistical data analysis,
|
||||
understanding errors and Monte Carlo methods are central elements in a proper understanding of many of
|
||||
algorithms and methods we will discuss.
|
||||
justification, opening up thereby large rooms for experimenting and
|
||||
trial and error and thereby exciting new developments. However, a
|
||||
solid command of linear algebra, multivariate theory, probability
|
||||
theory, statistical data analysis, understanding errors and Monte
|
||||
Carlo methods are central elements in a proper understanding of many
|
||||
of algorithms and methods we will discuss.
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
@@ -184,7 +188,7 @@ algorithms and methods we will discuss.
|
||||
<h2 id="___sec1" class="anchor">Learning outcomes </h2>
|
||||
|
||||
<p>
|
||||
These lectures aim at giving you an overview of central aspects of
|
||||
These setsof lectures aim at giving you an overview of central aspects of
|
||||
statistical data analysis as well as some of the central algorithms
|
||||
used in machine learning. We will introduce a variety of central
|
||||
algorithms and methods essential for studies of data analysis and
|
||||
@@ -235,23 +239,22 @@ and jackknife methods.
|
||||
The second part of the material covers several algorithms used in
|
||||
machine learning.
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec2" class="anchor">Types of Machine Learning </h2>
|
||||
|
||||
<p>
|
||||
The approaches to machine learning are many, but are often split into two main categories.
|
||||
In <em>supervised learning</em> we know the answer to a problem,
|
||||
and let the computer deduce the logic behind it. On the other hand, <em>unsupervised learning</em>
|
||||
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
|
||||
Some authours also operate with a third category, namely <em>reinforcement learning</em>. This is a paradigm
|
||||
of learning inspired by behavioral psychology, where learning is achieved by trial-and-error,
|
||||
The approaches to machine learning are many, but are often split into
|
||||
two main categories. In <em>supervised learning</em> we know the answer to a
|
||||
problem, and let the computer deduce the logic behind it. On the other
|
||||
hand, <em>unsupervised learning</em> is a method for finding patterns and
|
||||
relationship in data sets without any prior knowledge of the system.
|
||||
Some authours also operate with a third category, namely
|
||||
<em>reinforcement learning</em>. This is a paradigm of learning inspired by
|
||||
behavioral psychology, where learning is achieved by trial-and-error,
|
||||
solely from rewards and punishment.
|
||||
|
||||
<p>
|
||||
Another way to categorize machine learning tasks is to consider the desired output of a system.
|
||||
Some of the most common tasks are:
|
||||
Another way to categorize machine learning tasks is to consider the
|
||||
desired output of a system. Some of the most common tasks are:
|
||||
|
||||
<ul>
|
||||
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
|
||||
@@ -261,10 +264,14 @@ Some of the most common tasks are:
|
||||
|
||||
The methods we cover have three main topics in common, irrespective of
|
||||
whether we deal with supervised or unsupervised learning. The first
|
||||
ingredient is normally our data set, the second is a model which is
|
||||
normally a function of some parameters. The last ingredient is a
|
||||
so-called <b>cost</b> function which allows us to present an estimate on
|
||||
how good our model is in reproducing the data it is supposed to train.
|
||||
ingredient is normally our data set (which can be subdivided into
|
||||
training and test data), the second item is a model which is normally a
|
||||
function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.
|
||||
|
||||
<p>
|
||||
The last ingredient is a so-called <b>cost</b>
|
||||
function which allows us to present an estimate on how good our model
|
||||
is in reproducing the data it is supposed to train.
|
||||
|
||||
<p>
|
||||
Here we will build our machine learning approach on elements of the
|
||||
@@ -279,28 +286,161 @@ machine learning algorithms
|
||||
<li> Bayesian statistics and regression</li>
|
||||
<li> Support vector machines and finally various variants of</li>
|
||||
<li> Artifical neural networks and deep learning</li>
|
||||
<li> Networks for unsupervised learning using for example reduced Boltzmann machines.</li>
|
||||
</ol>
|
||||
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec3" class="anchor">Why this text? </h2>
|
||||
<h2 id="___sec3" class="anchor">Choice of programming language </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec4" class="anchor">Choice of programming language </h2>
|
||||
Python plays nowadays a central role in the development of machine
|
||||
learning techniques and tools for data analysis. In particular, seen
|
||||
the wealth of machine learning and data analysis packages written in
|
||||
Python, easy to use libraries with immediate visualization(and not the
|
||||
least impressive galleries of existing example), the popularity of the
|
||||
Jupyter notebook framework with the possibility to run <b>R</b> codes or
|
||||
compiled programs written in C++, and much more made our choice of
|
||||
programming language for this series of lectures of easy. However,
|
||||
since the focus here is not only on using existing Python tools such
|
||||
as <b>scikit-learn</b> or <b>tensorflow</b>, but also on developing your own
|
||||
algorithms and codes, we will as far as possible present many of these
|
||||
algorithms eithers a Python codes or C++ codes. Finally, we will, as
|
||||
far as possible keep parallel versions of the data analysis and
|
||||
machine larning programming aspects in <b>R</b> as
|
||||
well. <a href="https://www.r-project.org/" target="_self">R</a> is a language and environment
|
||||
for statistical computing and graphics which is widely used in
|
||||
statistics and mathematics applications.
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec5" class="anchor">Data handling, machine learning and ethical aspects </h2>
|
||||
The reason we also focus on compiled languages like C++ (or
|
||||
Fortran), is that Python is still notoriously slow when we do not
|
||||
utilize highly streamlined computational libraries like
|
||||
<a href="http://www.netlib.org/lapack/" target="_self">Lapack</a> or other numerical libraries
|
||||
written in compiled languages (many of these libraries are written in
|
||||
Fortran). Although a project like <a href="https://numba.pydata.org/" target="_self">Numba</a>
|
||||
holds great promise for speeding up the unrolling of lengthy loops, C+
|
||||
and Fortran are presently still the performance winners. Numba gives
|
||||
you potentially the power to speed up your applications with high
|
||||
performance functions written directly in Python. In particular,
|
||||
array-oriented and math-heavy Python code can achieve similar
|
||||
performance to C, C++ and Fortran. However, even with these speed-ups,
|
||||
for codes involving heavy Markov Chain Monte Carlo analyses and
|
||||
optimizations of cost functions, C++/C or Fortran codes tend to
|
||||
outperform Python codes.
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec6" class="anchor">Acknowledgements </h2>
|
||||
Presently thus, the community tends to let
|
||||
code written in C++/C or Fortran do the heavy duty numerical
|
||||
number crunching and leave the post-analysis of the data to the above
|
||||
mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen
|
||||
the changes during the last decade, the above situation may change swiftly in the not too distant future.
|
||||
|
||||
<p>
|
||||
Many of the examples we discuss in this series of lectures come with
|
||||
existing data files or provide code examples which produce the data to
|
||||
be analyzed. Most of the applications we will discuss deal with
|
||||
small data sets (less than a terabyte of information) and can easily
|
||||
be analyzed and tested on standard off the shelf laptops you find in general
|
||||
grocery stores.
|
||||
|
||||
<h2 id="___sec4" class="anchor">Data handling, machine learning and ethical aspects </h2>
|
||||
|
||||
<p>
|
||||
In most of the cases we will study, we will either generate the data
|
||||
to analyze ourselves (both for supervised learning and unsupervised
|
||||
learning) or we will recur again and again to data present in say
|
||||
<b>scikit-learn</b> or <b>tensorflow</b>. Many of the examples we end up
|
||||
dealing with are from a privacy and data protection point of view,
|
||||
rather inoccuous and boring results of numerical
|
||||
calculations. However, this does not hinder us from developing a sound
|
||||
ethical attitude to the data we use, how we analyze the data and how
|
||||
we handle the data.
|
||||
|
||||
<p>
|
||||
The most immediate and simplest possible ethical aspects deal with our
|
||||
approach to the scientific process. Nowadays, with version control
|
||||
software like <a href="https://git-scm.com/" target="_self">Git</a> and various online
|
||||
repositories like <a href="https://github.com/" target="_self">Github</a>,
|
||||
<a href="https://about.gitlab.com/" target="_self">Gitlab</a> etc, we can easily make our codes
|
||||
and data sets we have used, freely and easily accessible to a wider
|
||||
community. This helps us almost automagically in making our science
|
||||
reproducible. The large open-source development communities involved
|
||||
in say <a href="http://scikit-learn.org/stable/" target="_self">Scikit-learn</a>,
|
||||
<a href="https://www.tensorflow.org/" target="_self">Tensorflow</a>,
|
||||
<a href="http://pytorch.org/" target="_self">PyTorch</a> and <a href="https://keras.io/" target="_self">Keras</a>, are
|
||||
all excellent examples of this. The codes can be tested and improved
|
||||
upon continuosly, helping thereby our scientific community at large in
|
||||
developing data analysis and machine learning tools. It is much
|
||||
easier today to gain traction and acceptance for making your science
|
||||
reproducible. From a societal stand, this is an important element
|
||||
since many of the developers are employees of large public institutions like
|
||||
universities and research labs. Our taxpayer do deserve to get
|
||||
something back for their bucks.
|
||||
|
||||
<p>
|
||||
However, this more mechanical aspect of the ethics of science (in
|
||||
particular the reproducibility of scientific results) is something
|
||||
which is obvious and everybody should do as part of the dialectics of
|
||||
science. The fact that many scientists are not willing to share their codes or
|
||||
data is detrimental to the scientific discourse.
|
||||
|
||||
<p>
|
||||
Before we proceed, we should add a disclaimer. Even though
|
||||
we may dream of computers developing some kind of higher learning
|
||||
capabilities, at the end (even if the artificial intelligence
|
||||
community keeps touting our ears full of fancy futuristic avenues), it is we
|
||||
who end up constructing and instructing, via various algorithms, the
|
||||
computers. Self-driving cars for example, rely on sofisticated
|
||||
programs which take into account all possible situations a car can
|
||||
encounter. In addition, extensive usage of training datas from GPS
|
||||
information, maps etc, are typically fed into the software for
|
||||
self-driving cars. Adding to this various sensors and cameras that
|
||||
feed information to the programs, there are zillions of ethical issues
|
||||
which arise from this.
|
||||
|
||||
<p>
|
||||
For self-driving cars, where basically many of the standard machine
|
||||
learning algorithms discussed here enter into the codes, at a certain
|
||||
stage we have to make choices. Yes, we , the lads and lasses who wrote
|
||||
a program for a specific brand of a self-driving car. As an example,
|
||||
a most carmakers have as their utmost priority the security of the
|
||||
driver and the accompanying passengers. A famous carmaker, which is
|
||||
one of the leaders in the market of self-driving cars, had <b>if</b>
|
||||
statements of the following type: suppose there are two obstacles in
|
||||
front of you and you cannot avoid to collide with one of them. One of
|
||||
the obstacles is a monstertruck while the other one is a kindergarten
|
||||
class trying to cross the road. The self-driving car algo would then
|
||||
opt for the hitting the small folks instead of the monstertruck, since
|
||||
the likelihood of surving a collision with our future citizens, is
|
||||
much higher.
|
||||
|
||||
<p>
|
||||
This brings us leads then to serious ethical aspects. Why should we
|
||||
opt for such an option? Who decides and who is entitled to make such
|
||||
choices? Keep in mind that many of the algorithms you will about in
|
||||
this series of lectures or hear about later, are indeed based on
|
||||
simple programming instructions. And you are very likely to be one of
|
||||
the people who may end up writing such a code. Thus, developing a
|
||||
sound ethical attitude to what we do, an approach well beyond the
|
||||
simple mechanistic one of making our science available and
|
||||
reproducible, is much needed. The example of the self-driving cars is
|
||||
just one of infinitely many cases where we have to make choices. When
|
||||
you analyze data on economic inequalities, who guarantees that you are
|
||||
not weighting some data in a particular way, perhaps because you dearly want a
|
||||
specific conclusion which may support your political views?
|
||||
|
||||
<p>
|
||||
We do not have the answers here, but we want you think over these
|
||||
topics in a more overarching way. A statistical data analysis with
|
||||
its dry numbers and graphs meant to guide the eye, do not necessarily
|
||||
reflect the truth, whatever that is. As a scientist, and after a
|
||||
university education, you are supposedly a better citizen, with an
|
||||
improved critical view and understanding of the scientific method, and
|
||||
perhaps some deeper understandings of the ethics of science at
|
||||
large. Use these insights. Be a critical citizen. You owe it to our
|
||||
societies.
|
||||
|
||||
<p>
|
||||
To do: Add references and acknowledgements
|
||||
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
|
||||
@@ -3,9 +3,9 @@
|
||||
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Representing data">
|
||||
<meta name="description" content="Introduction to Applied Data Analysis and Machine Learning">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Representing data</title>
|
||||
<title>Introduction to Applied Data Analysis and Machine Learning</title>
|
||||
|
||||
|
||||
|
||||
@@ -116,7 +116,7 @@ td.padding {
|
||||
|
||||
|
||||
|
||||
<center><h1 style="text-align: center;">Data Analysis and Machine Learning: Representing data</h1></center> <!-- document title -->
|
||||
<center><h1 style="text-align: center;">Introduction to Applied Data Analysis and Machine Learning</h1></center> <!-- document title -->
|
||||
|
||||
<p>
|
||||
<!-- author(s): Morten Hjorth-Jensen -->
|
||||
@@ -132,8 +132,79 @@ td.padding {
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p> <br>
|
||||
<center><h4>May 22, 2018</h4></center> <!-- date -->
|
||||
<center><h4>May 28, 2018</h4></center> <!-- date -->
|
||||
<br>
|
||||
|
||||
<h2 id="___sec0">Introduction </h2>
|
||||
|
||||
<p>
|
||||
Statistics, data science and machine learning form important fields of
|
||||
research in modern science. They describe how to learn and make
|
||||
predictions from data, as well as allowing us to extract important
|
||||
correlations about physical process and the underlying laws of motion
|
||||
in large data sets. The latter, big data sets, appear frequently in
|
||||
essentially all disciplines, from the traditional Science, Technology,
|
||||
Mathematics and Engineering fields to Life Science, Law, education
|
||||
research, the Humanities and the Social Sciences.
|
||||
|
||||
<p>
|
||||
It has become more
|
||||
and more common to see research projects on big data in for example
|
||||
the Social Sciences where extracting patterns from complicated survey
|
||||
data is one of many research directions. Having a solid grasp of data
|
||||
analysis and machine learning is thus becoming central to scientific
|
||||
computing in many fields, and competences and skills within the fields
|
||||
of machine learning and scientific computing are nowadays strongly
|
||||
requested by many potential employers. The latter cannot be
|
||||
overstated, familiarity with machine learning has almost become a
|
||||
prerequisite for many of the most exciting employment opportunities,
|
||||
whether they are in bioinformatics, life science, physics or finance,
|
||||
in the private or the public sector. This author has had several
|
||||
students or met students who have been hired recently based on their
|
||||
skills and competences in scientific computing and data science, often
|
||||
with marginal knowledge of machine learning.
|
||||
|
||||
<p>
|
||||
Machine learning is a subfield of computer science, and is closely
|
||||
related to computational statistics. It evolved from the study of
|
||||
pattern recognition in artificial intelligence (AI) research, and has
|
||||
made contributions to AI tasks like computer vision, natural language
|
||||
processing and speech recognition. Many of the methods we will study are also
|
||||
strongly rooted in basic mathematics and physics research.
|
||||
|
||||
<p>
|
||||
Ideally, machine learning represents the science of giving computers
|
||||
the ability to learn without being explicitly programmed. The idea is
|
||||
that there exist generic algorithms which can be used to find patterns
|
||||
in a broad class of data sets without having to write code
|
||||
specifically for each problem. The algorithm will build its own logic
|
||||
based on the data. You should however always keep in mind that
|
||||
machines and algorithms are to a large extent developed by humans. The
|
||||
insights and knowledge we have about a specific system, play a central
|
||||
role when we develop a specific machine learning algorithm.
|
||||
|
||||
<p>
|
||||
Machine learning is an extremely rich field, in spite of its young
|
||||
age. The increases we have seen during the last three decades in
|
||||
computational capabilities have been followed by developments of
|
||||
methods and techniques for analyzing and handling large date sets,
|
||||
relying heavily on statistics, computer science and mathematics. The
|
||||
field is rather new and developing rapidly. Popular software packages
|
||||
written in Python for machine learning like
|
||||
<a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a>,
|
||||
<a href="https://www.tensorflow.org/" target="_blank">Tensorflow</a>,
|
||||
<a href="http://pytorch.org/" target="_blank">PyTorch</a> and <a href="https://keras.io/" target="_blank">Keras</a>, all
|
||||
freely available at their respective GitHub sites, encompass
|
||||
communities of developers in the thousands or more. And the number of
|
||||
code developers and contributors keeps increasing. Not all the
|
||||
algorithms and methods can be given a rigorous mathematical
|
||||
justification, opening up thereby large rooms for experimenting and
|
||||
trial and error and thereby exciting new developments. However, a
|
||||
solid command of linear algebra, multivariate theory, probability
|
||||
theory, statistical data analysis, understanding errors and Monte
|
||||
Carlo methods are central elements in a proper understanding of many
|
||||
of algorithms and methods we will discuss.
|
||||
|
||||
<p>
|
||||
|
||||
<center style="font-size:80%">
|
||||
@@ -142,73 +213,11 @@ td.padding {
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec0">Introduction </h2>
|
||||
|
||||
<p>
|
||||
Statistics, data science and machine learning form important fields of
|
||||
research in modern science. They describe how to learn and make
|
||||
predictions from data, as well as allowing us to extract important
|
||||
correlations about physical process and the underlying laws of motion
|
||||
in large data sets. The latter, big data sets, appear
|
||||
frequently in essentially all disciplines, from the traditional Science,
|
||||
Technology, Mathematics and Engineering fields to Life Science, Law, education research,
|
||||
the Humanities and
|
||||
the Social Sciences. It has become more and more common to see
|
||||
research projects on big data in for example the Social
|
||||
Sciences where extracting patterns from complicated survey data is one of many research directions.
|
||||
Having a solid grasp of data analysis and machine learning
|
||||
is thus becoming central to scientific computing in many
|
||||
fields, and competences and skills within the fields of machine learning
|
||||
and scientific computing are nowadays strongly requested by many
|
||||
potential employers. The latter cannot be overstated, familiarity with
|
||||
machine learning has almost become a prerequisite for many of the most
|
||||
exciting employment opportunities, whether they are in bioinformatics,
|
||||
life science, physics or finance, in the private or the public
|
||||
sector. This author has had several students or met students who have
|
||||
been hired recently based on their skills and competences in
|
||||
scientific computing and data science, often with marginal knowledge
|
||||
of machine learning.
|
||||
|
||||
<p>
|
||||
Machine learning is a subfield of computer science, and is closely
|
||||
related to computational statistics. It evolved from the study of
|
||||
pattern recognition in artificial intelligence (AI) research, and has
|
||||
made contributions to AI tasks like computer vision, natural language
|
||||
processing and speech recognition.
|
||||
Machine learning represents the
|
||||
science of giving computers the ability to learn without being
|
||||
explicitly programmed. The idea is that there exist generic
|
||||
algorithms which can be used to find patterns in a broad class of data
|
||||
sets without having to write code specifically for each problem. The
|
||||
algorithm will build its own logic based on the data.
|
||||
|
||||
<p>
|
||||
Machine learning is an extremely rich field, in spite of its young age. The
|
||||
increases we have seen during the last three decades in computational
|
||||
capabilities have been followed by developments of methods and
|
||||
techniques for analyzing and handling large date sets, relying heavily
|
||||
on statistics, computer science and mathematics. The field is rather
|
||||
new and developing rapidly. Popular software packages written in
|
||||
Python for machine learning like <a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a>, <a href="https://www.tensorflow.org/" target="_blank">Tensorflow</a>,
|
||||
<a href="http://pytorch.org/" target="_blank">PyTorch</a> and <a href="https://keras.io/" target="_blank">Keras</a>, all freely available at their respective GitHub sites,
|
||||
encompass communities of developers in the thousands or more. And the number
|
||||
of code developers and contributors keeps increasing. Not all the
|
||||
algorithms and methods can be given a rigorous mathematical
|
||||
justification, opening up thereby large rooms for experimenting
|
||||
and trial and error and thereby exciting new developments.
|
||||
However, a solid command of linear algebra, multivariate theory,
|
||||
probability theory, statistical data analysis,
|
||||
understanding errors and Monte Carlo methods are central elements in a proper understanding of many of
|
||||
algorithms and methods we will discuss.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec1">Learning outcomes </h2>
|
||||
|
||||
<p>
|
||||
These lectures aim at giving you an overview of central aspects of
|
||||
These setsof lectures aim at giving you an overview of central aspects of
|
||||
statistical data analysis as well as some of the central algorithms
|
||||
used in machine learning. We will introduce a variety of central
|
||||
algorithms and methods essential for studies of data analysis and
|
||||
@@ -259,24 +268,23 @@ and jackknife methods.
|
||||
<p>
|
||||
The second part of the material covers several algorithms used in
|
||||
machine learning.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec2">Types of Machine Learning </h2>
|
||||
|
||||
<p>
|
||||
The approaches to machine learning are many, but are often split into two main categories.
|
||||
In <em>supervised learning</em> we know the answer to a problem,
|
||||
and let the computer deduce the logic behind it. On the other hand, <em>unsupervised learning</em>
|
||||
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
|
||||
Some authours also operate with a third category, namely <em>reinforcement learning</em>. This is a paradigm
|
||||
of learning inspired by behavioral psychology, where learning is achieved by trial-and-error,
|
||||
The approaches to machine learning are many, but are often split into
|
||||
two main categories. In <em>supervised learning</em> we know the answer to a
|
||||
problem, and let the computer deduce the logic behind it. On the other
|
||||
hand, <em>unsupervised learning</em> is a method for finding patterns and
|
||||
relationship in data sets without any prior knowledge of the system.
|
||||
Some authours also operate with a third category, namely
|
||||
<em>reinforcement learning</em>. This is a paradigm of learning inspired by
|
||||
behavioral psychology, where learning is achieved by trial-and-error,
|
||||
solely from rewards and punishment.
|
||||
|
||||
<p>
|
||||
Another way to categorize machine learning tasks is to consider the desired output of a system.
|
||||
Some of the most common tasks are:
|
||||
Another way to categorize machine learning tasks is to consider the
|
||||
desired output of a system. Some of the most common tasks are:
|
||||
|
||||
<ul>
|
||||
|
||||
@@ -290,10 +298,14 @@ Some of the most common tasks are:
|
||||
|
||||
The methods we cover have three main topics in common, irrespective of
|
||||
whether we deal with supervised or unsupervised learning. The first
|
||||
ingredient is normally our data set, the second is a model which is
|
||||
normally a function of some parameters. The last ingredient is a
|
||||
so-called <b>cost</b> function which allows us to present an estimate on
|
||||
how good our model is in reproducing the data it is supposed to train.
|
||||
ingredient is normally our data set (which can be subdivided into
|
||||
training and test data), the second item is a model which is normally a
|
||||
function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.
|
||||
|
||||
<p>
|
||||
The last ingredient is a so-called <b>cost</b>
|
||||
function which allows us to present an estimate on how good our model
|
||||
is in reproducing the data it is supposed to train.
|
||||
|
||||
<p>
|
||||
Here we will build our machine learning approach on elements of the
|
||||
@@ -308,27 +320,161 @@ machine learning algorithms
|
||||
<p><li> Bayesian statistics and regression</li>
|
||||
<p><li> Support vector machines and finally various variants of</li>
|
||||
<p><li> Artifical neural networks and deep learning</li>
|
||||
<p><li> Networks for unsupervised learning using for example reduced Boltzmann machines.</li>
|
||||
</ol>
|
||||
</section>
|
||||
|
||||
<h2 id="___sec3">Choice of programming language </h2>
|
||||
|
||||
<section>
|
||||
<h2 id="___sec3">Why this text? </h2>
|
||||
</section>
|
||||
<p>
|
||||
Python plays nowadays a central role in the development of machine
|
||||
learning techniques and tools for data analysis. In particular, seen
|
||||
the wealth of machine learning and data analysis packages written in
|
||||
Python, easy to use libraries with immediate visualization(and not the
|
||||
least impressive galleries of existing example), the popularity of the
|
||||
Jupyter notebook framework with the possibility to run <b>R</b> codes or
|
||||
compiled programs written in C++, and much more made our choice of
|
||||
programming language for this series of lectures of easy. However,
|
||||
since the focus here is not only on using existing Python tools such
|
||||
as <b>scikit-learn</b> or <b>tensorflow</b>, but also on developing your own
|
||||
algorithms and codes, we will as far as possible present many of these
|
||||
algorithms eithers a Python codes or C++ codes. Finally, we will, as
|
||||
far as possible keep parallel versions of the data analysis and
|
||||
machine larning programming aspects in <b>R</b> as
|
||||
well. <a href="https://www.r-project.org/" target="_blank">R</a> is a language and environment
|
||||
for statistical computing and graphics which is widely used in
|
||||
statistics and mathematics applications.
|
||||
|
||||
<p>
|
||||
The reason we also focus on compiled languages like C++ (or
|
||||
Fortran), is that Python is still notoriously slow when we do not
|
||||
utilize highly streamlined computational libraries like
|
||||
<a href="http://www.netlib.org/lapack/" target="_blank">Lapack</a> or other numerical libraries
|
||||
written in compiled languages (many of these libraries are written in
|
||||
Fortran). Although a project like <a href="https://numba.pydata.org/" target="_blank">Numba</a>
|
||||
holds great promise for speeding up the unrolling of lengthy loops, C+
|
||||
and Fortran are presently still the performance winners. Numba gives
|
||||
you potentially the power to speed up your applications with high
|
||||
performance functions written directly in Python. In particular,
|
||||
array-oriented and math-heavy Python code can achieve similar
|
||||
performance to C, C++ and Fortran. However, even with these speed-ups,
|
||||
for codes involving heavy Markov Chain Monte Carlo analyses and
|
||||
optimizations of cost functions, C++/C or Fortran codes tend to
|
||||
outperform Python codes.
|
||||
|
||||
<section>
|
||||
<h2 id="___sec4">Choice of programming language </h2>
|
||||
</section>
|
||||
<p>
|
||||
Presently thus, the community tends to let
|
||||
code written in C++/C or Fortran do the heavy duty numerical
|
||||
number crunching and leave the post-analysis of the data to the above
|
||||
mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen
|
||||
the changes during the last decade, the above situation may change swiftly in the not too distant future.
|
||||
|
||||
<p>
|
||||
Many of the examples we discuss in this series of lectures come with
|
||||
existing data files or provide code examples which produce the data to
|
||||
be analyzed. Most of the applications we will discuss deal with
|
||||
small data sets (less than a terabyte of information) and can easily
|
||||
be analyzed and tested on standard off the shelf laptops you find in general
|
||||
grocery stores.
|
||||
|
||||
<section>
|
||||
<h2 id="___sec5">Data handling, machine learning and ethical aspects </h2>
|
||||
</section>
|
||||
<h2 id="___sec4">Data handling, machine learning and ethical aspects </h2>
|
||||
|
||||
<p>
|
||||
In most of the cases we will study, we will either generate the data
|
||||
to analyze ourselves (both for supervised learning and unsupervised
|
||||
learning) or we will recur again and again to data present in say
|
||||
<b>scikit-learn</b> or <b>tensorflow</b>. Many of the examples we end up
|
||||
dealing with are from a privacy and data protection point of view,
|
||||
rather inoccuous and boring results of numerical
|
||||
calculations. However, this does not hinder us from developing a sound
|
||||
ethical attitude to the data we use, how we analyze the data and how
|
||||
we handle the data.
|
||||
|
||||
<section>
|
||||
<h2 id="___sec6">Acknowledgements </h2>
|
||||
<p>
|
||||
The most immediate and simplest possible ethical aspects deal with our
|
||||
approach to the scientific process. Nowadays, with version control
|
||||
software like <a href="https://git-scm.com/" target="_blank">Git</a> and various online
|
||||
repositories like <a href="https://github.com/" target="_blank">Github</a>,
|
||||
<a href="https://about.gitlab.com/" target="_blank">Gitlab</a> etc, we can easily make our codes
|
||||
and data sets we have used, freely and easily accessible to a wider
|
||||
community. This helps us almost automagically in making our science
|
||||
reproducible. The large open-source development communities involved
|
||||
in say <a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a>,
|
||||
<a href="https://www.tensorflow.org/" target="_blank">Tensorflow</a>,
|
||||
<a href="http://pytorch.org/" target="_blank">PyTorch</a> and <a href="https://keras.io/" target="_blank">Keras</a>, are
|
||||
all excellent examples of this. The codes can be tested and improved
|
||||
upon continuosly, helping thereby our scientific community at large in
|
||||
developing data analysis and machine learning tools. It is much
|
||||
easier today to gain traction and acceptance for making your science
|
||||
reproducible. From a societal stand, this is an important element
|
||||
since many of the developers are employees of large public institutions like
|
||||
universities and research labs. Our taxpayer do deserve to get
|
||||
something back for their bucks.
|
||||
|
||||
<p>
|
||||
However, this more mechanical aspect of the ethics of science (in
|
||||
particular the reproducibility of scientific results) is something
|
||||
which is obvious and everybody should do as part of the dialectics of
|
||||
science. The fact that many scientists are not willing to share their codes or
|
||||
data is detrimental to the scientific discourse.
|
||||
|
||||
<p>
|
||||
Before we proceed, we should add a disclaimer. Even though
|
||||
we may dream of computers developing some kind of higher learning
|
||||
capabilities, at the end (even if the artificial intelligence
|
||||
community keeps touting our ears full of fancy futuristic avenues), it is we
|
||||
who end up constructing and instructing, via various algorithms, the
|
||||
computers. Self-driving cars for example, rely on sofisticated
|
||||
programs which take into account all possible situations a car can
|
||||
encounter. In addition, extensive usage of training datas from GPS
|
||||
information, maps etc, are typically fed into the software for
|
||||
self-driving cars. Adding to this various sensors and cameras that
|
||||
feed information to the programs, there are zillions of ethical issues
|
||||
which arise from this.
|
||||
|
||||
<p>
|
||||
For self-driving cars, where basically many of the standard machine
|
||||
learning algorithms discussed here enter into the codes, at a certain
|
||||
stage we have to make choices. Yes, we , the lads and lasses who wrote
|
||||
a program for a specific brand of a self-driving car. As an example,
|
||||
a most carmakers have as their utmost priority the security of the
|
||||
driver and the accompanying passengers. A famous carmaker, which is
|
||||
one of the leaders in the market of self-driving cars, had <b>if</b>
|
||||
statements of the following type: suppose there are two obstacles in
|
||||
front of you and you cannot avoid to collide with one of them. One of
|
||||
the obstacles is a monstertruck while the other one is a kindergarten
|
||||
class trying to cross the road. The self-driving car algo would then
|
||||
opt for the hitting the small folks instead of the monstertruck, since
|
||||
the likelihood of surving a collision with our future citizens, is
|
||||
much higher.
|
||||
|
||||
<p>
|
||||
This brings us leads then to serious ethical aspects. Why should we
|
||||
opt for such an option? Who decides and who is entitled to make such
|
||||
choices? Keep in mind that many of the algorithms you will about in
|
||||
this series of lectures or hear about later, are indeed based on
|
||||
simple programming instructions. And you are very likely to be one of
|
||||
the people who may end up writing such a code. Thus, developing a
|
||||
sound ethical attitude to what we do, an approach well beyond the
|
||||
simple mechanistic one of making our science available and
|
||||
reproducible, is much needed. The example of the self-driving cars is
|
||||
just one of infinitely many cases where we have to make choices. When
|
||||
you analyze data on economic inequalities, who guarantees that you are
|
||||
not weighting some data in a particular way, perhaps because you dearly want a
|
||||
specific conclusion which may support your political views?
|
||||
|
||||
<p>
|
||||
We do not have the answers here, but we want you think over these
|
||||
topics in a more overarching way. A statistical data analysis with
|
||||
its dry numbers and graphs meant to guide the eye, do not necessarily
|
||||
reflect the truth, whatever that is. As a scientist, and after a
|
||||
university education, you are supposedly a better citizen, with an
|
||||
improved critical view and understanding of the scientific method, and
|
||||
perhaps some deeper understandings of the ethics of science at
|
||||
large. Use these insights. Be a critical citizen. You owe it to our
|
||||
societies.
|
||||
|
||||
<p>
|
||||
To do: Add references and acknowledgements
|
||||
</section>
|
||||
|
||||
|
||||
|
||||
@@ -6,9 +6,9 @@ Automatically generated HTML file from DocOnce source
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Representing data">
|
||||
<meta name="description" content="Introduction to Applied Data Analysis and Machine Learning">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Representing data</title>
|
||||
<title>Introduction to Applied Data Analysis and Machine Learning</title>
|
||||
|
||||
|
||||
<link href="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
|
||||
@@ -37,13 +37,11 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
'sections': [('Introduction', 2, None, '___sec0'),
|
||||
('Learning outcomes', 2, None, '___sec1'),
|
||||
('Types of Machine Learning', 2, None, '___sec2'),
|
||||
('Why this text?', 2, None, '___sec3'),
|
||||
('Choice of programming language', 2, None, '___sec4'),
|
||||
('Choice of programming language', 2, None, '___sec3'),
|
||||
('Data handling, machine learning and ethical aspects',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Acknowledgements', 2, None, '___sec6')]}
|
||||
'___sec4')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -53,7 +51,7 @@ end of tocinfo -->
|
||||
|
||||
|
||||
|
||||
<center><h1>Data Analysis and Machine Learning: Representing data</h1></center> <!-- document title -->
|
||||
<center><h1>Introduction to Applied Data Analysis and Machine Learning</h1></center> <!-- document title -->
|
||||
|
||||
<p>
|
||||
<!-- author(s): Morten Hjorth-Jensen -->
|
||||
@@ -69,10 +67,8 @@ end of tocinfo -->
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>May 22, 2018</h4></center> <!-- date -->
|
||||
<center><h4>May 28, 2018</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec0">Introduction </h2>
|
||||
|
||||
@@ -81,57 +77,68 @@ Statistics, data science and machine learning form important fields of
|
||||
research in modern science. They describe how to learn and make
|
||||
predictions from data, as well as allowing us to extract important
|
||||
correlations about physical process and the underlying laws of motion
|
||||
in large data sets. The latter, big data sets, appear
|
||||
frequently in essentially all disciplines, from the traditional Science,
|
||||
Technology, Mathematics and Engineering fields to Life Science, Law, education research,
|
||||
the Humanities and
|
||||
the Social Sciences. It has become more and more common to see
|
||||
research projects on big data in for example the Social
|
||||
Sciences where extracting patterns from complicated survey data is one of many research directions.
|
||||
Having a solid grasp of data analysis and machine learning
|
||||
is thus becoming central to scientific computing in many
|
||||
fields, and competences and skills within the fields of machine learning
|
||||
and scientific computing are nowadays strongly requested by many
|
||||
potential employers. The latter cannot be overstated, familiarity with
|
||||
machine learning has almost become a prerequisite for many of the most
|
||||
exciting employment opportunities, whether they are in bioinformatics,
|
||||
life science, physics or finance, in the private or the public
|
||||
sector. This author has had several students or met students who have
|
||||
been hired recently based on their skills and competences in
|
||||
scientific computing and data science, often with marginal knowledge
|
||||
of machine learning.
|
||||
in large data sets. The latter, big data sets, appear frequently in
|
||||
essentially all disciplines, from the traditional Science, Technology,
|
||||
Mathematics and Engineering fields to Life Science, Law, education
|
||||
research, the Humanities and the Social Sciences.
|
||||
|
||||
<p>
|
||||
It has become more
|
||||
and more common to see research projects on big data in for example
|
||||
the Social Sciences where extracting patterns from complicated survey
|
||||
data is one of many research directions. Having a solid grasp of data
|
||||
analysis and machine learning is thus becoming central to scientific
|
||||
computing in many fields, and competences and skills within the fields
|
||||
of machine learning and scientific computing are nowadays strongly
|
||||
requested by many potential employers. The latter cannot be
|
||||
overstated, familiarity with machine learning has almost become a
|
||||
prerequisite for many of the most exciting employment opportunities,
|
||||
whether they are in bioinformatics, life science, physics or finance,
|
||||
in the private or the public sector. This author has had several
|
||||
students or met students who have been hired recently based on their
|
||||
skills and competences in scientific computing and data science, often
|
||||
with marginal knowledge of machine learning.
|
||||
|
||||
<p>
|
||||
Machine learning is a subfield of computer science, and is closely
|
||||
related to computational statistics. It evolved from the study of
|
||||
pattern recognition in artificial intelligence (AI) research, and has
|
||||
made contributions to AI tasks like computer vision, natural language
|
||||
processing and speech recognition.
|
||||
Machine learning represents the
|
||||
science of giving computers the ability to learn without being
|
||||
explicitly programmed. The idea is that there exist generic
|
||||
algorithms which can be used to find patterns in a broad class of data
|
||||
sets without having to write code specifically for each problem. The
|
||||
algorithm will build its own logic based on the data.
|
||||
processing and speech recognition. Many of the methods we will study are also
|
||||
strongly rooted in basic mathematics and physics research.
|
||||
|
||||
<p>
|
||||
Machine learning is an extremely rich field, in spite of its young age. The
|
||||
increases we have seen during the last three decades in computational
|
||||
capabilities have been followed by developments of methods and
|
||||
techniques for analyzing and handling large date sets, relying heavily
|
||||
on statistics, computer science and mathematics. The field is rather
|
||||
new and developing rapidly. Popular software packages written in
|
||||
Python for machine learning like <a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a>, <a href="https://www.tensorflow.org/" target="_blank">Tensorflow</a>,
|
||||
<a href="http://pytorch.org/" target="_blank">PyTorch</a> and <a href="https://keras.io/" target="_blank">Keras</a>, all freely available at their respective GitHub sites,
|
||||
encompass communities of developers in the thousands or more. And the number
|
||||
of code developers and contributors keeps increasing. Not all the
|
||||
Ideally, machine learning represents the science of giving computers
|
||||
the ability to learn without being explicitly programmed. The idea is
|
||||
that there exist generic algorithms which can be used to find patterns
|
||||
in a broad class of data sets without having to write code
|
||||
specifically for each problem. The algorithm will build its own logic
|
||||
based on the data. You should however always keep in mind that
|
||||
machines and algorithms are to a large extent developed by humans. The
|
||||
insights and knowledge we have about a specific system, play a central
|
||||
role when we develop a specific machine learning algorithm.
|
||||
|
||||
<p>
|
||||
Machine learning is an extremely rich field, in spite of its young
|
||||
age. The increases we have seen during the last three decades in
|
||||
computational capabilities have been followed by developments of
|
||||
methods and techniques for analyzing and handling large date sets,
|
||||
relying heavily on statistics, computer science and mathematics. The
|
||||
field is rather new and developing rapidly. Popular software packages
|
||||
written in Python for machine learning like
|
||||
<a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a>,
|
||||
<a href="https://www.tensorflow.org/" target="_blank">Tensorflow</a>,
|
||||
<a href="http://pytorch.org/" target="_blank">PyTorch</a> and <a href="https://keras.io/" target="_blank">Keras</a>, all
|
||||
freely available at their respective GitHub sites, encompass
|
||||
communities of developers in the thousands or more. And the number of
|
||||
code developers and contributors keeps increasing. Not all the
|
||||
algorithms and methods can be given a rigorous mathematical
|
||||
justification, opening up thereby large rooms for experimenting
|
||||
and trial and error and thereby exciting new developments.
|
||||
However, a solid command of linear algebra, multivariate theory,
|
||||
probability theory, statistical data analysis,
|
||||
understanding errors and Monte Carlo methods are central elements in a proper understanding of many of
|
||||
algorithms and methods we will discuss.
|
||||
justification, opening up thereby large rooms for experimenting and
|
||||
trial and error and thereby exciting new developments. However, a
|
||||
solid command of linear algebra, multivariate theory, probability
|
||||
theory, statistical data analysis, understanding errors and Monte
|
||||
Carlo methods are central elements in a proper understanding of many
|
||||
of algorithms and methods we will discuss.
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
@@ -139,7 +146,7 @@ algorithms and methods we will discuss.
|
||||
<h2 id="___sec1">Learning outcomes </h2>
|
||||
|
||||
<p>
|
||||
These lectures aim at giving you an overview of central aspects of
|
||||
These setsof lectures aim at giving you an overview of central aspects of
|
||||
statistical data analysis as well as some of the central algorithms
|
||||
used in machine learning. We will introduce a variety of central
|
||||
algorithms and methods essential for studies of data analysis and
|
||||
@@ -190,23 +197,22 @@ and jackknife methods.
|
||||
The second part of the material covers several algorithms used in
|
||||
machine learning.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec2">Types of Machine Learning </h2>
|
||||
|
||||
<p>
|
||||
The approaches to machine learning are many, but are often split into two main categories.
|
||||
In <em>supervised learning</em> we know the answer to a problem,
|
||||
and let the computer deduce the logic behind it. On the other hand, <em>unsupervised learning</em>
|
||||
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
|
||||
Some authours also operate with a third category, namely <em>reinforcement learning</em>. This is a paradigm
|
||||
of learning inspired by behavioral psychology, where learning is achieved by trial-and-error,
|
||||
The approaches to machine learning are many, but are often split into
|
||||
two main categories. In <em>supervised learning</em> we know the answer to a
|
||||
problem, and let the computer deduce the logic behind it. On the other
|
||||
hand, <em>unsupervised learning</em> is a method for finding patterns and
|
||||
relationship in data sets without any prior knowledge of the system.
|
||||
Some authours also operate with a third category, namely
|
||||
<em>reinforcement learning</em>. This is a paradigm of learning inspired by
|
||||
behavioral psychology, where learning is achieved by trial-and-error,
|
||||
solely from rewards and punishment.
|
||||
|
||||
<p>
|
||||
Another way to categorize machine learning tasks is to consider the desired output of a system.
|
||||
Some of the most common tasks are:
|
||||
Another way to categorize machine learning tasks is to consider the
|
||||
desired output of a system. Some of the most common tasks are:
|
||||
|
||||
<ul>
|
||||
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
|
||||
@@ -216,10 +222,14 @@ Some of the most common tasks are:
|
||||
|
||||
The methods we cover have three main topics in common, irrespective of
|
||||
whether we deal with supervised or unsupervised learning. The first
|
||||
ingredient is normally our data set, the second is a model which is
|
||||
normally a function of some parameters. The last ingredient is a
|
||||
so-called <b>cost</b> function which allows us to present an estimate on
|
||||
how good our model is in reproducing the data it is supposed to train.
|
||||
ingredient is normally our data set (which can be subdivided into
|
||||
training and test data), the second item is a model which is normally a
|
||||
function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.
|
||||
|
||||
<p>
|
||||
The last ingredient is a so-called <b>cost</b>
|
||||
function which allows us to present an estimate on how good our model
|
||||
is in reproducing the data it is supposed to train.
|
||||
|
||||
<p>
|
||||
Here we will build our machine learning approach on elements of the
|
||||
@@ -234,28 +244,161 @@ machine learning algorithms
|
||||
<li> Bayesian statistics and regression</li>
|
||||
<li> Support vector machines and finally various variants of</li>
|
||||
<li> Artifical neural networks and deep learning</li>
|
||||
<li> Networks for unsupervised learning using for example reduced Boltzmann machines.</li>
|
||||
</ol>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec3">Why this text? </h2>
|
||||
<h2 id="___sec3">Choice of programming language </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec4">Choice of programming language </h2>
|
||||
Python plays nowadays a central role in the development of machine
|
||||
learning techniques and tools for data analysis. In particular, seen
|
||||
the wealth of machine learning and data analysis packages written in
|
||||
Python, easy to use libraries with immediate visualization(and not the
|
||||
least impressive galleries of existing example), the popularity of the
|
||||
Jupyter notebook framework with the possibility to run <b>R</b> codes or
|
||||
compiled programs written in C++, and much more made our choice of
|
||||
programming language for this series of lectures of easy. However,
|
||||
since the focus here is not only on using existing Python tools such
|
||||
as <b>scikit-learn</b> or <b>tensorflow</b>, but also on developing your own
|
||||
algorithms and codes, we will as far as possible present many of these
|
||||
algorithms eithers a Python codes or C++ codes. Finally, we will, as
|
||||
far as possible keep parallel versions of the data analysis and
|
||||
machine larning programming aspects in <b>R</b> as
|
||||
well. <a href="https://www.r-project.org/" target="_blank">R</a> is a language and environment
|
||||
for statistical computing and graphics which is widely used in
|
||||
statistics and mathematics applications.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec5">Data handling, machine learning and ethical aspects </h2>
|
||||
The reason we also focus on compiled languages like C++ (or
|
||||
Fortran), is that Python is still notoriously slow when we do not
|
||||
utilize highly streamlined computational libraries like
|
||||
<a href="http://www.netlib.org/lapack/" target="_blank">Lapack</a> or other numerical libraries
|
||||
written in compiled languages (many of these libraries are written in
|
||||
Fortran). Although a project like <a href="https://numba.pydata.org/" target="_blank">Numba</a>
|
||||
holds great promise for speeding up the unrolling of lengthy loops, C+
|
||||
and Fortran are presently still the performance winners. Numba gives
|
||||
you potentially the power to speed up your applications with high
|
||||
performance functions written directly in Python. In particular,
|
||||
array-oriented and math-heavy Python code can achieve similar
|
||||
performance to C, C++ and Fortran. However, even with these speed-ups,
|
||||
for codes involving heavy Markov Chain Monte Carlo analyses and
|
||||
optimizations of cost functions, C++/C or Fortran codes tend to
|
||||
outperform Python codes.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec6">Acknowledgements </h2>
|
||||
Presently thus, the community tends to let
|
||||
code written in C++/C or Fortran do the heavy duty numerical
|
||||
number crunching and leave the post-analysis of the data to the above
|
||||
mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen
|
||||
the changes during the last decade, the above situation may change swiftly in the not too distant future.
|
||||
|
||||
<p>
|
||||
Many of the examples we discuss in this series of lectures come with
|
||||
existing data files or provide code examples which produce the data to
|
||||
be analyzed. Most of the applications we will discuss deal with
|
||||
small data sets (less than a terabyte of information) and can easily
|
||||
be analyzed and tested on standard off the shelf laptops you find in general
|
||||
grocery stores.
|
||||
|
||||
<h2 id="___sec4">Data handling, machine learning and ethical aspects </h2>
|
||||
|
||||
<p>
|
||||
In most of the cases we will study, we will either generate the data
|
||||
to analyze ourselves (both for supervised learning and unsupervised
|
||||
learning) or we will recur again and again to data present in say
|
||||
<b>scikit-learn</b> or <b>tensorflow</b>. Many of the examples we end up
|
||||
dealing with are from a privacy and data protection point of view,
|
||||
rather inoccuous and boring results of numerical
|
||||
calculations. However, this does not hinder us from developing a sound
|
||||
ethical attitude to the data we use, how we analyze the data and how
|
||||
we handle the data.
|
||||
|
||||
<p>
|
||||
The most immediate and simplest possible ethical aspects deal with our
|
||||
approach to the scientific process. Nowadays, with version control
|
||||
software like <a href="https://git-scm.com/" target="_blank">Git</a> and various online
|
||||
repositories like <a href="https://github.com/" target="_blank">Github</a>,
|
||||
<a href="https://about.gitlab.com/" target="_blank">Gitlab</a> etc, we can easily make our codes
|
||||
and data sets we have used, freely and easily accessible to a wider
|
||||
community. This helps us almost automagically in making our science
|
||||
reproducible. The large open-source development communities involved
|
||||
in say <a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a>,
|
||||
<a href="https://www.tensorflow.org/" target="_blank">Tensorflow</a>,
|
||||
<a href="http://pytorch.org/" target="_blank">PyTorch</a> and <a href="https://keras.io/" target="_blank">Keras</a>, are
|
||||
all excellent examples of this. The codes can be tested and improved
|
||||
upon continuosly, helping thereby our scientific community at large in
|
||||
developing data analysis and machine learning tools. It is much
|
||||
easier today to gain traction and acceptance for making your science
|
||||
reproducible. From a societal stand, this is an important element
|
||||
since many of the developers are employees of large public institutions like
|
||||
universities and research labs. Our taxpayer do deserve to get
|
||||
something back for their bucks.
|
||||
|
||||
<p>
|
||||
However, this more mechanical aspect of the ethics of science (in
|
||||
particular the reproducibility of scientific results) is something
|
||||
which is obvious and everybody should do as part of the dialectics of
|
||||
science. The fact that many scientists are not willing to share their codes or
|
||||
data is detrimental to the scientific discourse.
|
||||
|
||||
<p>
|
||||
Before we proceed, we should add a disclaimer. Even though
|
||||
we may dream of computers developing some kind of higher learning
|
||||
capabilities, at the end (even if the artificial intelligence
|
||||
community keeps touting our ears full of fancy futuristic avenues), it is we
|
||||
who end up constructing and instructing, via various algorithms, the
|
||||
computers. Self-driving cars for example, rely on sofisticated
|
||||
programs which take into account all possible situations a car can
|
||||
encounter. In addition, extensive usage of training datas from GPS
|
||||
information, maps etc, are typically fed into the software for
|
||||
self-driving cars. Adding to this various sensors and cameras that
|
||||
feed information to the programs, there are zillions of ethical issues
|
||||
which arise from this.
|
||||
|
||||
<p>
|
||||
For self-driving cars, where basically many of the standard machine
|
||||
learning algorithms discussed here enter into the codes, at a certain
|
||||
stage we have to make choices. Yes, we , the lads and lasses who wrote
|
||||
a program for a specific brand of a self-driving car. As an example,
|
||||
a most carmakers have as their utmost priority the security of the
|
||||
driver and the accompanying passengers. A famous carmaker, which is
|
||||
one of the leaders in the market of self-driving cars, had <b>if</b>
|
||||
statements of the following type: suppose there are two obstacles in
|
||||
front of you and you cannot avoid to collide with one of them. One of
|
||||
the obstacles is a monstertruck while the other one is a kindergarten
|
||||
class trying to cross the road. The self-driving car algo would then
|
||||
opt for the hitting the small folks instead of the monstertruck, since
|
||||
the likelihood of surving a collision with our future citizens, is
|
||||
much higher.
|
||||
|
||||
<p>
|
||||
This brings us leads then to serious ethical aspects. Why should we
|
||||
opt for such an option? Who decides and who is entitled to make such
|
||||
choices? Keep in mind that many of the algorithms you will about in
|
||||
this series of lectures or hear about later, are indeed based on
|
||||
simple programming instructions. And you are very likely to be one of
|
||||
the people who may end up writing such a code. Thus, developing a
|
||||
sound ethical attitude to what we do, an approach well beyond the
|
||||
simple mechanistic one of making our science available and
|
||||
reproducible, is much needed. The example of the self-driving cars is
|
||||
just one of infinitely many cases where we have to make choices. When
|
||||
you analyze data on economic inequalities, who guarantees that you are
|
||||
not weighting some data in a particular way, perhaps because you dearly want a
|
||||
specific conclusion which may support your political views?
|
||||
|
||||
<p>
|
||||
We do not have the answers here, but we want you think over these
|
||||
topics in a more overarching way. A statistical data analysis with
|
||||
its dry numbers and graphs meant to guide the eye, do not necessarily
|
||||
reflect the truth, whatever that is. As a scientist, and after a
|
||||
university education, you are supposedly a better citizen, with an
|
||||
improved critical view and understanding of the scientific method, and
|
||||
perhaps some deeper understandings of the ethics of science at
|
||||
large. Use these insights. Be a critical citizen. You owe it to our
|
||||
societies.
|
||||
|
||||
<p>
|
||||
To do: Add references and acknowledgements
|
||||
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
|
||||
@@ -6,9 +6,9 @@ Automatically generated HTML file from DocOnce source
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Representing data">
|
||||
<meta name="description" content="Introduction to Applied Data Analysis and Machine Learning">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Representing data</title>
|
||||
<title>Introduction to Applied Data Analysis and Machine Learning</title>
|
||||
|
||||
|
||||
<style type="text/css">
|
||||
@@ -42,13 +42,11 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
'sections': [('Introduction', 2, None, '___sec0'),
|
||||
('Learning outcomes', 2, None, '___sec1'),
|
||||
('Types of Machine Learning', 2, None, '___sec2'),
|
||||
('Why this text?', 2, None, '___sec3'),
|
||||
('Choice of programming language', 2, None, '___sec4'),
|
||||
('Choice of programming language', 2, None, '___sec3'),
|
||||
('Data handling, machine learning and ethical aspects',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Acknowledgements', 2, None, '___sec6')]}
|
||||
'___sec4')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -58,7 +56,7 @@ end of tocinfo -->
|
||||
|
||||
|
||||
|
||||
<center><h1>Data Analysis and Machine Learning: Representing data</h1></center> <!-- document title -->
|
||||
<center><h1>Introduction to Applied Data Analysis and Machine Learning</h1></center> <!-- document title -->
|
||||
|
||||
<p>
|
||||
<!-- author(s): Morten Hjorth-Jensen -->
|
||||
@@ -74,10 +72,8 @@ end of tocinfo -->
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>May 22, 2018</h4></center> <!-- date -->
|
||||
<center><h4>May 28, 2018</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec0">Introduction </h2>
|
||||
|
||||
@@ -86,57 +82,68 @@ Statistics, data science and machine learning form important fields of
|
||||
research in modern science. They describe how to learn and make
|
||||
predictions from data, as well as allowing us to extract important
|
||||
correlations about physical process and the underlying laws of motion
|
||||
in large data sets. The latter, big data sets, appear
|
||||
frequently in essentially all disciplines, from the traditional Science,
|
||||
Technology, Mathematics and Engineering fields to Life Science, Law, education research,
|
||||
the Humanities and
|
||||
the Social Sciences. It has become more and more common to see
|
||||
research projects on big data in for example the Social
|
||||
Sciences where extracting patterns from complicated survey data is one of many research directions.
|
||||
Having a solid grasp of data analysis and machine learning
|
||||
is thus becoming central to scientific computing in many
|
||||
fields, and competences and skills within the fields of machine learning
|
||||
and scientific computing are nowadays strongly requested by many
|
||||
potential employers. The latter cannot be overstated, familiarity with
|
||||
machine learning has almost become a prerequisite for many of the most
|
||||
exciting employment opportunities, whether they are in bioinformatics,
|
||||
life science, physics or finance, in the private or the public
|
||||
sector. This author has had several students or met students who have
|
||||
been hired recently based on their skills and competences in
|
||||
scientific computing and data science, often with marginal knowledge
|
||||
of machine learning.
|
||||
in large data sets. The latter, big data sets, appear frequently in
|
||||
essentially all disciplines, from the traditional Science, Technology,
|
||||
Mathematics and Engineering fields to Life Science, Law, education
|
||||
research, the Humanities and the Social Sciences.
|
||||
|
||||
<p>
|
||||
It has become more
|
||||
and more common to see research projects on big data in for example
|
||||
the Social Sciences where extracting patterns from complicated survey
|
||||
data is one of many research directions. Having a solid grasp of data
|
||||
analysis and machine learning is thus becoming central to scientific
|
||||
computing in many fields, and competences and skills within the fields
|
||||
of machine learning and scientific computing are nowadays strongly
|
||||
requested by many potential employers. The latter cannot be
|
||||
overstated, familiarity with machine learning has almost become a
|
||||
prerequisite for many of the most exciting employment opportunities,
|
||||
whether they are in bioinformatics, life science, physics or finance,
|
||||
in the private or the public sector. This author has had several
|
||||
students or met students who have been hired recently based on their
|
||||
skills and competences in scientific computing and data science, often
|
||||
with marginal knowledge of machine learning.
|
||||
|
||||
<p>
|
||||
Machine learning is a subfield of computer science, and is closely
|
||||
related to computational statistics. It evolved from the study of
|
||||
pattern recognition in artificial intelligence (AI) research, and has
|
||||
made contributions to AI tasks like computer vision, natural language
|
||||
processing and speech recognition.
|
||||
Machine learning represents the
|
||||
science of giving computers the ability to learn without being
|
||||
explicitly programmed. The idea is that there exist generic
|
||||
algorithms which can be used to find patterns in a broad class of data
|
||||
sets without having to write code specifically for each problem. The
|
||||
algorithm will build its own logic based on the data.
|
||||
processing and speech recognition. Many of the methods we will study are also
|
||||
strongly rooted in basic mathematics and physics research.
|
||||
|
||||
<p>
|
||||
Machine learning is an extremely rich field, in spite of its young age. The
|
||||
increases we have seen during the last three decades in computational
|
||||
capabilities have been followed by developments of methods and
|
||||
techniques for analyzing and handling large date sets, relying heavily
|
||||
on statistics, computer science and mathematics. The field is rather
|
||||
new and developing rapidly. Popular software packages written in
|
||||
Python for machine learning like <a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a>, <a href="https://www.tensorflow.org/" target="_blank">Tensorflow</a>,
|
||||
<a href="http://pytorch.org/" target="_blank">PyTorch</a> and <a href="https://keras.io/" target="_blank">Keras</a>, all freely available at their respective GitHub sites,
|
||||
encompass communities of developers in the thousands or more. And the number
|
||||
of code developers and contributors keeps increasing. Not all the
|
||||
Ideally, machine learning represents the science of giving computers
|
||||
the ability to learn without being explicitly programmed. The idea is
|
||||
that there exist generic algorithms which can be used to find patterns
|
||||
in a broad class of data sets without having to write code
|
||||
specifically for each problem. The algorithm will build its own logic
|
||||
based on the data. You should however always keep in mind that
|
||||
machines and algorithms are to a large extent developed by humans. The
|
||||
insights and knowledge we have about a specific system, play a central
|
||||
role when we develop a specific machine learning algorithm.
|
||||
|
||||
<p>
|
||||
Machine learning is an extremely rich field, in spite of its young
|
||||
age. The increases we have seen during the last three decades in
|
||||
computational capabilities have been followed by developments of
|
||||
methods and techniques for analyzing and handling large date sets,
|
||||
relying heavily on statistics, computer science and mathematics. The
|
||||
field is rather new and developing rapidly. Popular software packages
|
||||
written in Python for machine learning like
|
||||
<a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a>,
|
||||
<a href="https://www.tensorflow.org/" target="_blank">Tensorflow</a>,
|
||||
<a href="http://pytorch.org/" target="_blank">PyTorch</a> and <a href="https://keras.io/" target="_blank">Keras</a>, all
|
||||
freely available at their respective GitHub sites, encompass
|
||||
communities of developers in the thousands or more. And the number of
|
||||
code developers and contributors keeps increasing. Not all the
|
||||
algorithms and methods can be given a rigorous mathematical
|
||||
justification, opening up thereby large rooms for experimenting
|
||||
and trial and error and thereby exciting new developments.
|
||||
However, a solid command of linear algebra, multivariate theory,
|
||||
probability theory, statistical data analysis,
|
||||
understanding errors and Monte Carlo methods are central elements in a proper understanding of many of
|
||||
algorithms and methods we will discuss.
|
||||
justification, opening up thereby large rooms for experimenting and
|
||||
trial and error and thereby exciting new developments. However, a
|
||||
solid command of linear algebra, multivariate theory, probability
|
||||
theory, statistical data analysis, understanding errors and Monte
|
||||
Carlo methods are central elements in a proper understanding of many
|
||||
of algorithms and methods we will discuss.
|
||||
|
||||
<p>
|
||||
<!-- !split -->
|
||||
@@ -144,7 +151,7 @@ algorithms and methods we will discuss.
|
||||
<h2 id="___sec1">Learning outcomes </h2>
|
||||
|
||||
<p>
|
||||
These lectures aim at giving you an overview of central aspects of
|
||||
These setsof lectures aim at giving you an overview of central aspects of
|
||||
statistical data analysis as well as some of the central algorithms
|
||||
used in machine learning. We will introduce a variety of central
|
||||
algorithms and methods essential for studies of data analysis and
|
||||
@@ -195,23 +202,22 @@ and jackknife methods.
|
||||
The second part of the material covers several algorithms used in
|
||||
machine learning.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec2">Types of Machine Learning </h2>
|
||||
|
||||
<p>
|
||||
The approaches to machine learning are many, but are often split into two main categories.
|
||||
In <em>supervised learning</em> we know the answer to a problem,
|
||||
and let the computer deduce the logic behind it. On the other hand, <em>unsupervised learning</em>
|
||||
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
|
||||
Some authours also operate with a third category, namely <em>reinforcement learning</em>. This is a paradigm
|
||||
of learning inspired by behavioral psychology, where learning is achieved by trial-and-error,
|
||||
The approaches to machine learning are many, but are often split into
|
||||
two main categories. In <em>supervised learning</em> we know the answer to a
|
||||
problem, and let the computer deduce the logic behind it. On the other
|
||||
hand, <em>unsupervised learning</em> is a method for finding patterns and
|
||||
relationship in data sets without any prior knowledge of the system.
|
||||
Some authours also operate with a third category, namely
|
||||
<em>reinforcement learning</em>. This is a paradigm of learning inspired by
|
||||
behavioral psychology, where learning is achieved by trial-and-error,
|
||||
solely from rewards and punishment.
|
||||
|
||||
<p>
|
||||
Another way to categorize machine learning tasks is to consider the desired output of a system.
|
||||
Some of the most common tasks are:
|
||||
Another way to categorize machine learning tasks is to consider the
|
||||
desired output of a system. Some of the most common tasks are:
|
||||
|
||||
<ul>
|
||||
<li> Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.</li>
|
||||
@@ -221,10 +227,14 @@ Some of the most common tasks are:
|
||||
|
||||
The methods we cover have three main topics in common, irrespective of
|
||||
whether we deal with supervised or unsupervised learning. The first
|
||||
ingredient is normally our data set, the second is a model which is
|
||||
normally a function of some parameters. The last ingredient is a
|
||||
so-called <b>cost</b> function which allows us to present an estimate on
|
||||
how good our model is in reproducing the data it is supposed to train.
|
||||
ingredient is normally our data set (which can be subdivided into
|
||||
training and test data), the second item is a model which is normally a
|
||||
function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.
|
||||
|
||||
<p>
|
||||
The last ingredient is a so-called <b>cost</b>
|
||||
function which allows us to present an estimate on how good our model
|
||||
is in reproducing the data it is supposed to train.
|
||||
|
||||
<p>
|
||||
Here we will build our machine learning approach on elements of the
|
||||
@@ -239,28 +249,161 @@ machine learning algorithms
|
||||
<li> Bayesian statistics and regression</li>
|
||||
<li> Support vector machines and finally various variants of</li>
|
||||
<li> Artifical neural networks and deep learning</li>
|
||||
<li> Networks for unsupervised learning using for example reduced Boltzmann machines.</li>
|
||||
</ol>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec3">Why this text? </h2>
|
||||
<h2 id="___sec3">Choice of programming language </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec4">Choice of programming language </h2>
|
||||
Python plays nowadays a central role in the development of machine
|
||||
learning techniques and tools for data analysis. In particular, seen
|
||||
the wealth of machine learning and data analysis packages written in
|
||||
Python, easy to use libraries with immediate visualization(and not the
|
||||
least impressive galleries of existing example), the popularity of the
|
||||
Jupyter notebook framework with the possibility to run <b>R</b> codes or
|
||||
compiled programs written in C++, and much more made our choice of
|
||||
programming language for this series of lectures of easy. However,
|
||||
since the focus here is not only on using existing Python tools such
|
||||
as <b>scikit-learn</b> or <b>tensorflow</b>, but also on developing your own
|
||||
algorithms and codes, we will as far as possible present many of these
|
||||
algorithms eithers a Python codes or C++ codes. Finally, we will, as
|
||||
far as possible keep parallel versions of the data analysis and
|
||||
machine larning programming aspects in <b>R</b> as
|
||||
well. <a href="https://www.r-project.org/" target="_blank">R</a> is a language and environment
|
||||
for statistical computing and graphics which is widely used in
|
||||
statistics and mathematics applications.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec5">Data handling, machine learning and ethical aspects </h2>
|
||||
The reason we also focus on compiled languages like C++ (or
|
||||
Fortran), is that Python is still notoriously slow when we do not
|
||||
utilize highly streamlined computational libraries like
|
||||
<a href="http://www.netlib.org/lapack/" target="_blank">Lapack</a> or other numerical libraries
|
||||
written in compiled languages (many of these libraries are written in
|
||||
Fortran). Although a project like <a href="https://numba.pydata.org/" target="_blank">Numba</a>
|
||||
holds great promise for speeding up the unrolling of lengthy loops, C+
|
||||
and Fortran are presently still the performance winners. Numba gives
|
||||
you potentially the power to speed up your applications with high
|
||||
performance functions written directly in Python. In particular,
|
||||
array-oriented and math-heavy Python code can achieve similar
|
||||
performance to C, C++ and Fortran. However, even with these speed-ups,
|
||||
for codes involving heavy Markov Chain Monte Carlo analyses and
|
||||
optimizations of cost functions, C++/C or Fortran codes tend to
|
||||
outperform Python codes.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec6">Acknowledgements </h2>
|
||||
Presently thus, the community tends to let
|
||||
code written in C++/C or Fortran do the heavy duty numerical
|
||||
number crunching and leave the post-analysis of the data to the above
|
||||
mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen
|
||||
the changes during the last decade, the above situation may change swiftly in the not too distant future.
|
||||
|
||||
<p>
|
||||
Many of the examples we discuss in this series of lectures come with
|
||||
existing data files or provide code examples which produce the data to
|
||||
be analyzed. Most of the applications we will discuss deal with
|
||||
small data sets (less than a terabyte of information) and can easily
|
||||
be analyzed and tested on standard off the shelf laptops you find in general
|
||||
grocery stores.
|
||||
|
||||
<h2 id="___sec4">Data handling, machine learning and ethical aspects </h2>
|
||||
|
||||
<p>
|
||||
In most of the cases we will study, we will either generate the data
|
||||
to analyze ourselves (both for supervised learning and unsupervised
|
||||
learning) or we will recur again and again to data present in say
|
||||
<b>scikit-learn</b> or <b>tensorflow</b>. Many of the examples we end up
|
||||
dealing with are from a privacy and data protection point of view,
|
||||
rather inoccuous and boring results of numerical
|
||||
calculations. However, this does not hinder us from developing a sound
|
||||
ethical attitude to the data we use, how we analyze the data and how
|
||||
we handle the data.
|
||||
|
||||
<p>
|
||||
The most immediate and simplest possible ethical aspects deal with our
|
||||
approach to the scientific process. Nowadays, with version control
|
||||
software like <a href="https://git-scm.com/" target="_blank">Git</a> and various online
|
||||
repositories like <a href="https://github.com/" target="_blank">Github</a>,
|
||||
<a href="https://about.gitlab.com/" target="_blank">Gitlab</a> etc, we can easily make our codes
|
||||
and data sets we have used, freely and easily accessible to a wider
|
||||
community. This helps us almost automagically in making our science
|
||||
reproducible. The large open-source development communities involved
|
||||
in say <a href="http://scikit-learn.org/stable/" target="_blank">Scikit-learn</a>,
|
||||
<a href="https://www.tensorflow.org/" target="_blank">Tensorflow</a>,
|
||||
<a href="http://pytorch.org/" target="_blank">PyTorch</a> and <a href="https://keras.io/" target="_blank">Keras</a>, are
|
||||
all excellent examples of this. The codes can be tested and improved
|
||||
upon continuosly, helping thereby our scientific community at large in
|
||||
developing data analysis and machine learning tools. It is much
|
||||
easier today to gain traction and acceptance for making your science
|
||||
reproducible. From a societal stand, this is an important element
|
||||
since many of the developers are employees of large public institutions like
|
||||
universities and research labs. Our taxpayer do deserve to get
|
||||
something back for their bucks.
|
||||
|
||||
<p>
|
||||
However, this more mechanical aspect of the ethics of science (in
|
||||
particular the reproducibility of scientific results) is something
|
||||
which is obvious and everybody should do as part of the dialectics of
|
||||
science. The fact that many scientists are not willing to share their codes or
|
||||
data is detrimental to the scientific discourse.
|
||||
|
||||
<p>
|
||||
Before we proceed, we should add a disclaimer. Even though
|
||||
we may dream of computers developing some kind of higher learning
|
||||
capabilities, at the end (even if the artificial intelligence
|
||||
community keeps touting our ears full of fancy futuristic avenues), it is we
|
||||
who end up constructing and instructing, via various algorithms, the
|
||||
computers. Self-driving cars for example, rely on sofisticated
|
||||
programs which take into account all possible situations a car can
|
||||
encounter. In addition, extensive usage of training datas from GPS
|
||||
information, maps etc, are typically fed into the software for
|
||||
self-driving cars. Adding to this various sensors and cameras that
|
||||
feed information to the programs, there are zillions of ethical issues
|
||||
which arise from this.
|
||||
|
||||
<p>
|
||||
For self-driving cars, where basically many of the standard machine
|
||||
learning algorithms discussed here enter into the codes, at a certain
|
||||
stage we have to make choices. Yes, we , the lads and lasses who wrote
|
||||
a program for a specific brand of a self-driving car. As an example,
|
||||
a most carmakers have as their utmost priority the security of the
|
||||
driver and the accompanying passengers. A famous carmaker, which is
|
||||
one of the leaders in the market of self-driving cars, had <b>if</b>
|
||||
statements of the following type: suppose there are two obstacles in
|
||||
front of you and you cannot avoid to collide with one of them. One of
|
||||
the obstacles is a monstertruck while the other one is a kindergarten
|
||||
class trying to cross the road. The self-driving car algo would then
|
||||
opt for the hitting the small folks instead of the monstertruck, since
|
||||
the likelihood of surving a collision with our future citizens, is
|
||||
much higher.
|
||||
|
||||
<p>
|
||||
This brings us leads then to serious ethical aspects. Why should we
|
||||
opt for such an option? Who decides and who is entitled to make such
|
||||
choices? Keep in mind that many of the algorithms you will about in
|
||||
this series of lectures or hear about later, are indeed based on
|
||||
simple programming instructions. And you are very likely to be one of
|
||||
the people who may end up writing such a code. Thus, developing a
|
||||
sound ethical attitude to what we do, an approach well beyond the
|
||||
simple mechanistic one of making our science available and
|
||||
reproducible, is much needed. The example of the self-driving cars is
|
||||
just one of infinitely many cases where we have to make choices. When
|
||||
you analyze data on economic inequalities, who guarantees that you are
|
||||
not weighting some data in a particular way, perhaps because you dearly want a
|
||||
specific conclusion which may support your political views?
|
||||
|
||||
<p>
|
||||
We do not have the answers here, but we want you think over these
|
||||
topics in a more overarching way. A statistical data analysis with
|
||||
its dry numbers and graphs meant to guide the eye, do not necessarily
|
||||
reflect the truth, whatever that is. As a scientist, and after a
|
||||
university education, you are supposedly a better citizen, with an
|
||||
improved critical view and understanding of the scientific method, and
|
||||
perhaps some deeper understandings of the ethics of science at
|
||||
large. Use these insights. Be a critical citizen. You owe it to our
|
||||
societies.
|
||||
|
||||
<p>
|
||||
To do: Add references and acknowledgements
|
||||
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
|
||||
@@ -4,79 +4,90 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<!-- dom:TITLE: Data Analysis and Machine Learning: Representing data -->\n",
|
||||
"# Data Analysis and Machine Learning: Representing data\n",
|
||||
"<!-- dom:TITLE: Introduction to Applied Data Analysis and Machine Learning -->\n",
|
||||
"# Introduction to Applied Data Analysis and Machine Learning\n",
|
||||
"<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
|
||||
"<!-- Author: --> \n",
|
||||
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
||||
"\n",
|
||||
"Date: **May 22, 2018**\n",
|
||||
"Date: **May 28, 2018**\n",
|
||||
"\n",
|
||||
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Introduction\n",
|
||||
"\n",
|
||||
"Statistics, data science and machine learning form important fields of\n",
|
||||
"research in modern science. They describe how to learn and make\n",
|
||||
"predictions from data, as well as allowing us to extract important\n",
|
||||
"correlations about physical process and the underlying laws of motion\n",
|
||||
"in large data sets. The latter, big data sets, appear\n",
|
||||
"frequently in essentially all disciplines, from the traditional Science,\n",
|
||||
"Technology, Mathematics and Engineering fields to Life Science, Law, education research, \n",
|
||||
"the Humanities and\n",
|
||||
"the Social Sciences. It has become more and more common to see\n",
|
||||
"research projects on big data in for example the Social\n",
|
||||
"Sciences where extracting patterns from complicated survey data is one of many research directions. \n",
|
||||
"Having a solid grasp of data analysis and machine learning\n",
|
||||
"is thus becoming central to scientific computing in many\n",
|
||||
"fields, and competences and skills within the fields of machine learning\n",
|
||||
"and scientific computing are nowadays strongly requested by many\n",
|
||||
"potential employers. The latter cannot be overstated, familiarity with\n",
|
||||
"machine learning has almost become a prerequisite for many of the most\n",
|
||||
"exciting employment opportunities, whether they are in bioinformatics,\n",
|
||||
"life science, physics or finance, in the private or the public\n",
|
||||
"sector. This author has had several students or met students who have\n",
|
||||
"been hired recently based on their skills and competences in\n",
|
||||
"scientific computing and data science, often with marginal knowledge\n",
|
||||
"of machine learning.\n",
|
||||
"in large data sets. The latter, big data sets, appear frequently in\n",
|
||||
"essentially all disciplines, from the traditional Science, Technology,\n",
|
||||
"Mathematics and Engineering fields to Life Science, Law, education\n",
|
||||
"research, the Humanities and the Social Sciences. \n",
|
||||
"\n",
|
||||
"It has become more\n",
|
||||
"and more common to see research projects on big data in for example\n",
|
||||
"the Social Sciences where extracting patterns from complicated survey\n",
|
||||
"data is one of many research directions. Having a solid grasp of data\n",
|
||||
"analysis and machine learning is thus becoming central to scientific\n",
|
||||
"computing in many fields, and competences and skills within the fields\n",
|
||||
"of machine learning and scientific computing are nowadays strongly\n",
|
||||
"requested by many potential employers. The latter cannot be\n",
|
||||
"overstated, familiarity with machine learning has almost become a\n",
|
||||
"prerequisite for many of the most exciting employment opportunities,\n",
|
||||
"whether they are in bioinformatics, life science, physics or finance,\n",
|
||||
"in the private or the public sector. This author has had several\n",
|
||||
"students or met students who have been hired recently based on their\n",
|
||||
"skills and competences in scientific computing and data science, often\n",
|
||||
"with marginal knowledge of machine learning.\n",
|
||||
"\n",
|
||||
"Machine learning is a subfield of computer science, and is closely\n",
|
||||
"related to computational statistics. It evolved from the study of\n",
|
||||
"pattern recognition in artificial intelligence (AI) research, and has\n",
|
||||
"made contributions to AI tasks like computer vision, natural language\n",
|
||||
"processing and speech recognition. \n",
|
||||
"Machine learning represents the\n",
|
||||
"science of giving computers the ability to learn without being\n",
|
||||
"explicitly programmed. The idea is that there exist generic\n",
|
||||
"algorithms which can be used to find patterns in a broad class of data\n",
|
||||
"sets without having to write code specifically for each problem. The\n",
|
||||
"algorithm will build its own logic based on the data. \n",
|
||||
"processing and speech recognition. Many of the methods we will study are also \n",
|
||||
"strongly rooted in basic mathematics and physics research. \n",
|
||||
"\n",
|
||||
"Machine learning is an extremely rich field, in spite of its young age. The\n",
|
||||
"increases we have seen during the last three decades in computational\n",
|
||||
"capabilities have been followed by developments of methods and\n",
|
||||
"techniques for analyzing and handling large date sets, relying heavily\n",
|
||||
"on statistics, computer science and mathematics. The field is rather\n",
|
||||
"new and developing rapidly. Popular software packages written in\n",
|
||||
"Python for machine learning like [Scikit-learn](http://scikit-learn.org/stable/), [Tensorflow](https://www.tensorflow.org/),\n",
|
||||
"[PyTorch](http://pytorch.org/) and [Keras](https://keras.io/), all freely available at their respective GitHub sites,\n",
|
||||
"encompass communities of developers in the thousands or more. And the number\n",
|
||||
"of code developers and contributors keeps increasing. Not all the\n",
|
||||
"Ideally, machine learning represents the science of giving computers\n",
|
||||
"the ability to learn without being explicitly programmed. The idea is\n",
|
||||
"that there exist generic algorithms which can be used to find patterns\n",
|
||||
"in a broad class of data sets without having to write code\n",
|
||||
"specifically for each problem. The algorithm will build its own logic\n",
|
||||
"based on the data. You should however always keep in mind that\n",
|
||||
"machines and algorithms are to a large extent developed by humans. The\n",
|
||||
"insights and knowledge we have about a specific system, play a central\n",
|
||||
"role when we develop a specific machine learning algorithm. \n",
|
||||
"\n",
|
||||
"Machine learning is an extremely rich field, in spite of its young\n",
|
||||
"age. The increases we have seen during the last three decades in\n",
|
||||
"computational capabilities have been followed by developments of\n",
|
||||
"methods and techniques for analyzing and handling large date sets,\n",
|
||||
"relying heavily on statistics, computer science and mathematics. The\n",
|
||||
"field is rather new and developing rapidly. Popular software packages\n",
|
||||
"written in Python for machine learning like\n",
|
||||
"[Scikit-learn](http://scikit-learn.org/stable/),\n",
|
||||
"[Tensorflow](https://www.tensorflow.org/),\n",
|
||||
"[PyTorch](http://pytorch.org/) and [Keras](https://keras.io/), all\n",
|
||||
"freely available at their respective GitHub sites, encompass\n",
|
||||
"communities of developers in the thousands or more. And the number of\n",
|
||||
"code developers and contributors keeps increasing. Not all the\n",
|
||||
"algorithms and methods can be given a rigorous mathematical\n",
|
||||
"justification, opening up thereby large rooms for experimenting\n",
|
||||
"and trial and error and thereby exciting new developments. \n",
|
||||
"However, a solid command of linear algebra, multivariate theory, \n",
|
||||
"probability theory, statistical data analysis,\n",
|
||||
"understanding errors and Monte Carlo methods are central elements in a proper understanding of many of \n",
|
||||
"algorithms and methods we will discuss. \n",
|
||||
"justification, opening up thereby large rooms for experimenting and\n",
|
||||
"trial and error and thereby exciting new developments. However, a\n",
|
||||
"solid command of linear algebra, multivariate theory, probability\n",
|
||||
"theory, statistical data analysis, understanding errors and Monte\n",
|
||||
"Carlo methods are central elements in a proper understanding of many\n",
|
||||
"of algorithms and methods we will discuss.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"<!-- !split -->\n",
|
||||
"## Learning outcomes\n",
|
||||
"\n",
|
||||
"These lectures aim at giving you an overview of central aspects of\n",
|
||||
"These setsof lectures aim at giving you an overview of central aspects of\n",
|
||||
"statistical data analysis as well as some of the central algorithms\n",
|
||||
"used in machine learning. We will introduce a variety of central\n",
|
||||
"algorithms and methods essential for studies of data analysis and\n",
|
||||
@@ -132,19 +143,22 @@
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Types of Machine Learning\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The approaches to machine learning are many, but are often split into two main categories. \n",
|
||||
"In *supervised learning* we know the answer to a problem,\n",
|
||||
"and let the computer deduce the logic behind it. On the other hand, *unsupervised learning*\n",
|
||||
"is a method for finding patterns and relationship in data sets without any prior knowledge of the system.\n",
|
||||
"Some authours also operate with a third category, namely *reinforcement learning*. This is a paradigm \n",
|
||||
"of learning inspired by behavioral psychology, where learning is achieved by trial-and-error, \n",
|
||||
"The approaches to machine learning are many, but are often split into\n",
|
||||
"two main categories. In *supervised learning* we know the answer to a\n",
|
||||
"problem, and let the computer deduce the logic behind it. On the other\n",
|
||||
"hand, *unsupervised learning* is a method for finding patterns and\n",
|
||||
"relationship in data sets without any prior knowledge of the system.\n",
|
||||
"Some authours also operate with a third category, namely\n",
|
||||
"*reinforcement learning*. This is a paradigm of learning inspired by\n",
|
||||
"behavioral psychology, where learning is achieved by trial-and-error,\n",
|
||||
"solely from rewards and punishment.\n",
|
||||
"\n",
|
||||
"Another way to categorize machine learning tasks is to consider the desired output of a system.\n",
|
||||
"Some of the most common tasks are:\n",
|
||||
"Another way to categorize machine learning tasks is to consider the\n",
|
||||
"desired output of a system. Some of the most common tasks are:\n",
|
||||
"\n",
|
||||
" * Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.\n",
|
||||
"\n",
|
||||
@@ -154,10 +168,13 @@
|
||||
"\n",
|
||||
"The methods we cover have three main topics in common, irrespective of\n",
|
||||
"whether we deal with supervised or unsupervised learning. The first\n",
|
||||
"ingredient is normally our data set, the second is a model which is\n",
|
||||
"normally a function of some parameters. The last ingredient is a\n",
|
||||
"so-called **cost** function which allows us to present an estimate on\n",
|
||||
"how good our model is in reproducing the data it is supposed to train.\n",
|
||||
"ingredient is normally our data set (which can be subdivided into\n",
|
||||
"training and test data), the second item is a model which is normally a\n",
|
||||
"function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model. \n",
|
||||
"\n",
|
||||
"The last ingredient is a so-called **cost**\n",
|
||||
"function which allows us to present an estimate on how good our model\n",
|
||||
"is in reproducing the data it is supposed to train. \n",
|
||||
"\n",
|
||||
"Here we will build our machine learning approach on elements of the\n",
|
||||
"statistical foundation discussed above, with elements from data\n",
|
||||
@@ -176,15 +193,150 @@
|
||||
"\n",
|
||||
"6. Artifical neural networks and deep learning\n",
|
||||
"\n",
|
||||
"## Why this text?\n",
|
||||
"\n",
|
||||
"7. Networks for unsupervised learning using for example reduced Boltzmann machines.\n",
|
||||
"\n",
|
||||
"## Choice of programming language\n",
|
||||
"\n",
|
||||
"Python plays nowadays a central role in the development of machine\n",
|
||||
"learning techniques and tools for data analysis. In particular, seen\n",
|
||||
"the wealth of machine learning and data analysis packages written in\n",
|
||||
"Python, easy to use libraries with immediate visualization(and not the\n",
|
||||
"least impressive galleries of existing example), the popularity of the\n",
|
||||
"Jupyter notebook framework with the possibility to run **R** codes or\n",
|
||||
"compiled programs written in C++, and much more made our choice of\n",
|
||||
"programming language for this series of lectures of easy. However,\n",
|
||||
"since the focus here is not only on using existing Python tools such\n",
|
||||
"as **scikit-learn** or **tensorflow**, but also on developing your own\n",
|
||||
"algorithms and codes, we will as far as possible present many of these\n",
|
||||
"algorithms eithers a Python codes or C++ codes. Finally, we will, as\n",
|
||||
"far as possible keep parallel versions of the data analysis and\n",
|
||||
"machine larning programming aspects in **R** as\n",
|
||||
"well. [R](https://www.r-project.org/) is a language and environment\n",
|
||||
"for statistical computing and graphics which is widely used in\n",
|
||||
"statistics and mathematics applications.\n",
|
||||
"\n",
|
||||
"The reason we also focus on compiled languages like C++ (or\n",
|
||||
"Fortran), is that Python is still notoriously slow when we do not\n",
|
||||
"utilize highly streamlined computational libraries like\n",
|
||||
"[Lapack](http://www.netlib.org/lapack/) or other numerical libraries\n",
|
||||
"written in compiled languages (many of these libraries are written in\n",
|
||||
"Fortran). Although a project like [Numba](https://numba.pydata.org/)\n",
|
||||
"holds great promise for speeding up the unrolling of lengthy loops, C+\n",
|
||||
"and Fortran are presently still the performance winners. Numba gives\n",
|
||||
"you potentially the power to speed up your applications with high\n",
|
||||
"performance functions written directly in Python. In particular,\n",
|
||||
"array-oriented and math-heavy Python code can achieve similar\n",
|
||||
"performance to C, C++ and Fortran. However, even with these speed-ups,\n",
|
||||
"for codes involving heavy Markov Chain Monte Carlo analyses and\n",
|
||||
"optimizations of cost functions, C++/C or Fortran codes tend to\n",
|
||||
"outperform Python codes. \n",
|
||||
"\n",
|
||||
"Presently thus, the community tends to let\n",
|
||||
"code written in C++/C or Fortran do the heavy duty numerical\n",
|
||||
"number crunching and leave the post-analysis of the data to the above\n",
|
||||
"mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen\n",
|
||||
"the changes during the last decade, the above situation may change swiftly in the not too distant future. \n",
|
||||
"\n",
|
||||
"Many of the examples we discuss in this series of lectures come with\n",
|
||||
"existing data files or provide code examples which produce the data to\n",
|
||||
"be analyzed. Most of the applications we will discuss deal with\n",
|
||||
"small data sets (less than a terabyte of information) and can easily\n",
|
||||
"be analyzed and tested on standard off the shelf laptops you find in general \n",
|
||||
"grocery stores.\n",
|
||||
"\n",
|
||||
"## Data handling, machine learning and ethical aspects\n",
|
||||
"\n",
|
||||
"In most of the cases we will study, we will either generate the data\n",
|
||||
"to analyze ourselves (both for supervised learning and unsupervised\n",
|
||||
"learning) or we will recur again and again to data present in say\n",
|
||||
"**scikit-learn** or **tensorflow**. Many of the examples we end up\n",
|
||||
"dealing with are from a privacy and data protection point of view,\n",
|
||||
"rather inoccuous and boring results of numerical\n",
|
||||
"calculations. However, this does not hinder us from developing a sound\n",
|
||||
"ethical attitude to the data we use, how we analyze the data and how\n",
|
||||
"we handle the data.\n",
|
||||
"\n",
|
||||
"## Acknowledgements"
|
||||
"The most immediate and simplest possible ethical aspects deal with our\n",
|
||||
"approach to the scientific process. Nowadays, with version control\n",
|
||||
"software like [Git](https://git-scm.com/) and various online\n",
|
||||
"repositories like [Github](https://github.com/),\n",
|
||||
"[Gitlab](https://about.gitlab.com/) etc, we can easily make our codes\n",
|
||||
"and data sets we have used, freely and easily accessible to a wider\n",
|
||||
"community. This helps us almost automagically in making our science\n",
|
||||
"reproducible. The large open-source development communities involved\n",
|
||||
"in say [Scikit-learn](http://scikit-learn.org/stable/),\n",
|
||||
"[Tensorflow](https://www.tensorflow.org/),\n",
|
||||
"[PyTorch](http://pytorch.org/) and [Keras](https://keras.io/), are\n",
|
||||
"all excellent examples of this. The codes can be tested and improved\n",
|
||||
"upon continuosly, helping thereby our scientific community at large in\n",
|
||||
"developing data analysis and machine learning tools. It is much\n",
|
||||
"easier today to gain traction and acceptance for making your science\n",
|
||||
"reproducible. From a societal stand, this is an important element\n",
|
||||
"since many of the developers are employees of large public institutions like\n",
|
||||
"universities and research labs. Our taxpayer do deserve to get\n",
|
||||
"something back for their bucks.\n",
|
||||
"\n",
|
||||
"However, this more mechanical aspect of the ethics of science (in\n",
|
||||
"particular the reproducibility of scientific results) is something\n",
|
||||
"which is obvious and everybody should do as part of the dialectics of\n",
|
||||
"science. The fact that many scientists are not willing to share their codes or \n",
|
||||
"data is detrimental to the scientific discourse.\n",
|
||||
"\n",
|
||||
"Before we proceed, we should add a disclaimer. Even though\n",
|
||||
"we may dream of computers developing some kind of higher learning\n",
|
||||
"capabilities, at the end (even if the artificial intelligence\n",
|
||||
"community keeps touting our ears full of fancy futuristic avenues), it is we\n",
|
||||
"who end up constructing and instructing, via various algorithms, the\n",
|
||||
"computers. Self-driving cars for example, rely on sofisticated\n",
|
||||
"programs which take into account all possible situations a car can\n",
|
||||
"encounter. In addition, extensive usage of training datas from GPS\n",
|
||||
"information, maps etc, are typically fed into the software for\n",
|
||||
"self-driving cars. Adding to this various sensors and cameras that\n",
|
||||
"feed information to the programs, there are zillions of ethical issues\n",
|
||||
"which arise from this.\n",
|
||||
"\n",
|
||||
"For self-driving cars, where basically many of the standard machine\n",
|
||||
"learning algorithms discussed here enter into the codes, at a certain\n",
|
||||
"stage we have to make choices. Yes, we , the lads and lasses who wrote\n",
|
||||
"a program for a specific brand of a self-driving car. As an example,\n",
|
||||
"a most carmakers have as their utmost priority the security of the\n",
|
||||
"driver and the accompanying passengers. A famous carmaker, which is\n",
|
||||
"one of the leaders in the market of self-driving cars, had **if**\n",
|
||||
"statements of the following type: suppose there are two obstacles in\n",
|
||||
"front of you and you cannot avoid to collide with one of them. One of\n",
|
||||
"the obstacles is a monstertruck while the other one is a kindergarten\n",
|
||||
"class trying to cross the road. The self-driving car algo would then\n",
|
||||
"opt for the hitting the small folks instead of the monstertruck, since\n",
|
||||
"the likelihood of surving a collision with our future citizens, is\n",
|
||||
"much higher.\n",
|
||||
"\n",
|
||||
"This brings us leads then to serious ethical aspects. Why should we\n",
|
||||
"opt for such an option? Who decides and who is entitled to make such\n",
|
||||
"choices? Keep in mind that many of the algorithms you will about in\n",
|
||||
"this series of lectures or hear about later, are indeed based on\n",
|
||||
"simple programming instructions. And you are very likely to be one of\n",
|
||||
"the people who may end up writing such a code. Thus, developing a\n",
|
||||
"sound ethical attitude to what we do, an approach well beyond the\n",
|
||||
"simple mechanistic one of making our science available and\n",
|
||||
"reproducible, is much needed. The example of the self-driving cars is\n",
|
||||
"just one of infinitely many cases where we have to make choices. When\n",
|
||||
"you analyze data on economic inequalities, who guarantees that you are\n",
|
||||
"not weighting some data in a particular way, perhaps because you dearly want a\n",
|
||||
"specific conclusion which may support your political views?\n",
|
||||
"\n",
|
||||
"We do not have the answers here, but we want you think over these\n",
|
||||
"topics in a more overarching way. A statistical data analysis with\n",
|
||||
"its dry numbers and graphs meant to guide the eye, do not necessarily\n",
|
||||
"reflect the truth, whatever that is. As a scientist, and after a\n",
|
||||
"university education, you are supposedly a better citizen, with an\n",
|
||||
"improved critical view and understanding of the scientific method, and\n",
|
||||
"perhaps some deeper understandings of the ethics of science at\n",
|
||||
"large. Use these insights. Be a critical citizen. You owe it to our\n",
|
||||
"societies.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"To do: Add references and acknowledgements"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
Binary file not shown.
Binary file not shown.
@@ -225,7 +225,7 @@ function $y$ to
|
||||
|
||||
!bt
|
||||
\[
|
||||
y = 10x+0.01N(0,1),
|
||||
y = 10x+0.01 \times N(0,1),
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -236,7 +236,7 @@ set. However, this testing 'by the eye' is obviouly not satisfactory in the
|
||||
long run. Here we have only defined the training data and our model, and
|
||||
have not discussed a more rigorous approach to the _cost_ function.
|
||||
|
||||
We need more rigorous criteria in defining whether we have succeed or
|
||||
We need more rigorous criteria in defining whether we have succeeded or
|
||||
not in modeling our training data. You will be surprised to see that
|
||||
many scientists seldomly venture beyond this 'by the eye' approach. A
|
||||
standard approach for the *cost* function is the so-called $\chi^2$
|
||||
|
||||
@@ -1,69 +1,79 @@
|
||||
TITLE: Data Analysis and Machine Learning: Representing data
|
||||
TITLE: Introduction to Applied Data Analysis and Machine Learning
|
||||
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
|
||||
DATE: today
|
||||
|
||||
|
||||
!split
|
||||
|
||||
===== Introduction =====
|
||||
|
||||
Statistics, data science and machine learning form important fields of
|
||||
research in modern science. They describe how to learn and make
|
||||
predictions from data, as well as allowing us to extract important
|
||||
correlations about physical process and the underlying laws of motion
|
||||
in large data sets. The latter, big data sets, appear
|
||||
frequently in essentially all disciplines, from the traditional Science,
|
||||
Technology, Mathematics and Engineering fields to Life Science, Law, education research,
|
||||
the Humanities and
|
||||
the Social Sciences. It has become more and more common to see
|
||||
research projects on big data in for example the Social
|
||||
Sciences where extracting patterns from complicated survey data is one of many research directions.
|
||||
Having a solid grasp of data analysis and machine learning
|
||||
is thus becoming central to scientific computing in many
|
||||
fields, and competences and skills within the fields of machine learning
|
||||
and scientific computing are nowadays strongly requested by many
|
||||
potential employers. The latter cannot be overstated, familiarity with
|
||||
machine learning has almost become a prerequisite for many of the most
|
||||
exciting employment opportunities, whether they are in bioinformatics,
|
||||
life science, physics or finance, in the private or the public
|
||||
sector. This author has had several students or met students who have
|
||||
been hired recently based on their skills and competences in
|
||||
scientific computing and data science, often with marginal knowledge
|
||||
of machine learning.
|
||||
in large data sets. The latter, big data sets, appear frequently in
|
||||
essentially all disciplines, from the traditional Science, Technology,
|
||||
Mathematics and Engineering fields to Life Science, Law, education
|
||||
research, the Humanities and the Social Sciences.
|
||||
|
||||
It has become more
|
||||
and more common to see research projects on big data in for example
|
||||
the Social Sciences where extracting patterns from complicated survey
|
||||
data is one of many research directions. Having a solid grasp of data
|
||||
analysis and machine learning is thus becoming central to scientific
|
||||
computing in many fields, and competences and skills within the fields
|
||||
of machine learning and scientific computing are nowadays strongly
|
||||
requested by many potential employers. The latter cannot be
|
||||
overstated, familiarity with machine learning has almost become a
|
||||
prerequisite for many of the most exciting employment opportunities,
|
||||
whether they are in bioinformatics, life science, physics or finance,
|
||||
in the private or the public sector. This author has had several
|
||||
students or met students who have been hired recently based on their
|
||||
skills and competences in scientific computing and data science, often
|
||||
with marginal knowledge of machine learning.
|
||||
|
||||
Machine learning is a subfield of computer science, and is closely
|
||||
related to computational statistics. It evolved from the study of
|
||||
pattern recognition in artificial intelligence (AI) research, and has
|
||||
made contributions to AI tasks like computer vision, natural language
|
||||
processing and speech recognition.
|
||||
Machine learning represents the
|
||||
science of giving computers the ability to learn without being
|
||||
explicitly programmed. The idea is that there exist generic
|
||||
algorithms which can be used to find patterns in a broad class of data
|
||||
sets without having to write code specifically for each problem. The
|
||||
algorithm will build its own logic based on the data.
|
||||
processing and speech recognition. Many of the methods we will study are also
|
||||
strongly rooted in basic mathematics and physics research.
|
||||
|
||||
Machine learning is an extremely rich field, in spite of its young age. The
|
||||
increases we have seen during the last three decades in computational
|
||||
capabilities have been followed by developments of methods and
|
||||
techniques for analyzing and handling large date sets, relying heavily
|
||||
on statistics, computer science and mathematics. The field is rather
|
||||
new and developing rapidly. Popular software packages written in
|
||||
Python for machine learning like "Scikit-learn":"http://scikit-learn.org/stable/", "Tensorflow":"https://www.tensorflow.org/",
|
||||
"PyTorch":"http://pytorch.org/" and "Keras":"https://keras.io/", all freely available at their respective GitHub sites,
|
||||
encompass communities of developers in the thousands or more. And the number
|
||||
of code developers and contributors keeps increasing. Not all the
|
||||
Ideally, machine learning represents the science of giving computers
|
||||
the ability to learn without being explicitly programmed. The idea is
|
||||
that there exist generic algorithms which can be used to find patterns
|
||||
in a broad class of data sets without having to write code
|
||||
specifically for each problem. The algorithm will build its own logic
|
||||
based on the data. You should however always keep in mind that
|
||||
machines and algorithms are to a large extent developed by humans. The
|
||||
insights and knowledge we have about a specific system, play a central
|
||||
role when we develop a specific machine learning algorithm.
|
||||
|
||||
Machine learning is an extremely rich field, in spite of its young
|
||||
age. The increases we have seen during the last three decades in
|
||||
computational capabilities have been followed by developments of
|
||||
methods and techniques for analyzing and handling large date sets,
|
||||
relying heavily on statistics, computer science and mathematics. The
|
||||
field is rather new and developing rapidly. Popular software packages
|
||||
written in Python for machine learning like
|
||||
"Scikit-learn":"http://scikit-learn.org/stable/",
|
||||
"Tensorflow":"https://www.tensorflow.org/",
|
||||
"PyTorch":"http://pytorch.org/" and "Keras":"https://keras.io/", all
|
||||
freely available at their respective GitHub sites, encompass
|
||||
communities of developers in the thousands or more. And the number of
|
||||
code developers and contributors keeps increasing. Not all the
|
||||
algorithms and methods can be given a rigorous mathematical
|
||||
justification, opening up thereby large rooms for experimenting
|
||||
and trial and error and thereby exciting new developments.
|
||||
However, a solid command of linear algebra, multivariate theory,
|
||||
probability theory, statistical data analysis,
|
||||
understanding errors and Monte Carlo methods are central elements in a proper understanding of many of
|
||||
algorithms and methods we will discuss.
|
||||
justification, opening up thereby large rooms for experimenting and
|
||||
trial and error and thereby exciting new developments. However, a
|
||||
solid command of linear algebra, multivariate theory, probability
|
||||
theory, statistical data analysis, understanding errors and Monte
|
||||
Carlo methods are central elements in a proper understanding of many
|
||||
of algorithms and methods we will discuss.
|
||||
|
||||
|
||||
!split
|
||||
===== Learning outcomes =====
|
||||
|
||||
These lectures aim at giving you an overview of central aspects of
|
||||
These setsof lectures aim at giving you an overview of central aspects of
|
||||
statistical data analysis as well as some of the central algorithms
|
||||
used in machine learning. We will introduce a variety of central
|
||||
algorithms and methods essential for studies of data analysis and
|
||||
@@ -113,20 +123,22 @@ machine learning.
|
||||
|
||||
|
||||
|
||||
!split
|
||||
|
||||
===== Types of Machine Learning =====
|
||||
|
||||
|
||||
The approaches to machine learning are many, but are often split into two main categories.
|
||||
In *supervised learning* we know the answer to a problem,
|
||||
and let the computer deduce the logic behind it. On the other hand, *unsupervised learning*
|
||||
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
|
||||
Some authours also operate with a third category, namely *reinforcement learning*. This is a paradigm
|
||||
of learning inspired by behavioral psychology, where learning is achieved by trial-and-error,
|
||||
The approaches to machine learning are many, but are often split into
|
||||
two main categories. In *supervised learning* we know the answer to a
|
||||
problem, and let the computer deduce the logic behind it. On the other
|
||||
hand, *unsupervised learning* is a method for finding patterns and
|
||||
relationship in data sets without any prior knowledge of the system.
|
||||
Some authours also operate with a third category, namely
|
||||
*reinforcement learning*. This is a paradigm of learning inspired by
|
||||
behavioral psychology, where learning is achieved by trial-and-error,
|
||||
solely from rewards and punishment.
|
||||
|
||||
Another way to categorize machine learning tasks is to consider the desired output of a system.
|
||||
Some of the most common tasks are:
|
||||
Another way to categorize machine learning tasks is to consider the
|
||||
desired output of a system. Some of the most common tasks are:
|
||||
|
||||
* Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.
|
||||
|
||||
@@ -137,10 +149,13 @@ Some of the most common tasks are:
|
||||
|
||||
The methods we cover have three main topics in common, irrespective of
|
||||
whether we deal with supervised or unsupervised learning. The first
|
||||
ingredient is normally our data set, the second is a model which is
|
||||
normally a function of some parameters. The last ingredient is a
|
||||
so-called _cost_ function which allows us to present an estimate on
|
||||
how good our model is in reproducing the data it is supposed to train.
|
||||
ingredient is normally our data set (which can be subdivided into
|
||||
training and test data), the second item is a model which is normally a
|
||||
function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.
|
||||
|
||||
The last ingredient is a so-called _cost_
|
||||
function which allows us to present an estimate on how good our model
|
||||
is in reproducing the data it is supposed to train.
|
||||
|
||||
Here we will build our machine learning approach on elements of the
|
||||
statistical foundation discussed above, with elements from data
|
||||
@@ -153,20 +168,149 @@ o Nearest neighbors models
|
||||
o Bayesian statistics and regression
|
||||
o Support vector machines and finally various variants of
|
||||
o Artifical neural networks and deep learning
|
||||
o Networks for unsupervised learning using for example reduced Boltzmann machines.
|
||||
|
||||
|
||||
|
||||
!split
|
||||
===== Why this text? =====
|
||||
|
||||
|
||||
!split
|
||||
===== Choice of programming language =====
|
||||
|
||||
!split
|
||||
Python plays nowadays a central role in the development of machine
|
||||
learning techniques and tools for data analysis. In particular, seen
|
||||
the wealth of machine learning and data analysis packages written in
|
||||
Python, easy to use libraries with immediate visualization(and not the
|
||||
least impressive galleries of existing example), the popularity of the
|
||||
Jupyter notebook framework with the possibility to run _R_ codes or
|
||||
compiled programs written in C++, and much more made our choice of
|
||||
programming language for this series of lectures of easy. However,
|
||||
since the focus here is not only on using existing Python tools such
|
||||
as _scikit-learn_ or _tensorflow_, but also on developing your own
|
||||
algorithms and codes, we will as far as possible present many of these
|
||||
algorithms eithers a Python codes or C++ codes. Finally, we will, as
|
||||
far as possible keep parallel versions of the data analysis and
|
||||
machine larning programming aspects in _R_ as
|
||||
well. "R":"https://www.r-project.org/" is a language and environment
|
||||
for statistical computing and graphics which is widely used in
|
||||
statistics and mathematics applications.
|
||||
|
||||
The reason we also focus on compiled languages like C++ (or
|
||||
Fortran), is that Python is still notoriously slow when we do not
|
||||
utilize highly streamlined computational libraries like
|
||||
"Lapack":"http://www.netlib.org/lapack/" or other numerical libraries
|
||||
written in compiled languages (many of these libraries are written in
|
||||
Fortran). Although a project like "Numba":"https://numba.pydata.org/"
|
||||
holds great promise for speeding up the unrolling of lengthy loops, C+
|
||||
and Fortran are presently still the performance winners. Numba gives
|
||||
you potentially the power to speed up your applications with high
|
||||
performance functions written directly in Python. In particular,
|
||||
array-oriented and math-heavy Python code can achieve similar
|
||||
performance to C, C++ and Fortran. However, even with these speed-ups,
|
||||
for codes involving heavy Markov Chain Monte Carlo analyses and
|
||||
optimizations of cost functions, C++/C or Fortran codes tend to
|
||||
outperform Python codes.
|
||||
|
||||
Presently thus, the community tends to let
|
||||
code written in C++/C or Fortran do the heavy duty numerical
|
||||
number crunching and leave the post-analysis of the data to the above
|
||||
mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen
|
||||
the changes during the last decade, the above situation may change swiftly in the not too distant future.
|
||||
|
||||
Many of the examples we discuss in this series of lectures come with
|
||||
existing data files or provide code examples which produce the data to
|
||||
be analyzed. Most of the applications we will discuss deal with
|
||||
small data sets (less than a terabyte of information) and can easily
|
||||
be analyzed and tested on standard off the shelf laptops you find in general
|
||||
grocery stores.
|
||||
|
||||
===== Data handling, machine learning and ethical aspects =====
|
||||
|
||||
In most of the cases we will study, we will either generate the data
|
||||
to analyze ourselves (both for supervised learning and unsupervised
|
||||
learning) or we will recur again and again to data present in say
|
||||
_scikit-learn_ or _tensorflow_. Many of the examples we end up
|
||||
dealing with are from a privacy and data protection point of view,
|
||||
rather inoccuous and boring results of numerical
|
||||
calculations. However, this does not hinder us from developing a sound
|
||||
ethical attitude to the data we use, how we analyze the data and how
|
||||
we handle the data.
|
||||
|
||||
The most immediate and simplest possible ethical aspects deal with our
|
||||
approach to the scientific process. Nowadays, with version control
|
||||
software like "Git":"https://git-scm.com/" and various online
|
||||
repositories like "Github":"https://github.com/",
|
||||
"Gitlab":"https://about.gitlab.com/" etc, we can easily make our codes
|
||||
and data sets we have used, freely and easily accessible to a wider
|
||||
community. This helps us almost automagically in making our science
|
||||
reproducible. The large open-source development communities involved
|
||||
in say "Scikit-learn":"http://scikit-learn.org/stable/",
|
||||
"Tensorflow":"https://www.tensorflow.org/",
|
||||
"PyTorch":"http://pytorch.org/" and "Keras":"https://keras.io/", are
|
||||
all excellent examples of this. The codes can be tested and improved
|
||||
upon continuosly, helping thereby our scientific community at large in
|
||||
developing data analysis and machine learning tools. It is much
|
||||
easier today to gain traction and acceptance for making your science
|
||||
reproducible. From a societal stand, this is an important element
|
||||
since many of the developers are employees of large public institutions like
|
||||
universities and research labs. Our taxpayer do deserve to get
|
||||
something back for their bucks.
|
||||
|
||||
However, this more mechanical aspect of the ethics of science (in
|
||||
particular the reproducibility of scientific results) is something
|
||||
which is obvious and everybody should do as part of the dialectics of
|
||||
science. The fact that many scientists are not willing to share their codes or
|
||||
data is detrimental to the scientific discourse.
|
||||
|
||||
Before we proceed, we should add a disclaimer. Even though
|
||||
we may dream of computers developing some kind of higher learning
|
||||
capabilities, at the end (even if the artificial intelligence
|
||||
community keeps touting our ears full of fancy futuristic avenues), it is we
|
||||
who end up constructing and instructing, via various algorithms, the
|
||||
computers. Self-driving cars for example, rely on sofisticated
|
||||
programs which take into account all possible situations a car can
|
||||
encounter. In addition, extensive usage of training datas from GPS
|
||||
information, maps etc, are typically fed into the software for
|
||||
self-driving cars. Adding to this various sensors and cameras that
|
||||
feed information to the programs, there are zillions of ethical issues
|
||||
which arise from this.
|
||||
|
||||
For self-driving cars, where basically many of the standard machine
|
||||
learning algorithms discussed here enter into the codes, at a certain
|
||||
stage we have to make choices. Yes, we , the lads and lasses who wrote
|
||||
a program for a specific brand of a self-driving car. As an example,
|
||||
a most carmakers have as their utmost priority the security of the
|
||||
driver and the accompanying passengers. A famous carmaker, which is
|
||||
one of the leaders in the market of self-driving cars, had _if_
|
||||
statements of the following type: suppose there are two obstacles in
|
||||
front of you and you cannot avoid to collide with one of them. One of
|
||||
the obstacles is a monstertruck while the other one is a kindergarten
|
||||
class trying to cross the road. The self-driving car algo would then
|
||||
opt for the hitting the small folks instead of the monstertruck, since
|
||||
the likelihood of surving a collision with our future citizens, is
|
||||
much higher.
|
||||
|
||||
This brings us leads then to serious ethical aspects. Why should we
|
||||
opt for such an option? Who decides and who is entitled to make such
|
||||
choices? Keep in mind that many of the algorithms you will about in
|
||||
this series of lectures or hear about later, are indeed based on
|
||||
simple programming instructions. And you are very likely to be one of
|
||||
the people who may end up writing such a code. Thus, developing a
|
||||
sound ethical attitude to what we do, an approach well beyond the
|
||||
simple mechanistic one of making our science available and
|
||||
reproducible, is much needed. The example of the self-driving cars is
|
||||
just one of infinitely many cases where we have to make choices. When
|
||||
you analyze data on economic inequalities, who guarantees that you are
|
||||
not weighting some data in a particular way, perhaps because you dearly want a
|
||||
specific conclusion which may support your political views?
|
||||
|
||||
We do not have the answers here, but we want you think over these
|
||||
topics in a more overarching way. A statistical data analysis with
|
||||
its dry numbers and graphs meant to guide the eye, do not necessarily
|
||||
reflect the truth, whatever that is. As a scientist, and after a
|
||||
university education, you are supposedly a better citizen, with an
|
||||
improved critical view and understanding of the scientific method, and
|
||||
perhaps some deeper understandings of the ethics of science at
|
||||
large. Use these insights. Be a critical citizen. You owe it to our
|
||||
societies.
|
||||
|
||||
!split
|
||||
===== Acknowledgements =====
|
||||
|
||||
|
||||
To do: Add references and acknowledgements
|
||||
|
||||
@@ -49,29 +49,6 @@ system doconce format html $name --html_style=bootstrap --pygments_html_style=de
|
||||
# IPython notebook
|
||||
system doconce format ipynb $name $opt
|
||||
|
||||
# LaTeX Beamer slides
|
||||
beamertheme=red_plain
|
||||
system doconce format pdflatex $name --latex_title_layout=beamer --latex_table_format=footnotesize $opt
|
||||
system doconce ptex2tex $name envir=minted
|
||||
# Add special packages
|
||||
doconce subst "% Add user's preamble" "\g<1>\n\\usepackage{simplewick}" $name.tex
|
||||
system doconce slides_beamer $name --beamer_slide_theme=$beamertheme
|
||||
system pdflatex -shell-escape ${name}
|
||||
system pdflatex -shell-escape ${name}
|
||||
cp $name.pdf ${name}-beamer.pdf
|
||||
cp $name.tex ${name}-beamer.tex
|
||||
|
||||
# Handouts
|
||||
system doconce format pdflatex $name --latex_title_layout=beamer --latex_table_format=footnotesize $opt
|
||||
system doconce ptex2tex $name envir=minted
|
||||
# Add special packages
|
||||
doconce subst "% Add user's preamble" "\g<1>\n\\usepackage{simplewick}" $name.tex
|
||||
system doconce slides_beamer $name --beamer_slide_theme=red_shadow --handout
|
||||
system pdflatex -shell-escape $name
|
||||
pdflatex -shell-escape $name
|
||||
pdflatex -shell-escape $name
|
||||
pdfnup --nup 2x3 --frame true --delta "1cm 1cm" --scale 0.9 --outfile ${name}-beamer-handouts2x3.pdf ${name}.pdf
|
||||
rm -f ${name}.pdf
|
||||
|
||||
# Ordinary plain LaTeX document
|
||||
rm -f *.aux # important after beamer
|
||||
|
||||
Reference in New Issue
Block a user