lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week34

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-08-26 09:03:43 +02:00
parent e514b736c5
commit f373c9bed0
41 changed files with 4157 additions and 2836 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week34/week34.do.txt
+244 -13
View File
@@ -1,4 +1,4 @@
TITLE: Week 34: Introduction to the course, Logistics and Practicalities
TITLE: Week 35: Introduction to the course, Logistics and Practicalities
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
@@ -187,7 +187,7 @@ These topics will be scattered thorughout the course and may not necessarily be
!bblock Statistical analysis and optimization of data
The following topics will be covered
We plan to cover the following topics:
* Basic concepts, expectation values, variance, covariance, correlation functions and errors;
* Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;
* Central elements of Bayesian statistics and modeling;
@@ -376,16 +376,106 @@ desired output of a system. Some of the most common tasks are:
* Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.
The methods we cover have three main topics in common, irrespective of
whether we deal with supervised or unsupervised learning. The first
ingredient is normally our data set (which can be subdivided into
training and test data), the second item is a model which is normally a
function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.
!split
===== Essential elements of ML =====
The methods we cover have three main topics in common, irrespective of
whether we deal with supervised or unsupervised learning.
!bpop
* The first ingredient is normally our data set (which can be subdivided into training, validation and test data). Many find the most difficult part of using Machine Learning to be the set up of your data in a meaningful way.
* The second item is a model which is normally a function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.
* The last ingredient is a so-called _cost/loss_ function (or error or risk function) which allows us to present an estimate on how good our model is in reproducing the data it is supposed to train.
!epop
!split
===== An optimization/minimization problem =====
At the heart of basically all Machine Learning algorithms we will encounter so-called minimization or optimization algorithms. A large family of such methods are so-called _gradient methods_.
!split
===== A Frequentist approach to data analysis =====
When you hear phrases like _predictions and estimations_ and
_correlations and causations_, what do you think of? May be you think
of the difference between classifying new data points and generating
new data points.
Or perhaps you consider that correlations represent some kind of symmetric statements like
if $A$ is correlated with $B$, then $B$ is correlated with
$A$. Causation on the other hand is directional, that is if $A$ causes $B$, $B$ does not
necessarily cause $A$.
These concepts are in some sense the difference between machine
learning and statistics. In machine learning and prediction based
tasks, we are often interested in developing algorithms that are
capable of learning patterns from given data in an automated fashion,
and then using these learned patterns to make predictions or
assessments of newly given data. In many cases, our primary concern
is the quality of the predictions or assessments, and we are less
concerned about the underlying patterns that were learned in order
to make these predictions.
In machine learning we normally use "a so-called frequentist approach":"https://en.wikipedia.org/wiki/Frequentist_inference",
where the aim is to make predictions and find correlations. We focus
less on for example extracting a probability distribution function (PDF). The PDF can be
used in turn to make estimations and find causations such as given $A$
what is the likelihood of finding $B$.
!split
===== What is a good model? =====
In science and engineering we often end up in situations where we want to infer (or learn) a
quantitative model $M$ for a given set of sample points $\bm{X} \in [x_1, x_2,\dots x_N]$.
As we will see repeatedely in these lectures, we could try to fit these data points to a model given by a
straight line, or if we wish to be more sophisticated to a more complex
function.
The reason for inferring such a model is that it
serves many useful purposes. On the one hand, the model can reveal information
encoded in the data or underlying mechanisms from which the data were generated. For instance, we could discover important
corelations that relate interesting physics interpretations.
In addition, it can simplify the representation of the given data set and help
us in making predictions about future data samples.
A first important consideration to keep in mind is that inferring the *correct* model
for a given data set is an elusive, if not impossible, task. The fundamental difficulty
is that if we are not specific about what we mean by a *correct* model, there
could easily be many different models that fit the given data set *equally well*.
!split
===== What is a good model? Can we define it? =====
The central question is this: what leads us to say that a model is correct or
optimal for a given data set? To make the model inference problem well posed, i.e.,
to guarantee that there is a unique optimal model for the given data, we need to
impose additional assumptions or restrictions on the class of models considered. To
this end, we should not be looking for just any model that can describe the data.
Instead, we should look for a _model_ $M$ that is the best among a restricted class
of models. In addition, to make the model inference problem computationally
tractable, we need to specify how restricted the class of models needs to be. A
common strategy is to start
with the simplest possible class of models that is just necessary to describe the data
or solve the problem at hand. More precisely, the model class should be rich enough
to contain at least one model that can fit the data to a desired accuracy and yet be
restricted enough that it is relatively simple to find the best model for the given data.
Thus, the most popular strategy is to start from the
simplest class of models and increase the complexity of the models only when the
simpler models become inadequate. For instance, if we work with a regression problem to fit a set of sample points, one
may first try the simplest class of models, namely linear models, followed obviously by more complex models.
How to evaluate which model fits best the data is something we will come back to over and over again in these sets of lectures.
The last ingredient is a so-called _cost_
function which allows us to present an estimate on how good our model
is in reproducing the data it is supposed to train.
At the heart of basically all ML algorithms there are so-called minimization algorithms, often we end up with various variants of _gradient_ methods.
@@ -897,9 +987,9 @@ For multidimensional arrays, we recommend strongly "xarray":"http://xarray.pydat
!split
===== Friday August 21 =====
===== Friday August 27 =====
"Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/LectureAug21.mp4?vrtx=view-as-webpage" and "Handwritten notes":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesAugust21.pdf"
"Video of Lecture from fall 2020":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/LectureAug21.mp4?vrtx=view-as-webpage" and "Handwritten notes":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesAugust21.pdf"
@@ -1599,5 +1689,146 @@ Machine Learning algorithms for supervised learning. Later we will meet _Tensorf
Now it is time to dive more into the details of various methods. We will start with linear regression and try to take a deeper look at what it entails.
!split
===== Exercise: Setting up various Python environments =====
The first exercise here is of a mere technical art. We want you to have
* git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo "GitHub facilities":"https://www.uio.no/tjenester/it/maskin/filer/versjonskontroll/github.html".
* Install various Python packages
We will make extensive use of Python as programming language and its
myriad of available libraries. You will find
IPython/Jupyter notebooks invaluable in your work. You can run _R_
codes in the Jupyter/IPython notebooks, with the immediate benefit of
visualizing your data. You can also use compiled languages like C++,
Rust, Fortran etc if you prefer. The focus in these lectures will be
on Python.
If you have Python installed (we recommend Python3) and you feel
pretty familiar with installing different packages, we recommend that
you install the following Python packages via _pip_ as
o pip install numpy scipy matplotlib ipython scikit-learn sympy pandas pillow
For _Tensorflow_, we recommend following the instructions in the text of
"Aurelien Geron, Hands‑On Machine Learning with Scikit‑Learn and TensorFlow, O'Reilly":"http://shop.oreilly.com/product/0636920052289.do"
We will come back to _tensorflow_ later.
For Python3, replace _pip_ with _pip3_.
For OSX users we recommend, after having installed Xcode, to
install _brew_. Brew allows for a seamless installation of additional
software via for example
o brew install python3
For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,
you can use _pip_ as well and simply install Python as
o sudo apt-get install python3 (or python for Python2.7)
If you don't want to perform these operations separately and venture
into the hassle of exploring how to set up dependencies and paths, we
recommend two widely used distrubutions which set up all relevant
dependencies for Python, namely
* "Anaconda":"https://docs.anaconda.com/",
which is an open source
distribution of the Python and R programming languages for large-scale
data processing, predictive analytics, and scientific computing, that
aims to simplify package management and deployment. Package versions
are managed by the package management system _conda_.
* "Enthought canopy":"https://www.enthought.com/product/canopy/"
is a Python
distribution for scientific and analytic computing distribution and
analysis environment, available for free and under a commercial
license.
We recommend using _Anaconda_.
===== Exercise: making your own data and exploring scikit-learn =====
We will generate our own dataset for a function $y(x)$ where $x \in [0,1]$ and defined by random numbers computed with the uniform distribution. The function $y$ is a quadratic polynomial in $x$ with added stochastic noise according to the normal distribution $\cal {N}(0,1)$.
The following simple Python instructions define our $x$ and $y$ values (with 100 data points).
!bc pycod
x = np.random.rand(100,1)
y = 2.0+5*x*x+0.1*np.random.randn(100,1)
!ec
o Write your own code (following the examples under the "regression slides":"https://compphysics.github.io/MachineLearningECT/doc/pub/Day1/html/Day1-bs.html") for computing the parametrization of the data set fitting a second-order polynomial.
o Use thereafter _scikit-learn_ (see again the examples in the regression slides) and compare with your own code.
o Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
!bt
\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
\]
!et
and the $R^2$ score function.
If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
!bt
\[
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
\]
!et
where we have defined the mean value of $\hat{y}$ as
!bt
\[
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
\]
!et
You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions.
Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.
!bsol
The code here is an example of where we define our own design matrix and fit parameters $\beta$.
!bc pycod
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
def save_fig(fig_id):
plt.savefig(image_path(fig_id) + ".png", format='png')
def R2(y_data, y_model):
return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
def MSE(y_data,y_model):
n = np.size(y_model)
return np.sum((y_data-y_model)**2)/n
x = np.random.rand(100)
y = 2.0+5*x*x+0.1*np.random.randn(100)
# The design matrix now as function of a given polynomial
X = np.zeros((len(x),3))
X[:,0] = 1.0
X[:,1] = x
X[:,2] = x**2
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# matrix inversion to find beta
beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
print(beta)
# and then make the prediction
ytilde = X_train @ beta
print("Training R2")
print(R2(y_train,ytilde))
print("Training MSE")
print(MSE(y_train,ytilde))
ypredict = X_test @ beta
print("Test R2")
print(R2(y_test,ypredict))
print("Test MSE")
print(MSE(y_test,ypredict))
!ec
!esol