lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update week34.do.txt

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-08-20 20:52:00 +02:00
parent aba7c2cd5e
commit 5b55f9a5c3
1 changed files with 6 additions and 267 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week34/week34.do.txt
+6 -267
View File
@@ -635,77 +635,6 @@ developed in the 1970s, namely EISPACK and LINPACK. We describe them shortly he
* LAPACK:package for solving symmetric, unsymmetric and generalized eigenvalue problems. From LAPACK's website URL: "http://www.netlib.org" it is possible to download for free all source codes from this library. Both C/C++ and Fortran versions are available.
* BLAS (I, II and III): (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations. Blas I is vector operations, II vector-matrix operations and III matrix-matrix operations. Highly parallelized and efficient codes, all available for download from URL: "http://www.netlib.org".
!split
===== Basic Matrix Features =====
!bblock Matrix properties reminder
!bt
\[
\mathbf{A} =
\begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\
a_{21} & a_{22} & a_{23} & a_{24} \\
a_{31} & a_{32} & a_{33} & a_{34} \\
a_{41} & a_{42} & a_{43} & a_{44}
\end{bmatrix}\qquad
\mathbf{I} =
\begin{bmatrix} 1 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 \\
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 1
\end{bmatrix}
\]
!et
The inverse of a matrix is defined by
!bt
\[
\mathbf{A}^{-1} \cdot \mathbf{A} = I
\]
!et
|----------------------------------------------------------------------|
| Relations | Name | matrix elements |
|----------------------------------------------------------------------|
| $A=A^{T}$ | symmetric | $a_{ij}=a_{ji}$ |
| $A=\left (A^{T}\right )^{-1}$ | real orthogonal | $\sum_k a_{ik}a_{jk}=\sum_k a_{ki} a_{kj}=\delta_{ij}$ |
| $A=A^*$ | real matrix | $a_{ij}=a_{ij}^*$ |
| $A=A^{\dagger}$ | hermitian | $a_{ij}=a_{ji}^*$ |
| $A=\left(A^{\dagger}\right )^{-1}$ | unitary | $\sum_k a_{ik}a_{jk}^*=\sum_k a_{ki}^* a_{kj}=\delta_{ij}$ |
|----------------------------------------------------------------------|
!eblock
!split
=== Some famous Matrices ===
* Diagonal if $a_{ij}=0$ for $i\ne j$
* Upper triangular if $a_{ij}=0$ for $i>j$
* Lower triangular if $a_{ij}=0$ for $i<j$
* Upper Hessenberg if $a_{ij}=0$ for $i>j+1$
* Lower Hessenberg if $a_{ij}=0$ for $i<j+1$
* Tridiagonal if $a_{ij}=0$ for $|i -j|>1$
* Lower banded with bandwidth $p$: $a_{ij}=0$ for $i>j+p$
* Upper banded with bandwidth $p$: $a_{ij}=0$ for $i<j+p$
* Banded, block upper triangular, block lower triangular....
!split
=== More Basic Matrix Features ===
!bblock Some Equivalent Statements
For an $N\times N$ matrix $\mathbf{A}$ the following properties are all equivalent
* If the inverse of $\mathbf{A}$ exists, $\mathbf{A}$ is nonsingular.
* The equation $\mathbf{Ax}=0$ implies $\mathbf{x}=0$.
* The rows of $\mathbf{A}$ form a basis of $R^N$.
* The columns of $\mathbf{A}$ form a basis of $R^N$.
* $\mathbf{A}$ is a product of elementary matrices.
* $0$ is not eigenvalue of $\mathbf{A}$.
!eblock
!split
===== Numpy and arrays =====
@@ -997,17 +926,6 @@ As we will see below it leads also to a very concice code close to the mathemati
For multidimensional arrays, we recommend strongly "xarray":"http://xarray.pydata.org/en/stable/". _xarray_ has much of the same flexibility as _pandas_, but allows for the extension to higher dimensions than two. We will see examples later of the usage of both _pandas_ and _xarray_.
!split
===== Friday August 27 =====
"Video of Lecture August 27, 2021":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h21/forelesningsvideoer/LectureThursdayAugust27.mp4?vrtx=view-as-webpage
"Video of Lecture from fall 2020":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/LectureAug21.mp4?vrtx=view-as-webpage" and "Handwritten notes":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesAugust21.pdf"
!split
@@ -1283,42 +1201,7 @@ Here $\bm{a}=\bm{y} - \bm{\tilde{y}}$.
We will discuss in more detail these and other functions in the
various lectures. We conclude this part with another example. Instead
of a linear $x$-dependence we study now a cubic polynomial and use the
polynomial regression analysis tools of scikit-learn.
!bc pycod
import matplotlib.pyplot as plt
import numpy as np
import random
from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LinearRegression
x=np.linspace(0.02,0.98,200)
noise = np.asarray(random.sample((range(200)),200))
y=x**3*noise
yn=x**3*100
poly3 = PolynomialFeatures(degree=3)
X = poly3.fit_transform(x[:,np.newaxis])
clf3 = LinearRegression()
clf3.fit(X,y)
Xplot=poly3.fit_transform(x[:,np.newaxis])
poly3_plot=plt.plot(x, clf3.predict(Xplot), label='Cubic Fit')
plt.plot(x,yn, color='red', label="True Cubic")
plt.scatter(x, y, label='Data', color='orange', s=15)
plt.legend()
plt.show()
def error(a):
for i in y:
err=(y-yn)/yn
return abs(np.sum(err))/len(err)
print (error(y))
!ec
various lectures and lab sessions.
@@ -1578,44 +1461,6 @@ plt.show()
!ec
=== Seeing the wood for the trees ===
As a teaser, let us now see how we can do this with decision trees using _scikit-learn_. Later we will switch to so-called _random forests_!
!bc pycod
#Decision Tree Regression
from sklearn.tree import DecisionTreeRegressor
regr_1=DecisionTreeRegressor(max_depth=5)
regr_2=DecisionTreeRegressor(max_depth=7)
regr_3=DecisionTreeRegressor(max_depth=9)
regr_1.fit(X, Energies)
regr_2.fit(X, Energies)
regr_3.fit(X, Energies)
y_1 = regr_1.predict(X)
y_2 = regr_2.predict(X)
y_3=regr_3.predict(X)
Masses['Eapprox'] = y_3
# Plot the results
plt.figure()
plt.plot(A, Energies, color="blue", label="Data", linewidth=2)
plt.plot(A, y_1, color="red", label="max_depth=5", linewidth=2)
plt.plot(A, y_2, color="green", label="max_depth=7", linewidth=2)
plt.plot(A, y_3, color="m", label="max_depth=9", linewidth=2)
plt.xlabel("$A$")
plt.ylabel("$E$[MeV]")
plt.title("Decision Tree Regression")
plt.legend()
save_fig("Masses2016Trees")
plt.show()
print(Masses)
print(np.mean( (Energies-y_1)**2))
!ec
=== And what about using neural networks? ===
The _seaborn_ package allows us to visualize data in an efficient way. Note that we use _scikit-learn_'s multi-layer perceptron (or feed forward neural network)
@@ -1735,7 +1580,7 @@ Linear regression gives us a set of analytical equations for the parameters $\be
!split
===== Examples =====
!bblock
In order to understand the relation among the predictors $p$, the set of data $n$ and the target (outcome, output etc) $\bm{y}$,
In order to understand the relation among the predictors (or features or properties) $p$, the set of data $n$ and the target (outcome, output etc) $\bm{y}$,
consider the model we discussed for describing nuclear binding energies.
There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model.
@@ -2131,29 +1976,10 @@ _Small question_: Do you think the example we have at hand here (the nuclear bin
!split
===== Some useful matrix and vector expressions =====
The following matrix and vector relation will be useful here and for the rest of the course. Vectors are always written as boldfaced lower case letters and
matrices as upper case boldfaced letters.
See the handwritten notes at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2022/NotesExercise5Week452022.pdf"
These notes will be discussed during one of the lectures.
!bt
\[
\frac{\partial (\bm{b}^T\bm{a})}{\partial \bm{a}} = \bm{b},
\]
!et
!bt
\[
\frac{\partial (\bm{a}^T\bm{A}\bm{a})}{\partial \bm{a}} = (\bm{A}+\bm{A}^T)\bm{a},
\]
!et
!bt
\[
\frac{\partial tr(\bm{B}\bm{A})}{\partial \bm{A}} = \bm{B}^T,
\]
!et
!bt
\[
\frac{\partial \log{\vert\bm{A}\vert}}{\partial \bm{A}} = (\bm{A}^{-1})^T.
\]
!et
!split
===== Interpretations and optimizing our parameters =====
!bblock
@@ -2637,7 +2463,7 @@ print(MSE(y_test,ypredict))
!split
===== Exercises =====
Here are three possible exercises for weeks 34 and 35.
Here are three possible exercises for week 34
===== Exercise: Setting up various Python environments =====
@@ -2784,90 +2610,3 @@ print(MSE(y_test,ypredict))
!esol
===== Exercise: Normalizing our data =====
A much used approach before starting to train the data is to preprocess our
data. Normally the data may need a rescaling and/or may be sensitive
to extreme values. Scaling the data renders our inputs much more
suitable for the algorithms we want to employ.
_Scikit-Learn_ has several functions which allow us to rescale the
data, normally resulting in much better results in terms of various
accuracy scores. The _StandardScaler_ function in _Scikit-Learn_
ensures that for each feature/predictor we study the mean value is
zero and the variance is one (every column in the design/feature
matrix). This scaling has the drawback that it does not ensure that
we have a particular maximum or minimum in our data set. Another
function included in _Scikit-Learn_ is the _MinMaxScaler_ which
ensures that all features are exactly between $0$ and $1$. The
The _Normalizer_ scales each data
point such that the feature vector has a euclidean length of one. In other words, it
projects a data point on the circle (or sphere in the case of higher dimensions) with a
radius of 1. This means every data point is scaled by a different number (by the
inverse of it’s length).
This normalization is often used when only the direction (or angle) of the data matters,
not the length of the feature vector.
The _RobustScaler_ works similarly to the StandardScaler in that it
ensures statistical properties for each feature that guarantee that
they are on the same scale. However, the RobustScaler uses the median
and quartiles, instead of mean and variance. This makes the
RobustScaler ignore data points that are very different from the rest
(like measurement errors). These odd data points are also called
outliers, and might often lead to trouble for other scaling
techniques.
It also common to split the data in a _training_ set and a _testing_ set. A typical split is to use $80\%$ of the data for training and the rest
for testing. This can be done as follows with our design matrix $\bm{X}$ and data $\bm{y}$ (remember to import _scikit-learn_)
!bc pycod
# split in training and test data
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)
!ec
Then we can use the standard scaler to scale our data as
!bc pycod
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
!ec
In this exercise we want you to to compute the MSE for the training
data and the test data as function of the complexity of a polynomial,
that is the degree of a given polynomial. We want you also to compute the $R2$ score as function of the complexity of the model for both training data and test data. You should also run the calculation with and without scaling.
One of
the aims is to reproduce Figure 2.11 of "Hastie et al":"https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf".
Our data is defined by $x\in [-3,3]$ with a total of for example $100$ data points.
!bc pycod
np.random.seed()
n = 100
maxdegree = 14
# Make data set.
x = np.linspace(-3, 3, n).reshape(-1, 1)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
!ec
where $y$ is the function we want to fit with a given polynomial.
!bsubex
Write a first code which sets up a design matrix $X$ defined by a fifth-order polynomial. Scale your data and split it in training and test data.
!esubex
!bsubex
Perform an ordinary least squares and compute the means squared error and the $R2$ factor for the training data and the test data, with and without scaling.
!esubex
!bsubex
Add now a model which allows you to make polynomials up to degree $15$. Perform a standard OLS fitting of the training data and compute the MSE and $R2$ for the training and test data and plot both test and training data MSE and $R2$ as functions of the polynomial degree. Compare what you see with Figure 2.11 of Hastie et al. Comment your results. For which polynomial degree do you find an optimal MSE (smallest value)?
!esubex