lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-13 14:00:16 +02:00
parent 1af325e7fb
commit 30d664a82b
9 changed files with 2392 additions and 942 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/chapter1.ipynb
+200 -166
View File
@@ -13,38 +13,37 @@
"\n",
"\n",
"\n",
"Our emphasis throughout this series of lectures \n",
"is on understanding the mathematical aspects of\n",
"different algorithms used in the fields of data analysis and machine learning. \n",
"Our emphasis throughout this series of lectures is on understanding\n",
"the mathematical aspects of different algorithms used in the fields of\n",
"data analysis and machine learning.\n",
"\n",
"However, where possible we will emphasize the\n",
"importance of using available software. We start thus with a hands-on\n",
"and top-down approach to machine learning. The aim is thus to start with\n",
"relevant data or data we have produced \n",
"and use these to introduce statistical data analysis\n",
"concepts and machine learning algorithms before we delve into the\n",
"algorithms themselves. The examples we will use in the beginning, start with simple\n",
"polynomials with random noise added. We will use the Python\n",
"software package [Scikit-Learn](http://scikit-learn.org/stable/) and\n",
"introduce various machine learning algorithms to make fits of\n",
"the data and predictions. We move thereafter to more interesting\n",
"cases such as data from say experiments (below we will look at experimental nuclear binding energies as an example).\n",
"These are examples where we can easily set up the data and\n",
"then use machine learning algorithms included in for example\n",
"**Scikit-Learn**. \n",
"However, where possible we will emphasize the importance of using\n",
"available software. We start thus with a hands-on and top-down\n",
"approach to machine learning. The aim is thus to start with relevant\n",
"data or data we have produced and use these to introduce statistical\n",
"data analysis concepts and machine learning algorithms before we delve\n",
"into the algorithms themselves. The examples we will use in the\n",
"beginning, start with simple polynomials with random noise added. We\n",
"will use the Python software package\n",
"[Scikit-Learn](http://scikit-learn.org/stable/) and introduce various\n",
"machine learning algorithms to make fits of the data and\n",
"predictions. We move thereafter to more interesting cases such as data\n",
"from say experiments (below we will look at experimental nuclear\n",
"binding energies as an example). These are examples where we can\n",
"easily set up the data and then use machine learning algorithms\n",
"included in for example **Scikit-Learn**.\n",
"\n",
"These examples will serve us the purpose of getting\n",
"started. Furthermore, they allow us to catch more than two birds with\n",
"a stone. They will allow us to bring in some programming specific\n",
"topics and tools as well as showing the power of various Python \n",
"libraries for machine learning and statistical data analysis. \n",
"topics and tools as well as showing the power of various Python\n",
"libraries for machine learning and statistical data analysis.\n",
"\n",
"Here, we will mainly focus on two\n",
"specific Python packages for Machine Learning, Scikit-Learn and\n",
"Tensorflow (see below for links etc). Moreover, the examples we\n",
"introduce will serve as inputs to many of our discussions later, as\n",
"well as allowing you to set up models and produce your own data and\n",
"get started with programming.\n",
"Here, we will mainly focus on two specific Python packages for Machine\n",
"Learning, Scikit-Learn and Tensorflow (see below for links etc).\n",
"Moreover, the examples we introduce will serve as inputs to many of\n",
"our discussions later, as well as allowing you to set up models and\n",
"produce your own data and get started with programming.\n",
"\n",
"\n",
"\n",
@@ -286,7 +285,17 @@
"[gallery](https://matplotlib.org/gallery/index.html) of examples. In\n",
"this example we plot our original values of $x$ and $y$ as well as the\n",
"prediction **ypredict** ($\\tilde{y}$), which attempts at fitting our\n",
"data with a straight line.\n",
"data with a straight line. Note also that **Scikit-Learn** requires a\n",
"matrix as input for the input values $x$ and $y$. In the above code we\n",
"have solved this by declaring $x$ and $y$ as arrays of dimension\n",
"$n\\times 1$.\n",
"\n",
"In the code here we have also made a new array for $x\\in [0,1]$. Our\n",
"prediction is computed for these values, meaning that they were not\n",
"included in the data set used to *train* (or fit) the model.\n",
"This is a recurrring theme in machine learning and data analysis. We would like to train a model on a specific given data set.\n",
"Thereafter we wish to apply it to data which were not included in the training. Below we will encounter this again in the so-called *train-validate-test* spliting. We will typically split our data into different sets, oen for training, one for validation and finally, our data from the untouched test vault!\n",
"\n",
"\n",
"The Python code follows here."
]
@@ -311,6 +320,7 @@
"y = 2*x+np.random.randn(100,1)\n",
"linreg = LinearRegression()\n",
"linreg.fit(x,y)\n",
"# This is our new x-array to which we test our model\n",
"xnew = np.array([[0],[1]])\n",
"ypredict = linreg.predict(xnew)\n",
"\n",
@@ -460,8 +470,7 @@
"Depending on the parameter in front of the normal distribution, we may\n",
"have a small or larger relative error. Try to play around with\n",
"different training data sets and study (graphically) the value of the\n",
"relative error. Note also that **Scikit-Learn** requires a matrix as input for the input values $x$ and $y$. In the above code we have\n",
"solved this by declaring $x$ and $y$ as arrays of dimension $n\\times 1$.\n",
"relative error.\n",
"\n",
"As mentioned above, **Scikit-Learn** has an impressive functionality.\n",
"We can for example extract the values of $\\alpha$ and $\\beta$ and\n",
@@ -1039,7 +1048,7 @@
"metadata": {},
"source": [
"The next step, and we will define this mathematically later, is to set up the so-called **design matrix**. We will throughout call this matrix $\\boldsymbol{X}$.\n",
"It has dimensionality $p\\times n$, where $n$ is the number of data points and $p$ are the so-called predictors. In our case here they are given by the number of polynomials in $A$ we wish to include in the fit."
"It has dimensionality $n\\times p$, where $n$ is the number of data points and $p$ are the so-called predictors. In our case here they are given by the number of polynomials in $A$ we wish to include in the fit."
]
},
{
@@ -1064,9 +1073,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Note well that we have made life simple here. We perform a fit in terms of the number of nucleons only. A more sophisticated fit can be done by including an explicit dependence on the number of protons and neutrons in the asymmetry and Coulomb terms.\n",
"Note well that we have made life simple here. We perform a fit in\n",
"terms of the number of nucleons only. A more sophisticated fit can be\n",
"done by including an explicit dependence on the number of protons and\n",
"neutrons in the asymmetry and Coulomb terms. We leave this as an exercise to you the reader.\n",
"\n",
"With **scikitlearn** we are now ready to use linear regression and fit our data."
"With **Scikit-Learn** we are now ready to use linear regression and fit our data."
]
},
{
@@ -1124,7 +1136,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"As a teaser, let us now see how we can do this with decision trees using **scikit-learn**. Later we will switch to so-called **random forests**!"
"As a teaser, let us now see how we can do this with decision trees using **Scikit-Learn**. Later we will switch to so-called **random forests**!"
]
},
{
@@ -1172,6 +1184,17 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"With a deeper and deeper tree level, we can almost reproduce every\n",
"single data point by increasing the max depth of the tree.\n",
"We can actually decide to make a decision tree which fits every single point.\n",
"As we will\n",
"see later, this has the benefit that we can really train a model which\n",
"traverses every single data point. However, the price we pay is that\n",
"we will easily overfit. That is, if we apply our model to unseen data,\n",
"we will most likely fail miserably in our attempt at making\n",
"predictions. As an exercise, try to make the tree level larger by adjusting the maximum depth variable. When printing out the predicition, you will note that the binding energy of every nucleus is accurately reproduced.\n",
"\n",
"\n",
"The **seaborn** package allows us to visualize data in an efficient way. Note that we use **scikit-learn**'s multi-layer perceptron (or feed forward neural network) \n",
"functionality."
]
@@ -1977,6 +2000,47 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We can then compute the second derivative of the cost function, which in our case is the second derivative\n",
"of the means squared error. This leads to"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial^2 C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).\n",
"\n",
"The Hessian matrix plays an important role and is defined for the mean squared error as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The Hessian matrix for ordinary least squares is also proportional to\n",
"the covariance matrix. As we will see in the chapter on Ridge and Lasso regression, This means that we can use the Singular Value Decomposition of a matrix to find\n",
"the eigenvalues of the covariance matrix and the Hessian matrix in\n",
"terms of the singular values.\n",
"\n",
"\n",
"The residuals $\\boldsymbol{\\epsilon}$ are in turn given by"
]
},
@@ -2529,9 +2593,7 @@
"**pandas** again, rather extensively in order to organize our data.\n",
"\n",
"The difference now is that we use **Scikit-Learn's** regression tools\n",
"instead of our own matrix inversion implementation. Furthermore, we\n",
"sneak in **Ridge** regression (to be discussed below) which includes a\n",
"hyperparameter $\\lambda$, also to be explained below."
"instead of our own matrix inversion implementation."
]
},
{
@@ -2603,18 +2665,6 @@
"print('Mean absolute error: %.2f' % mean_absolute_error(Energies, ytilde))\n",
"print(clf.coef_, clf.intercept_)\n",
"\n",
"# The Ridge regression with a hyperparameter lambda = 0.1\n",
"_lambda = 0.1\n",
"clf_ridge = skl.Ridge(alpha=_lambda).fit(X, Energies)\n",
"yridge = clf_ridge.predict(X)\n",
"EoS['Eridge'] = yridge\n",
"# The mean squared error \n",
"print(\"Mean squared error: %.2f\" % mean_squared_error(Energies, yridge))\n",
"# Explained variance score: 1 is perfect prediction \n",
"print('Variance score: %.2f' % r2_score(Energies, yridge))\n",
"# Mean absolute error \n",
"print('Mean absolute error: %.2f' % mean_absolute_error(Energies, yridge))\n",
"print(clf_ridge.coef_, clf_ridge.intercept_)\n",
"\n",
"fig, ax = plt.subplots()\n",
"ax.set_xlabel(r'$\\rho[\\mathrm{fm}^{-3}]$')\n",
@@ -2623,8 +2673,6 @@
" label='Theoretical data')\n",
"ax.plot(EoS['Density'], EoS['Eols'], alpha=0.7, lw=2, c='m',\n",
" label='OLS')\n",
"ax.plot(EoS['Density'], EoS['Eridge'], alpha=0.7, lw=2, c='g',\n",
" label='Ridge $\\lambda = 0.1$')\n",
"ax.legend()\n",
"save_fig(\"EoSfitting\")\n",
"plt.show()"
@@ -2637,13 +2685,11 @@
"The above simple polynomial in density $\\rho$ gives an excellent fit\n",
"to the data. \n",
"\n",
"We note also that there is a small deviation between the\n",
"standard OLS and the Ridge regression at higher densities. We discuss this in more detail\n",
"below.\n",
"\n",
"\n",
"## Splitting our Data in Training and Test data\n",
"\n",
"\n",
"It is normal in essentially all Machine Learning studies to split the\n",
"data in a training set and a test set (sometimes also an additional\n",
"validation set). **Scikit-Learn** has an own function for this. There\n",
@@ -2653,6 +2699,107 @@
"postpone a discussion of this splitting to the end of these notes and\n",
"our discussion of the so-called **bias-variance** tradeoff. Here we\n",
"limit ourselves to repeat the above equation of state fitting example\n",
"but now splitting the data into a training set and a test set.\n",
"\n",
"Let us study some examples. The first code here takes a simple\n",
"one-dimensional second-order polynomial and we fit it to a\n",
"second-order polynomial. Depending on the strength of the added noise,\n",
"the various measures like the $R2$ score or the mean-squared error,\n",
"the fit becomes better or worse."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import os\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"\n",
"def R2(y_data, y_model):\n",
" return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)\n",
"def MSE(y_data,y_model):\n",
" n = np.size(y_model)\n",
" return np.sum((y_data-y_model)**2)/n\n",
"\n",
"x = np.random.rand(100)\n",
"y = 2.0+5*x*x+0.1*np.random.randn(100)\n",
"\n",
"\n",
"# The design matrix now as function of a given polynomial\n",
"X = np.zeros((len(x),3))\n",
"X[:,0] = 1.0\n",
"X[:,1] = x\n",
"X[:,2] = x**2\n",
"# We split the data in test and training data\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
"# matrix inversion to find beta\n",
"beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
"print(beta)\n",
"# and then make the prediction\n",
"ytilde = X_train @ beta\n",
"print(\"Training R2\")\n",
"print(R2(y_train,ytilde))\n",
"print(\"Training MSE\")\n",
"print(MSE(y_train,ytilde))\n",
"ypredict = X_test @ beta\n",
"print(\"Test R2\")\n",
"print(R2(y_test,ypredict))\n",
"print(\"Test MSE\")\n",
"print(MSE(y_test,ypredict))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively, you could write your own test-train splitting function as shown here."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# equivalently in numpy\n",
"def train_test_split_numpy(inputs, labels, train_size, test_size):\n",
" n_inputs = len(inputs)\n",
" inputs_shuffled = inputs.copy()\n",
" labels_shuffled = labels.copy()\n",
"\n",
" np.random.shuffle(inputs_shuffled)\n",
" np.random.shuffle(labels_shuffled)\n",
"\n",
" train_end = int(n_inputs*train_size)\n",
" X_train, X_test = inputs_shuffled[:train_end], inputs_shuffled[train_end:]\n",
" Y_train, Y_test = labels_shuffled[:train_end], labels_shuffled[train_end:]\n",
"\n",
" return X_train, X_test, Y_train, Y_test"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"But since **scikit-learn** has its own function for doing this and since\n",
"it interfaces easily with **tensorflow** and other libraries, we\n",
"normally recommend using the latter functionality.\n",
"\n",
"\n",
"As another example, we apply the training and testing split to \n",
"to the above equation of state fitting example\n",
"but now splitting the data into a training set and a test set."
]
},
@@ -3050,119 +3197,6 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Splitting our Data in Training and Test data\n",
"\n",
"\n",
"It is normal in essentially all Machine Learning studies to split the\n",
"data in a training set and a test set (sometimes also an additional\n",
"validation set). **Scikit-Learn** has an own function for this. There\n",
"is no explicit recipe for how much data should be included as training\n",
"data and say test data. An accepted rule of thumb is to use\n",
"approximately $2/3$ to $4/5$ of the data as training data. We will\n",
"postpone a discussion of this splitting to the end of these notes and\n",
"our discussion of the so-called **bias-variance** tradeoff. Here we\n",
"limit ourselves to repeat the above equation of state fitting example\n",
"but now splitting the data into a training set and a test set.\n",
"\n",
"Let us study some examples. The first code here takes a simple\n",
"one-dimensional second-order polynomial and we fit it to a\n",
"second-order polynomial. Depending on the strength of the added noise,\n",
"the various measures like the $R2$ score or the mean-squared error,\n",
"the fit becomes better or worse."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import os\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"\n",
"def R2(y_data, y_model):\n",
" return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)\n",
"def MSE(y_data,y_model):\n",
" n = np.size(y_model)\n",
" return np.sum((y_data-y_model)**2)/n\n",
"\n",
"x = np.random.rand(100)\n",
"y = 2.0+5*x*x+0.1*np.random.randn(100)\n",
"\n",
"\n",
"# The design matrix now as function of a given polynomial\n",
"X = np.zeros((len(x),3))\n",
"X[:,0] = 1.0\n",
"X[:,1] = x\n",
"X[:,2] = x**2\n",
"# We split the data in test and training data\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
"# matrix inversion to find beta\n",
"beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
"print(beta)\n",
"# and then make the prediction\n",
"ytilde = X_train @ beta\n",
"print(\"Training R2\")\n",
"print(R2(y_train,ytilde))\n",
"print(\"Training MSE\")\n",
"print(MSE(y_train,ytilde))\n",
"ypredict = X_test @ beta\n",
"print(\"Test R2\")\n",
"print(R2(y_test,ypredict))\n",
"print(\"Test MSE\")\n",
"print(MSE(y_test,ypredict))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively, you could write your own test-train splitting function as shown here."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# equivalently in numpy\n",
"def train_test_split_numpy(inputs, labels, train_size, test_size):\n",
" n_inputs = len(inputs)\n",
" inputs_shuffled = inputs.copy()\n",
" labels_shuffled = labels.copy()\n",
"\n",
" np.random.shuffle(inputs_shuffled)\n",
" np.random.shuffle(labels_shuffled)\n",
"\n",
" train_end = int(n_inputs*train_size)\n",
" X_train, X_test = inputs_shuffled[:train_end], inputs_shuffled[train_end:]\n",
" Y_train, Y_test = labels_shuffled[:train_end], labels_shuffled[train_end:]\n",
"\n",
" return X_train, X_test, Y_train, Y_test"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"But since **scikit-learn** has its own function for doing this and since\n",
"it interfaces easily with **tensorflow** and other libraries, we\n",
"normally recommend using the latter functionality.\n",
"\n",
"\n",
"\n",
"\n",
"## Reducing the number of degrees of freedom, overarching view\n",
"\n",
"Many Machine Learning problems involve thousands or even millions of\n",