updating figures
This commit is contained in:
Morten Hjorth-Jensen
Binary file not shown.
|
Before Width: | Height: | Size: 6.4 KiB After Width: | Height: | Size: 6.2 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 9.1 KiB After Width: | Height: | Size: 9.4 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 22 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 14 KiB After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 9.7 KiB After Width: | Height: | Size: 9.3 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 24 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 9.0 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 5.0 KiB After Width: | Height: | Size: 5.0 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 9.5 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 92 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 163 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
@@ -6,8 +6,6 @@
|
||||
"source": [
|
||||
"# Linear Regression\n",
|
||||
"\n",
|
||||
"[Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/LectureAug21.mp4?vrtx=view-as-webpage)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Introduction\n",
|
||||
"\n",
|
||||
@@ -15,7 +13,7 @@
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Our emphasis throughout this series of lectures (small change) \n",
|
||||
"Our emphasis throughout this series of lectures \n",
|
||||
"is on understanding the mathematical aspects of\n",
|
||||
"different algorithms used in the fields of data analysis and machine learning. \n",
|
||||
"\n",
|
||||
@@ -235,7 +233,9 @@
|
||||
"\n",
|
||||
"## Simple linear regression model using **scikit-learn**\n",
|
||||
"\n",
|
||||
"We start with perhaps our simplest possible example, using **Scikit-Learn** to perform linear regression analysis on a data set produced by us. \n",
|
||||
"We start with perhaps our simplest possible example, using\n",
|
||||
"**Scikit-Learn** to perform linear regression analysis on a data set\n",
|
||||
"produced by us.\n",
|
||||
"\n",
|
||||
"What follows is a simple Python code where we have defined a function\n",
|
||||
"$y$ in terms of the variable $x$. Both are defined as vectors with $100$ entries. \n",
|
||||
@@ -437,7 +437,8 @@
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from sklearn.linear_model import LinearRegression\n",
|
||||
"\n",
|
||||
"# Number of data points\n",
|
||||
"n = 100\n",
|
||||
"x = np.random.rand(100,1)\n",
|
||||
"y = 5*x+0.01*np.random.randn(100,1)\n",
|
||||
"linreg = LinearRegression()\n",
|
||||
@@ -459,7 +460,8 @@
|
||||
"Depending on the parameter in front of the normal distribution, we may\n",
|
||||
"have a small or larger relative error. Try to play around with\n",
|
||||
"different training data sets and study (graphically) the value of the\n",
|
||||
"relative error.\n",
|
||||
"relative error. Note also that **Scikit-Learn** requires a matrix as input for the input values $x$ and $y$. In the above code we have\n",
|
||||
"solved this by declaring $x$ and $y$ as arrays of dimension $n\\times 1$.\n",
|
||||
"\n",
|
||||
"As mentioned above, **Scikit-Learn** has an impressive functionality.\n",
|
||||
"We can for example extract the values of $\\alpha$ and $\\beta$ and\n",
|
||||
@@ -629,8 +631,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"H_{\\delta}(\\boldsymbol{a})=\\left\\{\\begin{array}{cc}\\frac{1}{2} \\boldsymbol{a}^{2}& \\text{for }|\\boldsymbol{a}|\\leq \\delta\\\\ \\delta (|\\b\\\n",
|
||||
"m{a}|-\\frac{1}{2}\\delta ),&\\text{otherwise}.\\end{array}\\right.\n",
|
||||
"H_{\\delta}(\\boldsymbol{a})=\\left\\{\\begin{array}{cc}\\frac{1}{2} \\boldsymbol{a}^{2}& \\text{for }|\\boldsymbol{a}|\\leq \\delta\\\\ \\delta (|\\boldsymbol{a}|-\\frac{1}{2}\\delta ),&\\text{otherwise}.\\end{array}\\right.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -641,6 +642,8 @@
|
||||
"Here $\\boldsymbol{a}=\\boldsymbol{y} - \\boldsymbol{\\tilde{y}}$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"We will discuss in more\n",
|
||||
"detail these and other functions in the various lectures. We conclude this part with another example. Instead of \n",
|
||||
"a linear $x$-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn."
|
||||
@@ -1061,6 +1064,8 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Note well that we have made life simple here. We perform a fit in terms of the number of nucleons only. A more sophisticated fit can be done by including an explicit dependence on the number of protons and neutrons in the asymmetry and Coulomb terms.\n",
|
||||
"\n",
|
||||
"With **scikitlearn** we are now ready to use linear regression and fit our data."
|
||||
]
|
||||
},
|
||||
@@ -1100,7 +1105,6 @@
|
||||
"print('Variance score: %.2f' % r2_score(Energies, fity))\n",
|
||||
"# Mean absolute error \n",
|
||||
"print('Mean absolute error: %.2f' % mean_absolute_error(Energies, fity))\n",
|
||||
"print(clf.coef_, clf.intercept_)\n",
|
||||
"\n",
|
||||
"Masses['Eapprox'] = fity\n",
|
||||
"# Generate a plot comparing the experimental with the fitted values values.\n",
|
||||
@@ -3046,6 +3050,119 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Splitting our Data in Training and Test data\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"It is normal in essentially all Machine Learning studies to split the\n",
|
||||
"data in a training set and a test set (sometimes also an additional\n",
|
||||
"validation set). **Scikit-Learn** has an own function for this. There\n",
|
||||
"is no explicit recipe for how much data should be included as training\n",
|
||||
"data and say test data. An accepted rule of thumb is to use\n",
|
||||
"approximately $2/3$ to $4/5$ of the data as training data. We will\n",
|
||||
"postpone a discussion of this splitting to the end of these notes and\n",
|
||||
"our discussion of the so-called **bias-variance** tradeoff. Here we\n",
|
||||
"limit ourselves to repeat the above equation of state fitting example\n",
|
||||
"but now splitting the data into a training set and a test set.\n",
|
||||
"\n",
|
||||
"Let us study some examples. The first code here takes a simple\n",
|
||||
"one-dimensional second-order polynomial and we fit it to a\n",
|
||||
"second-order polynomial. Depending on the strength of the added noise,\n",
|
||||
"the various measures like the $R2$ score or the mean-squared error,\n",
|
||||
"the fit becomes better or worse."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def R2(y_data, y_model):\n",
|
||||
" return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)\n",
|
||||
"def MSE(y_data,y_model):\n",
|
||||
" n = np.size(y_model)\n",
|
||||
" return np.sum((y_data-y_model)**2)/n\n",
|
||||
"\n",
|
||||
"x = np.random.rand(100)\n",
|
||||
"y = 2.0+5*x*x+0.1*np.random.randn(100)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# The design matrix now as function of a given polynomial\n",
|
||||
"X = np.zeros((len(x),3))\n",
|
||||
"X[:,0] = 1.0\n",
|
||||
"X[:,1] = x\n",
|
||||
"X[:,2] = x**2\n",
|
||||
"# We split the data in test and training data\n",
|
||||
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
|
||||
"# matrix inversion to find beta\n",
|
||||
"beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
|
||||
"print(beta)\n",
|
||||
"# and then make the prediction\n",
|
||||
"ytilde = X_train @ beta\n",
|
||||
"print(\"Training R2\")\n",
|
||||
"print(R2(y_train,ytilde))\n",
|
||||
"print(\"Training MSE\")\n",
|
||||
"print(MSE(y_train,ytilde))\n",
|
||||
"ypredict = X_test @ beta\n",
|
||||
"print(\"Test R2\")\n",
|
||||
"print(R2(y_test,ypredict))\n",
|
||||
"print(\"Test MSE\")\n",
|
||||
"print(MSE(y_test,ypredict))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Alternatively, you could write your own test-train splitting function as shown here."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# equivalently in numpy\n",
|
||||
"def train_test_split_numpy(inputs, labels, train_size, test_size):\n",
|
||||
" n_inputs = len(inputs)\n",
|
||||
" inputs_shuffled = inputs.copy()\n",
|
||||
" labels_shuffled = labels.copy()\n",
|
||||
"\n",
|
||||
" np.random.shuffle(inputs_shuffled)\n",
|
||||
" np.random.shuffle(labels_shuffled)\n",
|
||||
"\n",
|
||||
" train_end = int(n_inputs*train_size)\n",
|
||||
" X_train, X_test = inputs_shuffled[:train_end], inputs_shuffled[train_end:]\n",
|
||||
" Y_train, Y_test = labels_shuffled[:train_end], labels_shuffled[train_end:]\n",
|
||||
"\n",
|
||||
" return X_train, X_test, Y_train, Y_test"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"But since **scikit-learn** has its own function for doing this and since\n",
|
||||
"it interfaces easily with **tensorflow** and other libraries, we\n",
|
||||
"normally recommend using the latter functionality.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Reducing the number of degrees of freedom, overarching view\n",
|
||||
"\n",
|
||||
"Many Machine Learning problems involve thousands or even millions of\n",
|
||||
@@ -3069,6 +3186,7 @@
|
||||
"visualization.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Before we proceed however, we will discuss how to preprocess our\n",
|
||||
"data. Till now and in connection with our previous examples we have\n",
|
||||
"not met so many cases where we are too sensitive to the scaling of our\n",
|
||||
@@ -3076,6 +3194,15 @@
|
||||
"to extreme values. Scaling the data renders our inputs much more\n",
|
||||
"suitable for the algorithms we want to employ.\n",
|
||||
"\n",
|
||||
"For data sets gathered for real world applications, it is rather normal that\n",
|
||||
"different features have very different units and\n",
|
||||
"numerical scales. For example, a data set detailing health habits may include\n",
|
||||
"features such as **age** in the range $0-80$, and **caloric intake** of order $2000$.\n",
|
||||
"Many machine learning methods sensitive to the scales of the features and may perform poorly if they\n",
|
||||
"are very different scales. Therefore, it is typical to scale\n",
|
||||
"the features in a way to avoid such outlier values.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"**Scikit-Learn** has several functions which allow us to rescale the\n",
|
||||
"data, normally resulting in much better results in terms of various\n",
|
||||
"accuracy scores. The **StandardScaler** function in **Scikit-Learn**\n",
|
||||
@@ -3105,7 +3232,38 @@
|
||||
"techniques.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Simple preprocessing examples, Franke function and regression"
|
||||
"Many features are often scaled using standardization to improve\n",
|
||||
"performance. In **Scikit-Learn** this is given by the **StandardScaler**\n",
|
||||
"function as discussed above. It is easy however to write your own.\n",
|
||||
"Mathematically, this involves subtracting the mean and divide by the\n",
|
||||
"standard deviation over the data set, for each feature:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"x_j^{(i)} \\rightarrow \\frac{x_j^{(i)} - \\overline{x}_j}{\\sigma(x_j)},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"where $\\overline{x}_j$ and $\\sigma(x_j)$ are the mean and standard\n",
|
||||
"deviation, respectively, of the feature $x_j$. This ensures that each\n",
|
||||
"feature has zero mean and unit standard deviation. For data sets\n",
|
||||
"where we do not have the standard deviation or don't wish to calculate\n",
|
||||
"it, it is then common to simply set it to one.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Let us consider the following vanilla example where we use both\n",
|
||||
"**Scikit-Learn** and write our own function as well. We produce a\n",
|
||||
"simple test design matrix with random numbers. Each column could then\n",
|
||||
"represent a specific feature whose mean value is subracted."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -3117,98 +3275,126 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Common imports\n",
|
||||
"import os\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import sklearn.linear_model as skl\n",
|
||||
"from sklearn.metrics import mean_squared_error\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"from sklearn.preprocessing import MinMaxScaler, StandardScaler, Normalizer\n",
|
||||
"\n",
|
||||
"# Where to save the figures and data files\n",
|
||||
"PROJECT_ROOT_DIR = \"Results\"\n",
|
||||
"FIGURE_ID = \"Results/FigureFiles\"\n",
|
||||
"DATA_ID = \"DataFiles/\"\n",
|
||||
"\n",
|
||||
"if not os.path.exists(PROJECT_ROOT_DIR):\n",
|
||||
" os.mkdir(PROJECT_ROOT_DIR)\n",
|
||||
"\n",
|
||||
"if not os.path.exists(FIGURE_ID):\n",
|
||||
" os.makedirs(FIGURE_ID)\n",
|
||||
"\n",
|
||||
"if not os.path.exists(DATA_ID):\n",
|
||||
" os.makedirs(DATA_ID)\n",
|
||||
"\n",
|
||||
"def image_path(fig_id):\n",
|
||||
" return os.path.join(FIGURE_ID, fig_id)\n",
|
||||
"\n",
|
||||
"def data_path(dat_id):\n",
|
||||
" return os.path.join(DATA_ID, dat_id)\n",
|
||||
"\n",
|
||||
"def save_fig(fig_id):\n",
|
||||
" plt.savefig(image_path(fig_id) + \".png\", format='png')\n",
|
||||
"import numpy as np\n",
|
||||
"import pandas as pd\n",
|
||||
"from IPython.display import display\n",
|
||||
"np.random.seed(100)\n",
|
||||
"# setting up a 10 x 5 matrix\n",
|
||||
"rows = 10\n",
|
||||
"cols = 5\n",
|
||||
"X = np.random.randn(rows,cols)\n",
|
||||
"XPandas = pd.DataFrame(X)\n",
|
||||
"display(XPandas)\n",
|
||||
"print(XPandas.mean())\n",
|
||||
"print(XPandas.std())\n",
|
||||
"XPandas = (XPandas -XPandas.mean())\n",
|
||||
"display(XPandas)\n",
|
||||
"# This option does not include the standard deviation\n",
|
||||
"scaler = StandardScaler(with_std=False)\n",
|
||||
"scaler.fit(X)\n",
|
||||
"Xscaled = scaler.transform(X)\n",
|
||||
"display(XPandas-Xscaled)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def FrankeFunction(x,y):\n",
|
||||
"\tterm1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))\n",
|
||||
"\tterm2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))\n",
|
||||
"\tterm3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))\n",
|
||||
"\tterm4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)\n",
|
||||
"\treturn term1 + term2 + term3 + term4\n",
|
||||
"\n",
|
||||
"Another commonly used scaling method is min-max scaling. This is very\n",
|
||||
"useful for when we want the features to lie in a certain interval. To\n",
|
||||
"scale the feature $x_j$ to the interval $[a, b]$, we can apply the\n",
|
||||
"transformation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"x_j^{(i)} \\rightarrow (b-a)\\frac{x_j^{(i)} - \\min(x_j)}{\\max(x_j) - \\min(x_j)} - a\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"where $\\min(x_j)$ and $\\max(x_j)$ return the minimum and maximum value of $x_j$ over the data set, respectively.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def create_X(x, y, n ):\n",
|
||||
"\tif len(x.shape) > 1:\n",
|
||||
"\t\tx = np.ravel(x)\n",
|
||||
"\t\ty = np.ravel(y)\n",
|
||||
"\n",
|
||||
"\tN = len(x)\n",
|
||||
"\tl = int((n+1)*(n+2)/2)\t\t# Number of elements in beta\n",
|
||||
"\tX = np.ones((N,l))\n",
|
||||
"\n",
|
||||
"\tfor i in range(1,n+1):\n",
|
||||
"\t\tq = int((i)*(i+1)/2)\n",
|
||||
"\t\tfor k in range(i+1):\n",
|
||||
"\t\t\tX[:,q+k] = (x**(i-k))*(y**k)\n",
|
||||
"\n",
|
||||
"\treturn X\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Making meshgrid of datapoints and compute Franke's function\n",
|
||||
"n = 5\n",
|
||||
"N = 1000\n",
|
||||
"x = np.sort(np.random.uniform(0, 1, N))\n",
|
||||
"y = np.sort(np.random.uniform(0, 1, N))\n",
|
||||
"z = FrankeFunction(x, y)\n",
|
||||
"X = create_X(x, y, n=n) \n",
|
||||
"# split in training and test data\n",
|
||||
"X_train, X_test, y_train, y_test = train_test_split(X,z,test_size=0.2)\n",
|
||||
"## Testing the Means Squared Error as function of Complexity\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"clf = skl.LinearRegression().fit(X_train, y_train)\n",
|
||||
"Before we proceed with a more detailed analysis of the so-called\n",
|
||||
"Bias-Variance tradeoff, we present here an example of the relation\n",
|
||||
"between model complexity and the mean squared error for the triaining\n",
|
||||
"data and the test data.\n",
|
||||
"\n",
|
||||
"# The mean squared error and R2 score\n",
|
||||
"print(\"MSE before scaling: {:.2f}\".format(mean_squared_error(clf.predict(X_test), y_test)))\n",
|
||||
"print(\"R2 score before scaling {:.2f}\".format(clf.score(X_test,y_test)))\n",
|
||||
"The results here tell us clearly that for the data not included in the\n",
|
||||
"training, there is an optimal model as function of the complexity of\n",
|
||||
"ourmodel (here in terms of the polynomial degree of the model).\n",
|
||||
"\n",
|
||||
"The results here will vary as function of model complexity and the amount od data used for training. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Our data is defined by $x\\in [-3,3]$ with a total of for example $100$ data points."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import numpy as np\n",
|
||||
"from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",
|
||||
"from sklearn.preprocessing import PolynomialFeatures\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"from sklearn.pipeline import make_pipeline\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"np.random.seed(2018)\n",
|
||||
"n = 100\n",
|
||||
"maxdegree = 14\n",
|
||||
"# Make data set.\n",
|
||||
"x = np.linspace(-3, 3, n).reshape(-1, 1)\n",
|
||||
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)\n",
|
||||
"TestError = np.zeros(maxdegree)\n",
|
||||
"TrainError = np.zeros(maxdegree)\n",
|
||||
"polydegree = np.zeros(maxdegree)\n",
|
||||
"x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)\n",
|
||||
"scaler = StandardScaler()\n",
|
||||
"scaler.fit(X_train)\n",
|
||||
"X_train_scaled = scaler.transform(X_train)\n",
|
||||
"X_test_scaled = scaler.transform(X_test)\n",
|
||||
"scaler.fit(x_train)\n",
|
||||
"x_train_scaled = scaler.transform(x_train)\n",
|
||||
"x_test_scaled = scaler.transform(x_test)\n",
|
||||
"\n",
|
||||
"print(\"Feature min values before scaling:\\n {}\".format(X_train.min(axis=0)))\n",
|
||||
"print(\"Feature max values before scaling:\\n {}\".format(X_train.max(axis=0)))\n",
|
||||
"for degree in range(maxdegree):\n",
|
||||
" model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))\n",
|
||||
" clf = model.fit(x_train_scaled,y_train)\n",
|
||||
" y_fit = clf.predict(x_train_scaled)\n",
|
||||
" y_pred = clf.predict(x_test_scaled) \n",
|
||||
" polydegree[degree] = degree\n",
|
||||
" TestError[degree] = np.mean( np.mean((y_test - y_pred)**2) )\n",
|
||||
" TrainError[degree] = np.mean( np.mean((y_train - y_fit)**2) )\n",
|
||||
"\n",
|
||||
"print(\"Feature min values after scaling:\\n {}\".format(X_train_scaled.min(axis=0)))\n",
|
||||
"print(\"Feature max values after scaling:\\n {}\".format(X_train_scaled.max(axis=0)))\n",
|
||||
"\n",
|
||||
"clf = skl.LinearRegression().fit(X_train_scaled, y_train)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"print(\"MSE after scaling: {:.2f}\".format(mean_squared_error(clf.predict(X_test_scaled), y_test)))\n",
|
||||
"print(\"R2 score for scaled data: {:.2f}\".format(clf.score(X_test_scaled,y_test)))"
|
||||
"plt.plot(polydegree, TestError, label='Test Error')\n",
|
||||
"plt.plot(polydegree, TrainError, label='Train Error')\n",
|
||||
"plt.legend()\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -54,7 +54,7 @@
|
||||
<script async="async" src="_static/sphinx-thebe.js"></script>
|
||||
<link rel="index" title="Index" href="genindex.html" />
|
||||
<link rel="search" title="Search" href="search.html" />
|
||||
<link rel="next" title="4. Resampling Methods" href="chapter2.html" />
|
||||
<link rel="next" title="4. Ridge and Lasso Regression" href="chapter2.html" />
|
||||
<link rel="prev" title="2. Linear Algebra, Handling of Arrays and more Python Features" href="linalg.html" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<meta name="docsearch:language" content="en" />
|
||||
@@ -144,12 +144,12 @@
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter2.html">
|
||||
4. Resampling Methods
|
||||
4. Ridge and Lasso Regression
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter3.html">
|
||||
5. Ridge and Lasso Regression
|
||||
5. Resampling Methods
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
@@ -358,35 +358,38 @@
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#reducing-the-number-of-degrees-of-freedom-overarching-view">
|
||||
3.8. Reducing the number of degrees of freedom, overarching view
|
||||
<a class="reference internal nav-link" href="#id1">
|
||||
3.8. Splitting our Data in Training and Test data
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#reducing-the-number-of-degrees-of-freedom-overarching-view">
|
||||
3.9. Reducing the number of degrees of freedom, overarching view
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#testing-the-means-squared-error-as-function-of-complexity">
|
||||
3.10. Testing the Means Squared Error as function of Complexity
|
||||
</a>
|
||||
<ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#simple-preprocessing-examples-franke-function-and-regression">
|
||||
3.8.1. Simple preprocessing examples, Franke function and regression
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercises">
|
||||
3.9. Exercises
|
||||
3.11. Exercises
|
||||
</a>
|
||||
<ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-setting-up-various-python-environments">
|
||||
3.9.1. Exercise: Setting up various Python environments
|
||||
3.11.1. Exercise: Setting up various Python environments
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-making-your-own-data-and-exploring-scikit-learn">
|
||||
3.9.2. Exercise: making your own data and exploring scikit-learn
|
||||
3.11.2. Exercise: making your own data and exploring scikit-learn
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#exercise-normalizing-our-data">
|
||||
3.9.3. Exercise: Normalizing our data
|
||||
3.11.3. Exercise: Normalizing our data
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
@@ -404,10 +407,9 @@
|
||||
|
||||
<div class="section" id="linear-regression">
|
||||
<h1><span class="section-number">3. </span>Linear Regression<a class="headerlink" href="#linear-regression" title="Permalink to this headline">¶</a></h1>
|
||||
<p><a class="reference external" href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/LectureAug21.mp4?vrtx=view-as-webpage">Video of Lecture</a></p>
|
||||
<div class="section" id="introduction">
|
||||
<h2><span class="section-number">3.1. </span>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Our emphasis throughout this series of lectures (small change)<br />
|
||||
<p>Our emphasis throughout this series of lectures<br />
|
||||
is on understanding the mathematical aspects of
|
||||
different algorithms used in the fields of data analysis and machine learning.</p>
|
||||
<p>However, where possible we will emphasize the
|
||||
@@ -585,7 +587,9 @@ may first try the simplest class of models, namely linear models, followed obvio
|
||||
</div>
|
||||
<div class="section" id="simple-linear-regression-model-using-scikit-learn">
|
||||
<h2><span class="section-number">3.3. </span>Simple linear regression model using <strong>scikit-learn</strong><a class="headerlink" href="#simple-linear-regression-model-using-scikit-learn" title="Permalink to this headline">¶</a></h2>
|
||||
<p>We start with perhaps our simplest possible example, using <strong>Scikit-Learn</strong> to perform linear regression analysis on a data set produced by us.</p>
|
||||
<p>We start with perhaps our simplest possible example, using
|
||||
<strong>Scikit-Learn</strong> to perform linear regression analysis on a data set
|
||||
produced by us.</p>
|
||||
<p>What follows is a simple Python code where we have defined a function
|
||||
<span class="math notranslate nohighlight">\(y\)</span> in terms of the variable <span class="math notranslate nohighlight">\(x\)</span>. Both are defined as vectors with <span class="math notranslate nohighlight">\(100\)</span> entries.
|
||||
The numbers in the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> are given
|
||||
@@ -717,7 +721,8 @@ to be dominated by outliers.</p>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LinearRegression</span>
|
||||
|
||||
<span class="c1"># Number of data points</span>
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mi">5</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mf">0.01</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">100</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">linreg</span> <span class="o">=</span> <span class="n">LinearRegression</span><span class="p">()</span>
|
||||
@@ -740,7 +745,8 @@ to be dominated by outliers.</p>
|
||||
<p>Depending on the parameter in front of the normal distribution, we may
|
||||
have a small or larger relative error. Try to play around with
|
||||
different training data sets and study (graphically) the value of the
|
||||
relative error.</p>
|
||||
relative error. Note also that <strong>Scikit-Learn</strong> requires a matrix as input for the input values <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span>. In the above code we have
|
||||
solved this by declaring <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> as arrays of dimension <span class="math notranslate nohighlight">\(n\times 1\)</span>.</p>
|
||||
<p>As mentioned above, <strong>Scikit-Learn</strong> has an impressive functionality.
|
||||
We can for example extract the values of <span class="math notranslate nohighlight">\(\alpha\)</span> and <span class="math notranslate nohighlight">\(\beta\)</span> and
|
||||
their error estimates, or the variance and standard deviation and many
|
||||
@@ -781,13 +787,13 @@ example of the functionality of <strong>Scikit-Learn</strong>.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The intercept alpha:
|
||||
[1.99723611]
|
||||
[1.84860939]
|
||||
Coefficient beta :
|
||||
[[5.1007125]]
|
||||
Mean squared error: 0.22
|
||||
[[5.2571699]]
|
||||
Mean squared error: 0.25
|
||||
Variance score: 0.90
|
||||
Mean squared log error: 0.01
|
||||
Mean absolute error: 0.36
|
||||
Mean absolute error: 0.40
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_13_1.png" src="_images/chapter1_13_1.png" />
|
||||
@@ -844,8 +850,7 @@ ways of dealing with outliers.</p>
|
||||
<p>The Huber cost function is defined as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
H_{\delta}(\boldsymbol{a})=\left\{\begin{array}{cc}\frac{1}{2} \boldsymbol{a}^{2}& \text{for }|\boldsymbol{a}|\leq \delta\\ \delta (|\b\
|
||||
m{a}|-\frac{1}{2}\delta ),&\text{otherwise}.\end{array}\right.
|
||||
H_{\delta}(\boldsymbol{a})=\left\{\begin{array}{cc}\frac{1}{2} \boldsymbol{a}^{2}& \text{for }|\boldsymbol{a}|\leq \delta\\ \delta (|\boldsymbol{a}|-\frac{1}{2}\delta ),&\text{otherwise}.\end{array}\right.
|
||||
\end{split}\]</div>
|
||||
<p>Here <span class="math notranslate nohighlight">\(\boldsymbol{a}=\boldsymbol{y} - \boldsymbol{\tilde{y}}\)</span>.</p>
|
||||
<p>We will discuss in more
|
||||
@@ -888,7 +893,7 @@ a linear <span class="math notranslate nohighlight">\(x\)</span>-dependence we s
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter1_27_0.png" src="_images/chapter1_27_0.png" />
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.005000000000000007
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.0050000000000000044
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1126,6 +1131,7 @@ It has dimensionality <span class="math notranslate nohighlight">\(p\times n\)</
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Note well that we have made life simple here. We perform a fit in terms of the number of nucleons only. A more sophisticated fit can be done by including an explicit dependence on the number of protons and neutrons in the asymmetry and Coulomb terms.</p>
|
||||
<p>With <strong>scikitlearn</strong> we are now ready to use linear regression and fit our data.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
@@ -1145,7 +1151,6 @@ Now we can print measures of how our fit is doing, the coefficients from the fit
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Variance score: </span><span class="si">%.2f</span><span class="s1">'</span> <span class="o">%</span> <span class="n">r2_score</span><span class="p">(</span><span class="n">Energies</span><span class="p">,</span> <span class="n">fity</span><span class="p">))</span>
|
||||
<span class="c1"># Mean absolute error </span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s1">'Mean absolute error: </span><span class="si">%.2f</span><span class="s1">'</span> <span class="o">%</span> <span class="n">mean_absolute_error</span><span class="p">(</span><span class="n">Energies</span><span class="p">,</span> <span class="n">fity</span><span class="p">))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="n">coef_</span><span class="p">,</span> <span class="n">clf</span><span class="o">.</span><span class="n">intercept_</span><span class="p">)</span>
|
||||
|
||||
<span class="n">Masses</span><span class="p">[</span><span class="s1">'Eapprox'</span><span class="p">]</span> <span class="o">=</span> <span class="n">fity</span>
|
||||
<span class="c1"># Generate a plot comparing the experimental with the fitted values values.</span>
|
||||
@@ -1166,8 +1171,6 @@ Now we can print measures of how our fit is doing, the coefficients from the fit
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Mean squared error: 0.04
|
||||
Variance score: 0.95
|
||||
Mean absolute error: 0.05
|
||||
[ 0.00000000e+00 7.06492086e-03 -1.73091052e-01 -1.66020213e+01
|
||||
1.17385778e+00] 15.212327334149492
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_57_1.png" src="_images/chapter1_57_1.png" />
|
||||
@@ -1225,7 +1228,7 @@ A
|
||||
270 3344 160 110 270 Ds 7.253775 7.253775
|
||||
|
||||
[267 rows x 6 columns]
|
||||
0.009883615646716182
|
||||
0.009883615646716186
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1273,8 +1276,6 @@ functionality.</p>
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
@@ -1339,7 +1340,9 @@ functionality.</p>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
@@ -1357,7 +1360,7 @@ functionality.</p>
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter1_61_10.png" src="_images/chapter1_61_10.png" />
|
||||
<img alt="_images/chapter1_61_11.png" src="_images/chapter1_61_11.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2332,13 +2335,13 @@ but now splitting the data into a training set and a test set.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Training R2
|
||||
0.9999878260589065
|
||||
0.9999869956119286
|
||||
Training MSE
|
||||
5.415634483874318
|
||||
5.745136489050356
|
||||
Test R2
|
||||
0.999974281880048
|
||||
0.9999787681537219
|
||||
Test MSE
|
||||
11.653095390463358
|
||||
9.712199063818309
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2600,8 +2603,107 @@ R2 score is 0.6628996975186952
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="id1">
|
||||
<h2><span class="section-number">3.8. </span>Splitting our Data in Training and Test data<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h2>
|
||||
<p>It is normal in essentially all Machine Learning studies to split the
|
||||
data in a training set and a test set (sometimes also an additional
|
||||
validation set). <strong>Scikit-Learn</strong> has an own function for this. There
|
||||
is no explicit recipe for how much data should be included as training
|
||||
data and say test data. An accepted rule of thumb is to use
|
||||
approximately <span class="math notranslate nohighlight">\(2/3\)</span> to <span class="math notranslate nohighlight">\(4/5\)</span> of the data as training data. We will
|
||||
postpone a discussion of this splitting to the end of these notes and
|
||||
our discussion of the so-called <strong>bias-variance</strong> tradeoff. Here we
|
||||
limit ourselves to repeat the above equation of state fitting example
|
||||
but now splitting the data into a training set and a test set.</p>
|
||||
<p>Let us study some examples. The first code here takes a simple
|
||||
one-dimensional second-order polynomial and we fit it to a
|
||||
second-order polynomial. Depending on the strength of the added noise,
|
||||
the various measures like the <span class="math notranslate nohighlight">\(R2\)</span> score or the mean-squared error,
|
||||
the fit becomes better or worse.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">R2</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span> <span class="n">y_model</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">y_model</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y_data</span><span class="p">))</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
|
||||
<span class="k">def</span> <span class="nf">MSE</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span><span class="n">y_model</span><span class="p">):</span>
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">y_model</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span><span class="o">-</span><span class="n">y_model</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
|
||||
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">5</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mf">0.1</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="c1"># The design matrix now as function of a given polynomial</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">),</span><span class="mi">3</span><span class="p">))</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mf">1.0</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="mi">2</span>
|
||||
<span class="c1"># We split the data in test and training data</span>
|
||||
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
|
||||
<span class="c1"># matrix inversion to find beta</span>
|
||||
<span class="n">beta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X_train</span><span class="p">)</span> <span class="o">@</span> <span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y_train</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">beta</span><span class="p">)</span>
|
||||
<span class="c1"># and then make the prediction</span>
|
||||
<span class="n">ytilde</span> <span class="o">=</span> <span class="n">X_train</span> <span class="o">@</span> <span class="n">beta</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Training R2"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">R2</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytilde</span><span class="p">))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Training MSE"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytilde</span><span class="p">))</span>
|
||||
<span class="n">ypredict</span> <span class="o">=</span> <span class="n">X_test</span> <span class="o">@</span> <span class="n">beta</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Test R2"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">R2</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredict</span><span class="p">))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Test MSE"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredict</span><span class="p">))</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 2.05802766 -0.30626706 5.27262964]
|
||||
Training R2
|
||||
0.9967292151090247
|
||||
Training MSE
|
||||
0.007931713723267314
|
||||
Test R2
|
||||
0.9966744029509663
|
||||
Test MSE
|
||||
0.006968283076248407
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Alternatively, you could write your own test-train splitting function as shown here.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># equivalently in numpy</span>
|
||||
<span class="k">def</span> <span class="nf">train_test_split_numpy</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">labels</span><span class="p">,</span> <span class="n">train_size</span><span class="p">,</span> <span class="n">test_size</span><span class="p">):</span>
|
||||
<span class="n">n_inputs</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">inputs</span><span class="p">)</span>
|
||||
<span class="n">inputs_shuffled</span> <span class="o">=</span> <span class="n">inputs</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
|
||||
<span class="n">labels_shuffled</span> <span class="o">=</span> <span class="n">labels</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
|
||||
|
||||
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">shuffle</span><span class="p">(</span><span class="n">inputs_shuffled</span><span class="p">)</span>
|
||||
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">shuffle</span><span class="p">(</span><span class="n">labels_shuffled</span><span class="p">)</span>
|
||||
|
||||
<span class="n">train_end</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n_inputs</span><span class="o">*</span><span class="n">train_size</span><span class="p">)</span>
|
||||
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span> <span class="o">=</span> <span class="n">inputs_shuffled</span><span class="p">[:</span><span class="n">train_end</span><span class="p">],</span> <span class="n">inputs_shuffled</span><span class="p">[</span><span class="n">train_end</span><span class="p">:]</span>
|
||||
<span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_test</span> <span class="o">=</span> <span class="n">labels_shuffled</span><span class="p">[:</span><span class="n">train_end</span><span class="p">],</span> <span class="n">labels_shuffled</span><span class="p">[</span><span class="n">train_end</span><span class="p">:]</span>
|
||||
|
||||
<span class="k">return</span> <span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_test</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>But since <strong>scikit-learn</strong> has its own function for doing this and since
|
||||
it interfaces easily with <strong>tensorflow</strong> and other libraries, we
|
||||
normally recommend using the latter functionality.</p>
|
||||
</div>
|
||||
<div class="section" id="reducing-the-number-of-degrees-of-freedom-overarching-view">
|
||||
<h2><span class="section-number">3.8. </span>Reducing the number of degrees of freedom, overarching view<a class="headerlink" href="#reducing-the-number-of-degrees-of-freedom-overarching-view" title="Permalink to this headline">¶</a></h2>
|
||||
<h2><span class="section-number">3.9. </span>Reducing the number of degrees of freedom, overarching view<a class="headerlink" href="#reducing-the-number-of-degrees-of-freedom-overarching-view" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Many Machine Learning problems involve thousands or even millions of
|
||||
features for each training instance. Not only does this make training
|
||||
extremely slow, it can also make it much harder to find a good
|
||||
@@ -2624,6 +2726,13 @@ not met so many cases where we are too sensitive to the scaling of our
|
||||
data. Normally the data may need a rescaling and/or may be sensitive
|
||||
to extreme values. Scaling the data renders our inputs much more
|
||||
suitable for the algorithms we want to employ.</p>
|
||||
<p>For data sets gathered for real world applications, it is rather normal that
|
||||
different features have very different units and
|
||||
numerical scales. For example, a data set detailing health habits may include
|
||||
features such as <strong>age</strong> in the range <span class="math notranslate nohighlight">\(0-80\)</span>, and <strong>caloric intake</strong> of order <span class="math notranslate nohighlight">\(2000\)</span>.
|
||||
Many machine learning methods sensitive to the scales of the features and may perform poorly if they
|
||||
are very different scales. Therefore, it is typical to scale
|
||||
the features in a way to avoid such outlier values.</p>
|
||||
<p><strong>Scikit-Learn</strong> has several functions which allow us to rescale the
|
||||
data, normally resulting in much better results in terms of various
|
||||
accuracy scores. The <strong>StandardScaler</strong> function in <strong>Scikit-Learn</strong>
|
||||
@@ -2648,142 +2757,465 @@ RobustScaler ignore data points that are very different from the rest
|
||||
(like measurement errors). These odd data points are also called
|
||||
outliers, and might often lead to trouble for other scaling
|
||||
techniques.</p>
|
||||
<div class="section" id="simple-preprocessing-examples-franke-function-and-regression">
|
||||
<h3><span class="section-number">3.8.1. </span>Simple preprocessing examples, Franke function and regression<a class="headerlink" href="#simple-preprocessing-examples-franke-function-and-regression" title="Permalink to this headline">¶</a></h3>
|
||||
<p>Many features are often scaled using standardization to improve
|
||||
performance. In <strong>Scikit-Learn</strong> this is given by the <strong>StandardScaler</strong>
|
||||
function as discussed above. It is easy however to write your own.
|
||||
Mathematically, this involves subtracting the mean and divide by the
|
||||
standard deviation over the data set, for each feature:</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
x_j^{(i)} \rightarrow \frac{x_j^{(i)} - \overline{x}_j}{\sigma(x_j)},
|
||||
\]</div>
|
||||
<p>where <span class="math notranslate nohighlight">\(\overline{x}_j\)</span> and <span class="math notranslate nohighlight">\(\sigma(x_j)\)</span> are the mean and standard
|
||||
deviation, respectively, of the feature <span class="math notranslate nohighlight">\(x_j\)</span>. This ensures that each
|
||||
feature has zero mean and unit standard deviation. For data sets
|
||||
where we do not have the standard deviation or don’t wish to calculate
|
||||
it, it is then common to simply set it to one.</p>
|
||||
<p>Let us consider the following vanilla example where we use both
|
||||
<strong>Scikit-Learn</strong> and write our own function as well. We produce a
|
||||
simple test design matrix with random numbers. Each column could then
|
||||
represent a specific feature whose mean value is subracted.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Common imports</span>
|
||||
<span class="kn">import</span> <span class="nn">os</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">import</span> <span class="nn">sklearn.linear_model</span> <span class="k">as</span> <span class="nn">skl</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sklearn.linear_model</span> <span class="k">as</span> <span class="nn">skl</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.metrics</span> <span class="kn">import</span> <span class="n">mean_squared_error</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">MinMaxScaler</span><span class="p">,</span> <span class="n">StandardScaler</span><span class="p">,</span> <span class="n">Normalizer</span>
|
||||
|
||||
<span class="c1"># Where to save the figures and data files</span>
|
||||
<span class="n">PROJECT_ROOT_DIR</span> <span class="o">=</span> <span class="s2">"Results"</span>
|
||||
<span class="n">FIGURE_ID</span> <span class="o">=</span> <span class="s2">"Results/FigureFiles"</span>
|
||||
<span class="n">DATA_ID</span> <span class="o">=</span> <span class="s2">"DataFiles/"</span>
|
||||
|
||||
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">PROJECT_ROOT_DIR</span><span class="p">):</span>
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">PROJECT_ROOT_DIR</span><span class="p">)</span>
|
||||
|
||||
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">):</span>
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">)</span>
|
||||
|
||||
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">):</span>
|
||||
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">)</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">FIGURE_ID</span><span class="p">,</span> <span class="n">fig_id</span><span class="p">)</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">data_path</span><span class="p">(</span><span class="n">dat_id</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">DATA_ID</span><span class="p">,</span> <span class="n">dat_id</span><span class="p">)</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">save_fig</span><span class="p">(</span><span class="n">fig_id</span><span class="p">):</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="n">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">)</span> <span class="o">+</span> <span class="s2">".png"</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="s1">'png'</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">FrankeFunction</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">):</span>
|
||||
<span class="n">term1</span> <span class="o">=</span> <span class="mf">0.75</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="mf">0.25</span><span class="o">*</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">-</span> <span class="mf">0.25</span><span class="o">*</span><span class="p">((</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">))</span>
|
||||
<span class="n">term2</span> <span class="o">=</span> <span class="mf">0.75</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">((</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="mf">49.0</span> <span class="o">-</span> <span class="mf">0.1</span><span class="o">*</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">+</span><span class="mi">1</span><span class="p">))</span>
|
||||
<span class="n">term3</span> <span class="o">=</span> <span class="mf">0.5</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">-</span><span class="mi">7</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="o">/</span><span class="mf">4.0</span> <span class="o">-</span> <span class="mf">0.25</span><span class="o">*</span><span class="p">((</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">-</span><span class="mi">3</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">))</span>
|
||||
<span class="n">term4</span> <span class="o">=</span> <span class="o">-</span><span class="mf">0.2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">x</span><span class="o">-</span><span class="mi">4</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span> <span class="o">-</span> <span class="p">(</span><span class="mi">9</span><span class="o">*</span><span class="n">y</span><span class="o">-</span><span class="mi">7</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="n">term1</span> <span class="o">+</span> <span class="n">term2</span> <span class="o">+</span> <span class="n">term3</span> <span class="o">+</span> <span class="n">term4</span>
|
||||
|
||||
|
||||
<span class="k">def</span> <span class="nf">create_X</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">n</span> <span class="p">):</span>
|
||||
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span> <span class="o">></span> <span class="mi">1</span><span class="p">:</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ravel</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ravel</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
|
||||
|
||||
<span class="n">N</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
|
||||
<span class="n">l</span> <span class="o">=</span> <span class="nb">int</span><span class="p">((</span><span class="n">n</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="p">(</span><span class="n">n</span><span class="o">+</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="mi">2</span><span class="p">)</span> <span class="c1"># Number of elements in beta</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">N</span><span class="p">,</span><span class="n">l</span><span class="p">))</span>
|
||||
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="n">n</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
|
||||
<span class="n">q</span> <span class="o">=</span> <span class="nb">int</span><span class="p">((</span><span class="n">i</span><span class="p">)</span><span class="o">*</span><span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">/</span><span class="mi">2</span><span class="p">)</span>
|
||||
<span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
|
||||
<span class="n">X</span><span class="p">[:,</span><span class="n">q</span><span class="o">+</span><span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="n">x</span><span class="o">**</span><span class="p">(</span><span class="n">i</span><span class="o">-</span><span class="n">k</span><span class="p">))</span><span class="o">*</span><span class="p">(</span><span class="n">y</span><span class="o">**</span><span class="n">k</span><span class="p">)</span>
|
||||
|
||||
<span class="k">return</span> <span class="n">X</span>
|
||||
|
||||
|
||||
<span class="c1"># Making meshgrid of datapoints and compute Franke's function</span>
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="mi">5</span>
|
||||
<span class="n">N</span> <span class="o">=</span> <span class="mi">1000</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">N</span><span class="p">))</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">uniform</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="n">N</span><span class="p">))</span>
|
||||
<span class="n">z</span> <span class="o">=</span> <span class="n">FrankeFunction</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">create_X</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">n</span><span class="o">=</span><span class="n">n</span><span class="p">)</span>
|
||||
<span class="c1"># split in training and test data</span>
|
||||
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span><span class="n">z</span><span class="p">,</span><span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="n">clf</span> <span class="o">=</span> <span class="n">skl</span><span class="o">.</span><span class="n">LinearRegression</span><span class="p">()</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># The mean squared error and R2 score</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"MSE before scaling: </span><span class="si">{:.2f}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">mean_squared_error</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">),</span> <span class="n">y_test</span><span class="p">)))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"R2 score before scaling </span><span class="si">{:.2f}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X_test</span><span class="p">,</span><span class="n">y_test</span><span class="p">)))</span>
|
||||
|
||||
<span class="n">scaler</span> <span class="o">=</span> <span class="n">StandardScaler</span><span class="p">()</span>
|
||||
<span class="n">scaler</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
|
||||
<span class="n">X_train_scaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
|
||||
<span class="n">X_test_scaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
|
||||
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Feature min values before scaling:</span><span class="se">\n</span><span class="s2"> </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">min</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Feature max values before scaling:</span><span class="se">\n</span><span class="s2"> </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)))</span>
|
||||
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Feature min values after scaling:</span><span class="se">\n</span><span class="s2"> </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="o">.</span><span class="n">min</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Feature max values after scaling:</span><span class="se">\n</span><span class="s2"> </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)))</span>
|
||||
|
||||
<span class="n">clf</span> <span class="o">=</span> <span class="n">skl</span><span class="o">.</span><span class="n">LinearRegression</span><span class="p">()</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train_scaled</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
|
||||
|
||||
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"MSE after scaling: </span><span class="si">{:.2f}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">mean_squared_error</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">),</span> <span class="n">y_test</span><span class="p">)))</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"R2 score for scaled data: </span><span class="si">{:.2f}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">clf</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">,</span><span class="n">y_test</span><span class="p">)))</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
|
||||
<span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">display</span>
|
||||
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
|
||||
<span class="c1"># setting up a 10 x 5 matrix</span>
|
||||
<span class="n">rows</span> <span class="o">=</span> <span class="mi">10</span>
|
||||
<span class="n">cols</span> <span class="o">=</span> <span class="mi">5</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">rows</span><span class="p">,</span><span class="n">cols</span><span class="p">)</span>
|
||||
<span class="n">XPandas</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
|
||||
<span class="n">display</span><span class="p">(</span><span class="n">XPandas</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">XPandas</span><span class="o">.</span><span class="n">mean</span><span class="p">())</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="n">XPandas</span><span class="o">.</span><span class="n">std</span><span class="p">())</span>
|
||||
<span class="n">XPandas</span> <span class="o">=</span> <span class="p">(</span><span class="n">XPandas</span> <span class="o">-</span><span class="n">XPandas</span><span class="o">.</span><span class="n">mean</span><span class="p">())</span>
|
||||
<span class="n">display</span><span class="p">(</span><span class="n">XPandas</span><span class="p">)</span>
|
||||
<span class="c1"># This option does not include the standard deviation</span>
|
||||
<span class="n">scaler</span> <span class="o">=</span> <span class="n">StandardScaler</span><span class="p">(</span><span class="n">with_std</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
|
||||
<span class="n">scaler</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
|
||||
<span class="n">Xscaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
|
||||
<span class="n">display</span><span class="p">(</span><span class="n">XPandas</span><span class="o">-</span><span class="n">Xscaled</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>MSE before scaling: 0.00
|
||||
R2 score before scaling 0.99
|
||||
Feature min values before scaling:
|
||||
[1.00000000e+00 2.31109204e-04 5.13266452e-03 5.34114641e-08
|
||||
1.18620601e-06 2.63442451e-05 1.23438810e-11 2.74143127e-10
|
||||
6.08839750e-09 1.35216172e-07 2.85278450e-15 6.33569998e-14
|
||||
1.40708470e-12 3.12497018e-11 6.94019247e-10 6.59304755e-19
|
||||
1.46423858e-17 3.25190225e-16 7.22209371e-15 1.60394236e-13
|
||||
3.56216797e-12]
|
||||
Feature max values before scaling:
|
||||
[1. 0.99997737 0.9950799 0.99995475 0.99505739 0.99018401
|
||||
0.99993212 0.99503487 0.99016161 0.98531221 0.9999095 0.99501236
|
||||
0.99013921 0.98528992 0.98046438 0.99988687 0.99498985 0.9901168
|
||||
0.98526763 0.9804422 0.9756404 ]
|
||||
Feature min values after scaling:
|
||||
[ 0. -1.7621419 -1.74382593 -1.13803928 -1.14662074 -1.15475232
|
||||
-0.89942598 -0.90825063 -0.9171356 -0.92605247 -0.76570177 -0.77317984
|
||||
-0.78080633 -0.78857629 -0.79648291 -0.67708423 -0.68316185 -0.68937695
|
||||
-0.69573183 -0.7022283 -0.70886748]
|
||||
Feature max values after scaling:
|
||||
[0. 1.71669651 1.63315151 2.21623344 2.15324631 2.088871
|
||||
2.62856593 2.5755035 2.52158335 2.46675058 2.987902 2.94034531
|
||||
2.8923 2.84372853 2.79459035 3.30968181 3.2653678 3.22076362
|
||||
3.1758506 3.13060772 3.08501147]
|
||||
MSE after scaling: 0.00
|
||||
R2 score for scaled data: 0.99
|
||||
<div class="output text_html"><div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.dataframe tbody tr th {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.dataframe thead th {
|
||||
text-align: right;
|
||||
}
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;">
|
||||
<th></th>
|
||||
<th>0</th>
|
||||
<th>1</th>
|
||||
<th>2</th>
|
||||
<th>3</th>
|
||||
<th>4</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>0</th>
|
||||
<td>-1.749765</td>
|
||||
<td>0.342680</td>
|
||||
<td>1.153036</td>
|
||||
<td>-0.252436</td>
|
||||
<td>0.981321</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>1</th>
|
||||
<td>0.514219</td>
|
||||
<td>0.221180</td>
|
||||
<td>-1.070043</td>
|
||||
<td>-0.189496</td>
|
||||
<td>0.255001</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>2</th>
|
||||
<td>-0.458027</td>
|
||||
<td>0.435163</td>
|
||||
<td>-0.583595</td>
|
||||
<td>0.816847</td>
|
||||
<td>0.672721</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>3</th>
|
||||
<td>-0.104411</td>
|
||||
<td>-0.531280</td>
|
||||
<td>1.029733</td>
|
||||
<td>-0.438136</td>
|
||||
<td>-1.118318</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>4</th>
|
||||
<td>1.618982</td>
|
||||
<td>1.541605</td>
|
||||
<td>-0.251879</td>
|
||||
<td>-0.842436</td>
|
||||
<td>0.184519</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>5</th>
|
||||
<td>0.937082</td>
|
||||
<td>0.731000</td>
|
||||
<td>1.361556</td>
|
||||
<td>-0.326238</td>
|
||||
<td>0.055676</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>6</th>
|
||||
<td>0.222400</td>
|
||||
<td>-1.443217</td>
|
||||
<td>-0.756352</td>
|
||||
<td>0.816454</td>
|
||||
<td>0.750445</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>7</th>
|
||||
<td>-0.455947</td>
|
||||
<td>1.189622</td>
|
||||
<td>-1.690617</td>
|
||||
<td>-1.356399</td>
|
||||
<td>-1.232435</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>8</th>
|
||||
<td>-0.544439</td>
|
||||
<td>-0.668172</td>
|
||||
<td>0.007315</td>
|
||||
<td>-0.612939</td>
|
||||
<td>1.299748</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>9</th>
|
||||
<td>-1.733096</td>
|
||||
<td>-0.983310</td>
|
||||
<td>0.357508</td>
|
||||
<td>-1.613579</td>
|
||||
<td>1.470714</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div></div><div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0 -0.175300
|
||||
1 0.083527
|
||||
2 -0.044334
|
||||
3 -0.399836
|
||||
4 0.331939
|
||||
dtype: float64
|
||||
0 1.069584
|
||||
1 0.965548
|
||||
2 1.018232
|
||||
3 0.793167
|
||||
4 0.918992
|
||||
dtype: float64
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output text_html"><div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.dataframe tbody tr th {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.dataframe thead th {
|
||||
text-align: right;
|
||||
}
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;">
|
||||
<th></th>
|
||||
<th>0</th>
|
||||
<th>1</th>
|
||||
<th>2</th>
|
||||
<th>3</th>
|
||||
<th>4</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>0</th>
|
||||
<td>-1.574465</td>
|
||||
<td>0.259153</td>
|
||||
<td>1.197370</td>
|
||||
<td>0.147400</td>
|
||||
<td>0.649382</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>1</th>
|
||||
<td>0.689519</td>
|
||||
<td>0.137652</td>
|
||||
<td>-1.025709</td>
|
||||
<td>0.210340</td>
|
||||
<td>-0.076938</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>2</th>
|
||||
<td>-0.282727</td>
|
||||
<td>0.351636</td>
|
||||
<td>-0.539261</td>
|
||||
<td>1.216683</td>
|
||||
<td>0.340782</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>3</th>
|
||||
<td>0.070889</td>
|
||||
<td>-0.614808</td>
|
||||
<td>1.074067</td>
|
||||
<td>-0.038300</td>
|
||||
<td>-1.450257</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>4</th>
|
||||
<td>1.794282</td>
|
||||
<td>1.458078</td>
|
||||
<td>-0.207545</td>
|
||||
<td>-0.442600</td>
|
||||
<td>-0.147420</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>5</th>
|
||||
<td>1.112383</td>
|
||||
<td>0.647473</td>
|
||||
<td>1.405890</td>
|
||||
<td>0.073598</td>
|
||||
<td>-0.276263</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>6</th>
|
||||
<td>0.397700</td>
|
||||
<td>-1.526744</td>
|
||||
<td>-0.712018</td>
|
||||
<td>1.216290</td>
|
||||
<td>0.418506</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>7</th>
|
||||
<td>-0.280647</td>
|
||||
<td>1.106095</td>
|
||||
<td>-1.646283</td>
|
||||
<td>-0.956563</td>
|
||||
<td>-1.564374</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>8</th>
|
||||
<td>-0.369139</td>
|
||||
<td>-0.751699</td>
|
||||
<td>0.051649</td>
|
||||
<td>-0.213103</td>
|
||||
<td>0.967809</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>9</th>
|
||||
<td>-1.557795</td>
|
||||
<td>-1.066837</td>
|
||||
<td>0.401842</td>
|
||||
<td>-1.213743</td>
|
||||
<td>1.138775</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div></div><div class="output text_html"><div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.dataframe tbody tr th {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.dataframe thead th {
|
||||
text-align: right;
|
||||
}
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;">
|
||||
<th></th>
|
||||
<th>0</th>
|
||||
<th>1</th>
|
||||
<th>2</th>
|
||||
<th>3</th>
|
||||
<th>4</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>0</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>1</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>2</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>3</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>4</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>5</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>6</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>7</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>8</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>9</th>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
<td>0.0</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div></div></div>
|
||||
</div>
|
||||
<p>Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.</p>
|
||||
<p>Another commonly used scaling method is min-max scaling. This is very
|
||||
useful for when we want the features to lie in a certain interval. To
|
||||
scale the feature <span class="math notranslate nohighlight">\(x_j\)</span> to the interval <span class="math notranslate nohighlight">\([a, b]\)</span>, we can apply the
|
||||
transformation</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
x_j^{(i)} \rightarrow (b-a)\frac{x_j^{(i)} - \min(x_j)}{\max(x_j) - \min(x_j)} - a
|
||||
\]</div>
|
||||
<p>where <span class="math notranslate nohighlight">\(\min(x_j)\)</span> and <span class="math notranslate nohighlight">\(\max(x_j)\)</span> return the minimum and maximum value of <span class="math notranslate nohighlight">\(x_j\)</span> over the data set, respectively.</p>
|
||||
</div>
|
||||
<div class="section" id="testing-the-means-squared-error-as-function-of-complexity">
|
||||
<h2><span class="section-number">3.10. </span>Testing the Means Squared Error as function of Complexity<a class="headerlink" href="#testing-the-means-squared-error-as-function-of-complexity" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Before we proceed with a more detailed analysis of the so-called
|
||||
Bias-Variance tradeoff, we present here an example of the relation
|
||||
between model complexity and the mean squared error for the triaining
|
||||
data and the test data.</p>
|
||||
<p>The results here tell us clearly that for the data not included in the
|
||||
training, there is an optimal model as function of the complexity of
|
||||
ourmodel (here in terms of the polynomial degree of the model).</p>
|
||||
<p>The results here will vary as function of model complexity and the amount od data used for training.</p>
|
||||
<p>Our data is defined by <span class="math notranslate nohighlight">\(x\in [-3,3]\)</span> with a total of for example <span class="math notranslate nohighlight">\(100\)</span> data points.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LinearRegression</span><span class="p">,</span> <span class="n">Ridge</span><span class="p">,</span> <span class="n">Lasso</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">PolynomialFeatures</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.pipeline</span> <span class="kn">import</span> <span class="n">make_pipeline</span>
|
||||
|
||||
|
||||
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">2018</span><span class="p">)</span>
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
|
||||
<span class="n">maxdegree</span> <span class="o">=</span> <span class="mi">14</span>
|
||||
<span class="c1"># Make data set.</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linspace</span><span class="p">(</span><span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mf">1.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
|
||||
<span class="n">TestError</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">maxdegree</span><span class="p">)</span>
|
||||
<span class="n">TrainError</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">maxdegree</span><span class="p">)</span>
|
||||
<span class="n">polydegree</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">maxdegree</span><span class="p">)</span>
|
||||
<span class="n">x_train</span><span class="p">,</span> <span class="n">x_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
|
||||
<span class="n">scaler</span> <span class="o">=</span> <span class="n">StandardScaler</span><span class="p">()</span>
|
||||
<span class="n">scaler</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">x_train</span><span class="p">)</span>
|
||||
<span class="n">x_train_scaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">x_train</span><span class="p">)</span>
|
||||
<span class="n">x_test_scaled</span> <span class="o">=</span> <span class="n">scaler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">x_test</span><span class="p">)</span>
|
||||
|
||||
<span class="k">for</span> <span class="n">degree</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">maxdegree</span><span class="p">):</span>
|
||||
<span class="n">model</span> <span class="o">=</span> <span class="n">make_pipeline</span><span class="p">(</span><span class="n">PolynomialFeatures</span><span class="p">(</span><span class="n">degree</span><span class="o">=</span><span class="n">degree</span><span class="p">),</span> <span class="n">LinearRegression</span><span class="p">(</span><span class="n">fit_intercept</span><span class="o">=</span><span class="kc">False</span><span class="p">))</span>
|
||||
<span class="n">clf</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">x_train_scaled</span><span class="p">,</span><span class="n">y_train</span><span class="p">)</span>
|
||||
<span class="n">y_fit</span> <span class="o">=</span> <span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">x_train_scaled</span><span class="p">)</span>
|
||||
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">x_test_scaled</span><span class="p">)</span>
|
||||
<span class="n">polydegree</span><span class="p">[</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">degree</span>
|
||||
<span class="n">TestError</span><span class="p">[</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">((</span><span class="n">y_test</span> <span class="o">-</span> <span class="n">y_pred</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="p">)</span>
|
||||
<span class="n">TrainError</span><span class="p">[</span><span class="n">degree</span><span class="p">]</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">((</span><span class="n">y_train</span> <span class="o">-</span> <span class="n">y_fit</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="p">)</span>
|
||||
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">polydegree</span><span class="p">,</span> <span class="n">TestError</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s1">'Test Error'</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">polydegree</span><span class="p">,</span> <span class="n">TrainError</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s1">'Train Error'</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter1_214_0.png" src="_images/chapter1_214_0.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="exercises">
|
||||
<h2><span class="section-number">3.9. </span>Exercises<a class="headerlink" href="#exercises" title="Permalink to this headline">¶</a></h2>
|
||||
<h2><span class="section-number">3.11. </span>Exercises<a class="headerlink" href="#exercises" title="Permalink to this headline">¶</a></h2>
|
||||
<div class="section" id="exercise-setting-up-various-python-environments">
|
||||
<h3><span class="section-number">3.9.1. </span>Exercise: Setting up various Python environments<a class="headerlink" href="#exercise-setting-up-various-python-environments" title="Permalink to this headline">¶</a></h3>
|
||||
<h3><span class="section-number">3.11.1. </span>Exercise: Setting up various Python environments<a class="headerlink" href="#exercise-setting-up-various-python-environments" title="Permalink to this headline">¶</a></h3>
|
||||
<p>The first exercise here is of a mere technical art. We want you to have</p>
|
||||
<ul class="simple">
|
||||
<li><p>git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo <a class="reference external" href="https://www.uio.no/tjenester/it/maskin/filer/versjonskontroll/github.html">GitHub facilities</a>.</p></li>
|
||||
@@ -2839,7 +3271,7 @@ license.</p>
|
||||
<p>We recommend using <strong>Anaconda</strong> if you are not too familiar with setting paths in a terminal environment.</p>
|
||||
</div>
|
||||
<div class="section" id="exercise-making-your-own-data-and-exploring-scikit-learn">
|
||||
<h3><span class="section-number">3.9.2. </span>Exercise: making your own data and exploring scikit-learn<a class="headerlink" href="#exercise-making-your-own-data-and-exploring-scikit-learn" title="Permalink to this headline">¶</a></h3>
|
||||
<h3><span class="section-number">3.11.2. </span>Exercise: making your own data and exploring scikit-learn<a class="headerlink" href="#exercise-making-your-own-data-and-exploring-scikit-learn" title="Permalink to this headline">¶</a></h3>
|
||||
<p>We will generate our own dataset for a function <span class="math notranslate nohighlight">\(y(x)\)</span> where <span class="math notranslate nohighlight">\(x \in [0,1]\)</span> and defined by random numbers computed with the uniform distribution. The function <span class="math notranslate nohighlight">\(y\)</span> is a quadratic polynomial in <span class="math notranslate nohighlight">\(x\)</span> with added stochastic noise according to the normal distribution <span class="math notranslate nohighlight">\(\cal {N}(0,1)\)</span>.
|
||||
The following simple Python instructions define our <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> values (with 100 data points).</p>
|
||||
<div class="cell docutils container">
|
||||
@@ -2875,7 +3307,7 @@ R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i
|
||||
Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.</p>
|
||||
</div>
|
||||
<div class="section" id="exercise-normalizing-our-data">
|
||||
<h3><span class="section-number">3.9.3. </span>Exercise: Normalizing our data<a class="headerlink" href="#exercise-normalizing-our-data" title="Permalink to this headline">¶</a></h3>
|
||||
<h3><span class="section-number">3.11.3. </span>Exercise: Normalizing our data<a class="headerlink" href="#exercise-normalizing-our-data" title="Permalink to this headline">¶</a></h3>
|
||||
<p>A much used approach before starting to train the data is to preprocess our
|
||||
data. Normally the data may need a rescaling and/or may be sensitive
|
||||
to extreme values. Scaling the data renders our inputs much more
|
||||
@@ -2987,7 +3419,7 @@ your results. For which polynomial degree do you find an optimal MSE
|
||||
<div class='prev-next-bottom'>
|
||||
|
||||
<a class='left-prev' id="prev-link" href="linalg.html" title="previous page"><span class="section-number">2. </span>Linear Algebra, Handling of Arrays and more Python Features</a>
|
||||
<a class='right-next' id="next-link" href="chapter2.html" title="next page"><span class="section-number">4. </span>Resampling Methods</a>
|
||||
<a class='right-next' id="next-link" href="chapter2.html" title="next page"><span class="section-number">4. </span>Ridge and Lasso Regression</a>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -587,10 +587,10 @@ number <span class="math notranslate nohighlight">\(i\)</span> is left out. Usin
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.139462 sec
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.136687 sec
|
||||
Jackknife Statistics :
|
||||
original bias std. error
|
||||
100.065 100.055 0.149894
|
||||
99.9196 99.9096 0.149076
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -718,10 +718,10 @@ theorem.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 2.40859 sec
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 1.78082 sec
|
||||
Bootstrap Statistics :
|
||||
original bias std. error
|
||||
100.148 14.7733 100.151 0.147722
|
||||
99.9769 14.9969 99.9765 0.149832
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
|
||||
|
||||
@@ -55,7 +55,7 @@
|
||||
<link rel="index" title="Index" href="genindex.html" />
|
||||
<link rel="search" title="Search" href="search.html" />
|
||||
<link rel="next" title="7. Support Vector Machines, overarching aims" href="chapter5.html" />
|
||||
<link rel="prev" title="5. Ridge and Lasso Regression" href="chapter3.html" />
|
||||
<link rel="prev" title="5. Resampling Methods" href="chapter3.html" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<meta name="docsearch:language" content="en" />
|
||||
|
||||
@@ -144,12 +144,12 @@
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter2.html">
|
||||
4. Resampling Methods
|
||||
4. Ridge and Lasso Regression
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter3.html">
|
||||
5. Ridge and Lasso Regression
|
||||
5. Resampling Methods
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1 current active">
|
||||
@@ -725,18 +725,112 @@ the probability of a given category. This leads us to the logistic function.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
|
||||
<span class="ne">FileNotFoundError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
|
||||
<span class="o"><</span><span class="n">ipython</span><span class="o">-</span><span class="nb">input</span><span class="o">-</span><span class="mi">1</span><span class="o">-</span><span class="n">a77d5ac269b2</span><span class="o">></span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">38</span> <span class="n">plt</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="n">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">)</span> <span class="o">+</span> <span class="s2">".png"</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="s1">'png'</span><span class="p">)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">39</span>
|
||||
<span class="ne">---> </span><span class="mi">40</span> <span class="n">infile</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">data_path</span><span class="p">(</span><span class="s2">"chddata.csv"</span><span class="p">),</span><span class="s1">'r'</span><span class="p">)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">41</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">42</span> <span class="c1"># Read the chd data as csv file and organize the data into arrays with age group, age, and chd</span>
|
||||
<div class="output text_html"><div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
<span class="ne">FileNotFoundError</span>: [Errno 2] No such file or directory: 'DataFiles/chddata.csv'
|
||||
</pre></div>
|
||||
</div>
|
||||
.dataframe tbody tr th {
|
||||
vertical-align: top;
|
||||
}
|
||||
|
||||
.dataframe thead th {
|
||||
text-align: right;
|
||||
}
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;">
|
||||
<th></th>
|
||||
<th>ID</th>
|
||||
<th>Age</th>
|
||||
<th>Agegroup</th>
|
||||
<th>CHD</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>0</th>
|
||||
<td>1</td>
|
||||
<td>21</td>
|
||||
<td>1</td>
|
||||
<td>0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>1</th>
|
||||
<td>2</td>
|
||||
<td>23</td>
|
||||
<td>1</td>
|
||||
<td>0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>2</th>
|
||||
<td>3</td>
|
||||
<td>25</td>
|
||||
<td>1</td>
|
||||
<td>1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>3</th>
|
||||
<td>4</td>
|
||||
<td>29</td>
|
||||
<td>1</td>
|
||||
<td>0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>4</th>
|
||||
<td>5</td>
|
||||
<td>21</td>
|
||||
<td>1</td>
|
||||
<td>0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>...</th>
|
||||
<td>...</td>
|
||||
<td>...</td>
|
||||
<td>...</td>
|
||||
<td>...</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>95</th>
|
||||
<td>96</td>
|
||||
<td>61</td>
|
||||
<td>8</td>
|
||||
<td>1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>96</th>
|
||||
<td>97</td>
|
||||
<td>69</td>
|
||||
<td>8</td>
|
||||
<td>1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>97</th>
|
||||
<td>98</td>
|
||||
<td>65</td>
|
||||
<td>8</td>
|
||||
<td>1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>98</th>
|
||||
<td>99</td>
|
||||
<td>64</td>
|
||||
<td>8</td>
|
||||
<td>1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>99</th>
|
||||
<td>100</td>
|
||||
<td>63</td>
|
||||
<td>8</td>
|
||||
<td>0</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>100 rows × 4 columns</p>
|
||||
</div></div><img alt="_images/chapter4_5_1.png" src="_images/chapter4_5_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>What we could attempt however is to plot the mean value for each group.</p>
|
||||
@@ -753,6 +847,9 @@ the probability of a given category. This leads us to the logistic function.</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter4_7_0.png" src="_images/chapter4_7_0.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>We are now trying to find a function <span class="math notranslate nohighlight">\(f(y\vert x)\)</span>, that is a function which gives us an expected value for the output <span class="math notranslate nohighlight">\(y\)</span> with a given input <span class="math notranslate nohighlight">\(x\)</span>.
|
||||
In standard linear regression with a linear dependence on <span class="math notranslate nohighlight">\(x\)</span>, we would write this in terms of our model</p>
|
||||
@@ -852,6 +949,11 @@ p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter4_13_0.png" src="_images/chapter4_13_0.png" />
|
||||
<img alt="_images/chapter4_13_1.png" src="_images/chapter4_13_1.png" />
|
||||
<img alt="_images/chapter4_13_2.png" src="_images/chapter4_13_2.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>We assume now that we have two classes with <span class="math notranslate nohighlight">\(y_i\)</span> either <span class="math notranslate nohighlight">\(0\)</span> or <span class="math notranslate nohighlight">\(1\)</span>. Furthermore we assume also that we have only two parameters <span class="math notranslate nohighlight">\(\beta\)</span> in our fitting of the Sigmoid function, that is we define probabilities</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
@@ -1016,6 +1118,24 @@ classification.</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>(426, 30)
|
||||
(143, 30)
|
||||
Test set accuracy with Logistic Regression: 0.94
|
||||
Test set accuracy Logistic Regression with scaled data: 0.96
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:762: ConvergenceWarning: lbfgs failed to converge (status=1):
|
||||
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
|
||||
|
||||
Increase the number of iterations (max_iter) or scale the data as shown in:
|
||||
https://scikit-learn.org/stable/modules/preprocessing.html
|
||||
Please also refer to the documentation for alternative solver options:
|
||||
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
|
||||
n_iter_i = _check_optimize_result(
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>In addition to the above scores, we could also study the covariance (and the correlation matrix).
|
||||
We use <strong>Pandas</strong> to compute the correlation matrix.</p>
|
||||
@@ -1058,6 +1178,10 @@ We use <strong>Pandas</strong> to compute the correlation matrix.</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter4_51_0.png" src="_images/chapter4_51_0.png" />
|
||||
<img alt="_images/chapter4_51_1.png" src="_images/chapter4_51_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>In the above example we note two things. In the first plot we display
|
||||
the overlap of benign and malignant tumors as functions of the various
|
||||
@@ -1138,6 +1262,39 @@ applications. This will be discussed later this semester (<a class="reference ex
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>(426, 30)
|
||||
(143, 30)
|
||||
Test set accuracy with Logistic Regression: 0.94
|
||||
Test set accuracy Logistic Regression with scaled data: 0.96
|
||||
[1. 1. 1. 1. 1. 1.
|
||||
1. 1. 0.92857143 0.92857143]
|
||||
Test set accuracy with Logistic Regression and scaled data: 0.96
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:762: ConvergenceWarning: lbfgs failed to converge (status=1):
|
||||
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
|
||||
|
||||
Increase the number of iterations (max_iter) or scale the data as shown in:
|
||||
https://scikit-learn.org/stable/modules/preprocessing.html
|
||||
Please also refer to the documentation for alternative solver options:
|
||||
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
|
||||
n_iter_i = _check_optimize_result(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
|
||||
<span class="ne">ModuleNotFoundError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
|
||||
<span class="o"><</span><span class="n">ipython</span><span class="o">-</span><span class="nb">input</span><span class="o">-</span><span class="mi">8</span><span class="o">-</span><span class="mi">12</span><span class="n">adb44b1c20</span><span class="o">></span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">34</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">35</span>
|
||||
<span class="ne">---> </span><span class="mi">36</span> <span class="kn">import</span> <span class="nn">scikitplot</span> <span class="k">as</span> <span class="nn">skplt</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">37</span> <span class="n">y_pred</span> <span class="o">=</span> <span class="n">logreg</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test_scaled</span><span class="p">)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">38</span> <span class="n">skplt</span><span class="o">.</span><span class="n">metrics</span><span class="o">.</span><span class="n">plot_confusion_matrix</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">,</span> <span class="n">normalize</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
|
||||
|
||||
<span class="ne">ModuleNotFoundError</span>: No module named 'scikitplot'
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="optimization-the-central-part-of-any-machine-learning-algortithm">
|
||||
@@ -2256,7 +2413,7 @@ function.</p>
|
||||
|
||||
<div class='prev-next-bottom'>
|
||||
|
||||
<a class='left-prev' id="prev-link" href="chapter3.html" title="previous page"><span class="section-number">5. </span>Ridge and Lasso Regression</a>
|
||||
<a class='left-prev' id="prev-link" href="chapter3.html" title="previous page"><span class="section-number">5. </span>Resampling Methods</a>
|
||||
<a class='right-next' id="next-link" href="chapter5.html" title="next page"><span class="section-number">7. </span>Support Vector Machines, overarching aims</a>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -17,73 +17,61 @@ Traceback (most recent call last):
|
||||
raise CellExecutionError.from_cell_and_msg(cell, exec_reply['content'])
|
||||
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
|
||||
------------------
|
||||
%matplotlib inline
|
||||
|
||||
# Common imports
|
||||
import os
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import matplotlib.pyplot as plt
|
||||
from sklearn.linear_model import LinearRegression, Ridge, Lasso
|
||||
from sklearn.model_selection import train_test_split
|
||||
from sklearn.utils import resample
|
||||
from sklearn.metrics import mean_squared_error
|
||||
from IPython.display import display
|
||||
from pylab import plt, mpl
|
||||
plt.style.use('seaborn')
|
||||
mpl.rcParams['font.family'] = 'serif'
|
||||
import numpy as np
|
||||
from sklearn.model_selection import train_test_split
|
||||
from sklearn.datasets import load_breast_cancer
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
|
||||
# Where to save the figures and data files
|
||||
PROJECT_ROOT_DIR = "Results"
|
||||
FIGURE_ID = "Results/FigureFiles"
|
||||
DATA_ID = "DataFiles/"
|
||||
# Load the data
|
||||
cancer = load_breast_cancer()
|
||||
|
||||
if not os.path.exists(PROJECT_ROOT_DIR):
|
||||
os.mkdir(PROJECT_ROOT_DIR)
|
||||
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
|
||||
print(X_train.shape)
|
||||
print(X_test.shape)
|
||||
# Logistic Regression
|
||||
logreg = LogisticRegression(solver='lbfgs')
|
||||
logreg.fit(X_train, y_train)
|
||||
print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
|
||||
#now scale the data
|
||||
from sklearn.preprocessing import StandardScaler
|
||||
scaler = StandardScaler()
|
||||
scaler.fit(X_train)
|
||||
X_train_scaled = scaler.transform(X_train)
|
||||
X_test_scaled = scaler.transform(X_test)
|
||||
# Logistic Regression
|
||||
logreg.fit(X_train_scaled, y_train)
|
||||
print("Test set accuracy Logistic Regression with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
|
||||
|
||||
if not os.path.exists(FIGURE_ID):
|
||||
os.makedirs(FIGURE_ID)
|
||||
|
||||
if not os.path.exists(DATA_ID):
|
||||
os.makedirs(DATA_ID)
|
||||
from sklearn.preprocessing import LabelEncoder
|
||||
from sklearn.model_selection import cross_validate
|
||||
#Cross validation
|
||||
accuracy = cross_validate(logreg,X_test_scaled,y_test,cv=10)['test_score']
|
||||
print(accuracy)
|
||||
print("Test set accuracy with Logistic Regression and scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
|
||||
|
||||
def image_path(fig_id):
|
||||
return os.path.join(FIGURE_ID, fig_id)
|
||||
|
||||
def data_path(dat_id):
|
||||
return os.path.join(DATA_ID, dat_id)
|
||||
|
||||
def save_fig(fig_id):
|
||||
plt.savefig(image_path(fig_id) + ".png", format='png')
|
||||
|
||||
infile = open(data_path("chddata.csv"),'r')
|
||||
|
||||
# Read the chd data as csv file and organize the data into arrays with age group, age, and chd
|
||||
chd = pd.read_csv(infile, names=('ID', 'Age', 'Agegroup', 'CHD'))
|
||||
chd.columns = ['ID', 'Age', 'Agegroup', 'CHD']
|
||||
output = chd['CHD']
|
||||
age = chd['Age']
|
||||
agegroup = chd['Agegroup']
|
||||
numberID = chd['ID']
|
||||
display(chd)
|
||||
|
||||
plt.scatter(age, output, marker='o')
|
||||
plt.axis([18,70.0,-0.1, 1.2])
|
||||
plt.xlabel(r'Age')
|
||||
plt.ylabel(r'CHD')
|
||||
plt.title(r'Age distribution and Coronary heart disease')
|
||||
import scikitplot as skplt
|
||||
y_pred = logreg.predict(X_test_scaled)
|
||||
skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
|
||||
plt.show()
|
||||
y_probas = logreg.predict_proba(X_test_scaled)
|
||||
skplt.metrics.plot_roc(y_test, y_probas)
|
||||
plt.show()
|
||||
skplt.metrics.plot_cumulative_gain(y_test, y_probas)
|
||||
plt.show()
|
||||
------------------
|
||||
|
||||
[0;31m---------------------------------------------------------------------------[0m
|
||||
[0;31mFileNotFoundError[0m Traceback (most recent call last)
|
||||
[0;32m<ipython-input-1-a77d5ac269b2>[0m in [0;36m<module>[0;34m[0m
|
||||
[1;32m 38[0m [0mplt[0m[0;34m.[0m[0msavefig[0m[0;34m([0m[0mimage_path[0m[0;34m([0m[0mfig_id[0m[0;34m)[0m [0;34m+[0m [0;34m".png"[0m[0;34m,[0m [0mformat[0m[0;34m=[0m[0;34m'png'[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
|
||||
[1;32m 39[0m [0;34m[0m[0m
|
||||
[0;32m---> 40[0;31m [0minfile[0m [0;34m=[0m [0mopen[0m[0;34m([0m[0mdata_path[0m[0;34m([0m[0;34m"chddata.csv"[0m[0;34m)[0m[0;34m,[0m[0;34m'r'[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
|
||||
[0m[1;32m 41[0m [0;34m[0m[0m
|
||||
[1;32m 42[0m [0;31m# Read the chd data as csv file and organize the data into arrays with age group, age, and chd[0m[0;34m[0m[0;34m[0m[0;34m[0m[0m
|
||||
[0;31mModuleNotFoundError[0m Traceback (most recent call last)
|
||||
[0;32m<ipython-input-8-12adb44b1c20>[0m in [0;36m<module>[0;34m[0m
|
||||
[1;32m 34[0m [0;34m[0m[0m
|
||||
[1;32m 35[0m [0;34m[0m[0m
|
||||
[0;32m---> 36[0;31m [0;32mimport[0m [0mscikitplot[0m [0;32mas[0m [0mskplt[0m[0;34m[0m[0;34m[0m[0m
|
||||
[0m[1;32m 37[0m [0my_pred[0m [0;34m=[0m [0mlogreg[0m[0;34m.[0m[0mpredict[0m[0;34m([0m[0mX_test_scaled[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
|
||||
[1;32m 38[0m [0mskplt[0m[0;34m.[0m[0mmetrics[0m[0;34m.[0m[0mplot_confusion_matrix[0m[0;34m([0m[0my_test[0m[0;34m,[0m [0my_pred[0m[0;34m,[0m [0mnormalize[0m[0;34m=[0m[0;32mTrue[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
|
||||
|
||||
[0;31mFileNotFoundError[0m: [Errno 2] No such file or directory: 'DataFiles/chddata.csv'
|
||||
FileNotFoundError: [Errno 2] No such file or directory: 'DataFiles/chddata.csv'
|
||||
[0;31mModuleNotFoundError[0m: No module named 'scikitplot'
|
||||
ModuleNotFoundError: No module named 'scikitplot'
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user