diff --git a/doc/BookChapters/chapter1.dlog b/doc/BookChapters/chapter1.dlog index c30a72cfc..ce4fec57d 100644 --- a/doc/BookChapters/chapter1.dlog +++ b/doc/BookChapters/chapter1.dlog @@ -87,3 +87,13 @@ found info about 5 exercises *** warning: latex envir \begin{bmatrix} does not work well in Markdown. Stick to \[ ... \], equation, equation*, align, or align* environments in math environments. output in chapter1.ipynb +Translating doconce text in chapter1.do.txt to ipynb +*** replacing \bm{...} by \boldsymbol{...} (\bm is not supported by MathJax) +found info about 5 exercises + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. Stick to \[ ... \], equation, equation*, align, or align* environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. Stick to \[ ... \], equation, equation*, align, or align* environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. Stick to \[ ... \], equation, equation*, align, or align* environments in math environments. +output in chapter1.ipynb diff --git a/doc/BookChapters/chapter1.do.txt b/doc/BookChapters/chapter1.do.txt index 4ac46ba31..31f5fc14c 100644 --- a/doc/BookChapters/chapter1.do.txt +++ b/doc/BookChapters/chapter1.do.txt @@ -684,21 +684,6 @@ infile = open(data_path("MassEval2016.dat"),'r') !ec -Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various _matplotlib_ commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function. -!bc pycod -from pylab import plt, mpl -plt.style.use('seaborn') -mpl.rcParams['font.family'] = 'serif' - -def MakePlot(x,y, styles, labels, axlabels): - plt.figure(figsize=(10,6)) - for i in range(len(x)): - plt.plot(x[i], y[i], styles[i], label = labels[i]) - plt.xlabel(axlabels[0]) - plt.ylabel(axlabels[1]) - plt.legend(loc=0) -!ec - Our next step is to read the data on experimental binding energies and reorganize them as functions of the mass number $A$, the number of protons $Z$ and neutrons $N$ using _pandas_. Before we do this it is @@ -903,10 +888,10 @@ plt.show() "Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureAug27.mp4?vrtx=view-as-webpage". -Fitting a continuous function with linear parameterization in terms of the parameters $\bm{\beta}$. +Fitting a continuous function with linear parameterization in terms of the parameters $\bm{\theta}$. * Method of choice for fitting a continuous function! * Gives an excellent introduction to central Machine Learning features with _understandable pedagogical_ links to other methods like _Neural Networks_, _Support Vector Machines_ etc -* Analytical expression for the fitting parameters $\bm{\beta}$ +* Analytical expression for the fitting parameters $\bm{\theta}$ * Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more * Analytical relation with probabilistic interpretations * Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics @@ -920,12 +905,12 @@ Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also re Regression modeling deals with the description of the sampling distribution of a given random variable $y$ and how it varies as function of another variable or a set of such variables $\bm{x} =[x_0, x_1,\dots, x_{n-1}]^T$. -The first variable is called the _dependent_, the _outcome_ or the _response_ variable while the set of variables $\bm{x}$ is called the independent variable, or the predictor variable or the explanatory variable. +The first variable is called the _dependent_, the _outcome_ or the _response_ or just the output variable while the set of variables $\bm{x}$ is called the independent variable, or the predictor variable or the explanatory variable, or just the input variable. We will hereafter call $\bm{y}$ and $\bm{x}x$ for the output and input variables, respectively. -A regression model aims at finding a likelihood function $p(\bm{y}\vert \bm{x})$, that is the conditional distribution for $\bm{y}$ with a given $\bm{x}$. The estimation of $p(\bm{y}\vert \bm{x})$ is made using a data set with +A regression model aims at finding a likelihood function $p(\bm{y}\vert \bm{x})$ (or just a function $f(\bm{x}$) , that is the conditional distribution for $\bm{y}$ with a given $\bm{x}$. The estimation of $p(\bm{y}\vert \bm{x})$ is made using a data set with * $n$ cases $i = 0, 1, 2, \dots, n-1$ -* Response (target, dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \dots, n-1$ -* $p$ so-called explanatory (independent or predictor) variables $\bm{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}]$ with $i = 0, 1, 2, \dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples. +* Response/output (target, dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \dots, n-1$ +* $p$ so-called explanatory/input (independent or predictor) variables $\bm{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}]$ with $i = 0, 1, 2, \dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples. The goal of the regression analysis is to extract/exploit relationship between $\bm{y}$ and $\bm{x}$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things. @@ -942,10 +927,10 @@ $\bm{X}$ through a functional relationship like $y_i = f(\mathbf{X}_{i,\ast})$. When no prior knowledge on the form of $f(\cdot)$ is available, it is common to assume a linear relationship between $\bm{X}$ and $\bm{y}$. This assumption gives rise to -the *linear regression model* where $\bm{\beta} = [\beta_0, \ldots, -\beta_{p-1}]^{T}$ are the *regression parameters*. +the *linear regression model* where $\bm{\theta} = [\theta_0, \ldots, +\theta_{p-1}]^{T}$ are the *regression parameters*. -Linear regression gives us a set of analytical equations for the parameters $\beta_j$. +Linear regression gives us a set of analytical equations for the parameters $\theta_j$. In order to understand the relation among the predictors $p$, the set of data $n$ and the target (outcome, output etc) $\bm{y}$, @@ -971,7 +956,7 @@ Before we proceed let us study a case from linear algebra where we aim at fittin Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of $y$ which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree $n-1$ with $n$ points, that is !bt \[ -y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i, +y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \theta_j x_i^j+\epsilon_i, \] !et where $\epsilon_i$ is the error in our approximation. @@ -980,11 +965,11 @@ where $\epsilon_i$ is the error in our approximation. For every set of values $y_i,x_i$ we have thus the corresponding set of equations !bt \begin{align*} -y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\ -y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\ -y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\ +y_0&=\theta_0+\theta_1x_0^1+\theta_2x_0^2+\dots+\theta_{n-1}x_0^{n-1}+\epsilon_0\\ +y_1&=\theta_0+\theta_1x_1^1+\theta_2x_1^2+\dots+\theta_{n-1}x_1^{n-1}+\epsilon_1\\ +y_2&=\theta_0+\theta_1x_2^1+\theta_2x_2^2+\dots+\theta_{n-1}x_2^{n-1}+\epsilon_2\\ \dots & \dots \\ -y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\ +y_{n-1}&=\theta_0+\theta_1x_{n-1}^1+\theta_2x_{n-1}^2+\dots+\theta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\ \end{align*} !et @@ -1000,7 +985,7 @@ Defining the vectors and !bt \[ -\bm{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T, +\bm{\theta} = [\theta_0,\theta_1, \theta_2,\dots, \theta_{n-1}]^T, \] !et and @@ -1025,7 +1010,7 @@ and the design matrix we can rewrite our equations as !bt \[ -\bm{y} = \bm{X}\bm{\beta}+\bm{\epsilon}. +\bm{y} = \bm{X}\bm{\theta}+\bm{\epsilon}. \] !et The above design matrix is called a "Vandermonde matrix":"https://en.wikipedia.org/wiki/Vandermonde_matrix". @@ -1038,13 +1023,13 @@ of values $y_i,x_i$ we can then generalize the equations to !bt \begin{align*} -y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ -y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ -y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\ +y_0&=\theta_0x_{00}+\theta_1x_{01}+\theta_2x_{02}+\dots+\theta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\theta_0x_{10}+\theta_1x_{11}+\theta_2x_{12}+\dots+\theta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\theta_0x_{20}+\theta_1x_{21}+\theta_2x_{22}+\dots+\theta_{n-1}x_{2n-1}+\epsilon_2\\ \dots & \dots \\ -y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\ +y_{i}&=\theta_0x_{i0}+\theta_1x_{i1}+\theta_2x_{i2}+\dots+\theta_{n-1}x_{in-1}+\epsilon_i\\ \dots & \dots \\ -y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +y_{n-1}&=\theta_0x_{n-1,0}+\theta_1x_{n-1,2}+\theta_2x_{n-1,2}+\dots+\theta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ \end{align*} !et @@ -1066,21 +1051,21 @@ x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\ and without loss of generality we rewrite again our equations as !bt \[ -\bm{y} = \bm{X}\bm{\beta}+\bm{\epsilon}. +\bm{y} = \bm{X}\bm{\theta}+\bm{\epsilon}. \] !et -The left-hand side of this equation is kwown. Our error vector $\bm{\epsilon}$ and the parameter vector $\bm{\beta}$ are our unknow quantities. How can we obtain the optimal set of $\beta_i$ values? +The left-hand side of this equation is kwown. Our error vector $\bm{\epsilon}$ and the parameter vector $\bm{\theta}$ are our unknow quantities. How can we obtain the optimal set of $\theta_i$ values? We have defined the matrix $\bm{X}$ via the equations !bt \begin{align*} -y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ -y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ -y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\ +y_0&=\theta_0x_{00}+\theta_1x_{01}+\theta_2x_{02}+\dots+\theta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\theta_0x_{10}+\theta_1x_{11}+\theta_2x_{12}+\dots+\theta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\theta_0x_{20}+\theta_1x_{21}+\theta_2x_{22}+\dots+\theta_{n-1}x_{2n-1}+\epsilon_1\\ \dots & \dots \\ -y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\ +y_{i}&=\theta_0x_{i0}+\theta_1x_{i1}+\theta_2x_{i2}+\dots+\theta_{n-1}x_{in-1}+\epsilon_1\\ \dots & \dots \\ -y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +y_{n-1}&=\theta_0x_{n-1,0}+\theta_1x_{n-1,2}+\theta_2x_{n-1,2}+\dots+\theta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ \end{align*} !et @@ -1163,30 +1148,30 @@ DesignMatrix.columns = ['1', 'A', 'A^(2/3)', 'A^(-1/3)', '1/A'] display(DesignMatrix) !ec -With $\bm{\beta}\in {\mathbb{R}}^{p\times 1}$, it means that we will hereafter write our equations for the approximation as +With $\bm{\theta}\in {\mathbb{R}}^{p\times 1}$, it means that we will hereafter write our equations for the approximation as !bt \[ -\bm{\tilde{y}}= \bm{X}\bm{\beta}, +\bm{\tilde{y}}= \bm{X}\bm{\theta}, \] !et throughout these lectures. -With the above we use the design matrix to define the approximation $\bm{\tilde{y}}$ via the unknown quantity $\bm{\beta}$ as +With the above we use the design matrix to define the approximation $\bm{\tilde{y}}$ via the unknown quantity $\bm{\theta}$ as !bt \[ -\bm{\tilde{y}}= \bm{X}\bm{\beta}, +\bm{\tilde{y}}= \bm{X}\bm{\theta}, \] !et -and in order to find the optimal parameters $\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\tilde{y}_i$, namely +and in order to find the optimal parameters $\theta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\tilde{y}_i$, namely !bt \[ -C(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\}, +C(\bm{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\}, \] !et or using the matrix $\bm{X}$ and in a more compact matrix-vector notation as !bt \[ -C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}. +C(\bm{\theta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}. \] !et This function is one possible way to define the so-called cost function. @@ -1198,22 +1183,22 @@ the function $C$ as !bt \[ -C(\bm{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2, +C(\bm{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2, \] !et -since when taking the first derivative with respect to the unknown parameters $\beta$, the factor of $2$ cancels out. +since when taking the first derivative with respect to the unknown parameters $\theta$, the factor of $2$ cancels out. The function !bt \[ -C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}, +C(\bm{\theta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}, \] !et can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value. When linking (see the discussion below) with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value !bt \[ -y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i, +y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i, \] !et @@ -1226,29 +1211,29 @@ error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat $y_i$ as our exact value for the response variable. -In order to find the parameters $\beta_i$ we will then minimize the spread of $C(\bm{\beta})$, that is we are going to solve the problem +In order to find the parameters $\theta_i$ we will then minimize the spread of $C(\bm{\theta})$, that is we are going to solve the problem !bt \[ -{\displaystyle \min_{\bm{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}. +{\displaystyle \min_{\bm{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}. \] !et In practical terms it means we will require !bt \[ -\frac{\partial C(\bm{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, +\frac{\partial C(\bm{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0, \] !et which results in !bt \[ -\frac{\partial C(\bm{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, +\frac{\partial C(\bm{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0, \] !et or in a matrix-vector form as !bt \[ -\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right). +\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right). \] !et @@ -1257,19 +1242,19 @@ or in a matrix-vector form as We can rewrite !bt \[ -\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right), +\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right), \] !et as !bt \[ -\bm{X}^T\bm{y} = \bm{X}^T\bm{X}\bm{\beta}, +\bm{X}^T\bm{y} = \bm{X}^T\bm{X}\bm{\theta}, \] !et and if the matrix $\bm{X}^T\bm{X}$ is invertible we have the solution !bt \[ -\bm{\beta} =\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\bm{\theta} =\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. \] !et @@ -1314,7 +1299,7 @@ of the means squared error. This leads to !bt \[ -\frac{\partial^2 C(\bm{\beta})}{\partial \bm{\beta}^T\partial \bm{\beta}} =\frac{2}{n}\bm{X}^T\bm{X}. +\frac{\partial^2 C(\bm{\theta})}{\partial \bm{\theta}^T\partial \bm{\theta}} =\frac{2}{n}\bm{X}^T\bm{X}. \] !et This quantity defines the so- called the Hessian matrix. @@ -1337,34 +1322,34 @@ terms of the singular values. The residuals $\bm{\epsilon}$ are in turn given by !bt \[ -\bm{\epsilon} = \bm{y}-\bm{\tilde{y}} = \bm{y}-\bm{X}\bm{\beta}, +\bm{\epsilon} = \bm{y}-\bm{\tilde{y}} = \bm{y}-\bm{X}\bm{\theta}, \] !et and with !bt \[ -\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0, +\bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right)= 0, \] !et we have !bt \[ -\bm{X}^T\bm{\epsilon}=\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0, +\bm{X}^T\bm{\epsilon}=\bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right)= 0, \] !et -meaning that the solution for $\bm{\beta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. +meaning that the solution for $\bm{\theta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. Let us now return to our nuclear binding energies and simply code the above equations. -It is rather straightforward to implement the matrix inversion and obtain the parameters $\bm{\beta}$. After having defined the matrix $\bm{X}$ we simply need to +It is rather straightforward to implement the matrix inversion and obtain the parameters $\bm{\theta}$. After having defined the matrix $\bm{X}$ we simply need to write !bc pycod -# matrix inversion to find beta -beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies) +# matrix inversion to find theta +theta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies) # and then make the prediction -ytilde = X @ beta +ytilde = X @ theta !ec Alternatively, you can use the least squares functionality in _Numpy_ as !bc pycod @@ -1432,28 +1417,28 @@ as !bt \[ -\chi^2(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\frac{1}{\bm{\Sigma^2}}\left(\bm{y}-\bm{\tilde{y}}\right)\right\}, +\chi^2(\bm{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\frac{1}{\bm{\Sigma^2}}\left(\bm{y}-\bm{\tilde{y}}\right)\right\}, \] !et where the matrix $\bm{\Sigma}$ is a diagonal matrix with $\sigma_i$ as matrix elements. -In order to find the parameters $\beta_i$ we will then minimize the spread of $\chi^2(\bm{\beta})$ by requiring +In order to find the parameters $\theta_i$ we will then minimize the spread of $\chi^2(\bm{\theta})$ by requiring !bt \[ -\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0, +\frac{\partial \chi^2(\bm{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0, \] !et which results in !bt \[ -\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0, +\frac{\partial \chi^2(\bm{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0, \] !et or in a matrix-vector form as !bt \[ -\frac{\partial \chi^2(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{A}^T\left( \bm{b}-\bm{A}\bm{\beta}\right). +\frac{\partial \chi^2(\bm{\theta})}{\partial \bm{\theta}} = 0 = \bm{A}^T\left( \bm{b}-\bm{A}\bm{\theta}\right). \] !et where we have defined the matrix $\bm{A} =\bm{X}/\bm{\Sigma}$ with matrix elements $a_{ij} = x_{ij}/\sigma_i$ and the vector $\bm{b}$ with elements $b_i = y_i/\sigma_i$. @@ -1461,19 +1446,19 @@ where we have defined the matrix $\bm{A} =\bm{X}/\bm{\Sigma}$ with matrix elemen We can rewrite !bt \[ -\frac{\partial \chi^2(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{A}^T\left( \bm{b}-\bm{A}\bm{\beta}\right), +\frac{\partial \chi^2(\bm{\theta})}{\partial \bm{\theta}} = 0 = \bm{A}^T\left( \bm{b}-\bm{A}\bm{\theta}\right), \] !et as !bt \[ -\bm{A}^T\bm{b} = \bm{A}^T\bm{A}\bm{\beta}, +\bm{A}^T\bm{b} = \bm{A}^T\bm{A}\bm{\theta}, \] !et and if the matrix $\bm{A}^T\bm{A}$ is invertible we have the solution !bt \[ -\bm{\beta} =\left(\bm{A}^T\bm{A}\right)^{-1}\bm{A}^T\bm{b}. +\bm{\theta} =\left(\bm{A}^T\bm{A}\right)^{-1}\bm{A}^T\bm{b}. \] !et @@ -1484,41 +1469,41 @@ If we then introduce the matrix \bm{H} = \left(\bm{A}^T\bm{A}\right)^{-1}, \] !et -we have then the following expression for the parameters $\beta_j$ (the matrix elements of $\bm{H}$ are $h_{ij}$) +we have then the following expression for the parameters $\theta_j$ (the matrix elements of $\bm{H}$ are $h_{ij}$) !bt \[ -\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik} +\theta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik} \] !et -We state without proof the expression for the uncertainty in the parameters $\beta_j$ as (we leave this as an exercise) +We state without proof the expression for the uncertainty in the parameters $\theta_j$ as (we leave this as an exercise) !bt \[ -\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2, +\sigma^2(\theta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \theta_j}{\partial y_i}\right)^2, \] !et resulting in !bt \[ -\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}! +\sigma^2(\theta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}! \] !et The first step here is to approximate the function $y$ with a first-order polynomial, that is we write !bt \[ -y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i. +y=y(x) \rightarrow y(x_i) \approx \theta_0+\theta_1 x_i. \] !et -By computing the derivatives of $\chi^2$ with respect to $\beta_0$ and $\beta_1$ show that these are given by +By computing the derivatives of $\chi^2$ with respect to $\theta_0$ and $\theta_1$ show that these are given by !bt \[ -\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0, +\frac{\partial \chi^2(\bm{\theta})}{\partial \theta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\theta_0-\theta_1x_{i}}{\sigma_i^2}\right)\right]=0, \] !et and !bt \[ -\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0. +\frac{\partial \chi^2(\bm{\theta})}{\partial \theta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\theta_0-\theta_1x_{i}}{\sigma_i^2}\right)\right]=0. \] !et @@ -1558,19 +1543,19 @@ we obtain !bt \[ -\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}, +\theta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}, \] !et !bt \[ -\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}. +\theta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}. \] !et This approach (different linear and non-linear regression) suffers often from both being underdetermined and overdetermined in the -unknown coefficients $\beta_i$. A better approach is to use the +unknown coefficients $\theta_i$. A better approach is to use the Singular Value Decomposition (SVD) method discussed below. Or using Lasso and Ridge regression. See below. @@ -1718,16 +1703,16 @@ X[:,1] = x X[:,2] = x**2 # We split the data in test and training data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) -# matrix inversion to find beta -beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train -print(beta) +# matrix inversion to find theta +theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train +print(theta) # and then make the prediction -ytilde = X_train @ beta +ytilde = X_train @ theta print("Training R2") print(R2(y_train,ytilde)) print("Training MSE") print(MSE(y_train,ytilde)) -ypredict = X_test @ beta +ypredict = X_test @ theta print("Test R2") print(R2(y_test,ypredict)) print("Test MSE") @@ -1814,15 +1799,15 @@ X[:,3] = Density**(4.0/3.0) X[:,4] = Density**(5.0/3.0) # We split the data in test and training data X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2) -# matrix inversion to find beta -beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train) +# matrix inversion to find theta +theta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train) # and then make the prediction -ytilde = X_train @ beta +ytilde = X_train @ theta print("Training R2") print(R2(y_train,ytilde)) print("Training MSE") print(MSE(y_train,ytilde)) -ypredict = X_test @ beta +ypredict = X_test @ theta print("Test R2") print(R2(y_test,ypredict)) print("Test MSE") @@ -2284,7 +2269,7 @@ You can use the functionality included in scikit-learn. If you feel for it, you Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits. !bsol -The code here is an example of where we define our own design matrix and fit parameters $\beta$. +The code here is an example of where we define our own design matrix and fit parameters $\theta$. !bc pycod import os import numpy as np @@ -2312,16 +2297,16 @@ X[:,1] = x X[:,2] = x**2 # We split the data in test and training data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) -# matrix inversion to find beta -beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train -print(beta) +# matrix inversion to find theta +theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train +print(theta) # and then make the prediction -ytilde = X_train @ beta +ytilde = X_train @ theta print("Training R2") print(R2(y_train,ytilde)) print("Training MSE") print(MSE(y_train,ytilde)) -ypredict = X_test @ beta +ypredict = X_test @ theta print("Test R2") print(R2(y_test,ypredict)) print("Test MSE") @@ -2549,16 +2534,16 @@ X[:,2] = x*x # We split the data in test and training data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) -# matrix inversion to find beta -OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train -print(OLSbeta) +# matrix inversion to find theta +OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train +print(OLStheta) # and then make the prediction -ytildeOLS = X_train @ OLSbeta +ytildeOLS = X_train @ OLStheta print("Training R2 for OLS") print(R2(y_train,ytildeOLS)) print("Training MSE for OLS") print(MSE(y_train,ytildeOLS)) -ypredictOLS = X_test @ OLSbeta +ypredictOLS = X_test @ OLStheta print("Test R2 for OLS") print(R2(y_test,ypredictOLS)) print("Test MSE OLS") @@ -2575,10 +2560,10 @@ MSERidgePredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 1, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train + OwnRidgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train # and then make the prediction - OwnytildeRidge = X_train @ OwnRidgebeta - OwnypredictRidge = X_test @ OwnRidgebeta + OwnytildeRidge = X_train @ OwnRidgetheta + OwnypredictRidge = X_test @ OwnRidgetheta OwnMSEPredict[i] = MSE(y_test,OwnypredictRidge) OwnMSETrain[i] = MSE(y_train,OwnytildeRidge) # Make the fit using Ridge from Sklearn diff --git a/doc/BookChapters/chapter2.do.txt b/doc/BookChapters/chapter2.do.txt index 45c884f59..bae6f49a6 100644 --- a/doc/BookChapters/chapter2.do.txt +++ b/doc/BookChapters/chapter2.do.txt @@ -7,21 +7,21 @@ What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). -We have shown that in ordinary least squares (OLS) the optimal parameters $\beta$ are given by +We have shown that in ordinary least squares (OLS) the optimal parameters $\theta$ are given by !bt \[ -\hat{\bm{\beta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\hat{\bm{\theta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. \] !et -The _hat_ over $\bm{\beta}$ means we have the optimal parameters after minimization of the cost function. +The _hat_ over $\bm{\theta}$ means we have the optimal parameters after minimization of the cost function. This means that our best model is defined as !bt \[ -\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\tilde{\bm{y}}=\bm{X}\hat{\bm{\theta}} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. \] !et @@ -35,7 +35,7 @@ We now define a matrix We can rewrite !bt \[ -\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{A}\bm{y}. +\tilde{\bm{y}}=\bm{X}\hat{\bm{\theta}} = \bm{A}\bm{y}. \] !et @@ -165,14 +165,14 @@ This is equivalent to saying that the matrix $\bm{X}$ has at least an eigenvalue If our design matrix $\bm{X}$ which enters the linear regression problem !bt \begin{align} -\bm{\beta} & = (\bm{X}^{T} \bm{X})^{-1} \bm{X}^{T} \bm{y}, +\bm{\theta} & = (\bm{X}^{T} \bm{X})^{-1} \bm{X}^{T} \bm{y}, \end{align} !et has linearly dependent column vectors, we will not be able to compute the inverse -of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\beta_i$. +of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\theta_i$. The estimators are only well-defined if $(\bm{X}^{T}\bm{X})$ can be inverted. This is more likely to happen when the matrix $\bm{X}$ is high-dimensional. In this case it is likely to encounter a situation where -the regression parameters $\beta_i$ cannot be estimated. +the regression parameters $\theta_i$ cannot be estimated. A cheap *ad hoc* approach is simply to add a small diagonal component to the matrix to invert, that is we change !bt @@ -652,7 +652,7 @@ function, that is we have !bt \[ -\frac{\partial^2 C(\bm{\beta})}{\partial \bm{\beta}^T\partial \bm{\beta}} =\frac{2}{n}\bm{X}^T\bm{X}. +\frac{\partial^2 C(\bm{\theta})}{\partial \bm{\theta}^T\partial \bm{\theta}} =\frac{2}{n}\bm{X}^T\bm{X}. \] !et This quantity defines what is called the Hessian matrix (the second derivative of the cost function we want to optimize). @@ -924,7 +924,7 @@ def create_X(x, y, n ): y = np.ravel(y) N = len(x) - l = int((n+1)*(n+2)/2) # Number of elements in beta + l = int((n+1)*(n+2)/2) # Number of elements in theta X = np.ones((N,l)) for i in range(1,n+1): @@ -1131,14 +1131,14 @@ Let us remind ourselves about the expression for the standard Mean Squared Error our optimization problem is !bt \[ -{\displaystyle \min_{\bm{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}. +{\displaystyle \min_{\bm{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}. \] !et or we can state it as !bt \[ -{\displaystyle \min_{\bm{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2, +{\displaystyle \min_{\bm{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2, \] !et where we have used the definition of a norm-2 vector, that is @@ -1151,32 +1151,32 @@ where we have used the definition of a norm-2 vector, that is By minimizing the above equation with respect to the parameters -$\bm{\beta}$ we could then obtain an analytical expression for the -parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by +$\bm{\theta}$ we could then obtain an analytical expression for the +parameters $\bm{\theta}$. We can add a regularization parameter $\lambda$ by defining a new cost function to be optimized, that is !bt \[ -{\displaystyle \min_{\bm{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_2^2 +{\displaystyle \min_{\bm{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2+\lambda\vert\vert \bm{\theta}\vert\vert_2^2 \] !et which leads to the Ridge regression minimization problem where we -require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is +require that $\vert\vert \bm{\theta}\vert\vert_2^2\le t$, where $t$ is a finite number larger than zero. By defining !bt \[ -C(\bm{X},\bm{\beta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1, +C(\bm{X},\bm{\theta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2+\lambda\vert\vert \bm{\theta}\vert\vert_1, \] !et we have a new optimization equation !bt \[ -{\displaystyle \min_{\bm{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1 +{\displaystyle \min_{\bm{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2+\lambda\vert\vert \bm{\theta}\vert\vert_1 \] !et which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. @@ -1196,17 +1196,17 @@ Using the matrix-vector expression for Ridge regression and dropping the paramet !bt \[ -C(\bm{X},\bm{\beta})=\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta}, +C(\bm{X},\bm{\theta})=\left\{(\bm{y}-\bm{X}\bm{\theta})^T(\bm{y}-\bm{X}\bm{\theta})\right\}+\lambda\bm{\theta}^T\bm{\theta}, \] !et and -taking the derivatives with respect to $\bm{\beta}$ we obtain then +taking the derivatives with respect to $\bm{\theta}$ we obtain then a slightly modified matrix inversion problem which for finite values of $\lambda$ does not suffer from singularity problems. We obtain the optimal parameters !bt \[ -\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}, +\hat{\bm{\theta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}, \] !et @@ -1214,7 +1214,7 @@ with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that !bt \[ -\sum_{i=0}^{p-1} \beta_i^2 \leq t, +\sum_{i=0}^{p-1} \theta_i^2 \leq t, \] !et @@ -1223,7 +1223,7 @@ with $t$ a finite positive number. When we compare this with the ordinary least squares result we have !bt \[ -\hat{\bm{\beta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}, +\hat{\bm{\theta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}, \] !et which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\bm{X}^T\bm{X}$. @@ -1233,7 +1233,7 @@ We see that Ridge regression is nothing but the standard OLS with a modified diagonal term added to $\bm{X}^T\bm{X}$. The consequences, in particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of $\lambda$, we may -even reduce the variance of the optimal parameters $\bm{\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +even reduce the variance of the optimal parameters $\bm{\theta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. @@ -1241,7 +1241,7 @@ Using our insights about the SVD of the design matrix $\bm{X}$ We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as !bt \[ -\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\beta} =\bm{U}\bm{U}^T\bm{y}. +\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\theta} =\bm{U}\bm{U}^T\bm{y}. \] !et @@ -1250,7 +1250,7 @@ For Ridge regression this becomes !bt \[ -\tilde{\bm{y}}_{\mathrm{Ridge}}=\bm{X}\bm{\beta}_{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{\Sigma}^2\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y}, +\tilde{\bm{y}}_{\mathrm{Ridge}}=\bm{X}\bm{\theta}_{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{\Sigma}^2\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y}, \] !et @@ -1287,7 +1287,7 @@ For the sake of simplicity, let us assume that the design matrix is orthonormal, In this case the standard OLS results in !bt \[ -\bm{\beta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_i\bm{u}_i^T\bm{y}, +\bm{\theta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_i\bm{u}_i^T\bm{y}, \] !et @@ -1295,7 +1295,7 @@ and !bt \[ -\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\beta}^{\mathrm{OLS}}, +\bm{\theta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\theta}^{\mathrm{OLS}}, \] !et @@ -1311,27 +1311,27 @@ Using the matrix-vector expression for Lasso regression and dropping the paramet !bt \[ -C(\bm{X},\bm{\beta})=\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\vert\vert\bm{\beta}\vert\vert_1, +C(\bm{X},\bm{\theta})=\left\{(\bm{y}-\bm{X}\bm{\theta})^T(\bm{y}-\bm{X}\bm{\theta})\right\}+\lambda\vert\vert\bm{\theta}\vert\vert_1, \] !et -Taking the derivative with respect to $\bm{\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty) +Taking the derivative with respect to $\bm{\theta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty) !bt \[ -\frac{d \vert \beta\vert}{d \bm{\beta}}=\mathrm{sgn}(\bm{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right. +\frac{d \vert \theta\vert}{d \bm{\theta}}=\mathrm{sgn}(\bm{\theta})=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right. \] !et we have that the derivative of the cost function is !bt \[ -\frac{\partial C(\bm{X},\bm{\beta})}{\partial \bm{\beta}}=-2\bm{X}^T(\bm{y}-\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=0, +\frac{\partial C(\bm{X},\bm{\theta})}{\partial \bm{\theta}}=-2\bm{X}^T(\bm{y}-\bm{X}\bm{\theta})+\lambda sgn(\bm{\theta})=0, \] !et and reordering we have !bt \[ -\bm{X}^T\bm{X}\bm{\beta}+\lambda sgn(\bm{\beta})=2\bm{X}^T\bm{y}. +\bm{X}^T\bm{X}\bm{\theta}+\lambda sgn(\bm{\theta})=2\bm{X}^T\bm{y}. \] !et This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package "CVXOPT":"https://cvxopt.org/". We will discuss this later. @@ -1343,16 +1343,16 @@ This equation does not lead to a nice analytical equation as in Ridge regression Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the diagonal. In this case we have an equal number of rows and columns $n=p$. -Our model approximation is just $\tilde{\bm{y}}=\bm{\beta}$ and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term $1/n$) +Our model approximation is just $\tilde{\bm{y}}=\bm{\theta}$ and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term $1/n$) !bt \[ -C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2, +C(\bm{\theta})=\sum_{i=0}^{p-1}(y_i-\theta_i)^2, \] !et and minimizing we have that !bt \[ -\hat{\beta}_i^{\mathrm{OLS}} = y_i. +\hat{\theta}_i^{\mathrm{OLS}} = y_i. \] !et @@ -1360,13 +1360,13 @@ and minimizing we have that For Ridge regression our cost function is !bt \[ -C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, +C(\bm{\theta})=\sum_{i=0}^{p-1}(y_i-\theta_i)^2+\lambda\sum_{i=0}^{p-1}\theta_i^2, \] !et and minimizing we have that !bt \[ -\hat{\beta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}. +\hat{\theta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}. \] !et @@ -1374,25 +1374,25 @@ and minimizing we have that For Lasso regression our cost function is !bt \[ -C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, +C(\bm{\theta})=\sum_{i=0}^{p-1}(y_i-\theta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\theta_i\vert=\sum_{i=0}^{p-1}(y_i-\theta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\theta_i^2}, \] !et and minimizing we have that !bt \[ --2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, +-2\sum_{i=0}^{p-1}(y_i-\theta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\theta_i)}{\vert\theta_i\vert}=0, \] !et which leads to !bt \[ -\hat{\bm{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ +\hat{\bm{\theta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ y_i+\frac{\lambda}{2} &\mathrm{if} & y_i< -\frac{\lambda}{2}\\ 0 &\mathrm{if} & \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\. \] !et -Plotting these results ("figure in handwritten notes for week 36":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf") shows clearly that Lasso regression suppresses (sets to zero) values of $\beta_i$ for specific values of $\lambda$. Ridge regression reduces on the other hand the values of $\beta_i$ as function of $\lambda$. +Plotting these results ("figure in handwritten notes for week 36":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf") shows clearly that Lasso regression suppresses (sets to zero) values of $\theta_i$ for specific values of $\lambda$. Ridge regression reduces on the other hand the values of $\theta_i$ as function of $\lambda$. As another example, @@ -1409,21 +1409,21 @@ and our inputs as a $3\times 2$ design matrix \bm{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix}, \] !et -meaning that we have two features and two unknown parameters $\beta_0$ and $\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression. +meaning that we have two features and two unknown parameters $\theta_0$ and $\theta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression. For ordinary least squares (OLS) we know that the optimal solution is !bt \[ -\hat{\bm{\beta}}^{\mathrm{OLS}}=\left( \bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\hat{\bm{\theta}}^{\mathrm{OLS}}=\left( \bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. \] !et Inserting the above values we obtain that !bt \[ -\hat{\bm{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, +\hat{\bm{\theta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, \] !et @@ -1434,19 +1434,19 @@ For Ridge regression we have !bt \[ -\hat{\bm{\beta}}^{\mathrm{Ridge}}=\left( \bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}. +\hat{\bm{\theta}}^{\mathrm{Ridge}}=\left( \bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}. \] !et Inserting the above values we obtain that !bt \[ -\hat{\bm{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, +\hat{\bm{\theta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, \] !et -There is normally a constraint on the value of $\vert\vert \bm{\beta}\vert\vert_2$ via the parameter $\lambda$. -Let us for simplicity assume that $\beta_0^2+\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\beta$ and $\lambda$. +There is normally a constraint on the value of $\vert\vert \bm{\theta}\vert\vert_2$ via the parameter $\lambda$. +Let us for simplicity assume that $\theta_0^2+\theta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\theta$ and $\lambda$. To see this, let us write the cost function for Ridge regression. @@ -1455,90 +1455,90 @@ To see this, let us write the cost function for Ridge regression. We define the MSE without the $1/n$ factor and have then, using that !bt \[ -\bm{X}\bm{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix}, +\bm{X}\bm{\theta}=\begin{bmatrix} 2\theta_0 \\ \theta_1 \\0 \end{bmatrix}, \] !et !bt \[ -C(\bm{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2), +C(\bm{\theta})=(4-2\theta_0)^2+(2-\theta_1)^2+\lambda(\theta_0^2+\theta_1^2), \] !et -and taking the derivative with respect to $\beta_0$ we get +and taking the derivative with respect to $\theta_0$ we get !bt \[ -\beta_0=\frac{8}{4+\lambda}, +\theta_0=\frac{8}{4+\lambda}, \] !et -and for $\beta_1$ we obtain +and for $\theta_1$ we obtain !bt \[ -\beta_1=\frac{2}{1+\lambda}, +\theta_1=\frac{2}{1+\lambda}, \] !et -Using the constraint for $\beta_0^2+\beta_1^2=1$ we can constrain $\lambda$ by solving +Using the constraint for $\theta_0^2+\theta_1^2=1$ we can constrain $\lambda$ by solving !bt \[ \left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1, \] !et -which gives $\lambda=4.571$ and $\beta_0=0.933$ and $\beta_1=0.359$. +which gives $\lambda=4.571$ and $\theta_0=0.933$ and $\theta_1=0.359$. -For Lasso we need now, keeping a constraint on $\vert\beta_0\vert+\vert\beta_1\vert=1$, to take the derivative of the absolute values of $\beta_0$ -and $\beta_1$. This gives us the following derivatives of the cost function +For Lasso we need now, keeping a constraint on $\vert\theta_0\vert+\vert\theta_1\vert=1$, to take the derivative of the absolute values of $\theta_0$ +and $\theta_1$. This gives us the following derivatives of the cost function !bt \[ -C(\bm{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), +C(\bm{\theta})=(4-2\theta_0)^2+(2-\theta_1)^2+\lambda(\vert\theta_0\vert+\vert\theta_1\vert), \] !et !bt \[ -\frac{\partial C(\bm{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0, +\frac{\partial C(\bm{\theta})}{\partial \theta_0}=-4(4-2\theta_0)+\lambda\mathrm{sgn}(\theta_0)=0, \] !et and !bt \[ -\frac{\partial C(\bm{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0. +\frac{\partial C(\bm{\theta})}{\partial \theta_1}=-2(2-\theta_1)+\lambda\mathrm{sgn}(\theta_1)=0. \] !et -We have now four cases to solve besides the trivial cases $\beta_0$ and/or $\beta_1$ are zero, namely -o $\beta_0 > 0$ and $\beta_1 > 0$, -o $\beta_0 > 0$ and $\beta_1 < 0$, -o $\beta_0 < 0$ and $\beta_1 > 0$, -o $\beta_0 < 0$ and $\beta_1 < 0$. +We have now four cases to solve besides the trivial cases $\theta_0$ and/or $\theta_1$ are zero, namely +o $\theta_0 > 0$ and $\theta_1 > 0$, +o $\theta_0 > 0$ and $\theta_1 < 0$, +o $\theta_0 < 0$ and $\theta_1 > 0$, +o $\theta_0 < 0$ and $\theta_1 < 0$. If we consider the first case, we have then !bt \[ --4(4-2\beta_0)+\lambda=0, +-4(4-2\theta_0)+\lambda=0, \] !et and !bt \[ --2(2-\beta_1)+\lambda=0. +-2(2-\theta_1)+\lambda=0. \] !et which yields !bt \[ -\beta_0=\frac{16+\lambda}{8}, +\theta_0=\frac{16+\lambda}{8}, \] !et and !bt \[ -\beta_1=\frac{4+\lambda}{2}. +\theta_1=\frac{4+\lambda}{2}. \] !et -Using the constraint on $\beta_0$ and $\beta_1$ we can then find the optimal value of $\lambda$ for the different cases. We leave this as an exercise to you. +Using the constraint on $\theta_0$ and $\theta_1$ we can then find the optimal value of $\lambda$ for the different cases. We leave this as an exercise to you. Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of $\lambda$, meaning that we need to perform a search in order to find the optimal values. @@ -1566,14 +1566,14 @@ X = np.array( [ [ 2, 0], [0, 1], [0,0]]) y = np.array( [4, 2, 3]) -# matrix inversion to find beta -OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y -print(OLSbeta) +# matrix inversion to find theta +OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y +print(OLStheta) # and then make the prediction -ytildeOLS = X @ OLSbeta +ytildeOLS = X @ OLStheta print("Training MSE for OLS") print(MSE(y,ytildeOLS)) -ypredictOLS = X @ OLSbeta +ypredictOLS = X @ OLStheta # Repeat now for Ridge regression and various values of the regularization parameter I = np.eye(2,2) @@ -1583,10 +1583,10 @@ MSEPredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 4, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y -# print(Ridgebeta) + Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y +# print(Ridgetheta) # and then make the prediction - ypredictRidge = X @ Ridgebeta + ypredictRidge = X @ Ridgetheta MSEPredict[i] = MSE(y,ypredictRidge) # print(MSEPredict[i]) # Now plot the results @@ -1599,10 +1599,10 @@ plt.show() !ec -We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\bm{\beta}$, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of $\tilde{\bm{y}}$ approach zero. +We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\bm{\theta}$, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of $\tilde{\bm{y}}$ approach zero. This happens also for Lasso regression, as seen from the next code -output. The difference is that Lasso shrinks the values of $\beta$ to +output. The difference is that Lasso shrinks the values of $\theta$ to zero at a much earlier stage and the results flatten out. We see that Lasso gives also an excellent fit for small values of $\lambda$ and shows the best performance of the three regression methods. @@ -1628,14 +1628,14 @@ X = np.array( [ [ 2, 0], [0, 1], [0,0]]) y = np.array( [4, 2, 3]) -# matrix inversion to find beta -OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y -print(OLSbeta) +# matrix inversion to find theta +OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y +print(OLStheta) # and then make the prediction -ytildeOLS = X @ OLSbeta +ytildeOLS = X @ OLStheta print("Training MSE for OLS") print(MSE(y,ytildeOLS)) -ypredictOLS = X @ OLSbeta +ypredictOLS = X @ OLStheta # Repeat now for Ridge regression and various values of the regularization parameter I = np.eye(2,2) @@ -1646,10 +1646,10 @@ MSELassoPredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 4, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y - print(Ridgebeta) + Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y + print(Ridgetheta) # and then make the prediction - ypredictRidge = X @ Ridgebeta + ypredictRidge = X @ Ridgetheta MSERidgePredict[i] = MSE(y,ypredictRidge) RegLasso = linear_model.Lasso(lmb) RegLasso.fit(X,y) @@ -1673,7 +1673,7 @@ three regression methods. Depending on the level of noise, we note that for small values of the hyperparameter $\lambda$ all three methods produce the same mean squared error. Again, Lasso shrinks the parameter values to zero much earlier than Ridge regression and the -Lasso results flatten out much earlier since all $\beta_j=0$ (check +Lasso results flatten out much earlier since all $\theta_j=0$ (check this by printing the values). This case is an example of where OLS performs best. Lasso and Ridge reproduce the OLS results for a limited set of $\lambda$ values. @@ -1710,14 +1710,14 @@ X[:,2] = x*x # We split the data in test and training data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) -# matrix inversion to find beta -OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train -print(OLSbeta) +# matrix inversion to find theta +OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train +print(OLStheta) # and then make the prediction -ytildeOLS = X_train @ OLSbeta +ytildeOLS = X_train @ OLStheta print("Training MSE for OLS") print(MSE(y_train,ytildeOLS)) -ypredictOLS = X_test @ OLSbeta +ypredictOLS = X_test @ OLStheta print("Test MSE OLS") print(MSE(y_test,ypredictOLS)) @@ -1732,13 +1732,13 @@ MSELassoTrain = np.zeros(nlambdas) lambdas = np.logspace(-4, 4, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train + Ridgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train # include lasso using Scikit-Learn RegLasso = linear_model.Lasso(lmb) RegLasso.fit(X_train,y_train) # and then make the prediction - ytildeRidge = X_train @ Ridgebeta - ypredictRidge = X_test @ Ridgebeta + ytildeRidge = X_train @ Ridgetheta + ypredictRidge = X_test @ Ridgetheta ytildeLasso = RegLasso.predict(X_train) ypredictLasso = RegLasso.predict(X_test) MSEPredict[i] = MSE(y_test,ypredictRidge) @@ -1777,7 +1777,7 @@ move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters -$\beta$. +$\theta$. The @@ -1801,7 +1801,7 @@ independent, i.e.: The randomness of $\varepsilon_i$ implies that $\mathbf{y}_i$ is also a random variable. In particular, $\mathbf{y}_i$ is normally distributed, because $\varepsilon_i \sim -\mathcal{N}(0, \sigma^2)$ and $\mathbf{X}_{i,\ast} \, \bm{\beta}$ is a +\mathcal{N}(0, \sigma^2)$ and $\mathbf{X}_{i,\ast} \, \bm{\theta}$ is a non-random scalar. To specify the parameters of the distribution of $\mathbf{y}_i$ we need to calculate its first two moments. @@ -1824,7 +1824,7 @@ We approximate this function with our model from the solution of the linear regr function $f$ is approximated by $\bm{\tilde{y}}$ where we want to minimize $(\bm{y}-\bm{\tilde{y}})^2$, our MSE, with !bt \[ -\bm{\tilde{y}} = \bm{X}\bm{\beta}. +\bm{\tilde{y}} = \bm{X}\bm{\theta}. \] !et @@ -1833,8 +1833,8 @@ We can calculate the expectation value of $\bm{y}$ for a given element $i$ !bt \begin{align*} \mathbb{E}(y_i) & = -\mathbb{E}(\mathbf{X}_{i, \ast} \, \bm{\beta}) + \mathbb{E}(\varepsilon_i) -\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\mathbb{E}(\mathbf{X}_{i, \ast} \, \bm{\theta}) + \mathbb{E}(\varepsilon_i) +\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta, \end{align*} !et while @@ -1843,95 +1843,95 @@ its variance is \begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i - \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - [\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, -\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 \\ & -= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 + 2 \varepsilon_i -\mathbf{X}_{i, \ast} \, \bm{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, -\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 + 2 -\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \bm{\beta} + -\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 +\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \bm{\theta})^2 \\ & += \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \bm{\theta})^2 + 2 \varepsilon_i +\mathbf{X}_{i, \ast} \, \bm{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, +\ast} \, \theta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \bm{\theta})^2 + 2 +\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \bm{\theta} + +\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \bm{\theta})^2 \\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, \mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. \end{align*} !et -Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \bm{\beta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with -mean value $\bm{X}\bm{\beta}$ and variance $\sigma^2$ (not be confused with the singular values of the SVD). +Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \bm{\theta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with +mean value $\bm{X}\bm{\theta}$ and variance $\sigma^2$ (not be confused with the singular values of the SVD). -With the OLS expressions for the parameters $\bm{\beta}$ we can evaluate the expectation value +With the OLS expressions for the parameters $\bm{\theta}$ we can evaluate the expectation value !bt \[ -\mathbb{E}(\bm{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\bm{\beta}=\bm{\beta}. +\mathbb{E}(\bm{\theta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\bm{\theta}=\bm{\theta}. \] !et This means that the estimator of the regression parameters is unbiased. We can also calculate the variance -The variance of $\bm{\beta}$ is +The variance of $\bm{\theta}$ is !bt \begin{eqnarray*} -\mbox{Var}(\bm{\beta}) & = & \mathbb{E} \{ [\bm{\beta} - \mathbb{E}(\bm{\beta})] [\bm{\beta} - \mathbb{E}(\bm{\beta})]^{T} \} +\mbox{Var}(\bm{\theta}) & = & \mathbb{E} \{ [\bm{\theta} - \mathbb{E}(\bm{\theta})] [\bm{\theta} - \mathbb{E}(\bm{\theta})]^{T} \} \\ -& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\beta}]^{T} \} +& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\theta}]^{T} \} \\ -% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \bm{\beta} \, \bm{\beta}^{T} +% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \bm{\theta} \, \bm{\theta}^{T} % \\ -% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \bm{\beta} \, \bm{\beta}^{T} +% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \bm{\theta} \, \bm{\theta}^{T} % \\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T} +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\theta} \, \bm{\theta}^{T} \\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \bm{\beta} \, \bm{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T} +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \bm{\theta} \, \bm{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\theta} \, \bm{\theta}^{T} % \\ -% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \bm{\beta} \, \bm{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} +% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \bm{\theta} \, \bm{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} % \\ -% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \bm{\beta} \bm{\beta}^T +% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \bm{\theta} \bm{\theta}^T \\ -& = & \bm{\beta} \, \bm{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T} +& = & \bm{\theta} \, \bm{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\theta} \, \bm{\theta}^{T} \, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, \end{eqnarray*} !et where we have used that $\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = -\mathbf{X} \, \bm{\beta} \, \bm{\beta}^{T} \, \mathbf{X}^{T} + -\sigma^2 \, \mathbf{I}_{nn}$. From $\mbox{Var}(\bm{\beta}) = \sigma^2 +\mathbf{X} \, \bm{\theta} \, \bm{\theta}^{T} \, \mathbf{X}^{T} + +\sigma^2 \, \mathbf{I}_{nn}$. From $\mbox{Var}(\bm{\theta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}$, one obtains an estimate of the variance of the estimate of the $j$-th regression coefficient: -$\bm{\sigma}^2 (\bm{\beta}_j ) = \bm{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} $. This may be used to +$\bm{\sigma}^2 (\bm{\theta}_j ) = \bm{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} $. This may be used to construct a confidence interval for the estimates. In a similar way, we can obtain analytical expressions for say the -expectation values of the parameters $\bm{\beta}$ and their variance +expectation values of the parameters $\bm{\theta}$ and their variance when we employ Ridge regression, allowing us again to define a confidence interval. It is rather straightforward to show that !bt \[ -\mathbb{E} \big[ \bm{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\bm{\beta}^{\mathrm{OLS}}. +\mathbb{E} \big[ \bm{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\bm{\theta}^{\mathrm{OLS}}. \] !et We see clearly that -$\mathbb{E} \big[ \bm{\beta}^{\mathrm{Ridge}} \big] \not= \bm{\beta}^{\mathrm{OLS}}$ for any $\lambda > 0$. We say then that the ridge estimator is biased. +$\mathbb{E} \big[ \bm{\theta}^{\mathrm{Ridge}} \big] \not= \bm{\theta}^{\mathrm{OLS}}$ for any $\lambda > 0$. We say then that the ridge estimator is biased. We can also compute the variance as !bt \[ -\mbox{Var}[\bm{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +\mbox{Var}[\bm{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, \] !et -and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\beta}$ goes to zero. +and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\theta}$ goes to zero. With this, we can compute the difference !bt \[ -\mbox{Var}[\bm{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\bm{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +\mbox{Var}[\bm{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\bm{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. \] !et The difference is non-negative definite since each component of the matrix product is non-negative definite. -This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\bm{\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. +This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\bm{\theta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. @@ -1944,15 +1944,15 @@ distribution with zero mean value and an undetermined variance $\sigma^2$. We found above that the outputs $\bm{y}$ have a mean value given by -$\bm{X}\hat{\bm{\beta}}$ and variance $\sigma^2$. Since the entries to +$\bm{X}\hat{\bm{\theta}}$ and variance $\sigma^2$. Since the entries to the design matrix are not stochastic variables, we can assume that the probability distribution of our targets is also a normal distribution -but now with mean value $\bm{X}\hat{\bm{\beta}}$. This means that a +but now with mean value $\bm{X}\hat{\bm{\theta}}$. This means that a single output $y_i$ is given by the Gaussian distribution !bt \[ -y_i\sim \mathcal{N}(\bm{X}_{i,*}\bm{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}. +y_i\sim \mathcal{N}(\bm{X}_{i,*}\bm{\theta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\theta})^2}{2\sigma^2}\right]}. \] !et @@ -1961,16 +1961,16 @@ We assume now that the various $y_i$ values are stochastically distributed accor We define this distribution as !bt \[ -p(y_i, \bm{X}\vert\bm{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}, +p(y_i, \bm{X}\vert\bm{\theta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\theta})^2}{2\sigma^2}\right]}, \] !et -which reads as finding the likelihood of an event $y_i$ with the input variables $\bm{X}$ given the parameters (to be determined) $\bm{\beta}$. +which reads as finding the likelihood of an event $y_i$ with the input variables $\bm{X}$ given the parameters (to be determined) $\bm{\theta}$. Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event $\bm{y}$ as the product of the single events, that is we have !bt \[ -p(\bm{y},\bm{X}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\bm{X}\vert\bm{\beta}). +p(\bm{y},\bm{X}\vert\bm{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\theta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\bm{X}\vert\bm{\theta}). \] !et @@ -1985,13 +1985,13 @@ In the more general case the various inputs should be replaced by the possible f We can now rewrite the above probability as !bt \[ -p(\bm{D}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}. +p(\bm{D}\vert\bm{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\theta})^2}{2\sigma^2}\right]}. \] !et It is a conditional probability (see below) and reads as the likelihood of a domain of events $\bm{D}$ given a set of parameters -$\bm{\beta}$. +$\bm{\theta}$. In statistics, maximum likelihood estimation (MLE) is a method of @@ -2002,7 +2002,7 @@ data is the most probable. We will assume here that our events are given by the above Gaussian -distribution and we will determine the optimal parameters $\beta$ by +distribution and we will determine the optimal parameters $\theta$ by maximizing the above PDF. However, computing the derivatives of a product function is cumbersome and can easily lead to overflow and/or underflowproblems, with potentials for loss of numerical precision. @@ -2024,27 +2024,27 @@ We could now define a new cost function to minimize, namely the negative logarit !bt \[ -C(\bm{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\bm{X}\vert\bm{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\bm{X}\vert\bm{\beta})}, +C(\bm{\theta}=-\log{\prod_{i=0}^{n-1}p(y_i,\bm{X}\vert\bm{\theta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\bm{X}\vert\bm{\theta})}, \] !et which becomes !bt \[ -C(\bm{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}. +C(\bm{\theta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\bm{y}-\bm{X}\bm{\theta})\vert\vert_2^2}{2\sigma^2}. \] !et -Taking the derivative of the *new* cost function with respect to the parameters $\beta$ we recognize our familiar OLS equation, namely +Taking the derivative of the *new* cost function with respect to the parameters $\theta$ we recognize our familiar OLS equation, namely !bt \[ -\bm{X}^T\left(\bm{y}-\bm{X}\bm{\beta}\right) =0, +\bm{X}^T\left(\bm{y}-\bm{X}\bm{\theta}\right) =0, \] !et -which leads to the well-known OLS equation for the optimal paramters $\beta$ +which leads to the well-known OLS equation for the optimal paramters $\theta$ !bt \[ -\hat{\bm{\beta}}^{\mathrm{OLS}}=\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}! +\hat{\bm{\theta}}^{\mathrm{OLS}}=\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}! \] !et @@ -2203,9 +2203,9 @@ Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomia We will play around with a study of the values for the optimal -parameters $\bm{\beta}$ using OLS, Ridge and Lasso regression. For +parameters $\bm{\theta}$ using OLS, Ridge and Lasso regression. For OLS, you will notice as function of the noise and polynomial degree, -that the parameters $\beta$ will fluctuate from order to order in the +that the parameters $\theta$ will fluctuate from order to order in the polynomial fit and that for larger and larger polynomial degrees of freedom, the parameters will tend to increase in value for OLS. @@ -2242,10 +2242,10 @@ for polydegree in range(1, Maxpolydegree): # We split the data in test and training data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) -# matrix inversion to find beta -OLSbeta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train -print(OLSbeta) -ypredictOLS = X_test @ OLSbeta +# matrix inversion to find theta +OLStheta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train +print(OLStheta) +ypredictOLS = X_test @ OLStheta print("Test MSE OLS") print(MSE(y_test,ypredictOLS)) # Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn @@ -2283,12 +2283,12 @@ plt.show() How can we understand this? -Let us write out the values of the coefficients $\beta_i$ as functions +Let us write out the values of the coefficients $\theta_i$ as functions of the polynomial degree and noise. We will focus only on the Ridge results and some few selected values of the hyperparameter $\lambda$. If we don't include any noise and run this code for different values -of the polynomial degree, we notice that the results for $\beta_i$ do +of the polynomial degree, we notice that the results for $\theta_i$ do not show great changes from one order to the next. This is an indication that for higher polynomial orders, our parameters become less important. @@ -2296,10 +2296,10 @@ less important. If we however add noise, what happens is that the polynomial fit is trying to adjust the fit to traverse in the best possible way all data points. This can lead to large fluctuations in the parameters -$\beta_i$ as functions of polynomial order. It will also be reflected -in a larger value of the variance of each parameter $\beta_i$. What +$\theta_i$ as functions of polynomial order. It will also be reflected +in a larger value of the variance of each parameter $\theta_i$. What Ridge regression (and Lasso as well) are doing then is to try to -quench the fluctuations in the parameters of $\beta_i$ which have a +quench the fluctuations in the parameters of $\theta_i$ which have a large variance (normally for higher orders in the polynomial). !bc pycod @@ -2338,19 +2338,19 @@ for i in range(nlambdas): # and then make the prediction ypredictRidge = RegRidge.predict(X_test) Coeffs = np.array(RegRidge.coef_) - BetaValues = pd.DataFrame(Coeffs) - BetaValues.columns = ['beta'] - display(BetaValues) + ThetaValues = pd.DataFrame(Coeffs) + ThetaValues.columns = ['theta'] + display(ThetaValues) !ec As an exercise, repeat these calculations with ordinary least squares only with and without noise. Calculate thereafter the variance of the -parameters $\beta_j$ as function of polynomial order and of the added +parameters $\theta_j$ as function of polynomial order and of the added noise. Here we recommend to use $\sigma^2=1$ as variance for the added noise (which follows a normal distribution with mean value zero). -Comment your results. If you have a large noise term, do the parameters $\beta_j$ vary more as function +Comment your results. If you have a large noise term, do the parameters $\theta_j$ vary more as function of model complexity? And what about their variance? @@ -2360,7 +2360,7 @@ of model complexity? And what about their variance? We have seen that Ridge regression suppresses those features which have a small singular value. This corresponds to a feature which exhibits -a large variance in the parameters $\beta_j$. +a large variance in the parameters $\theta_j$. Our analysis hitherto has been based on linear algebra. To add to our intuition, we will use Bayes' theorem in order to deepen our understanding of Ridge and Lasso regression. @@ -2373,26 +2373,26 @@ For ordinary least squares we postulated that the maximum likelihood for the dom is given by !bt \[ -p(\bm{D}\vert\bm{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}. +p(\bm{D}\vert\bm{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\theta})^2}{2\sigma^2}\right]}. \] !et -In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\bm{\beta}$ given a domain of events $\bm{D}$? That is, how can we define the posterior probability +In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\bm{\theta}$ given a domain of events $\bm{D}$? That is, how can we define the posterior probability !bt \[ -p(\bm{\beta}\vert\bm{D}). +p(\bm{\theta}\vert\bm{D}). \] !et Bayes' theorem comes to our rescue here since (omitting the normalization constant) !bt \[ -p(\bm{\beta}\vert\bm{D})\propto p(\bm{D}\vert\bm{\beta})p(\bm{\beta}). +p(\bm{\theta}\vert\bm{D})\propto p(\bm{D}\vert\bm{\theta})p(\bm{\theta}). \] !et -We have a model for $p(\bm{D}\vert\bm{\beta})$ but need one for the _prior_ $p(\bm{\beta})$! +We have a model for $p(\bm{D}\vert\bm{\theta})$ but need one for the _prior_ $p(\bm{\theta})$! @@ -2400,41 +2400,41 @@ With the posterior probability defined by a likelihood which we have already modeled and an unknown prior, we are now ready to make additional models for the prior. -We can, based on our discussions of the variance of $\bm{\beta}$ and -the mean value, assume that the prior for the values $\bm{\beta}$ is +We can, based on our discussions of the variance of $\bm{\theta}$ and +the mean value, assume that the prior for the values $\bm{\theta}$ is given by a Gaussian with mean value zero and variance $\tau^2$, that is !bt \[ -p(\bm{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. +p(\bm{\theta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\theta_j^2}{2\tau^2}\right)}. \] !et Our posterior probability becomes then (omitting the normalization factor which is just a constant) !bt \[ -p(\bm{\beta\vert\bm{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. +p(\bm{\theta\vert\bm{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\theta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\theta_j^2}{2\tau^2}\right)}. \] !et -We can now optimize this quantity with respect to $\bm{\beta}$. As we +We can now optimize this quantity with respect to $\bm{\theta}$. As we did for OLS, this is most conveniently done by taking the negative logarithm of the posterior probability. Doing so and leaving out the -terms that do not depend on $\beta$, we have +terms that do not depend on $\theta$, we have !bt \[ -C(\bm{\beta})=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\bm{\beta}\vert\vert_2^2, +C(\bm{\theta})=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\theta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\bm{\theta}\vert\vert_2^2, \] !et and replacing $1/2\tau^2$ with $\lambda$ we have !bt \[ -C(\bm{\beta})=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\bm{\beta}\vert\vert_2^2, +C(\bm{\theta})=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\theta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\bm{\theta}\vert\vert_2^2, \] !et which is our Ridge cost function! Nice, isn't it? @@ -2444,33 +2444,33 @@ To derive the Lasso cost function, we simply replace the Gaussian prior with an !bt \[ -p(\bm{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. +p(\bm{\theta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\theta_j\vert}{\tau}\right)}. \] !et Our posterior probability becomes then (omitting the normalization factor which is just a constant) !bt \[ -p(\bm{\beta}\vert\bm{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. +p(\bm{\theta}\vert\bm{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\bm{X}_{i,*}\bm{\theta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\theta_j\vert}{\tau}\right)}. \] !et Taking the negative logarithm of the posterior probability and leaving out the -constants terms that do not depend on $\beta$, we have +constants terms that do not depend on $\theta$, we have !bt \[ -C(\bm{\beta})=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\bm{\beta}\vert\vert_1, +C(\bm{\theta})=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\theta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\bm{\theta}\vert\vert_1, \] !et and replacing $1/\tau$ with $\lambda$ we have !bt \[ -C(\bm{\beta})=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\bm{\beta}\vert\vert_1, +C(\bm{\theta})=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\theta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\bm{\theta}\vert\vert_1, \] !et which is our Lasso cost function! @@ -2478,7 +2478,7 @@ which is our Lasso cost function! Plotting these prior functions shows us that we can use the parameter $\lambda$ to shrink or increase the role of a given parameter -$\beta_j$. The variance for the Laplace distribution is +$\theta_j$. The variance for the Laplace distribution is $2\tau^2=1/\lambda$ while for the Gaussian distribution it is $\sigma^2=1/(2\lambda)$. Thus, increasing the variance means decreasing $\lambda$ and shrinking the variance means increasing diff --git a/doc/BookChapters/chapter3.do.txt b/doc/BookChapters/chapter3.do.txt index 9813418c7..f14ec7002 100644 --- a/doc/BookChapters/chapter3.do.txt +++ b/doc/BookChapters/chapter3.do.txt @@ -72,7 +72,7 @@ independent, i.e.: The randomness of $\varepsilon_i$ implies that $\mathbf{y}_i$ is also a random variable. In particular, $\mathbf{y}_i$ is normally distributed, because $\varepsilon_i \sim -\mathcal{N}(0, \sigma^2)$ and $\mathbf{X}_{i,\ast} \, \bm{\beta}$ is a +\mathcal{N}(0, \sigma^2)$ and $\mathbf{X}_{i,\ast} \, \bm{\theta}$ is a non-random scalar. To specify the parameters of the distribution of $\mathbf{y}_i$ we need to calculate its first two moments. @@ -94,7 +94,7 @@ We approximate this function with our model from the solution of the linear regr function $f$ is approximated by $\bm{\tilde{y}}$ where we want to minimize $(\bm{y}-\bm{\tilde{y}})^2$, our MSE, with !bt \[ -\bm{\tilde{y}} = \bm{X}\bm{\beta}. +\bm{\tilde{y}} = \bm{X}\bm{\theta}. \] !et @@ -103,8 +103,8 @@ We can calculate the expectation value of $\bm{y}$ for a given element $i$ !bt \begin{align*} \mathbb{E}(y_i) & = -\mathbb{E}(\mathbf{X}_{i, \ast} \, \bm{\beta}) + \mathbb{E}(\varepsilon_i) -\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\mathbb{E}(\mathbf{X}_{i, \ast} \, \bm{\theta}) + \mathbb{E}(\varepsilon_i) +\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta, \end{align*} !et while @@ -113,96 +113,96 @@ its variance is \begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i - \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - [\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, -\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 \\ & -= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 + 2 \varepsilon_i -\mathbf{X}_{i, \ast} \, \bm{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, -\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 + 2 -\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \bm{\beta} + -\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 +\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \bm{\theta})^2 \\ & += \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \bm{\theta})^2 + 2 \varepsilon_i +\mathbf{X}_{i, \ast} \, \bm{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, +\ast} \, \theta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \bm{\theta})^2 + 2 +\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \bm{\theta} + +\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \bm{\theta})^2 \\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, \mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. \end{align*} !et -Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \bm{\beta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with -mean value $\bm{X}\bm{\beta}$ and variance $\sigma^2$ (not be confused with the singular values of the SVD). +Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \bm{\theta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with +mean value $\bm{X}\bm{\theta}$ and variance $\sigma^2$ (not be confused with the singular values of the SVD). -With the OLS expressions for the parameters $\bm{\beta}$ we can evaluate the expectation value +With the OLS expressions for the parameters $\bm{\theta}$ we can evaluate the expectation value !bt \[ -\mathbb{E}(\bm{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\bm{\beta}=\bm{\beta}. +\mathbb{E}(\bm{\theta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\bm{\theta}=\bm{\theta}. \] !et This means that the estimator of the regression parameters is unbiased. We can also calculate the variance -The variance of $\bm{\beta}$ is +The variance of $\bm{\theta}$ is !bt \begin{eqnarray*} -\mbox{Var}(\bm{\beta}) & = & \mathbb{E} \{ [\bm{\beta} - \mathbb{E}(\bm{\beta})] [\bm{\beta} - \mathbb{E}(\bm{\beta})]^{T} \} +\mbox{Var}(\bm{\theta}) & = & \mathbb{E} \{ [\bm{\theta} - \mathbb{E}(\bm{\theta})] [\bm{\theta} - \mathbb{E}(\bm{\theta})]^{T} \} \\ -& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\beta}]^{T} \} +& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\theta}]^{T} \} \\ -% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \bm{\beta} \, \bm{\beta}^{T} +% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \bm{\theta} \, \bm{\theta}^{T} % \\ -% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \bm{\beta} \, \bm{\beta}^{T} +% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \bm{\theta} \, \bm{\theta}^{T} % \\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T} +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\theta} \, \bm{\theta}^{T} \\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \bm{\beta} \, \bm{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T} +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \bm{\theta} \, \bm{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\theta} \, \bm{\theta}^{T} % \\ -% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \bm{\beta} \, \bm{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} +% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \bm{\theta} \, \bm{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} % \\ -% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \bm{\beta} \bm{\beta}^T +% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \bm{\theta} \bm{\theta}^T \\ -& = & \bm{\beta} \, \bm{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T} +& = & \bm{\theta} \, \bm{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\theta} \, \bm{\theta}^{T} \, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, \end{eqnarray*} !et where we have used that $\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = -\mathbf{X} \, \bm{\beta} \, \bm{\beta}^{T} \, \mathbf{X}^{T} + -\sigma^2 \, \mathbf{I}_{nn}$. From $\mbox{Var}(\bm{\beta}) = \sigma^2 +\mathbf{X} \, \bm{\theta} \, \bm{\theta}^{T} \, \mathbf{X}^{T} + +\sigma^2 \, \mathbf{I}_{nn}$. From $\mbox{Var}(\bm{\theta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}$, one obtains an estimate of the variance of the estimate of the $j$-th regression coefficient: -$\bm{\sigma}^2 (\bm{\beta}_j ) = \bm{\sigma}^2 \sqrt{ +$\bm{\sigma}^2 (\bm{\theta}_j ) = \bm{\sigma}^2 \sqrt{ [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} }$. This may be used to construct a confidence interval for the estimates. In a similar way, we can obtain analytical expressions for say the -expectation values of the parameters $\bm{\beta}$ and their variance +expectation values of the parameters $\bm{\theta}$ and their variance when we employ Ridge regression, allowing us again to define a confidence interval. It is rather straightforward to show that !bt \[ -\mathbb{E} \big[ \bm{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\bm{\beta}^{\mathrm{OLS}}. +\mathbb{E} \big[ \bm{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\bm{\theta}^{\mathrm{OLS}}. \] !et We see clearly that -$\mathbb{E} \big[ \bm{\beta}^{\mathrm{Ridge}} \big] \not= \bm{\beta}^{\mathrm{OLS}}$ for any $\lambda > 0$. We say then that the ridge estimator is biased. +$\mathbb{E} \big[ \bm{\theta}^{\mathrm{Ridge}} \big] \not= \bm{\theta}^{\mathrm{OLS}}$ for any $\lambda > 0$. We say then that the ridge estimator is biased. We can also compute the variance as !bt \[ -\mbox{Var}[\bm{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +\mbox{Var}[\bm{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, \] !et -and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\beta}$ goes to zero. +and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\theta}$ goes to zero. With this, we can compute the difference !bt \[ -\mbox{Var}[\bm{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\bm{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +\mbox{Var}[\bm{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\bm{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. \] !et The difference is non-negative definite since each component of the matrix product is non-negative definite. -This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\bm{\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. +This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\bm{\theta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. @@ -244,7 +244,7 @@ variance of $\overline{X}$ (which often is the case), then there is no need for bootstrapping. -The Jackknife works by making many replicas of the estimator $\widehat{\beta}$. +The Jackknife works by making many replicas of the estimator $\widehat{\theta}$. The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\bm{x} = (x_1,x_2,\cdots,X_n)$. Let $\bm{x}_i$ denote the vector !bt @@ -255,8 +255,8 @@ Let $\bm{x}_i$ denote the vector which equals the vector $\bm{x}$ with the exception that observation number $i$ is left out. Using this notation, define -$\widehat{\beta}_i$ to be the estimator -$\widehat{\beta}$ computed using $\vec{X}_i$. +$\widehat{\theta}_i$ to be the estimator +$\widehat{\theta}$ computed using $\vec{X}_i$. @@ -306,11 +306,11 @@ o It is relatively simple to apply the bootstrap to complex data-collection plan -Since $\widehat{\beta} = \widehat{\beta}(\bm{X})$ is a function of random variables, -$\widehat{\beta}$ itself must be a random variable. Thus it has +Since $\widehat{\theta} = \widehat{\theta}(\bm{X})$ is a function of random variables, +$\widehat{\theta}$ itself must be a random variable. Thus it has a pdf, call this function $p(\bm{t})$. The aim of the bootstrap is to estimate $p(\bm{t})$ by the relative frequency of -$\widehat{\beta}$. You can think of this as using a histogram +$\widehat{\theta}$. You can think of this as using a histogram in the place of $p(\bm{t})$. If the relative frequency closely resembles $p(\vec{t})$, then using numerics, it is straight forward to estimate all the interesting parameters of $p(\bm{t})$ using point @@ -318,17 +318,17 @@ estimators. -In the case that $\widehat{\beta}$ has +In the case that $\widehat{\theta}$ has more than one component, and the components are independent, we use the same estimator on each component separately. If the probability density function of $X_i$, $p(x)$, had been known, then it would have been straight forward to do this by: o Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$. -o Then using these numbers, we could compute a replica of $\widehat{\beta}$ called $\widehat{\beta}^*$. +o Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$. By repeated use of (1) and (2), many -estimates of $\widehat{\beta}$ could have been obtained. The -idea is to use the relative frequency of $\widehat{\beta}^*$ +estimates of $\widehat{\theta}$ could have been obtained. The +idea is to use the relative frequency of $\widehat{\theta}^*$ (think of a histogram) as an estimate of $p(\bm{t})$. @@ -352,18 +352,18 @@ The independent bootstrap works like this: o Draw with replacement $n$ numbers for the observed variables $\bm{x} = (x_1,x_2,\cdots,x_n)$. o Define a vector $\bm{x}^*$ containing the values which were drawn from $\bm{x}$. -o Using the vector $\bm{x}^*$ compute $\widehat{\beta}^*$ by evaluating $\widehat \beta$ under the observations $\bm{x}^*$. +o Using the vector $\bm{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\bm{x}^*$. o Repeat this process $k$ times. When you are done, you can draw a histogram of the relative frequency -of $\widehat \beta^*$. This is your estimate of the probability +of $\widehat \theta^*$. This is your estimate of the probability distribution $p(t)$. Using this probability distribution you can estimate any statistics thereof. In principle you never draw the -histogram of the relative frequency of $\widehat{\beta}^*$. Instead +histogram of the relative frequency of $\widehat{\theta}^*$. Instead you use the estimators corresponding to the statistic of interest. For example, if you are interested in estimating the variance of $\widehat -\beta$, apply the estimator $\widehat \sigma^2$ to the values -$\widehat \beta^*$. +\theta$, apply the estimator $\widehat \sigma^2$ to the values +$\widehat \theta^*$. Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called _central limit theorem_. @@ -587,13 +587,13 @@ where $\epsilon$ is normally distributed with mean zero and standard deviation $ In our derivation of the ordinary least squares method we defined then an approximation to the function $f$ in terms of the parameters -$\bm{\beta}$ and the design matrix $\bm{X}$ which embody our model, -that is $\bm{\tilde{y}}=\bm{X}\bm{\beta}$. +$\bm{\theta}$ and the design matrix $\bm{X}$ which embody our model, +that is $\bm{\tilde{y}}=\bm{X}\bm{\theta}$. -Thereafter we found the parameters $\bm{\beta}$ by optimizing the means squared error via the so-called cost function +Thereafter we found the parameters $\bm{\theta}$ by optimizing the means squared error via the so-called cost function !bt \[ -C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. +C(\bm{X},\bm{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. \] !et @@ -970,20 +970,20 @@ cross-validation (LOOCV). * Fit the linear regression model by means of ridge estimation for each $\lambda$ in the grid using the training set, and the corresponding estimate of the error variance $\bm{\sigma}_{-i}^2(\lambda)$, as !bt \begin{align*} -\bm{\beta}_{-i}(\lambda) & = ( \bm{X}_{-i, \ast}^{T} +\bm{\theta}_{-i}(\lambda) & = ( \bm{X}_{-i, \ast}^{T} \bm{X}_{-i, \ast} + \lambda \bm{I}_{pp})^{-1} \bm{X}_{-i, \ast}^{T} \bm{y}_{-i} \end{align*} !et -* Evaluate the prediction performance of these models on the test set by $\log\{L[y_i, \bm{X}_{i, \ast}; \bm{\beta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]\}$. Or, by the prediction error $|y_i - \bm{X}_{i, \ast} \bm{\beta}_{-i}(\lambda)|$, the relative error, the error squared or the R2 score function. +* Evaluate the prediction performance of these models on the test set by $\log\{L[y_i, \bm{X}_{i, \ast}; \bm{\theta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]\}$. Or, by the prediction error $|y_i - \bm{X}_{i, \ast} \bm{\theta}_{-i}(\lambda)|$, the relative error, the error squared or the R2 score function. * Repeat the first three steps such that each sample plays the role of the test set once. * Average the prediction performances of the test sets at each grid point of the penalty bias/parameter. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as !bt \begin{align*} -\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \bm{\beta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]\}. +\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \bm{\theta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]\}. \end{align*} !et @@ -1190,8 +1190,8 @@ Thus, if we cannot assume that the expected outputs/targets are zero when all predictors are zero (the columns in the design matrix), it may be a bad idea to implement a model which penalizes the intercept. Furthermore, in for example Ridge and Lasso regression, the default solutions -from the library _Scikit-Learn_ (when not shrinking $\beta_0$) for the unknown parameters -$\bm{\beta}$, are derived under the assumption that both $\bm{y}$ and +from the library _Scikit-Learn_ (when not shrinking $\theta_0$) for the unknown parameters +$\bm{\theta}$, are derived under the assumption that both $\bm{y}$ and $\bm{X}$ are zero centered, that is we subtract the mean values. @@ -1253,7 +1253,7 @@ simplicity, we will focus on ordinary regression, as done in the above example. The cost/loss function for regression is !bt \[ -C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,. +C(\theta_0, \theta_1, ... , \theta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij}\theta_j\right)^2,. \] !et @@ -1261,42 +1261,42 @@ Recall also that we use the squared value. This expression can lead to an increased penalty for higher differences between predicted and output/target values. -What we have done is to single out the $\beta_0$ term in the +What we have done is to single out the $\theta_0$ term in the definition of the mean squared error (MSE). The design matrix $X$ does in this case not contain any intercept column. When we take the -derivative with respect to $\beta_0$, we want the derivative to obey +derivative with respect to $\theta_0$, we want the derivative to obey !bt \[ -\frac{\partial C}{\partial \beta_j} = 0, +\frac{\partial C}{\partial \theta_j} = 0, \] !et -for all $j$. For $\beta_0$ we have +for all $j$. For $\theta_0$ we have !bt \[ -\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right). +\frac{\partial C}{\partial \theta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij} \theta_j\right). \] !et Multiplying away the constant $2/n$, we obtain !bt \[ -\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j. +\sum_{i=0}^{n-1} \theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \theta_j. \] !et -Let us specialize first to the case where we have only two parameters $\beta_0$ and $\beta_1$. -Our result for $\beta_0$ simplifies then to +Let us specialize first to the case where we have only two parameters $\theta_0$ and $\theta_1$. +Our result for $\theta_0$ simplifies then to !bt \[ -n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1. +n\theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \theta_1. \] !et We obtain then !bt \[ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \theta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. \] !et If we define @@ -1314,20 +1314,20 @@ and the mean value of the outputs as we have !bt \[ -\beta_0 = \mu_y - \beta_1\mu_{\bm{x}_1}. +\theta_0 = \mu_y - \theta_1\mu_{\bm{x}_1}. \] !et -In the general case with more parameters than $\beta_0$ and $\beta_1$, we have +In the general case with more parameters than $\theta_0$ and $\theta_1$, we have !bt \[ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j. +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\theta_j. \] !et We can rewrite the latter equation as !bt \[ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\bm{x}_j}\beta_j, +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\bm{x}_j}\theta_j, \] !et where we have defined @@ -1343,27 +1343,27 @@ the mean value for all elements of the column vector $\bm{x}_j$. Replacing $y_i$ with $y_i - y_i - \overline{\bm{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise) !bt \[ -C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}). +C(\boldsymbol{\theta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta}). \] !et -If we minimize with respect to $\bm{\beta}$ we have then +If we minimize with respect to $\bm{\theta}$ we have then !bt \[ -\hat{\bm{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, +\hat{\bm{\theta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, \] !et where $\boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\bm{y}}$ and $\tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj}$. -For Ridge regression we need to add $\lambda \boldsymbol{\beta}^T\boldsymbol{\beta}$ to the cost function and get then +For Ridge regression we need to add $\lambda \boldsymbol{\theta}^T\boldsymbol{\theta}$ to the cost function and get then !bt \[ -\hat{\bm{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. +\hat{\bm{\theta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. \] !et @@ -1387,15 +1387,15 @@ def MSE(y_data,y_model): return np.sum((y_data-y_model)**2)/n -def fit_beta(X, y): +def fit_theta(X, y): return np.linalg.pinv(X.T @ X) @ X.T @ y -true_beta = [2, 0.5, 3.7] +true_theta = [2, 0.5, 3.7] x = np.linspace(0, 1, 11) y = np.sum( - np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0 + np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0 ) + 0.1 * np.random.normal(size=len(x)) degree = 3 @@ -1405,15 +1405,15 @@ X = np.zeros((len(x), degree)) for p in range(degree): X[:, p] = x ** p -beta = fit_beta(X, y) +theta = fit_theta(X, y) # Intercept is included in the design matrix skl = LinearRegression(fit_intercept=False).fit(X, y) -print(f"True beta: {true_beta}") -print(f"Fitted beta: {beta}") -print(f"Sklearn fitted beta: {skl.coef_}") -ypredictOwn = X @ beta +print(f"True theta: {true_theta}") +print(f"Fitted theta: {theta}") +print(f"Sklearn fitted theta: {skl.coef_}") +ypredictOwn = X @ theta ypredictSKL = skl.predict(X) print(f"MSE with intercept column") print(MSE(y,ypredictOwn)) @@ -1423,7 +1423,7 @@ print(MSE(y,ypredictSKL)) plt.figure() plt.scatter(x, y, label="Data") -plt.plot(x, X @ beta, label="Fit") +plt.plot(x, X @ theta, label="Fit") plt.plot(x, skl.predict(X), label="Sklearn (fit_intercept=False)") @@ -1440,21 +1440,21 @@ skl = LinearRegression(fit_intercept=True).fit(X, y) y_offset = np.average(y, axis=0) X_offset = np.average(X, axis=0) -beta = fit_beta(X - X_offset, y - y_offset) -intercept = np.mean(y_offset - X_offset @ beta) +theta = fit_theta(X - X_offset, y - y_offset) +intercept = np.mean(y_offset - X_offset @ theta) print(f"Manual intercept: {intercept}") -print(f"Fitted beta (wiothout intercept): {beta}") +print(f"Fitted theta (wiothout intercept): {theta}") print(f"Sklearn intercept: {skl.intercept_}") -print(f"Sklearn fitted beta (without intercept): {skl.coef_}") -ypredictOwn = X @ beta +print(f"Sklearn fitted theta (without intercept): {skl.coef_}") +ypredictOwn = X @ theta ypredictSKL = skl.predict(X) print(f"MSE with Manual intercept") print(MSE(y,ypredictOwn+intercept)) print(f"MSE with Sklearn intercept") print(MSE(y,ypredictSKL)) -plt.plot(x, X @ beta + intercept, "--", label="Fit (manual intercept)") +plt.plot(x, X @ theta + intercept, "--", label="Fit (manual intercept)") plt.plot(x, skl.predict(X), "--", label="Sklearn (fit_intercept=True)") plt.grid() plt.legend() @@ -1471,26 +1471,26 @@ they should. However, when we move to for example Ridge regression, the way we treat the intercept may give a larger or smaller MSE, meaning that the MSE can be penalized by the value of the intercept. Not including the intercept in the fit, means that the -regularization term does not include $\beta_0$. For different values +regularization term does not include $\theta_0$. For different values of $\lambda$, this may lead to different MSE values. To remind the reader, the regularization term, with the intercept in Ridge regression, is given by !bt \[ -\lambda \vert\vert \bm{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2, +\lambda \vert\vert \bm{\theta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\theta_j^2, \] !et but when we take out the intercept, this equation becomes !bt \[ -\lambda \vert\vert \bm{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2. +\lambda \vert\vert \bm{\theta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\theta_j^2. \] !et For Lasso regression we have !bt \[ -\lambda \vert\vert \bm{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert. +\lambda \vert\vert \bm{\theta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\theta_j\vert. \] !et @@ -1541,20 +1541,20 @@ MSERidgePredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train + OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train # Note: we include the intercept column and no scaling RegRidge = linear_model.Ridge(lmb,fit_intercept=False) RegRidge.fit(X_train,y_train) # and then make the prediction - ytildeOwnRidge = X_train @ OwnRidgeBeta - ypredictOwnRidge = X_test @ OwnRidgeBeta + ytildeOwnRidge = X_train @ OwnRidgeTheta + ypredictOwnRidge = X_test @ OwnRidgeTheta ytildeRidge = RegRidge.predict(X_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) - print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) - print("Beta values for Scikit-Learn Ridge implementation") + print("Theta values for own Ridge implementation") + print(OwnRidgeTheta) + print("Theta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) print("MSE values for own Ridge implementation") print(MSEOwnRidgePredict[i]) @@ -1626,18 +1626,18 @@ MSERidgePredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) - intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data + OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) + intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data #Add intercept to prediction - ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler + ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler RegRidge = linear_model.Ridge(lmb) RegRidge.fit(X_train,y_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) - print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) #Intercept is given by mean of target variable - print("Beta values for Scikit-Learn Ridge implementation") + print("Theta values for own Ridge implementation") + print(OwnRidgeTheta) #Intercept is given by mean of target variable + print("Theta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) print('Intercept from own implementation:') print(intercept_) @@ -1661,7 +1661,7 @@ plt.show() We see here, when compared to the code which includes explicitely the intercept column, that our MSE value is actually smaller. This is because the regularization term does not include the intercept value -$\beta_0$ in the fitting. This applies to Lasso regularization as +$\theta_0$ in the fitting. This applies to Lasso regularization as well. It means that our optimization is now done only with the centered matrix and/or vector that enter the fitting procedure. Note also that the problem with the intercept occurs mainly in these type @@ -1741,7 +1741,7 @@ with the form utilized in linear regression, that is !bt \begin{align} - \bm{y} = \bm{X}\bm{\beta} + \bm{\epsilon}, + \bm{y} = \bm{X}\bm{\theta} + \bm{\epsilon}, \end{align} !et @@ -1760,16 +1760,16 @@ In the ordinary least squares method we choose the cost function !bt \begin{align} - C(\bm{X}, \bm{\beta})= \frac{1}{n}\left\{(\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y})\right\}. + C(\bm{X}, \bm{\theta})= \frac{1}{n}\left\{(\bm{X}\bm{\theta} - \bm{y})^T(\bm{X}\bm{\theta} - \bm{y})\right\}. \end{align} !et -We then find the extremal point of $C$ by taking the derivative with respect to $\bm{\beta}$ as discussed above. -This yields the expression for $\bm{\beta}$ to be +We then find the extremal point of $C$ by taking the derivative with respect to $\bm{\theta}$ as discussed above. +This yields the expression for $\bm{\theta}$ to be !bt \[ - \bm{\beta} = \frac{\bm{X}^T \bm{y}}{\bm{X}^T \bm{X}}, + \bm{\theta} = \frac{\bm{X}^T \bm{y}}{\bm{X}^T \bm{X}}, \] !et @@ -1794,11 +1794,11 @@ X_test_own = np.concatenate( Doing the inversion directly turns out to be a bad idea since the matrix $\bm{X}^T\bm{X}$ is singular. An alternative approach is to use the _singular value decomposition_. Using the definition of the Moore-Penrose -pseudoinverse we can write the equation for $\bm{\beta}$ as +pseudoinverse we can write the equation for $\bm{\theta}$ as !bt \[ - \bm{\beta} = \bm{X}^{+}\bm{y}, + \bm{\theta} = \bm{X}^{+}\bm{y}, \] !et @@ -1816,7 +1816,7 @@ where $X^{+} = V\Sigma^{+} U^T$. This reduces the equation for $\omega$ to !bt \begin{align} - \bm{\beta} = \bm{V}\bm{\Sigma}^{+} \bm{U}^T \bm{y}. + \bm{\theta} = \bm{V}\bm{\Sigma}^{+} \bm{U}^T \bm{y}. \end{align} !et @@ -1835,13 +1835,13 @@ def ols_svd(x: np.ndarray, y: np.ndarray) -> np.ndarray: !ec !bc pycod -beta = ols_svd(X_train_own,y_train) +theta = ols_svd(X_train_own,y_train) !ec When extracting the $J$-matrix we need to make sure that we remove the intercept, as is done here !bc pycod -J = beta[1:].reshape(L, L) +J = theta[1:].reshape(L, L) !ec A way of looking at the coefficients in $J$ is to plot the matrices as images. @@ -1932,7 +1932,7 @@ elements $-J_{jk}$. This form of writing the energy fits perfectly with the form utilized in linear regression, viz. !bt \begin{align} - \bm{y} = \bm{X}\bm{\beta} + \bm{\epsilon}. + \bm{y} = \bm{X}\bm{\theta} + \bm{\epsilon}. \end{align} !et We organize the data as we did above @@ -1980,12 +1980,12 @@ The results agree perfectly with our previous discussion where we used our own c Having explored the ordinary least squares we move on to ridge regression. In ridge regression we include a _regularizer_. This involves a new cost function which leads to a new estimate for the -weights $\bm{\beta}$. This results in a penalized regression problem. The +weights $\bm{\theta}$. This results in a penalized regression problem. The cost function is given by !bt \begin{align} - C(\bm{X}, \bm{\beta}; \lambda) = (\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y}) + \lambda \bm{\beta}^T\bm{\beta}. + C(\bm{X}, \bm{\theta}; \lambda) = (\bm{X}\bm{\theta} - \bm{y})^T(\bm{X}\bm{\theta} - \bm{y}) + \lambda \bm{\theta}^T\bm{\theta}. \end{align} !et !bc pycod @@ -2007,7 +2007,7 @@ In the _Least Absolute Shrinkage and Selection Operator_ (LASSO)-method we get a !bt \begin{align} - C(\bm{X}, \bm{\beta}; \lambda) = (\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y}) + \lambda \sqrt{\bm{\beta}^T\bm{\beta}}. + C(\bm{X}, \bm{\theta}; \lambda) = (\bm{X}\bm{\theta} - \bm{y})^T(\bm{X}\bm{\theta} - \bm{y}) + \lambda \sqrt{\bm{\theta}^T\bm{\theta}}. \end{align} !et @@ -2237,7 +2237,7 @@ distribution $N(0,1)$. value decomposition from e.g., _numpy_ ) or use your code from homeworks 1 and 2 and perform a standard least square regression analysis using polynomials in $x$ and $y$ up to fifth order. Find the -"confidence intervals":"https://en.wikipedia.org/wiki/Confidence_interval" of the parameters (estimators) $\beta$ by computing their +"confidence intervals":"https://en.wikipedia.org/wiki/Confidence_interval" of the parameters (estimators) $\theta$ by computing their variances, evaluate the Mean Squared error (MSE) !bt @@ -2319,15 +2319,15 @@ deviation $\sigma^2$. In our derivation of the ordinary least squares method we defined then an approximation to the function $f$ in terms of the parameters -$\bm{\beta}$ and the design matrix $\bm{X}$ which embody our model, -that is $\bm{\tilde{y}}=\bm{X}\bm{\beta}$. +$\bm{\theta}$ and the design matrix $\bm{X}$ which embody our model, +that is $\bm{\tilde{y}}=\bm{X}\bm{\theta}$. -The parameters $\bm{\beta}$ are in turn found by optimizing the means +The parameters $\bm{\theta}$ are in turn found by optimizing the means squared error via the so-called cost function !bt \[ -C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. +C(\bm{X},\bm{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. \] !et Here the expected value $\mathbb{E}$ is the sample value. diff --git a/doc/BookChapters/chapter4.do.txt b/doc/BookChapters/chapter4.do.txt index 0fa9fc831..115767f12 100644 --- a/doc/BookChapters/chapter4.do.txt +++ b/doc/BookChapters/chapter4.do.txt @@ -12,7 +12,7 @@ independent variables $x_i$. Linear regression resulted in analytical expressions for standard ordinary Least Squares or Ridge regression (in terms of matrices to invert) for several quantities, ranging from the variance and thereby the confidence intervals of the -optimal parameters $\hat{\beta}$ to the mean squared error. If we can invert +optimal parameters $\hat{\theta}$ to the mean squared error. If we can invert the product of the design matrices, linear regression gives then a simple recipe for fitting our data. @@ -37,7 +37,7 @@ failure etc. Logistic regression will also serve as our stepping stone towards neural network algorithms and supervised deep learning. For logistic learning, the minimization of the cost function leads to a non-linear -equation in the parameters $\hat{\beta}$. The optimization of the +equation in the parameters $\hat{\theta}$. The optimization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the @@ -86,11 +86,11 @@ We would then have our weighted linear combination, namely !bt \begin{equation} -\bm{y} = \bm{X}^T\bm{\beta} + \bm{\epsilon}, +\bm{y} = \bm{X}^T\bm{\theta} + \bm{\epsilon}, \end{equation} !et where $\bm{y}$ is a vector representing the possible outcomes, $\bm{X}$ is our -$n\times p$ design matrix and $\bm{\beta}$ represents our estimators/predictors. +$n\times p$ design matrix and $\bm{\theta}$ represents our estimators/predictors. The main problem with our function is that it takes values on the @@ -186,7 +186,7 @@ We are now trying to find a function $f(y\vert x)$, that is a function which giv In standard linear regression with a linear dependence on $x$, we would write this in terms of our model !bt \[ -f(y_i\vert x_i)=\beta_0+\beta_1 x_i. +f(y_i\vert x_i)=\theta_0+\theta_1 x_i. \] !et @@ -291,19 +291,19 @@ plt.show() -We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\beta$ in our fitting of the Sigmoid function, that is we define probabilities +We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\theta$ in our fitting of the Sigmoid function, that is we define probabilities !bt \begin{align*} -p(y_i=1|x_i,\bm{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ -p(y_i=0|x_i,\bm{\beta}) &= 1 - p(y_i=1|x_i,\bm{\beta}), +p(y_i=1|x_i,\bm{\theta}) &= \frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\bm{\theta}) &= 1 - p(y_i=1|x_i,\bm{\theta}), \end{align*} !et -where $\bm{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$. +where $\bm{\theta}$ are the weights we wish to extract from data, in our case $\theta_0$ and $\theta_1$. Note that we used !bt \[ -p(y_i=0\vert x_i, \bm{\beta}) = 1-p(y_i=1\vert x_i, \bm{\beta}). +p(y_i=0\vert x_i, \bm{\theta}) = 1-p(y_i=1\vert x_i, \bm{\theta}). \] !et @@ -316,13 +316,13 @@ the probability of seeing the observed data. We can then approximate the likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is !bt \begin{align*} -P(\mathcal{D}|\bm{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\bm{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\bm{\beta}))\right]^{1-y_i}\nonumber \\ +P(\mathcal{D}|\bm{\theta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\bm{\theta})\right]^{y_i}\left[1-p(y_i=1|x_i,\bm{\theta}))\right]^{1-y_i}\nonumber \\ \end{align*} !et from which we obtain the log-likelihood and our _cost/loss_ function !bt \[ -\mathcal{C}(\bm{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\bm{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\bm{\beta}))\right]\right). +\mathcal{C}(\bm{\theta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\bm{\theta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\bm{\theta}))\right]\right). \] !et @@ -330,58 +330,58 @@ from which we obtain the log-likelihood and our _cost/loss_ function Reordering the logarithms, we can rewrite the _cost/loss_ function as !bt \[ -\mathcal{C}(\bm{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\mathcal{C}(\bm{\theta}) = \sum_{i=1}^n \left(y_i(\theta_0+\theta_1x_i) -\log{(1+\exp{(\theta_0+\theta_1x_i)})}\right). \] !et -The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\beta$. +The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\theta$. Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that !bt \[ -\mathcal{C}(\bm{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\mathcal{C}(\bm{\theta})=-\sum_{i=1}^n \left(y_i(\theta_0+\theta_1x_i) -\log{(1+\exp{(\theta_0+\theta_1x_i)})}\right). \] !et This equation is known in statistics as the _cross entropy_. Finally, we note that just as in linear regression, in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression. -The cross entropy is a convex function of the weights $\bm{\beta}$ and, +The cross entropy is a convex function of the weights $\bm{\theta}$ and, therefore, any local minimizer is a global minimizer. Minimizing this -cost function with respect to the two parameters $\beta_0$ and $\beta_1$ we obtain +cost function with respect to the two parameters $\theta_0$ and $\theta_1$ we obtain !bt \[ -\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), +\frac{\partial \mathcal{C}(\bm{\theta})}{\partial \theta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}}\right), \] !et and !bt \[ -\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). +\frac{\partial \mathcal{C}(\bm{\theta})}{\partial \theta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}}\right). \] !et Let us now define a vector $\bm{y}$ with $n$ elements $y_i$, an $n\times p$ matrix $\bm{X}$ which contains the $x_i$ values and a -vector $\bm{p}$ of fitted probabilities $p(y_i\vert x_i,\bm{\beta})$. We can rewrite in a more compact form the first +vector $\bm{p}$ of fitted probabilities $p(y_i\vert x_i,\bm{\theta})$. We can rewrite in a more compact form the first derivative of cost function as !bt \[ -\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}} = -\bm{X}^T\left(\bm{y}-\bm{p}\right). +\frac{\partial \mathcal{C}(\bm{\theta})}{\partial \bm{\theta}} = -\bm{X}^T\left(\bm{y}-\bm{p}\right). \] !et If we in addition define a diagonal matrix $\bm{W}$ with elements -$p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compact expression of the second derivative as +$p(y_i\vert x_i,\bm{\theta})(1-p(y_i\vert x_i,\bm{\theta})$, we can obtain a compact expression of the second derivative as !bt \[ -\frac{\partial^2 \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T} = \bm{X}^T\bm{W}\bm{X}. +\frac{\partial^2 \mathcal{C}(\bm{\theta})}{\partial \bm{\theta}\partial \bm{\theta}^T} = \bm{X}^T\bm{W}\bm{X}. \] !et @@ -389,13 +389,13 @@ $p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compa Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors !bt \[ -\log{ \frac{p(\bm{\beta}\bm{x})}{1-p(\bm{\beta}\bm{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. +\log{ \frac{p(\bm{\theta}\bm{x})}{1-p(\bm{\theta}\bm{x})}} = \theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p. \] !et -Here we defined $\bm{x}=[1,x_1,x_2,\dots,x_p]$ and $\bm{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to +Here we defined $\bm{x}=[1,x_1,x_2,\dots,x_p]$ and $\bm{\theta}=[\theta_0, \theta_1, \dots, \theta_p]$ leading to !bt \[ -p(\bm{\beta}\bm{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. +p(\bm{\theta}\bm{x})=\frac{ \exp{(\theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p)}}{1+\exp{(\theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p)}}. \] !et @@ -406,19 +406,19 @@ of simplicity assume we have only two predictors. We have then following model !bt \[ -\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1, +\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \theta_{10}+\theta_{11}x_1, \] !et and !bt \[ -\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1, +\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \theta_{20}+\theta_{21}x_1, \] !et and so on till the class $C=K-1$ class !bt \[ -\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1, +\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \theta_{(K-1)0}+\theta_{(K-1)1}x_1, \] !et @@ -437,18 +437,18 @@ Bayes classifiers, and artificial neural networks. Specifically, in multinomial logistic regression and linear discriminant analysis, the input to the function is the result of $K$ distinct linear functions, and the predicted probability for the $k$-th class given a sample -vector $\bm{x}$ and a weighting vector $\bm{\beta}$ is (with two +vector $\bm{x}$ and a weighting vector $\bm{\theta}$ is (with two predictors): !bt \[ -p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}. +p(C=k\vert \mathbf {x} )=\frac{\exp{(\theta_{k0}+\theta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\theta_{l0}+\theta_{l1}x_1)}}. \] !et It is easy to extend to more predictors. The final class is !bt \[ -p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}, +p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\theta_{l0}+\theta_{l1}x_1)}}, \] !et diff --git a/doc/LectureNotes/.ipynb_checkpoints/E2-checkpoint.ipynb b/doc/LectureNotes/.ipynb_checkpoints/E2-checkpoint.ipynb new file mode 100644 index 000000000..f7405e0df --- /dev/null +++ b/doc/LectureNotes/.ipynb_checkpoints/E2-checkpoint.ipynb @@ -0,0 +1,517 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "b4005770", + "metadata": {}, + "source": [ + "# Exercises week 35\n", + "\n", + "## Deriving and Implementing Ordinary Least Squares" + ] + }, + { + "cell_type": "markdown", + "id": "2ca1b589", + "metadata": {}, + "source": [ + "This week you will be deriving the analytical expressions for linear regression, building up the model from scratch. This will include taking several derivatives of products of vectors and matrices. Such derivatives are central to the optimization of many machine learning models. Although we will often use automatic differentiation in actual calculations, to be able to have analytical expressions is extremely helpful in case we have simpler derivatives as well as when we analyze various properties (like second derivatives) of the chosen cost functions.\n", + "\n", + "Vectors are always written as boldfaced lower case letters and matrices as upper case boldfaced letters. You will find useful the notes from week 35 on derivatives of vectors and matrices. See also the textbook of Faisal at al, chapter 5 and in particular sections 5.3-5.5 at " + ] + }, + { + "cell_type": "markdown", + "id": "41e92bf9", + "metadata": {}, + "source": [ + "### Learning goals\n", + "\n", + "After completing these exercises, you will know how to\n", + "- Take the derivatives of simple products between vectors and matrices\n", + "- Implement OLS using the analytical expressions\n", + "- Create a feature matrix from a set of data\n", + "- Create a feature matrix for a polynomial model\n", + "- Evaluate the MSE score of various model on training and test data, and comparing their performance\n", + "\n", + "### Deliverables\n", + "\n", + "Complete the following exercises while working in a jupyter notebook. Then, in canvas, include\n", + "- The jupyter notebook with the exercises completed\n", + "- An exported PDF of the notebook (https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_export-your-jupyter-notebook)" + ] + }, + { + "cell_type": "markdown", + "id": "f7a9209d", + "metadata": {}, + "source": [ + "## How to take derivatives of Matrix-Vector expressions" + ] + }, + { + "cell_type": "markdown", + "id": "45f3712e", + "metadata": {}, + "source": [ + "In these exercises it is always useful to write out with summation indices the various quantities. Take also a look at the weekly slides from week 35 and the various examples included there.\n", + "\n", + "As an example, consider the function\n", + "\n", + "$$\n", + "f(\\boldsymbol{x}) =\\boldsymbol{A}\\boldsymbol{x},\n", + "$$\n", + "\n", + "which reads for a specific component $f_i$ (we define the matrix $\\boldsymbol{A}$ to have dimension $n\\times n$ and the vector $\\boldsymbol{x}$ to have length $n$)\n", + "\n", + "$$\n", + "f_i =\\sum_{j=0}^{n-1}a_{ij}x_j,\n", + "$$\n", + "\n", + "which leads to\n", + "\n", + "$$\n", + "\\frac{\\partial f_i}{\\partial x_j}= a_{ij},\n", + "$$\n", + "\n", + "and written out in terms of the vector $\\boldsymbol{x}$ we have\n", + "\n", + "$$\n", + "\\frac{\\partial f(\\boldsymbol{x})}{\\partial \\boldsymbol{x}}= \\boldsymbol{A}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5fa8a4e6", + "metadata": {}, + "source": [ + "## Exercise 1 - Finding the derivative of Matrix-Vector expressions" + ] + }, + { + "cell_type": "markdown", + "id": "df7a2270", + "metadata": {}, + "source": [ + "**a)** Consider the expression\n", + "\n", + "$$\n", + "\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}},\n", + "$$\n", + "\n", + "Where $\\boldsymbol{a}$ and $\\boldsymbol{x}$ are column-vectors with length $n$.\n", + "\n", + "What is the *shape* of the expression we are taking the derivative of?\n", + "\n", + "What is the *shape* of the thing we are taking the derivative with respect to?\n", + "\n", + "What is the *shape* of the result of the expression?" + ] + }, + { + "cell_type": "markdown", + "id": "c0396734", + "metadata": {}, + "source": [ + "**b)** Show that\n", + "\n", + "$$\n", + "\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "dc39d541", + "metadata": {}, + "source": [ + "**c)** Show that\n", + "\n", + "$$\n", + "\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{A}\\boldsymbol{a})}{\\partial \\boldsymbol{a}} = \\boldsymbol{a}^T(\\boldsymbol{A}+\\boldsymbol{A}^T),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "498d13ec", + "metadata": {}, + "source": [ + "## Exercise 2 - Deriving the expression for OLS" + ] + }, + { + "cell_type": "markdown", + "id": "f3f771de", + "metadata": {}, + "source": [ + "The ordinary least squares method finds the parameters $\\boldsymbol{\\beta}$ which minimizes the squared error between our model $\\boldsymbol{X\\beta}$ and the true values $\\boldsymbol{y}$.\n", + "\n", + "To find the parameters $\\boldsymbol{\\beta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\\boldsymbol{\\beta}$, and set it equal to 0." + ] + }, + { + "cell_type": "markdown", + "id": "49690237", + "metadata": {}, + "source": [ + "**a)** Very briefly explain why the approach above finds the parameters $\\boldsymbol{\\beta}$ which minimizes this error." + ] + }, + { + "cell_type": "markdown", + "id": "b7cccc9d", + "metadata": {}, + "source": [ + "We typically write the squared error as\n", + "\n", + "$$\n", + "\\vert\\vert\\boldsymbol{y} - \\boldsymbol{X\\beta}\\vert\\vert^2\n", + "$$\n", + "\n", + "which we can rewrite in matrix-vector form as\n", + "\n", + "$$\n", + "\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "8fbecf74", + "metadata": {}, + "source": [ + "**b)** If $\\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\\boldsymbol{\\beta}$? (**Hint:** Don't compute any derivatives, but solve $\\boldsymbol{X\\beta}=\\boldsymbol{y}$ for $\\boldsymbol{\\beta}$)" + ] + }, + { + "cell_type": "markdown", + "id": "f37af8f0", + "metadata": {}, + "source": [ + "**c)** Show that\n", + "\n", + "$$\n", + "\\frac{\\partial \\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)^T\\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)}{\\partial \\boldsymbol{s}} = -2\\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)^T\\boldsymbol{A},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "869fca4d", + "metadata": {}, + "source": [ + "**d)** Using the expression from **c)**, but substituting back in $\\boldsymbol{\\beta}$, $\\boldsymbol{y}$ and $\\boldsymbol{X}$, find the expression for the optimal parameters $\\boldsymbol{\\beta}$ in the case that $\\boldsymbol{X}$ is not invertible, but $\\boldsymbol{X^T X}$ is, which is most often the case.\n", + "\n", + "$$\n", + "\\boldsymbol{\\hat{\\beta}_{OLS}} = ...\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "57ca3d74", + "metadata": {}, + "source": [ + "## Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression" + ] + }, + { + "cell_type": "markdown", + "id": "5dc179f7", + "metadata": {}, + "source": [ + "With the expression for $\\boldsymbol{\\hat{\\beta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\\boldsymbol{X}$.\n", + "\n", + "In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, with the montly income and the number of children as columns.\n", + "\n", + "We typically also include an intercept in our models. The intercept is a value that is added to our prediction regardless of the value of the other features. The intercept tries to account for constant effects in our data that are not dependant on anything else. In our current example, the intercept could account for living expenses which are typical regardless of income or childcare expenses.\n", + "\n", + "We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter $\\beta_0$ from the OLS method into the optimal intercept value (which will be $\\beta_0$). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e5ff2a69", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a3cf2792", + "metadata": {}, + "outputs": [], + "source": [ + "n = 20\n", + "income = np.array([116., 161., 167., 118., 172., 163., 179., 173., 162., 116., 101., 176., 178., 172., 143., 135., 160., 101., 149., 125.])\n", + "children = np.array([5, 3, 0, 4, 5, 3, 0, 4, 4, 3, 3, 5, 1, 0, 2, 3, 2, 1, 5, 4])\n", + "spending = np.array([152., 141., 102., 136., 161., 129., 99., 159., 160., 107., 98., 164., 121., 93., 112., 127., 117., 69., 156., 131.])\n" + ] + }, + { + "cell_type": "markdown", + "id": "5da61481", + "metadata": {}, + "source": [ + "**a)** Create a feature matrix $\\boldsymbol{X}$ for the features income and children, including an intercept column of ones at the start." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5ad87a65", + "metadata": {}, + "outputs": [], + "source": [ + "X = np.zeros((n, 3))\n", + "#X[:, 0] = ...\n", + "#X[:, 1] = ...\n", + "#X[:, 2] = ..." + ] + }, + { + "cell_type": "markdown", + "id": "e0ddfac2", + "metadata": {}, + "source": [ + "**b)** Use the expression from **3d)** to find the optimal parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ for predicting spending based on these features. Create a function for this operation, as you are going to need to use it a lot." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8f3f68aa", + "metadata": {}, + "outputs": [], + "source": [ + "def OLS_parameters(X, y):\n", + " return ...\n", + "\n", + "#beta = OLS_parameters(X, y)" + ] + }, + { + "cell_type": "markdown", + "id": "0cb6da80", + "metadata": {}, + "source": [ + "## Exercise 4 - Fitting a polynomial" + ] + }, + { + "cell_type": "markdown", + "id": "71015064", + "metadata": {}, + "source": [ + "In this course, we typically do linear regression using polynomials, though in real world applications it is also very common to make linear models based on measured features like you did in the previous exercise.\n", + "\n", + "When fitting a polynomial with linear regression, we make each polynomial degree($x, x^2, x^3, ..., x^p$) its own feature." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d7476c84", + "metadata": {}, + "outputs": [], + "source": [ + "n = 100\n", + "x = np.linspace(-3, 3, n)\n", + "y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1)" + ] + }, + { + "cell_type": "markdown", + "id": "8321451b", + "metadata": {}, + "source": [ + "**a)** Create a feature matrix $\\boldsymbol{X}$ for the features $x, x^2, x^3, x^4, x^5$, including an intercept column of ones at the start. Make this into a function, as you will do this a lot over the next weeks." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "91496e40", + "metadata": {}, + "outputs": [], + "source": [ + "def polynomial_features(x, p):\n", + " n = len(x)\n", + " X = np.zeros((n, p + 1))\n", + " #X[:, 0] = ...\n", + " #X[:, 1] = ...\n", + " #X[:, 2] = ...\n", + " # could this be a loop?\n", + "\n", + "#X = polynomial_features(x, 5)" + ] + }, + { + "cell_type": "markdown", + "id": "b84b1e31", + "metadata": {}, + "source": [ + "**b)** Use the expression from **3d)** to find the optimal parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ for predicting $\\boldsymbol{y}$ based on these features. If you have done everything right so far, this code will not need changing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "034f502c", + "metadata": {}, + "outputs": [], + "source": [ + "#beta = OLS_parameters(X, y)" + ] + }, + { + "cell_type": "markdown", + "id": "d703f788", + "metadata": {}, + "source": [ + "**c)** Like in exercise 4 last week, split your feature matrix and target data into a training split and test split." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "29171358", + "metadata": {}, + "outputs": [], + "source": [ + "from sklearn.model_selection import train_test_split\n", + "\n", + "#X_train, X_test, y_train, y_test = ..." + ] + }, + { + "cell_type": "markdown", + "id": "a0e3509f", + "metadata": {}, + "source": [ + "**d)** Train your model on the training data(find the parameters which best fit) and compute the MSE on both the training and test data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1e346f4c", + "metadata": {}, + "outputs": [], + "source": [ + "..." + ] + }, + { + "cell_type": "markdown", + "id": "7e431889", + "metadata": {}, + "source": [ + "**e)** Do the same for each polynomial degree from 2 to 10, and plot the MSE on both the training and test data as a function of polynomial degree. The aim is to reproduce Figure 2.11 of [Hastie et al](https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf). Feel free to read the discussions leading to figure 2.11 of Hastie et al. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ceb57457", + "metadata": {}, + "outputs": [], + "source": [ + "..." + ] + }, + { + "cell_type": "markdown", + "id": "5e5b5954", + "metadata": {}, + "source": [ + "**f)** Interpret the graph. Why do the lines move as they do? What does it tell us about model performance and generalizability?" + ] + }, + { + "cell_type": "markdown", + "id": "ad2acfb9", + "metadata": {}, + "source": [] + }, + { + "cell_type": "markdown", + "id": "5994f0c5", + "metadata": {}, + "source": [ + "## Exercise 5 - Comparing your code with sklearn" + ] + }, + { + "cell_type": "markdown", + "id": "8f595b7a", + "metadata": {}, + "source": [ + "When implementing different algorithms for the first time, it can be helpful to double check your results with established implementations before you go on to add more complexity." + ] + }, + { + "cell_type": "markdown", + "id": "8ab310c1", + "metadata": {}, + "source": [ + "**a)** Make sure your `polynomial_features` function creates the same feature matrix as sklearns PolynomialFeatures.\n", + "\n", + "(https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "85b964d1", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "id": "73c32c52", + "metadata": {}, + "source": [ + "**b)** Make sure your `OLS_parameters` function computes the same parameters as sklearns LinearRegression with fit_intercept set to False, since the intercept is included in the feature matrix. Use `your_model_object.coef_` to extract the computed parameters.\n", + "\n", + "(https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "35b04126", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/doc/LectureNotes/E2.ipynb b/doc/LectureNotes/E2.ipynb index f7405e0df..8ce6a6c3d 100644 --- a/doc/LectureNotes/E2.ipynb +++ b/doc/LectureNotes/E2.ipynb @@ -146,9 +146,9 @@ "id": "f3f771de", "metadata": {}, "source": [ - "The ordinary least squares method finds the parameters $\\boldsymbol{\\beta}$ which minimizes the squared error between our model $\\boldsymbol{X\\beta}$ and the true values $\\boldsymbol{y}$.\n", + "The ordinary least squares method finds the parameters $\\boldsymbol{\\theta}$ which minimizes the squared error between our model $\\boldsymbol{X\\theta}$ and the true values $\\boldsymbol{y}$.\n", "\n", - "To find the parameters $\\boldsymbol{\\beta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\\boldsymbol{\\beta}$, and set it equal to 0." + "To find the parameters $\\boldsymbol{\\theta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\\boldsymbol{\\theta}$, and set it equal to 0." ] }, { @@ -156,7 +156,7 @@ "id": "49690237", "metadata": {}, "source": [ - "**a)** Very briefly explain why the approach above finds the parameters $\\boldsymbol{\\beta}$ which minimizes this error." + "**a)** Very briefly explain why the approach above finds the parameters $\\boldsymbol{\\theta}$ which minimizes this error." ] }, { @@ -167,13 +167,13 @@ "We typically write the squared error as\n", "\n", "$$\n", - "\\vert\\vert\\boldsymbol{y} - \\boldsymbol{X\\beta}\\vert\\vert^2\n", + "\\vert\\vert\\boldsymbol{y} - \\boldsymbol{X\\theta}\\vert\\vert^2\n", "$$\n", "\n", "which we can rewrite in matrix-vector form as\n", "\n", "$$\n", - "\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\n", + "\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\n", "$$" ] }, @@ -182,7 +182,7 @@ "id": "8fbecf74", "metadata": {}, "source": [ - "**b)** If $\\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\\boldsymbol{\\beta}$? (**Hint:** Don't compute any derivatives, but solve $\\boldsymbol{X\\beta}=\\boldsymbol{y}$ for $\\boldsymbol{\\beta}$)" + "**b)** If $\\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\\boldsymbol{\\theta}$? (**Hint:** Don't compute any derivatives, but solve $\\boldsymbol{X\\theta}=\\boldsymbol{y}$ for $\\boldsymbol{\\theta}$)" ] }, { @@ -202,10 +202,10 @@ "id": "869fca4d", "metadata": {}, "source": [ - "**d)** Using the expression from **c)**, but substituting back in $\\boldsymbol{\\beta}$, $\\boldsymbol{y}$ and $\\boldsymbol{X}$, find the expression for the optimal parameters $\\boldsymbol{\\beta}$ in the case that $\\boldsymbol{X}$ is not invertible, but $\\boldsymbol{X^T X}$ is, which is most often the case.\n", + "**d)** Using the expression from **c)**, but substituting back in $\\boldsymbol{\\theta}$, $\\boldsymbol{y}$ and $\\boldsymbol{X}$, find the expression for the optimal parameters $\\boldsymbol{\\theta}$ in the case that $\\boldsymbol{X}$ is not invertible, but $\\boldsymbol{X^T X}$ is, which is most often the case.\n", "\n", "$$\n", - "\\boldsymbol{\\hat{\\beta}_{OLS}} = ...\n", + "\\boldsymbol{\\hat{\\theta}_{OLS}} = ...\n", "$$" ] }, @@ -222,18 +222,18 @@ "id": "5dc179f7", "metadata": {}, "source": [ - "With the expression for $\\boldsymbol{\\hat{\\beta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\\boldsymbol{X}$.\n", + "With the expression for $\\boldsymbol{\\hat{\\theta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\\boldsymbol{X}$.\n", "\n", "In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, with the montly income and the number of children as columns.\n", "\n", "We typically also include an intercept in our models. The intercept is a value that is added to our prediction regardless of the value of the other features. The intercept tries to account for constant effects in our data that are not dependant on anything else. In our current example, the intercept could account for living expenses which are typical regardless of income or childcare expenses.\n", "\n", - "We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter $\\beta_0$ from the OLS method into the optimal intercept value (which will be $\\beta_0$). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix." + "We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter $\\theta_0$ from the OLS method into the optimal intercept value (which will be $\\theta_0$). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "id": "e5ff2a69", "metadata": {}, "outputs": [], @@ -243,7 +243,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "id": "a3cf2792", "metadata": {}, "outputs": [], @@ -264,7 +264,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, "id": "5ad87a65", "metadata": {}, "outputs": [], @@ -285,7 +285,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, "id": "8f3f68aa", "metadata": {}, "outputs": [], @@ -316,7 +316,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "id": "d7476c84", "metadata": {}, "outputs": [], @@ -336,7 +336,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, "id": "91496e40", "metadata": {}, "outputs": [], @@ -362,7 +362,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 7, "id": "034f502c", "metadata": {}, "outputs": [], @@ -380,7 +380,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, "id": "29171358", "metadata": {}, "outputs": [], @@ -400,10 +400,21 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 9, "id": "1e346f4c", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "Ellipsis" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "..." ] @@ -418,10 +429,21 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 10, "id": "ceb57457", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "Ellipsis" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "..." ] @@ -495,7 +517,7 @@ ], "metadata": { "kernelspec": { - "display_name": ".venv", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -509,7 +531,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.13.0" + "version": "3.9.15" } }, "nbformat": 4, diff --git a/doc/LectureNotes/_build/.doctrees/E2.doctree b/doc/LectureNotes/_build/.doctrees/E2.doctree index 531a6f79d..5d4b13945 100644 Binary files a/doc/LectureNotes/_build/.doctrees/E2.doctree and b/doc/LectureNotes/_build/.doctrees/E2.doctree differ diff --git a/doc/LectureNotes/_build/.doctrees/chapter1.doctree b/doc/LectureNotes/_build/.doctrees/chapter1.doctree index 58bc87dbe..1fb9c6c33 100644 Binary files a/doc/LectureNotes/_build/.doctrees/chapter1.doctree and b/doc/LectureNotes/_build/.doctrees/chapter1.doctree differ diff --git a/doc/LectureNotes/_build/.doctrees/chapter2.doctree b/doc/LectureNotes/_build/.doctrees/chapter2.doctree index d7703ba96..10c950a54 100644 Binary files a/doc/LectureNotes/_build/.doctrees/chapter2.doctree and b/doc/LectureNotes/_build/.doctrees/chapter2.doctree differ diff --git a/doc/LectureNotes/_build/.doctrees/chapter3.doctree b/doc/LectureNotes/_build/.doctrees/chapter3.doctree index 0ec044610..640fcb0b0 100644 Binary files a/doc/LectureNotes/_build/.doctrees/chapter3.doctree and b/doc/LectureNotes/_build/.doctrees/chapter3.doctree differ diff --git a/doc/LectureNotes/_build/.doctrees/chapter4.doctree b/doc/LectureNotes/_build/.doctrees/chapter4.doctree index 88ef0d88e..70b0099ee 100644 Binary files a/doc/LectureNotes/_build/.doctrees/chapter4.doctree and b/doc/LectureNotes/_build/.doctrees/chapter4.doctree differ diff --git a/doc/LectureNotes/_build/.doctrees/environment.pickle b/doc/LectureNotes/_build/.doctrees/environment.pickle index ac58569bf..803277043 100644 Binary files a/doc/LectureNotes/_build/.doctrees/environment.pickle and b/doc/LectureNotes/_build/.doctrees/environment.pickle differ diff --git a/doc/LectureNotes/_build/html/E1.html b/doc/LectureNotes/_build/html/E1.html index b86e89e6a..7a0e6da1e 100644 --- a/doc/LectureNotes/_build/html/E1.html +++ b/doc/LectureNotes/_build/html/E1.html @@ -32,7 +32,7 @@ - + diff --git a/doc/LectureNotes/_build/html/E2.html b/doc/LectureNotes/_build/html/E2.html index 2779ea6ca..02f3149a3 100644 --- a/doc/LectureNotes/_build/html/E2.html +++ b/doc/LectureNotes/_build/html/E2.html @@ -469,37 +469,37 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,

Exercise 2 - Deriving the expression for OLS#

-

The ordinary least squares method finds the parameters \(\boldsymbol{\beta}\) which minimizes the squared error between our model \(\boldsymbol{X\beta}\) and the true values \(\boldsymbol{y}\).

-

To find the parameters \(\boldsymbol{\beta}\) which minimizes this error, we take the derivative of the squared error expression with respect to \(\boldsymbol{\beta}\), and set it equal to 0.

-

a) Very briefly explain why the approach above finds the parameters \(\boldsymbol{\beta}\) which minimizes this error.

+

The ordinary least squares method finds the parameters \(\boldsymbol{\theta}\) which minimizes the squared error between our model \(\boldsymbol{X\theta}\) and the true values \(\boldsymbol{y}\).

+

To find the parameters \(\boldsymbol{\theta}\) which minimizes this error, we take the derivative of the squared error expression with respect to \(\boldsymbol{\theta}\), and set it equal to 0.

+

a) Very briefly explain why the approach above finds the parameters \(\boldsymbol{\theta}\) which minimizes this error.

We typically write the squared error as

\[ -\vert\vert\boldsymbol{y} - \boldsymbol{X\beta}\vert\vert^2 +\vert\vert\boldsymbol{y} - \boldsymbol{X\theta}\vert\vert^2 \]

which we can rewrite in matrix-vector form as

\[ -\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) +\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right) \]
-

b) If \(\boldsymbol{X}\) is invertible, what is the expression for the optimal parameters \(\boldsymbol{\beta}\)? (Hint: Don’t compute any derivatives, but solve \(\boldsymbol{X\beta}=\boldsymbol{y}\) for \(\boldsymbol{\beta}\))

+

b) If \(\boldsymbol{X}\) is invertible, what is the expression for the optimal parameters \(\boldsymbol{\theta}\)? (Hint: Don’t compute any derivatives, but solve \(\boldsymbol{X\theta}=\boldsymbol{y}\) for \(\boldsymbol{\theta}\))

c) Show that

\[ \frac{\partial \left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)^T\left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)}{\partial \boldsymbol{s}} = -2\left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)^T\boldsymbol{A}, \]
-

d) Using the expression from c), but substituting back in \(\boldsymbol{\beta}\), \(\boldsymbol{y}\) and \(\boldsymbol{X}\), find the expression for the optimal parameters \(\boldsymbol{\beta}\) in the case that \(\boldsymbol{X}\) is not invertible, but \(\boldsymbol{X^T X}\) is, which is most often the case.

+

d) Using the expression from c), but substituting back in \(\boldsymbol{\theta}\), \(\boldsymbol{y}\) and \(\boldsymbol{X}\), find the expression for the optimal parameters \(\boldsymbol{\theta}\) in the case that \(\boldsymbol{X}\) is not invertible, but \(\boldsymbol{X^T X}\) is, which is most often the case.

\[ -\boldsymbol{\hat{\beta}_{OLS}} = ... +\boldsymbol{\hat{\theta}_{OLS}} = ... \]

Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression#

-

With the expression for \(\boldsymbol{\hat{\beta}_{OLS}}\), you now have what you need to implement OLS regression with your input data and target data \(\boldsymbol{y}\). But before you can do that, you need to set up you input data as a feature matrix \(\boldsymbol{X}\).

+

With the expression for \(\boldsymbol{\hat{\theta}_{OLS}}\), you now have what you need to implement OLS regression with your input data and target data \(\boldsymbol{y}\). But before you can do that, you need to set up you input data as a feature matrix \(\boldsymbol{X}\).

In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, with the montly income and the number of children as columns.

We typically also include an intercept in our models. The intercept is a value that is added to our prediction regardless of the value of the other features. The intercept tries to account for constant effects in our data that are not dependant on anything else. In our current example, the intercept could account for living expenses which are typical regardless of income or childcare expenses.

-

We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter \(\beta_0\) from the OLS method into the optimal intercept value (which will be \(\beta_0\)). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix.

+

We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter \(\theta_0\) from the OLS method into the optimal intercept value (which will be \(\theta_0\)). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix.

import numpy as np
@@ -594,6 +594,11 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
+
+
Ellipsis
+
+
+

e) Do the same for each polynomial degree from 2 to 10, and plot the MSE on both the training and test data as a function of polynomial degree. The aim is to reproduce Figure 2.11 of Hastie et al. Feel free to read the discussions leading to figure 2.11 of Hastie et al.

@@ -602,6 +607,11 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
+
+
Ellipsis
+
+
+

f) Interpret the graph. Why do the lines move as they do? What does it tell us about model performance and generalizability?

diff --git a/doc/LectureNotes/_build/html/_sources/E2.ipynb b/doc/LectureNotes/_build/html/_sources/E2.ipynb index f7405e0df..8ce6a6c3d 100644 --- a/doc/LectureNotes/_build/html/_sources/E2.ipynb +++ b/doc/LectureNotes/_build/html/_sources/E2.ipynb @@ -146,9 +146,9 @@ "id": "f3f771de", "metadata": {}, "source": [ - "The ordinary least squares method finds the parameters $\\boldsymbol{\\beta}$ which minimizes the squared error between our model $\\boldsymbol{X\\beta}$ and the true values $\\boldsymbol{y}$.\n", + "The ordinary least squares method finds the parameters $\\boldsymbol{\\theta}$ which minimizes the squared error between our model $\\boldsymbol{X\\theta}$ and the true values $\\boldsymbol{y}$.\n", "\n", - "To find the parameters $\\boldsymbol{\\beta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\\boldsymbol{\\beta}$, and set it equal to 0." + "To find the parameters $\\boldsymbol{\\theta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\\boldsymbol{\\theta}$, and set it equal to 0." ] }, { @@ -156,7 +156,7 @@ "id": "49690237", "metadata": {}, "source": [ - "**a)** Very briefly explain why the approach above finds the parameters $\\boldsymbol{\\beta}$ which minimizes this error." + "**a)** Very briefly explain why the approach above finds the parameters $\\boldsymbol{\\theta}$ which minimizes this error." ] }, { @@ -167,13 +167,13 @@ "We typically write the squared error as\n", "\n", "$$\n", - "\\vert\\vert\\boldsymbol{y} - \\boldsymbol{X\\beta}\\vert\\vert^2\n", + "\\vert\\vert\\boldsymbol{y} - \\boldsymbol{X\\theta}\\vert\\vert^2\n", "$$\n", "\n", "which we can rewrite in matrix-vector form as\n", "\n", "$$\n", - "\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\n", + "\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\n", "$$" ] }, @@ -182,7 +182,7 @@ "id": "8fbecf74", "metadata": {}, "source": [ - "**b)** If $\\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\\boldsymbol{\\beta}$? (**Hint:** Don't compute any derivatives, but solve $\\boldsymbol{X\\beta}=\\boldsymbol{y}$ for $\\boldsymbol{\\beta}$)" + "**b)** If $\\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\\boldsymbol{\\theta}$? (**Hint:** Don't compute any derivatives, but solve $\\boldsymbol{X\\theta}=\\boldsymbol{y}$ for $\\boldsymbol{\\theta}$)" ] }, { @@ -202,10 +202,10 @@ "id": "869fca4d", "metadata": {}, "source": [ - "**d)** Using the expression from **c)**, but substituting back in $\\boldsymbol{\\beta}$, $\\boldsymbol{y}$ and $\\boldsymbol{X}$, find the expression for the optimal parameters $\\boldsymbol{\\beta}$ in the case that $\\boldsymbol{X}$ is not invertible, but $\\boldsymbol{X^T X}$ is, which is most often the case.\n", + "**d)** Using the expression from **c)**, but substituting back in $\\boldsymbol{\\theta}$, $\\boldsymbol{y}$ and $\\boldsymbol{X}$, find the expression for the optimal parameters $\\boldsymbol{\\theta}$ in the case that $\\boldsymbol{X}$ is not invertible, but $\\boldsymbol{X^T X}$ is, which is most often the case.\n", "\n", "$$\n", - "\\boldsymbol{\\hat{\\beta}_{OLS}} = ...\n", + "\\boldsymbol{\\hat{\\theta}_{OLS}} = ...\n", "$$" ] }, @@ -222,18 +222,18 @@ "id": "5dc179f7", "metadata": {}, "source": [ - "With the expression for $\\boldsymbol{\\hat{\\beta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\\boldsymbol{X}$.\n", + "With the expression for $\\boldsymbol{\\hat{\\theta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\\boldsymbol{X}$.\n", "\n", "In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, with the montly income and the number of children as columns.\n", "\n", "We typically also include an intercept in our models. The intercept is a value that is added to our prediction regardless of the value of the other features. The intercept tries to account for constant effects in our data that are not dependant on anything else. In our current example, the intercept could account for living expenses which are typical regardless of income or childcare expenses.\n", "\n", - "We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter $\\beta_0$ from the OLS method into the optimal intercept value (which will be $\\beta_0$). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix." + "We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter $\\theta_0$ from the OLS method into the optimal intercept value (which will be $\\theta_0$). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "id": "e5ff2a69", "metadata": {}, "outputs": [], @@ -243,7 +243,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "id": "a3cf2792", "metadata": {}, "outputs": [], @@ -264,7 +264,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, "id": "5ad87a65", "metadata": {}, "outputs": [], @@ -285,7 +285,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, "id": "8f3f68aa", "metadata": {}, "outputs": [], @@ -316,7 +316,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "id": "d7476c84", "metadata": {}, "outputs": [], @@ -336,7 +336,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, "id": "91496e40", "metadata": {}, "outputs": [], @@ -362,7 +362,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 7, "id": "034f502c", "metadata": {}, "outputs": [], @@ -380,7 +380,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, "id": "29171358", "metadata": {}, "outputs": [], @@ -400,10 +400,21 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 9, "id": "1e346f4c", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "Ellipsis" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "..." ] @@ -418,10 +429,21 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 10, "id": "ceb57457", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "Ellipsis" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "..." ] @@ -495,7 +517,7 @@ ], "metadata": { "kernelspec": { - "display_name": ".venv", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -509,7 +531,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.13.0" + "version": "3.9.15" } }, "nbformat": 4, diff --git a/doc/LectureNotes/_build/html/_sources/chapter1.ipynb b/doc/LectureNotes/_build/html/_sources/chapter1.ipynb index 4d16c5ab1..231068020 100644 --- a/doc/LectureNotes/_build/html/_sources/chapter1.ipynb +++ b/doc/LectureNotes/_build/html/_sources/chapter1.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "d66e8057", + "id": "a453b968", "metadata": { "editable": true }, @@ -13,7 +13,7 @@ }, { "cell_type": "markdown", - "id": "3f6464d0", + "id": "499b2ddb", "metadata": { "editable": true }, @@ -23,7 +23,7 @@ }, { "cell_type": "markdown", - "id": "fab3a8f9", + "id": "c84cce7e", "metadata": { "editable": true }, @@ -65,7 +65,7 @@ }, { "cell_type": "markdown", - "id": "66513db3", + "id": "8419208e", "metadata": { "editable": true }, @@ -167,7 +167,7 @@ }, { "cell_type": "markdown", - "id": "0d1f19f9", + "id": "ceb7a805", "metadata": { "editable": true }, @@ -202,7 +202,7 @@ }, { "cell_type": "markdown", - "id": "0479c66e", + "id": "d6c1062f", "metadata": { "editable": true }, @@ -253,7 +253,7 @@ }, { "cell_type": "markdown", - "id": "609d8174", + "id": "50c1b706", "metadata": { "editable": true }, @@ -286,7 +286,7 @@ }, { "cell_type": "markdown", - "id": "7e580e18", + "id": "12e3ac84", "metadata": { "editable": true }, @@ -298,7 +298,7 @@ }, { "cell_type": "markdown", - "id": "bb8f332d", + "id": "10b4c333", "metadata": { "editable": true }, @@ -335,7 +335,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "9e479f11", + "id": "3e216ef0", "metadata": { "collapsed": false, "editable": true @@ -368,7 +368,7 @@ }, { "cell_type": "markdown", - "id": "8c7b818b", + "id": "af765a12", "metadata": { "editable": true }, @@ -385,7 +385,7 @@ }, { "cell_type": "markdown", - "id": "201888d1", + "id": "2f6fd730", "metadata": { "editable": true }, @@ -397,7 +397,7 @@ }, { "cell_type": "markdown", - "id": "e6a16258", + "id": "1770d85d", "metadata": { "editable": true }, @@ -418,7 +418,7 @@ }, { "cell_type": "markdown", - "id": "c3aab7c4", + "id": "684b72a0", "metadata": { "editable": true }, @@ -431,7 +431,7 @@ }, { "cell_type": "markdown", - "id": "f2957240", + "id": "e9fa8fd5", "metadata": { "editable": true }, @@ -462,7 +462,7 @@ }, { "cell_type": "markdown", - "id": "2a15d7ec", + "id": "15a4642f", "metadata": { "editable": true }, @@ -474,7 +474,7 @@ }, { "cell_type": "markdown", - "id": "12af21ef", + "id": "589ec9cb", "metadata": { "editable": true }, @@ -492,7 +492,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "16f86276", + "id": "cf21ae2b", "metadata": { "collapsed": false, "editable": true @@ -520,7 +520,7 @@ }, { "cell_type": "markdown", - "id": "87da55cc", + "id": "41c55cbb", "metadata": { "editable": true }, @@ -542,7 +542,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "221ebafe", + "id": "d62c9412", "metadata": { "collapsed": false, "editable": true @@ -580,7 +580,7 @@ }, { "cell_type": "markdown", - "id": "aa05a791", + "id": "ae0c6c2a", "metadata": { "editable": true }, @@ -591,7 +591,7 @@ }, { "cell_type": "markdown", - "id": "bac50889", + "id": "64cf13de", "metadata": { "editable": true }, @@ -604,7 +604,7 @@ }, { "cell_type": "markdown", - "id": "825fa9b8", + "id": "18439dae", "metadata": { "editable": true }, @@ -625,7 +625,7 @@ }, { "cell_type": "markdown", - "id": "a3fa2a46", + "id": "3e9fb291", "metadata": { "editable": true }, @@ -637,7 +637,7 @@ }, { "cell_type": "markdown", - "id": "7627c75a", + "id": "46d8744e", "metadata": { "editable": true }, @@ -647,7 +647,7 @@ }, { "cell_type": "markdown", - "id": "52686744", + "id": "38a29b65", "metadata": { "editable": true }, @@ -659,7 +659,7 @@ }, { "cell_type": "markdown", - "id": "f6cb9efa", + "id": "438e73b8", "metadata": { "editable": true }, @@ -671,7 +671,7 @@ }, { "cell_type": "markdown", - "id": "4822580d", + "id": "1403cc6a", "metadata": { "editable": true }, @@ -683,7 +683,7 @@ }, { "cell_type": "markdown", - "id": "fb61578b", + "id": "9fbd1c1b", "metadata": { "editable": true }, @@ -694,7 +694,7 @@ }, { "cell_type": "markdown", - "id": "6cb0d520", + "id": "d25da1d3", "metadata": { "editable": true }, @@ -706,7 +706,7 @@ }, { "cell_type": "markdown", - "id": "87835e2a", + "id": "c43c7452", "metadata": { "editable": true }, @@ -728,7 +728,7 @@ }, { "cell_type": "markdown", - "id": "69fc895e", + "id": "ad0d5004", "metadata": { "editable": true }, @@ -740,7 +740,7 @@ }, { "cell_type": "markdown", - "id": "debfac94", + "id": "eb9e2efb", "metadata": { "editable": true }, @@ -755,7 +755,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "96164cb3", + "id": "963f4f84", "metadata": { "collapsed": false, "editable": true @@ -796,7 +796,7 @@ }, { "cell_type": "markdown", - "id": "7cac7e62", + "id": "d530f60a", "metadata": { "editable": true }, @@ -811,7 +811,7 @@ }, { "cell_type": "markdown", - "id": "8b9b62fe", + "id": "f8951307", "metadata": { "editable": true }, @@ -823,7 +823,7 @@ }, { "cell_type": "markdown", - "id": "81afb7da", + "id": "913bb6a6", "metadata": { "editable": true }, @@ -833,7 +833,7 @@ }, { "cell_type": "markdown", - "id": "266b9b6d", + "id": "25cb82ce", "metadata": { "editable": true }, @@ -845,7 +845,7 @@ }, { "cell_type": "markdown", - "id": "26bfbb29", + "id": "d04f788d", "metadata": { "editable": true }, @@ -855,7 +855,7 @@ }, { "cell_type": "markdown", - "id": "7e4ee0e4", + "id": "866bf69f", "metadata": { "editable": true }, @@ -867,7 +867,7 @@ }, { "cell_type": "markdown", - "id": "40627d20", + "id": "a0d3421f", "metadata": { "editable": true }, @@ -877,7 +877,7 @@ }, { "cell_type": "markdown", - "id": "b4fbd650", + "id": "680eefd0", "metadata": { "editable": true }, @@ -889,7 +889,7 @@ }, { "cell_type": "markdown", - "id": "40eecb7a", + "id": "de97e875", "metadata": { "editable": true }, @@ -905,7 +905,7 @@ }, { "cell_type": "markdown", - "id": "97f2b230", + "id": "5d7dbc49", "metadata": { "editable": true }, @@ -917,7 +917,7 @@ }, { "cell_type": "markdown", - "id": "34eed0f8", + "id": "5a9a6d4d", "metadata": { "editable": true }, @@ -928,7 +928,7 @@ }, { "cell_type": "markdown", - "id": "37ca2a5b", + "id": "fef247ce", "metadata": { "editable": true }, @@ -940,7 +940,7 @@ }, { "cell_type": "markdown", - "id": "36071972", + "id": "3ed61266", "metadata": { "editable": true }, @@ -954,7 +954,7 @@ }, { "cell_type": "markdown", - "id": "32a6cbaf", + "id": "9882fb4e", "metadata": { "editable": true }, @@ -966,7 +966,7 @@ }, { "cell_type": "markdown", - "id": "68d9be7d", + "id": "992b3ae7", "metadata": { "editable": true }, @@ -991,7 +991,7 @@ }, { "cell_type": "markdown", - "id": "7ec5f81c", + "id": "6d772376", "metadata": { "editable": true }, @@ -1008,7 +1008,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "fb39ace0", + "id": "f861cd3f", "metadata": { "collapsed": false, "editable": true @@ -1052,40 +1052,7 @@ }, { "cell_type": "markdown", - "id": "fc71e4e1", - "metadata": { - "editable": true - }, - "source": [ - "Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various **matplotlib** commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "30e59ca4", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], - "source": [ - "from pylab import plt, mpl\n", - "plt.style.use('seaborn')\n", - "mpl.rcParams['font.family'] = 'serif'\n", - "\n", - "def MakePlot(x,y, styles, labels, axlabels):\n", - " plt.figure(figsize=(10,6))\n", - " for i in range(len(x)):\n", - " plt.plot(x[i], y[i], styles[i], label = labels[i])\n", - " plt.xlabel(axlabels[0])\n", - " plt.ylabel(axlabels[1])\n", - " plt.legend(loc=0)" - ] - }, - { - "cell_type": "markdown", - "id": "e58a020b", + "id": "da27a328", "metadata": { "editable": true }, @@ -1101,8 +1068,8 @@ }, { "cell_type": "code", - "execution_count": 7, - "id": "de0f27ba", + "execution_count": 6, + "id": "b4ef8b5a", "metadata": { "collapsed": false, "editable": true @@ -1123,7 +1090,7 @@ }, { "cell_type": "markdown", - "id": "5da52984", + "id": "1352f7d2", "metadata": { "editable": true }, @@ -1136,8 +1103,8 @@ }, { "cell_type": "code", - "execution_count": 8, - "id": "ba67d882", + "execution_count": 7, + "id": "c5b3333d", "metadata": { "collapsed": false, "editable": true @@ -1166,7 +1133,7 @@ }, { "cell_type": "markdown", - "id": "b75681d2", + "id": "6652db3a", "metadata": { "editable": true }, @@ -1185,8 +1152,8 @@ }, { "cell_type": "code", - "execution_count": 9, - "id": "d9760898", + "execution_count": 8, + "id": "715c03ae", "metadata": { "collapsed": false, "editable": true @@ -1203,7 +1170,7 @@ }, { "cell_type": "markdown", - "id": "5199cc2b", + "id": "eaff6065", "metadata": { "editable": true }, @@ -1214,8 +1181,8 @@ }, { "cell_type": "code", - "execution_count": 10, - "id": "d48c42f2", + "execution_count": 9, + "id": "4053e205", "metadata": { "collapsed": false, "editable": true @@ -1233,7 +1200,7 @@ }, { "cell_type": "markdown", - "id": "91ca751e", + "id": "587289d6", "metadata": { "editable": true }, @@ -1248,8 +1215,8 @@ }, { "cell_type": "code", - "execution_count": 11, - "id": "ea82f2ad", + "execution_count": 10, + "id": "52f1c9dc", "metadata": { "collapsed": false, "editable": true @@ -1262,7 +1229,7 @@ }, { "cell_type": "markdown", - "id": "0fb84108", + "id": "25241a4e", "metadata": { "editable": true }, @@ -1273,8 +1240,8 @@ }, { "cell_type": "code", - "execution_count": 12, - "id": "3b24153e", + "execution_count": 11, + "id": "06020cef", "metadata": { "collapsed": false, "editable": true @@ -1304,7 +1271,7 @@ }, { "cell_type": "markdown", - "id": "d9acc526", + "id": "8d07d616", "metadata": { "editable": true }, @@ -1314,8 +1281,8 @@ }, { "cell_type": "code", - "execution_count": 13, - "id": "942b4d36", + "execution_count": 12, + "id": "1c4a2fd6", "metadata": { "collapsed": false, "editable": true @@ -1356,7 +1323,7 @@ }, { "cell_type": "markdown", - "id": "8fe6a29e", + "id": "b9728189", "metadata": { "editable": true }, @@ -1377,8 +1344,8 @@ }, { "cell_type": "code", - "execution_count": 14, - "id": "5dc4b33c", + "execution_count": 13, + "id": "8b5b6d13", "metadata": { "collapsed": false, "editable": true @@ -1418,7 +1385,7 @@ }, { "cell_type": "markdown", - "id": "8dd1a395", + "id": "ec654f7a", "metadata": { "editable": true }, @@ -1427,12 +1394,12 @@ "\n", "[Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureAug27.mp4?vrtx=view-as-webpage).\n", "\n", - "Fitting a continuous function with linear parameterization in terms of the parameters $\\boldsymbol{\\beta}$.\n", + "Fitting a continuous function with linear parameterization in terms of the parameters $\\boldsymbol{\\theta}$.\n", "* Method of choice for fitting a continuous function!\n", "\n", "* Gives an excellent introduction to central Machine Learning features with **understandable pedagogical** links to other methods like **Neural Networks**, **Support Vector Machines** etc\n", "\n", - "* Analytical expression for the fitting parameters $\\boldsymbol{\\beta}$\n", + "* Analytical expression for the fitting parameters $\\boldsymbol{\\theta}$\n", "\n", "* Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more\n", "\n", @@ -1450,14 +1417,14 @@ "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n", "\n", "Regression modeling deals with the description of the sampling distribution of a given random variable $y$ and how it varies as function of another variable or a set of such variables $\\boldsymbol{x} =[x_0, x_1,\\dots, x_{n-1}]^T$. \n", - "The first variable is called the **dependent**, the **outcome** or the **response** variable while the set of variables $\\boldsymbol{x}$ is called the independent variable, or the predictor variable or the explanatory variable. \n", + "The first variable is called the **dependent**, the **outcome** or the **response** or just the output variable while the set of variables $\\boldsymbol{x}$ is called the independent variable, or the predictor variable or the explanatory variable, or just the input variable. We will hereafter call $\\boldsymbol{y}$ and $\\boldsymbol{x}x$ for the output and input variables, respectively. \n", "\n", - "A regression model aims at finding a likelihood function $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$, that is the conditional distribution for $\\boldsymbol{y}$ with a given $\\boldsymbol{x}$. The estimation of $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$ is made using a data set with \n", + "A regression model aims at finding a likelihood function $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$ (or just a function $f(\\boldsymbol{x}$) , that is the conditional distribution for $\\boldsymbol{y}$ with a given $\\boldsymbol{x}$. The estimation of $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$ is made using a data set with \n", "* $n$ cases $i = 0, 1, 2, \\dots, n-1$ \n", "\n", - "* Response (target, dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \\dots, n-1$ \n", + "* Response/output (target, dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \\dots, n-1$ \n", "\n", - "* $p$ so-called explanatory (independent or predictor) variables $\\boldsymbol{x}_i=[x_{i0}, x_{i1}, \\dots, x_{ip-1}]$ with $i = 0, 1, 2, \\dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples. \n", + "* $p$ so-called explanatory/input (independent or predictor) variables $\\boldsymbol{x}_i=[x_{i0}, x_{i1}, \\dots, x_{ip-1}]$ with $i = 0, 1, 2, \\dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples. \n", "\n", " The goal of the regression analysis is to extract/exploit relationship between $\\boldsymbol{y}$ and $\\boldsymbol{x}$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things.\n", "\n", @@ -1474,10 +1441,10 @@ "f(\\mathbf{X}_{i,\\ast})$. When no prior knowledge on the form of\n", "$f(\\cdot)$ is available, it is common to assume a linear relationship\n", "between $\\boldsymbol{X}$ and $\\boldsymbol{y}$. This assumption gives rise to\n", - "the *linear regression model* where $\\boldsymbol{\\beta} = [\\beta_0, \\ldots,\n", - "\\beta_{p-1}]^{T}$ are the *regression parameters*. \n", + "the *linear regression model* where $\\boldsymbol{\\theta} = [\\theta_0, \\ldots,\n", + "\\theta_{p-1}]^{T}$ are the *regression parameters*. \n", "\n", - "Linear regression gives us a set of analytical equations for the parameters $\\beta_j$.\n", + "Linear regression gives us a set of analytical equations for the parameters $\\theta_j$.\n", "\n", "In order to understand the relation among the predictors $p$, the set of data $n$ and the target (outcome, output etc) $\\boldsymbol{y}$,\n", "consider the model we discussed for describing nuclear binding energies. \n", @@ -1488,7 +1455,7 @@ }, { "cell_type": "markdown", - "id": "9845070b", + "id": "08d4a54f", "metadata": { "editable": true }, @@ -1500,7 +1467,7 @@ }, { "cell_type": "markdown", - "id": "fc1de086", + "id": "df21cf23", "metadata": { "editable": true }, @@ -1519,19 +1486,19 @@ }, { "cell_type": "markdown", - "id": "e9b27dfb", + "id": "6359332f", "metadata": { "editable": true }, "source": [ "$$\n", - "y=y(x) \\rightarrow y(x_i)=\\tilde{y}_i+\\epsilon_i=\\sum_{j=0}^{n-1} \\beta_j x_i^j+\\epsilon_i,\n", + "y=y(x) \\rightarrow y(x_i)=\\tilde{y}_i+\\epsilon_i=\\sum_{j=0}^{n-1} \\theta_j x_i^j+\\epsilon_i,\n", "$$" ] }, { "cell_type": "markdown", - "id": "d80ac08b", + "id": "6c8e31ed", "metadata": { "editable": true }, @@ -1543,25 +1510,25 @@ }, { "cell_type": "markdown", - "id": "59de586a", + "id": "0cc24030", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{align*}\n", - "y_0&=\\beta_0+\\beta_1x_0^1+\\beta_2x_0^2+\\dots+\\beta_{n-1}x_0^{n-1}+\\epsilon_0\\\\\n", - "y_1&=\\beta_0+\\beta_1x_1^1+\\beta_2x_1^2+\\dots+\\beta_{n-1}x_1^{n-1}+\\epsilon_1\\\\\n", - "y_2&=\\beta_0+\\beta_1x_2^1+\\beta_2x_2^2+\\dots+\\beta_{n-1}x_2^{n-1}+\\epsilon_2\\\\\n", + "y_0&=\\theta_0+\\theta_1x_0^1+\\theta_2x_0^2+\\dots+\\theta_{n-1}x_0^{n-1}+\\epsilon_0\\\\\n", + "y_1&=\\theta_0+\\theta_1x_1^1+\\theta_2x_1^2+\\dots+\\theta_{n-1}x_1^{n-1}+\\epsilon_1\\\\\n", + "y_2&=\\theta_0+\\theta_1x_2^1+\\theta_2x_2^2+\\dots+\\theta_{n-1}x_2^{n-1}+\\epsilon_2\\\\\n", "\\dots & \\dots \\\\\n", - "y_{n-1}&=\\beta_0+\\beta_1x_{n-1}^1+\\beta_2x_{n-1}^2+\\dots+\\beta_{n-1}x_{n-1}^{n-1}+\\epsilon_{n-1}.\\\\\n", + "y_{n-1}&=\\theta_0+\\theta_1x_{n-1}^1+\\theta_2x_{n-1}^2+\\dots+\\theta_{n-1}x_{n-1}^{n-1}+\\epsilon_{n-1}.\\\\\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", - "id": "3a0a6d11", + "id": "146809cb", "metadata": { "editable": true }, @@ -1571,7 +1538,7 @@ }, { "cell_type": "markdown", - "id": "f545dd3f", + "id": "4796a596", "metadata": { "editable": true }, @@ -1583,7 +1550,7 @@ }, { "cell_type": "markdown", - "id": "a4484e53", + "id": "edfabcdf", "metadata": { "editable": true }, @@ -1593,19 +1560,19 @@ }, { "cell_type": "markdown", - "id": "7a1e5488", + "id": "0b4a6062", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\beta} = [\\beta_0,\\beta_1, \\beta_2,\\dots, \\beta_{n-1}]^T,\n", + "\\boldsymbol{\\theta} = [\\theta_0,\\theta_1, \\theta_2,\\dots, \\theta_{n-1}]^T,\n", "$$" ] }, { "cell_type": "markdown", - "id": "65f8b020", + "id": "e419ff68", "metadata": { "editable": true }, @@ -1615,7 +1582,7 @@ }, { "cell_type": "markdown", - "id": "0ded825d", + "id": "dd55bff1", "metadata": { "editable": true }, @@ -1627,7 +1594,7 @@ }, { "cell_type": "markdown", - "id": "4747b482", + "id": "df571b73", "metadata": { "editable": true }, @@ -1637,7 +1604,7 @@ }, { "cell_type": "markdown", - "id": "1a50e6b1", + "id": "c1c6ffaa", "metadata": { "editable": true }, @@ -1656,7 +1623,7 @@ }, { "cell_type": "markdown", - "id": "2404cda8", + "id": "592bff5d", "metadata": { "editable": true }, @@ -1666,19 +1633,19 @@ }, { "cell_type": "markdown", - "id": "23bc5fc5", + "id": "9b776152", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta}+\\boldsymbol{\\epsilon}.\n", + "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\theta}+\\boldsymbol{\\epsilon}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "ae3d3214", + "id": "29c95a7a", "metadata": { "editable": true }, @@ -1694,27 +1661,27 @@ }, { "cell_type": "markdown", - "id": "bd07fd5c", + "id": "b53b0001", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{align*}\n", - "y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n", - "y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n", - "y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_2\\\\\n", + "y_0&=\\theta_0x_{00}+\\theta_1x_{01}+\\theta_2x_{02}+\\dots+\\theta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n", + "y_1&=\\theta_0x_{10}+\\theta_1x_{11}+\\theta_2x_{12}+\\dots+\\theta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n", + "y_2&=\\theta_0x_{20}+\\theta_1x_{21}+\\theta_2x_{22}+\\dots+\\theta_{n-1}x_{2n-1}+\\epsilon_2\\\\\n", "\\dots & \\dots \\\\\n", - "y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_i\\\\\n", + "y_{i}&=\\theta_0x_{i0}+\\theta_1x_{i1}+\\theta_2x_{i2}+\\dots+\\theta_{n-1}x_{in-1}+\\epsilon_i\\\\\n", "\\dots & \\dots \\\\\n", - "y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n", + "y_{n-1}&=\\theta_0x_{n-1,0}+\\theta_1x_{n-1,2}+\\theta_2x_{n-1,2}+\\dots+\\theta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", - "id": "cd7bdfc0", + "id": "f2fea940", "metadata": { "editable": true }, @@ -1726,7 +1693,7 @@ }, { "cell_type": "markdown", - "id": "acb43d02", + "id": "26411342", "metadata": { "editable": true }, @@ -1745,7 +1712,7 @@ }, { "cell_type": "markdown", - "id": "f1e9d7fc", + "id": "d432e651", "metadata": { "editable": true }, @@ -1755,51 +1722,51 @@ }, { "cell_type": "markdown", - "id": "c41dcd08", + "id": "e536d0b3", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta}+\\boldsymbol{\\epsilon}.\n", + "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\theta}+\\boldsymbol{\\epsilon}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "89ebc548", + "id": "bbe0dd39", "metadata": { "editable": true }, "source": [ - "The left-hand side of this equation is kwown. Our error vector $\\boldsymbol{\\epsilon}$ and the parameter vector $\\boldsymbol{\\beta}$ are our unknow quantities. How can we obtain the optimal set of $\\beta_i$ values? \n", + "The left-hand side of this equation is kwown. Our error vector $\\boldsymbol{\\epsilon}$ and the parameter vector $\\boldsymbol{\\theta}$ are our unknow quantities. How can we obtain the optimal set of $\\theta_i$ values? \n", "\n", "We have defined the matrix $\\boldsymbol{X}$ via the equations" ] }, { "cell_type": "markdown", - "id": "8e0a376a", + "id": "51d53970", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{align*}\n", - "y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n", - "y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n", - "y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_1\\\\\n", + "y_0&=\\theta_0x_{00}+\\theta_1x_{01}+\\theta_2x_{02}+\\dots+\\theta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n", + "y_1&=\\theta_0x_{10}+\\theta_1x_{11}+\\theta_2x_{12}+\\dots+\\theta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n", + "y_2&=\\theta_0x_{20}+\\theta_1x_{21}+\\theta_2x_{22}+\\dots+\\theta_{n-1}x_{2n-1}+\\epsilon_1\\\\\n", "\\dots & \\dots \\\\\n", - "y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_1\\\\\n", + "y_{i}&=\\theta_0x_{i0}+\\theta_1x_{i1}+\\theta_2x_{i2}+\\dots+\\theta_{n-1}x_{in-1}+\\epsilon_1\\\\\n", "\\dots & \\dots \\\\\n", - "y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n", + "y_{n-1}&=\\theta_0x_{n-1,0}+\\theta_1x_{n-1,2}+\\theta_2x_{n-1,2}+\\dots+\\theta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", - "id": "b2c90704", + "id": "4778eaaf", "metadata": { "editable": true }, @@ -1815,8 +1782,8 @@ }, { "cell_type": "code", - "execution_count": 15, - "id": "e89031dc", + "execution_count": 14, + "id": "87ed061d", "metadata": { "collapsed": false, "editable": true @@ -1896,75 +1863,75 @@ }, { "cell_type": "markdown", - "id": "7a1e8f92", + "id": "42b8f7c4", "metadata": { "editable": true }, "source": [ - "With $\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p\\times 1}$, it means that we will hereafter write our equations for the approximation as" + "With $\\boldsymbol{\\theta}\\in {\\mathbb{R}}^{p\\times 1}$, it means that we will hereafter write our equations for the approximation as" ] }, { "cell_type": "markdown", - "id": "84bf3673", + "id": "73ea9a01", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n", + "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\theta},\n", "$$" ] }, { "cell_type": "markdown", - "id": "e10c4fc5", + "id": "253891dd", "metadata": { "editable": true }, "source": [ "throughout these lectures. \n", "\n", - "With the above we use the design matrix to define the approximation $\\boldsymbol{\\tilde{y}}$ via the unknown quantity $\\boldsymbol{\\beta}$ as" + "With the above we use the design matrix to define the approximation $\\boldsymbol{\\tilde{y}}$ via the unknown quantity $\\boldsymbol{\\theta}$ as" ] }, { "cell_type": "markdown", - "id": "20866df8", + "id": "d2d20886", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n", + "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\theta},\n", "$$" ] }, { "cell_type": "markdown", - "id": "a075809a", + "id": "480c4c58", "metadata": { "editable": true }, "source": [ - "and in order to find the optimal parameters $\\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\\tilde{y}_i$, namely" + "and in order to find the optimal parameters $\\theta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\\tilde{y}_i$, namely" ] }, { "cell_type": "markdown", - "id": "ba0e251b", + "id": "7a782da9", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n", + "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n", "$$" ] }, { "cell_type": "markdown", - "id": "93a04fb7", + "id": "c1c60d77", "metadata": { "editable": true }, @@ -1974,19 +1941,19 @@ }, { "cell_type": "markdown", - "id": "56c4fcf1", + "id": "60d26064", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", + "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "e8fc405e", + "id": "8a19c134", "metadata": { "editable": true }, @@ -1999,43 +1966,43 @@ }, { "cell_type": "markdown", - "id": "fefb3fd3", + "id": "f21d525f", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\frac{1}{2n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2,\n", + "C(\\boldsymbol{\\theta})=\\frac{1}{2n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2,\n", "$$" ] }, { "cell_type": "markdown", - "id": "d0eeb2dc", + "id": "f41c45c4", "metadata": { "editable": true }, "source": [ - "since when taking the first derivative with respect to the unknown parameters $\\beta$, the factor of $2$ cancels out. \n", + "since when taking the first derivative with respect to the unknown parameters $\\theta$, the factor of $2$ cancels out. \n", "\n", "The function" ] }, { "cell_type": "markdown", - "id": "1f408b23", + "id": "b356426a", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\},\n", + "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\},\n", "$$" ] }, { "cell_type": "markdown", - "id": "dee2a248", + "id": "5169a2dd", "metadata": { "editable": true }, @@ -2046,19 +2013,19 @@ }, { "cell_type": "markdown", - "id": "f1d1d292", + "id": "19839cdf", "metadata": { "editable": true }, "source": [ "$$\n", - "y_{i}=\\langle y_i \\rangle = \\beta_0x_{i,0}+\\beta_1x_{i,1}+\\beta_2x_{i,2}+\\dots+\\beta_{n-1}x_{i,n-1}+\\epsilon_i,\n", + "y_{i}=\\langle y_i \\rangle = \\theta_0x_{i,0}+\\theta_1x_{i,1}+\\theta_2x_{i,2}+\\dots+\\theta_{n-1}x_{i,n-1}+\\epsilon_i,\n", "$$" ] }, { "cell_type": "markdown", - "id": "8b2075d1", + "id": "e664d67f", "metadata": { "editable": true }, @@ -2072,25 +2039,25 @@ "the standard deviation discussed earlier. In the discussion here we\n", "will treat $y_i$ as our exact value for the response variable.\n", "\n", - "In order to find the parameters $\\beta_i$ we will then minimize the spread of $C(\\boldsymbol{\\beta})$, that is we are going to solve the problem" + "In order to find the parameters $\\theta_i$ we will then minimize the spread of $C(\\boldsymbol{\\theta})$, that is we are going to solve the problem" ] }, { "cell_type": "markdown", - "id": "0c3145a6", + "id": "a14a28ab", "metadata": { "editable": true }, "source": [ "$$\n", - "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", - "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "e72f34b6", + "id": "e2f643ef", "metadata": { "editable": true }, @@ -2100,19 +2067,19 @@ }, { "cell_type": "markdown", - "id": "f2f9ee3f", + "id": "869ceba9", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n", + "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_j} = \\frac{\\partial }{\\partial \\theta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "f15f85e7", + "id": "371f2221", "metadata": { "editable": true }, @@ -2122,19 +2089,19 @@ }, { "cell_type": "markdown", - "id": "ded06fd1", + "id": "b4bf3615", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)\\right]=0,\n", + "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}\\right)\\right]=0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "dc9c099b", + "id": "3baf0d9a", "metadata": { "editable": true }, @@ -2144,19 +2111,19 @@ }, { "cell_type": "markdown", - "id": "40bc212d", + "id": "5e98ea5f", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right).\n", + "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right).\n", "$$" ] }, { "cell_type": "markdown", - "id": "7bc3483f", + "id": "6e515735", "metadata": { "editable": true }, @@ -2166,19 +2133,19 @@ }, { "cell_type": "markdown", - "id": "22662e2a", + "id": "8363a7ce", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right),\n", + "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right),\n", "$$" ] }, { "cell_type": "markdown", - "id": "6798151b", + "id": "77eb2d81", "metadata": { "editable": true }, @@ -2188,19 +2155,19 @@ }, { "cell_type": "markdown", - "id": "e63dc09e", + "id": "a94cb0f2", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta},\n", + "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta},\n", "$$" ] }, { "cell_type": "markdown", - "id": "a2f541a1", + "id": "09bdeb45", "metadata": { "editable": true }, @@ -2210,19 +2177,19 @@ }, { "cell_type": "markdown", - "id": "fd8bba77", + "id": "d44f06e9", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\beta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "\\boldsymbol{\\theta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "2163c93e", + "id": "1a83dceb", "metadata": { "editable": true }, @@ -2246,7 +2213,7 @@ }, { "cell_type": "markdown", - "id": "48634f87", + "id": "efa701a0", "metadata": { "editable": true }, @@ -2258,7 +2225,7 @@ }, { "cell_type": "markdown", - "id": "9117c119", + "id": "84be9b76", "metadata": { "editable": true }, @@ -2268,7 +2235,7 @@ }, { "cell_type": "markdown", - "id": "c7cd5b11", + "id": "194914da", "metadata": { "editable": true }, @@ -2280,7 +2247,7 @@ }, { "cell_type": "markdown", - "id": "1159b476", + "id": "da2cf79d", "metadata": { "editable": true }, @@ -2290,7 +2257,7 @@ }, { "cell_type": "markdown", - "id": "75098194", + "id": "7f331eee", "metadata": { "editable": true }, @@ -2302,7 +2269,7 @@ }, { "cell_type": "markdown", - "id": "3bd3d52b", + "id": "ad13d0dc", "metadata": { "editable": true }, @@ -2315,19 +2282,19 @@ }, { "cell_type": "markdown", - "id": "b086e3f6", + "id": "d00b722b", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial^2 C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n", + "\\frac{\\partial^2 C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}^T\\partial \\boldsymbol{\\theta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "6fe2d631", + "id": "8e486d44", "metadata": { "editable": true }, @@ -2339,7 +2306,7 @@ }, { "cell_type": "markdown", - "id": "26bd4db6", + "id": "29dcd5db", "metadata": { "editable": true }, @@ -2351,7 +2318,7 @@ }, { "cell_type": "markdown", - "id": "141de40c", + "id": "1a45048f", "metadata": { "editable": true }, @@ -2366,19 +2333,19 @@ }, { "cell_type": "markdown", - "id": "d44398a3", + "id": "3c8c91cc", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\epsilon} = \\boldsymbol{y}-\\boldsymbol{\\tilde{y}} = \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta},\n", + "\\boldsymbol{\\epsilon} = \\boldsymbol{y}-\\boldsymbol{\\tilde{y}} = \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta},\n", "$$" ] }, { "cell_type": "markdown", - "id": "fb4c7d58", + "id": "4adc4243", "metadata": { "editable": true }, @@ -2388,19 +2355,19 @@ }, { "cell_type": "markdown", - "id": "3c1a0371", + "id": "9c998ca0", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n", + "\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)= 0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "84a0c456", + "id": "4894c857", "metadata": { "editable": true }, @@ -2410,50 +2377,50 @@ }, { "cell_type": "markdown", - "id": "1c5edf48", + "id": "80ea22e9", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{X}^T\\boldsymbol{\\epsilon}=\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n", + "\\boldsymbol{X}^T\\boldsymbol{\\epsilon}=\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)= 0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "332a3b12", + "id": "e398b8f7", "metadata": { "editable": true }, "source": [ - "meaning that the solution for $\\boldsymbol{\\beta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.\n", + "meaning that the solution for $\\boldsymbol{\\theta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.\n", "\n", "Let us now return to our nuclear binding energies and simply code the above equations. \n", "\n", - "It is rather straightforward to implement the matrix inversion and obtain the parameters $\\boldsymbol{\\beta}$. After having defined the matrix $\\boldsymbol{X}$ we simply need to \n", + "It is rather straightforward to implement the matrix inversion and obtain the parameters $\\boldsymbol{\\theta}$. After having defined the matrix $\\boldsymbol{X}$ we simply need to \n", "write" ] }, { "cell_type": "code", - "execution_count": 16, - "id": "f52e02a6", + "execution_count": 15, + "id": "27d5b629", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ - "# matrix inversion to find beta\n", - "beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)\n", + "# matrix inversion to find theta\n", + "theta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)\n", "# and then make the prediction\n", - "ytilde = X @ beta" + "ytilde = X @ theta" ] }, { "cell_type": "markdown", - "id": "36ae01f6", + "id": "a95d91a9", "metadata": { "editable": true }, @@ -2463,8 +2430,8 @@ }, { "cell_type": "code", - "execution_count": 17, - "id": "142d2f33", + "execution_count": 16, + "id": "4d42fedc", "metadata": { "collapsed": false, "editable": true @@ -2477,7 +2444,7 @@ }, { "cell_type": "markdown", - "id": "46296414", + "id": "d15ca047", "metadata": { "editable": true }, @@ -2487,8 +2454,8 @@ }, { "cell_type": "code", - "execution_count": 18, - "id": "86ed7e92", + "execution_count": 17, + "id": "414ab312", "metadata": { "collapsed": false, "editable": true @@ -2511,7 +2478,7 @@ }, { "cell_type": "markdown", - "id": "942647c0", + "id": "11bc6803", "metadata": { "editable": true }, @@ -2522,8 +2489,8 @@ }, { "cell_type": "code", - "execution_count": 19, - "id": "e26cee7f", + "execution_count": 18, + "id": "aaa2eeb3", "metadata": { "collapsed": false, "editable": true @@ -2536,7 +2503,7 @@ }, { "cell_type": "markdown", - "id": "462de17c", + "id": "b9eb34c0", "metadata": { "editable": true }, @@ -2546,8 +2513,8 @@ }, { "cell_type": "code", - "execution_count": 20, - "id": "e50560cf", + "execution_count": 19, + "id": "1a22333a", "metadata": { "collapsed": false, "editable": true @@ -2559,7 +2526,7 @@ }, { "cell_type": "markdown", - "id": "ad325590", + "id": "067660ad", "metadata": { "editable": true }, @@ -2569,8 +2536,8 @@ }, { "cell_type": "code", - "execution_count": 21, - "id": "682dcbff", + "execution_count": 20, + "id": "25d43913", "metadata": { "collapsed": false, "editable": true @@ -2586,7 +2553,7 @@ }, { "cell_type": "markdown", - "id": "7a3c40b6", + "id": "73c9fe40", "metadata": { "editable": true }, @@ -2596,8 +2563,8 @@ }, { "cell_type": "code", - "execution_count": 22, - "id": "1cc29e48", + "execution_count": 21, + "id": "0ecca752", "metadata": { "collapsed": false, "editable": true @@ -2611,7 +2578,7 @@ }, { "cell_type": "markdown", - "id": "1ddbddcd", + "id": "da2c5259", "metadata": { "editable": true }, @@ -2633,43 +2600,43 @@ }, { "cell_type": "markdown", - "id": "90a8edfe", + "id": "acd25436", "metadata": { "editable": true }, "source": [ "$$\n", - "\\chi^2(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\frac{\\left(y_i-\\tilde{y}_i\\right)^2}{\\sigma_i^2}=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\frac{1}{\\boldsymbol{\\Sigma^2}}\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n", + "\\chi^2(\\boldsymbol{\\theta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\frac{\\left(y_i-\\tilde{y}_i\\right)^2}{\\sigma_i^2}=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\frac{1}{\\boldsymbol{\\Sigma^2}}\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n", "$$" ] }, { "cell_type": "markdown", - "id": "4ce63e21", + "id": "c3333906", "metadata": { "editable": true }, "source": [ "where the matrix $\\boldsymbol{\\Sigma}$ is a diagonal matrix with $\\sigma_i$ as matrix elements. \n", "\n", - "In order to find the parameters $\\beta_i$ we will then minimize the spread of $\\chi^2(\\boldsymbol{\\beta})$ by requiring" + "In order to find the parameters $\\theta_i$ we will then minimize the spread of $\\chi^2(\\boldsymbol{\\theta})$ by requiring" ] }, { "cell_type": "markdown", - "id": "d4c3d110", + "id": "d9069e54", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)^2\\right]=0,\n", + "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\theta_j} = \\frac{\\partial }{\\partial \\theta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)^2\\right]=0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "cf9ad9c2", + "id": "2d4b51c7", "metadata": { "editable": true }, @@ -2679,19 +2646,19 @@ }, { "cell_type": "markdown", - "id": "3b1f5fc1", + "id": "365750e2", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}\\frac{x_{ij}}{\\sigma_i}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)\\right]=0,\n", + "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\theta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}\\frac{x_{ij}}{\\sigma_i}\\left(\\frac{y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)\\right]=0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "559f6a04", + "id": "16a7ed8d", "metadata": { "editable": true }, @@ -2701,19 +2668,19 @@ }, { "cell_type": "markdown", - "id": "bba9e13a", + "id": "ce115915", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\beta}\\right).\n", + "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\theta}\\right).\n", "$$" ] }, { "cell_type": "markdown", - "id": "c97cf725", + "id": "10c88bf7", "metadata": { "editable": true }, @@ -2725,19 +2692,19 @@ }, { "cell_type": "markdown", - "id": "6256d393", + "id": "c13f349e", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\beta}\\right),\n", + "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\theta}\\right),\n", "$$" ] }, { "cell_type": "markdown", - "id": "4274f29f", + "id": "e1cfa827", "metadata": { "editable": true }, @@ -2747,19 +2714,19 @@ }, { "cell_type": "markdown", - "id": "6424e32f", + "id": "228294e8", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{A}^T\\boldsymbol{b} = \\boldsymbol{A}^T\\boldsymbol{A}\\boldsymbol{\\beta},\n", + "\\boldsymbol{A}^T\\boldsymbol{b} = \\boldsymbol{A}^T\\boldsymbol{A}\\boldsymbol{\\theta},\n", "$$" ] }, { "cell_type": "markdown", - "id": "2babfc95", + "id": "dd1aa581", "metadata": { "editable": true }, @@ -2769,19 +2736,19 @@ }, { "cell_type": "markdown", - "id": "1a25daa2", + "id": "4e631fe5", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\beta} =\\left(\\boldsymbol{A}^T\\boldsymbol{A}\\right)^{-1}\\boldsymbol{A}^T\\boldsymbol{b}.\n", + "\\boldsymbol{\\theta} =\\left(\\boldsymbol{A}^T\\boldsymbol{A}\\right)^{-1}\\boldsymbol{A}^T\\boldsymbol{b}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "d2a18a90", + "id": "a1c651fa", "metadata": { "editable": true }, @@ -2791,7 +2758,7 @@ }, { "cell_type": "markdown", - "id": "c2730e09", + "id": "0ba91c05", "metadata": { "editable": true }, @@ -2803,51 +2770,51 @@ }, { "cell_type": "markdown", - "id": "dab1abb8", + "id": "9c7d45d8", "metadata": { "editable": true }, "source": [ - "we have then the following expression for the parameters $\\beta_j$ (the matrix elements of $\\boldsymbol{H}$ are $h_{ij}$)" + "we have then the following expression for the parameters $\\theta_j$ (the matrix elements of $\\boldsymbol{H}$ are $h_{ij}$)" ] }, { "cell_type": "markdown", - "id": "be9b74c2", + "id": "1148dda1", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_j = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}\\frac{y_i}{\\sigma_i}\\frac{x_{ik}}{\\sigma_i} = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}b_ia_{ik}\n", + "\\theta_j = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}\\frac{y_i}{\\sigma_i}\\frac{x_{ik}}{\\sigma_i} = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}b_ia_{ik}\n", "$$" ] }, { "cell_type": "markdown", - "id": "38e7188c", + "id": "899ad725", "metadata": { "editable": true }, "source": [ - "We state without proof the expression for the uncertainty in the parameters $\\beta_j$ as (we leave this as an exercise)" + "We state without proof the expression for the uncertainty in the parameters $\\theta_j$ as (we leave this as an exercise)" ] }, { "cell_type": "markdown", - "id": "c568814d", + "id": "442cefde", "metadata": { "editable": true }, "source": [ "$$\n", - "\\sigma^2(\\beta_j) = \\sum_{i=0}^{n-1}\\sigma_i^2\\left( \\frac{\\partial \\beta_j}{\\partial y_i}\\right)^2,\n", + "\\sigma^2(\\theta_j) = \\sum_{i=0}^{n-1}\\sigma_i^2\\left( \\frac{\\partial \\theta_j}{\\partial y_i}\\right)^2,\n", "$$" ] }, { "cell_type": "markdown", - "id": "37fb03f1", + "id": "303d558b", "metadata": { "editable": true }, @@ -2857,19 +2824,19 @@ }, { "cell_type": "markdown", - "id": "753cb5d9", + "id": "c8361a08", "metadata": { "editable": true }, "source": [ "$$\n", - "\\sigma^2(\\beta_j) = \\left(\\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}a_{ik}\\right)\\left(\\sum_{l=0}^{p-1}h_{jl}\\sum_{m=0}^{n-1}a_{ml}\\right) = h_{jj}!\n", + "\\sigma^2(\\theta_j) = \\left(\\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}a_{ik}\\right)\\left(\\sum_{l=0}^{p-1}h_{jl}\\sum_{m=0}^{n-1}a_{ml}\\right) = h_{jj}!\n", "$$" ] }, { "cell_type": "markdown", - "id": "a887be4f", + "id": "fad61a6c", "metadata": { "editable": true }, @@ -2879,41 +2846,41 @@ }, { "cell_type": "markdown", - "id": "c99c1576", + "id": "29731faf", "metadata": { "editable": true }, "source": [ "$$\n", - "y=y(x) \\rightarrow y(x_i) \\approx \\beta_0+\\beta_1 x_i.\n", + "y=y(x) \\rightarrow y(x_i) \\approx \\theta_0+\\theta_1 x_i.\n", "$$" ] }, { "cell_type": "markdown", - "id": "ddb752ed", + "id": "ecb0789a", "metadata": { "editable": true }, "source": [ - "By computing the derivatives of $\\chi^2$ with respect to $\\beta_0$ and $\\beta_1$ show that these are given by" + "By computing the derivatives of $\\chi^2$ with respect to $\\theta_0$ and $\\theta_1$ show that these are given by" ] }, { "cell_type": "markdown", - "id": "d5d9bc16", + "id": "bf04c0eb", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -2\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0,\n", + "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\theta_0} = -2\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\theta_0-\\theta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "971e3cbe", + "id": "7fba1a68", "metadata": { "editable": true }, @@ -2923,19 +2890,19 @@ }, { "cell_type": "markdown", - "id": "14a81f3f", + "id": "cb8528d5", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_i\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0.\n", + "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\theta_1} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_i\\left(\\frac{y_i-\\theta_0-\\theta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0.\n", "$$" ] }, { "cell_type": "markdown", - "id": "daa744f2", + "id": "2771f01a", "metadata": { "editable": true }, @@ -2946,7 +2913,7 @@ }, { "cell_type": "markdown", - "id": "bcdd91b0", + "id": "dcf74fcf", "metadata": { "editable": true }, @@ -2958,7 +2925,7 @@ }, { "cell_type": "markdown", - "id": "78646001", + "id": "a7e95721", "metadata": { "editable": true }, @@ -2970,7 +2937,7 @@ }, { "cell_type": "markdown", - "id": "8eb6b0de", + "id": "a636424d", "metadata": { "editable": true }, @@ -2982,7 +2949,7 @@ }, { "cell_type": "markdown", - "id": "f5641168", + "id": "294e5692", "metadata": { "editable": true }, @@ -2994,7 +2961,7 @@ }, { "cell_type": "markdown", - "id": "c14fcf77", + "id": "923ebdc5", "metadata": { "editable": true }, @@ -3006,7 +2973,7 @@ }, { "cell_type": "markdown", - "id": "9b374023", + "id": "d605b9e0", "metadata": { "editable": true }, @@ -3016,45 +2983,45 @@ }, { "cell_type": "markdown", - "id": "5f34ff5d", + "id": "d9e8c477", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_0 = \\frac{\\gamma_{xx}\\gamma_y-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2},\n", + "\\theta_0 = \\frac{\\gamma_{xx}\\gamma_y-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2},\n", "$$" ] }, { "cell_type": "markdown", - "id": "349a1654", + "id": "2831c92f", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_1 = \\frac{\\gamma_{xy}\\gamma-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2}.\n", + "\\theta_1 = \\frac{\\gamma_{xy}\\gamma-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "bdc8cc84", + "id": "36edba71", "metadata": { "editable": true }, "source": [ "This approach (different linear and non-linear regression) suffers\n", "often from both being underdetermined and overdetermined in the\n", - "unknown coefficients $\\beta_i$. A better approach is to use the\n", + "unknown coefficients $\\theta_i$. A better approach is to use the\n", "Singular Value Decomposition (SVD) method discussed below. Or using\n", "Lasso and Ridge regression. See below." ] }, { "cell_type": "markdown", - "id": "8e31d846", + "id": "f209ca1f", "metadata": { "editable": true }, @@ -3079,8 +3046,8 @@ }, { "cell_type": "code", - "execution_count": 23, - "id": "ba56fb3a", + "execution_count": 22, + "id": "5d230f93", "metadata": { "collapsed": false, "editable": true @@ -3162,7 +3129,7 @@ }, { "cell_type": "markdown", - "id": "982e0c69", + "id": "b3d6f5a2", "metadata": { "editable": true }, @@ -3173,7 +3140,7 @@ }, { "cell_type": "markdown", - "id": "acb30447", + "id": "4b61d1b0", "metadata": { "editable": true }, @@ -3200,8 +3167,8 @@ }, { "cell_type": "code", - "execution_count": 24, - "id": "301a7664", + "execution_count": 23, + "id": "8c25e039", "metadata": { "collapsed": false, "editable": true @@ -3232,16 +3199,16 @@ "X[:,2] = x**2\n", "# We split the data in test and training data\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n", - "# matrix inversion to find beta\n", - "beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", - "print(beta)\n", + "# matrix inversion to find theta\n", + "theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", + "print(theta)\n", "# and then make the prediction\n", - "ytilde = X_train @ beta\n", + "ytilde = X_train @ theta\n", "print(\"Training R2\")\n", "print(R2(y_train,ytilde))\n", "print(\"Training MSE\")\n", "print(MSE(y_train,ytilde))\n", - "ypredict = X_test @ beta\n", + "ypredict = X_test @ theta\n", "print(\"Test R2\")\n", "print(R2(y_test,ypredict))\n", "print(\"Test MSE\")\n", @@ -3250,7 +3217,7 @@ }, { "cell_type": "markdown", - "id": "655f046e", + "id": "3c09d70c", "metadata": { "editable": true }, @@ -3260,8 +3227,8 @@ }, { "cell_type": "code", - "execution_count": 25, - "id": "413acb4d", + "execution_count": 24, + "id": "3e9b7ad8", "metadata": { "collapsed": false, "editable": true @@ -3286,7 +3253,7 @@ }, { "cell_type": "markdown", - "id": "f46a080f", + "id": "18804e9f", "metadata": { "editable": true }, @@ -3302,8 +3269,8 @@ }, { "cell_type": "code", - "execution_count": 26, - "id": "96391e08", + "execution_count": 25, + "id": "90f453cd", "metadata": { "collapsed": false, "editable": true @@ -3361,15 +3328,15 @@ "X[:,4] = Density**(5.0/3.0)\n", "# We split the data in test and training data\n", "X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)\n", - "# matrix inversion to find beta\n", - "beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)\n", + "# matrix inversion to find theta\n", + "theta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)\n", "# and then make the prediction\n", - "ytilde = X_train @ beta\n", + "ytilde = X_train @ theta\n", "print(\"Training R2\")\n", "print(R2(y_train,ytilde))\n", "print(\"Training MSE\")\n", "print(MSE(y_train,ytilde))\n", - "ypredict = X_test @ beta\n", + "ypredict = X_test @ theta\n", "print(\"Test R2\")\n", "print(R2(y_test,ypredict))\n", "print(\"Test MSE\")\n", @@ -3378,7 +3345,7 @@ }, { "cell_type": "markdown", - "id": "46a1885f", + "id": "3d672d7a", "metadata": { "editable": true }, @@ -3422,7 +3389,7 @@ }, { "cell_type": "markdown", - "id": "a2041293", + "id": "e2426f64", "metadata": { "editable": true }, @@ -3433,8 +3400,8 @@ }, { "cell_type": "code", - "execution_count": 27, - "id": "f813a977", + "execution_count": 26, + "id": "519e0c09", "metadata": { "collapsed": false, "editable": true @@ -3450,7 +3417,7 @@ }, { "cell_type": "markdown", - "id": "5fc086ec", + "id": "7b6fd188", "metadata": { "editable": true }, @@ -3460,8 +3427,8 @@ }, { "cell_type": "code", - "execution_count": 28, - "id": "bf72511a", + "execution_count": 27, + "id": "3665e1b0", "metadata": { "collapsed": false, "editable": true @@ -3479,7 +3446,7 @@ }, { "cell_type": "markdown", - "id": "70823078", + "id": "96f2fe7b", "metadata": { "editable": true }, @@ -3489,8 +3456,8 @@ }, { "cell_type": "code", - "execution_count": 29, - "id": "85e192bf", + "execution_count": 28, + "id": "df2c84ca", "metadata": { "collapsed": false, "editable": true @@ -3504,7 +3471,7 @@ }, { "cell_type": "markdown", - "id": "13fd362f", + "id": "0c4588d5", "metadata": { "editable": true }, @@ -3514,8 +3481,8 @@ }, { "cell_type": "code", - "execution_count": 30, - "id": "8f379d9b", + "execution_count": 29, + "id": "209438db", "metadata": { "collapsed": false, "editable": true @@ -3528,7 +3495,7 @@ }, { "cell_type": "markdown", - "id": "c5f9dc18", + "id": "1b67c46a", "metadata": { "editable": true }, @@ -3538,8 +3505,8 @@ }, { "cell_type": "code", - "execution_count": 31, - "id": "cb9f3445", + "execution_count": 30, + "id": "236343bf", "metadata": { "collapsed": false, "editable": true @@ -3556,7 +3523,7 @@ }, { "cell_type": "markdown", - "id": "7a16f238", + "id": "c8726877", "metadata": { "editable": true }, @@ -3566,8 +3533,8 @@ }, { "cell_type": "code", - "execution_count": 32, - "id": "a5386de5", + "execution_count": 31, + "id": "aaa47b00", "metadata": { "collapsed": false, "editable": true @@ -3583,7 +3550,7 @@ }, { "cell_type": "markdown", - "id": "1d2e90ee", + "id": "0283f3f1", "metadata": { "editable": true }, @@ -3593,8 +3560,8 @@ }, { "cell_type": "code", - "execution_count": 33, - "id": "b494903c", + "execution_count": 32, + "id": "c0823ed1", "metadata": { "collapsed": false, "editable": true @@ -3618,7 +3585,7 @@ }, { "cell_type": "markdown", - "id": "c580e7c5", + "id": "dc2cf448", "metadata": { "editable": true }, @@ -3628,8 +3595,8 @@ }, { "cell_type": "code", - "execution_count": 34, - "id": "45830e09", + "execution_count": 33, + "id": "ee945b00", "metadata": { "collapsed": false, "editable": true @@ -3642,7 +3609,7 @@ }, { "cell_type": "markdown", - "id": "fe0e34ba", + "id": "288a1417", "metadata": { "editable": true }, @@ -3652,8 +3619,8 @@ }, { "cell_type": "code", - "execution_count": 35, - "id": "f73e5b0f", + "execution_count": 34, + "id": "0b640ff9", "metadata": { "collapsed": false, "editable": true @@ -3673,7 +3640,7 @@ }, { "cell_type": "markdown", - "id": "a99e4f96", + "id": "32cfad5b", "metadata": { "editable": true }, @@ -3683,8 +3650,8 @@ }, { "cell_type": "code", - "execution_count": 36, - "id": "8c972941", + "execution_count": 35, + "id": "792df674", "metadata": { "collapsed": false, "editable": true @@ -3726,8 +3693,8 @@ }, { "cell_type": "code", - "execution_count": 37, - "id": "91ef994e", + "execution_count": 36, + "id": "6f9196e5", "metadata": { "collapsed": false, "editable": true @@ -3742,7 +3709,7 @@ }, { "cell_type": "markdown", - "id": "b2302654", + "id": "b72d5080", "metadata": { "editable": true }, @@ -3819,7 +3786,7 @@ }, { "cell_type": "markdown", - "id": "9f2998ff", + "id": "009b948b", "metadata": { "editable": true }, @@ -3831,7 +3798,7 @@ }, { "cell_type": "markdown", - "id": "bc8c9873", + "id": "4b02a26a", "metadata": { "editable": true }, @@ -3850,8 +3817,8 @@ }, { "cell_type": "code", - "execution_count": 38, - "id": "810f6b81", + "execution_count": 37, + "id": "efd083fc", "metadata": { "collapsed": false, "editable": true @@ -3885,7 +3852,7 @@ }, { "cell_type": "markdown", - "id": "565a1cd1", + "id": "b3590aea", "metadata": { "editable": true }, @@ -3900,7 +3867,7 @@ }, { "cell_type": "markdown", - "id": "0a65c309", + "id": "9592b7bf", "metadata": { "editable": true }, @@ -3912,7 +3879,7 @@ }, { "cell_type": "markdown", - "id": "8b0d4688", + "id": "0941f045", "metadata": { "editable": true }, @@ -3922,7 +3889,7 @@ }, { "cell_type": "markdown", - "id": "d1f9ca89", + "id": "d47fc8d3", "metadata": { "editable": true }, @@ -3945,8 +3912,8 @@ }, { "cell_type": "code", - "execution_count": 39, - "id": "28d4c3a0", + "execution_count": 38, + "id": "1ddb9cb5", "metadata": { "collapsed": false, "editable": true @@ -3990,7 +3957,7 @@ }, { "cell_type": "markdown", - "id": "dfe532c5", + "id": "725b78e9", "metadata": { "editable": true }, @@ -4000,7 +3967,7 @@ }, { "cell_type": "markdown", - "id": "86f0a026", + "id": "b08f94e5", "metadata": { "editable": true }, @@ -4069,7 +4036,7 @@ }, { "cell_type": "markdown", - "id": "fd6857da", + "id": "95e72a9e", "metadata": { "editable": true }, @@ -4082,8 +4049,8 @@ }, { "cell_type": "code", - "execution_count": 40, - "id": "c84d4e1f", + "execution_count": 39, + "id": "fb3ad8e5", "metadata": { "collapsed": false, "editable": true @@ -4096,7 +4063,7 @@ }, { "cell_type": "markdown", - "id": "10579805", + "id": "02fe1db6", "metadata": { "editable": true }, @@ -4110,7 +4077,7 @@ }, { "cell_type": "markdown", - "id": "30c31580", + "id": "33a4aed5", "metadata": { "editable": true }, @@ -4123,7 +4090,7 @@ }, { "cell_type": "markdown", - "id": "af797776", + "id": "78a3bc86", "metadata": { "editable": true }, @@ -4134,7 +4101,7 @@ }, { "cell_type": "markdown", - "id": "9c6eebe7", + "id": "38c3a27d", "metadata": { "editable": true }, @@ -4146,7 +4113,7 @@ }, { "cell_type": "markdown", - "id": "06a2f8be", + "id": "7eb5c51b", "metadata": { "editable": true }, @@ -4156,7 +4123,7 @@ }, { "cell_type": "markdown", - "id": "0b5633b3", + "id": "3597b20a", "metadata": { "editable": true }, @@ -4168,7 +4135,7 @@ }, { "cell_type": "markdown", - "id": "108886bb", + "id": "82922e13", "metadata": { "editable": true }, @@ -4178,13 +4145,13 @@ "\n", "\n", "**Solution.**\n", - "The code here is an example of where we define our own design matrix and fit parameters $\\beta$." + "The code here is an example of where we define our own design matrix and fit parameters $\\theta$." ] }, { "cell_type": "code", - "execution_count": 41, - "id": "94716007", + "execution_count": 40, + "id": "61cd693e", "metadata": { "collapsed": false, "editable": true @@ -4217,16 +4184,16 @@ "X[:,2] = x**2\n", "# We split the data in test and training data\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n", - "# matrix inversion to find beta\n", - "beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", - "print(beta)\n", + "# matrix inversion to find theta\n", + "theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", + "print(theta)\n", "# and then make the prediction\n", - "ytilde = X_train @ beta\n", + "ytilde = X_train @ theta\n", "print(\"Training R2\")\n", "print(R2(y_train,ytilde))\n", "print(\"Training MSE\")\n", "print(MSE(y_train,ytilde))\n", - "ypredict = X_test @ beta\n", + "ypredict = X_test @ theta\n", "print(\"Test R2\")\n", "print(R2(y_test,ypredict))\n", "print(\"Test MSE\")\n", @@ -4235,7 +4202,7 @@ }, { "cell_type": "markdown", - "id": "3aa5be35", + "id": "2f8d2e6e", "metadata": { "editable": true }, @@ -4245,7 +4212,7 @@ }, { "cell_type": "markdown", - "id": "1af6d7cd", + "id": "c81f5caf", "metadata": { "editable": true }, @@ -4290,8 +4257,8 @@ }, { "cell_type": "code", - "execution_count": 42, - "id": "1f4f0115", + "execution_count": 41, + "id": "37ca3335", "metadata": { "collapsed": false, "editable": true @@ -4304,7 +4271,7 @@ }, { "cell_type": "markdown", - "id": "8904ba25", + "id": "e6925e8c", "metadata": { "editable": true }, @@ -4314,8 +4281,8 @@ }, { "cell_type": "code", - "execution_count": 43, - "id": "04fdf05f", + "execution_count": 42, + "id": "caecb70a", "metadata": { "collapsed": false, "editable": true @@ -4330,7 +4297,7 @@ }, { "cell_type": "markdown", - "id": "a22f1ef9", + "id": "deabdf0c", "metadata": { "editable": true }, @@ -4347,8 +4314,8 @@ }, { "cell_type": "code", - "execution_count": 44, - "id": "ad96211c", + "execution_count": 43, + "id": "b083bb84", "metadata": { "collapsed": false, "editable": true @@ -4365,7 +4332,7 @@ }, { "cell_type": "markdown", - "id": "a103eff1", + "id": "21102c44", "metadata": { "editable": true }, @@ -4380,8 +4347,8 @@ }, { "cell_type": "code", - "execution_count": 45, - "id": "d9b5eed5", + "execution_count": 44, + "id": "b9108dab", "metadata": { "collapsed": false, "editable": true @@ -4425,7 +4392,7 @@ }, { "cell_type": "markdown", - "id": "9decc489", + "id": "cd19b575", "metadata": { "editable": true }, @@ -4435,7 +4402,7 @@ }, { "cell_type": "markdown", - "id": "f2ffecd1", + "id": "eb2f6352", "metadata": { "editable": true }, @@ -4446,7 +4413,7 @@ }, { "cell_type": "markdown", - "id": "356683e0", + "id": "6e09ea94", "metadata": { "editable": true }, @@ -4457,7 +4424,7 @@ }, { "cell_type": "markdown", - "id": "01abcbae", + "id": "38857156", "metadata": { "editable": true }, @@ -4468,7 +4435,7 @@ }, { "cell_type": "markdown", - "id": "8b2a42d8", + "id": "6b9115a6", "metadata": { "editable": true }, @@ -4490,8 +4457,8 @@ }, { "cell_type": "code", - "execution_count": 46, - "id": "4ae34751", + "execution_count": 45, + "id": "ae7a71c1", "metadata": { "collapsed": false, "editable": true @@ -4504,7 +4471,7 @@ }, { "cell_type": "markdown", - "id": "559e2cff", + "id": "ca615d39", "metadata": { "editable": true }, @@ -4521,7 +4488,7 @@ }, { "cell_type": "markdown", - "id": "03b05ffa", + "id": "d83c8354", "metadata": { "editable": true }, @@ -4534,7 +4501,7 @@ }, { "cell_type": "markdown", - "id": "1ff2cf84", + "id": "74f6e912", "metadata": { "editable": true }, @@ -4545,7 +4512,7 @@ }, { "cell_type": "markdown", - "id": "fe18f837", + "id": "93761664", "metadata": { "editable": true }, @@ -4557,7 +4524,7 @@ }, { "cell_type": "markdown", - "id": "49eff4b4", + "id": "be729d32", "metadata": { "editable": true }, @@ -4567,7 +4534,7 @@ }, { "cell_type": "markdown", - "id": "acc25706", + "id": "08549523", "metadata": { "editable": true }, @@ -4579,7 +4546,7 @@ }, { "cell_type": "markdown", - "id": "5f945b64", + "id": "03cc0ca0", "metadata": { "editable": true }, @@ -4595,8 +4562,8 @@ }, { "cell_type": "code", - "execution_count": 47, - "id": "f9578f4e", + "execution_count": 46, + "id": "9b1b9378", "metadata": { "collapsed": false, "editable": true @@ -4635,16 +4602,16 @@ "# We split the data in test and training data\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n", "\n", - "# matrix inversion to find beta\n", - "OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", - "print(OLSbeta)\n", + "# matrix inversion to find theta\n", + "OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", + "print(OLStheta)\n", "# and then make the prediction\n", - "ytildeOLS = X_train @ OLSbeta\n", + "ytildeOLS = X_train @ OLStheta\n", "print(\"Training R2 for OLS\")\n", "print(R2(y_train,ytildeOLS))\n", "print(\"Training MSE for OLS\")\n", "print(MSE(y_train,ytildeOLS))\n", - "ypredictOLS = X_test @ OLSbeta\n", + "ypredictOLS = X_test @ OLStheta\n", "print(\"Test R2 for OLS\")\n", "print(R2(y_test,ypredictOLS))\n", "print(\"Test MSE OLS\")\n", @@ -4661,10 +4628,10 @@ "lambdas = np.logspace(-4, 1, nlambdas)\n", "for i in range(nlambdas):\n", " lmb = lambdas[i]\n", - " OwnRidgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n", + " OwnRidgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n", " # and then make the prediction\n", - " OwnytildeRidge = X_train @ OwnRidgebeta\n", - " OwnypredictRidge = X_test @ OwnRidgebeta\n", + " OwnytildeRidge = X_train @ OwnRidgetheta\n", + " OwnypredictRidge = X_test @ OwnRidgetheta\n", " OwnMSEPredict[i] = MSE(y_test,OwnypredictRidge)\n", " OwnMSETrain[i] = MSE(y_train,OwnytildeRidge)\n", " # Make the fit using Ridge from Sklearn\n", @@ -4688,7 +4655,7 @@ }, { "cell_type": "markdown", - "id": "88be64bc", + "id": "341e9820", "metadata": { "editable": true }, @@ -4698,7 +4665,7 @@ }, { "cell_type": "markdown", - "id": "3a0e09e7", + "id": "6feb372f", "metadata": { "editable": true }, @@ -4720,7 +4687,7 @@ }, { "cell_type": "markdown", - "id": "fd41f7f2", + "id": "3c12062d", "metadata": { "editable": true }, @@ -4732,7 +4699,7 @@ }, { "cell_type": "markdown", - "id": "49d2bdba", + "id": "edfcd8ff", "metadata": { "editable": true }, @@ -4742,7 +4709,7 @@ }, { "cell_type": "markdown", - "id": "fd5a139b", + "id": "e8689eb2", "metadata": { "editable": true }, @@ -4754,7 +4721,7 @@ }, { "cell_type": "markdown", - "id": "af22b4d2", + "id": "62145d09", "metadata": { "editable": true }, @@ -4764,7 +4731,7 @@ }, { "cell_type": "markdown", - "id": "b120000f", + "id": "ec992c25", "metadata": { "editable": true }, @@ -4776,7 +4743,7 @@ }, { "cell_type": "markdown", - "id": "0e7df31e", + "id": "4a922090", "metadata": { "editable": true }, @@ -4791,7 +4758,7 @@ }, { "cell_type": "markdown", - "id": "faa1b4d0", + "id": "10f294a3", "metadata": { "editable": true }, @@ -4803,7 +4770,7 @@ }, { "cell_type": "markdown", - "id": "02f78802", + "id": "1dfa508a", "metadata": { "editable": true }, @@ -4813,7 +4780,7 @@ }, { "cell_type": "markdown", - "id": "e85b6015", + "id": "b5474b9f", "metadata": { "editable": true }, @@ -4825,7 +4792,7 @@ }, { "cell_type": "markdown", - "id": "12ae8696", + "id": "4dee1baf", "metadata": { "editable": true }, @@ -4835,7 +4802,7 @@ }, { "cell_type": "markdown", - "id": "d90de117", + "id": "9949ee71", "metadata": { "editable": true }, @@ -4847,7 +4814,7 @@ }, { "cell_type": "markdown", - "id": "15a6c139", + "id": "73ce2a98", "metadata": { "editable": true }, @@ -4857,7 +4824,7 @@ }, { "cell_type": "markdown", - "id": "f5e7aa62", + "id": "41e1008d", "metadata": { "editable": true }, @@ -4869,7 +4836,7 @@ }, { "cell_type": "markdown", - "id": "5a1f86bc", + "id": "597c4ebb", "metadata": { "editable": true }, @@ -4879,7 +4846,7 @@ }, { "cell_type": "markdown", - "id": "57018aa2", + "id": "e9163f17", "metadata": { "editable": true }, @@ -4891,7 +4858,7 @@ }, { "cell_type": "markdown", - "id": "6a9c70a8", + "id": "fc3d2fdb", "metadata": { "editable": true }, @@ -4901,7 +4868,7 @@ }, { "cell_type": "markdown", - "id": "75aaf0ec", + "id": "e7cec690", "metadata": { "editable": true }, @@ -4913,7 +4880,7 @@ }, { "cell_type": "markdown", - "id": "31a142ea", + "id": "1a74bac9", "metadata": { "editable": true }, @@ -4923,7 +4890,7 @@ }, { "cell_type": "markdown", - "id": "2552b361", + "id": "7087bc76", "metadata": { "editable": true }, @@ -4935,7 +4902,7 @@ }, { "cell_type": "markdown", - "id": "91b53dc5", + "id": "1e5a21fd", "metadata": { "editable": true }, @@ -4945,7 +4912,7 @@ }, { "cell_type": "markdown", - "id": "bf83b147", + "id": "624d40e0", "metadata": { "editable": true }, @@ -4957,7 +4924,7 @@ }, { "cell_type": "markdown", - "id": "0c8668a6", + "id": "d871f62f", "metadata": { "editable": true }, @@ -4967,7 +4934,7 @@ }, { "cell_type": "markdown", - "id": "89704701", + "id": "675aea37", "metadata": { "editable": true }, @@ -4979,7 +4946,7 @@ }, { "cell_type": "markdown", - "id": "0054bb58", + "id": "746b74e2", "metadata": { "editable": true }, @@ -4989,7 +4956,7 @@ }, { "cell_type": "markdown", - "id": "24f3ee5f", + "id": "6381d6ee", "metadata": { "editable": true }, @@ -5001,7 +4968,7 @@ }, { "cell_type": "markdown", - "id": "0827a002", + "id": "4331c20e", "metadata": { "editable": true }, diff --git a/doc/LectureNotes/_build/html/_sources/chapter2.ipynb b/doc/LectureNotes/_build/html/_sources/chapter2.ipynb index 31a6d7135..d663cda6a 100644 --- a/doc/LectureNotes/_build/html/_sources/chapter2.ipynb +++ b/doc/LectureNotes/_build/html/_sources/chapter2.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "74d0d498", + "id": "d5cb084b", "metadata": { "editable": true }, @@ -13,7 +13,7 @@ }, { "cell_type": "markdown", - "id": "acd544cd", + "id": "10ec5810", "metadata": { "editable": true }, @@ -23,7 +23,7 @@ }, { "cell_type": "markdown", - "id": "0acaa875", + "id": "607f5658", "metadata": { "editable": true }, @@ -32,48 +32,48 @@ "\n", "What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). \n", "\n", - "We have shown that in ordinary least squares (OLS) the optimal parameters $\\beta$ are given by" + "We have shown that in ordinary least squares (OLS) the optimal parameters $\\theta$ are given by" ] }, { "cell_type": "markdown", - "id": "0924b488", + "id": "d5cf7dd7", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "b7307eb7", + "id": "4bccceb2", "metadata": { "editable": true }, "source": [ - "The **hat** over $\\boldsymbol{\\beta}$ means we have the optimal parameters after minimization of the cost function.\n", + "The **hat** over $\\boldsymbol{\\theta}$ means we have the optimal parameters after minimization of the cost function.\n", "\n", "This means that our best model is defined as" ] }, { "cell_type": "markdown", - "id": "8096b1f0", + "id": "a51f281a", "metadata": { "editable": true }, "source": [ "$$\n", - "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "d64adaf2", + "id": "66dd8fd1", "metadata": { "editable": true }, @@ -83,7 +83,7 @@ }, { "cell_type": "markdown", - "id": "78b35483", + "id": "a4914b87", "metadata": { "editable": true }, @@ -95,7 +95,7 @@ }, { "cell_type": "markdown", - "id": "f4a3ff68", + "id": "12e868a7", "metadata": { "editable": true }, @@ -105,19 +105,19 @@ }, { "cell_type": "markdown", - "id": "5704d260", + "id": "a76003c4", "metadata": { "editable": true }, "source": [ "$$\n", - "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{A}\\boldsymbol{y}.\n", + "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}} = \\boldsymbol{A}\\boldsymbol{y}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "12fa775a", + "id": "f894a9be", "metadata": { "editable": true }, @@ -130,7 +130,7 @@ }, { "cell_type": "markdown", - "id": "a694774d", + "id": "1c7cf524", "metadata": { "editable": true }, @@ -142,7 +142,7 @@ }, { "cell_type": "markdown", - "id": "e0775433", + "id": "de9e3352", "metadata": { "editable": true }, @@ -154,7 +154,7 @@ }, { "cell_type": "markdown", - "id": "ec16ae7d", + "id": "4ece2993", "metadata": { "editable": true }, @@ -166,7 +166,7 @@ }, { "cell_type": "markdown", - "id": "258fbf6a", + "id": "f525eb65", "metadata": { "editable": true }, @@ -176,7 +176,7 @@ }, { "cell_type": "markdown", - "id": "05ea3605", + "id": "7d2849f0", "metadata": { "editable": true }, @@ -188,7 +188,7 @@ }, { "cell_type": "markdown", - "id": "db5ed2d4", + "id": "b214cf3a", "metadata": { "editable": true }, @@ -198,7 +198,7 @@ }, { "cell_type": "markdown", - "id": "e99ad8e8", + "id": "d566c8b4", "metadata": { "editable": true }, @@ -210,7 +210,7 @@ }, { "cell_type": "markdown", - "id": "1cc7938a", + "id": "cac5aeb2", "metadata": { "editable": true }, @@ -220,7 +220,7 @@ }, { "cell_type": "markdown", - "id": "0e3fc9d5", + "id": "b349c4ef", "metadata": { "editable": true }, @@ -267,7 +267,7 @@ }, { "cell_type": "markdown", - "id": "6d2bc570", + "id": "b5c5014d", "metadata": { "editable": true }, @@ -290,7 +290,7 @@ }, { "cell_type": "markdown", - "id": "fdda638f", + "id": "2b07f548", "metadata": { "editable": true }, @@ -307,7 +307,7 @@ }, { "cell_type": "markdown", - "id": "29a3a620", + "id": "f8223e8b", "metadata": { "editable": true }, @@ -326,7 +326,7 @@ }, { "cell_type": "markdown", - "id": "3278b038", + "id": "c1c28b92", "metadata": { "editable": true }, @@ -339,7 +339,7 @@ }, { "cell_type": "markdown", - "id": "7b66f623", + "id": "8cebf32c", "metadata": { "editable": true }, @@ -349,7 +349,7 @@ "\n", "$$\n", "\\begin{equation}\n", - "\\boldsymbol{\\beta} = (\\boldsymbol{X}^{T} \\boldsymbol{X})^{-1} \\boldsymbol{X}^{T} \\boldsymbol{y},\n", + "\\boldsymbol{\\theta} = (\\boldsymbol{X}^{T} \\boldsymbol{X})^{-1} \\boldsymbol{X}^{T} \\boldsymbol{y},\n", "\\label{_auto1} \\tag{1}\n", "\\end{equation}\n", "$$" @@ -357,23 +357,23 @@ }, { "cell_type": "markdown", - "id": "c97ae6c5", + "id": "4c2f125e", "metadata": { "editable": true }, "source": [ "has linearly dependent column vectors, we will not be able to compute the inverse\n", - "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\beta_i$. \n", + "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\theta_i$. \n", "The estimators are only well-defined if $(\\boldsymbol{X}^{T}\\boldsymbol{X})$ can be inverted. \n", "This is more likely to happen when the matrix $\\boldsymbol{X}$ is high-dimensional. In this case it is likely to encounter a situation where \n", - "the regression parameters $\\beta_i$ cannot be estimated.\n", + "the regression parameters $\\theta_i$ cannot be estimated.\n", "\n", "A cheap *ad hoc* approach is simply to add a small diagonal component to the matrix to invert, that is we change" ] }, { "cell_type": "markdown", - "id": "af0bfc59", + "id": "28b01b73", "metadata": { "editable": true }, @@ -385,7 +385,7 @@ }, { "cell_type": "markdown", - "id": "6469bfe8", + "id": "df72f01b", "metadata": { "editable": true }, @@ -395,7 +395,7 @@ }, { "cell_type": "markdown", - "id": "75d45f30", + "id": "62fc7dce", "metadata": { "editable": true }, @@ -410,7 +410,7 @@ }, { "cell_type": "markdown", - "id": "28b4fac2", + "id": "5b0639a7", "metadata": { "editable": true }, @@ -422,7 +422,7 @@ }, { "cell_type": "markdown", - "id": "085b2de7", + "id": "397cb734", "metadata": { "editable": true }, @@ -432,7 +432,7 @@ }, { "cell_type": "markdown", - "id": "1fb42fdc", + "id": "05ac3f20", "metadata": { "editable": true }, @@ -444,7 +444,7 @@ }, { "cell_type": "markdown", - "id": "0f7b8c0d", + "id": "ee59ca38", "metadata": { "editable": true }, @@ -454,7 +454,7 @@ }, { "cell_type": "markdown", - "id": "057f3ed2", + "id": "6c60f584", "metadata": { "editable": true }, @@ -466,7 +466,7 @@ }, { "cell_type": "markdown", - "id": "628031b9", + "id": "b7031193", "metadata": { "editable": true }, @@ -478,7 +478,7 @@ }, { "cell_type": "markdown", - "id": "3177dcfe", + "id": "4c0dce45", "metadata": { "editable": true }, @@ -493,7 +493,7 @@ }, { "cell_type": "markdown", - "id": "a392a54d", + "id": "ce330467", "metadata": { "editable": true }, @@ -514,7 +514,7 @@ }, { "cell_type": "markdown", - "id": "3496079e", + "id": "e42d67ce", "metadata": { "editable": true }, @@ -526,7 +526,7 @@ }, { "cell_type": "markdown", - "id": "77b61c68", + "id": "d43e350b", "metadata": { "editable": true }, @@ -536,7 +536,7 @@ }, { "cell_type": "markdown", - "id": "dbb84089", + "id": "4ad05081", "metadata": { "editable": true }, @@ -548,7 +548,7 @@ }, { "cell_type": "markdown", - "id": "04fc9eec", + "id": "7831a051", "metadata": { "editable": true }, @@ -594,7 +594,7 @@ }, { "cell_type": "markdown", - "id": "4f1e3bb4", + "id": "706c9e36", "metadata": { "editable": true }, @@ -605,7 +605,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "a39cbeb0", + "id": "9605f17b", "metadata": { "collapsed": false, "editable": true @@ -645,7 +645,7 @@ }, { "cell_type": "markdown", - "id": "dca36481", + "id": "d6dc015b", "metadata": { "editable": true }, @@ -675,7 +675,7 @@ }, { "cell_type": "markdown", - "id": "b9b3c7db", + "id": "2b38c905", "metadata": { "editable": true }, @@ -689,7 +689,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "6aa5981e", + "id": "3b82ddeb", "metadata": { "collapsed": false, "editable": true @@ -701,7 +701,7 @@ }, { "cell_type": "markdown", - "id": "ae58132b", + "id": "8bdcf281", "metadata": { "editable": true }, @@ -712,7 +712,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "761d48f5", + "id": "8090ac1c", "metadata": { "collapsed": false, "editable": true @@ -751,7 +751,7 @@ }, { "cell_type": "markdown", - "id": "6297c4ec", + "id": "35ee1cc9", "metadata": { "editable": true }, @@ -769,7 +769,7 @@ }, { "cell_type": "markdown", - "id": "2bb1f4b8", + "id": "13c4c73e", "metadata": { "editable": true }, @@ -781,7 +781,7 @@ }, { "cell_type": "markdown", - "id": "832a6404", + "id": "392f8342", "metadata": { "editable": true }, @@ -792,7 +792,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "d393c8c7", + "id": "3d64803c", "metadata": { "collapsed": false, "editable": true @@ -826,7 +826,7 @@ }, { "cell_type": "markdown", - "id": "3d0570c2", + "id": "a48cdbe7", "metadata": { "editable": true }, @@ -836,7 +836,7 @@ }, { "cell_type": "markdown", - "id": "071ff3b6", + "id": "22f55640", "metadata": { "editable": true }, @@ -850,7 +850,7 @@ }, { "cell_type": "markdown", - "id": "113247f9", + "id": "88772ea4", "metadata": { "editable": true }, @@ -869,7 +869,7 @@ }, { "cell_type": "markdown", - "id": "05603c38", + "id": "4f144230", "metadata": { "editable": true }, @@ -879,7 +879,7 @@ }, { "cell_type": "markdown", - "id": "8848e319", + "id": "c1a77d69", "metadata": { "editable": true }, @@ -891,7 +891,7 @@ }, { "cell_type": "markdown", - "id": "5b4146e5", + "id": "c77977a5", "metadata": { "editable": true }, @@ -905,7 +905,7 @@ }, { "cell_type": "markdown", - "id": "43701d21", + "id": "b3bac40c", "metadata": { "editable": true }, @@ -917,7 +917,7 @@ }, { "cell_type": "markdown", - "id": "7cdca00d", + "id": "a5803728", "metadata": { "editable": true }, @@ -929,7 +929,7 @@ }, { "cell_type": "markdown", - "id": "add83821", + "id": "21edee39", "metadata": { "editable": true }, @@ -946,7 +946,7 @@ }, { "cell_type": "markdown", - "id": "3746dd8d", + "id": "3e23d609", "metadata": { "editable": true }, @@ -956,7 +956,7 @@ }, { "cell_type": "markdown", - "id": "ae591c49", + "id": "64ec1fc5", "metadata": { "editable": true }, @@ -972,7 +972,7 @@ }, { "cell_type": "markdown", - "id": "63009724", + "id": "56d47d93", "metadata": { "editable": true }, @@ -982,7 +982,7 @@ }, { "cell_type": "markdown", - "id": "c674e8da", + "id": "2fda48ae", "metadata": { "editable": true }, @@ -998,7 +998,7 @@ }, { "cell_type": "markdown", - "id": "9f60659a", + "id": "824ffed2", "metadata": { "editable": true }, @@ -1008,7 +1008,7 @@ }, { "cell_type": "markdown", - "id": "27e17584", + "id": "0ac7a41e", "metadata": { "editable": true }, @@ -1024,7 +1024,7 @@ }, { "cell_type": "markdown", - "id": "3efc62c4", + "id": "8d0dd964", "metadata": { "editable": true }, @@ -1034,7 +1034,7 @@ }, { "cell_type": "markdown", - "id": "195d30c4", + "id": "30282069", "metadata": { "editable": true }, @@ -1051,7 +1051,7 @@ }, { "cell_type": "markdown", - "id": "a00d95e8", + "id": "064c0e6b", "metadata": { "editable": true }, @@ -1065,7 +1065,7 @@ }, { "cell_type": "markdown", - "id": "985428ec", + "id": "273056c4", "metadata": { "editable": true }, @@ -1077,7 +1077,7 @@ }, { "cell_type": "markdown", - "id": "0affd014", + "id": "e38034e4", "metadata": { "editable": true }, @@ -1087,7 +1087,7 @@ }, { "cell_type": "markdown", - "id": "146dcd49", + "id": "8cc943db", "metadata": { "editable": true }, @@ -1099,7 +1099,7 @@ }, { "cell_type": "markdown", - "id": "63bc8186", + "id": "9dfc2966", "metadata": { "editable": true }, @@ -1111,7 +1111,7 @@ }, { "cell_type": "markdown", - "id": "4bea8a7d", + "id": "8ca54207", "metadata": { "editable": true }, @@ -1123,7 +1123,7 @@ }, { "cell_type": "markdown", - "id": "5321e8fc", + "id": "a986b780", "metadata": { "editable": true }, @@ -1133,7 +1133,7 @@ }, { "cell_type": "markdown", - "id": "c16d0109", + "id": "2c31d500", "metadata": { "editable": true }, @@ -1145,7 +1145,7 @@ }, { "cell_type": "markdown", - "id": "4ca7bc79", + "id": "9e4536a0", "metadata": { "editable": true }, @@ -1155,7 +1155,7 @@ }, { "cell_type": "markdown", - "id": "795969aa", + "id": "92636687", "metadata": { "editable": true }, @@ -1167,7 +1167,7 @@ }, { "cell_type": "markdown", - "id": "36fd11ec", + "id": "a665d593", "metadata": { "editable": true }, @@ -1177,7 +1177,7 @@ }, { "cell_type": "markdown", - "id": "8b59e361", + "id": "769bb394", "metadata": { "editable": true }, @@ -1189,7 +1189,7 @@ }, { "cell_type": "markdown", - "id": "4212728f", + "id": "baf742bb", "metadata": { "editable": true }, @@ -1201,7 +1201,7 @@ }, { "cell_type": "markdown", - "id": "8d6940eb", + "id": "7fd2be6b", "metadata": { "editable": true }, @@ -1211,7 +1211,7 @@ }, { "cell_type": "markdown", - "id": "eabfdb9c", + "id": "e488ea83", "metadata": { "editable": true }, @@ -1223,7 +1223,7 @@ }, { "cell_type": "markdown", - "id": "bbc85eab", + "id": "76ac57c1", "metadata": { "editable": true }, @@ -1234,7 +1234,7 @@ }, { "cell_type": "markdown", - "id": "f5b1bf52", + "id": "b789d545", "metadata": { "editable": true }, @@ -1246,7 +1246,7 @@ }, { "cell_type": "markdown", - "id": "efcc256a", + "id": "fd7008f9", "metadata": { "editable": true }, @@ -1256,7 +1256,7 @@ }, { "cell_type": "markdown", - "id": "0650a070", + "id": "9d37f0bf", "metadata": { "editable": true }, @@ -1268,7 +1268,7 @@ }, { "cell_type": "markdown", - "id": "c168d425", + "id": "0fd2b28c", "metadata": { "editable": true }, @@ -1278,7 +1278,7 @@ }, { "cell_type": "markdown", - "id": "30fb2230", + "id": "98432443", "metadata": { "editable": true }, @@ -1290,7 +1290,7 @@ }, { "cell_type": "markdown", - "id": "d46b1fd5", + "id": "29174a30", "metadata": { "editable": true }, @@ -1301,7 +1301,7 @@ }, { "cell_type": "markdown", - "id": "16a93afd", + "id": "d07b5e5c", "metadata": { "editable": true }, @@ -1313,7 +1313,7 @@ }, { "cell_type": "markdown", - "id": "80e47710", + "id": "21892e7e", "metadata": { "editable": true }, @@ -1331,7 +1331,7 @@ }, { "cell_type": "markdown", - "id": "1dc0f98b", + "id": "a4ed7dd4", "metadata": { "editable": true }, @@ -1347,19 +1347,19 @@ }, { "cell_type": "markdown", - "id": "c5f9ce8b", + "id": "c7aafbc8", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial^2 C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n", + "\\frac{\\partial^2 C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}^T\\partial \\boldsymbol{\\theta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "c803fdb7", + "id": "f80de4f9", "metadata": { "editable": true }, @@ -1371,7 +1371,7 @@ }, { "cell_type": "markdown", - "id": "eef3c89e", + "id": "8ec45bf8", "metadata": { "editable": true }, @@ -1383,7 +1383,7 @@ }, { "cell_type": "markdown", - "id": "e886f303", + "id": "4554f86d", "metadata": { "editable": true }, @@ -1402,7 +1402,7 @@ }, { "cell_type": "markdown", - "id": "efcb2b9e", + "id": "c5c46053", "metadata": { "editable": true }, @@ -1416,7 +1416,7 @@ }, { "cell_type": "markdown", - "id": "caf380d7", + "id": "0d367725", "metadata": { "editable": true }, @@ -1426,7 +1426,7 @@ }, { "cell_type": "markdown", - "id": "9d676665", + "id": "ca783893", "metadata": { "editable": true }, @@ -1438,7 +1438,7 @@ }, { "cell_type": "markdown", - "id": "301b0d53", + "id": "81dacef8", "metadata": { "editable": true }, @@ -1448,7 +1448,7 @@ }, { "cell_type": "markdown", - "id": "7c134c39", + "id": "10790e67", "metadata": { "editable": true }, @@ -1460,7 +1460,7 @@ }, { "cell_type": "markdown", - "id": "12aec852", + "id": "0df66dcf", "metadata": { "editable": true }, @@ -1470,7 +1470,7 @@ }, { "cell_type": "markdown", - "id": "54a5de6b", + "id": "021fd105", "metadata": { "editable": true }, @@ -1484,7 +1484,7 @@ }, { "cell_type": "markdown", - "id": "3edc1dd1", + "id": "9cee5e0c", "metadata": { "editable": true }, @@ -1507,7 +1507,7 @@ }, { "cell_type": "markdown", - "id": "9d2d7b4e", + "id": "30ea5811", "metadata": { "editable": true }, @@ -1519,7 +1519,7 @@ }, { "cell_type": "markdown", - "id": "dcee4258", + "id": "a129d192", "metadata": { "editable": true }, @@ -1532,7 +1532,7 @@ }, { "cell_type": "markdown", - "id": "25100476", + "id": "6d5c0a69", "metadata": { "editable": true }, @@ -1546,7 +1546,7 @@ }, { "cell_type": "markdown", - "id": "5c46fdef", + "id": "79a613b9", "metadata": { "editable": true }, @@ -1559,7 +1559,7 @@ }, { "cell_type": "markdown", - "id": "dcfa6d43", + "id": "2d0dabde", "metadata": { "editable": true }, @@ -1578,7 +1578,7 @@ }, { "cell_type": "markdown", - "id": "89dd5e4e", + "id": "2731cd83", "metadata": { "editable": true }, @@ -1590,7 +1590,7 @@ }, { "cell_type": "markdown", - "id": "7edacd6c", + "id": "82de2dcc", "metadata": { "editable": true }, @@ -1602,7 +1602,7 @@ }, { "cell_type": "markdown", - "id": "94e8929f", + "id": "e8bf54be", "metadata": { "editable": true }, @@ -1612,7 +1612,7 @@ }, { "cell_type": "markdown", - "id": "3f10861a", + "id": "a0597d71", "metadata": { "editable": true }, @@ -1624,7 +1624,7 @@ }, { "cell_type": "markdown", - "id": "173fbc84", + "id": "d74275c9", "metadata": { "editable": true }, @@ -1637,7 +1637,7 @@ }, { "cell_type": "markdown", - "id": "0913dc51", + "id": "df0589bd", "metadata": { "editable": true }, @@ -1656,7 +1656,7 @@ }, { "cell_type": "markdown", - "id": "70346587", + "id": "c23bd593", "metadata": { "editable": true }, @@ -1666,7 +1666,7 @@ }, { "cell_type": "markdown", - "id": "86feec9f", + "id": "8806c741", "metadata": { "editable": true }, @@ -1685,7 +1685,7 @@ }, { "cell_type": "markdown", - "id": "25ed76e9", + "id": "ae1e6181", "metadata": { "editable": true }, @@ -1701,7 +1701,7 @@ }, { "cell_type": "markdown", - "id": "cea26675", + "id": "9a4af97a", "metadata": { "editable": true }, @@ -1715,7 +1715,7 @@ }, { "cell_type": "markdown", - "id": "1fa1423f", + "id": "607da328", "metadata": { "editable": true }, @@ -1730,7 +1730,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "f277b044", + "id": "afa9a0d9", "metadata": { "collapsed": false, "editable": true @@ -1751,7 +1751,7 @@ }, { "cell_type": "markdown", - "id": "02205795", + "id": "8acdd66a", "metadata": { "editable": true }, @@ -1766,7 +1766,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "6c182a7d", + "id": "4886841b", "metadata": { "collapsed": false, "editable": true @@ -1798,7 +1798,7 @@ }, { "cell_type": "markdown", - "id": "16ff9454", + "id": "90b67d9e", "metadata": { "editable": true }, @@ -1815,7 +1815,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "903635fb", + "id": "2dc76e27", "metadata": { "collapsed": false, "editable": true @@ -1840,7 +1840,7 @@ }, { "cell_type": "markdown", - "id": "034c38ef", + "id": "3b9c94f0", "metadata": { "editable": true }, @@ -1851,7 +1851,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "91afb8cb", + "id": "f12e4764", "metadata": { "collapsed": false, "editable": true @@ -1877,7 +1877,7 @@ "\t\ty = np.ravel(y)\n", "\n", "\tN = len(x)\n", - "\tl = int((n+1)*(n+2)/2)\t\t# Number of elements in beta\n", + "\tl = int((n+1)*(n+2)/2)\t\t# Number of elements in theta\n", "\tX = np.ones((N,l))\n", "\n", "\tfor i in range(1,n+1):\n", @@ -1905,7 +1905,7 @@ }, { "cell_type": "markdown", - "id": "9d8768f8", + "id": "6d8aa859", "metadata": { "editable": true }, @@ -1924,7 +1924,7 @@ }, { "cell_type": "markdown", - "id": "a7de38a0", + "id": "d4715f97", "metadata": { "editable": true }, @@ -1936,7 +1936,7 @@ }, { "cell_type": "markdown", - "id": "4855248a", + "id": "1800f79e", "metadata": { "editable": true }, @@ -1946,7 +1946,7 @@ }, { "cell_type": "markdown", - "id": "3a150f73", + "id": "dac62d99", "metadata": { "editable": true }, @@ -1963,7 +1963,7 @@ }, { "cell_type": "markdown", - "id": "6ab3f1f7", + "id": "97c7d1d3", "metadata": { "editable": true }, @@ -1973,7 +1973,7 @@ }, { "cell_type": "markdown", - "id": "ae35f475", + "id": "5645cf35", "metadata": { "editable": true }, @@ -1988,7 +1988,7 @@ }, { "cell_type": "markdown", - "id": "9f6b4b6b", + "id": "ae8afe59", "metadata": { "editable": true }, @@ -1998,7 +1998,7 @@ }, { "cell_type": "markdown", - "id": "af0c59a3", + "id": "3bae9bfc", "metadata": { "editable": true }, @@ -2012,7 +2012,7 @@ }, { "cell_type": "markdown", - "id": "824ae63b", + "id": "b60c8bb6", "metadata": { "editable": true }, @@ -2026,7 +2026,7 @@ }, { "cell_type": "markdown", - "id": "5b480160", + "id": "d8c6fa53", "metadata": { "editable": true }, @@ -2038,7 +2038,7 @@ }, { "cell_type": "markdown", - "id": "506a98da", + "id": "074c5673", "metadata": { "editable": true }, @@ -2050,7 +2050,7 @@ }, { "cell_type": "markdown", - "id": "72b1e665", + "id": "8320630a", "metadata": { "editable": true }, @@ -2060,7 +2060,7 @@ }, { "cell_type": "markdown", - "id": "6ea9123f", + "id": "2eebcb57", "metadata": { "editable": true }, @@ -2072,7 +2072,7 @@ }, { "cell_type": "markdown", - "id": "ab471475", + "id": "7a7c1b9c", "metadata": { "editable": true }, @@ -2082,7 +2082,7 @@ }, { "cell_type": "markdown", - "id": "ff364443", + "id": "c89ccbc8", "metadata": { "editable": true }, @@ -2099,7 +2099,7 @@ }, { "cell_type": "markdown", - "id": "4619ec8f", + "id": "b4fc87f6", "metadata": { "editable": true }, @@ -2109,7 +2109,7 @@ }, { "cell_type": "markdown", - "id": "529d5ed0", + "id": "15d7da17", "metadata": { "editable": true }, @@ -2121,7 +2121,7 @@ }, { "cell_type": "markdown", - "id": "762e07ea", + "id": "ee6e0550", "metadata": { "editable": true }, @@ -2131,7 +2131,7 @@ }, { "cell_type": "markdown", - "id": "f45c2ce1", + "id": "2789c2b4", "metadata": { "editable": true }, @@ -2143,7 +2143,7 @@ }, { "cell_type": "markdown", - "id": "35835939", + "id": "1b0a4cce", "metadata": { "editable": true }, @@ -2155,7 +2155,7 @@ }, { "cell_type": "markdown", - "id": "901c3505", + "id": "ad73966d", "metadata": { "editable": true }, @@ -2167,7 +2167,7 @@ }, { "cell_type": "markdown", - "id": "63486657", + "id": "107fdb0a", "metadata": { "editable": true }, @@ -2189,7 +2189,7 @@ }, { "cell_type": "markdown", - "id": "b8cb7b04", + "id": "8d070a5b", "metadata": { "editable": true }, @@ -2201,7 +2201,7 @@ }, { "cell_type": "markdown", - "id": "8962eeb3", + "id": "356bded4", "metadata": { "editable": true }, @@ -2218,7 +2218,7 @@ }, { "cell_type": "markdown", - "id": "ff83bf5a", + "id": "2b232f6c", "metadata": { "editable": true }, @@ -2230,7 +2230,7 @@ }, { "cell_type": "markdown", - "id": "67c405ee", + "id": "9ec41351", "metadata": { "editable": true }, @@ -2240,7 +2240,7 @@ }, { "cell_type": "markdown", - "id": "d2279b06", + "id": "5fd89419", "metadata": { "editable": true }, @@ -2252,7 +2252,7 @@ }, { "cell_type": "markdown", - "id": "c08bd913", + "id": "22a2cb66", "metadata": { "editable": true }, @@ -2262,7 +2262,7 @@ }, { "cell_type": "markdown", - "id": "585fcec4", + "id": "ac6721a2", "metadata": { "editable": true }, @@ -2274,7 +2274,7 @@ }, { "cell_type": "markdown", - "id": "c5ee7c18", + "id": "da33b69e", "metadata": { "editable": true }, @@ -2284,7 +2284,7 @@ }, { "cell_type": "markdown", - "id": "250d6a7b", + "id": "44110c4b", "metadata": { "editable": true }, @@ -2296,7 +2296,7 @@ }, { "cell_type": "markdown", - "id": "26dca3eb", + "id": "de0ee363", "metadata": { "editable": true }, @@ -2313,7 +2313,7 @@ }, { "cell_type": "markdown", - "id": "43ae52cf", + "id": "3c2b4cb7", "metadata": { "editable": true }, @@ -2326,19 +2326,19 @@ }, { "cell_type": "markdown", - "id": "fa5c2da2", + "id": "b8979ebf", "metadata": { "editable": true }, "source": [ "$$\n", - "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "ef125654", + "id": "f635713d", "metadata": { "editable": true }, @@ -2348,20 +2348,20 @@ }, { "cell_type": "markdown", - "id": "3e398c3d", + "id": "e335573e", "metadata": { "editable": true }, "source": [ "$$\n", - "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", - "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2,\n", "$$" ] }, { "cell_type": "markdown", - "id": "6e4285d1", + "id": "197171c0", "metadata": { "editable": true }, @@ -2371,7 +2371,7 @@ }, { "cell_type": "markdown", - "id": "c1d8fde6", + "id": "494553da", "metadata": { "editable": true }, @@ -2383,57 +2383,57 @@ }, { "cell_type": "markdown", - "id": "5b082cd2", + "id": "d131f255", "metadata": { "editable": true }, "source": [ "By minimizing the above equation with respect to the parameters\n", - "$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n", - "parameters $\\boldsymbol{\\beta}$. We can add a regularization parameter $\\lambda$ by\n", + "$\\boldsymbol{\\theta}$ we could then obtain an analytical expression for the\n", + "parameters $\\boldsymbol{\\theta}$. We can add a regularization parameter $\\lambda$ by\n", "defining a new cost function to be optimized, that is" ] }, { "cell_type": "markdown", - "id": "306b6062", + "id": "3992861b", "metadata": { "editable": true }, "source": [ "$$\n", - "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", - "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\n", "$$" ] }, { "cell_type": "markdown", - "id": "f9c22f99", + "id": "e210e429", "metadata": { "editable": true }, "source": [ "which leads to the Ridge regression minimization problem where we\n", - "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n", + "require that $\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\\le t$, where $t$ is\n", "a finite number larger than zero. By defining" ] }, { "cell_type": "markdown", - "id": "8fa998f9", + "id": "960bf071", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n", + "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1,\n", "$$" ] }, { "cell_type": "markdown", - "id": "954b3b63", + "id": "69ff9b18", "metadata": { "editable": true }, @@ -2443,20 +2443,20 @@ }, { "cell_type": "markdown", - "id": "40d226a6", + "id": "3b619870", "metadata": { "editable": true }, "source": [ "$$\n", - "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", - "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1\n", "$$" ] }, { "cell_type": "markdown", - "id": "571654e9", + "id": "020995b7", "metadata": { "editable": true }, @@ -2468,7 +2468,7 @@ }, { "cell_type": "markdown", - "id": "4b96f90f", + "id": "dc41832a", "metadata": { "editable": true }, @@ -2480,7 +2480,7 @@ }, { "cell_type": "markdown", - "id": "6af80b33", + "id": "e4a20e48", "metadata": { "editable": true }, @@ -2490,25 +2490,25 @@ }, { "cell_type": "markdown", - "id": "ed4e7687", + "id": "03ea4d0d", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", + "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\boldsymbol{\\theta}^T\\boldsymbol{\\theta},\n", "$$" ] }, { "cell_type": "markdown", - "id": "2eb319cc", + "id": "6734e20a", "metadata": { "editable": true }, "source": [ "and \n", - "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", + "taking the derivatives with respect to $\\boldsymbol{\\theta}$ we obtain then\n", "a slightly modified matrix inversion problem which for finite values\n", "of $\\lambda$ does not suffer from singularity problems. We obtain\n", "the optimal parameters" @@ -2516,19 +2516,19 @@ }, { "cell_type": "markdown", - "id": "8528934c", + "id": "502e8acf", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", + "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", "$$" ] }, { "cell_type": "markdown", - "id": "22a7e810", + "id": "fcb7eeca", "metadata": { "editable": true }, @@ -2538,19 +2538,19 @@ }, { "cell_type": "markdown", - "id": "01d9c1c9", + "id": "f9789400", "metadata": { "editable": true }, "source": [ "$$\n", - "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n", + "\\sum_{i=0}^{p-1} \\theta_i^2 \\leq t,\n", "$$" ] }, { "cell_type": "markdown", - "id": "5d910fd1", + "id": "cce5edcd", "metadata": { "editable": true }, @@ -2562,19 +2562,19 @@ }, { "cell_type": "markdown", - "id": "b8bbfa39", + "id": "8b0e4f12", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", + "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", "$$" ] }, { "cell_type": "markdown", - "id": "51ac6ef1", + "id": "c59da228", "metadata": { "editable": true }, @@ -2585,7 +2585,7 @@ "modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n", "particular for our discussion of the bias-variance tradeoff are rather\n", "interesting. We will see that for specific values of $\\lambda$, we may\n", - "even reduce the variance of the optimal parameters $\\boldsymbol{\\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n", + "even reduce the variance of the optimal parameters $\\boldsymbol{\\theta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n", "\n", "Using our insights about the SVD of the design matrix $\\boldsymbol{X}$ \n", "We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\\boldsymbol{U}$ as" @@ -2593,19 +2593,19 @@ }, { "cell_type": "markdown", - "id": "12ee6646", + "id": "96960524", "metadata": { "editable": true }, "source": [ "$$\n", - "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", + "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\theta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "22852ccf", + "id": "ef0014d1", "metadata": { "editable": true }, @@ -2615,19 +2615,19 @@ }, { "cell_type": "markdown", - "id": "68d87eac", + "id": "57ba3f2e", "metadata": { "editable": true }, "source": [ "$$\n", - "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", + "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\theta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", "$$" ] }, { "cell_type": "markdown", - "id": "0219a1a8", + "id": "86313c14", "metadata": { "editable": true }, @@ -2639,7 +2639,7 @@ }, { "cell_type": "markdown", - "id": "be1f339c", + "id": "c5c6f72d", "metadata": { "editable": true }, @@ -2651,7 +2651,7 @@ }, { "cell_type": "markdown", - "id": "7e1d5221", + "id": "4f780155", "metadata": { "editable": true }, @@ -2669,7 +2669,7 @@ }, { "cell_type": "markdown", - "id": "0b4bc87c", + "id": "08413d65", "metadata": { "editable": true }, @@ -2681,7 +2681,7 @@ }, { "cell_type": "markdown", - "id": "de7e0986", + "id": "10f8be87", "metadata": { "editable": true }, @@ -2691,19 +2691,19 @@ }, { "cell_type": "markdown", - "id": "0ae7dd26", + "id": "9d2b90ba", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n", + "\\boldsymbol{\\theta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n", "$$" ] }, { "cell_type": "markdown", - "id": "5caa1086", + "id": "367f8b0e", "metadata": { "editable": true }, @@ -2713,19 +2713,19 @@ }, { "cell_type": "markdown", - "id": "c737b14f", + "id": "cd4334f5", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n", + "\\boldsymbol{\\theta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\theta}^{\\mathrm{OLS}},\n", "$$" ] }, { "cell_type": "markdown", - "id": "60570cf0", + "id": "06f7db12", "metadata": { "editable": true }, @@ -2741,41 +2741,41 @@ }, { "cell_type": "markdown", - "id": "f0c2e386", + "id": "b81118eb", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", + "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_1,\n", "$$" ] }, { "cell_type": "markdown", - "id": "5f3e91d2", + "id": "7d2b22a9", "metadata": { "editable": true }, "source": [ - "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)" + "Taking the derivative with respect to $\\boldsymbol{\\theta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)" ] }, { "cell_type": "markdown", - "id": "66b637e8", + "id": "2648d39f", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{d \\vert \\beta\\vert}{d \\boldsymbol{\\beta}}=\\mathrm{sgn}(\\boldsymbol{\\beta})=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\-1 & \\beta < 0, \\end{array}\\right.\n", + "\\frac{d \\vert \\theta\\vert}{d \\boldsymbol{\\theta}}=\\mathrm{sgn}(\\boldsymbol{\\theta})=\\left\\{\\begin{array}{cc} 1 & \\theta > 0 \\\\-1 & \\theta < 0, \\end{array}\\right.\n", "$$" ] }, { "cell_type": "markdown", - "id": "4082c969", + "id": "ea227065", "metadata": { "editable": true }, @@ -2785,19 +2785,19 @@ }, { "cell_type": "markdown", - "id": "2c0b085b", + "id": "ce21fc5d", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n", + "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})+\\lambda sgn(\\boldsymbol{\\theta})=0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "bf355533", + "id": "f4d13288", "metadata": { "editable": true }, @@ -2807,19 +2807,19 @@ }, { "cell_type": "markdown", - "id": "928d6cb3", + "id": "116661fd", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta}+\\lambda sgn(\\boldsymbol{\\theta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "39a40fd9", + "id": "dc54e1a7", "metadata": { "editable": true }, @@ -2829,24 +2829,24 @@ "Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the\n", "diagonal. In this case we have an equal number of rows and columns $n=p$.\n", "\n", - "Our model approximation is just $\\tilde{\\boldsymbol{y}}=\\boldsymbol{\\beta}$ and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term $1/n$)" + "Our model approximation is just $\\tilde{\\boldsymbol{y}}=\\boldsymbol{\\theta}$ and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term $1/n$)" ] }, { "cell_type": "markdown", - "id": "5d891e5c", + "id": "87c27103", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2,\n", + "C(\\boldsymbol{\\theta})=\\sum_{i=0}^{p-1}(y_i-\\theta_i)^2,\n", "$$" ] }, { "cell_type": "markdown", - "id": "f9ea4d16", + "id": "530dada7", "metadata": { "editable": true }, @@ -2856,19 +2856,19 @@ }, { "cell_type": "markdown", - "id": "59b56446", + "id": "2e70789b", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\beta}_i^{\\mathrm{OLS}} = y_i.\n", + "\\hat{\\theta}_i^{\\mathrm{OLS}} = y_i.\n", "$$" ] }, { "cell_type": "markdown", - "id": "b178aedb", + "id": "a69b1fda", "metadata": { "editable": true }, @@ -2878,19 +2878,19 @@ }, { "cell_type": "markdown", - "id": "3c7c60e4", + "id": "6dbb9f75", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,\n", + "C(\\boldsymbol{\\theta})=\\sum_{i=0}^{p-1}(y_i-\\theta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\theta_i^2,\n", "$$" ] }, { "cell_type": "markdown", - "id": "2f5d3c17", + "id": "fe4b2167", "metadata": { "editable": true }, @@ -2900,19 +2900,19 @@ }, { "cell_type": "markdown", - "id": "2a612078", + "id": "c7ad140a", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\beta}_i^{\\mathrm{Ridge}} = \\frac{y_i}{1+\\lambda}.\n", + "\\hat{\\theta}_i^{\\mathrm{Ridge}} = \\frac{y_i}{1+\\lambda}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "dfb11d07", + "id": "6dab591b", "metadata": { "editable": true }, @@ -2922,19 +2922,19 @@ }, { "cell_type": "markdown", - "id": "9331f4a6", + "id": "bfa5b9f9", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\beta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\beta_i^2},\n", + "C(\\boldsymbol{\\theta})=\\sum_{i=0}^{p-1}(y_i-\\theta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\theta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\theta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\theta_i^2},\n", "$$" ] }, { "cell_type": "markdown", - "id": "b4661ad6", + "id": "3125ab9c", "metadata": { "editable": true }, @@ -2944,19 +2944,19 @@ }, { "cell_type": "markdown", - "id": "60adb333", + "id": "309aed47", "metadata": { "editable": true }, "source": [ "$$\n", - "-2\\sum_{i=0}^{p-1}(y_i-\\beta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\beta_i)}{\\vert\\beta_i\\vert}=0,\n", + "-2\\sum_{i=0}^{p-1}(y_i-\\theta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\theta_i)}{\\vert\\theta_i\\vert}=0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "d5fa48d0", + "id": "b863d5f9", "metadata": { "editable": true }, @@ -2966,13 +2966,13 @@ }, { "cell_type": "markdown", - "id": "bee441a6", + "id": "ffaed247", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n", + "\\hat{\\boldsymbol{\\theta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n", " y_i+\\frac{\\lambda}{2} &\\mathrm{if} & y_i< -\\frac{\\lambda}{2}\\\\\n", "\t\t\t\t\t\t\t 0 &\\mathrm{if} & \\vert y_i\\vert\\le \\frac{\\lambda}{2}\\end{array}\\right.\\\\.\n", "$$" @@ -2980,12 +2980,12 @@ }, { "cell_type": "markdown", - "id": "dc6929bb", + "id": "6b89a6c5", "metadata": { "editable": true }, "source": [ - "Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$.\n", + "Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\\theta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\theta_i$ as function of $\\lambda$.\n", "\n", "As another example, \n", "let us assume we have a data set with outputs/targets given by the vector" @@ -2993,7 +2993,7 @@ }, { "cell_type": "markdown", - "id": "954061f9", + "id": "adadb0c3", "metadata": { "editable": true }, @@ -3005,7 +3005,7 @@ }, { "cell_type": "markdown", - "id": "97efa82b", + "id": "63968be0", "metadata": { "editable": true }, @@ -3015,7 +3015,7 @@ }, { "cell_type": "markdown", - "id": "f2ed5f9b", + "id": "48e3a74e", "metadata": { "editable": true }, @@ -3027,31 +3027,31 @@ }, { "cell_type": "markdown", - "id": "1c424fc1", + "id": "5948f0d4", "metadata": { "editable": true }, "source": [ - "meaning that we have two features and two unknown parameters $\\beta_0$ and $\\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression.\n", + "meaning that we have two features and two unknown parameters $\\theta_0$ and $\\theta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression.\n", "\n", "For ordinary least squares (OLS) we know that the optimal solution is" ] }, { "cell_type": "markdown", - "id": "95467447", + "id": "55c9c8c1", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "5da0739a", + "id": "42fd5011", "metadata": { "editable": true }, @@ -3061,19 +3061,19 @@ }, { "cell_type": "markdown", - "id": "47e7e72f", + "id": "cf6a68ad", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n", + "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n", "$$" ] }, { "cell_type": "markdown", - "id": "974570c8", + "id": "cf208d17", "metadata": { "editable": true }, @@ -3085,19 +3085,19 @@ }, { "cell_type": "markdown", - "id": "dd9e2341", + "id": "60413571", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "70f08735", + "id": "8ff207f9", "metadata": { "editable": true }, @@ -3107,25 +3107,25 @@ }, { "cell_type": "markdown", - "id": "c0a08b76", + "id": "846f76e7", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n", + "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n", "$$" ] }, { "cell_type": "markdown", - "id": "1d8ed72a", + "id": "c3d07bb7", "metadata": { "editable": true }, "source": [ - "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2$ via the parameter $\\lambda$.\n", - "Let us for simplicity assume that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\beta$ and $\\lambda$.\n", + "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2$ via the parameter $\\lambda$.\n", + "Let us for simplicity assume that $\\theta_0^2+\\theta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\theta$ and $\\lambda$.\n", "\n", "To see this, let us write the cost function for Ridge regression. \n", "\n", @@ -3134,85 +3134,85 @@ }, { "cell_type": "markdown", - "id": "9d4ef6fd", + "id": "ec07f63c", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{X}\\boldsymbol{\\beta}=\\begin{bmatrix} 2\\beta_0 \\\\ \\beta_1 \\\\0 \\end{bmatrix},\n", + "\\boldsymbol{X}\\boldsymbol{\\theta}=\\begin{bmatrix} 2\\theta_0 \\\\ \\theta_1 \\\\0 \\end{bmatrix},\n", "$$" ] }, { "cell_type": "markdown", - "id": "3ba1fd07", + "id": "40bd6afb", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\beta_0^2+\\beta_1^2),\n", + "C(\\boldsymbol{\\theta})=(4-2\\theta_0)^2+(2-\\theta_1)^2+\\lambda(\\theta_0^2+\\theta_1^2),\n", "$$" ] }, { "cell_type": "markdown", - "id": "2c6a0484", + "id": "b1d4b933", "metadata": { "editable": true }, "source": [ - "and taking the derivative with respect to $\\beta_0$ we get" + "and taking the derivative with respect to $\\theta_0$ we get" ] }, { "cell_type": "markdown", - "id": "7f86e745", + "id": "63371364", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_0=\\frac{8}{4+\\lambda},\n", + "\\theta_0=\\frac{8}{4+\\lambda},\n", "$$" ] }, { "cell_type": "markdown", - "id": "46c8a1af", + "id": "71fb4eb4", "metadata": { "editable": true }, "source": [ - "and for $\\beta_1$ we obtain" + "and for $\\theta_1$ we obtain" ] }, { "cell_type": "markdown", - "id": "33d6b311", + "id": "cacea391", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_1=\\frac{2}{1+\\lambda},\n", + "\\theta_1=\\frac{2}{1+\\lambda},\n", "$$" ] }, { "cell_type": "markdown", - "id": "1e4eae3a", + "id": "a65f9b33", "metadata": { "editable": true }, "source": [ - "Using the constraint for $\\beta_0^2+\\beta_1^2=1$ we can constrain $\\lambda$ by solving" + "Using the constraint for $\\theta_0^2+\\theta_1^2=1$ we can constrain $\\lambda$ by solving" ] }, { "cell_type": "markdown", - "id": "7a68a399", + "id": "96471d24", "metadata": { "editable": true }, @@ -3224,44 +3224,44 @@ }, { "cell_type": "markdown", - "id": "89874fbd", + "id": "ff4a5368", "metadata": { "editable": true }, "source": [ - "which gives $\\lambda=4.571$ and $\\beta_0=0.933$ and $\\beta_1=0.359$.\n", + "which gives $\\lambda=4.571$ and $\\theta_0=0.933$ and $\\theta_1=0.359$.\n", "\n", - "For Lasso we need now, keeping a constraint on $\\vert\\beta_0\\vert+\\vert\\beta_1\\vert=1$, to take the derivative of the absolute values of $\\beta_0$\n", - "and $\\beta_1$. This gives us the following derivatives of the cost function" + "For Lasso we need now, keeping a constraint on $\\vert\\theta_0\\vert+\\vert\\theta_1\\vert=1$, to take the derivative of the absolute values of $\\theta_0$\n", + "and $\\theta_1$. This gives us the following derivatives of the cost function" ] }, { "cell_type": "markdown", - "id": "a849fabc", + "id": "cbaab2b3", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\vert\\beta_0\\vert+\\vert\\beta_1\\vert),\n", + "C(\\boldsymbol{\\theta})=(4-2\\theta_0)^2+(2-\\theta_1)^2+\\lambda(\\vert\\theta_0\\vert+\\vert\\theta_1\\vert),\n", "$$" ] }, { "cell_type": "markdown", - "id": "445f0b08", + "id": "8074c920", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-4(4-2\\beta_0)+\\lambda\\mathrm{sgn}(\\beta_0)=0,\n", + "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_0}=-4(4-2\\theta_0)+\\lambda\\mathrm{sgn}(\\theta_0)=0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "669385c5", + "id": "339ebabc", "metadata": { "editable": true }, @@ -3271,50 +3271,50 @@ }, { "cell_type": "markdown", - "id": "d5d14c30", + "id": "8803be3d", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}(\\beta_1)=0.\n", + "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_1}=-2(2-\\theta_1)+\\lambda\\mathrm{sgn}(\\theta_1)=0.\n", "$$" ] }, { "cell_type": "markdown", - "id": "1db05ce9", + "id": "a3af4674", "metadata": { "editable": true }, "source": [ - "We have now four cases to solve besides the trivial cases $\\beta_0$ and/or $\\beta_1$ are zero, namely\n", - "1. $\\beta_0 > 0$ and $\\beta_1 > 0$,\n", + "We have now four cases to solve besides the trivial cases $\\theta_0$ and/or $\\theta_1$ are zero, namely\n", + "1. $\\theta_0 > 0$ and $\\theta_1 > 0$,\n", "\n", - "2. $\\beta_0 > 0$ and $\\beta_1 < 0$,\n", + "2. $\\theta_0 > 0$ and $\\theta_1 < 0$,\n", "\n", - "3. $\\beta_0 < 0$ and $\\beta_1 > 0$,\n", + "3. $\\theta_0 < 0$ and $\\theta_1 > 0$,\n", "\n", - "4. $\\beta_0 < 0$ and $\\beta_1 < 0$.\n", + "4. $\\theta_0 < 0$ and $\\theta_1 < 0$.\n", "\n", "If we consider the first case, we have then" ] }, { "cell_type": "markdown", - "id": "48598bde", + "id": "c77ef35e", "metadata": { "editable": true }, "source": [ "$$\n", - "-4(4-2\\beta_0)+\\lambda=0,\n", + "-4(4-2\\theta_0)+\\lambda=0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "458cc863", + "id": "69c8185d", "metadata": { "editable": true }, @@ -3324,19 +3324,19 @@ }, { "cell_type": "markdown", - "id": "ff948eca", + "id": "304a2df1", "metadata": { "editable": true }, "source": [ "$$\n", - "-2(2-\\beta_1)+\\lambda=0.\n", + "-2(2-\\theta_1)+\\lambda=0.\n", "$$" ] }, { "cell_type": "markdown", - "id": "38dfba54", + "id": "55dd00fc", "metadata": { "editable": true }, @@ -3346,19 +3346,19 @@ }, { "cell_type": "markdown", - "id": "bc715180", + "id": "d8f3cb2f", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_0=\\frac{16+\\lambda}{8},\n", + "\\theta_0=\\frac{16+\\lambda}{8},\n", "$$" ] }, { "cell_type": "markdown", - "id": "62e9a17c", + "id": "2db2c43e", "metadata": { "editable": true }, @@ -3368,24 +3368,24 @@ }, { "cell_type": "markdown", - "id": "e564b775", + "id": "d875947f", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_1=\\frac{4+\\lambda}{2}.\n", + "\\theta_1=\\frac{4+\\lambda}{2}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "2bdbcac1", + "id": "a4ebed7b", "metadata": { "editable": true }, "source": [ - "Using the constraint on $\\beta_0$ and $\\beta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you.\n", + "Using the constraint on $\\theta_0$ and $\\theta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you.\n", "\n", "Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of $\\lambda$, meaning that we need to perform a search in order to find the optimal values.\n", "\n", @@ -3396,7 +3396,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "a1e3dee1", + "id": "ce6cdf5c", "metadata": { "collapsed": false, "editable": true @@ -3424,14 +3424,14 @@ "y = np.array( [4, 2, 3])\n", "\n", "\n", - "# matrix inversion to find beta\n", - "OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y\n", - "print(OLSbeta)\n", + "# matrix inversion to find theta\n", + "OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y\n", + "print(OLStheta)\n", "# and then make the prediction\n", - "ytildeOLS = X @ OLSbeta\n", + "ytildeOLS = X @ OLStheta\n", "print(\"Training MSE for OLS\")\n", "print(MSE(y,ytildeOLS))\n", - "ypredictOLS = X @ OLSbeta\n", + "ypredictOLS = X @ OLStheta\n", "\n", "# Repeat now for Ridge regression and various values of the regularization parameter\n", "I = np.eye(2,2)\n", @@ -3441,10 +3441,10 @@ "lambdas = np.logspace(-4, 4, nlambdas)\n", "for i in range(nlambdas):\n", " lmb = lambdas[i]\n", - " Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n", - "# print(Ridgebeta)\n", + " Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n", + "# print(Ridgetheta)\n", " # and then make the prediction\n", - " ypredictRidge = X @ Ridgebeta\n", + " ypredictRidge = X @ Ridgetheta\n", " MSEPredict[i] = MSE(y,ypredictRidge)\n", "# print(MSEPredict[i])\n", " # Now plot the results\n", @@ -3458,15 +3458,15 @@ }, { "cell_type": "markdown", - "id": "5973148f", + "id": "79720acf", "metadata": { "editable": true }, "source": [ - "We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\\boldsymbol{\\beta}$, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of $\\tilde{\\boldsymbol{y}}$ approach zero.\n", + "We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\\boldsymbol{\\theta}$, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of $\\tilde{\\boldsymbol{y}}$ approach zero.\n", "\n", "This happens also for Lasso regression, as seen from the next code\n", - "output. The difference is that Lasso shrinks the values of $\\beta$ to\n", + "output. The difference is that Lasso shrinks the values of $\\theta$ to\n", "zero at a much earlier stage and the results flatten out. We see that\n", "Lasso gives also an excellent fit for small values of $\\lambda$ and\n", "shows the best performance of the three regression methods." @@ -3475,7 +3475,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "a961f69c", + "id": "ac1089e2", "metadata": { "collapsed": false, "editable": true @@ -3502,14 +3502,14 @@ "y = np.array( [4, 2, 3])\n", "\n", "\n", - "# matrix inversion to find beta\n", - "OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y\n", - "print(OLSbeta)\n", + "# matrix inversion to find theta\n", + "OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y\n", + "print(OLStheta)\n", "# and then make the prediction\n", - "ytildeOLS = X @ OLSbeta\n", + "ytildeOLS = X @ OLStheta\n", "print(\"Training MSE for OLS\")\n", "print(MSE(y,ytildeOLS))\n", - "ypredictOLS = X @ OLSbeta\n", + "ypredictOLS = X @ OLStheta\n", "\n", "# Repeat now for Ridge regression and various values of the regularization parameter\n", "I = np.eye(2,2)\n", @@ -3520,10 +3520,10 @@ "lambdas = np.logspace(-4, 4, nlambdas)\n", "for i in range(nlambdas):\n", " lmb = lambdas[i]\n", - " Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n", - " print(Ridgebeta)\n", + " Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n", + " print(Ridgetheta)\n", " # and then make the prediction\n", - " ypredictRidge = X @ Ridgebeta\n", + " ypredictRidge = X @ Ridgetheta\n", " MSERidgePredict[i] = MSE(y,ypredictRidge)\n", " RegLasso = linear_model.Lasso(lmb)\n", " RegLasso.fit(X,y)\n", @@ -3542,7 +3542,7 @@ }, { "cell_type": "markdown", - "id": "60cfd641", + "id": "fbc293e8", "metadata": { "editable": true }, @@ -3552,7 +3552,7 @@ "that for small values of the hyperparameter $\\lambda$ all three\n", "methods produce the same mean squared error. Again, Lasso shrinks the\n", "parameter values to zero much earlier than Ridge regression and the\n", - "Lasso results flatten out much earlier since all $\\beta_j=0$ (check\n", + "Lasso results flatten out much earlier since all $\\theta_j=0$ (check\n", "this by printing the values). This case is an example of where OLS\n", "performs best. Lasso and Ridge reproduce the OLS results for a limited\n", "set of $\\lambda$ values." @@ -3561,7 +3561,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "171876b3", + "id": "b8d962a8", "metadata": { "collapsed": false, "editable": true @@ -3599,14 +3599,14 @@ "# We split the data in test and training data\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n", "\n", - "# matrix inversion to find beta\n", - "OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", - "print(OLSbeta)\n", + "# matrix inversion to find theta\n", + "OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", + "print(OLStheta)\n", "# and then make the prediction\n", - "ytildeOLS = X_train @ OLSbeta\n", + "ytildeOLS = X_train @ OLStheta\n", "print(\"Training MSE for OLS\")\n", "print(MSE(y_train,ytildeOLS))\n", - "ypredictOLS = X_test @ OLSbeta\n", + "ypredictOLS = X_test @ OLStheta\n", "print(\"Test MSE OLS\")\n", "print(MSE(y_test,ypredictOLS))\n", "\n", @@ -3621,13 +3621,13 @@ "lambdas = np.logspace(-4, 4, nlambdas)\n", "for i in range(nlambdas):\n", " lmb = lambdas[i]\n", - " Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n", + " Ridgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n", " # include lasso using Scikit-Learn\n", " RegLasso = linear_model.Lasso(lmb)\n", " RegLasso.fit(X_train,y_train)\n", " # and then make the prediction\n", - " ytildeRidge = X_train @ Ridgebeta\n", - " ypredictRidge = X_test @ Ridgebeta\n", + " ytildeRidge = X_train @ Ridgetheta\n", + " ypredictRidge = X_test @ Ridgetheta\n", " ytildeLasso = RegLasso.predict(X_train)\n", " ypredictLasso = RegLasso.predict(X_test)\n", " MSEPredict[i] = MSE(y_test,ypredictRidge)\n", @@ -3650,7 +3650,7 @@ }, { "cell_type": "markdown", - "id": "947928e7", + "id": "083abcdc", "metadata": { "editable": true }, @@ -3665,7 +3665,7 @@ }, { "cell_type": "markdown", - "id": "9559d0a8", + "id": "b23e5726", "metadata": { "editable": true }, @@ -3678,7 +3678,7 @@ "particular, we will focus on what the regularization terms can result\n", "in. We will amongst other things show that the regularization\n", "parameter can reduce considerably the variance of the parameters\n", - "$\\beta$.\n", + "$\\theta$.\n", "\n", "The\n", "advantage of doing linear regression is that we actually end up with\n", @@ -3694,7 +3694,7 @@ }, { "cell_type": "markdown", - "id": "6810eb7d", + "id": "2ea9005f", "metadata": { "editable": true }, @@ -3710,7 +3710,7 @@ }, { "cell_type": "markdown", - "id": "bd997167", + "id": "dda3af52", "metadata": { "editable": true }, @@ -3718,7 +3718,7 @@ "The randomness of $\\varepsilon_i$ implies that\n", "$\\mathbf{y}_i$ is also a random variable. In particular,\n", "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n", - "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n", + "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\theta}$ is a\n", "non-random scalar. To specify the parameters of the distribution of\n", "$\\mathbf{y}_i$ we need to calculate its first two moments. \n", "\n", @@ -3733,7 +3733,7 @@ }, { "cell_type": "markdown", - "id": "4ff740b9", + "id": "094aef6f", "metadata": { "editable": true }, @@ -3745,7 +3745,7 @@ }, { "cell_type": "markdown", - "id": "fe0b2250", + "id": "677a8764", "metadata": { "editable": true }, @@ -3756,19 +3756,19 @@ }, { "cell_type": "markdown", - "id": "87e6b9b3", + "id": "396b620b", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n", + "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\theta}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "68d0de57", + "id": "9c73587c", "metadata": { "editable": true }, @@ -3778,7 +3778,7 @@ }, { "cell_type": "markdown", - "id": "2f239890", + "id": "1a2d8f88", "metadata": { "editable": true }, @@ -3786,15 +3786,15 @@ "$$\n", "\\begin{align*} \n", "\\mathbb{E}(y_i) & =\n", - "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n", - "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n", + "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta}) + \\mathbb{E}(\\varepsilon_i)\n", + "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\theta, \n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", - "id": "2a724679", + "id": "f9c84479", "metadata": { "editable": true }, @@ -3805,7 +3805,7 @@ }, { "cell_type": "markdown", - "id": "2d710e45", + "id": "7b39b6a3", "metadata": { "editable": true }, @@ -3814,12 +3814,12 @@ "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n", "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n", "[\\mathbb{E}(y_i)]^2 \\\\ & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n", - "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n", - "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n", - "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n", - "\\ast} \\, \\beta)^2 \\\\ & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n", - "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n", - "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n", + "\\theta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 \\\\ &\n", + "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 + 2 \\varepsilon_i\n", + "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n", + "\\ast} \\, \\theta)^2 \\\\ & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 + 2\n", + "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta} +\n", + "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 \n", "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n", "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2. \n", "\\end{align*}\n", @@ -3828,32 +3828,32 @@ }, { "cell_type": "markdown", - "id": "488a73d8", + "id": "b89a5103", "metadata": { "editable": true }, "source": [ - "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", - "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n", + "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", + "mean value $\\boldsymbol{X}\\boldsymbol{\\theta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n", "\n", - "With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ we can evaluate the expectation value" + "With the OLS expressions for the parameters $\\boldsymbol{\\theta}$ we can evaluate the expectation value" ] }, { "cell_type": "markdown", - "id": "9b1dca9a", + "id": "311e8043", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathbb{E}(\\boldsymbol{\\beta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n", + "\\mathbb{E}(\\boldsymbol{\\theta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\theta}=\\boldsymbol{\\theta}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "07089a59", + "id": "2b32d6f7", "metadata": { "editable": true }, @@ -3862,35 +3862,35 @@ "\n", "We can also calculate the variance\n", "\n", - "The variance of $\\boldsymbol{\\beta}$ is" + "The variance of $\\boldsymbol{\\theta}$ is" ] }, { "cell_type": "markdown", - "id": "690bd104", + "id": "f5a7fb75", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{eqnarray*}\n", - "\\mbox{Var}(\\boldsymbol{\\beta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n", + "\\mbox{Var}(\\boldsymbol{\\theta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\theta} - \\mathbb{E}(\\boldsymbol{\\theta})] [\\boldsymbol{\\theta} - \\mathbb{E}(\\boldsymbol{\\theta})]^{T} \\}\n", "\\\\\n", - "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}]^{T} \\}\n", + "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\theta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\theta}]^{T} \\}\n", "\\\\\n", - "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n", "% \\\\\n", - "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n", "% \\\\\n", - "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n", "\\\\\n", - "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} \\, \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n", "% \\\\\n", - "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n", + "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^T \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n", "% \\\\\n", - "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n", + "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\boldsymbol{\\theta}^T\n", "\\\\\n", - "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "& = & \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n", "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n", "\\end{eqnarray*}\n", "$$" @@ -3898,21 +3898,21 @@ }, { "cell_type": "markdown", - "id": "6a9132ce", + "id": "6944f707", "metadata": { "editable": true }, "source": [ "where we have used that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n", - "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n", - "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n", + "\\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} \\, \\mathbf{X}^{T} +\n", + "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\theta}) = \\sigma^2\n", "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n", "variance of the estimate of the $j$-th regression coefficient:\n", - "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n", + "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\theta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n", "construct a confidence interval for the estimates.\n", "\n", "In a similar way, we can obtain analytical expressions for say the\n", - "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n", + "expectation values of the parameters $\\boldsymbol{\\theta}$ and their variance\n", "when we employ Ridge regression, allowing us again to define a confidence interval. \n", "\n", "It is rather straightforward to show that" @@ -3920,80 +3920,80 @@ }, { "cell_type": "markdown", - "id": "68cce775", + "id": "9c8d4134", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n", + "\\mathbb{E} \\big[ \\boldsymbol{\\theta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\theta}^{\\mathrm{OLS}}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "a9c3f89a", + "id": "6d47f172", "metadata": { "editable": true }, "source": [ "We see clearly that \n", - "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n", + "$\\mathbb{E} \\big[ \\boldsymbol{\\theta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\theta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n", "\n", "We can also compute the variance as" ] }, { "cell_type": "markdown", - "id": "f9e2f9d7", + "id": "e41732d4", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", + "\\mbox{Var}[\\boldsymbol{\\theta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", "$$" ] }, { "cell_type": "markdown", - "id": "58443fe8", + "id": "2e3123fe", "metadata": { "editable": true }, "source": [ - "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n", + "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\theta}$ goes to zero. \n", "\n", "With this, we can compute the difference" ] }, { "cell_type": "markdown", - "id": "cc34c059", + "id": "9ba7ba08", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n", + "\\mbox{Var}[\\boldsymbol{\\theta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\theta}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "6ad9c8e3", + "id": "f4077e50", "metadata": { "editable": true }, "source": [ "The difference is non-negative definite since each component of the\n", "matrix product is non-negative definite. \n", - "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below." + "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\theta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below." ] }, { "cell_type": "markdown", - "id": "7c09657d", + "id": "2945f567", "metadata": { "editable": true }, @@ -4007,28 +4007,28 @@ "$\\sigma^2$.\n", "\n", "We found above that the outputs $\\boldsymbol{y}$ have a mean value given by\n", - "$\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}}$ and variance $\\sigma^2$. Since the entries to\n", + "$\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$ and variance $\\sigma^2$. Since the entries to\n", "the design matrix are not stochastic variables, we can assume that the\n", "probability distribution of our targets is also a normal distribution\n", - "but now with mean value $\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}}$. This means that a\n", + "but now with mean value $\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$. This means that a\n", "single output $y_i$ is given by the Gaussian distribution" ] }, { "cell_type": "markdown", - "id": "abe9915b", + "id": "1324fb2e", "metadata": { "editable": true }, "source": [ "$$\n", - "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", + "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "326e0c75", + "id": "f19032d0", "metadata": { "editable": true }, @@ -4039,43 +4039,43 @@ }, { "cell_type": "markdown", - "id": "567fb1b1", + "id": "a18bd251", "metadata": { "editable": true }, "source": [ "$$\n", - "p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n", + "p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\theta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]},\n", "$$" ] }, { "cell_type": "markdown", - "id": "107abe1c", + "id": "9f5be910", "metadata": { "editable": true }, "source": [ - "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\beta}$.\n", + "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\theta}$.\n", "\n", "Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event $\\boldsymbol{y}$ as the product of the single events, that is we have" ] }, { "cell_type": "markdown", - "id": "f11ddf78", + "id": "d92545ca", "metadata": { "editable": true }, "source": [ "$$\n", - "p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta}).\n", + "p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\theta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "2abd6e3b", + "id": "d30bf273", "metadata": { "editable": true }, @@ -4086,7 +4086,7 @@ }, { "cell_type": "markdown", - "id": "caddb652", + "id": "70795118", "metadata": { "editable": true }, @@ -4098,7 +4098,7 @@ }, { "cell_type": "markdown", - "id": "291e1dd6", + "id": "ecfb478e", "metadata": { "editable": true }, @@ -4109,26 +4109,26 @@ }, { "cell_type": "markdown", - "id": "73ac95c1", + "id": "89c5df6b", "metadata": { "editable": true }, "source": [ "$$\n", - "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", + "p(\\boldsymbol{D}\\vert\\boldsymbol{\\theta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "ce49493b", + "id": "082b8841", "metadata": { "editable": true }, "source": [ "It is a conditional probability (see below) and reads as the\n", "likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters\n", - "$\\boldsymbol{\\beta}$.\n", + "$\\boldsymbol{\\theta}$.\n", "\n", "In statistics, maximum likelihood estimation (MLE) is a method of\n", "estimating the parameters of an assumed probability distribution,\n", @@ -4137,7 +4137,7 @@ "data is the most probable. \n", "\n", "We will assume here that our events are given by the above Gaussian\n", - "distribution and we will determine the optimal parameters $\\beta$ by\n", + "distribution and we will determine the optimal parameters $\\theta$ by\n", "maximizing the above PDF. However, computing the derivatives of a\n", "product function is cumbersome and can easily lead to overflow and/or\n", "underflowproblems, with potentials for loss of numerical precision.\n", @@ -4156,19 +4156,19 @@ }, { "cell_type": "markdown", - "id": "1ff54861", + "id": "59d1eaca", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta}=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})},\n", + "C(\\boldsymbol{\\theta}=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\theta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\theta})},\n", "$$" ] }, { "cell_type": "markdown", - "id": "e8cdd425", + "id": "7142a2d9", "metadata": { "editable": true }, @@ -4178,63 +4178,63 @@ }, { "cell_type": "markdown", - "id": "95d54be7", + "id": "3e111f5b", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}.\n", + "C(\\boldsymbol{\\theta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "731e3e2a", + "id": "de59021f", "metadata": { "editable": true }, "source": [ - "Taking the derivative of the *new* cost function with respect to the parameters $\\beta$ we recognize our familiar OLS equation, namely" + "Taking the derivative of the *new* cost function with respect to the parameters $\\theta$ we recognize our familiar OLS equation, namely" ] }, { "cell_type": "markdown", - "id": "8c40a24c", + "id": "5e78bfd8", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right) =0,\n", + "\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right) =0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "1e298a02", + "id": "09836e12", "metadata": { "editable": true }, "source": [ - "which leads to the well-known OLS equation for the optimal paramters $\\beta$" + "which leads to the well-known OLS equation for the optimal paramters $\\theta$" ] }, { "cell_type": "markdown", - "id": "fd5c3e4f", + "id": "2f3404a5", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n", + "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n", "$$" ] }, { "cell_type": "markdown", - "id": "d3aab131", + "id": "0bc175f7", "metadata": { "editable": true }, @@ -4254,7 +4254,7 @@ }, { "cell_type": "markdown", - "id": "64646b7c", + "id": "2d0434aa", "metadata": { "editable": true }, @@ -4266,7 +4266,7 @@ }, { "cell_type": "markdown", - "id": "0ad4cc29", + "id": "42f4b84f", "metadata": { "editable": true }, @@ -4276,7 +4276,7 @@ }, { "cell_type": "markdown", - "id": "8dfd2150", + "id": "3677eed8", "metadata": { "editable": true }, @@ -4288,7 +4288,7 @@ }, { "cell_type": "markdown", - "id": "9c0313b7", + "id": "134f8f1a", "metadata": { "editable": true }, @@ -4302,7 +4302,7 @@ }, { "cell_type": "markdown", - "id": "af94800f", + "id": "9f6349f1", "metadata": { "editable": true }, @@ -4314,7 +4314,7 @@ }, { "cell_type": "markdown", - "id": "3ed2ccef", + "id": "557eae67", "metadata": { "editable": true }, @@ -4324,7 +4324,7 @@ }, { "cell_type": "markdown", - "id": "7a74ee19", + "id": "0859c412", "metadata": { "editable": true }, @@ -4336,7 +4336,7 @@ }, { "cell_type": "markdown", - "id": "5191a71e", + "id": "ef82304a", "metadata": { "editable": true }, @@ -4346,7 +4346,7 @@ }, { "cell_type": "markdown", - "id": "5d5de8f7", + "id": "9f2500ca", "metadata": { "editable": true }, @@ -4358,7 +4358,7 @@ }, { "cell_type": "markdown", - "id": "cca75f59", + "id": "b7b9d2d8", "metadata": { "editable": true }, @@ -4368,7 +4368,7 @@ }, { "cell_type": "markdown", - "id": "9113e675", + "id": "f79df283", "metadata": { "editable": true }, @@ -4380,7 +4380,7 @@ }, { "cell_type": "markdown", - "id": "a21d13da", + "id": "ca93f0cf", "metadata": { "editable": true }, @@ -4416,7 +4416,7 @@ }, { "cell_type": "markdown", - "id": "0c7abec6", + "id": "9092d777", "metadata": { "editable": true }, @@ -4428,7 +4428,7 @@ }, { "cell_type": "markdown", - "id": "2eccb1af", + "id": "017ca9fb", "metadata": { "editable": true }, @@ -4441,7 +4441,7 @@ }, { "cell_type": "markdown", - "id": "3c6635e5", + "id": "529014b2", "metadata": { "editable": true }, @@ -4453,7 +4453,7 @@ }, { "cell_type": "markdown", - "id": "ecf0a0b6", + "id": "23629d2d", "metadata": { "editable": true }, @@ -4467,7 +4467,7 @@ }, { "cell_type": "markdown", - "id": "166345a1", + "id": "80c6a09b", "metadata": { "editable": true }, @@ -4479,7 +4479,7 @@ }, { "cell_type": "markdown", - "id": "8a73e80e", + "id": "fe371d69", "metadata": { "editable": true }, @@ -4490,7 +4490,7 @@ }, { "cell_type": "markdown", - "id": "01441388", + "id": "ed9320dd", "metadata": { "editable": true }, @@ -4502,7 +4502,7 @@ }, { "cell_type": "markdown", - "id": "93fe1e0a", + "id": "710d72a1", "metadata": { "editable": true }, @@ -4514,7 +4514,7 @@ }, { "cell_type": "markdown", - "id": "d6860415", + "id": "331424a5", "metadata": { "editable": true }, @@ -4532,7 +4532,7 @@ }, { "cell_type": "markdown", - "id": "9710dd92", + "id": "683874d3", "metadata": { "editable": true }, @@ -4550,7 +4550,7 @@ }, { "cell_type": "markdown", - "id": "23e3912e", + "id": "2b32fd95", "metadata": { "editable": true }, @@ -4560,7 +4560,7 @@ }, { "cell_type": "markdown", - "id": "cf500f71", + "id": "9ed06a74", "metadata": { "editable": true }, @@ -4575,9 +4575,9 @@ "Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit. \n", "\n", "We will play around with a study of the values for the optimal\n", - "parameters $\\boldsymbol{\\beta}$ using OLS, Ridge and Lasso regression. For\n", + "parameters $\\boldsymbol{\\theta}$ using OLS, Ridge and Lasso regression. For\n", "OLS, you will notice as function of the noise and polynomial degree,\n", - "that the parameters $\\beta$ will fluctuate from order to order in the\n", + "that the parameters $\\theta$ will fluctuate from order to order in the\n", "polynomial fit and that for larger and larger polynomial degrees of\n", "freedom, the parameters will tend to increase in value for OLS.\n", "\n", @@ -4589,7 +4589,7 @@ { "cell_type": "code", "execution_count": 12, - "id": "134d0a22", + "id": "9a336ec4", "metadata": { "collapsed": false, "editable": true @@ -4624,10 +4624,10 @@ "# We split the data in test and training data\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n", "\n", - "# matrix inversion to find beta\n", - "OLSbeta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train\n", - "print(OLSbeta)\n", - "ypredictOLS = X_test @ OLSbeta\n", + "# matrix inversion to find theta\n", + "OLStheta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train\n", + "print(OLStheta)\n", + "ypredictOLS = X_test @ OLStheta\n", "print(\"Test MSE OLS\")\n", "print(MSE(y_test,ypredictOLS))\n", "# Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn\n", @@ -4663,19 +4663,19 @@ }, { "cell_type": "markdown", - "id": "7dcfe550", + "id": "87b78389", "metadata": { "editable": true }, "source": [ "How can we understand this?\n", "\n", - "Let us write out the values of the coefficients $\\beta_i$ as functions\n", + "Let us write out the values of the coefficients $\\theta_i$ as functions\n", "of the polynomial degree and noise. We will focus only on the Ridge\n", "results and some few selected values of the hyperparameter $\\lambda$.\n", "\n", "If we don't include any noise and run this code for different values\n", - "of the polynomial degree, we notice that the results for $\\beta_i$ do\n", + "of the polynomial degree, we notice that the results for $\\theta_i$ do\n", "not show great changes from one order to the next. This is an\n", "indication that for higher polynomial orders, our parameters become\n", "less important.\n", @@ -4683,17 +4683,17 @@ "If we however add noise, what happens is that the polynomial fit is\n", "trying to adjust the fit to traverse in the best possible way all data\n", "points. This can lead to large fluctuations in the parameters\n", - "$\\beta_i$ as functions of polynomial order. It will also be reflected\n", - "in a larger value of the variance of each parameter $\\beta_i$. What\n", + "$\\theta_i$ as functions of polynomial order. It will also be reflected\n", + "in a larger value of the variance of each parameter $\\theta_i$. What\n", "Ridge regression (and Lasso as well) are doing then is to try to\n", - "quench the fluctuations in the parameters of $\\beta_i$ which have a\n", + "quench the fluctuations in the parameters of $\\theta_i$ which have a\n", "large variance (normally for higher orders in the polynomial)." ] }, { "cell_type": "code", "execution_count": 13, - "id": "5ff5d7c2", + "id": "e2c403e8", "metadata": { "collapsed": false, "editable": true @@ -4735,30 +4735,30 @@ " # and then make the prediction\n", " ypredictRidge = RegRidge.predict(X_test)\n", " Coeffs = np.array(RegRidge.coef_)\n", - " BetaValues = pd.DataFrame(Coeffs)\n", - " BetaValues.columns = ['beta']\n", - " display(BetaValues)" + " ThetaValues = pd.DataFrame(Coeffs)\n", + " ThetaValues.columns = ['theta']\n", + " display(ThetaValues)" ] }, { "cell_type": "markdown", - "id": "f1680928", + "id": "4f4dd369", "metadata": { "editable": true }, "source": [ "As an exercise, repeat these calculations with ordinary least squares\n", "only with and without noise. Calculate thereafter the variance of the\n", - "parameters $\\beta_j$ as function of polynomial order and of the added\n", + "parameters $\\theta_j$ as function of polynomial order and of the added\n", "noise. Here we recommend to use $\\sigma^2=1$ as variance for the\n", "added noise (which follows a normal distribution with mean value zero).\n", - "Comment your results. If you have a large noise term, do the parameters $\\beta_j$ vary more as function\n", + "Comment your results. If you have a large noise term, do the parameters $\\theta_j$ vary more as function\n", "of model complexity? And what about their variance?" ] }, { "cell_type": "markdown", - "id": "5b458403", + "id": "67adfefc", "metadata": { "editable": true }, @@ -4767,7 +4767,7 @@ "\n", "We have seen that Ridge regression suppresses those features which\n", "have a small singular value. This corresponds to a feature which exhibits\n", - "a large variance in the parameters $\\beta_j$.\n", + "a large variance in the parameters $\\theta_j$.\n", "Our analysis hitherto has been based on linear algebra. To add to our intuition, we will use\n", "Bayes' theorem in order to deepen our understanding of Ridge and Lasso regression. \n", "\n", @@ -4776,7 +4776,7 @@ }, { "cell_type": "markdown", - "id": "6e2d6bf6", + "id": "ab781318", "metadata": { "editable": true }, @@ -4788,7 +4788,7 @@ }, { "cell_type": "markdown", - "id": "098b6cbd", + "id": "ccf25024", "metadata": { "editable": true }, @@ -4798,41 +4798,41 @@ }, { "cell_type": "markdown", - "id": "8a02d0aa", + "id": "9c8bc496", "metadata": { "editable": true }, "source": [ "$$\n", - "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", + "p(\\boldsymbol{D}\\vert\\boldsymbol{\\theta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "97f22408", + "id": "7dcf96d4", "metadata": { "editable": true }, "source": [ - "In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\beta}$ given a domain of events $\\boldsymbol{D}$? That is, how can we define the posterior probability" + "In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\theta}$ given a domain of events $\\boldsymbol{D}$? That is, how can we define the posterior probability" ] }, { "cell_type": "markdown", - "id": "27038459", + "id": "83af5cfb", "metadata": { "editable": true }, "source": [ "$$\n", - "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D}).\n", + "p(\\boldsymbol{\\theta}\\vert\\boldsymbol{D}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "f682a8c3", + "id": "baf02e7e", "metadata": { "editable": true }, @@ -4842,49 +4842,49 @@ }, { "cell_type": "markdown", - "id": "7fdaa748", + "id": "1ec2d48a", "metadata": { "editable": true }, "source": [ "$$\n", - "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})p(\\boldsymbol{\\beta}).\n", + "p(\\boldsymbol{\\theta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\theta})p(\\boldsymbol{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "1e7fa52c", + "id": "e77ae0c0", "metadata": { "editable": true }, "source": [ - "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta})$! \n", + "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\theta})$ but need one for the **prior** $p(\\boldsymbol{\\theta})$! \n", "\n", "With the posterior probability defined by a likelihood which we have\n", "already modeled and an unknown prior, we are now ready to make\n", "additional models for the prior.\n", "\n", - "We can, based on our discussions of the variance of $\\boldsymbol{\\beta}$ and\n", - "the mean value, assume that the prior for the values $\\boldsymbol{\\beta}$ is\n", + "We can, based on our discussions of the variance of $\\boldsymbol{\\theta}$ and\n", + "the mean value, assume that the prior for the values $\\boldsymbol{\\theta}$ is\n", "given by a Gaussian with mean value zero and variance $\\tau^2$, that" ] }, { "cell_type": "markdown", - "id": "501d66f4", + "id": "5ee70503", "metadata": { "editable": true }, "source": [ "$$\n", - "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", + "p(\\boldsymbol{\\theta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\theta_j^2}{2\\tau^2}\\right)}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "f029c143", + "id": "81b5ea11", "metadata": { "editable": true }, @@ -4894,44 +4894,44 @@ }, { "cell_type": "markdown", - "id": "7f7c3e11", + "id": "a15cac35", "metadata": { "editable": true }, "source": [ "$$\n", - "p(\\boldsymbol{\\beta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", + "p(\\boldsymbol{\\theta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\theta_j^2}{2\\tau^2}\\right)}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "1f39114c", + "id": "76f66371", "metadata": { "editable": true }, "source": [ - "We can now optimize this quantity with respect to $\\boldsymbol{\\beta}$. As we\n", + "We can now optimize this quantity with respect to $\\boldsymbol{\\theta}$. As we\n", "did for OLS, this is most conveniently done by taking the negative\n", "logarithm of the posterior probability. Doing so and leaving out the\n", - "terms that do not depend on $\\beta$, we have" + "terms that do not depend on $\\theta$, we have" ] }, { "cell_type": "markdown", - "id": "81cc7b03", + "id": "deaecaf5", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", + "C(\\boldsymbol{\\theta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_2^2,\n", "$$" ] }, { "cell_type": "markdown", - "id": "1e614b9b", + "id": "da1297cc", "metadata": { "editable": true }, @@ -4941,19 +4941,19 @@ }, { "cell_type": "markdown", - "id": "77252afc", + "id": "4c429a58", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", + "C(\\boldsymbol{\\theta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_2^2,\n", "$$" ] }, { "cell_type": "markdown", - "id": "14953579", + "id": "2c9fd875", "metadata": { "editable": true }, @@ -4965,19 +4965,19 @@ }, { "cell_type": "markdown", - "id": "36f1f63d", + "id": "350a62eb", "metadata": { "editable": true }, "source": [ "$$\n", - "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", + "p(\\boldsymbol{\\theta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\theta_j\\vert}{\\tau}\\right)}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "50dd90a5", + "id": "9bb76e79", "metadata": { "editable": true }, @@ -4987,43 +4987,43 @@ }, { "cell_type": "markdown", - "id": "de39cb12", + "id": "e08c07dd", "metadata": { "editable": true }, "source": [ "$$\n", - "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", + "p(\\boldsymbol{\\theta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\theta_j\\vert}{\\tau}\\right)}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "ad1fc46e", + "id": "e4bd6b2c", "metadata": { "editable": true }, "source": [ "Taking the negative\n", "logarithm of the posterior probability and leaving out the\n", - "constants terms that do not depend on $\\beta$, we have" + "constants terms that do not depend on $\\theta$, we have" ] }, { "cell_type": "markdown", - "id": "ff8695d4", + "id": "7c1bc34c", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", + "C(\\boldsymbol{\\theta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_1,\n", "$$" ] }, { "cell_type": "markdown", - "id": "0de8080e", + "id": "5f24477c", "metadata": { "editable": true }, @@ -5033,19 +5033,19 @@ }, { "cell_type": "markdown", - "id": "3965e5ef", + "id": "d9948bc0", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", + "C(\\boldsymbol{\\theta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_1,\n", "$$" ] }, { "cell_type": "markdown", - "id": "5c978cdf", + "id": "d690afa3", "metadata": { "editable": true }, @@ -5054,7 +5054,7 @@ "\n", "Plotting these prior functions shows us that we can use the parameter\n", "$\\lambda$ to shrink or increase the role of a given parameter\n", - "$\\beta_j$. The variance for the Laplace distribution is\n", + "$\\theta_j$. The variance for the Laplace distribution is\n", "$2\\tau^2=1/\\lambda$ while for the Gaussian distribution it is\n", "$\\sigma^2=1/(2\\lambda)$. Thus, increasing the variance means\n", "decreasing $\\lambda$ and shrinking the variance means increasing\n", diff --git a/doc/LectureNotes/_build/html/_sources/chapter3.ipynb b/doc/LectureNotes/_build/html/_sources/chapter3.ipynb index 6e1a008cf..a9b8c34f2 100644 --- a/doc/LectureNotes/_build/html/_sources/chapter3.ipynb +++ b/doc/LectureNotes/_build/html/_sources/chapter3.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "51186d57", + "id": "663483da", "metadata": { "editable": true }, @@ -13,7 +13,7 @@ }, { "cell_type": "markdown", - "id": "79a63fb4", + "id": "05ca2993", "metadata": { "editable": true }, @@ -23,7 +23,7 @@ }, { "cell_type": "markdown", - "id": "f3d916b4", + "id": "cea49fb9", "metadata": { "editable": true }, @@ -74,7 +74,7 @@ }, { "cell_type": "markdown", - "id": "46cb3279", + "id": "85adbc81", "metadata": { "editable": true }, @@ -105,7 +105,7 @@ }, { "cell_type": "markdown", - "id": "0fe38e07", + "id": "4d010e03", "metadata": { "editable": true }, @@ -121,7 +121,7 @@ }, { "cell_type": "markdown", - "id": "d9d6955b", + "id": "54c13e4f", "metadata": { "editable": true }, @@ -129,7 +129,7 @@ "The randomness of $\\varepsilon_i$ implies that\n", "$\\mathbf{y}_i$ is also a random variable. In particular,\n", "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n", - "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n", + "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\theta}$ is a\n", "non-random scalar. To specify the parameters of the distribution of\n", "$\\mathbf{y}_i$ we need to calculate its first two moments. \n", "\n", @@ -144,7 +144,7 @@ }, { "cell_type": "markdown", - "id": "f41f7049", + "id": "1705258c", "metadata": { "editable": true }, @@ -156,7 +156,7 @@ }, { "cell_type": "markdown", - "id": "2ee172f4", + "id": "f96c2ee1", "metadata": { "editable": true }, @@ -167,19 +167,19 @@ }, { "cell_type": "markdown", - "id": "1aecc768", + "id": "84fc8fc9", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n", + "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\theta}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "12e9bdea", + "id": "c846eb35", "metadata": { "editable": true }, @@ -189,7 +189,7 @@ }, { "cell_type": "markdown", - "id": "8f9db9db", + "id": "a7c728c5", "metadata": { "editable": true }, @@ -197,15 +197,15 @@ "$$\n", "\\begin{align*} \n", "\\mathbb{E}(y_i) & =\n", - "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n", - "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n", + "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta}) + \\mathbb{E}(\\varepsilon_i)\n", + "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\theta, \n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", - "id": "b10abe89", + "id": "d7d34864", "metadata": { "editable": true }, @@ -216,7 +216,7 @@ }, { "cell_type": "markdown", - "id": "bec51521", + "id": "f235a710", "metadata": { "editable": true }, @@ -225,12 +225,12 @@ "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n", "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n", "[\\mathbb{E}(y_i)]^2 \\\\ & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n", - "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n", - "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n", - "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n", - "\\ast} \\, \\beta)^2 \\\\ & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n", - "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n", - "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n", + "\\theta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 \\\\ &\n", + "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 + 2 \\varepsilon_i\n", + "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n", + "\\ast} \\, \\theta)^2 \\\\ & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 + 2\n", + "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta} +\n", + "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 \n", "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n", "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2. \n", "\\end{align*}\n", @@ -239,32 +239,32 @@ }, { "cell_type": "markdown", - "id": "4859640c", + "id": "faa4158d", "metadata": { "editable": true }, "source": [ - "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", - "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n", + "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", + "mean value $\\boldsymbol{X}\\boldsymbol{\\theta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n", "\n", - "With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ we can evaluate the expectation value" + "With the OLS expressions for the parameters $\\boldsymbol{\\theta}$ we can evaluate the expectation value" ] }, { "cell_type": "markdown", - "id": "69978823", + "id": "da189fae", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathbb{E}(\\boldsymbol{\\beta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n", + "\\mathbb{E}(\\boldsymbol{\\theta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\theta}=\\boldsymbol{\\theta}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "5caee9d9", + "id": "640c53e9", "metadata": { "editable": true }, @@ -273,35 +273,35 @@ "\n", "We can also calculate the variance\n", "\n", - "The variance of $\\boldsymbol{\\beta}$ is" + "The variance of $\\boldsymbol{\\theta}$ is" ] }, { "cell_type": "markdown", - "id": "6791e5b4", + "id": "e2eee7fb", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{eqnarray*}\n", - "\\mbox{Var}(\\boldsymbol{\\beta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n", + "\\mbox{Var}(\\boldsymbol{\\theta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\theta} - \\mathbb{E}(\\boldsymbol{\\theta})] [\\boldsymbol{\\theta} - \\mathbb{E}(\\boldsymbol{\\theta})]^{T} \\}\n", "\\\\\n", - "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}]^{T} \\}\n", + "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\theta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\theta}]^{T} \\}\n", "\\\\\n", - "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n", "% \\\\\n", - "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n", "% \\\\\n", - "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n", "\\\\\n", - "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} \\, \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n", "% \\\\\n", - "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n", + "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^T \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n", "% \\\\\n", - "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n", + "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\boldsymbol{\\theta}^T\n", "\\\\\n", - "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "& = & \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n", "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n", "\\end{eqnarray*}\n", "$$" @@ -309,22 +309,22 @@ }, { "cell_type": "markdown", - "id": "4f747992", + "id": "69660ffd", "metadata": { "editable": true }, "source": [ "where we have used that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n", - "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n", - "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n", + "\\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} \\, \\mathbf{X}^{T} +\n", + "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\theta}) = \\sigma^2\n", "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n", "variance of the estimate of the $j$-th regression coefficient:\n", - "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 \\sqrt{\n", + "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\theta}_j ) = \\boldsymbol{\\sigma}^2 \\sqrt{\n", "[(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} }$. This may be used to\n", "construct a confidence interval for the estimates.\n", "\n", "In a similar way, we can obtain analytical expressions for say the\n", - "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n", + "expectation values of the parameters $\\boldsymbol{\\theta}$ and their variance\n", "when we employ Ridge regression, allowing us again to define a confidence interval. \n", "\n", "It is rather straightforward to show that" @@ -332,80 +332,80 @@ }, { "cell_type": "markdown", - "id": "d4679496", + "id": "166fe7a6", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n", + "\\mathbb{E} \\big[ \\boldsymbol{\\theta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\theta}^{\\mathrm{OLS}}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "bd3534e1", + "id": "9b9e1336", "metadata": { "editable": true }, "source": [ "We see clearly that \n", - "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n", + "$\\mathbb{E} \\big[ \\boldsymbol{\\theta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\theta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n", "\n", "We can also compute the variance as" ] }, { "cell_type": "markdown", - "id": "125b139e", + "id": "d4548ed8", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", + "\\mbox{Var}[\\boldsymbol{\\theta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", "$$" ] }, { "cell_type": "markdown", - "id": "98b59ea0", + "id": "642f4b4a", "metadata": { "editable": true }, "source": [ - "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n", + "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\theta}$ goes to zero. \n", "\n", "With this, we can compute the difference" ] }, { "cell_type": "markdown", - "id": "dcfd7a6c", + "id": "e8a1528a", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n", + "\\mbox{Var}[\\boldsymbol{\\theta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\theta}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "30f43be3", + "id": "1bd2f713", "metadata": { "editable": true }, "source": [ "The difference is non-negative definite since each component of the\n", "matrix product is non-negative definite. \n", - "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below." + "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\theta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below." ] }, { "cell_type": "markdown", - "id": "e3c1bf3c", + "id": "8019b9c6", "metadata": { "editable": true }, @@ -446,14 +446,14 @@ "variance of $\\overline{X}$ (which often is the case), then there is no\n", "need for bootstrapping. \n", "\n", - "The Jackknife works by making many replicas of the estimator $\\widehat{\\beta}$. \n", + "The Jackknife works by making many replicas of the estimator $\\widehat{\\theta}$. \n", "The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\\boldsymbol{x} = (x_1,x_2,\\cdots,X_n)$. \n", "Let $\\boldsymbol{x}_i$ denote the vector" ] }, { "cell_type": "markdown", - "id": "3925f435", + "id": "38e804f2", "metadata": { "editable": true }, @@ -465,21 +465,21 @@ }, { "cell_type": "markdown", - "id": "cc6328ee", + "id": "35eb791c", "metadata": { "editable": true }, "source": [ "which equals the vector $\\boldsymbol{x}$ with the exception that observation\n", "number $i$ is left out. Using this notation, define\n", - "$\\widehat{\\beta}_i$ to be the estimator\n", - "$\\widehat{\\beta}$ computed using $\\vec{X}_i$." + "$\\widehat{\\theta}_i$ to be the estimator\n", + "$\\widehat{\\theta}$ computed using $\\vec{X}_i$." ] }, { "cell_type": "code", "execution_count": 1, - "id": "b751a941", + "id": "3261f074", "metadata": { "collapsed": false, "editable": true @@ -518,7 +518,7 @@ }, { "cell_type": "markdown", - "id": "25ff562a", + "id": "9d117c52", "metadata": { "editable": true }, @@ -537,28 +537,28 @@ "\n", "4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).\n", "\n", - "Since $\\widehat{\\beta} = \\widehat{\\beta}(\\boldsymbol{X})$ is a function of random variables,\n", - "$\\widehat{\\beta}$ itself must be a random variable. Thus it has\n", + "Since $\\widehat{\\theta} = \\widehat{\\theta}(\\boldsymbol{X})$ is a function of random variables,\n", + "$\\widehat{\\theta}$ itself must be a random variable. Thus it has\n", "a pdf, call this function $p(\\boldsymbol{t})$. The aim of the bootstrap is to\n", "estimate $p(\\boldsymbol{t})$ by the relative frequency of\n", - "$\\widehat{\\beta}$. You can think of this as using a histogram\n", + "$\\widehat{\\theta}$. You can think of this as using a histogram\n", "in the place of $p(\\boldsymbol{t})$. If the relative frequency closely\n", "resembles $p(\\vec{t})$, then using numerics, it is straight forward to\n", "estimate all the interesting parameters of $p(\\boldsymbol{t})$ using point\n", "estimators. \n", "\n", - "In the case that $\\widehat{\\beta}$ has\n", + "In the case that $\\widehat{\\theta}$ has\n", "more than one component, and the components are independent, we use the\n", "same estimator on each component separately. If the probability\n", "density function of $X_i$, $p(x)$, had been known, then it would have\n", "been straight forward to do this by: \n", "1. Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \\cdots, X_n^*)$. \n", "\n", - "2. Then using these numbers, we could compute a replica of $\\widehat{\\beta}$ called $\\widehat{\\beta}^*$. \n", + "2. Then using these numbers, we could compute a replica of $\\widehat{\\theta}$ called $\\widehat{\\theta}^*$. \n", "\n", "By repeated use of (1) and (2), many\n", - "estimates of $\\widehat{\\beta}$ could have been obtained. The\n", - "idea is to use the relative frequency of $\\widehat{\\beta}^*$\n", + "estimates of $\\widehat{\\theta}$ could have been obtained. The\n", + "idea is to use the relative frequency of $\\widehat{\\theta}^*$\n", "(think of a histogram) as an estimate of $p(\\boldsymbol{t})$.\n", "\n", "But\n", @@ -581,19 +581,19 @@ "\n", "2. Define a vector $\\boldsymbol{x}^*$ containing the values which were drawn from $\\boldsymbol{x}$. \n", "\n", - "3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\beta}^*$ by evaluating $\\widehat \\beta$ under the observations $\\boldsymbol{x}^*$. \n", + "3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\theta}^*$ by evaluating $\\widehat \\theta$ under the observations $\\boldsymbol{x}^*$. \n", "\n", "4. Repeat this process $k$ times. \n", "\n", "When you are done, you can draw a histogram of the relative frequency\n", - "of $\\widehat \\beta^*$. This is your estimate of the probability\n", + "of $\\widehat \\theta^*$. This is your estimate of the probability\n", "distribution $p(t)$. Using this probability distribution you can\n", "estimate any statistics thereof. In principle you never draw the\n", - "histogram of the relative frequency of $\\widehat{\\beta}^*$. Instead\n", + "histogram of the relative frequency of $\\widehat{\\theta}^*$. Instead\n", "you use the estimators corresponding to the statistic of interest. For\n", "example, if you are interested in estimating the variance of $\\widehat\n", - "\\beta$, apply the estimator $\\widehat \\sigma^2$ to the values\n", - "$\\widehat \\beta^*$.\n", + "\\theta$, apply the estimator $\\widehat \\sigma^2$ to the values\n", + "$\\widehat \\theta^*$.\n", "\n", "Before we proceed however, we need to remind ourselves about a central\n", "theorem in statistics, namely the so-called **central limit theorem**.\n", @@ -614,7 +614,7 @@ }, { "cell_type": "markdown", - "id": "fa55ab5a", + "id": "d864122f", "metadata": { "editable": true }, @@ -626,7 +626,7 @@ }, { "cell_type": "markdown", - "id": "697c0c94", + "id": "6750e014", "metadata": { "editable": true }, @@ -641,7 +641,7 @@ }, { "cell_type": "markdown", - "id": "bd26bbd1", + "id": "d5a74550", "metadata": { "editable": true }, @@ -654,7 +654,7 @@ }, { "cell_type": "markdown", - "id": "68664e4f", + "id": "c44dc3f1", "metadata": { "editable": true }, @@ -669,7 +669,7 @@ }, { "cell_type": "markdown", - "id": "742a107a", + "id": "202b1bba", "metadata": { "editable": true }, @@ -682,7 +682,7 @@ }, { "cell_type": "markdown", - "id": "b6d624c1", + "id": "da055544", "metadata": { "editable": true }, @@ -693,7 +693,7 @@ }, { "cell_type": "markdown", - "id": "46458586", + "id": "7773cf22", "metadata": { "editable": true }, @@ -707,7 +707,7 @@ }, { "cell_type": "markdown", - "id": "ca119461", + "id": "8f5e06ca", "metadata": { "editable": true }, @@ -717,7 +717,7 @@ }, { "cell_type": "markdown", - "id": "f75b40fc", + "id": "82157124", "metadata": { "editable": true }, @@ -731,7 +731,7 @@ }, { "cell_type": "markdown", - "id": "a4b64e20", + "id": "2af358ec", "metadata": { "editable": true }, @@ -742,7 +742,7 @@ }, { "cell_type": "markdown", - "id": "fc76951f", + "id": "4d4bafce", "metadata": { "editable": true }, @@ -755,7 +755,7 @@ }, { "cell_type": "markdown", - "id": "76979572", + "id": "628ec1b1", "metadata": { "editable": true }, @@ -765,7 +765,7 @@ }, { "cell_type": "markdown", - "id": "3edff7d2", + "id": "eb27434b", "metadata": { "editable": true }, @@ -778,7 +778,7 @@ }, { "cell_type": "markdown", - "id": "73a9341f", + "id": "8dbfd377", "metadata": { "editable": true }, @@ -788,7 +788,7 @@ }, { "cell_type": "markdown", - "id": "88f1cc30", + "id": "9eda7c0f", "metadata": { "editable": true }, @@ -801,7 +801,7 @@ }, { "cell_type": "markdown", - "id": "1aec913e", + "id": "6c97b3dc", "metadata": { "editable": true }, @@ -822,7 +822,7 @@ }, { "cell_type": "markdown", - "id": "42b317e7", + "id": "6d316b42", "metadata": { "editable": true }, @@ -835,7 +835,7 @@ }, { "cell_type": "markdown", - "id": "e5baf71d", + "id": "426cae9c", "metadata": { "editable": true }, @@ -847,7 +847,7 @@ }, { "cell_type": "markdown", - "id": "8e2b39e5", + "id": "3445fa66", "metadata": { "editable": true }, @@ -860,7 +860,7 @@ }, { "cell_type": "markdown", - "id": "472e7c2c", + "id": "fee204e8", "metadata": { "editable": true }, @@ -894,7 +894,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "0ff7b796", + "id": "b98fbaba", "metadata": { "collapsed": false, "editable": true @@ -933,7 +933,7 @@ }, { "cell_type": "markdown", - "id": "22680159", + "id": "bc56e30f", "metadata": { "editable": true }, @@ -946,7 +946,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "fe018bb3", + "id": "817ebac7", "metadata": { "collapsed": false, "editable": true @@ -966,7 +966,7 @@ }, { "cell_type": "markdown", - "id": "5bbd613c", + "id": "454fba01", "metadata": { "editable": true }, @@ -984,7 +984,7 @@ }, { "cell_type": "markdown", - "id": "ca3fde4a", + "id": "1716dfb2", "metadata": { "editable": true }, @@ -996,7 +996,7 @@ }, { "cell_type": "markdown", - "id": "8e11f84b", + "id": "5295a516", "metadata": { "editable": true }, @@ -1005,27 +1005,27 @@ "\n", "In our derivation of the ordinary least squares method we defined then\n", "an approximation to the function $f$ in terms of the parameters\n", - "$\\boldsymbol{\\beta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n", - "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\beta}$. \n", + "$\\boldsymbol{\\theta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n", + "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\theta}$. \n", "\n", - "Thereafter we found the parameters $\\boldsymbol{\\beta}$ by optimizing the means squared error via the so-called cost function" + "Thereafter we found the parameters $\\boldsymbol{\\theta}$ by optimizing the means squared error via the so-called cost function" ] }, { "cell_type": "markdown", - "id": "026a65c8", + "id": "b5953a4f", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n", + "C(\\boldsymbol{X},\\boldsymbol{\\theta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n", "$$" ] }, { "cell_type": "markdown", - "id": "e59918c7", + "id": "ea8014cf", "metadata": { "editable": true }, @@ -1035,7 +1035,7 @@ }, { "cell_type": "markdown", - "id": "2fd3f73c", + "id": "4159a3e3", "metadata": { "editable": true }, @@ -1047,7 +1047,7 @@ }, { "cell_type": "markdown", - "id": "7daf46c9", + "id": "3c18f501", "metadata": { "editable": true }, @@ -1064,7 +1064,7 @@ }, { "cell_type": "markdown", - "id": "6094266b", + "id": "09b0f8b1", "metadata": { "editable": true }, @@ -1076,7 +1076,7 @@ }, { "cell_type": "markdown", - "id": "643e0047", + "id": "80679145", "metadata": { "editable": true }, @@ -1086,7 +1086,7 @@ }, { "cell_type": "markdown", - "id": "1319bde5", + "id": "1f39a627", "metadata": { "editable": true }, @@ -1098,7 +1098,7 @@ }, { "cell_type": "markdown", - "id": "9c6d6da1", + "id": "b045c821", "metadata": { "editable": true }, @@ -1108,7 +1108,7 @@ }, { "cell_type": "markdown", - "id": "855756ef", + "id": "15cb1a46", "metadata": { "editable": true }, @@ -1120,7 +1120,7 @@ }, { "cell_type": "markdown", - "id": "34d24717", + "id": "d0c484eb", "metadata": { "editable": true }, @@ -1131,7 +1131,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "d51b6100", + "id": "7940a9b2", "metadata": { "collapsed": false, "editable": true @@ -1197,7 +1197,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "bd636def", + "id": "e9698e1e", "metadata": { "collapsed": false, "editable": true @@ -1254,7 +1254,7 @@ }, { "cell_type": "markdown", - "id": "8cf88b3a", + "id": "3edbafad", "metadata": { "editable": true }, @@ -1291,7 +1291,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "86bfc49a", + "id": "5aa48d95", "metadata": { "collapsed": false, "editable": true @@ -1374,7 +1374,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "0c2a183a", + "id": "50c5c640", "metadata": { "collapsed": false, "editable": true @@ -1463,7 +1463,7 @@ }, { "cell_type": "markdown", - "id": "2c6c9e89", + "id": "947db26f", "metadata": { "editable": true }, @@ -1494,14 +1494,14 @@ }, { "cell_type": "markdown", - "id": "71738b2a", + "id": "9974dc92", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{align*}\n", - "\\boldsymbol{\\beta}_{-i}(\\lambda) & = ( \\boldsymbol{X}_{-i, \\ast}^{T}\n", + "\\boldsymbol{\\theta}_{-i}(\\lambda) & = ( \\boldsymbol{X}_{-i, \\ast}^{T}\n", "\\boldsymbol{X}_{-i, \\ast} + \\lambda \\boldsymbol{I}_{pp})^{-1}\n", "\\boldsymbol{X}_{-i, \\ast}^{T} \\boldsymbol{y}_{-i}\n", "\\end{align*}\n", @@ -1510,12 +1510,12 @@ }, { "cell_type": "markdown", - "id": "14db46b6", + "id": "45b8c96c", "metadata": { "editable": true }, "source": [ - "* Evaluate the prediction performance of these models on the test set by $\\log\\{L[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\beta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n", + "* Evaluate the prediction performance of these models on the test set by $\\log\\{L[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\theta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\theta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n", "\n", "* Repeat the first three steps such that each sample plays the role of the test set once.\n", "\n", @@ -1524,21 +1524,21 @@ }, { "cell_type": "markdown", - "id": "95e5c8e4", + "id": "25b8400a", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{align*}\n", - "\\frac{1}{n} \\sum_{i = 1}^n \\log\\{L[y_i, \\mathbf{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}.\n", + "\\frac{1}{n} \\sum_{i = 1}^n \\log\\{L[y_i, \\mathbf{X}_{i, \\ast}; \\boldsymbol{\\theta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}.\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", - "id": "7e60f51d", + "id": "91b1405d", "metadata": { "editable": true }, @@ -1567,7 +1567,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "2cef0eb7", + "id": "48621261", "metadata": { "collapsed": false, "editable": true @@ -1667,7 +1667,7 @@ }, { "cell_type": "markdown", - "id": "f501c9cf", + "id": "2213dc0e", "metadata": { "editable": true }, @@ -1678,7 +1678,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "30e1e320", + "id": "6e7f3e92", "metadata": { "collapsed": false, "editable": true @@ -1756,7 +1756,7 @@ }, { "cell_type": "markdown", - "id": "383e5c2a", + "id": "9c98e1e5", "metadata": { "editable": true }, @@ -1766,7 +1766,7 @@ }, { "cell_type": "markdown", - "id": "1f9cd409", + "id": "b00dd83d", "metadata": { "editable": true }, @@ -1789,8 +1789,8 @@ "when all predictors are zero (the columns in the design matrix), it\n", "may be a bad idea to implement a model which penalizes the intercept.\n", "Furthermore, in for example Ridge and Lasso regression, the default solutions\n", - "from the library **Scikit-Learn** (when not shrinking $\\beta_0$) for the unknown parameters\n", - "$\\boldsymbol{\\beta}$, are derived under the assumption that both $\\boldsymbol{y}$ and\n", + "from the library **Scikit-Learn** (when not shrinking $\\theta_0$) for the unknown parameters\n", + "$\\boldsymbol{\\theta}$, are derived under the assumption that both $\\boldsymbol{y}$ and\n", "$\\boldsymbol{X}$ are zero centered, that is we subtract the mean values.\n", "\n", "If our predictors represent different scales, then it is important to\n", @@ -1825,7 +1825,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "eab81633", + "id": "bae38eac", "metadata": { "collapsed": false, "editable": true @@ -1852,7 +1852,7 @@ }, { "cell_type": "markdown", - "id": "0ca51a54", + "id": "61bd7f35", "metadata": { "editable": true }, @@ -1866,19 +1866,19 @@ }, { "cell_type": "markdown", - "id": "fa3a4110", + "id": "1a0f8db3", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n", + "C(\\theta_0, \\theta_1, ... , \\theta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\theta_0 - \\sum_{j=1}^{p-1} X_{ij}\\theta_j\\right)^2,.\n", "$$" ] }, { "cell_type": "markdown", - "id": "9f499302", + "id": "4b0c41b0", "metadata": { "editable": true }, @@ -1887,49 +1887,49 @@ "increased penalty for higher differences between predicted and\n", "output/target values.\n", "\n", - "What we have done is to single out the $\\beta_0$ term in the\n", + "What we have done is to single out the $\\theta_0$ term in the\n", "definition of the mean squared error (MSE). The design matrix $X$\n", "does in this case not contain any intercept column. When we take the\n", - "derivative with respect to $\\beta_0$, we want the derivative to obey" + "derivative with respect to $\\theta_0$, we want the derivative to obey" ] }, { "cell_type": "markdown", - "id": "b1f7ba52", + "id": "0aa213f5", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n", + "\\frac{\\partial C}{\\partial \\theta_j} = 0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "7ff75b7f", + "id": "d0808572", "metadata": { "editable": true }, "source": [ - "for all $j$. For $\\beta_0$ we have" + "for all $j$. For $\\theta_0$ we have" ] }, { "cell_type": "markdown", - "id": "eac37d3c", + "id": "cf1692d9", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n", + "\\frac{\\partial C}{\\partial \\theta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\theta_0 - \\sum_{j=1}^{p-1} X_{ij} \\theta_j\\right).\n", "$$" ] }, { "cell_type": "markdown", - "id": "6e653464", + "id": "cc8156be", "metadata": { "editable": true }, @@ -1939,42 +1939,42 @@ }, { "cell_type": "markdown", - "id": "8afa60a1", + "id": "cd70e62c", "metadata": { "editable": true }, "source": [ "$$\n", - "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n", + "\\sum_{i=0}^{n-1} \\theta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\theta_j.\n", "$$" ] }, { "cell_type": "markdown", - "id": "055b7975", + "id": "92c99c16", "metadata": { "editable": true }, "source": [ - "Let us specialize first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n", - "Our result for $\\beta_0$ simplifies then to" + "Let us specialize first to the case where we have only two parameters $\\theta_0$ and $\\theta_1$.\n", + "Our result for $\\theta_0$ simplifies then to" ] }, { "cell_type": "markdown", - "id": "a2bf6e5d", + "id": "cf522258", "metadata": { "editable": true }, "source": [ "$$\n", - "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n", + "n\\theta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\theta_1.\n", "$$" ] }, { "cell_type": "markdown", - "id": "e15afbd5", + "id": "cc86b9e1", "metadata": { "editable": true }, @@ -1984,19 +1984,19 @@ }, { "cell_type": "markdown", - "id": "d2f9a064", + "id": "58fd6599", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n", + "\\theta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\theta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "8e8035d8", + "id": "86b6ed8e", "metadata": { "editable": true }, @@ -2006,7 +2006,7 @@ }, { "cell_type": "markdown", - "id": "0bbcef44", + "id": "fac0cd1f", "metadata": { "editable": true }, @@ -2018,7 +2018,7 @@ }, { "cell_type": "markdown", - "id": "919462c2", + "id": "904fde54", "metadata": { "editable": true }, @@ -2028,7 +2028,7 @@ }, { "cell_type": "markdown", - "id": "3a0899ec", + "id": "21e40e22", "metadata": { "editable": true }, @@ -2040,7 +2040,7 @@ }, { "cell_type": "markdown", - "id": "9b255ec5", + "id": "1fd599b0", "metadata": { "editable": true }, @@ -2050,41 +2050,41 @@ }, { "cell_type": "markdown", - "id": "7716aca1", + "id": "69e81981", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_0 = \\mu_y - \\beta_1\\mu_{\\boldsymbol{x}_1}.\n", + "\\theta_0 = \\mu_y - \\theta_1\\mu_{\\boldsymbol{x}_1}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "24863f48", + "id": "b95e7089", "metadata": { "editable": true }, "source": [ - "In the general case with more parameters than $\\beta_0$ and $\\beta_1$, we have" + "In the general case with more parameters than $\\theta_0$ and $\\theta_1$, we have" ] }, { "cell_type": "markdown", - "id": "6c10f940", + "id": "29fb77c2", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n", + "\\theta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\theta_j.\n", "$$" ] }, { "cell_type": "markdown", - "id": "7afefe95", + "id": "6c5d2f58", "metadata": { "editable": true }, @@ -2094,19 +2094,19 @@ }, { "cell_type": "markdown", - "id": "a2e7bc1a", + "id": "cbd7ab47", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\beta_j,\n", + "\\theta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\theta_j,\n", "$$" ] }, { "cell_type": "markdown", - "id": "5ded7d78", + "id": "e1c475ee", "metadata": { "editable": true }, @@ -2116,7 +2116,7 @@ }, { "cell_type": "markdown", - "id": "bcac7366", + "id": "54ae8c6e", "metadata": { "editable": true }, @@ -2128,7 +2128,7 @@ }, { "cell_type": "markdown", - "id": "bafe8156", + "id": "c63feefb", "metadata": { "editable": true }, @@ -2140,41 +2140,41 @@ }, { "cell_type": "markdown", - "id": "7c91d280", + "id": "12498a19", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n", + "C(\\boldsymbol{\\theta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\theta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "1b81d6de", + "id": "75c5b8bb", "metadata": { "editable": true }, "source": [ - "If we minimize with respect to $\\boldsymbol{\\beta}$ we have then" + "If we minimize with respect to $\\boldsymbol{\\theta}$ we have then" ] }, { "cell_type": "markdown", - "id": "7d6d497e", + "id": "da602e6e", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n", + "\\hat{\\boldsymbol{\\theta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n", "$$" ] }, { "cell_type": "markdown", - "id": "772593b2", + "id": "3cbb6060", "metadata": { "editable": true }, @@ -2182,24 +2182,24 @@ "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n", "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n", "\n", - "For Ridge regression we need to add $\\lambda \\boldsymbol{\\beta}^T\\boldsymbol{\\beta}$ to the cost function and get then" + "For Ridge regression we need to add $\\lambda \\boldsymbol{\\theta}^T\\boldsymbol{\\theta}$ to the cost function and get then" ] }, { "cell_type": "markdown", - "id": "3edbdb56", + "id": "139ef52e", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n", + "\\hat{\\boldsymbol{\\theta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "7de9723e", + "id": "afd4d249", "metadata": { "editable": true }, @@ -2213,7 +2213,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "2e4a0363", + "id": "99a09833", "metadata": { "collapsed": false, "editable": true @@ -2233,15 +2233,15 @@ " return np.sum((y_data-y_model)**2)/n\n", "\n", "\n", - "def fit_beta(X, y):\n", + "def fit_theta(X, y):\n", " return np.linalg.pinv(X.T @ X) @ X.T @ y\n", "\n", "\n", - "true_beta = [2, 0.5, 3.7]\n", + "true_theta = [2, 0.5, 3.7]\n", "\n", "x = np.linspace(0, 1, 11)\n", "y = np.sum(\n", - " np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0\n", + " np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0\n", ") + 0.1 * np.random.normal(size=len(x))\n", "\n", "degree = 3\n", @@ -2251,15 +2251,15 @@ "for p in range(degree):\n", " X[:, p] = x ** p\n", "\n", - "beta = fit_beta(X, y)\n", + "theta = fit_theta(X, y)\n", "\n", "# Intercept is included in the design matrix\n", "skl = LinearRegression(fit_intercept=False).fit(X, y)\n", "\n", - "print(f\"True beta: {true_beta}\")\n", - "print(f\"Fitted beta: {beta}\")\n", - "print(f\"Sklearn fitted beta: {skl.coef_}\")\n", - "ypredictOwn = X @ beta\n", + "print(f\"True theta: {true_theta}\")\n", + "print(f\"Fitted theta: {theta}\")\n", + "print(f\"Sklearn fitted theta: {skl.coef_}\")\n", + "ypredictOwn = X @ theta\n", "ypredictSKL = skl.predict(X)\n", "print(f\"MSE with intercept column\")\n", "print(MSE(y,ypredictOwn))\n", @@ -2269,7 +2269,7 @@ "\n", "plt.figure()\n", "plt.scatter(x, y, label=\"Data\")\n", - "plt.plot(x, X @ beta, label=\"Fit\")\n", + "plt.plot(x, X @ theta, label=\"Fit\")\n", "plt.plot(x, skl.predict(X), label=\"Sklearn (fit_intercept=False)\")\n", "\n", "\n", @@ -2286,21 +2286,21 @@ "y_offset = np.average(y, axis=0)\n", "X_offset = np.average(X, axis=0)\n", "\n", - "beta = fit_beta(X - X_offset, y - y_offset)\n", - "intercept = np.mean(y_offset - X_offset @ beta)\n", + "theta = fit_theta(X - X_offset, y - y_offset)\n", + "intercept = np.mean(y_offset - X_offset @ theta)\n", "\n", "print(f\"Manual intercept: {intercept}\")\n", - "print(f\"Fitted beta (wiothout intercept): {beta}\")\n", + "print(f\"Fitted theta (wiothout intercept): {theta}\")\n", "print(f\"Sklearn intercept: {skl.intercept_}\")\n", - "print(f\"Sklearn fitted beta (without intercept): {skl.coef_}\")\n", - "ypredictOwn = X @ beta\n", + "print(f\"Sklearn fitted theta (without intercept): {skl.coef_}\")\n", + "ypredictOwn = X @ theta\n", "ypredictSKL = skl.predict(X)\n", "print(f\"MSE with Manual intercept\")\n", "print(MSE(y,ypredictOwn+intercept))\n", "print(f\"MSE with Sklearn intercept\")\n", "print(MSE(y,ypredictSKL))\n", "\n", - "plt.plot(x, X @ beta + intercept, \"--\", label=\"Fit (manual intercept)\")\n", + "plt.plot(x, X @ theta + intercept, \"--\", label=\"Fit (manual intercept)\")\n", "plt.plot(x, skl.predict(X), \"--\", label=\"Sklearn (fit_intercept=True)\")\n", "plt.grid()\n", "plt.legend()\n", @@ -2310,7 +2310,7 @@ }, { "cell_type": "markdown", - "id": "c6d3a071", + "id": "1bacfcef", "metadata": { "editable": true }, @@ -2323,7 +2323,7 @@ "the way we treat the intercept may give a larger or smaller MSE,\n", "meaning that the MSE can be penalized by the value of the\n", "intercept. Not including the intercept in the fit, means that the\n", - "regularization term does not include $\\beta_0$. For different values\n", + "regularization term does not include $\\theta_0$. For different values\n", "of $\\lambda$, this may lead to different MSE values. \n", "\n", "To remind the reader, the regularization term, with the intercept in Ridge regression, is given by" @@ -2331,19 +2331,19 @@ }, { "cell_type": "markdown", - "id": "db803b77", + "id": "8061073e", "metadata": { "editable": true }, "source": [ "$$\n", - "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\beta_j^2,\n", + "\\lambda \\vert\\vert \\boldsymbol{\\theta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\theta_j^2,\n", "$$" ] }, { "cell_type": "markdown", - "id": "d7a9c9d8", + "id": "fcea892f", "metadata": { "editable": true }, @@ -2353,19 +2353,19 @@ }, { "cell_type": "markdown", - "id": "e08af763", + "id": "af6d400e", "metadata": { "editable": true }, "source": [ "$$\n", - "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\beta_j^2.\n", + "\\lambda \\vert\\vert \\boldsymbol{\\theta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\theta_j^2.\n", "$$" ] }, { "cell_type": "markdown", - "id": "10fe24e4", + "id": "1bfb0c9e", "metadata": { "editable": true }, @@ -2375,19 +2375,19 @@ }, { "cell_type": "markdown", - "id": "6feb3bdd", + "id": "3b87c68e", "metadata": { "editable": true }, "source": [ "$$\n", - "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\beta_j\\vert.\n", + "\\lambda \\vert\\vert \\boldsymbol{\\theta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\theta_j\\vert.\n", "$$" ] }, { "cell_type": "markdown", - "id": "657a0777", + "id": "eae8012d", "metadata": { "editable": true }, @@ -2405,7 +2405,7 @@ { "cell_type": "code", "execution_count": 12, - "id": "9f21317d", + "id": "e02c5a51", "metadata": { "collapsed": false, "editable": true @@ -2448,20 +2448,20 @@ "lambdas = np.logspace(-4, 2, nlambdas)\n", "for i in range(nlambdas):\n", " lmb = lambdas[i]\n", - " OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n", + " OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n", " # Note: we include the intercept column and no scaling\n", " RegRidge = linear_model.Ridge(lmb,fit_intercept=False)\n", " RegRidge.fit(X_train,y_train)\n", " # and then make the prediction\n", - " ytildeOwnRidge = X_train @ OwnRidgeBeta\n", - " ypredictOwnRidge = X_test @ OwnRidgeBeta\n", + " ytildeOwnRidge = X_train @ OwnRidgeTheta\n", + " ypredictOwnRidge = X_test @ OwnRidgeTheta\n", " ytildeRidge = RegRidge.predict(X_train)\n", " ypredictRidge = RegRidge.predict(X_test)\n", " MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)\n", " MSERidgePredict[i] = MSE(y_test,ypredictRidge)\n", - " print(\"Beta values for own Ridge implementation\")\n", - " print(OwnRidgeBeta)\n", - " print(\"Beta values for Scikit-Learn Ridge implementation\")\n", + " print(\"Theta values for own Ridge implementation\")\n", + " print(OwnRidgeTheta)\n", + " print(\"Theta values for Scikit-Learn Ridge implementation\")\n", " print(RegRidge.coef_)\n", " print(\"MSE values for own Ridge implementation\")\n", " print(MSEOwnRidgePredict[i])\n", @@ -2481,7 +2481,7 @@ }, { "cell_type": "markdown", - "id": "20de82fd", + "id": "df4ec33b", "metadata": { "editable": true }, @@ -2495,7 +2495,7 @@ { "cell_type": "code", "execution_count": 13, - "id": "4138ed50", + "id": "019f59bd", "metadata": { "collapsed": false, "editable": true @@ -2549,18 +2549,18 @@ "lambdas = np.logspace(-4, 2, nlambdas)\n", "for i in range(nlambdas):\n", " lmb = lambdas[i]\n", - " OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)\n", - " intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data\n", + " OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)\n", + " intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data\n", " #Add intercept to prediction\n", - " ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler \n", + " ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler \n", " RegRidge = linear_model.Ridge(lmb)\n", " RegRidge.fit(X_train,y_train)\n", " ypredictRidge = RegRidge.predict(X_test)\n", " MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)\n", " MSERidgePredict[i] = MSE(y_test,ypredictRidge)\n", - " print(\"Beta values for own Ridge implementation\")\n", - " print(OwnRidgeBeta) #Intercept is given by mean of target variable\n", - " print(\"Beta values for Scikit-Learn Ridge implementation\")\n", + " print(\"Theta values for own Ridge implementation\")\n", + " print(OwnRidgeTheta) #Intercept is given by mean of target variable\n", + " print(\"Theta values for Scikit-Learn Ridge implementation\")\n", " print(RegRidge.coef_)\n", " print('Intercept from own implementation:')\n", " print(intercept_)\n", @@ -2584,7 +2584,7 @@ }, { "cell_type": "markdown", - "id": "7e9dfda5", + "id": "11a0c8f8", "metadata": { "editable": true }, @@ -2592,7 +2592,7 @@ "We see here, when compared to the code which includes explicitely the\n", "intercept column, that our MSE value is actually smaller. This is\n", "because the regularization term does not include the intercept value\n", - "$\\beta_0$ in the fitting. This applies to Lasso regularization as\n", + "$\\theta_0$ in the fitting. This applies to Lasso regularization as\n", "well. It means that our optimization is now done only with the\n", "centered matrix and/or vector that enter the fitting procedure. Note\n", "also that the problem with the intercept occurs mainly in these type\n", @@ -2603,7 +2603,7 @@ }, { "cell_type": "markdown", - "id": "a7eb252c", + "id": "fca71b1a", "metadata": { "editable": true }, @@ -2616,7 +2616,7 @@ }, { "cell_type": "markdown", - "id": "990ff280", + "id": "bea82853", "metadata": { "editable": true }, @@ -2634,7 +2634,7 @@ }, { "cell_type": "markdown", - "id": "5ea46b31", + "id": "d92e0742", "metadata": { "editable": true }, @@ -2651,7 +2651,7 @@ { "cell_type": "code", "execution_count": 14, - "id": "fa60aaab", + "id": "039e1f8c", "metadata": { "collapsed": false, "editable": true @@ -2682,7 +2682,7 @@ }, { "cell_type": "markdown", - "id": "b9e8214e", + "id": "94a8e6b6", "metadata": { "editable": true }, @@ -2698,7 +2698,7 @@ }, { "cell_type": "markdown", - "id": "a578cd63", + "id": "0ba92480", "metadata": { "editable": true }, @@ -2716,7 +2716,7 @@ }, { "cell_type": "markdown", - "id": "d90dcfee", + "id": "313c4354", "metadata": { "editable": true }, @@ -2728,7 +2728,7 @@ }, { "cell_type": "markdown", - "id": "c4eec69a", + "id": "da5f96ae", "metadata": { "editable": true }, @@ -2746,7 +2746,7 @@ }, { "cell_type": "markdown", - "id": "ae46a11b", + "id": "14375be5", "metadata": { "editable": true }, @@ -2758,7 +2758,7 @@ }, { "cell_type": "markdown", - "id": "87ebb9b8", + "id": "58f91570", "metadata": { "editable": true }, @@ -2768,7 +2768,7 @@ "\n", "$$\n", "\\begin{equation}\n", - " \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon},\n", + " \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\theta} + \\boldsymbol{\\epsilon},\n", "\\label{_auto4} \\tag{4}\n", "\\end{equation}\n", "$$" @@ -2776,7 +2776,7 @@ }, { "cell_type": "markdown", - "id": "856efd3f", + "id": "820108ea", "metadata": { "editable": true }, @@ -2787,7 +2787,7 @@ { "cell_type": "code", "execution_count": 15, - "id": "5ae2828f", + "id": "ec4efa81", "metadata": { "collapsed": false, "editable": true @@ -2803,7 +2803,7 @@ }, { "cell_type": "markdown", - "id": "0d6201d9", + "id": "aa3656f3", "metadata": { "editable": true }, @@ -2813,7 +2813,7 @@ }, { "cell_type": "markdown", - "id": "4af380a7", + "id": "0c1bbe7c", "metadata": { "editable": true }, @@ -2823,7 +2823,7 @@ "\n", "$$\n", "\\begin{equation}\n", - " C(\\boldsymbol{X}, \\boldsymbol{\\beta})= \\frac{1}{n}\\left\\{(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})\\right\\}.\n", + " C(\\boldsymbol{X}, \\boldsymbol{\\theta})= \\frac{1}{n}\\left\\{(\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y})\\right\\}.\n", "\\label{_auto5} \\tag{5}\n", "\\end{equation}\n", "$$" @@ -2831,30 +2831,30 @@ }, { "cell_type": "markdown", - "id": "8e8eefa3", + "id": "784bc4e4", "metadata": { "editable": true }, "source": [ - "We then find the extremal point of $C$ by taking the derivative with respect to $\\boldsymbol{\\beta}$ as discussed above.\n", - "This yields the expression for $\\boldsymbol{\\beta}$ to be" + "We then find the extremal point of $C$ by taking the derivative with respect to $\\boldsymbol{\\theta}$ as discussed above.\n", + "This yields the expression for $\\boldsymbol{\\theta}$ to be" ] }, { "cell_type": "markdown", - "id": "a1c0765c", + "id": "db1ec5fe", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\beta} = \\frac{\\boldsymbol{X}^T \\boldsymbol{y}}{\\boldsymbol{X}^T \\boldsymbol{X}},\n", + "\\boldsymbol{\\theta} = \\frac{\\boldsymbol{X}^T \\boldsymbol{y}}{\\boldsymbol{X}^T \\boldsymbol{X}},\n", "$$" ] }, { "cell_type": "markdown", - "id": "c754e28b", + "id": "5d2e6b17", "metadata": { "editable": true }, @@ -2868,7 +2868,7 @@ { "cell_type": "code", "execution_count": 16, - "id": "ff056cf9", + "id": "6a242f2e", "metadata": { "collapsed": false, "editable": true @@ -2887,7 +2887,7 @@ }, { "cell_type": "markdown", - "id": "524f259e", + "id": "329b50d2", "metadata": { "editable": true }, @@ -2895,24 +2895,24 @@ "Doing the inversion directly turns out to be a bad idea since the matrix\n", "$\\boldsymbol{X}^T\\boldsymbol{X}$ is singular. An alternative approach is to use the **singular\n", "value decomposition**. Using the definition of the Moore-Penrose\n", - "pseudoinverse we can write the equation for $\\boldsymbol{\\beta}$ as" + "pseudoinverse we can write the equation for $\\boldsymbol{\\theta}$ as" ] }, { "cell_type": "markdown", - "id": "52be4553", + "id": "2b544864", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\beta} = \\boldsymbol{X}^{+}\\boldsymbol{y},\n", + "\\boldsymbol{\\theta} = \\boldsymbol{X}^{+}\\boldsymbol{y},\n", "$$" ] }, { "cell_type": "markdown", - "id": "c37ff09a", + "id": "bf0d128d", "metadata": { "editable": true }, @@ -2922,7 +2922,7 @@ }, { "cell_type": "markdown", - "id": "ea49a105", + "id": "67e527de", "metadata": { "editable": true }, @@ -2934,7 +2934,7 @@ }, { "cell_type": "markdown", - "id": "bf31067a", + "id": "d4ef2ece", "metadata": { "editable": true }, @@ -2947,7 +2947,7 @@ }, { "cell_type": "markdown", - "id": "5ef2c839", + "id": "c4c467f8", "metadata": { "editable": true }, @@ -2957,7 +2957,7 @@ "\n", "$$\n", "\\begin{equation}\n", - " \\boldsymbol{\\beta} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^{+} \\boldsymbol{U}^T \\boldsymbol{y}.\n", + " \\boldsymbol{\\theta} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^{+} \\boldsymbol{U}^T \\boldsymbol{y}.\n", "\\label{_auto6} \\tag{6}\n", "\\end{equation}\n", "$$" @@ -2965,7 +2965,7 @@ }, { "cell_type": "markdown", - "id": "96bbf134", + "id": "ffd1ba3d", "metadata": { "editable": true }, @@ -2981,7 +2981,7 @@ { "cell_type": "code", "execution_count": 17, - "id": "5e729b1e", + "id": "9cdf3698", "metadata": { "collapsed": false, "editable": true @@ -2996,19 +2996,19 @@ { "cell_type": "code", "execution_count": 18, - "id": "80b923d1", + "id": "164ccc0c", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ - "beta = ols_svd(X_train_own,y_train)" + "theta = ols_svd(X_train_own,y_train)" ] }, { "cell_type": "markdown", - "id": "b4cf2f1a", + "id": "a23f2826", "metadata": { "editable": true }, @@ -3019,19 +3019,19 @@ { "cell_type": "code", "execution_count": 19, - "id": "93be2c0d", + "id": "a164aca2", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ - "J = beta[1:].reshape(L, L)" + "J = theta[1:].reshape(L, L)" ] }, { "cell_type": "markdown", - "id": "ae869089", + "id": "3bb35ae1", "metadata": { "editable": true }, @@ -3042,7 +3042,7 @@ { "cell_type": "code", "execution_count": 20, - "id": "8d6d8152", + "id": "93445ba4", "metadata": { "collapsed": false, "editable": true @@ -3061,7 +3061,7 @@ }, { "cell_type": "markdown", - "id": "8092ac84", + "id": "f602ed47", "metadata": { "editable": true }, @@ -3084,7 +3084,7 @@ }, { "cell_type": "markdown", - "id": "d9251cde", + "id": "86d25ab0", "metadata": { "editable": true }, @@ -3102,7 +3102,7 @@ }, { "cell_type": "markdown", - "id": "abe454c9", + "id": "2008a25b", "metadata": { "editable": true }, @@ -3115,7 +3115,7 @@ { "cell_type": "code", "execution_count": 21, - "id": "89f8fbd0", + "id": "01b2b75a", "metadata": { "collapsed": false, "editable": true @@ -3147,7 +3147,7 @@ }, { "cell_type": "markdown", - "id": "e06639d4", + "id": "d6b5a067", "metadata": { "editable": true }, @@ -3157,7 +3157,7 @@ }, { "cell_type": "markdown", - "id": "43ef0f40", + "id": "497d84f1", "metadata": { "editable": true }, @@ -3175,7 +3175,7 @@ }, { "cell_type": "markdown", - "id": "523dc03b", + "id": "38cc642a", "metadata": { "editable": true }, @@ -3187,7 +3187,7 @@ }, { "cell_type": "markdown", - "id": "0f389eef", + "id": "bec7f36d", "metadata": { "editable": true }, @@ -3205,7 +3205,7 @@ }, { "cell_type": "markdown", - "id": "b6e00f24", + "id": "14ad8f62", "metadata": { "editable": true }, @@ -3217,7 +3217,7 @@ }, { "cell_type": "markdown", - "id": "5ea1084d", + "id": "4ccac4ea", "metadata": { "editable": true }, @@ -3227,7 +3227,7 @@ "\n", "$$\n", "\\begin{equation}\n", - " \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon}.\n", + " \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\theta} + \\boldsymbol{\\epsilon}.\n", "\\label{_auto10} \\tag{10}\n", "\\end{equation}\n", "$$" @@ -3235,7 +3235,7 @@ }, { "cell_type": "markdown", - "id": "5551bb05", + "id": "cabb6ced", "metadata": { "editable": true }, @@ -3246,7 +3246,7 @@ { "cell_type": "code", "execution_count": 22, - "id": "f5dd7795", + "id": "ce90ccbe", "metadata": { "collapsed": false, "editable": true @@ -3272,7 +3272,7 @@ }, { "cell_type": "markdown", - "id": "f36c807b", + "id": "6f6d672f", "metadata": { "editable": true }, @@ -3283,7 +3283,7 @@ { "cell_type": "code", "execution_count": 23, - "id": "eb701dd4", + "id": "945777f9", "metadata": { "collapsed": false, "editable": true @@ -3295,7 +3295,7 @@ }, { "cell_type": "markdown", - "id": "bd356ada", + "id": "ffc8317f", "metadata": { "editable": true }, @@ -3306,7 +3306,7 @@ { "cell_type": "code", "execution_count": 24, - "id": "ae737db8", + "id": "e17c598f", "metadata": { "collapsed": false, "editable": true @@ -3318,7 +3318,7 @@ }, { "cell_type": "markdown", - "id": "d152a32a", + "id": "3396f010", "metadata": { "editable": true }, @@ -3329,7 +3329,7 @@ { "cell_type": "code", "execution_count": 25, - "id": "8d713977", + "id": "311d3cea", "metadata": { "collapsed": false, "editable": true @@ -3348,7 +3348,7 @@ }, { "cell_type": "markdown", - "id": "ede30c18", + "id": "13705eed", "metadata": { "editable": true }, @@ -3358,13 +3358,13 @@ "Having explored the ordinary least squares we move on to ridge\n", "regression. In ridge regression we include a **regularizer**. This\n", "involves a new cost function which leads to a new estimate for the\n", - "weights $\\boldsymbol{\\beta}$. This results in a penalized regression problem. The\n", + "weights $\\boldsymbol{\\theta}$. This results in a penalized regression problem. The\n", "cost function is given by" ] }, { "cell_type": "markdown", - "id": "eac772e9", + "id": "4cd9530c", "metadata": { "editable": true }, @@ -3374,7 +3374,7 @@ "\n", "$$\n", "\\begin{equation}\n", - " C(\\boldsymbol{X}, \\boldsymbol{\\beta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y}) + \\lambda \\boldsymbol{\\beta}^T\\boldsymbol{\\beta}.\n", + " C(\\boldsymbol{X}, \\boldsymbol{\\theta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y}) + \\lambda \\boldsymbol{\\theta}^T\\boldsymbol{\\theta}.\n", "\\label{_auto11} \\tag{11}\n", "\\end{equation}\n", "$$" @@ -3383,7 +3383,7 @@ { "cell_type": "code", "execution_count": 26, - "id": "b66675f0", + "id": "d1c9a462", "metadata": { "collapsed": false, "editable": true @@ -3406,7 +3406,7 @@ }, { "cell_type": "markdown", - "id": "71aa37b5", + "id": "97e6303c", "metadata": { "editable": true }, @@ -3416,7 +3416,7 @@ }, { "cell_type": "markdown", - "id": "8b70ef66", + "id": "8631d307", "metadata": { "editable": true }, @@ -3426,7 +3426,7 @@ "\n", "$$\n", "\\begin{equation}\n", - " C(\\boldsymbol{X}, \\boldsymbol{\\beta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y}) + \\lambda \\sqrt{\\boldsymbol{\\beta}^T\\boldsymbol{\\beta}}.\n", + " C(\\boldsymbol{X}, \\boldsymbol{\\theta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y}) + \\lambda \\sqrt{\\boldsymbol{\\theta}^T\\boldsymbol{\\theta}}.\n", "\\label{_auto12} \\tag{12}\n", "\\end{equation}\n", "$$" @@ -3434,7 +3434,7 @@ }, { "cell_type": "markdown", - "id": "079981ff", + "id": "dc2a06fb", "metadata": { "editable": true }, @@ -3445,7 +3445,7 @@ { "cell_type": "code", "execution_count": 27, - "id": "c8d4a180", + "id": "2f3333c6", "metadata": { "collapsed": false, "editable": true @@ -3467,7 +3467,7 @@ }, { "cell_type": "markdown", - "id": "57839941", + "id": "692844ac", "metadata": { "editable": true }, @@ -3482,7 +3482,7 @@ { "cell_type": "code", "execution_count": 28, - "id": "0e3bbb4e", + "id": "d40c4d10", "metadata": { "collapsed": false, "editable": true @@ -3529,7 +3529,7 @@ }, { "cell_type": "markdown", - "id": "14711abd", + "id": "965b51d2", "metadata": { "editable": true }, @@ -3550,7 +3550,7 @@ { "cell_type": "code", "execution_count": 29, - "id": "9ff29a72", + "id": "04fe5f87", "metadata": { "collapsed": false, "editable": true @@ -3591,7 +3591,7 @@ }, { "cell_type": "markdown", - "id": "9c10be56", + "id": "845d9279", "metadata": { "editable": true }, @@ -3603,7 +3603,7 @@ }, { "cell_type": "markdown", - "id": "efce3b63", + "id": "6d8674ba", "metadata": { "editable": true }, @@ -3629,7 +3629,7 @@ }, { "cell_type": "markdown", - "id": "ef5a4df0", + "id": "b26b187d", "metadata": { "editable": true }, @@ -3644,7 +3644,7 @@ }, { "cell_type": "markdown", - "id": "b085eff4", + "id": "76e8040d", "metadata": { "editable": true }, @@ -3674,7 +3674,7 @@ { "cell_type": "code", "execution_count": 30, - "id": "5056dccb", + "id": "7608edfe", "metadata": { "collapsed": false, "editable": true @@ -3724,7 +3724,7 @@ }, { "cell_type": "markdown", - "id": "f5e72aef", + "id": "c0d0b45f", "metadata": { "editable": true }, @@ -3741,13 +3741,13 @@ "value decomposition from e.g., **numpy** ) or use your code from\n", "homeworks 1 and 2 and perform a standard least square regression\n", "analysis using polynomials in $x$ and $y$ up to fifth order. Find the\n", - "[confidence intervals](https://en.wikipedia.org/wiki/Confidence_interval) of the parameters (estimators) $\\beta$ by computing their\n", + "[confidence intervals](https://en.wikipedia.org/wiki/Confidence_interval) of the parameters (estimators) $\\theta$ by computing their\n", "variances, evaluate the Mean Squared error (MSE)" ] }, { "cell_type": "markdown", - "id": "0ba33237", + "id": "76861250", "metadata": { "editable": true }, @@ -3760,7 +3760,7 @@ }, { "cell_type": "markdown", - "id": "1f9af2ef", + "id": "90074ad8", "metadata": { "editable": true }, @@ -3772,7 +3772,7 @@ }, { "cell_type": "markdown", - "id": "77d5b0f2", + "id": "31d7a9ad", "metadata": { "editable": true }, @@ -3784,7 +3784,7 @@ }, { "cell_type": "markdown", - "id": "6c1f9df9", + "id": "b6bcc7b7", "metadata": { "editable": true }, @@ -3794,7 +3794,7 @@ }, { "cell_type": "markdown", - "id": "1d06102c", + "id": "45f3893f", "metadata": { "editable": true }, @@ -3806,7 +3806,7 @@ }, { "cell_type": "markdown", - "id": "8fb40b08", + "id": "8fea6a0a", "metadata": { "editable": true }, @@ -3831,7 +3831,7 @@ }, { "cell_type": "markdown", - "id": "1701de47", + "id": "6283b437", "metadata": { "editable": true }, @@ -3863,7 +3863,7 @@ }, { "cell_type": "markdown", - "id": "15110cdf", + "id": "62f77e00", "metadata": { "editable": true }, @@ -3875,7 +3875,7 @@ }, { "cell_type": "markdown", - "id": "33046595", + "id": "7380e759", "metadata": { "editable": true }, @@ -3885,28 +3885,28 @@ "\n", "In our derivation of the ordinary least squares method we defined then\n", "an approximation to the function $f$ in terms of the parameters\n", - "$\\boldsymbol{\\beta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n", - "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\beta}$.\n", + "$\\boldsymbol{\\theta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n", + "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\theta}$.\n", "\n", - "The parameters $\\boldsymbol{\\beta}$ are in turn found by optimizing the means\n", + "The parameters $\\boldsymbol{\\theta}$ are in turn found by optimizing the means\n", "squared error via the so-called cost function" ] }, { "cell_type": "markdown", - "id": "84527747", + "id": "b5c040e0", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n", + "C(\\boldsymbol{X},\\boldsymbol{\\theta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n", "$$" ] }, { "cell_type": "markdown", - "id": "da2f876d", + "id": "2cf7f08f", "metadata": { "editable": true }, @@ -3918,7 +3918,7 @@ }, { "cell_type": "markdown", - "id": "1845a7dc", + "id": "8cf351f9", "metadata": { "editable": true }, @@ -3930,7 +3930,7 @@ }, { "cell_type": "markdown", - "id": "d2f1d7d7", + "id": "53fff2ad", "metadata": { "editable": true }, @@ -3950,7 +3950,7 @@ }, { "cell_type": "markdown", - "id": "e898c902", + "id": "89dec1fb", "metadata": { "editable": true }, @@ -3975,7 +3975,7 @@ }, { "cell_type": "markdown", - "id": "d23dc734", + "id": "2d5f7364", "metadata": { "editable": true }, @@ -3995,7 +3995,7 @@ }, { "cell_type": "markdown", - "id": "39a35330", + "id": "d56e6dc8", "metadata": { "editable": true }, @@ -4012,7 +4012,7 @@ }, { "cell_type": "markdown", - "id": "44d9e821", + "id": "608453fd", "metadata": { "editable": true }, @@ -4040,7 +4040,7 @@ { "cell_type": "code", "execution_count": 31, - "id": "0a4e6d7e", + "id": "9399d087", "metadata": { "collapsed": false, "editable": true @@ -4052,7 +4052,7 @@ }, { "cell_type": "markdown", - "id": "16a73292", + "id": "ae1d9c8f", "metadata": { "editable": true }, @@ -4064,7 +4064,7 @@ { "cell_type": "code", "execution_count": 32, - "id": "168356a4", + "id": "381e9da0", "metadata": { "collapsed": false, "editable": true @@ -4092,7 +4092,7 @@ }, { "cell_type": "markdown", - "id": "e9ec3e68", + "id": "6e1c4af1", "metadata": { "editable": true }, diff --git a/doc/LectureNotes/_build/html/_sources/chapter4.ipynb b/doc/LectureNotes/_build/html/_sources/chapter4.ipynb index bdfb97b2b..934d454cf 100644 --- a/doc/LectureNotes/_build/html/_sources/chapter4.ipynb +++ b/doc/LectureNotes/_build/html/_sources/chapter4.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "f63b8c1a", + "id": "225f0ca7", "metadata": { "editable": true }, @@ -13,7 +13,7 @@ }, { "cell_type": "markdown", - "id": "9a284cf5", + "id": "a48a2920", "metadata": { "editable": true }, @@ -23,7 +23,7 @@ }, { "cell_type": "markdown", - "id": "d78b73e2", + "id": "90b6e822", "metadata": { "editable": true }, @@ -38,7 +38,7 @@ "analytical expressions for standard ordinary Least Squares or Ridge\n", "regression (in terms of matrices to invert) for several quantities,\n", "ranging from the variance and thereby the confidence intervals of the\n", - "optimal parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n", + "optimal parameters $\\hat{\\theta}$ to the mean squared error. If we can invert\n", "the product of the design matrices, linear regression gives then a\n", "simple recipe for fitting our data.\n", "\n", @@ -61,7 +61,7 @@ "Logistic regression will also serve as our stepping stone towards\n", "neural network algorithms and supervised deep learning. For logistic\n", "learning, the minimization of the cost function leads to a non-linear\n", - "equation in the parameters $\\hat{\\beta}$. The optimization of the\n", + "equation in the parameters $\\hat{\\theta}$. The optimization of the\n", "problem calls therefore for minimization algorithms. This forms the\n", "bottle neck of all machine learning algorithms, namely how to find\n", "reliable minima of a multi-variable function. This leads us to the\n", @@ -75,7 +75,7 @@ }, { "cell_type": "markdown", - "id": "d0c829b4", + "id": "08ebdf8e", "metadata": { "editable": true }, @@ -100,7 +100,7 @@ }, { "cell_type": "markdown", - "id": "f1f4d5f2", + "id": "f08eca95", "metadata": { "editable": true }, @@ -112,7 +112,7 @@ }, { "cell_type": "markdown", - "id": "8e2f836b", + "id": "d82a2a29", "metadata": { "editable": true }, @@ -128,7 +128,7 @@ }, { "cell_type": "markdown", - "id": "6b58f1ba", + "id": "467f1d23", "metadata": { "editable": true }, @@ -138,7 +138,7 @@ "\n", "$$\n", "\\begin{equation}\n", - "\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon},\n", + "\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\theta} + \\boldsymbol{\\epsilon},\n", "\\label{_auto1} \\tag{1}\n", "\\end{equation}\n", "$$" @@ -146,13 +146,13 @@ }, { "cell_type": "markdown", - "id": "54e3604b", + "id": "0f2560d0", "metadata": { "editable": true }, "source": [ "where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n", - "$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors.\n", + "$n\\times p$ design matrix and $\\boldsymbol{\\theta}$ represents our estimators/predictors.\n", "\n", "The main problem with our function is that it takes values on the\n", "entire real axis. In the case of logistic regression, however, the\n", @@ -175,7 +175,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "cd1dd1c5", + "id": "3916f494", "metadata": { "collapsed": false, "editable": true @@ -242,7 +242,7 @@ }, { "cell_type": "markdown", - "id": "2e06ee82", + "id": "df52e736", "metadata": { "editable": true }, @@ -253,7 +253,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "a774f001", + "id": "07b5c0a7", "metadata": { "collapsed": false, "editable": true @@ -272,7 +272,7 @@ }, { "cell_type": "markdown", - "id": "e102fa56", + "id": "25ce17fc", "metadata": { "editable": true }, @@ -283,19 +283,19 @@ }, { "cell_type": "markdown", - "id": "75d995ba", + "id": "818e2f55", "metadata": { "editable": true }, "source": [ "$$\n", - "f(y_i\\vert x_i)=\\beta_0+\\beta_1 x_i.\n", + "f(y_i\\vert x_i)=\\theta_0+\\theta_1 x_i.\n", "$$" ] }, { "cell_type": "markdown", - "id": "e2cacca5", + "id": "b2616b20", "metadata": { "editable": true }, @@ -314,7 +314,7 @@ }, { "cell_type": "markdown", - "id": "af9ac586", + "id": "ab85db3c", "metadata": { "editable": true }, @@ -336,7 +336,7 @@ }, { "cell_type": "markdown", - "id": "80aa3cbe", + "id": "efbd4772", "metadata": { "editable": true }, @@ -348,7 +348,7 @@ }, { "cell_type": "markdown", - "id": "28d49915", + "id": "1b9231ea", "metadata": { "editable": true }, @@ -358,7 +358,7 @@ }, { "cell_type": "markdown", - "id": "eeff6cd9", + "id": "ad6acb90", "metadata": { "editable": true }, @@ -371,7 +371,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "5ecec05e", + "id": "bf66d0dd", "metadata": { "collapsed": false, "editable": true @@ -436,56 +436,56 @@ }, { "cell_type": "markdown", - "id": "5802efe3", + "id": "363760d3", "metadata": { "editable": true }, "source": [ - "We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\beta$ in our fitting of the Sigmoid function, that is we define probabilities" + "We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\theta$ in our fitting of the Sigmoid function, that is we define probabilities" ] }, { "cell_type": "markdown", - "id": "f85d14ea", + "id": "eefe7b8d", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{align*}\n", - "p(y_i=1|x_i,\\boldsymbol{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n", - "p(y_i=0|x_i,\\boldsymbol{\\beta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\beta}),\n", + "p(y_i=1|x_i,\\boldsymbol{\\theta}) &= \\frac{\\exp{(\\theta_0+\\theta_1x_i)}}{1+\\exp{(\\theta_0+\\theta_1x_i)}},\\nonumber\\\\\n", + "p(y_i=0|x_i,\\boldsymbol{\\theta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\theta}),\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", - "id": "6c4e0334", + "id": "102fb347", "metadata": { "editable": true }, "source": [ - "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n", + "where $\\boldsymbol{\\theta}$ are the weights we wish to extract from data, in our case $\\theta_0$ and $\\theta_1$. \n", "\n", "Note that we used" ] }, { "cell_type": "markdown", - "id": "df7facc9", + "id": "0b230504", "metadata": { "editable": true }, "source": [ "$$\n", - "p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n", + "p(y_i=0\\vert x_i, \\boldsymbol{\\theta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "a920e6d8", + "id": "669b4c9f", "metadata": { "editable": true }, @@ -500,21 +500,21 @@ }, { "cell_type": "markdown", - "id": "a780bfe1", + "id": "5dc36460", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{align*}\n", - "P(\\mathcal{D}|\\boldsymbol{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n", + "P(\\mathcal{D}|\\boldsymbol{\\theta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\theta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\theta}))\\right]^{1-y_i}\\nonumber \\\\\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", - "id": "9f810a44", + "id": "0874a9c6", "metadata": { "editable": true }, @@ -524,19 +524,19 @@ }, { "cell_type": "markdown", - "id": "3fac4ef4", + "id": "212397d3", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n", + "\\mathcal{C}(\\boldsymbol{\\theta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\theta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\theta}))\\right]\\right).\n", "$$" ] }, { "cell_type": "markdown", - "id": "75aa2c14", + "id": "f99396bf", "metadata": { "editable": true }, @@ -546,42 +546,42 @@ }, { "cell_type": "markdown", - "id": "780f2038", + "id": "fd5e7b5f", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", + "\\mathcal{C}(\\boldsymbol{\\theta}) = \\sum_{i=1}^n \\left(y_i(\\theta_0+\\theta_1x_i) -\\log{(1+\\exp{(\\theta_0+\\theta_1x_i)})}\\right).\n", "$$" ] }, { "cell_type": "markdown", - "id": "c8c940aa", + "id": "80dbca63", "metadata": { "editable": true }, "source": [ - "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n", + "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\theta$.\n", "Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that" ] }, { "cell_type": "markdown", - "id": "9d4a527b", + "id": "f5d6c58f", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", + "\\mathcal{C}(\\boldsymbol{\\theta})=-\\sum_{i=1}^n \\left(y_i(\\theta_0+\\theta_1x_i) -\\log{(1+\\exp{(\\theta_0+\\theta_1x_i)})}\\right).\n", "$$" ] }, { "cell_type": "markdown", - "id": "c1706606", + "id": "57e33c81", "metadata": { "editable": true }, @@ -589,28 +589,28 @@ "This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n", "in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.\n", "\n", - "The cross entropy is a convex function of the weights $\\boldsymbol{\\beta}$ and,\n", + "The cross entropy is a convex function of the weights $\\boldsymbol{\\theta}$ and,\n", "therefore, any local minimizer is a global minimizer. \n", "\n", "Minimizing this\n", - "cost function with respect to the two parameters $\\beta_0$ and $\\beta_1$ we obtain" + "cost function with respect to the two parameters $\\theta_0$ and $\\theta_1$ we obtain" ] }, { "cell_type": "markdown", - "id": "fe8fb387", + "id": "1833cc5d", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\theta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\theta_0+\\theta_1x_i)}}{1+\\exp{(\\theta_0+\\theta_1x_i)}}\\right),\n", "$$" ] }, { "cell_type": "markdown", - "id": "bb564e81", + "id": "ea34e8e6", "metadata": { "editable": true }, @@ -620,67 +620,67 @@ }, { "cell_type": "markdown", - "id": "700c5443", + "id": "f1bc373e", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\theta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\theta_0+\\theta_1x_i)}}{1+\\exp{(\\theta_0+\\theta_1x_i)}}\\right).\n", "$$" ] }, { "cell_type": "markdown", - "id": "f442c956", + "id": "617c6d86", "metadata": { "editable": true }, "source": [ "Let us now define a vector $\\boldsymbol{y}$ with $n$ elements $y_i$, an\n", "$n\\times p$ matrix $\\boldsymbol{X}$ which contains the $x_i$ values and a\n", - "vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\beta})$. We can rewrite in a more compact form the first\n", + "vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\theta})$. We can rewrite in a more compact form the first\n", "derivative of cost function as" ] }, { "cell_type": "markdown", - "id": "83937f6f", + "id": "e79426c6", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", "$$" ] }, { "cell_type": "markdown", - "id": "71819c1d", + "id": "b1dd9451", "metadata": { "editable": true }, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", - "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" + "$p(y_i\\vert x_i,\\boldsymbol{\\theta})(1-p(y_i\\vert x_i,\\boldsymbol{\\theta})$, we can obtain a compact expression of the second derivative as" ] }, { "cell_type": "markdown", - "id": "23dfd975", + "id": "ac6500cc", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", + "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}\\partial \\boldsymbol{\\theta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "6ca9f74b", + "id": "12f4d5d0", "metadata": { "editable": true }, @@ -690,41 +690,41 @@ }, { "cell_type": "markdown", - "id": "ddb3d93a", + "id": "b4733616", "metadata": { "editable": true }, "source": [ "$$\n", - "\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n", + "\\log{ \\frac{p(\\boldsymbol{\\theta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\theta}\\boldsymbol{x})}} = \\theta_0+\\theta_1x_1+\\theta_2x_2+\\dots+\\theta_px_p.\n", "$$" ] }, { "cell_type": "markdown", - "id": "0cfae560", + "id": "eec7d85e", "metadata": { "editable": true }, "source": [ - "Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to" + "Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\theta}=[\\theta_0, \\theta_1, \\dots, \\theta_p]$ leading to" ] }, { "cell_type": "markdown", - "id": "7c427fc3", + "id": "152e6fa3", "metadata": { "editable": true }, "source": [ "$$\n", - "p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n", + "p(\\boldsymbol{\\theta}\\boldsymbol{x})=\\frac{ \\exp{(\\theta_0+\\theta_1x_1+\\theta_2x_2+\\dots+\\theta_px_p)}}{1+\\exp{(\\theta_0+\\theta_1x_1+\\theta_2x_2+\\dots+\\theta_px_p)}}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "6a6814bf", + "id": "b8168ae0", "metadata": { "editable": true }, @@ -736,19 +736,19 @@ }, { "cell_type": "markdown", - "id": "85282137", + "id": "3af519ee", "metadata": { "editable": true }, "source": [ "$$\n", - "\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\beta_{10}+\\beta_{11}x_1,\n", + "\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\theta_{10}+\\theta_{11}x_1,\n", "$$" ] }, { "cell_type": "markdown", - "id": "4864f76d", + "id": "fa02c958", "metadata": { "editable": true }, @@ -758,19 +758,19 @@ }, { "cell_type": "markdown", - "id": "390e9a55", + "id": "c550e015", "metadata": { "editable": true }, "source": [ "$$\n", - "\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n", + "\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\theta_{20}+\\theta_{21}x_1,\n", "$$" ] }, { "cell_type": "markdown", - "id": "c3f39c44", + "id": "8398ec7a", "metadata": { "editable": true }, @@ -780,19 +780,19 @@ }, { "cell_type": "markdown", - "id": "92f32a03", + "id": "6be24150", "metadata": { "editable": true }, "source": [ "$$\n", - "\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n", + "\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\theta_{(K-1)0}+\\theta_{(K-1)1}x_1,\n", "$$" ] }, { "cell_type": "markdown", - "id": "677a5c4d", + "id": "2153c8e3", "metadata": { "editable": true }, @@ -810,25 +810,25 @@ "multinomial logistic regression and linear discriminant analysis, the\n", "input to the function is the result of $K$ distinct linear functions,\n", "and the predicted probability for the $k$-th class given a sample\n", - "vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\beta}$ is (with two\n", + "vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\theta}$ is (with two\n", "predictors):" ] }, { "cell_type": "markdown", - "id": "fa70e3bf", + "id": "ab540777", "metadata": { "editable": true }, "source": [ "$$\n", - "p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n", + "p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\theta_{k0}+\\theta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\theta_{l0}+\\theta_{l1}x_1)}}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "4c59400c", + "id": "b7bcd7e8", "metadata": { "editable": true }, @@ -838,19 +838,19 @@ }, { "cell_type": "markdown", - "id": "f4073652", + "id": "c9605782", "metadata": { "editable": true }, "source": [ "$$\n", - "p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n", + "p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\theta_{l0}+\\theta_{l1}x_1)}},\n", "$$" ] }, { "cell_type": "markdown", - "id": "810e872a", + "id": "856dd896", "metadata": { "editable": true }, @@ -867,7 +867,7 @@ }, { "cell_type": "markdown", - "id": "f4ebc1da", + "id": "73029fad", "metadata": { "editable": true }, @@ -882,7 +882,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "b7a8cbac", + "id": "d81a44e3", "metadata": { "collapsed": false, "editable": true @@ -918,7 +918,7 @@ }, { "cell_type": "markdown", - "id": "10f74b93", + "id": "83a2e903", "metadata": { "editable": true }, @@ -930,7 +930,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "d0b8025d", + "id": "976fc321", "metadata": { "collapsed": false, "editable": true @@ -975,7 +975,7 @@ }, { "cell_type": "markdown", - "id": "9d84a2aa", + "id": "ea10d953", "metadata": { "editable": true }, @@ -998,7 +998,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "6144ea0a", + "id": "42204f9c", "metadata": { "collapsed": false, "editable": true @@ -1010,7 +1010,7 @@ }, { "cell_type": "markdown", - "id": "5627f5ae", + "id": "a56de206", "metadata": { "editable": true }, @@ -1021,7 +1021,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "a7c8662a", + "id": "0f2ff030", "metadata": { "collapsed": false, "editable": true @@ -1033,7 +1033,7 @@ }, { "cell_type": "markdown", - "id": "d03ee2a7", + "id": "68abe92d", "metadata": { "editable": true }, @@ -1059,7 +1059,7 @@ }, { "cell_type": "markdown", - "id": "44ce5095", + "id": "3cea1c5a", "metadata": { "editable": true }, @@ -1072,7 +1072,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "24ff3dd3", + "id": "f42e1d87", "metadata": { "collapsed": false, "editable": true diff --git a/doc/LectureNotes/_build/html/chapter1.html b/doc/LectureNotes/_build/html/chapter1.html index 16f0be608..9533e1f33 100644 --- a/doc/LectureNotes/_build/html/chapter1.html +++ b/doc/LectureNotes/_build/html/chapter1.html @@ -992,24 +992,6 @@ infile = open(data_path("MassEval2016.dat"),'r') -

Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various matplotlib commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function.

-
-
-
from pylab import plt, mpl
-plt.style.use('seaborn')
-mpl.rcParams['font.family'] = 'serif'
-
-def MakePlot(x,y, styles, labels, axlabels):
-    plt.figure(figsize=(10,6))
-    for i in range(len(x)):
-        plt.plot(x[i], y[i], styles[i], label = labels[i])
-        plt.xlabel(axlabels[0])
-        plt.ylabel(axlabels[1])
-    plt.legend(loc=0)
-
-
-
-

Our next step is to read the data on experimental binding energies and reorganize them as functions of the mass number \(A\), the number of protons \(Z\) and neutrons \(N\) using pandas. Before we do this it is @@ -1224,11 +1206,11 @@ plt.show()

3.4. Linear Regression, basic elements#

Video of Lecture.

-

Fitting a continuous function with linear parameterization in terms of the parameters \(\boldsymbol{\beta}\).

+

Fitting a continuous function with linear parameterization in terms of the parameters \(\boldsymbol{\theta}\).

@@ -1939,16 +1921,16 @@ X[:,1] = x X[:,2] = x**2 # We split the data in test and training data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) -# matrix inversion to find beta -beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train -print(beta) +# matrix inversion to find theta +theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train +print(theta) # and then make the prediction -ytilde = X_train @ beta +ytilde = X_train @ theta print("Training R2") print(R2(y_train,ytilde)) print("Training MSE") print(MSE(y_train,ytilde)) -ypredict = X_test @ beta +ypredict = X_test @ theta print("Test R2") print(R2(y_test,ypredict)) print("Test MSE") @@ -2037,15 +2019,15 @@ X[:,3] = Density**(4.0/3.0) X[:,4] = Density**(5.0/3.0) # We split the data in test and training data X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2) -# matrix inversion to find beta -beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train) +# matrix inversion to find theta +theta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train) # and then make the prediction -ytilde = X_train @ beta +ytilde = X_train @ theta print("Training R2") print(R2(y_train,ytilde)) print("Training MSE") print(MSE(y_train,ytilde)) -ypredict = X_test @ beta +ypredict = X_test @ theta print("Test R2") print(R2(y_test,ypredict)) print("Test MSE") @@ -2506,7 +2488,7 @@ R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.

Solution. -The code here is an example of where we define our own design matrix and fit parameters \(\beta\).

+The code here is an example of where we define our own design matrix and fit parameters \(\theta\).

import os
@@ -2535,16 +2517,16 @@ X[:,1] = x
 X[:,2] = x**2
 # We split the data in test and training data
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
-# matrix inversion to find beta
-beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
-print(beta)
+# matrix inversion to find theta
+theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
+print(theta)
 # and then make the prediction
-ytilde = X_train @ beta
+ytilde = X_train @ theta
 print("Training R2")
 print(R2(y_train,ytilde))
 print("Training MSE")
 print(MSE(y_train,ytilde))
-ypredict = X_test @ beta
+ypredict = X_test @ theta
 print("Test R2")
 print(R2(y_test,ypredict))
 print("Test MSE")
@@ -2755,16 +2737,16 @@ X[:,2] = x*x
 # We split the data in test and training data
 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
 
-# matrix inversion to find beta
-OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
-print(OLSbeta)
+# matrix inversion to find theta
+OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
+print(OLStheta)
 # and then make the prediction
-ytildeOLS = X_train @ OLSbeta
+ytildeOLS = X_train @ OLStheta
 print("Training R2 for OLS")
 print(R2(y_train,ytildeOLS))
 print("Training MSE for OLS")
 print(MSE(y_train,ytildeOLS))
-ypredictOLS = X_test @ OLSbeta
+ypredictOLS = X_test @ OLStheta
 print("Test R2 for OLS")
 print(R2(y_test,ypredictOLS))
 print("Test MSE OLS")
@@ -2781,10 +2763,10 @@ MSERidgePredict =  np.zeros(nlambdas)
 lambdas = np.logspace(-4, 1, nlambdas)
 for i in range(nlambdas):
     lmb = lambdas[i]
-    OwnRidgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
+    OwnRidgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
     # and then make the prediction
-    OwnytildeRidge = X_train @ OwnRidgebeta
-    OwnypredictRidge = X_test @ OwnRidgebeta
+    OwnytildeRidge = X_train @ OwnRidgetheta
+    OwnypredictRidge = X_test @ OwnRidgetheta
     OwnMSEPredict[i] = MSE(y_test,OwnypredictRidge)
     OwnMSETrain[i] = MSE(y_train,OwnytildeRidge)
     # Make the fit using Ridge from Sklearn
diff --git a/doc/LectureNotes/_build/html/chapter2.html b/doc/LectureNotes/_build/html/chapter2.html
index 9de7d1941..ad1f80139 100644
--- a/doc/LectureNotes/_build/html/chapter2.html
+++ b/doc/LectureNotes/_build/html/chapter2.html
@@ -405,16 +405,16 @@ doconce format html chapter2.do.txt  -->

 
4.1. Mathematical Interpretation of Ordinary Least Squares#

 
What is presented here is a mathematical analysis of various regression algorithms (ordinary least  squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).

-
We have shown that in ordinary least squares (OLS) the optimal parameters \(\beta\) are given by

+
We have shown that in ordinary least squares (OLS) the optimal parameters \(\theta\) are given by

 

 \[
-\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
 \]

-
The hat over \(\boldsymbol{\beta}\) means we have the optimal parameters after minimization of the cost function.

+
The hat over \(\boldsymbol{\theta}\) means we have the optimal parameters after minimization of the cost function.

 
This means that our best model is defined as

 

 \[
-\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
 \]

 
We now define a matrix

 

@@ -424,7 +424,7 @@ doconce format html chapter2.do.txt  -->

 \[
-\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
+\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{A}\boldsymbol{y}.
 \]

 
The matrix \(\boldsymbol{A}\) has the important property that \(\boldsymbol{A}^2=\boldsymbol{A}\). This is the definition of a projection matrix.
 We can then interpret our optimal model \(\tilde{\boldsymbol{y}}\) as being represented  by an orthogonal  projection of \(\boldsymbol{y}\) onto a space defined by the column vectors of \(\boldsymbol{X}\).  In our case here the matrix \(\boldsymbol{A}\) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.

@@ -525,15 +525,15 @@ This is equivalent to saying that the matrix 
 \[
 \begin{equation}
-\boldsymbol{\beta}  =  (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
+\boldsymbol{\theta}  =  (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
 \label{_auto1} \tag{1}
 \end{equation}
 \]

has linearly dependent column vectors, we will not be able to compute the inverse -of \(\boldsymbol{X}^T\boldsymbol{X}\) and we cannot find the parameters (estimators) \(\beta_i\). +of \(\boldsymbol{X}^T\boldsymbol{X}\) and we cannot find the parameters (estimators) \(\theta_i\). The estimators are only well-defined if \((\boldsymbol{X}^{T}\boldsymbol{X})\) can be inverted. This is more likely to happen when the matrix \(\boldsymbol{X}\) is high-dimensional. In this case it is likely to encounter a situation where -the regression parameters \(\beta_i\) cannot be estimated.

+the regression parameters \(\theta_i\) cannot be estimated.

A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change

\[ @@ -923,7 +923,7 @@ least squares is proportional to the second derivative of the cost function, that is we have

\[ -\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. +\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T\partial \boldsymbol{\theta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. \]

This quantity defines what is called the Hessian matrix (the second derivative of the cost function we want to optimize).

The Hessian matrix plays an important role and is defined in this course as

@@ -1154,7 +1154,7 @@ def create_X(x, y, n ): y = np.ravel(y) N = len(x) - l = int((n+1)*(n+2)/2) # Number of elements in beta + l = int((n+1)*(n+2)/2) # Number of elements in theta X = np.ones((N,l)) for i in range(1,n+1): @@ -1318,13 +1318,13 @@ values and the column vectors of \(\b our optimization problem is

\[ -{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}. \]

or we can state it as

\[ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2, \]

where we have used the definition of a norm-2 vector, that is

@@ -1332,26 +1332,26 @@ our optimization problem is

\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. \]

By minimizing the above equation with respect to the parameters -\(\boldsymbol{\beta}\) we could then obtain an analytical expression for the -parameters \(\boldsymbol{\beta}\). We can add a regularization parameter \(\lambda\) by +\(\boldsymbol{\theta}\) we could then obtain an analytical expression for the +parameters \(\boldsymbol{\theta}\). We can add a regularization parameter \(\lambda\) by defining a new cost function to be optimized, that is

\[ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2 \]

which leads to the Ridge regression minimization problem where we -require that \(\vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t\), where \(t\) is +require that \(\vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t\), where \(t\) is a finite number larger than zero. By defining

\[ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, +C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1, \]

we have a new optimization equation

\[ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1 \]

which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.

Here we have defined the norm-1 as

@@ -1362,44 +1362,44 @@ C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\bolds

Using the matrix-vector expression for Ridge regression and dropping the parameter \(1/n\) in front of the standard means squared error equation, we have

\[ -C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta}, \]

and -taking the derivatives with respect to \(\boldsymbol{\beta}\) we obtain then +taking the derivatives with respect to \(\boldsymbol{\theta}\) we obtain then a slightly modified matrix inversion problem which for finite values of \(\lambda\) does not suffer from singularity problems. We obtain the optimal parameters

\[ -\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, \]

with \(\boldsymbol{I}\) being a \(p\times p\) identity matrix with the constraint that

\[ -\sum_{i=0}^{p-1} \beta_i^2 \leq t, +\sum_{i=0}^{p-1} \theta_i^2 \leq t, \]

with \(t\) a finite positive number.

When we compare this with the ordinary least squares result we have

\[ -\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, \]

which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \(\boldsymbol{X}^T\boldsymbol{X}\).

We see that Ridge regression is nothing but the standard OLS with a modified diagonal term added to \(\boldsymbol{X}^T\boldsymbol{X}\). The consequences, in particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \(\lambda\), we may -even reduce the variance of the optimal parameters \(\boldsymbol{\beta}\). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.

+even reduce the variance of the optimal parameters \(\boldsymbol{\theta}\). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.

Using our insights about the SVD of the design matrix \(\boldsymbol{X}\) We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \(\boldsymbol{U}\) as

\[ -\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. +\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\theta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. \]

For Ridge regression this becomes

\[ -\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\theta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, \]

with the vectors \(\boldsymbol{u}_j\) being the columns of \(\boldsymbol{U}\) from the SVD of the matrix \(\boldsymbol{X}\). Note that the sums goes to \(p-1\) since.

Since \(\lambda \geq 0\), it means that compared to OLS, we have

@@ -1421,12 +1421,12 @@ eigenvalues ordered in a descending way, that is \[ -\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y}, +\boldsymbol{\theta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y}, \]

and

\[ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, +\boldsymbol{\theta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\theta}^{\mathrm{OLS}}, \]

that is the Ridge estimator scales the OLS estimator by the inverse of a factor \(1+\lambda\), and the Ridge estimator converges to zero when the hyperparameter goes to @@ -1435,64 +1435,64 @@ infinity.

Using the matrix-vector expression for Lasso regression and dropping the parameter \(1/n\) in front of the standard mean squared error equation, we have the following cost function

\[ -C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, +C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1, \]
-

Taking the derivative with respect to \(\boldsymbol{\beta}\) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)

+

Taking the derivative with respect to \(\boldsymbol{\theta}\) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)

\[\begin{split} -\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right. +\frac{d \vert \theta\vert}{d \boldsymbol{\theta}}=\mathrm{sgn}(\boldsymbol{\theta})=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right. \end{split}\]

we have that the derivative of the cost function is

\[ -\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, +\frac{\partial C(\boldsymbol{X},\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})+\lambda sgn(\boldsymbol{\theta})=0, \]

and reordering we have

\[ -\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}. +\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}. \]

This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.

Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the diagonal. In this case we have an equal number of rows and columns \(n=p\).

-

Our model approximation is just \(\tilde{\boldsymbol{y}}=\boldsymbol{\beta}\) and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term \(1/n\))

+

Our model approximation is just \(\tilde{\boldsymbol{y}}=\boldsymbol{\theta}\) and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term \(1/n\))

\[ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2, +C(\boldsymbol{\theta})=\sum_{i=0}^{p-1}(y_i-\theta_i)^2, \]

and minimizing we have that

\[ -\hat{\beta}_i^{\mathrm{OLS}} = y_i. +\hat{\theta}_i^{\mathrm{OLS}} = y_i. \]

For Ridge regression our cost function is

\[ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, +C(\boldsymbol{\theta})=\sum_{i=0}^{p-1}(y_i-\theta_i)^2+\lambda\sum_{i=0}^{p-1}\theta_i^2, \]

and minimizing we have that

\[ -\hat{\beta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}. +\hat{\theta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}. \]

For Lasso regression our cost function is

\[ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, +C(\boldsymbol{\theta})=\sum_{i=0}^{p-1}(y_i-\theta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\theta_i\vert=\sum_{i=0}^{p-1}(y_i-\theta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\theta_i^2}, \]

and minimizing we have that

\[ --2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, +-2\sum_{i=0}^{p-1}(y_i-\theta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\theta_i)}{\vert\theta_i\vert}=0, \]

which leads to

\[\begin{split} -\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ +\hat{\boldsymbol{\theta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ y_i+\frac{\lambda}{2} &\mathrm{if} & y_i< -\frac{\lambda}{2}\\ 0 &\mathrm{if} & \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\. \end{split}\]
-

Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \(\beta_i\) for specific values of \(\lambda\). Ridge regression reduces on the other hand the values of \(\beta_i\) as function of \(\lambda\).

+

Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \(\theta_i\) for specific values of \(\lambda\). Ridge regression reduces on the other hand the values of \(\theta_i\) as function of \(\lambda\).

As another example, let us assume we have a data set with outputs/targets given by the vector

@@ -1504,99 +1504,99 @@ let us assume we have a data set with outputs/targets given by the vector

\[\begin{split} \boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix}, \end{split}\]
-

meaning that we have two features and two unknown parameters \(\beta_0\) and \(\beta_1\) to be determined either by ordinary least squares, Ridge or Lasso regression.

+

meaning that we have two features and two unknown parameters \(\theta_0\) and \(\theta_1\) to be determined either by ordinary least squares, Ridge or Lasso regression.

For ordinary least squares (OLS) we know that the optimal solution is

\[ -\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. \]

Inserting the above values we obtain that

\[\begin{split} -\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, +\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, \end{split}\]

The code which implements this simpler case is presented after the discussion of Ridge and Lasso.

For Ridge regression we have

\[ -\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\hat{\boldsymbol{\theta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. \]

Inserting the above values we obtain that

\[\begin{split} -\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, +\hat{\boldsymbol{\theta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, \end{split}\]
-

There is normally a constraint on the value of \(\vert\vert \boldsymbol{\beta}\vert\vert_2\) via the parameter \(\lambda\). -Let us for simplicity assume that \(\beta_0^2+\beta_1^2=1\) as constraint. This will allow us to find an expression for the optimal values of \(\beta\) and \(\lambda\).

+

There is normally a constraint on the value of \(\vert\vert \boldsymbol{\theta}\vert\vert_2\) via the parameter \(\lambda\). +Let us for simplicity assume that \(\theta_0^2+\theta_1^2=1\) as constraint. This will allow us to find an expression for the optimal values of \(\theta\) and \(\lambda\).

To see this, let us write the cost function for Ridge regression.

We define the MSE without the \(1/n\) factor and have then, using that

\[\begin{split} -\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix}, +\boldsymbol{X}\boldsymbol{\theta}=\begin{bmatrix} 2\theta_0 \\ \theta_1 \\0 \end{bmatrix}, \end{split}\]
\[ -C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2), +C(\boldsymbol{\theta})=(4-2\theta_0)^2+(2-\theta_1)^2+\lambda(\theta_0^2+\theta_1^2), \]
-

and taking the derivative with respect to \(\beta_0\) we get

+

and taking the derivative with respect to \(\theta_0\) we get

\[ -\beta_0=\frac{8}{4+\lambda}, +\theta_0=\frac{8}{4+\lambda}, \]
-

and for \(\beta_1\) we obtain

+

and for \(\theta_1\) we obtain

\[ -\beta_1=\frac{2}{1+\lambda}, +\theta_1=\frac{2}{1+\lambda}, \]
-

Using the constraint for \(\beta_0^2+\beta_1^2=1\) we can constrain \(\lambda\) by solving

+

Using the constraint for \(\theta_0^2+\theta_1^2=1\) we can constrain \(\lambda\) by solving

\[ \left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1, \]
-

which gives \(\lambda=4.571\) and \(\beta_0=0.933\) and \(\beta_1=0.359\).

-

For Lasso we need now, keeping a constraint on \(\vert\beta_0\vert+\vert\beta_1\vert=1\), to take the derivative of the absolute values of \(\beta_0\) -and \(\beta_1\). This gives us the following derivatives of the cost function

+

which gives \(\lambda=4.571\) and \(\theta_0=0.933\) and \(\theta_1=0.359\).

+

For Lasso we need now, keeping a constraint on \(\vert\theta_0\vert+\vert\theta_1\vert=1\), to take the derivative of the absolute values of \(\theta_0\) +and \(\theta_1\). This gives us the following derivatives of the cost function

\[ -C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), +C(\boldsymbol{\theta})=(4-2\theta_0)^2+(2-\theta_1)^2+\lambda(\vert\theta_0\vert+\vert\theta_1\vert), \]
\[ -\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0, +\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_0}=-4(4-2\theta_0)+\lambda\mathrm{sgn}(\theta_0)=0, \]

and

\[ -\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0. +\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_1}=-2(2-\theta_1)+\lambda\mathrm{sgn}(\theta_1)=0. \]
-

We have now four cases to solve besides the trivial cases \(\beta_0\) and/or \(\beta_1\) are zero, namely

+

We have now four cases to solve besides the trivial cases \(\theta_0\) and/or \(\theta_1\) are zero, namely

    -

  1. \(\beta_0 > 0\) and \(\beta_1 > 0\),

    2. -

  2. \(\beta_0 > 0\) and \(\beta_1 < 0\),

    3. -

  3. \(\beta_0 < 0\) and \(\beta_1 > 0\),

    4. -

  4. \(\beta_0 < 0\) and \(\beta_1 < 0\).

    5. +

  5. \(\theta_0 > 0\) and \(\theta_1 > 0\),

    6. +

  6. \(\theta_0 > 0\) and \(\theta_1 < 0\),

    7. +

  7. \(\theta_0 < 0\) and \(\theta_1 > 0\),

    8. +

  8. \(\theta_0 < 0\) and \(\theta_1 < 0\).

If we consider the first case, we have then

\[ --4(4-2\beta_0)+\lambda=0, +-4(4-2\theta_0)+\lambda=0, \]

and

\[ --2(2-\beta_1)+\lambda=0. +-2(2-\theta_1)+\lambda=0. \]

which yields

\[ -\beta_0=\frac{16+\lambda}{8}, +\theta_0=\frac{16+\lambda}{8}, \]

and

\[ -\beta_1=\frac{4+\lambda}{2}. +\theta_1=\frac{4+\lambda}{2}. \]
-

Using the constraint on \(\beta_0\) and \(\beta_1\) we can then find the optimal value of \(\lambda\) for the different cases. We leave this as an exercise to you.

+

Using the constraint on \(\theta_0\) and \(\theta_1\) we can then find the optimal value of \(\lambda\) for the different cases. We leave this as an exercise to you.

Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \(\lambda\), meaning that we need to perform a search in order to find the optimal values.

First we study and compare the OLS and Ridge results. The next code compares all three methods. We select values of the hyperparameter \(\lambda\in [10^{-4},10^4]\) and compute the predicted values for ordinary least squares and Ridge regression.

@@ -1623,14 +1623,14 @@ X = np.array( [ [ 2, 0], [0, 1], [0,0]]) y = np.array( [4, 2, 3]) -# matrix inversion to find beta -OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y -print(OLSbeta) +# matrix inversion to find theta +OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y +print(OLStheta) # and then make the prediction -ytildeOLS = X @ OLSbeta +ytildeOLS = X @ OLStheta print("Training MSE for OLS") print(MSE(y,ytildeOLS)) -ypredictOLS = X @ OLSbeta +ypredictOLS = X @ OLStheta # Repeat now for Ridge regression and various values of the regularization parameter I = np.eye(2,2) @@ -1640,10 +1640,10 @@ MSEPredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 4, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y -# print(Ridgebeta) + Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y +# print(Ridgetheta) # and then make the prediction - ypredictRidge = X @ Ridgebeta + ypredictRidge = X @ Ridgetheta MSEPredict[i] = MSE(y,ypredictRidge) # print(MSEPredict[i]) # Now plot the results @@ -1657,9 +1657,9 @@ plt.show()
-

We see here that we reach a plateau for the Ridge results. Writing out the coefficients \(\boldsymbol{\beta}\), we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of \(\tilde{\boldsymbol{y}}\) approach zero.

+

We see here that we reach a plateau for the Ridge results. Writing out the coefficients \(\boldsymbol{\theta}\), we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of \(\tilde{\boldsymbol{y}}\) approach zero.

This happens also for Lasso regression, as seen from the next code -output. The difference is that Lasso shrinks the values of \(\beta\) to +output. The difference is that Lasso shrinks the values of \(\theta\) to zero at a much earlier stage and the results flatten out. We see that Lasso gives also an excellent fit for small values of \(\lambda\) and shows the best performance of the three regression methods.

@@ -1685,14 +1685,14 @@ X = np.array( [ [ 2, 0], [0, 1], [0,0]]) y = np.array( [4, 2, 3]) -# matrix inversion to find beta -OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y -print(OLSbeta) +# matrix inversion to find theta +OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y +print(OLStheta) # and then make the prediction -ytildeOLS = X @ OLSbeta +ytildeOLS = X @ OLStheta print("Training MSE for OLS") print(MSE(y,ytildeOLS)) -ypredictOLS = X @ OLSbeta +ypredictOLS = X @ OLStheta # Repeat now for Ridge regression and various values of the regularization parameter I = np.eye(2,2) @@ -1703,10 +1703,10 @@ MSELassoPredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 4, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y - print(Ridgebeta) + Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y + print(Ridgetheta) # and then make the prediction - ypredictRidge = X @ Ridgebeta + ypredictRidge = X @ Ridgetheta MSERidgePredict[i] = MSE(y,ypredictRidge) RegLasso = linear_model.Lasso(lmb) RegLasso.fit(X,y) @@ -1730,7 +1730,7 @@ three regression methods. Depending on the level of noise, we note that for small values of the hyperparameter \(\lambda\) all three methods produce the same mean squared error. Again, Lasso shrinks the parameter values to zero much earlier than Ridge regression and the -Lasso results flatten out much earlier since all \(\beta_j=0\) (check +Lasso results flatten out much earlier since all \(\theta_j=0\) (check this by printing the values). This case is an example of where OLS performs best. Lasso and Ridge reproduce the OLS results for a limited set of \(\lambda\) values.

@@ -1767,14 +1767,14 @@ X[:,2] = x*x # We split the data in test and training data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) -# matrix inversion to find beta -OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train -print(OLSbeta) +# matrix inversion to find theta +OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train +print(OLStheta) # and then make the prediction -ytildeOLS = X_train @ OLSbeta +ytildeOLS = X_train @ OLStheta print("Training MSE for OLS") print(MSE(y_train,ytildeOLS)) -ypredictOLS = X_test @ OLSbeta +ypredictOLS = X_test @ OLStheta print("Test MSE OLS") print(MSE(y_test,ypredictOLS)) @@ -1789,13 +1789,13 @@ MSELassoTrain = np.zeros(nlambdas) lambdas = np.logspace(-4, 4, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train + Ridgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train # include lasso using Scikit-Learn RegLasso = linear_model.Lasso(lmb) RegLasso.fit(X_train,y_train) # and then make the prediction - ytildeRidge = X_train @ Ridgebeta - ypredictRidge = X_test @ Ridgebeta + ytildeRidge = X_train @ Ridgetheta + ypredictRidge = X_test @ Ridgetheta ytildeLasso = RegLasso.predict(X_train) ypredictLasso = RegLasso.predict(X_test) MSEPredict[i] = MSE(y_test,ypredictRidge) @@ -1832,7 +1832,7 @@ move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters -\(\beta\).

+\(\theta\).

The advantage of doing linear regression is that we actually end up with analytical expressions for several statistical quantities.
@@ -1853,7 +1853,7 @@ independent, i.e.:

The randomness of \(\varepsilon_i\) implies that \(\mathbf{y}_i\) is also a random variable. In particular, \(\mathbf{y}_i\) is normally distributed, because \(\varepsilon_i \sim -\mathcal{N}(0, \sigma^2)\) and \(\mathbf{X}_{i,\ast} \, \boldsymbol{\beta}\) is a +\mathcal{N}(0, \sigma^2)\) and \(\mathbf{X}_{i,\ast} \, \boldsymbol{\theta}\) is a non-random scalar. To specify the parameters of the distribution of \(\mathbf{y}_i\) we need to calculate its first two moments.

Recall that \(\boldsymbol{X}\) is a matrix of dimensionality \(n\times p\). The @@ -1870,15 +1870,15 @@ which describe our data

function \(f\) is approximated by \(\boldsymbol{\tilde{y}}\) where we want to minimize \((\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\), our MSE, with

\[ -\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\theta}. \]

We can calculate the expectation value of \(\boldsymbol{y}\) for a given element \(i\)

\[ \begin{align*} \mathbb{E}(y_i) & = -\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i) -\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\theta}) + \mathbb{E}(\varepsilon_i) +\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta, \end{align*} \]

while @@ -1888,80 +1888,80 @@ its variance is

\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i - \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - [\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, -\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ & -= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i -\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, -\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 -\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + -\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 +\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ & += \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 \varepsilon_i +\mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, +\ast} \, \theta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 +\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + +\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, \mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. \end{align*} \end{split}\] -

Hence, \(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2)\), that is \(\boldsymbol{y}\) follows a normal distribution with -mean value \(\boldsymbol{X}\boldsymbol{\beta}\) and variance \(\sigma^2\) (not be confused with the singular values of the SVD).

-

With the OLS expressions for the parameters \(\boldsymbol{\beta}\) we can evaluate the expectation value

+

Hence, \(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta}, \sigma^2)\), that is \(\boldsymbol{y}\) follows a normal distribution with +mean value \(\boldsymbol{X}\boldsymbol{\theta}\) and variance \(\sigma^2\) (not be confused with the singular values of the SVD).

+

With the OLS expressions for the parameters \(\boldsymbol{\theta}\) we can evaluate the expectation value

\[ -\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. +\mathbb{E}(\boldsymbol{\theta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\theta}=\boldsymbol{\theta}. \]

This means that the estimator of the regression parameters is unbiased.

We can also calculate the variance

-

The variance of \(\boldsymbol{\beta}\) is

+

The variance of \(\boldsymbol{\theta}\) is

\[\begin{split} \begin{eqnarray*} -\mbox{Var}(\boldsymbol{\beta}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} +\mbox{Var}(\boldsymbol{\theta}) & = & \mathbb{E} \{ [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})] [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})]^{T} \} \\ -& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \} +& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}]^{T} \} \\ -% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} % \\ -% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} % \\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} % \\ -% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} +% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} % \\ -% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T +% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\theta} \boldsymbol{\theta}^T \\ -& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +& = & \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, \end{eqnarray*} \end{split}\]

where we have used that \(\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = -\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + -\sigma^2 \, \mathbf{I}_{nn}\). From \(\mbox{Var}(\boldsymbol{\beta}) = \sigma^2 +\mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + +\sigma^2 \, \mathbf{I}_{nn}\). From \(\mbox{Var}(\boldsymbol{\theta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}\), one obtains an estimate of the variance of the estimate of the \(j\)-th regression coefficient: -\(\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to +\(\boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to construct a confidence interval for the estimates.

In a similar way, we can obtain analytical expressions for say the -expectation values of the parameters \(\boldsymbol{\beta}\) and their variance +expectation values of the parameters \(\boldsymbol{\theta}\) and their variance when we employ Ridge regression, allowing us again to define a confidence interval.

It is rather straightforward to show that

\[ -\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. +\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\theta}^{\mathrm{OLS}}. \]

We see clearly that -\(\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}}\) for any \(\lambda > 0\). We say then that the ridge estimator is biased.

+\(\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\theta}^{\mathrm{OLS}}\) for any \(\lambda > 0\). We say then that the ridge estimator is biased.

We can also compute the variance as

\[ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +\mbox{Var}[\boldsymbol{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, \]
-

and it is easy to see that if the parameter \(\lambda\) goes to infinity then the variance of Ridge parameters \(\boldsymbol{\beta}\) goes to zero.

+

and it is easy to see that if the parameter \(\lambda\) goes to infinity then the variance of Ridge parameters \(\boldsymbol{\theta}\) goes to zero.

With this, we can compute the difference

\[ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +\mbox{Var}[\boldsymbol{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. \]

The difference is non-negative definite since each component of the matrix product is non-negative definite. -This means the variance we obtain with the standard OLS will always for \(\lambda > 0\) be larger than the variance of \(\boldsymbol{\beta}\) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.

+This means the variance we obtain with the standard OLS will always for \(\lambda > 0\) be larger than the variance of \(\boldsymbol{\theta}\) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.

4.12. Deriving OLS from a probability distribution#

@@ -1971,26 +1971,26 @@ that our output is determined by a given continuous function distribution with zero mean value and an undetermined variance \(\sigma^2\).

We found above that the outputs \(\boldsymbol{y}\) have a mean value given by -\(\boldsymbol{X}\hat{\boldsymbol{\beta}}\) and variance \(\sigma^2\). Since the entries to +\(\boldsymbol{X}\hat{\boldsymbol{\theta}}\) and variance \(\sigma^2\). Since the entries to the design matrix are not stochastic variables, we can assume that the probability distribution of our targets is also a normal distribution -but now with mean value \(\boldsymbol{X}\hat{\boldsymbol{\beta}}\). This means that a +but now with mean value \(\boldsymbol{X}\hat{\boldsymbol{\theta}}\). This means that a single output \(y_i\) is given by the Gaussian distribution

\[ -y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. +y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\theta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}. \]

We assume now that the various \(y_i\) values are stochastically distributed according to the above Gaussian distribution. We define this distribution as

\[ -p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}, +p(y_i, \boldsymbol{X}\vert\boldsymbol{\theta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}, \]
-

which reads as finding the likelihood of an event \(y_i\) with the input variables \(\boldsymbol{X}\) given the parameters (to be determined) \(\boldsymbol{\beta}\).

+

which reads as finding the likelihood of an event \(y_i\) with the input variables \(\boldsymbol{X}\) given the parameters (to be determined) \(\boldsymbol{\theta}\).

Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event \(\boldsymbol{y}\) as the product of the single events, that is we have

\[ -p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}). +p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta}). \]

We will write this in a more compact form reserving \(\boldsymbol{D}\) for the domain of events, including the ouputs (targets) and the inputs. That is in case we have a simple one-dimensional input and output case

@@ -2002,18 +2002,18 @@ in case we have a simple one-dimensional input and output case

We can now rewrite the above probability as

\[ -p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. +p(\boldsymbol{D}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}. \]

It is a conditional probability (see below) and reads as the likelihood of a domain of events \(\boldsymbol{D}\) given a set of parameters -\(\boldsymbol{\beta}\).

+\(\boldsymbol{\theta}\).

In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is the most probable.

We will assume here that our events are given by the above Gaussian -distribution and we will determine the optimal parameters \(\beta\) by +distribution and we will determine the optimal parameters \(\theta\) by maximizing the above PDF. However, computing the derivatives of a product function is cumbersome and can easily lead to overflow and/or underflowproblems, with potentials for loss of numerical precision.

@@ -2027,22 +2027,22 @@ is equivalent to the maximization/minimization of the function itself.

We could now define a new cost function to minimize, namely the negative logarithm of the above PDF

\[ -C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}, +C(\boldsymbol{\theta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})}, \]

which becomes

\[ -C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}. +C(\boldsymbol{\theta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}. \]
-

Taking the derivative of the new cost function with respect to the parameters \(\beta\) we recognize our familiar OLS equation, namely

+

Taking the derivative of the new cost function with respect to the parameters \(\theta\) we recognize our familiar OLS equation, namely

\[ -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0, +\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right) =0, \]
-

which leads to the well-known OLS equation for the optimal paramters \(\beta\)

+

which leads to the well-known OLS equation for the optimal paramters \(\theta\)

\[ -\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! +\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! \]

Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics.

A central theorem in statistics is Bayes’ theorem. This theorem plays a similar role as the good old Pythagoras’ theorem in geometry. @@ -2160,9 +2160,9 @@ perhaps not that intuitive. The question is whether we can develop a more intuitive way of understanding what Ridge and Lasso express.

Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit.

We will play around with a study of the values for the optimal -parameters \(\boldsymbol{\beta}\) using OLS, Ridge and Lasso regression. For +parameters \(\boldsymbol{\theta}\) using OLS, Ridge and Lasso regression. For OLS, you will notice as function of the noise and polynomial degree, -that the parameters \(\beta\) will fluctuate from order to order in the +that the parameters \(\theta\) will fluctuate from order to order in the polynomial fit and that for larger and larger polynomial degrees of freedom, the parameters will tend to increase in value for OLS.

For Ridge and Lasso regression, the higher order parameters will @@ -2198,10 +2198,10 @@ for polydegree in range(1, Maxpolydegree): # We split the data in test and training data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) -# matrix inversion to find beta -OLSbeta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train -print(OLSbeta) -ypredictOLS = X_test @ OLSbeta +# matrix inversion to find theta +OLStheta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train +print(OLStheta) +ypredictOLS = X_test @ OLStheta print("Test MSE OLS") print(MSE(y_test,ypredictOLS)) # Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn @@ -2238,21 +2238,21 @@ plt.show()

How can we understand this?

-

Let us write out the values of the coefficients \(\beta_i\) as functions +

Let us write out the values of the coefficients \(\theta_i\) as functions of the polynomial degree and noise. We will focus only on the Ridge results and some few selected values of the hyperparameter \(\lambda\).

If we don’t include any noise and run this code for different values -of the polynomial degree, we notice that the results for \(\beta_i\) do +of the polynomial degree, we notice that the results for \(\theta_i\) do not show great changes from one order to the next. This is an indication that for higher polynomial orders, our parameters become less important.

If we however add noise, what happens is that the polynomial fit is trying to adjust the fit to traverse in the best possible way all data points. This can lead to large fluctuations in the parameters -\(\beta_i\) as functions of polynomial order. It will also be reflected -in a larger value of the variance of each parameter \(\beta_i\). What +\(\theta_i\) as functions of polynomial order. It will also be reflected +in a larger value of the variance of each parameter \(\theta_i\). What Ridge regression (and Lasso as well) are doing then is to try to -quench the fluctuations in the parameters of \(\beta_i\) which have a +quench the fluctuations in the parameters of \(\theta_i\) which have a large variance (normally for higher orders in the polynomial).

@@ -2291,26 +2291,26 @@ for i in range(nlambdas): # and then make the prediction ypredictRidge = RegRidge.predict(X_test) Coeffs = np.array(RegRidge.coef_) - BetaValues = pd.DataFrame(Coeffs) - BetaValues.columns = ['beta'] - display(BetaValues) + ThetaValues = pd.DataFrame(Coeffs) + ThetaValues.columns = ['theta'] + display(ThetaValues)

As an exercise, repeat these calculations with ordinary least squares only with and without noise. Calculate thereafter the variance of the -parameters \(\beta_j\) as function of polynomial order and of the added +parameters \(\theta_j\) as function of polynomial order and of the added noise. Here we recommend to use \(\sigma^2=1\) as variance for the added noise (which follows a normal distribution with mean value zero). -Comment your results. If you have a large noise term, do the parameters \(\beta_j\) vary more as function +Comment your results. If you have a large noise term, do the parameters \(\theta_j\) vary more as function of model complexity? And what about their variance?

4.14. Linking Bayes’ Theorem with Ridge and Lasso Regression#

We have seen that Ridge regression suppresses those features which have a small singular value. This corresponds to a feature which exhibits -a large variance in the parameters \(\beta_j\). +a large variance in the parameters \(\theta_j\). Our analysis hitherto has been based on linear algebra. To add to our intuition, we will use Bayes’ theorem in order to deepen our understanding of Ridge and Lasso regression.

For ordinary least squares we postulated that the maximum likelihood for the domain of events \(\boldsymbol{D}\) (one-dimensional case)

@@ -2321,74 +2321,74 @@ Bayes’ theorem in order to deepen our understanding of Ridge and Lasso regres

is given by

\[ -p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. +p(\boldsymbol{D}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}. \]
-

In Bayes’ theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set \(\boldsymbol{\beta}\) given a domain of events \(\boldsymbol{D}\)? That is, how can we define the posterior probability

+

In Bayes’ theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set \(\boldsymbol{\theta}\) given a domain of events \(\boldsymbol{D}\)? That is, how can we define the posterior probability

\[ -p(\boldsymbol{\beta}\vert\boldsymbol{D}). +p(\boldsymbol{\theta}\vert\boldsymbol{D}). \]

Bayes’ theorem comes to our rescue here since (omitting the normalization constant)

\[ -p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}). +p(\boldsymbol{\theta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\theta})p(\boldsymbol{\theta}). \]
-

We have a model for \(p(\boldsymbol{D}\vert\boldsymbol{\beta})\) but need one for the prior \(p(\boldsymbol{\beta})\)!

+

We have a model for \(p(\boldsymbol{D}\vert\boldsymbol{\theta})\) but need one for the prior \(p(\boldsymbol{\theta})\)!

With the posterior probability defined by a likelihood which we have already modeled and an unknown prior, we are now ready to make additional models for the prior.

-

We can, based on our discussions of the variance of \(\boldsymbol{\beta}\) and -the mean value, assume that the prior for the values \(\boldsymbol{\beta}\) is +

We can, based on our discussions of the variance of \(\boldsymbol{\theta}\) and +the mean value, assume that the prior for the values \(\boldsymbol{\theta}\) is given by a Gaussian with mean value zero and variance \(\tau^2\), that

\[ -p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. +p(\boldsymbol{\theta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\theta_j^2}{2\tau^2}\right)}. \]

Our posterior probability becomes then (omitting the normalization factor which is just a constant)

\[ -p(\boldsymbol{\beta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. +p(\boldsymbol{\theta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\theta_j^2}{2\tau^2}\right)}. \]
-

We can now optimize this quantity with respect to \(\boldsymbol{\beta}\). As we +

We can now optimize this quantity with respect to \(\boldsymbol{\theta}\). As we did for OLS, this is most conveniently done by taking the negative logarithm of the posterior probability. Doing so and leaving out the -terms that do not depend on \(\beta\), we have

+terms that do not depend on \(\theta\), we have

\[ -C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +C(\boldsymbol{\theta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\theta}\vert\vert_2^2, \]

and replacing \(1/2\tau^2\) with \(\lambda\) we have

\[ -C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +C(\boldsymbol{\theta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_2^2, \]

which is our Ridge cost function! Nice, isn’t it?

To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is

\[ -p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. +p(\boldsymbol{\theta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\theta_j\vert}{\tau}\right)}. \]

Our posterior probability becomes then (omitting the normalization factor which is just a constant)

\[ -p(\boldsymbol{\beta}\vert\boldsymbol{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. +p(\boldsymbol{\theta}\vert\boldsymbol{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\theta_j\vert}{\tau}\right)}. \]

Taking the negative logarithm of the posterior probability and leaving out the -constants terms that do not depend on \(\beta\), we have

+constants terms that do not depend on \(\theta\), we have

\[ -C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1, +C(\boldsymbol{\theta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\theta}\vert\vert_1, \]

and replacing \(1/\tau\) with \(\lambda\) we have

\[ -C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, +C(\boldsymbol{\theta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1, \]

which is our Lasso cost function!

Plotting these prior functions shows us that we can use the parameter \(\lambda\) to shrink or increase the role of a given parameter -\(\beta_j\). The variance for the Laplace distribution is +\(\theta_j\). The variance for the Laplace distribution is \(2\tau^2=1/\lambda\) while for the Gaussian distribution it is \(\sigma^2=1/(2\lambda)\). Thus, increasing the variance means decreasing \(\lambda\) and shrinking the variance means increasing diff --git a/doc/LectureNotes/_build/html/chapter3.html b/doc/LectureNotes/_build/html/chapter3.html index cb0a407e9..adac56980 100644 --- a/doc/LectureNotes/_build/html/chapter3.html +++ b/doc/LectureNotes/_build/html/chapter3.html @@ -479,7 +479,7 @@ independent, i.e.:

The randomness of \(\varepsilon_i\) implies that \(\mathbf{y}_i\) is also a random variable. In particular, \(\mathbf{y}_i\) is normally distributed, because \(\varepsilon_i \sim -\mathcal{N}(0, \sigma^2)\) and \(\mathbf{X}_{i,\ast} \, \boldsymbol{\beta}\) is a +\mathcal{N}(0, \sigma^2)\) and \(\mathbf{X}_{i,\ast} \, \boldsymbol{\theta}\) is a non-random scalar. To specify the parameters of the distribution of \(\mathbf{y}_i\) we need to calculate its first two moments.

Recall that \(\boldsymbol{X}\) is a matrix of dimensionality \(n\times p\). The @@ -496,15 +496,15 @@ which describe our data

function \(f\) is approximated by \(\boldsymbol{\tilde{y}}\) where we want to minimize \((\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\), our MSE, with

\[ -\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\theta}. \]

We can calculate the expectation value of \(\boldsymbol{y}\) for a given element \(i\)

\[ \begin{align*} \mathbb{E}(y_i) & = -\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i) -\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\theta}) + \mathbb{E}(\varepsilon_i) +\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta, \end{align*} \]

while @@ -514,81 +514,81 @@ its variance is

\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i - \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - [\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, -\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ & -= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i -\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, -\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 -\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + -\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 +\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ & += \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 \varepsilon_i +\mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, +\ast} \, \theta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 +\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + +\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, \mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. \end{align*} \end{split}\] -

Hence, \(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2)\), that is \(\boldsymbol{y}\) follows a normal distribution with -mean value \(\boldsymbol{X}\boldsymbol{\beta}\) and variance \(\sigma^2\) (not be confused with the singular values of the SVD).

-

With the OLS expressions for the parameters \(\boldsymbol{\beta}\) we can evaluate the expectation value

+

Hence, \(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta}, \sigma^2)\), that is \(\boldsymbol{y}\) follows a normal distribution with +mean value \(\boldsymbol{X}\boldsymbol{\theta}\) and variance \(\sigma^2\) (not be confused with the singular values of the SVD).

+

With the OLS expressions for the parameters \(\boldsymbol{\theta}\) we can evaluate the expectation value

\[ -\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. +\mathbb{E}(\boldsymbol{\theta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\theta}=\boldsymbol{\theta}. \]

This means that the estimator of the regression parameters is unbiased.

We can also calculate the variance

-

The variance of \(\boldsymbol{\beta}\) is

+

The variance of \(\boldsymbol{\theta}\) is

\[\begin{split} \begin{eqnarray*} -\mbox{Var}(\boldsymbol{\beta}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} +\mbox{Var}(\boldsymbol{\theta}) & = & \mathbb{E} \{ [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})] [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})]^{T} \} \\ -& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \} +& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}]^{T} \} \\ -% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} % \\ -% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} % \\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} % \\ -% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} +% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} % \\ -% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T +% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\theta} \boldsymbol{\theta}^T \\ -& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +& = & \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, \end{eqnarray*} \end{split}\]

where we have used that \(\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = -\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + -\sigma^2 \, \mathbf{I}_{nn}\). From \(\mbox{Var}(\boldsymbol{\beta}) = \sigma^2 +\mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + +\sigma^2 \, \mathbf{I}_{nn}\). From \(\mbox{Var}(\boldsymbol{\theta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}\), one obtains an estimate of the variance of the estimate of the \(j\)-th regression coefficient: -\(\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 \sqrt{ +\(\boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 \sqrt{ [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} }\). This may be used to construct a confidence interval for the estimates.

In a similar way, we can obtain analytical expressions for say the -expectation values of the parameters \(\boldsymbol{\beta}\) and their variance +expectation values of the parameters \(\boldsymbol{\theta}\) and their variance when we employ Ridge regression, allowing us again to define a confidence interval.

It is rather straightforward to show that

\[ -\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. +\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\theta}^{\mathrm{OLS}}. \]

We see clearly that -\(\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}}\) for any \(\lambda > 0\). We say then that the ridge estimator is biased.

+\(\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\theta}^{\mathrm{OLS}}\) for any \(\lambda > 0\). We say then that the ridge estimator is biased.

We can also compute the variance as

\[ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +\mbox{Var}[\boldsymbol{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, \]
-

and it is easy to see that if the parameter \(\lambda\) goes to infinity then the variance of Ridge parameters \(\boldsymbol{\beta}\) goes to zero.

+

and it is easy to see that if the parameter \(\lambda\) goes to infinity then the variance of Ridge parameters \(\boldsymbol{\theta}\) goes to zero.

With this, we can compute the difference

\[ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +\mbox{Var}[\boldsymbol{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. \]

The difference is non-negative definite since each component of the matrix product is non-negative definite. -This means the variance we obtain with the standard OLS will always for \(\lambda > 0\) be larger than the variance of \(\boldsymbol{\beta}\) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.

+This means the variance we obtain with the standard OLS will always for \(\lambda > 0\) be larger than the variance of \(\boldsymbol{\theta}\) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.

5.3. Resampling methods#

@@ -620,7 +620,7 @@ satisfied, the methods will fail. Yet, it should be said that if the data are independent, identically distributed, and we only want to estimate the variance of \(\overline{X}\) (which often is the case), then there is no need for bootstrapping.

-

The Jackknife works by making many replicas of the estimator \(\widehat{\beta}\). +

The Jackknife works by making many replicas of the estimator \(\widehat{\theta}\). The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \(\boldsymbol{x} = (x_1,x_2,\cdots,X_n)\). Let \(\boldsymbol{x}_i\) denote the vector

@@ -629,8 +629,8 @@ Let \(\boldsymbol{x}_i\) denot \]

which equals the vector \(\boldsymbol{x}\) with the exception that observation number \(i\) is left out. Using this notation, define -\(\widehat{\beta}_i\) to be the estimator -\(\widehat{\beta}\) computed using \(\vec{X}_i\).

+\(\widehat{\theta}_i\) to be the estimator +\(\widehat{\theta}\) computed using \(\vec{X}_i\).

from numpy import *
@@ -677,27 +677,27 @@ advantages:

 
  • It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.
    • 
 
  • It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).
    • 
 
-
    Since \(\widehat{\beta} = \widehat{\beta}(\boldsymbol{X})\) is a function of random variables,
-\(\widehat{\beta}\) itself must be a random variable. Thus it has
+
    Since \(\widehat{\theta} = \widehat{\theta}(\boldsymbol{X})\) is a function of random variables,
+\(\widehat{\theta}\) itself must be a random variable. Thus it has
 a pdf, call this function \(p(\boldsymbol{t})\). The aim of the bootstrap is to
 estimate \(p(\boldsymbol{t})\) by the relative frequency of
-\(\widehat{\beta}\). You can think of this as using a histogram
+\(\widehat{\theta}\). You can think of this as using a histogram
 in the place of \(p(\boldsymbol{t})\). If the relative frequency closely
 resembles \(p(\vec{t})\), then using numerics, it is straight forward to
 estimate all the interesting parameters of \(p(\boldsymbol{t})\) using point
 estimators.
    
-
    In the case that \(\widehat{\beta}\) has
+
    In the case that \(\widehat{\theta}\) has
 more than one component, and the components are independent, we use the
 same estimator on each component separately.  If the probability
 density function of \(X_i\), \(p(x)\), had been known, then it would have
 been straight forward to do this by:
    
 
      
 
    1. Drawing lots of numbers from \(p(x)\), suppose we call one such set of numbers \((X_1^*, X_2^*, \cdots, X_n^*)\).
      2. 
-
    2. Then using these numbers, we could compute a replica of \(\widehat{\beta}\) called \(\widehat{\beta}^*\).
      3. 
+
    3. Then using these numbers, we could compute a replica of \(\widehat{\theta}\) called \(\widehat{\theta}^*\).
      4. 
 
    
 
    By repeated use of (1) and (2), many
-estimates of \(\widehat{\beta}\) could have been obtained. The
-idea is to use the relative frequency of \(\widehat{\beta}^*\)
+estimates of \(\widehat{\theta}\) could have been obtained. The
+idea is to use the relative frequency of \(\widehat{\theta}^*\)
 (think of a histogram) as an estimate of \(p(\boldsymbol{t})\).
    
 
    But
 unless there is enough information available about the process that
@@ -715,18 +715,18 @@ frequency of the observation \(X_i\)<
 
      
 
    1. Draw with replacement \(n\) numbers for the observed variables \(\boldsymbol{x} = (x_1,x_2,\cdots,x_n)\).
      2. 
 
    2. Define a vector \(\boldsymbol{x}^*\) containing the values which were drawn from \(\boldsymbol{x}\).
      3. 
-
    3. Using the vector \(\boldsymbol{x}^*\) compute \(\widehat{\beta}^*\) by evaluating \(\widehat \beta\) under the observations \(\boldsymbol{x}^*\).
      4. 
+
    4. Using the vector \(\boldsymbol{x}^*\) compute \(\widehat{\theta}^*\) by evaluating \(\widehat \theta\) under the observations \(\boldsymbol{x}^*\).
      5. 
 
    5. Repeat this process \(k\) times.
      6. 
 
    
 
    When you are done, you can draw a histogram of the relative frequency
-of \(\widehat \beta^*\). This is your estimate of the probability
+of \(\widehat \theta^*\). This is your estimate of the probability
 distribution \(p(t)\). Using this probability distribution you can
 estimate any statistics thereof. In principle you never draw the
-histogram of the relative frequency of \(\widehat{\beta}^*\). Instead
+histogram of the relative frequency of \(\widehat{\theta}^*\). Instead
 you use the estimators corresponding to the statistic of interest. For
 example, if you are interested in estimating the variance of \(\widehat
-\beta\), apply the estimator \(\widehat \sigma^2\) to the values
-\(\widehat \beta^*\).
    
+\theta\), apply the estimator \(\widehat \sigma^2\) to the values
+\(\widehat \theta^*\).
    
 
    Before we proceed however, we need to remind ourselves about a central
 theorem in statistics, namely the so-called central limit theorem.
 This theorem plays a central role in understanding why the Bootstrap
@@ -911,12 +911,12 @@ tasks. Consider a dataset \(\mathcal{
 
    where \(\epsilon\) is normally distributed with mean zero and standard deviation \(\sigma^2\).
    
 
    In our derivation of the ordinary least squares method we defined then
 an approximation to the function \(f\) in terms of the parameters
-\(\boldsymbol{\beta}\) and the design matrix \(\boldsymbol{X}\) which embody our model,
-that is \(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta}\).
    
-
    Thereafter we found the parameters \(\boldsymbol{\beta}\) by optimizing the means squared error via the so-called cost function
    
+\(\boldsymbol{\theta}\) and the design matrix \(\boldsymbol{X}\) which embody our model,
+that is \(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\theta}\).
    
+
    Thereafter we found the parameters \(\boldsymbol{\theta}\) by optimizing the means squared error via the so-called cost function
    
 
    
 \[
-C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
+C(\boldsymbol{X},\boldsymbol{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
 \]
    
 
    We can rewrite this as
    
 
    
@@ -1267,20 +1267,20 @@ cross-validation (LOOCV).
    
 
    
 \[
 \begin{align*}
-\boldsymbol{\beta}_{-i}(\lambda) & =  ( \boldsymbol{X}_{-i, \ast}^{T}
+\boldsymbol{\theta}_{-i}(\lambda) & =  ( \boldsymbol{X}_{-i, \ast}^{T}
 \boldsymbol{X}_{-i, \ast} + \lambda \boldsymbol{I}_{pp})^{-1}
 \boldsymbol{X}_{-i, \ast}^{T} \boldsymbol{y}_{-i}
 \end{align*}
 \]
    
 
      
-
    • Evaluate the prediction performance of these models on the test set by \(\log\{L[y_i, \boldsymbol{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\}\). Or, by the prediction error \(|y_i - \boldsymbol{X}_{i, \ast} \boldsymbol{\beta}_{-i}(\lambda)|\), the relative error, the error squared or the R2 score function.
      • 
+
    • Evaluate the prediction performance of these models on the test set by \(\log\{L[y_i, \boldsymbol{X}_{i, \ast}; \boldsymbol{\theta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\}\). Or, by the prediction error \(|y_i - \boldsymbol{X}_{i, \ast} \boldsymbol{\theta}_{-i}(\lambda)|\), the relative error, the error squared or the R2 score function.
      • 
 
    • Repeat the first three steps  such that each sample plays the role of the test set once.
      • 
 
    • Average the prediction performances of the test sets at each grid point of the penalty bias/parameter. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as
      • 
 
    
 
    
 \[
 \begin{align*}
-\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\}.
+\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \boldsymbol{\theta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\}.
 \end{align*}
 \]
    
 
    For the various values of \(k\)
    
@@ -1483,8 +1483,8 @@ Thus, if we cannot assume that the expected outputs/targets are zero
 when all predictors are zero (the columns in the design matrix), it
 may be a bad idea to implement a model which penalizes the intercept.
 Furthermore, in for example Ridge and Lasso regression, the default solutions
-from the library Scikit-Learn (when not shrinking \(\beta_0\)) for the unknown parameters
-\(\boldsymbol{\beta}\), are derived under the assumption that both \(\boldsymbol{y}\) and
+from the library Scikit-Learn (when not shrinking \(\theta_0\)) for the unknown parameters
+\(\boldsymbol{\theta}\), are derived under the assumption that both \(\boldsymbol{y}\) and
 \(\boldsymbol{X}\) are zero centered, that is we subtract the mean values.
    
 
    If our predictors represent different scales, then it is important to
 standardize the design matrix \(\boldsymbol{X}\) by subtracting the mean of each
@@ -1537,39 +1537,39 @@ simplicity, we will focus on  ordinary regression, as done in the above example.
 
    The cost/loss function  for regression is
    
 
    
 \[
-C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,.
+C(\theta_0, \theta_1, ... , \theta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij}\theta_j\right)^2,.
 \]
    
 
    Recall also that we use the squared value. This expression can lead to an
 increased penalty for higher differences between predicted and
 output/target values.
    
-
    What we have done is to single out the \(\beta_0\) term in the
+
    What we have done is to single out the \(\theta_0\) term in the
 definition of the mean squared error (MSE).  The design matrix \(X\)
 does in this case not contain any intercept column.  When we take the
-derivative with respect to \(\beta_0\), we want the derivative to obey
    
+derivative with respect to \(\theta_0\), we want the derivative to obey
    
 
    
 \[
-\frac{\partial C}{\partial \beta_j} = 0,
+\frac{\partial C}{\partial \theta_j} = 0,
 \]
    
-
    for all \(j\). For \(\beta_0\) we have
    
+
    for all \(j\). For \(\theta_0\) we have
    
 
    
 \[
-\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right).
+\frac{\partial C}{\partial \theta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij} \theta_j\right).
 \]
    
 
    Multiplying away the constant \(2/n\), we obtain
    
 
    
 \[
-\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j.
+\sum_{i=0}^{n-1} \theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \theta_j.
 \]
    
-
    Let us specialize first to the case where we have only two parameters \(\beta_0\) and \(\beta_1\).
-Our result for \(\beta_0\) simplifies then to
    
+
    Let us specialize first to the case where we have only two parameters \(\theta_0\) and \(\theta_1\).
+Our result for \(\theta_0\) simplifies then to
    
 
    
 \[
-n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
+n\theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \theta_1.
 \]
    
 
    We obtain then
    
 
    
 \[
-\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}.
+\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \theta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}.
 \]
    
 
    If we define
    
 
    
@@ -1584,17 +1584,17 @@ n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
 
    we have
    
 
    
 \[
-\beta_0 = \mu_y - \beta_1\mu_{\boldsymbol{x}_1}.
+\theta_0 = \mu_y - \theta_1\mu_{\boldsymbol{x}_1}.
 \]
    
-
    In the general case with more parameters than \(\beta_0\) and \(\beta_1\), we have
    
+
    In the general case with more parameters than \(\theta_0\) and \(\theta_1\), we have
    
 
    
 \[
-\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j.
+\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\theta_j.
 \]
    
 
    We can rewrite the latter equation as
    
 
    
 \[
-\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\beta_j,
+\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\theta_j,
 \]
    
 
    where we have defined
    
 
    
@@ -1605,19 +1605,19 @@ n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
 
    Replacing \(y_i\) with \(y_i - y_i - \overline{\boldsymbol{y}}\) and centering also our design matrix results in a cost function (in vector-matrix disguise)
    
 
    
 \[
-C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}).
+C(\boldsymbol{\theta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta}).
 \]
    
-
    If we minimize with respect to \(\boldsymbol{\beta}\) we have then
    
+
    If we minimize with respect to \(\boldsymbol{\theta}\) we have then
    
 
    
 \[
-\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}},
+\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}},
 \]
    
 
    where \(\boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}}\)
 and \(\tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj}\).
    
-
    For Ridge regression we need to add \(\lambda \boldsymbol{\beta}^T\boldsymbol{\beta}\) to the cost function and get then
    
+
    For Ridge regression we need to add \(\lambda \boldsymbol{\theta}^T\boldsymbol{\theta}\) to the cost function and get then
    
 
    
 \[
-\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}.
+\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}.
 \]
    
 
    What does this mean? And why do we insist on all this? Let us look at some examples.
    
 
    This code shows a simple first-order fit to a data set using the above transformed data, where we consider the role of the intercept first, by either excluding it or including it (code example thanks to  Øyvind Sigmundson Schøyen). Here our scaling of the data is done by subtracting the mean values only.
@@ -1637,15 +1637,15 @@ def MSE(y_data,y_model):
     return np.sum((y_data-y_model)**2)/n
 
 
-def fit_beta(X, y):
+def fit_theta(X, y):
     return np.linalg.pinv(X.T @ X) @ X.T @ y
 
 
-true_beta = [2, 0.5, 3.7]
+true_theta = [2, 0.5, 3.7]
 
 x = np.linspace(0, 1, 11)
 y = np.sum(
-    np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0
+    np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0
 ) + 0.1 * np.random.normal(size=len(x))
 
 degree = 3
@@ -1655,15 +1655,15 @@ X = np.zeros((len(x), degree))
 for p in range(degree):
     X[:, p] = x ** p
 
-beta = fit_beta(X, y)
+theta = fit_theta(X, y)
 
 # Intercept is included in the design matrix
 skl = LinearRegression(fit_intercept=False).fit(X, y)
 
-print(f"True beta: {true_beta}")
-print(f"Fitted beta: {beta}")
-print(f"Sklearn fitted beta: {skl.coef_}")
-ypredictOwn = X @ beta
+print(f"True theta: {true_theta}")
+print(f"Fitted theta: {theta}")
+print(f"Sklearn fitted theta: {skl.coef_}")
+ypredictOwn = X @ theta
 ypredictSKL = skl.predict(X)
 print(f"MSE with intercept column")
 print(MSE(y,ypredictOwn))
@@ -1673,7 +1673,7 @@ print(MSE(y,ypredictSKL))
 
 plt.figure()
 plt.scatter(x, y, label="Data")
-plt.plot(x, X @ beta, label="Fit")
+plt.plot(x, X @ theta, label="Fit")
 plt.plot(x, skl.predict(X), label="Sklearn (fit_intercept=False)")
 
 
@@ -1690,21 +1690,21 @@ skl = LinearRegression(fit_intercept=True).fit(X, y)
 y_offset = np.average(y, axis=0)
 X_offset = np.average(X, axis=0)
 
-beta = fit_beta(X - X_offset, y - y_offset)
-intercept = np.mean(y_offset - X_offset @ beta)
+theta = fit_theta(X - X_offset, y - y_offset)
+intercept = np.mean(y_offset - X_offset @ theta)
 
 print(f"Manual intercept: {intercept}")
-print(f"Fitted beta (wiothout intercept): {beta}")
+print(f"Fitted theta (wiothout intercept): {theta}")
 print(f"Sklearn intercept: {skl.intercept_}")
-print(f"Sklearn fitted beta (without intercept): {skl.coef_}")
-ypredictOwn = X @ beta
+print(f"Sklearn fitted theta (without intercept): {skl.coef_}")
+ypredictOwn = X @ theta
 ypredictSKL = skl.predict(X)
 print(f"MSE with Manual intercept")
 print(MSE(y,ypredictOwn+intercept))
 print(f"MSE with Sklearn intercept")
 print(MSE(y,ypredictSKL))
 
-plt.plot(x, X @ beta + intercept, "--", label="Fit (manual intercept)")
+plt.plot(x, X @ theta + intercept, "--", label="Fit (manual intercept)")
 plt.plot(x, skl.predict(X), "--", label="Sklearn (fit_intercept=True)")
 plt.grid()
 plt.legend()
@@ -1721,22 +1721,22 @@ they should.  However, when we move to for example Ridge regression,
 the way we treat the intercept may give a larger or smaller MSE,
 meaning that the MSE can be penalized by the value of the
 intercept. Not including the intercept in the fit, means that the
-regularization term does not include \(\beta_0\). For different values
+regularization term does not include \(\theta_0\). For different values
 of \(\lambda\), this may lead to different MSE values.
    
 
    To remind the reader, the regularization term, with the intercept in Ridge regression, is given by
    
 
    
 \[
-\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2,
+\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\theta_j^2,
 \]
    
 
    but when we take out the intercept, this equation becomes
    
 
    
 \[
-\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2.
+\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\theta_j^2.
 \]
    
 
    For Lasso regression we have
    
 
    
 \[
-\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert.
+\lambda \vert\vert \boldsymbol{\theta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\theta_j\vert.
 \]
    
 
    It means that, when scaling the design matrix and the outputs/targets,
 by subtracting the mean values, we have an optimization problem which
@@ -1783,20 +1783,20 @@ MSERidgePredict = np.zeros(nlambdas)
 lambdas = np.logspace(-4, 2, nlambdas)
 for i in range(nlambdas):
     lmb = lambdas[i]
-    OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
+    OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
     # Note: we include the intercept column and no scaling
     RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
     RegRidge.fit(X_train,y_train)
     # and then make the prediction
-    ytildeOwnRidge = X_train @ OwnRidgeBeta
-    ypredictOwnRidge = X_test @ OwnRidgeBeta
+    ytildeOwnRidge = X_train @ OwnRidgeTheta
+    ypredictOwnRidge = X_test @ OwnRidgeTheta
     ytildeRidge = RegRidge.predict(X_train)
     ypredictRidge = RegRidge.predict(X_test)
     MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
     MSERidgePredict[i] = MSE(y_test,ypredictRidge)
-    print("Beta values for own Ridge implementation")
-    print(OwnRidgeBeta)
-    print("Beta values for Scikit-Learn Ridge implementation")
+    print("Theta values for own Ridge implementation")
+    print(OwnRidgeTheta)
+    print("Theta values for Scikit-Learn Ridge implementation")
     print(RegRidge.coef_)
     print("MSE values for own Ridge implementation")
     print(MSEOwnRidgePredict[i])
@@ -1869,18 +1869,18 @@ MSERidgePredict = np.zeros(nlambdas)
 lambdas = np.logspace(-4, 2, nlambdas)
 for i in range(nlambdas):
     lmb = lambdas[i]
-    OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)
-    intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data
+    OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)
+    intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data
     #Add intercept to prediction
-    ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler 
+    ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler 
     RegRidge = linear_model.Ridge(lmb)
     RegRidge.fit(X_train,y_train)
     ypredictRidge = RegRidge.predict(X_test)
     MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
     MSERidgePredict[i] = MSE(y_test,ypredictRidge)
-    print("Beta values for own Ridge implementation")
-    print(OwnRidgeBeta) #Intercept is given by mean of target variable
-    print("Beta values for Scikit-Learn Ridge implementation")
+    print("Theta values for own Ridge implementation")
+    print(OwnRidgeTheta) #Intercept is given by mean of target variable
+    print("Theta values for Scikit-Learn Ridge implementation")
     print(RegRidge.coef_)
     print('Intercept from own implementation:')
     print(intercept_)
@@ -1907,7 +1907,7 @@ plt.show()
 
    We see here, when compared to the code which includes explicitely the
 intercept column, that our MSE value is actually smaller. This is
 because the regularization term does not include the intercept value
-\(\beta_0\) in the fitting.  This applies to Lasso regularization as
+\(\theta_0\) in the fitting.  This applies to Lasso regularization as
 well.  It means that our optimization is now done only with the
 centered matrix and/or vector that enter the fitting procedure. Note
 also that the problem with the intercept occurs mainly in these type
@@ -1994,7 +1994,7 @@ with the form utilized in linear regression, that is
    
 
    
 \[
 \begin{equation}
-    \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon},
+    \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\theta} + \boldsymbol{\epsilon},
 \label{_auto4} \tag{4}
 \end{equation}
 \]
    
@@ -2016,15 +2016,15 @@ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
 
    
 \[
 \begin{equation}
-    C(\boldsymbol{X}, \boldsymbol{\beta})= \frac{1}{n}\left\{(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})\right\}.
+    C(\boldsymbol{X}, \boldsymbol{\theta})= \frac{1}{n}\left\{(\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y})\right\}.
 \label{_auto5} \tag{5}
 \end{equation}
 \]
    
-
    We then find the extremal point of \(C\) by taking the derivative with respect to \(\boldsymbol{\beta}\) as discussed above.
-This yields the expression for \(\boldsymbol{\beta}\) to be
    
+
    We then find the extremal point of \(C\) by taking the derivative with respect to \(\boldsymbol{\theta}\) as discussed above.
+This yields the expression for \(\boldsymbol{\theta}\) to be
    
 
    
 \[
-\boldsymbol{\beta} = \frac{\boldsymbol{X}^T \boldsymbol{y}}{\boldsymbol{X}^T \boldsymbol{X}},
+\boldsymbol{\theta} = \frac{\boldsymbol{X}^T \boldsymbol{y}}{\boldsymbol{X}^T \boldsymbol{X}},
 \]
    
 
    which immediately imposes some requirements on \(\boldsymbol{X}\) as there must exist
 an inverse of \(\boldsymbol{X}^T \boldsymbol{X}\). If the expression we are modeling contains an
@@ -2047,10 +2047,10 @@ X_test_own = np.concatenate(
 
    Doing the inversion directly turns out to be a bad idea since the matrix
 \(\boldsymbol{X}^T\boldsymbol{X}\) is singular. An alternative approach is to use the singular
 value decomposition. Using the definition of the Moore-Penrose
-pseudoinverse we can write the equation for \(\boldsymbol{\beta}\) as
    
+pseudoinverse we can write the equation for \(\boldsymbol{\theta}\) as
    
 
    
 \[
-\boldsymbol{\beta} = \boldsymbol{X}^{+}\boldsymbol{y},
+\boldsymbol{\theta} = \boldsymbol{X}^{+}\boldsymbol{y},
 \]
    
 
    where the pseudoinverse of \(\boldsymbol{X}\) is given by
    
 
    
@@ -2066,7 +2066,7 @@ where \(X^{+} = V\Sigma^{+} U^T\)
 \[
 \begin{equation}
-    \boldsymbol{\beta} = \boldsymbol{V}\boldsymbol{\Sigma}^{+} \boldsymbol{U}^T \boldsymbol{y}.
+    \boldsymbol{\theta} = \boldsymbol{V}\boldsymbol{\Sigma}^{+} \boldsymbol{U}^T \boldsymbol{y}.
 \label{_auto6} \tag{6}
 \end{equation}
 \]
    
@@ -2087,7 +2087,7 @@ linear system as an equation would reduce this down to
 
    
 
    
 
    
-
    beta = ols_svd(X_train_own,y_train)
+
    theta = ols_svd(X_train_own,y_train)
 
    
 
    
 
    
@@ -2095,7 +2095,7 @@ linear system as an equation would reduce this down to
 
    When extracting the \(J\)-matrix  we need to make sure that we remove the intercept, as is done here
    
 
    
 
    
-
    J = beta[1:].reshape(L, L)
+
    J = theta[1:].reshape(L, L)
 
    
 
    
 
    
@@ -2195,7 +2195,7 @@ with the form utilized in linear regression, viz.
    
 
    
 \[
 \begin{equation}
-    \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}.
+    \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\theta} + \boldsymbol{\epsilon}.
 \label{_auto10} \tag{10}
 \end{equation}
 \]
    
@@ -2256,14 +2256,14 @@ plt.show()
 
    Having explored the ordinary least squares we move on to ridge
 regression. In ridge regression we include a regularizer. This
 involves a new cost function which leads to a new estimate for the
-weights \(\boldsymbol{\beta}\). This results in a penalized regression problem. The
+weights \(\boldsymbol{\theta}\). This results in a penalized regression problem. The
 cost function is given by
    
 
 
    
 
    
 \[
 \begin{equation}
-    C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \boldsymbol{\beta}^T\boldsymbol{\beta}.
+    C(\boldsymbol{X}, \boldsymbol{\theta}; \lambda) = (\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y}) + \lambda \boldsymbol{\theta}^T\boldsymbol{\theta}.
 \label{_auto11} \tag{11}
 \end{equation}
 \]
    
@@ -2291,7 +2291,7 @@ plt.show()
 
    
 \[
 \begin{equation}
-    C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \sqrt{\boldsymbol{\beta}^T\boldsymbol{\beta}}.
+    C(\boldsymbol{X}, \boldsymbol{\theta}; \lambda) = (\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\theta} - \boldsymbol{y}) + \lambda \sqrt{\boldsymbol{\theta}^T\boldsymbol{\theta}}.
 \label{_auto12} \tag{12}
 \end{equation}
 \]
    
@@ -2505,7 +2505,7 @@ distribution \(N(0,1)\).
    
 value decomposition from e.g., numpy ) or use your code from
 homeworks 1 and 2 and perform a standard least square regression
 analysis using polynomials in \(x\) and \(y\) up to fifth order. Find the
-confidence intervals of the parameters (estimators) \(\beta\) by computing their
+confidence intervals of the parameters (estimators) \(\theta\) by computing their
 variances, evaluate the Mean Squared error (MSE)
    
 
    
 \[
@@ -2565,13 +2565,13 @@ dataset \(\mathcal{L}\) consis
 deviation \(\sigma^2\).
    
 
    In our derivation of the ordinary least squares method we defined then
 an approximation to the function \(f\) in terms of the parameters
-\(\boldsymbol{\beta}\) and the design matrix \(\boldsymbol{X}\) which embody our model,
-that is \(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta}\).
    
-
    The parameters \(\boldsymbol{\beta}\) are in turn found by optimizing the means
+\(\boldsymbol{\theta}\) and the design matrix \(\boldsymbol{X}\) which embody our model,
+that is \(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\theta}\).
    
+
    The parameters \(\boldsymbol{\theta}\) are in turn found by optimizing the means
 squared error via the so-called cost function
    
 
    
 \[
-C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
+C(\boldsymbol{X},\boldsymbol{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
 \]
    
 
    Here the expected value \(\mathbb{E}\) is the sample value.
    
 
    Show that you can rewrite  this as
    
diff --git a/doc/LectureNotes/_build/html/chapter4.html b/doc/LectureNotes/_build/html/chapter4.html
index 44c94c685..6cdd43fff 100644
--- a/doc/LectureNotes/_build/html/chapter4.html
+++ b/doc/LectureNotes/_build/html/chapter4.html
@@ -403,7 +403,7 @@ independent variables \(x_i\).
 analytical expressions for standard ordinary Least Squares or Ridge
 regression (in terms of matrices to invert) for several quantities,
 ranging from the variance and thereby the confidence intervals of the
-optimal parameters \(\hat{\beta}\) to the mean squared error. If we can invert
+optimal parameters \(\hat{\theta}\) to the mean squared error. If we can invert
 the product of the design matrices, linear regression gives then a
 simple recipe for fitting our data.
    
 
    Classification problems, however, are concerned with outcomes taking
@@ -423,7 +423,7 @@ failure etc.
    
 
    Logistic regression will also serve as our stepping stone towards
 neural network algorithms and supervised deep learning. For logistic
 learning, the minimization of the cost function leads to a non-linear
-equation in the parameters \(\hat{\beta}\). The optimization of the
+equation in the parameters \(\hat{\theta}\). The optimization of the
 problem calls therefore for minimization algorithms. This forms the
 bottle neck of all machine learning algorithms, namely how to find
 reliable minima of a multi-variable function. This leads us to the
@@ -462,12 +462,12 @@ weighted linear combination, namely
    
 
    
 \[
 \begin{equation}
-\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\beta} +  \boldsymbol{\epsilon},
+\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\theta} +  \boldsymbol{\epsilon},
 \label{_auto1} \tag{1}
 \end{equation}
 \]
    
 
    where \(\boldsymbol{y}\) is a vector representing the possible outcomes, \(\boldsymbol{X}\) is our
-\(n\times p\) design matrix and \(\boldsymbol{\beta}\) represents our estimators/predictors.
    
+\(n\times p\) design matrix and \(\boldsymbol{\theta}\) represents our estimators/predictors.
    
 
    The main problem with our function is that it takes values on the
 entire real axis. In the case of logistic regression, however, the
 labels \(y_i\) are discrete variables. A typical example is the credit
@@ -561,7 +561,7 @@ plt.show()
 In standard linear regression with a linear dependence on \(x\), we would write this in terms of our model
    
 
    
 \[
-f(y_i\vert x_i)=\beta_0+\beta_1 x_i.
+f(y_i\vert x_i)=\theta_0+\theta_1 x_i.
 \]
    
 
    This expression implies however that \(f(y_i\vert x_i)\) could take any
 value from minus infinity to plus infinity. If we however let
@@ -656,19 +656,19 @@ plt.show()
 
    
 
    
 
    
-
    We assume now that we have two classes with \(y_i\) either \(0\) or \(1\). Furthermore we assume also that we have only two parameters \(\beta\) in our fitting of the Sigmoid function, that is we define probabilities
    
+
    We assume now that we have two classes with \(y_i\) either \(0\) or \(1\). Furthermore we assume also that we have only two parameters \(\theta\) in our fitting of the Sigmoid function, that is we define probabilities
    
 
    
 \[\begin{split}
 \begin{align*}
-p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
-p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
+p(y_i=1|x_i,\boldsymbol{\theta}) &= \frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}},\nonumber\\
+p(y_i=0|x_i,\boldsymbol{\theta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\theta}),
 \end{align*}
 \end{split}\]
    
-
    where \(\boldsymbol{\beta}\) are the weights we wish to extract from data, in our case \(\beta_0\) and \(\beta_1\).
    
+
    where \(\boldsymbol{\theta}\) are the weights we wish to extract from data, in our case \(\theta_0\) and \(\theta_1\).
    
 
    Note that we used
    
 
    
 \[
-p(y_i=0\vert x_i, \boldsymbol{\beta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\beta}).
+p(y_i=0\vert x_i, \boldsymbol{\theta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\theta}).
 \]
    
 
    In order to define the total likelihood for all possible outcomes from a
    
 dataset \(\mathcal{D}=\{(y_i,x_i)\}\), with the binary labels
@@ -679,80 +679,80 @@ likelihood in terms of the product of the individual probabilities of a specific
 
    
 \[\begin{split}
 \begin{align*}
-P(\mathcal{D}|\boldsymbol{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]^{1-y_i}\nonumber \\
+P(\mathcal{D}|\boldsymbol{\theta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\theta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\theta}))\right]^{1-y_i}\nonumber \\
 \end{align*}
 \end{split}\]
    
 
    from which we obtain the log-likelihood and our cost/loss function
    
 
    
 \[
-\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\beta}))\right]\right).
+\mathcal{C}(\boldsymbol{\theta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\theta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\theta}))\right]\right).
 \]
    
 
    Reordering the logarithms, we can rewrite the cost/loss function as
    
 
    
 \[
-\mathcal{C}(\boldsymbol{\beta}) = \sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
+\mathcal{C}(\boldsymbol{\theta}) = \sum_{i=1}^n  \left(y_i(\theta_0+\theta_1x_i) -\log{(1+\exp{(\theta_0+\theta_1x_i)})}\right).
 \]
    
-
    The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \(\beta\).
+
    The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \(\theta\).
 Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
    
 
    
 \[
-\mathcal{C}(\boldsymbol{\beta})=-\sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
+\mathcal{C}(\boldsymbol{\theta})=-\sum_{i=1}^n  \left(y_i(\theta_0+\theta_1x_i) -\log{(1+\exp{(\theta_0+\theta_1x_i)})}\right).
 \]
    
 
    This equation is known in statistics as the cross entropy. Finally, we note that just as in linear regression,
 in practice we often supplement the cross-entropy with additional regularization terms, usually \(L_1\) and \(L_2\) regularization as we did for Ridge and Lasso regression.
    
-
    The cross entropy is a convex function of the weights \(\boldsymbol{\beta}\) and,
+
    The cross entropy is a convex function of the weights \(\boldsymbol{\theta}\) and,
 therefore, any local minimizer is a global minimizer.
    
 
    Minimizing this
-cost function with respect to the two parameters \(\beta_0\) and \(\beta_1\) we obtain
    
+cost function with respect to the two parameters \(\theta_0\) and \(\theta_1\) we obtain
    
 
    
 \[
-\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_0} = -\sum_{i=1}^n  \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
+\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \theta_0} = -\sum_{i=1}^n  \left(y_i -\frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}}\right),
 \]
    
 
    and
    
 
    
 \[
-\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \beta_1} = -\sum_{i=1}^n  \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
+\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \theta_1} = -\sum_{i=1}^n  \left(y_ix_i -x_i\frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}}\right).
 \]
    
 
    Let us now define a vector \(\boldsymbol{y}\) with \(n\) elements \(y_i\), an
 \(n\times p\) matrix \(\boldsymbol{X}\) which contains the \(x_i\) values and a
-vector \(\boldsymbol{p}\) of fitted probabilities \(p(y_i\vert x_i,\boldsymbol{\beta})\). We can rewrite in a more compact form the first
+vector \(\boldsymbol{p}\) of fitted probabilities \(p(y_i\vert x_i,\boldsymbol{\theta})\). We can rewrite in a more compact form the first
 derivative of cost function as
    
 
    
 \[
-\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
+\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
 \]
    
 
    If we in addition define a diagonal matrix \(\boldsymbol{W}\) with elements
-\(p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta})\), we can obtain a compact expression of the second derivative as
    
+\(p(y_i\vert x_i,\boldsymbol{\theta})(1-p(y_i\vert x_i,\boldsymbol{\theta})\), we can obtain a compact expression of the second derivative as
    
 
    
 \[
-\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
+\frac{\partial^2 \mathcal{C}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
 \]
    
 
    Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \(p\) predictors
    
 
    
 \[
-\log{ \frac{p(\boldsymbol{\beta}\boldsymbol{x})}{1-p(\boldsymbol{\beta}\boldsymbol{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
+\log{ \frac{p(\boldsymbol{\theta}\boldsymbol{x})}{1-p(\boldsymbol{\theta}\boldsymbol{x})}} = \theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p.
 \]
    
-
    Here we defined \(\boldsymbol{x}=[1,x_1,x_2,\dots,x_p]\) and \(\boldsymbol{\beta}=[\beta_0, \beta_1, \dots, \beta_p]\) leading to
    
+
    Here we defined \(\boldsymbol{x}=[1,x_1,x_2,\dots,x_p]\) and \(\boldsymbol{\theta}=[\theta_0, \theta_1, \dots, \theta_p]\) leading to
    
 
    
 \[
-p(\boldsymbol{\beta}\boldsymbol{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
+p(\boldsymbol{\theta}\boldsymbol{x})=\frac{ \exp{(\theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p)}}{1+\exp{(\theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p)}}.
 \]
    
 
    Till now we have mainly focused on two classes, the so-called binary
 system. Suppose we wish to extend to \(K\) classes.  Let us for the sake
 of simplicity assume we have only two predictors. We have then following model
    
 
    
 \[
-\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1,
+\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \theta_{10}+\theta_{11}x_1,
 \]
    
 
    and
    
 
    
 \[
-\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
+\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \theta_{20}+\theta_{21}x_1,
 \]
    
 
    and so on till the class \(C=K-1\) class
    
 
    
 \[
-\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
+\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \theta_{(K-1)0}+\theta_{(K-1)1}x_1,
 \]
    
 
    and the model is specified in term of \(K-1\) so-called log-odds or
 logit transformations.
    
@@ -765,16 +765,16 @@ Bayes classifiers, and artificial neural networks.  Specifically, in
 multinomial logistic regression and linear discriminant analysis, the
 input to the function is the result of \(K\) distinct linear functions,
 and the predicted probability for the \(k\)-th class given a sample
-vector \(\boldsymbol{x}\) and a weighting vector \(\boldsymbol{\beta}\) is (with two
+vector \(\boldsymbol{x}\) and a weighting vector \(\boldsymbol{\theta}\) is (with two
 predictors):
    
 
    
 \[
-p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
+p(C=k\vert \mathbf {x} )=\frac{\exp{(\theta_{k0}+\theta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\theta_{l0}+\theta_{l1}x_1)}}.
 \]
    
 
    It is easy to extend to more predictors. The final class is
    
 
    
 \[
-p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
+p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\theta_{l0}+\theta_{l1}x_1)}},
 \]
    
 
    and they sum to one. Our earlier discussions were all specialized to
 the case with two classes only. It is easy to see from the above that
diff --git a/doc/LectureNotes/_build/html/genindex.html b/doc/LectureNotes/_build/html/genindex.html
index 907e860ee..905369861 100644
--- a/doc/LectureNotes/_build/html/genindex.html
+++ b/doc/LectureNotes/_build/html/genindex.html
@@ -31,7 +31,7 @@
     
     
     
-    
+    
     
     
   
diff --git a/doc/LectureNotes/_build/html/intro.html b/doc/LectureNotes/_build/html/intro.html
index 19dc216b0..179fca5f2 100644
--- a/doc/LectureNotes/_build/html/intro.html
+++ b/doc/LectureNotes/_build/html/intro.html
@@ -32,7 +32,7 @@
     
     
     
-    
+    
     
     
   
diff --git a/doc/LectureNotes/_build/html/search.html b/doc/LectureNotes/_build/html/search.html
index 31483789c..506889057 100644
--- a/doc/LectureNotes/_build/html/search.html
+++ b/doc/LectureNotes/_build/html/search.html
@@ -30,7 +30,7 @@
     
     
     
-    
+    
     
     
   
diff --git a/doc/LectureNotes/_build/html/searchindex.js b/doc/LectureNotes/_build/html/searchindex.js
index 6e819f059..b29a6249f 100644
--- a/doc/LectureNotes/_build/html/searchindex.js
+++ b/doc/LectureNotes/_build/html/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"A Classification Tree": [[11, "a-classification-tree"]], "A Frequentist approach to data analysis": [[2, "a-frequentist-approach-to-data-analysis"], [23, "a-frequentist-approach-to-data-analysis"]], "A better approach": [[10, "a-better-approach"]], "A first summary": [[23, "a-first-summary"]], "A quick Reminder on Lagrangian Multipliers": [[10, "a-quick-reminder-on-lagrangian-multipliers"]], "A simple example": [[6, "a-simple-example"]], "A soft classifier": [[10, "a-soft-classifier"]], "A top-down perspective on Neural networks": [[3, "a-top-down-perspective-on-neural-networks"]], "ADAM optimizer": [[15, "adam-optimizer"]], "Activation functions": [[14, "activation-functions"]], "Adaptive boosting: AdaBoost, Basic Algorithm": [[12, "adaptive-boosting-adaboost-basic-algorithm"]], "Adding error analysis and training set up": [[23, "adding-error-analysis-and-training-set-up"]], "Adjust hyperparameters": [[3, "adjust-hyperparameters"]], "Algorithms for Setting up Decision Trees": [[11, "algorithms-for-setting-up-decision-trees"]], "An Overview of Ensemble Methods": [[12, "an-overview-of-ensemble-methods"]], "An extrapolation example": [[6, "an-extrapolation-example"]], "An optimization/minimization problem": [[23, "an-optimization-minimization-problem"]], "And what about using neural networks?": [[23, "and-what-about-using-neural-networks"]], "Another example, the moons again": [[11, "another-example-the-moons-again"]], "Applied Data Analysis and Machine Learning": [[17, null]], "Autocorrelation function": [[20, "autocorrelation-function"]], "Automatic differentiation": [[15, "automatic-differentiation"]], "Back to the Cancer Data": [[13, "back-to-the-cancer-data"]], "Bagging": [[12, "bagging"]], "Bagging Examples": [[12, "bagging-examples"]], "Basic Matrix Features": [[18, "basic-matrix-features"]], "Basic ideas of the Principal Component Analysis (PCA)": [[13, null]], "Basic math of the SVD": [[7, "basic-math-of-the-svd"]], "Basics": [[9, "basics"]], "Basics of a tree": [[11, "basics-of-a-tree"]], "Batch Normalization": [[3, "batch-normalization"]], "Bayes\u2019 Theorem and Ridge and Lasso Regression": [[7, "bayes-theorem-and-ridge-and-lasso-regression"]], "Boosting, a Bird\u2019s Eye View": [[12, "boosting-a-bird-s-eye-view"]], "Bootstrap": [[8, "bootstrap"]], "Bringing it together, first back propagation equation": [[14, "bringing-it-together-first-back-propagation-equation"]], "Building a Feed Forward Neural Network": [[3, null]], "Building a tree, regression": [[11, "building-a-tree-regression"]], "Building neural networks in Tensorflow and Keras": [[3, "building-neural-networks-in-tensorflow-and-keras"]], "CNNs in more detail, building convolutional neural networks in Tensorflow and Keras": [[5, "cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras"]], "Cancer Data again now with Decision Trees and other Methods": [[11, "cancer-data-again-now-with-decision-trees-and-other-methods"]], "Choose cost function and optimizer": [[3, "choose-cost-function-and-optimizer"]], "Classical PCA Theorem": [[13, "classical-pca-theorem"]], "Clustering and Unsupervised Learning": [[16, null]], "Code for SVD and Inversion of Matrices": [[7, "code-for-svd-and-inversion-of-matrices"]], "Codes and Approaches": [[16, "codes-and-approaches"]], "Codes for the SVD": [[7, "codes-for-the-svd"]], "Coding Setup and Linear Regression": [[0, "coding-setup-and-linear-regression"]], "Collect and pre-process data": [[3, "collect-and-pre-process-data"]], "Communication channels": [[23, "communication-channels"]], "Compare  Bagging on Trees with Random Forests": [[12, "compare-bagging-on-trees-with-random-forests"]], "Comparing with a numerical scheme": [[4, "comparing-with-a-numerical-scheme"]], "Computing the Gini index": [[11, "computing-the-gini-index"]], "Conjugate gradient method": [[15, "conjugate-gradient-method"]], "Convex functions": [[15, "convex-functions"]], "Convolution Examples: Polynomial multiplication": [[5, "convolution-examples-polynomial-multiplication"]], "Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)": [[5, "convolution-examples-principle-of-superposition-and-periodic-forces-fourier-transforms"]], "Convolutional Neural Network": [[14, "convolutional-neural-network"]], "Convolutional Neural Networks": [[5, null]], "Correlation Matrix": [[13, "correlation-matrix"]], "Course Format": [[23, "course-format"]], "Course setting": [[19, null]], "Cross-validation": [[8, "cross-validation"]], "Deadlines for projects (tentative)": [[23, "deadlines-for-projects-tentative"]], "Decision trees, overarching aims": [[11, null]], "Deep learning methods": [[23, "deep-learning-methods"]], "Define model and architecture": [[3, "define-model-and-architecture"]], "Defining the cost function": [[3, "defining-the-cost-function"]], "Deliverables": [[0, "deliverables"], [1, "deliverables"]], "Derivatives and the chain rule": [[14, "derivatives-and-the-chain-rule"]], "Deriving OLS from a probability distribution": [[7, "deriving-ols-from-a-probability-distribution"]], "Deriving and Implementing Ordinary Least Squares": [[1, "deriving-and-implementing-ordinary-least-squares"]], "Deriving the back propagation code for a multilayer perceptron model": [[14, "deriving-the-back-propagation-code-for-a-multilayer-perceptron-model"]], "Developing a code for doing neural networks with back propagation": [[3, "developing-a-code-for-doing-neural-networks-with-back-propagation"]], "Diagonalize the sample covariance matrix to obtain the principal components": [[13, "diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components"]], "Different kernels and Mercer\u2019s theorem": [[10, "different-kernels-and-mercer-s-theorem"]], "Disadvantages": [[11, "disadvantages"]], "Discriminative Modeling": [[23, "discriminative-modeling"]], "Domains and probabilities": [[20, "domains-and-probabilities"]], "Dropout": [[3, "dropout"]], "Elements of Probability Theory and Statistical Data Analysis": [[20, null]], "Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods": [[12, null]], "Entropy and the ID3 algorithm": [[11, "entropy-and-the-id3-algorithm"]], "Essential elements of ML": [[23, "essential-elements-of-ml"]], "Evaluate model performance on test data": [[3, "evaluate-model-performance-on-test-data"]], "Example of discriminative modeling, taken from Generative Deeep Learning by David Foster": [[23, "example-of-discriminative-modeling-taken-from-generative-deeep-learning-by-david-foster"]], "Example of generative modeling, taken from Generative Deep Learning by David Foster": [[23, "example-of-generative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example: Exponential decay": [[4, "example-exponential-decay"]], "Example: Population growth": [[4, "example-population-growth"]], "Example: The diffusion equation": [[4, "example-the-diffusion-equation"]], "Example: binary classification problem": [[3, "example-binary-classification-problem"]], "Examples": [[23, "examples"]], "Examples of likelihood functions used in logistic regression and neural networks": [[9, "examples-of-likelihood-functions-used-in-logistic-regression-and-neural-networks"]], "Exercise 1 - Finding the derivative of Matrix-Vector expressions": [[1, "exercise-1-finding-the-derivative-of-matrix-vector-expressions"]], "Exercise 1 - Github Setup": [[0, "exercise-1-github-setup"]], "Exercise 1: Setting up various Python environments": [[2, "exercise-1-setting-up-various-python-environments"]], "Exercise 2 - Deriving the expression for OLS": [[1, "exercise-2-deriving-the-expression-for-ols"]], "Exercise 2 - Setting up a Github repository": [[0, "exercise-2-setting-up-a-github-repository"]], "Exercise 2: making your own data and exploring scikit-learn": [[2, "exercise-2-making-your-own-data-and-exploring-scikit-learn"]], "Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression": [[1, "exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression"]], "Exercise 3 - Fitting an OLS model to data": [[0, "exercise-3-fitting-an-ols-model-to-data"]], "Exercise 3 - Setting up a Python virtual environment": [[0, "exercise-3-setting-up-a-python-virtual-environment"]], "Exercise 3: Normalizing our data": [[2, "exercise-3-normalizing-our-data"]], "Exercise 4 - Fitting a polynomial": [[1, "exercise-4-fitting-a-polynomial"]], "Exercise 4 - The train-test split": [[0, "exercise-4-the-train-test-split"]], "Exercise 4: Adding Ridge Regression": [[2, "exercise-4-adding-ridge-regression"]], "Exercise 5 - Comparing your code with sklearn": [[1, "exercise-5-comparing-your-code-with-sklearn"]], "Exercise 5: Analytical exercises": [[2, "exercise-5-analytical-exercises"]], "Exercise:  Cross-validation as resampling techniques, adding more complexity": [[8, "exercise-cross-validation-as-resampling-techniques-adding-more-complexity"]], "Exercise: Analysis of real data": [[8, "exercise-analysis-of-real-data"]], "Exercise: Bias-variance trade-off and resampling techniques": [[8, "exercise-bias-variance-trade-off-and-resampling-techniques"]], "Exercise: Lasso Regression on the Franke function  with resampling": [[8, "exercise-lasso-regression-on-the-franke-function-with-resampling"]], "Exercise: Ordinary Least Square (OLS) on the Franke function": [[8, "exercise-ordinary-least-square-ols-on-the-franke-function"]], "Exercise: Ridge Regression on the Franke function  with resampling": [[8, "exercise-ridge-regression-on-the-franke-function-with-resampling"]], "Exercises": [[2, "exercises"]], "Exercises and Projects": [[8, "exercises-and-projects"]], "Exercises week 34": [[0, null]], "Exercises week 35": [[1, null]], "Expectation values": [[20, "expectation-values"]], "Extremely useful tools, strongly recommended": [[23, "extremely-useful-tools-strongly-recommended"]], "Feed-forward neural networks": [[14, "feed-forward-neural-networks"]], "Feed-forward pass": [[3, "feed-forward-pass"]], "Final back propagating equation": [[14, "final-back-propagating-equation"]], "Fine-tuning neural network hyperparameters": [[3, "fine-tuning-neural-network-hyperparameters"]], "Fitting an Equation of State for Dense Nuclear Matter": [[2, "fitting-an-equation-of-state-for-dense-nuclear-matter"]], "From one to many layers, the universal approximation theorem": [[14, "from-one-to-many-layers-the-universal-approximation-theorem"]], "Further Dimensionality Remarks": [[5, "further-dimensionality-remarks"]], "Further properties (important for our analyses later)": [[7, "further-properties-important-for-our-analyses-later"]], "Gaussian Elimination": [[18, "gaussian-elimination"]], "General Features": [[11, "general-features"]], "General linear models and linear algebra": [[23, "general-linear-models-and-linear-algebra"]], "Generalizing the fitting procedure as a linear algebra problem": [[23, "generalizing-the-fitting-procedure-as-a-linear-algebra-problem"], [23, "id1"]], "Generative Adversarial Networks": [[6, "generative-adversarial-networks"]], "Generative Models": [[6, "generative-models"]], "Generative Versus Discriminative Modeling": [[23, "generative-versus-discriminative-modeling"]], "Geometric Interpretation and link with Singular Value Decomposition": [[13, "geometric-interpretation-and-link-with-singular-value-decomposition"]], "Gradient Boosting, Classification Example": [[12, "gradient-boosting-classification-example"]], "Gradient Boosting, Examples of Regression": [[12, "gradient-boosting-examples-of-regression"]], "Gradient Clipping": [[3, "gradient-clipping"]], "Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent": [[12, "gradient-boosting-basics-with-steepest-descent-functional-gradient-descent"]], "Gradient descent": [[4, "gradient-descent"]], "Grading": [[21, "grading"], [21, "id2"], [23, "grading"]], "Housing data, the code": [[2, "housing-data-the-code"]], "How to take derivatives of Matrix-Vector expressions": [[1, "how-to-take-derivatives-of-matrix-vector-expressions"]], "Hyperplanes and all that": [[10, "hyperplanes-and-all-that"]], "Important Matrix and vector handling packages": [[18, "important-matrix-and-vector-handling-packages"]], "Improving performance": [[3, "improving-performance"]], "In summary": [[21, "in-summary"]], "Including Stochastic Gradient Descent with Autograd": [[15, "including-stochastic-gradient-descent-with-autograd"]], "Incremental PCA": [[13, "incremental-pca"]], "Installing R, C++, cython or Julia": [[23, "installing-r-c-cython-or-julia"]], "Installing R, C++, cython, Numba etc": [[23, "installing-r-c-cython-numba-etc"]], "Instructor information": [[21, "instructor-information"]], "Interpretations and optimizing our parameters": [[23, "interpretations-and-optimizing-our-parameters"], [23, "id2"], [23, "id3"]], "Introducing JAX": [[15, "introducing-jax"]], "Introducing the Covariance and Correlation functions": [[13, "introducing-the-covariance-and-correlation-functions"]], "Introduction": [[2, "introduction"], [8, "introduction"], [17, "introduction"], [18, "introduction"]], "Iterative Fitting, Classification and AdaBoost": [[12, "iterative-fitting-classification-and-adaboost"]], "Iterative Fitting, Regression and Squared-error Cost Function": [[12, "iterative-fitting-regression-and-squared-error-cost-function"]], "Kernel PCA": [[13, "kernel-pca"]], "Kernels and non-linearity": [[10, "kernels-and-non-linearity"]], "LU Decomposition, the inverse of a matrix": [[18, "lu-decomposition-the-inverse-of-a-matrix"]], "Layers": [[3, "layers"]], "Layers used to build CNNs": [[5, "layers-used-to-build-cnns"]], "Learning goals": [[0, "learning-goals"], [1, "learning-goals"]], "Learning outcomes": [[17, "learning-outcomes"], [23, "learning-outcomes"]], "Lectures and ComputerLab": [[23, "lectures-and-computerlab"]], "Limitations of supervised learning with deep networks": [[3, "limitations-of-supervised-learning-with-deep-networks"]], "Linear Algebra, Handling of Arrays and more Python Features": [[18, null]], "Linear Regression": [[2, null]], "Linear Regression, basic elements": [[2, "linear-regression-basic-elements"]], "Linking Bayes\u2019 Theorem with Ridge and Lasso Regression": [[7, "linking-bayes-theorem-with-ridge-and-lasso-regression"]], "Linking the regression analysis with a statistical interpretation": [[7, "linking-the-regression-analysis-with-a-statistical-interpretation"]], "Linking with the SVD": [[7, "linking-with-the-svd"]], "Links to relevant courses at the University of Oslo": [[22, "links-to-relevant-courses-at-the-university-of-oslo"]], "Logistic Regression": [[9, null], [9, "id1"]], "MNIST and GANs": [[6, "mnist-and-gans"]], "Machine Learning": [[23, "machine-learning"]], "Machine learning": [[17, "machine-learning"]], "Main textbooks": [[23, "main-textbooks"]], "Making a tree": [[11, "making-a-tree"]], "Making your own Bootstrap: Changing the Level of the Decision Tree": [[12, "making-your-own-bootstrap-changing-the-level-of-the-decision-tree"]], "Mathematical Interpretation of Ordinary Least Squares": [[7, "mathematical-interpretation-of-ordinary-least-squares"]], "Mathematical optimization of convex functions": [[10, "mathematical-optimization-of-convex-functions"]], "Mathematics of CNNs": [[5, "mathematics-of-cnns"]], "Mathematics of the SVD and implications": [[7, "mathematics-of-the-svd-and-implications"]], "Matrices in Python": [[23, "matrices-in-python"]], "Matrix  multiplication": [[3, "matrix-multiplication"]], "Matrix-vector notation  and activation": [[14, "matrix-vector-notation-and-activation"]], "Meet the  covariance!": [[20, "meet-the-covariance"]], "Meet the Covariance Matrix": [[7, "meet-the-covariance-matrix"]], "Meet the Pandas": [[23, "meet-the-pandas"]], "Momentum based GD": [[15, "momentum-based-gd"]], "More complicated Example: The Ising model": [[8, "more-complicated-example-the-ising-model"]], "More on Dimensionalities": [[5, "more-on-dimensionalities"]], "More on Rescaling data": [[8, "more-on-rescaling-data"]], "Multilayer perceptrons": [[14, "multilayer-perceptrons"]], "Network requirements": [[4, "network-requirements"]], "Neural Networks vs CNNs": [[5, "neural-networks-vs-cnns"]], "Neural networks": [[14, null]], "Numerical experiments and the covariance, central limit theorem": [[20, "numerical-experiments-and-the-covariance-central-limit-theorem"]], "Numpy and arrays": [[18, "numpy-and-arrays"], [23, "numpy-and-arrays"]], "Numpy examples and Important Matrix and vector handling packages": [[23, "numpy-examples-and-important-matrix-and-vector-handling-packages"]], "Optimization, the central part of any Machine Learning algortithm": [[15, null]], "Optimizing our parameters": [[23, "optimizing-our-parameters"]], "Optimizing our parameters, more details": [[23, "optimizing-our-parameters-more-details"]], "Optimizing the cost function": [[3, "optimizing-the-cost-function"]], "Organizing our data": [[2, "organizing-our-data"], [23, "organizing-our-data"]], "Other Matrix and Vector Operations": [[18, "other-matrix-and-vector-operations"]], "Other Types of Recurrent Neural Networks": [[6, "other-types-of-recurrent-neural-networks"]], "Other courses on Data science and Machine Learning  at UiO": [[23, "other-courses-on-data-science-and-machine-learning-at-uio"]], "Other courses on Data science and Machine Learning  at UiO, contn": [[23, "other-courses-on-data-science-and-machine-learning-at-uio-contn"]], "Other popular texts": [[23, "other-popular-texts"]], "Other techniques": [[13, "other-techniques"]], "Other types of networks": [[14, "other-types-of-networks"]], "Other ways of visualizing the trees": [[11, "other-ways-of-visualizing-the-trees"]], "Our model for the nuclear binding energies": [[23, "our-model-for-the-nuclear-binding-energies"]], "Overview of first week": [[23, "overview-of-first-week"]], "Own code for Ordinary Least Squares": [[23, "own-code-for-ordinary-least-squares"]], "PCA and scikit-learn": [[13, "pca-and-scikit-learn"]], "Pandas AI": [[23, "pandas-ai"]], "Partial Differential Equations": [[4, "partial-differential-equations"]], "Practical tips": [[15, "practical-tips"]], "Practicalities": [[21, "practicalities"], [21, "id1"]], "Predicting New Points With A Trained Recurrent Neural Network": [[6, "predicting-new-points-with-a-trained-recurrent-neural-network"]], "Prerequisites": [[23, "prerequisites"]], "Prerequisites and background": [[17, "prerequisites-and-background"]], "Prerequisites: Collect and pre-process data": [[5, "prerequisites-collect-and-pre-process-data"]], "Probability Distribution Functions": [[20, "probability-distribution-functions"]], "Program for stochastic gradient": [[15, "program-for-stochastic-gradient"]], "Properties of PDFs": [[20, "properties-of-pdfs"]], "Pros and cons of trees, pros": [[11, "pros-and-cons-of-trees-pros"]], "Python installers": [[17, "python-installers"], [23, "python-installers"]], "RMS prop": [[15, "rms-prop"]], "Random Numbers": [[20, "random-numbers"]], "Random forests": [[12, "random-forests"]], "Randomized PCA": [[13, "randomized-pca"]], "Reading material": [[23, "reading-material"]], "Reading suggestions week 34": [[23, "reading-suggestions-week-34"]], "Recurrent neural networks": [[14, "recurrent-neural-networks"]], "Recurrent neural networks: Overarching view": [[6, null]], "Reducing the number of degrees of freedom, overarching view": [[2, "reducing-the-number-of-degrees-of-freedom-overarching-view"]], "Reformulating the problem": [[4, "reformulating-the-problem"]], "Regression Case": [[12, "regression-case"]], "Regression analysis, overarching aims": [[23, "regression-analysis-overarching-aims"]], "Regression analysis, overarching aims II": [[23, "regression-analysis-overarching-aims-ii"]], "Regularization": [[3, "regularization"]], "Reminder on Statistics": [[8, "reminder-on-statistics"]], "Replace or not": [[15, "replace-or-not"]], "Required Technologies": [[17, "required-technologies"]], "Resampling Methods": [[8, null]], "Resampling methods": [[8, "id1"]], "Resources on differential equations and deep learning": [[4, "resources-on-differential-equations-and-deep-learning"]], "Revisiting our Linear Regression Solvers": [[15, "revisiting-our-linear-regression-solvers"]], "Rewriting the fitting procedure as a linear algebra problem": [[23, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem"]], "Rewriting the fitting procedure as a linear algebra problem, more details": [[23, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem-more-details"]], "Ridge and Lasso Regression": [[7, null], [7, "id1"]], "Same code but now with momentum gradient descent": [[15, "same-code-but-now-with-momentum-gradient-descent"]], "Schedule first week": [[23, "schedule-first-week"]], "Schematic Regression Procedure": [[11, "schematic-regression-procedure"]], "Setting up the Back propagation algorithm": [[14, "setting-up-the-back-propagation-algorithm"]], "Setting up the network using Autograd; The full program": [[4, "setting-up-the-network-using-autograd-the-full-program"]], "Similar (second order function now) problem but now with AdaGrad": [[15, "similar-second-order-function-now-problem-but-now-with-adagrad"]], "Simple Python Code to read in Data and perform Classification": [[11, "simple-python-code-to-read-in-data-and-perform-classification"]], "Simple linear regression model using scikit-learn": [[2, "simple-linear-regression-model-using-scikit-learn"], [23, "simple-linear-regression-model-using-scikit-learn"]], "Software and needed installations": [[23, "software-and-needed-installations"]], "Solving Differential Equations  with Deep Learning": [[4, null]], "Solving the one dimensional Poisson equation": [[4, "solving-the-one-dimensional-poisson-equation"]], "Solving the wave equation with Neural Networks": [[4, "solving-the-wave-equation-with-neural-networks"]], "Some famous Matrices": [[18, "some-famous-matrices"]], "Some simple problems": [[15, "some-simple-problems"]], "Splitting our Data in Training and Test data": [[2, "splitting-our-data-in-training-and-test-data"]], "Standard steepest descent": [[15, "standard-steepest-descent"]], "Statistical analysis and optimization of data": [[17, "statistical-analysis-and-optimization-of-data"], [23, "statistical-analysis-and-optimization-of-data"]], "Steepest descent": [[15, "steepest-descent"]], "Stochastic Gradient Descent (SGD)": [[15, "stochastic-gradient-descent-sgd"]], "Stochastic variables and the main concepts, the discrete case": [[20, "stochastic-variables-and-the-main-concepts-the-discrete-case"]], "Support Vector Machines, overarching aims": [[10, null]], "Systematic reduction": [[5, "systematic-reduction"]], "Teachers": [[23, "teachers"]], "Teachers and Grading": [[21, null]], "Teaching Assistants Fall semester 2023": [[21, "teaching-assistants-fall-semester-2023"]], "Tentative deadllines for projects": [[21, "tentative-deadllines-for-projects"]], "Testing the Means Squared Error as function of Complexity": [[2, "testing-the-means-squared-error-as-function-of-complexity"]], "Textbooks": [[22, null]], "The Algorithm before theorem": [[13, "the-algorithm-before-theorem"]], "The Boston housing data example": [[2, "the-boston-housing-data-example"]], "The Breast Cancer Data, now with Keras": [[3, "the-breast-cancer-data-now-with-keras"]], "The CART algorithm for Classification": [[11, "the-cart-algorithm-for-classification"]], "The CART algorithm for Regression": [[11, "the-cart-algorithm-for-regression"]], "The CIFAR01 data set": [[5, "the-cifar01-data-set"]], "The MNIST dataset again": [[5, "the-mnist-dataset-again"]], "The RELU function family": [[3, "the-relu-function-family"]], "The Softmax function": [[3, "the-softmax-function"]], "The \\chi^2 function": [[2, "the-chi-2-function"], [23, "the-chi-2-function"], [23, "id4"], [23, "id5"], [23, "id6"], [23, "id7"], [23, "id8"]], "The bias-variance tradeoff": [[8, "the-bias-variance-tradeoff"]], "The code for solving the ODE": [[4, "the-code-for-solving-the-ode"]], "The course has two central parts": [[17, "the-course-has-two-central-parts"]], "The logistic function": [[9, "the-logistic-function"]], "The moons example": [[10, "the-moons-example"]], "The multilayer  perceptron (MLP)": [[14, "the-multilayer-perceptron-mlp"]], "The network with one input layer, specified number of hidden layers, and one output layer": [[4, "the-network-with-one-input-layer-specified-number-of-hidden-layers-and-one-output-layer"]], "The plethora  of machine learning algorithms/methods": [[23, "the-plethora-of-machine-learning-algorithms-methods"]], "The singular value decomposition": [[7, "the-singular-value-decomposition"]], "The two-dimensional case": [[10, "the-two-dimensional-case"]], "To our real data: nuclear binding energies. Brief reminder on masses and binding energies": [[23, "to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies"]], "Topics covered in this course: Statistical analysis and optimization of data": [[23, "topics-covered-in-this-course-statistical-analysis-and-optimization-of-data"]], "Towards the PCA theorem": [[13, "towards-the-pca-theorem"]], "Train and test datasets": [[3, "train-and-test-datasets"]], "Two-dimensional Objects": [[5, "two-dimensional-objects"]], "Type of problem": [[4, "type-of-problem"]], "Types of Machine Learning": [[23, "types-of-machine-learning"]], "Useful Python libraries": [[17, "useful-python-libraries"], [23, "useful-python-libraries"]], "Using Autograd": [[15, "using-autograd"]], "Using forward Euler to solve the ODE": [[4, "using-forward-euler-to-solve-the-ode"]], "Using gradient descent methods, limitations": [[15, "using-gradient-descent-methods-limitations"]], "Visualization": [[3, "visualization"], [3, "id1"]], "Visualizing the Tree, Classification": [[11, "visualizing-the-tree-classification"]], "Week 34: Introduction to the course, Logistics and Practicalities": [[23, null]], "What Is Generative Modeling?": [[23, "what-is-generative-modeling"]], "What is Machine Learning?": [[2, "what-is-machine-learning"]], "What is a good model?": [[2, "what-is-a-good-model"], [23, "what-is-a-good-model"]], "What is a good model? Can we define it?": [[23, "what-is-a-good-model-can-we-define-it"]], "Which activation function should I use?": [[3, "which-activation-function-should-i-use"]], "Why Linear Regression (aka Ordinary Least Squares and family)": [[23, "why-linear-regression-aka-ordinary-least-squares-and-family"]], "Wisconsin Cancer Data": [[9, "wisconsin-cancer-data"]], "Writing Our First Generative Adversarial Network": [[6, "writing-our-first-generative-adversarial-network"]], "Writing our own PCA code": [[13, "writing-our-own-pca-code"]], "XGBoost: Extreme Gradient Boosting": [[12, "xgboost-extreme-gradient-boosting"]], "scikit-learn implementation": [[3, "scikit-learn-implementation"]]}, "docnames": ["E1", "E2", "chapter1", "chapter10", "chapter11", "chapter12", "chapter13", "chapter2", "chapter3", "chapter4", "chapter5", "chapter6", "chapter7", "chapter8", "chapter9", "chapteroptimization", "clustering", "intro", "linalg", "schedule", "statistics", "teachers", "textbooks", "week34"], "envversion": {"sphinx": 62, "sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1}, "filenames": ["E1.ipynb", "E2.ipynb", "chapter1.ipynb", "chapter10.ipynb", "chapter11.ipynb", "chapter12.ipynb", "chapter13.ipynb", "chapter2.ipynb", "chapter3.ipynb", "chapter4.ipynb", "chapter5.ipynb", "chapter6.ipynb", "chapter7.ipynb", "chapter8.ipynb", "chapter9.ipynb", "chapteroptimization.ipynb", "clustering.ipynb", "intro.md", "linalg.ipynb", "schedule.md", "statistics.ipynb", "teachers.md", "textbooks.md", "week34.ipynb"], "indexentries": {}, "objects": {}, "objnames": {}, "objtypes": {}, "terms": {"": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 15, 17, 18, 20, 21, 23], "0": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23], "00": [2, 3, 7, 13, 23], "000": [3, 5], "00000000e": 23, "001": [4, 10, 15], "004": 7, "00727646693": [2, 23], "0086649156": [2, 23], "01": [2, 3, 4, 7, 11, 13, 15, 22, 23], "0110": 20, "01719003e": 23, "02": [2, 6, 9, 14, 23], "02334824": 23, "02857": 6, "02f": 8, "03077640549": 6, "03097597e": 23, "031": 7, "04": 13, "0458": 11, "05": [6, 8, 23], "062292565": 6, "062435": 23, "06730814": 23, "07": 23, "0713": [2, 23], "07285": 5, "08": 20, "08078025e": 23, "08336233266": 6, "0917": 11, "0n": [2, 23], "1": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23], "10": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 19, 20, 21, 23], "100": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 18, 20, 21, 23], "1000": [2, 3, 4, 6, 7, 10, 13, 15, 16, 17, 20, 23], "10000": [4, 7, 8, 12, 13, 15, 20], "100000": 10, "10001": 12, "1001": 20, "1002": 20, "1003": 20, "1005": 20, "1009": 20, "101": 1, "1011": 20, "1013": 20, "1013904243": 20, "1015": 20, "102": 1, "1023": 20, "1024": 5, "1026": 20, "1027": 20, "103": 3, "1030": 20, "1037": 20, "1038": 20, "1040": 20, "1047": 20, "107": 1, "108": 23, "10th": 11, "10x": [2, 23], "11": [1, 2, 4, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 22, 23], "110": 23, "1100": 20, "1101": 20, "111": [3, 9, 14], "112": 1, "11340253": 23, "11590451": 23, "116": 1, "117": 1, "118": 1, "12": [2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 18, 20, 22, 23], "120": 5, "121": [1, 10, 11, 12], "1215pm": [21, 23], "122": [10, 11, 12, 23], "124": [2, 23], "125": 1, "127": [1, 6], "128": [5, 6, 15], "129": 1, "1298": 11, "12pm": [21, 23], "13": [2, 4, 11, 14, 18, 20, 23], "131": 1, "133": 9, "135": 1, "136": 1, "14": [2, 4, 6, 8, 10, 11, 12, 14, 18, 20, 22], "141": 1, "143": 1, "1446729567": 6, "149": 1, "14g": 8, "15": [2, 4, 6, 8, 9, 10, 11, 14, 15, 20, 23], "150": [6, 10], "152": [1, 23], "153760": 23, "156": [1, 23], "157": 23, "158": 23, "159": [1, 23], "15g": 8, "15pm": 23, "16": [3, 4, 5, 6, 7, 10, 11, 12, 20, 23], "160": [1, 23], "1603": 5, "161": 1, "162": 1, "16231451": 6, "163": 1, "16384": 5, "164": 1, "167": 1, "17": [3, 4, 10, 20, 23], "172": 1, "173": 1, "176": 1, "178": 1, "179": 1, "1797": 3, "18": [4, 8, 9, 10, 11, 12, 20, 23], "1807": 6, "18392847": 23, "19": [4, 20, 23], "1940": 2, "1943": 14, "1970": [18, 23], "1973": 11, "1979": 8, "1_1": 14, "1_2": 14, "1_3": 14, "1cm": [2, 10, 12, 20, 23], "1d": [3, 4, 5], "1e": [4, 6, 15, 16], "1e10": 16, "1e4": 8, "1f": 3, "1k": 18, "1n": [2, 23], "1x": [2, 23], "2": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22], "20": [1, 2, 3, 4, 8, 9, 10, 20, 21, 23], "200": [2, 4, 5, 6, 10, 11, 12], "2000": 2, "2004": 15, "2006": 22, "20072279": 23, "2008": 23, "2010": 3, "2011": 3, "2014": 6, "2015": 3, "2016": [2, 23], "2018": [2, 8], "2021": [8, 16], "2022": 23, "2025": 23, "21": [2, 3, 7, 9, 11, 14, 18, 23], "2116753732": 6, "215pm": [21, 23], "2167072": 23, "22": [2, 3, 7, 14, 15, 18, 23], "221": 10, "225": 6, "22948497": 23, "23": [3, 14, 18, 23], "24": [2, 3, 18, 23], "25": [4, 5, 6, 7, 8, 10, 11, 13], "250": [4, 6, 9, 11], "25000": 2, "250154": 23, "253775": 23, "255": 5, "256": 6, "26": 23, "26303845": 23, "264": 23, "265": 23, "265109911": 6, "266": 23, "269": 23, "27": [2, 3], "270": 23, "27n_": 20, "28": [3, 5, 6], "2830637392": 6, "2861": 20, "2873": 11, "2882": 20, "2886": 20, "2890": [2, 23], "2892": 20, "29": 23, "2915": 20, "2931": 23, "29364655": 23, "294399745619595": 23, "296247": 23, "2968": 23, "2980": 23, "298273": 23, "298375": 23, "2990": 23, "2_": 14, "2_1": 14, "2_2": 14, "2_3": 14, "2_i": 14, "2_m": [8, 20], "2_t": 15, "2_x": 20, "2b": 20, "2cm": 10, "2d": [3, 5, 13, 14, 17, 23], "2e": 8, "2f": [2, 9, 11, 12, 13, 14, 23], "2g": 4, "2g_i": 4, "2k": 5, "2m": 8, "2n": [2, 4, 5, 23], "2nd": 11, "2p": 20, "2pt": 6, "2x": [2, 5, 10, 15, 23], "2x_ix_jy_iy_j": 10, "2x_j": 10, "2y_i": 12, "2y_j": 10, "3": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23], "30": [2, 3, 6, 8, 9, 12, 15, 21], "30000": [2, 23], "3072": 5, "31": [14, 18, 20], "315": 8, "3155": [2, 7, 8], "32": [5, 6, 8, 14, 15, 18, 20], "3200": 3, "3250": 3, "3297": 23, "33": [14, 18, 21], "3303": 23, "3310": 23, "332331": 23, "333": 9, "3331": 23, "3337": 23, "34": 18, "3436": [2, 23], "3437": [2, 23], "35": [2, 8, 23], "3581341341": 6, "359": 7, "36": [2, 7, 8, 20], "370782966": 6, "38": 20, "39": [2, 21, 23], "3d": [1, 4, 5, 6, 8, 15], "3f": [3, 5, 11], "3n": 18, "3x": [4, 10], "3x_i": 4, "3y": 10, "4": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "40": [3, 8, 21, 23], "400": 6, "4000": 23, "4050": [22, 23], "41": 18, "4155": [0, 4], "41589548": 23, "42": [3, 6, 10, 11, 12, 18, 23], "43": [2, 9, 18], "4310": 23, "436462435": 6, "44": [2, 18], "45": [21, 23], "46": [21, 23], "462": 9, "47": [21, 23], "479465113": 6, "47958494": 23, "48": 23, "48257387": [21, 23], "49": [7, 8, 13], "49152": 5, "4940954": [2, 23], "4990": 20, "4992": 20, "4997": 20, "4c4c7f": [11, 12], "4d": 5, "4f": 8, "4pm": [21, 23], "4y": 10, "4y_i": 12, "5": [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "50": [3, 4, 5, 6, 8, 9, 10, 12, 15, 23], "500": [3, 5, 6, 8, 11, 12, 15], "5018": 20, "506": 2, "507d50": [11, 12], "50j": 15, "50x10": 3, "51": [12, 23], "510": 3, "512132": 23, "5177783846": 6, "53": 11, "54": [8, 20, 23], "5411205": 23, "54894451": 23, "55": [3, 23], "56": 3, "56536": [2, 23], "569": 3, "57": [2, 10, 21, 23], "571": 7, "58": [12, 21, 23], "591317992": 6, "5cm": 20, "5f": 10, "5x": 10, "5y": 10, "6": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 21, 23], "60": [3, 5], "60000": 6, "6019067271": 6, "606439": 23, "625": 9, "63": [2, 3], "64": [3, 5, 6, 15, 18, 23], "64x50": 3, "65": [3, 10, 11], "6887363571": 6, "69": [1, 20], "69069n_": 20, "691": 23, "6n_": 20, "7": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 22, 23], "70": [3, 9], "70653767": 6, "71": 3, "724": 5, "73": 23, "7304881": 23, "75": [7, 8, 10, 13], "76": [21, 23], "765": 9, "77": [21, 23], "7718": 11, "7782028952": 6, "77893972": 23, "78": 23, "7d7d58": [11, 12], "8": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 21, 23], "80": [2, 3, 7, 10], "800": [6, 9], "81": 3, "815am": [21, 23], "85": 3, "8702784034": 6, "88": 23, "8f": 8, "8g": 8, "8n": 18, "8x8": 3, "9": [2, 3, 4, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 23], "90": 3, "9040": 11, "91": [21, 23], "92": [21, 23], "93": 1, "931": [2, 23], "933": 7, "937": 20, "938": 20, "939": [2, 20, 23], "94": 20, "95": [3, 13, 23], "954": 20, "955820c21e8b": 6, "96": 8, "960": 20, "961": 20, "962": 20, "9649652536": 6, "96611194e": 23, "9780387310732": 22, "9780387848570": 22, "9781098134174": 23, "9781492032632": 22, "9781801819312": 23, "98": [1, 2, 3], "985": 20, "986": 20, "989": 20, "9898ff": [11, 12], "99": [1, 15], "991": 20, "992": 20, "993": 20, "996": 7, "999": [11, 20], "9x": 8, "9y": 8, "A": [0, 1, 4, 5, 7, 8, 9, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22], "AND": 4, "And": [2, 5, 6, 7, 8, 11, 15, 17, 20], "As": [1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 15, 18, 20, 23], "At": [2, 6, 8, 15, 23], "BE": [2, 23], "Be": [4, 17, 23], "Being": 15, "But": [1, 2, 3, 4, 5, 7, 8, 11, 12, 20], "By": [2, 5, 7, 8, 14, 15, 18, 23], "For": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23], "IF": 8, "IN": 22, "If": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23], "In": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23], "Ising": [7, 14], "It": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "Its": [3, 4, 6, 13], "No": [8, 11, 23], "Not": [2, 3, 7, 8], "OR": 20, "Of": 20, "On": [2, 5, 20, 21, 22, 23], "One": [2, 3, 5, 6, 7, 8, 9, 10, 13, 14, 15, 20], "Or": [2, 3, 8, 23], "Such": [1, 2, 8, 14, 20], "That": [2, 7, 9, 12, 13, 14, 16, 20, 23], "The": [1, 6, 12, 15, 16, 18, 19, 20, 21, 22], "Then": [0, 1, 2, 3, 8, 10, 11, 12, 13, 14, 15, 16, 18, 23], "There": [0, 2, 5, 6, 7, 8, 10, 11, 13, 14, 16, 18, 20, 21, 23], "These": [2, 5, 6, 7, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23], "To": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20], "With": [1, 2, 7, 8, 10, 11, 12, 13, 14, 16, 18, 20, 23], "_": [1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 23], "_0": [7, 10, 12, 13, 15], "_1": [4, 7, 8, 10, 12, 13, 14, 15, 16, 18], "_2": [4, 7, 10, 13, 14, 15, 18], "_3": 18, "_4": 18, "_9": 15, "__class__": 12, "__doc__": 8, "__future__": [10, 11], "__init__": 3, "__main__": 4, "__name__": [4, 12], "_auto1": [4, 5, 6, 7, 8, 9, 14, 15, 18, 20], "_auto10": [8, 14], "_auto11": 8, "_auto12": 8, "_auto2": [4, 5, 6, 7, 8, 14, 15, 18, 20], "_auto3": [5, 6, 7, 8, 14, 15, 18], "_auto4": [6, 8, 14, 15, 18], "_auto5": [6, 8, 14, 15, 18], "_auto6": [6, 8, 14, 18], "_auto7": [6, 8, 14, 18], "_auto8": [8, 14], "_auto9": [8, 14], "_build": [2, 17, 22, 23], "_c": 3, "_compon": 13, "_depth": 11, "_export": [0, 1], "_fraction": 11, "_i": [2, 3, 4, 7, 8, 9, 10, 13, 14, 15, 23], "_j": [2, 3, 4, 5, 7, 8, 10, 15], "_k": 15, "_l": 14, "_lambda": 8, "_leaf": 11, "_m": 12, "_multilayer_perceptron": 23, "_n": [4, 7, 10, 13, 15], "_node": 11, "_p": [7, 10], "_ratio": 13, "_sampl": 11, "_split": [8, 11], "_t": 15, "_test": 8, "_varianc": 13, "_weight": 11, "a0": 5, "a0faa0": [11, 12], "a1": [2, 23], "a2": [2, 23], "a3": [2, 23], "a4": [2, 23], "a_": [1, 2, 3, 18, 23], "a_0": [2, 23], "a_1a": [2, 23], "a_2a": [2, 23], "a_3": [2, 23], "a_3a": [2, 23], "a_4": [2, 23], "a_4a": [2, 23], "a_h": 3, "a_i": [2, 3, 4, 14, 23], "a_j": [3, 14], "a_k": [2, 3, 14], "aaron": 22, "ab": [2, 4, 7, 15, 16, 23], "ab_channel": 17, "abandon": 3, "abid": 20, "abil": [2, 12], "abl": [1, 2, 3, 6, 7, 8, 9, 12, 14, 15], "about": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 21], "abov": [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 22, 23], "abovement": [8, 23], "abscissa": 15, "absolut": [2, 4, 7, 8, 15, 23], "abstract": 3, "acceler": 15, "accept": [2, 5, 8, 11], "access": [2, 5, 13, 20, 23], "accid": [6, 8], "accompani": [2, 23], "accomplish": [10, 11, 15], "accord": [2, 3, 4, 7, 8, 11, 14, 15, 16, 20, 23], "accordingli": 13, "account": [0, 1, 2, 5, 7, 15, 20, 23], "accumul": [14, 15, 20], "accur": [2, 5, 6, 8, 12, 15], "accuraci": [2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 23], "accuracy_scor": [2, 3, 12, 23], "accuracy_score_numpi": 3, "achiev": [2, 3, 7, 8, 10, 14, 18, 23], "aco": 20, "acquaint": 17, "acquir": [3, 17, 23], "acr": 2, "across": [3, 5, 8, 11, 17, 23], "act": [3, 5, 18], "action": 20, "activ": [0, 2, 4, 5, 6, 11, 19, 21, 23], "actual": [0, 1, 2, 3, 6, 7, 8, 10, 13, 18, 20, 23], "ad": [0, 1, 3, 5, 6, 7, 10, 15, 18], "ada_clf": 12, "adaboostclassifi": 12, "adadelta": 15, "adam": [3, 5, 6, 23], "adapt": [6, 8, 15, 22], "add": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 20, 21, 23], "add_subplot": [3, 9, 14, 16], "addendum": 7, "addit": [0, 2, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18, 20, 21, 22, 23], "addition": [14, 15], "address": [3, 11, 13, 15, 23], "adjac": [5, 14], "adjoint": 7, "adjust": [2, 7, 14, 15], "admir": [2, 23], "advanc": [6, 8, 14, 22, 23], "advantag": [3, 5, 7, 8, 12, 15, 18], "adversari": 23, "afecionado": 23, "affect": [0, 5], "affin": [2, 5, 10, 13], "afford": 5, "aficionado": 23, "aforement": 16, "african": 2, "after": [0, 1, 2, 3, 4, 6, 7, 8, 11, 13, 14, 15, 17, 18, 20, 23], "afterward": [2, 23], "ag": [2, 9, 23], "ag_0": 4, "again": [2, 3, 6, 7, 8, 9, 10, 12, 13, 14, 15, 20, 23], "against": [3, 6, 9, 12], "agegroup": 9, "agegroupmean": 9, "aggreg": [11, 12], "agorithm": 12, "agre": [7, 8, 20], "agreement": 15, "ahead": 11, "ai": [2, 22], "aid": 13, "aim": [1, 2, 3, 6, 8, 9, 13, 16, 17, 18], "ainv": 7, "airplan": 5, "aka": 7, "al": [1, 2, 4, 6, 22, 23], "alarm": [7, 9], "algebra": [2, 5, 7, 15, 17], "algorithm": [1, 2, 3, 4, 6, 7, 8, 9, 10, 15, 16, 17, 18, 20, 22], "align": [2, 4, 7, 8, 9, 10, 15, 20, 23], "all": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23], "allevi": [3, 15], "alloc": [5, 18], "allow": [2, 3, 4, 5, 7, 8, 10, 12, 15, 17, 18, 23], "almost": [2, 3, 8, 10, 13, 15, 20], "alon": [4, 11], "along": [0, 4, 5, 6, 7, 8, 11, 12, 13, 17, 18, 23], "alpha": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 15, 16, 20, 23], "alpha_": 12, "alpha_0": 5, "alpha_1": 5, "alpha_2": 5, "alpha_i": [5, 15], "alpha_k": 15, "alpha_m": 12, "alpha_n": 5, "alpha_opt": 15, "alreadi": [0, 4, 5, 6, 7, 8, 12, 14, 17, 18, 20, 23], "also": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "alter": 3, "altern": [2, 3, 6, 7, 8, 10, 11, 13, 15, 18, 23], "although": [1, 2, 3, 7, 8, 10, 12, 15, 23], "alwai": [1, 2, 5, 7, 8, 14, 15, 20, 23], "am": 6, "ame2016": [2, 23], "american": 2, "among": [2, 5, 7, 11, 12, 14, 18, 23], "amongst": 7, "amount": [2, 3, 5, 6, 8, 10, 12, 16, 17], "an": [1, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 20, 21, 22], "an_": 20, "anaconda": [2, 3, 17, 23], "analogi": 15, "analys": 8, "analysi": [3, 5, 6, 9, 16, 18, 22], "analyt": [4, 5, 7, 8, 9, 14, 15, 17, 23], "analyz": [1, 2, 3, 5, 6, 7, 8, 20], "andrew": 3, "angl": [2, 5, 11], "anharmon": 5, "ani": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 20, 23], "anim": [6, 14], "ann": 14, "annot": [2, 3, 5, 9, 10, 23], "announc": 23, "anoth": [0, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 23], "ansatz": [2, 23], "answer": [2, 3, 5, 7, 8, 18, 21, 23], "antialias": [4, 8], "anticip": 6, "anymor": [3, 10], "anyon": [0, 6, 10], "anyth": [0, 1, 3, 20], "anytim": [21, 23], "apach": 3, "apart": [13, 15], "api": [3, 17, 23], "appar": 4, "appear": [2, 3, 5, 15, 18, 20], "append": [3, 5, 6, 10, 11, 15, 23], "appli": [2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 20, 22, 23], "applic": [1, 2, 3, 5, 6, 7, 8, 9, 11, 14, 15, 18, 20, 22, 23], "apply_gradi": 6, "approach": [0, 1, 3, 4, 6, 7, 8, 11, 12, 13, 14, 15, 17, 20, 22], "appropri": [4, 8, 11, 14, 15, 17, 20], "approv": 23, "approx": [2, 4, 5, 8, 12, 13, 15, 20, 23], "approxim": [2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 15, 20, 23], "apt": [2, 17, 23], "aq": 20, "ar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23], "aragorn": 23, "arang": [3, 5, 6, 8, 9, 11, 12, 14, 15, 23], "arbitrari": [3, 6, 8, 10, 14, 15, 20], "arbitrarili": [2, 3, 13, 23], "arc": 8, "architectur": [5, 6, 14], "area": [2, 5, 8, 22, 23], "argmax": [3, 13], "argmin": [6, 12, 16], "argsort": 13, "argu": [3, 15], "argument": [2, 4, 5, 7, 13, 14, 15, 23], "aris": [2, 8, 14, 15, 20, 23], "arithmet": [2, 15, 18, 23], "arm": 8, "armadillo": 18, "around": [2, 3, 6, 7, 8, 13, 20, 23], "arrai": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 15, 16, 17, 20], "arrang": [5, 23], "arraybox": 15, "arriv": [2, 8, 11, 13, 18, 20, 23], "arrow": 14, "arrowprop": 10, "art": [2, 3, 17], "articl": [2, 5, 6, 8, 12, 23], "artifici": [2, 4, 9, 14, 22, 23], "artificialneuron": 14, "arug": 15, "arxiv": [5, 6], "asarrai": [2, 8, 11], "ask": [0, 7, 8, 13, 14], "aspect": [2, 8, 17, 23], "assembl": 5, "assembli": [2, 23], "assert": 6, "assess": [2, 8, 23], "assici": 6, "assign": [0, 2, 9, 10, 11, 14, 15, 16, 19, 21, 22, 23], "associ": [2, 8, 11, 14, 16, 20, 23], "assum": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "assumpt": [2, 5, 7, 8, 11, 13, 20, 23], "ast": [2, 7, 8, 23], "astyp": [6, 11, 12], "asymmetri": [2, 23], "asymptot": [6, 8], "atom": [2, 23], "attempt": [2, 6, 8, 9, 10, 12, 23], "attend": 23, "attent": [2, 18, 23], "attract": [2, 12, 23], "attribut": [2, 11, 23], "audi": [2, 23], "audio": [5, 6], "august": 23, "aurelien": [2, 22, 23], "austfjel": 8, "auth": 0, "authent": 0, "author": [2, 3, 12, 20], "authour": 23, "auto": [11, 12, 20], "autocor": 20, "autocorrelation_tim": 20, "autocorrelform": 20, "autocovari": 20, "autoencod": [6, 17, 23], "autoencond": 17, "autograd": [17, 23], "autom": [2, 17, 22, 23], "automac": 18, "automag": 23, "automat": [1, 2, 3, 4, 5, 6, 13, 17, 18, 23], "automobil": 5, "autonom": 6, "avail": [2, 3, 6, 8, 12, 13, 17, 18, 19, 21, 22, 23], "averag": [2, 3, 5, 8, 11, 12, 15, 16, 20, 21, 23], "avoid": [2, 6, 7, 8, 11, 13, 15, 18], "awai": [4, 5, 8], "awar": [4, 12], "award": [21, 23], "ax": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 23], "axes3d": [4, 8, 15], "axes_grid1": 8, "axhlin": 10, "axi": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "axiom": 7, "axlabel": 2, "axvlin": [6, 10], "axvspan": 6, "b": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 15, 16, 20, 21, 23], "b1": 10, "b2": 10, "b3": 10, "b_": [2, 3, 18], "b_0": 2, "b_1": [2, 4, 14, 15], "b_2": [2, 15], "b_5": 15, "b_group": 11, "b_i": [2, 3, 4, 14, 23], "b_ia_": [2, 23], "b_ia_i": 2, "b_index": 11, "b_j": [3, 14], "b_k": [2, 3, 14, 15], "b_m": 14, "b_score": 11, "b_valu": 11, "babcock": 23, "bachelor": [19, 21], "back": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 18, 20, 23], "backbon": 18, "backend": [3, 6], "background": [22, 23], "backpropag": 3, "backtrack": 11, "backup": 18, "backward": [3, 4, 6, 14, 18], "bad": 8, "badli": 20, "bag": [11, 17, 23], "bag_clf": 12, "baggin": 23, "baggingboot": 12, "baggingclassifi": 12, "baggingtre": 12, "balanc": 8, "band": 18, "bandwidth": 18, "bar": [2, 8, 13, 23], "barber": 22, "bare": [6, 12], "base": [1, 2, 3, 5, 6, 7, 9, 10, 11, 12, 16, 17, 20, 21, 22, 23], "basi": [7, 9, 10, 12, 13, 14, 15, 18], "basic": [0, 8, 10, 14, 15, 16, 17, 20, 23], "batch": [5, 6, 13, 14, 15], "batch_shap": 6, "batch_siz": [3, 5, 6], "batchnorm": 6, "bay": 9, "bayesian": [7, 17, 22, 23], "becaus": [2, 3, 4, 5, 6, 7, 8, 10, 11, 14, 15, 16, 23], "becom": [2, 3, 4, 7, 8, 9, 11, 14, 15, 20, 23], "been": [2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 17, 18, 23], "befor": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 18, 20, 23], "beforehand": [2, 20, 23], "begin": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 20, 21, 23], "behav": [3, 8, 15], "behavior": [2, 3, 15, 23], "behaviour": 14, "behind": [2, 3, 8, 10, 15, 23], "being": [2, 3, 4, 5, 6, 7, 9, 10, 12, 13, 14, 15, 20, 23], "believ": [11, 18], "belong": [9, 10, 11, 15, 16], "below": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "benchmark": 12, "benefici": [3, 15], "benefit": [2, 3, 6, 13, 15, 17, 23], "bengio": [3, 22, 23], "benign": [3, 9], "besid": [6, 7], "bessel": 7, "best": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 21, 23], "beta": [1, 2, 3, 5, 7, 8, 9, 12, 13, 15, 23], "beta_": [2, 5, 8, 9, 15], "beta_0": [1, 2, 3, 5, 7, 8, 9, 15], "beta_0x_": 2, "beta_1": [2, 3, 5, 7, 8, 9, 12, 15], "beta_1x_": 2, "beta_1x_0": 2, "beta_1x_1": [2, 9], "beta_1x_2": 2, "beta_1x_i": [9, 15], "beta_2": [2, 5, 15], "beta_2x_": 2, "beta_2x_0": 2, "beta_2x_1": 2, "beta_2x_2": [2, 9], "beta_3": 5, "beta_i": [2, 5, 7], "beta_j": [2, 7, 8, 15], "beta_k": 15, "beta_linreg": 15, "beta_m": 12, "beta_mg_m": 12, "beta_n": 5, "beta_p": 9, "beta_px_p": 9, "betavalu": 7, "better": [2, 3, 4, 5, 6, 8, 11, 12, 13, 14, 15, 23], "between": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 20, 23], "beyond": [2, 3, 7, 8, 10, 15, 23], "bf": [15, 16, 18, 20], "bg": 23, "bgd": 15, "bia": [2, 3, 4, 5, 7, 10, 11, 12, 14, 15, 23], "bias": [3, 4, 5, 7, 8, 11, 14], "big": [2, 3, 4, 7, 8, 16], "bigger": [3, 8], "bigr": 14, "bike": 11, "bilbo": 23, "billion": [5, 14, 17], "bin": [2, 9, 20], "binari": [2, 5, 7, 9, 11, 12, 14, 23], "binarycrossentropi": 6, "bind": 2, "binomi": [17, 20, 23], "binsboot": 8, "bioinformat": 2, "biolog": [3, 14], "bios1100": [17, 23], "bird": [2, 5], "birth": 23, "bishop": [22, 23], "bit": [3, 6, 18, 20, 23], "bitwis": 20, "bivari": 4, "bk": [2, 15], "bla": [18, 23], "black": [10, 11, 16], "block": [8, 12, 17, 18, 20, 23], "blog": 23, "blogpost": 6, "blue": [2, 5], "bmatrix": [2, 3, 5, 7, 9, 10, 13, 15, 18, 23], "bmi": 3, "bodi": [2, 3, 6, 14], "bold": 3, "boldfac": [1, 2, 7], "boldsymbol": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 15, 16, 23], "boltzmann": [14, 17, 23], "book": [22, 23], "book1": 22, "boolean": 6, "boost": [3, 11, 17, 23], "boostrap": 12, "bootstrap": [3, 15, 17, 23], "borrow": 23, "boston_dataset": 2, "bot": 10, "both": [0, 1, 2, 3, 6, 7, 8, 10, 11, 12, 15, 16, 17, 18, 20, 21, 23], "bottl": 9, "bound": [2, 10, 14], "boundari": [4, 6, 10, 13, 14], "box": [6, 11], "boyd": [10, 15], "bracket": [6, 20], "brain": [3, 9, 14], "branch": [11, 23], "break": [2, 6, 8, 13, 16, 23], "breast": [7, 9, 13], "breviti": 15, "brew": [2, 17, 23], "brg": 10, "briefli": [1, 2, 23], "bring": [2, 7, 8, 12], "britt": [21, 23], "broad": 2, "broadli": 23, "brought": [15, 17, 23], "brownle": 6, "browser": [0, 23], "brute": [5, 7, 13], "buffer_s": 6, "bui": 6, "build": [1, 2, 6, 7, 8, 12, 18, 20, 23], "built": [2, 3, 5, 6, 8], "bunch": 13, "busi": 2, "byte": [18, 23], "c": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22], "c1": [10, 13], "c2": [10, 13], "c_": [2, 10, 11, 12, 15, 20], "c_0": 20, "c_1": 14, "c_2": 14, "c_3": 14, "c_4": 14, "c_i": [14, 15], "c_k": 20, "ca": [3, 23], "cach": 12, "cal": [2, 10, 12, 14, 15], "calcul": [1, 2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "call": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "calor": 2, "cambridg": [15, 22], "can": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22], "cancel": [2, 15, 23], "cancer": [7, 12], "cancerpd": 9, "candid": [10, 11, 12], "cannot": [2, 3, 6, 7, 8, 9, 10, 11, 20], "canopi": [2, 17, 23], "canva": [0, 1, 23], "cap": 7, "capabl": [2, 3, 10, 15, 17, 23], "capac": [4, 21], "capita": 2, "captur": [6, 13, 14, 23], "car": [5, 6], "card": [2, 9, 23], "cardin": 3, "care": [0, 13], "carefulli": 15, "carlo": [2, 8, 17, 20, 22, 23], "carri": [4, 8, 9], "cart": 12, "case": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15, 16, 17, 18, 23], "casella": 22, "cast": 3, "cat": [5, 6], "catch": 2, "categor": [2, 3, 5, 11, 13, 23], "categori": [2, 3, 5, 9, 12, 14, 16, 23], "categorical_crossentropi": [3, 5], "caus": [2, 7, 8, 20, 23], "causal": 2, "causat": [2, 23], "cax": 3, "cb": [8, 23], "cbar": 3, "cc": [2, 3, 7, 15, 23], "ccc": [7, 14], "cdf": 20, "cdot": [2, 4, 8, 14, 15, 16, 18, 20, 23], "celebr": 15, "cell": 6, "center": [2, 3, 8, 9, 10, 11, 13, 16, 20, 23], "central": [1, 2, 5, 7, 8, 10, 18, 23], "centroid": [16, 20], "centroid_differ": 16, "centuri": 5, "certain": [2, 5, 8, 9, 11, 20, 23], "cg": 15, "cha": 2, "chain": [2, 3, 15, 17, 20, 23], "challeng": 0, "chanc": [3, 7, 15, 20], "chang": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 15, 16, 18, 20, 23], "channel": 5, "chapter": [1, 2, 8, 12, 13, 18, 22, 23], "chapter3": 2, "charact": [2, 5, 7, 23], "character": [10, 11, 12, 14, 20], "characterist": [2, 3, 5, 12, 15, 23], "charg": [2, 23], "charl": 2, "chase": 6, "chatgpt": 0, "chd": 9, "chddata": 9, "cheap": 7, "cheaper": [3, 15], "check": [0, 1, 2, 3, 5, 6, 7, 13, 15, 18, 23], "checkmark": 5, "checkpoint": 6, "checkpoint_dir": 6, "checkpoint_prefix": 6, "chen": 12, "chiaramont": 4, "childcar": 1, "children": 1, "choic": [2, 3, 4, 5, 6, 8, 11, 14, 15, 16, 18, 23], "choleski": [7, 18], "choos": [0, 4, 5, 8, 11, 12, 13, 15, 16], "chosen": [1, 2, 3, 4, 8, 10, 11, 12, 15, 20, 23], "chosen_datapoint": 3, "christian": 22, "christoph": [22, 23], "cifar": 5, "cifar10": 5, "circ": [3, 14], "circl": [2, 10, 14], "circuit": 5, "circumfer": 11, "circumv": [3, 7, 15], "ckpt": 6, "clariti": 20, "class": [2, 3, 5, 6, 8, 9, 10, 11, 13, 14, 15, 20, 23], "class_nam": [5, 11], "class_val": 11, "class_valu": 11, "classic": [9, 11, 15], "classif": [2, 5, 7, 8, 9, 10, 13, 14, 17, 22, 23], "classifi": [2, 3, 6, 9, 11, 12, 13, 23], "classificaton": 3, "classifii": 12, "clean": 3, "clear": [3, 7, 12, 14, 15], "clearli": [2, 5, 7, 8, 9, 10, 20], "clever": [3, 12], "clf": [2, 8, 10, 11, 12, 23], "clf3": 2, "clf_lasso": 8, "clf_ridg": 8, "cli": 0, "clip": [5, 20], "clone": [0, 21], "close": [2, 3, 4, 6, 8, 10, 11, 13, 14, 15, 16, 20, 22, 23], "closer": [5, 7, 15], "closest": [10, 13, 15, 16], "closur": [17, 23], "cloud": [17, 23], "cluster": [2, 3, 6, 8, 13, 17, 23], "cluster_label": 16, "cm": [3, 4, 5, 8, 10, 15], "cmap": [2, 3, 4, 5, 6, 8, 10, 11, 12, 23], "cmap_arg": 8, "cmd": [0, 11], "cn_": 20, "cnn": 14, "cnn_kera": 5, "cntk": [17, 23], "co": [2, 4, 5, 8, 11, 15, 23], "code": [5, 6, 8, 9, 10, 17, 18, 20, 22], "coef": [2, 23], "coef0": 10, "coef_": [1, 2, 7, 8, 10, 11, 15, 23], "coeff": 7, "coeffici": [2, 5, 7, 8, 9, 10, 11, 15, 18, 23], "coerc": [2, 8, 23], "coin": [12, 20], "coin_toss": 12, "col": [2, 13, 23], "colab": [17, 23], "cold": 11, "colinear": 2, "collaps": 10, "collect": [2, 4, 8, 12, 13, 17, 20, 22, 23], "collinear": 7, "color": [2, 5, 6, 8, 10, 11, 12, 20], "color_channel": 5, "color_cod": 8, "colorbar": [3, 8], "colsample_bytre": 12, "colsaobject": 12, "column": [1, 2, 3, 4, 7, 8, 9, 10, 11, 13, 14, 18, 23], "columntransform": 11, "com": [0, 1, 6, 8, 17, 22, 23], "combin": [3, 4, 7, 8, 9, 12, 20], "come": [0, 2, 3, 5, 6, 7, 14, 15, 16, 23], "command": [0, 2, 3], "comment": [2, 6, 7, 8], "commerci": [2, 17, 23], "commit": 0, "commod": [2, 23], "common": [1, 2, 3, 5, 7, 8, 9, 11, 13, 15, 16, 20, 23], "commonli": [2, 3, 6, 8, 9, 11, 15, 16], "commun": [2, 14], "commut": 5, "commutatitav": 5, "compact": [2, 3, 5, 7, 8, 9, 11, 13, 14, 15, 16, 23], "compair": 2, "compar": [2, 5, 6, 7, 8, 13, 15, 18, 23], "comparison": [4, 6, 15], "compat": 9, "compet": 2, "competit": 12, "compil": [2, 3, 5, 6, 15, 17, 18, 23], "complet": [0, 1, 2, 4, 5, 6, 11, 14, 23], "completenn": 14, "complex": [1, 3, 7, 10, 11, 13, 14, 15, 23], "complic": [2, 3, 11, 15, 23], "compon": [1, 2, 3, 5, 6, 7, 8, 9, 11, 16, 17, 23], "components_": 13, "compos": [11, 14, 15, 16, 17, 23], "compphys": [1, 2, 8, 17, 19, 21, 22, 23], "compress": [2, 23], "compris": 8, "compromis": 7, "compulsori": [17, 23], "comput": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 18, 19, 20, 22, 23], "computation": [2, 5, 8, 11, 15, 20, 23], "computationalscienceuio": 23, "concaten": [4, 6, 8, 16], "concav": [3, 15], "concentr": [2, 12], "concept": [2, 4, 17, 23], "conceptu": [14, 15], "concern": [2, 3, 6, 9, 23], "concic": 23, "conclud": [2, 7, 15], "conclus": 3, "cond": 4, "conda": [2, 3, 17, 23], "condit": [2, 4, 6, 7, 8, 10, 11, 13, 15, 20, 23], "conduct": 17, "condwav": 4, "confid": [2, 7, 8, 9, 10, 23], "configur": 5, "confirm": [7, 14], "confus": [7, 8, 9, 12, 18], "confusion_matrix": 11, "congruenti": 20, "conjug": [6, 10], "conjugaci": 15, "conjunct": 5, "connect": [2, 3, 5, 6, 11, 13, 14, 15, 18, 23], "consequ": [7, 8, 10, 12, 14, 15], "conserv": [7, 16], "consid": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 23], "consider": [2, 3, 7, 15, 23], "consist": [2, 3, 4, 5, 6, 8, 14, 15, 20], "constant": [1, 2, 4, 6, 7, 8, 10, 14, 15, 20, 23], "constitu": [2, 23], "constitut": [4, 8], "constrain": [3, 5, 7, 9, 13], "constraint": [7, 8, 10, 15], "construct": [2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 18, 20, 23], "contact": [2, 23], "contain": [0, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 22, 23], "contemporari": 23, "content": [0, 3, 17, 18, 23], "context": [8, 12, 15], "contigu": 18, "continu": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 23], "contour": [11, 12, 15], "contourf": [10, 11, 12], "contrast": [3, 6, 11, 12, 14, 23], "contribut": [2, 5, 7, 15, 20, 23], "contributor": 2, "control": [0, 2, 3, 5, 11, 15, 17, 23], "conv": [5, 6], "conv2d": [5, 6], "conv2dtranspos": 6, "convei": 23, "conveni": [2, 7, 8, 14, 15, 18, 23], "convent": 14, "converg": [3, 4, 6, 7, 10, 15, 16, 23], "convergencewarn": 23, "convert": [2, 3, 6, 7, 11, 13, 15, 18, 23], "converttomatrix": 6, "convex": [6, 7, 9], "convinc": 15, "convolut": [3, 6, 17, 23], "cool": [6, 11], "coolwarm": 8, "coordin": [7, 14, 16], "coorel": 2, "copi": [0, 2, 3, 16], "core": 12, "corel": 23, "coronari": 9, "corr": [2, 7, 9, 13], "correalt": [13, 17], "correct": [0, 2, 3, 4, 5, 6, 7, 9, 15, 18, 20, 23], "correctli": [3, 4, 8, 9, 12], "correl": [2, 3, 5, 7, 8, 9, 12, 14, 15, 17, 20, 23], "correlation_matrix": [2, 7, 9, 13], "correspond": [2, 5, 7, 8, 10, 11, 13, 14, 17, 18, 20, 23], "cortex": 14, "cosin": [5, 8], "cost": [1, 2, 4, 5, 7, 8, 9, 10, 11, 14, 15, 23], "cost_deep_grad": 4, "cost_funct": 4, "cost_function_deep": 4, "cost_function_deep_grad": 4, "cost_function_grad": 4, "cost_grad": 4, "cost_sum": 4, "costol": 15, "could": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "coulomb": [2, 23], "count": [0, 2, 11, 19, 20, 21, 23], "counterpart": 23, "countor": 15, "coupl": [6, 7, 8], "cours": [0, 1, 2, 3, 5, 7, 13, 21], "coursework": 0, "courvil": [22, 23], "cov": [7, 8, 13, 18, 20, 23], "cov_xi": [7, 13], "cov_xx": [7, 13], "cov_yi": [7, 13], "covari": [2, 9, 17, 18, 23], "covariance_matrix": [7, 13, 16], "cover": [2, 7, 17, 21, 22], "covert": [2, 23], "covxi": 20, "covxx": 20, "covxz": 20, "covyi": 20, "covyz": 20, "covzz": 20, "cpu": 3, "craft": 5, "creat": [0, 3, 5, 6, 7, 11, 12, 13, 14, 17, 23], "create_biases_and_weight": 3, "create_convolutional_neural_network_kera": 5, "create_neural_network_kera": 3, "create_x": [7, 13], "credit": [2, 9, 21, 23], "crim": 2, "crime": 2, "criteria": [2, 6, 11, 12, 16, 20, 23], "criterion": [11, 12, 15], "critic": 8, "cross": [0, 2, 3, 5, 9, 11, 12, 15, 17, 20, 23], "cross_entropi": 6, "cross_val_scor": 8, "cross_valid": [9, 12], "crossvalid": 8, "crucial": [3, 20], "cs231": 5, "csr_matrix": [18, 23], "csv": [2, 6, 8, 9, 11], "ctnk": 3, "cubic": 2, "cumbersom": 7, "cumsum": [12, 13, 23], "cumul": [9, 12, 20], "cumulative_heads_ratio": 12, "cup": 7, "current": [0, 1, 3, 4, 5, 6, 15, 16, 22], "curs": 2, "curv": [8, 9, 12, 14], "curvatur": 15, "custom": [8, 16], "custom_cmap": [11, 12], "custom_cmap2": [11, 12], "cutpoint": 11, "cv": [8, 9, 12], "cvxbook": 15, "cvxopt": [7, 10], "cycl": [3, 14], "d": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 23], "d2_g_t": 4, "d_f": 15, "d_g_t": 4, "d_net_out": 4, "da": 5, "dagger": [7, 18], "dai": [3, 11, 17], "damp": 5, "darget": 11, "darkr": 20, "dat": [2, 23], "dat_id": [2, 8, 9, 11, 23], "data": [1, 4, 6, 7, 10, 12, 14, 15, 16, 18, 22], "data1": 16, "data2": 16, "data3": 16, "data4": 16, "data_id": [2, 8, 9, 11, 23], "data_indic": 3, "data_panda": 23, "data_path": [2, 8, 9, 11, 23], "databas": 3, "datafil": [2, 8, 9, 11, 23], "datafram": [2, 6, 7, 9, 11, 13, 23], "datapoint": [1, 3, 7, 8, 9, 13, 15], "datasci": [0, 1], "dataset": [1, 2, 6, 8, 9, 10, 11, 12, 13, 15, 16, 23], "datatyp": 6, "date": [0, 23], "daughter": 12, "david": 22, "dbh": 3, "dbo": 3, "dcomposit": 18, "ddot": 4, "dead": 3, "deadlin": 0, "deal": [2, 3, 5, 7, 8, 10, 13, 15, 16, 18, 20, 23], "dealt": 2, "debt": 9, "debug": [2, 7, 8], "decad": [2, 5], "decai": [2, 15, 20, 23], "decemb": [21, 23], "decent": 12, "decid": [2, 4, 5, 7, 8, 11], "decim": [2, 23], "decis": [2, 3, 10, 13, 17, 22, 23], "decision_funct": 10, "decision_tre": 11, "decisiontreeclassifi": [11, 12], "decisiontreeregressor": [2, 11, 12], "declar": [2, 6, 18, 23], "decompos": [7, 8, 18], "decomposit": [2, 8, 14, 23], "decompost": 7, "deconvolut": 5, "decorrel": [12, 15], "decreas": [3, 4, 6, 7, 8, 12, 13, 15], "deduc": [2, 23], "deep": [5, 9, 14, 15, 17, 22], "deep_neural_network": 4, "deep_param": 4, "deep_tree_clf": [11, 12], "deep_tree_clf1": 11, "deep_tree_clf2": 11, "deepen": [7, 17, 23], "deeper": [2, 5, 6, 23], "deeplearningbook": [22, 23], "deer": 5, "def": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 20, 23], "def_covari": 20, "default": [2, 3, 4, 6, 8, 9, 18, 23], "default_tim": 6, "defect": 7, "defici": 7, "defin": [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20], "definit": [3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20], "defint": 20, "degre": [0, 1, 5, 7, 8, 10, 11, 12, 13, 20, 23], "del": 3, "delet": [0, 8], "delimit": 6, "deliv": [0, 19, 23], "delta": [2, 4, 5, 8, 10, 14, 15, 16, 23], "delta_": [3, 18], "delta_0": 5, "delta_1": 5, "delta_2": 5, "delta_3": 5, "delta_4": 5, "delta_5": 5, "delta_h": [2, 3, 23], "delta_j": [5, 14], "delta_k": 14, "delta_l": [3, 5], "delta_momentum": 15, "delta_n": [2, 5, 23], "delug": 17, "delv": 2, "demand": 15, "demonstr": [2, 5, 7, 8, 9, 13, 14, 17, 23], "den": 6, "denomin": [3, 7], "denot": [3, 4, 8, 9, 15, 20], "dens": [3, 5, 6], "densiti": [2, 4, 8, 20], "depart": [21, 23], "depend": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 20, 23], "depict": 20, "deploy": [2, 17, 23], "depth": [2, 5, 11, 12, 18], "deriv": [2, 3, 4, 8, 9, 10, 12, 13, 15, 17, 23], "derivati": 15, "derivative_fn": 15, "descend": [7, 11, 13], "descent": [2, 3, 5, 9, 10, 14, 23], "describ": [2, 4, 6, 7, 8, 10, 12, 13, 14, 15, 18, 23], "descript": [2, 10, 11, 23], "design": [2, 3, 5, 6, 7, 8, 9, 12, 13, 14, 15, 23], "designmatrix": [2, 23], "desir": [2, 4, 6, 7, 15, 16, 23], "desktop": 0, "despit": [3, 14], "destroi": 18, "det": [7, 18], "detail": [2, 8, 13, 15, 16, 18], "detect": [5, 10, 14], "determin": [2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 18, 20, 23], "determinist": [9, 15, 20], "dev": 3, "develop": [2, 5, 7, 10, 12, 13, 14, 17, 18, 23], "deviat": [2, 3, 4, 6, 7, 8, 20, 23], "devis": 14, "df": [6, 10, 13, 15, 23], "df1": 23, "di": 2, "diag": [7, 10], "diagnost": [3, 12], "diagon": [2, 7, 9, 15, 18, 20, 23], "diagonaliz": 7, "diagram": 12, "diagsvd": 8, "dice": [8, 20], "dict": [8, 10], "dictionari": 2, "did": [1, 2, 3, 7, 8, 9, 12, 13, 16, 23], "die": 3, "diff": 4, "diff1": 4, "diff2": 4, "diff_ag": 4, "diffeent": 10, "differ": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23], "differenti": [1, 2, 5, 17, 18, 23], "difficult": [2, 3, 8, 12, 15, 20, 23], "difficulti": [2, 3, 15, 23], "diffonedim": 4, "digit": [2, 3, 5, 6, 8, 21, 23], "dilemma": 15, "dilut": 3, "dim": [6, 13, 16, 18], "dimens": [1, 2, 3, 4, 5, 6, 7, 10, 13, 16, 18, 23], "dimension": [2, 6, 7, 8, 11, 13, 15, 16, 17, 18, 23], "dimensionless": [2, 5, 23], "diment": 18, "dimnsion": 6, "diod": 5, "direct": [2, 3, 4, 6, 13, 14, 15, 16, 23], "directli": [3, 6, 7, 8, 20], "disadvantag": [2, 23], "disappear": [5, 8], "disc_loss": 6, "disc_tap": 6, "discard": [8, 13], "disciplin": [2, 5, 14], "disclaim": 20, "discord": 23, "discourag": [0, 15], "discov": [2, 23], "discover": 7, "discret": [3, 5, 7, 9, 15], "discrimin": [6, 9, 12, 13], "discriminator_loss": 6, "discriminator_loss_list": 6, "discriminator_model": 6, "discriminator_optim": 6, "discuss": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23], "diseas": 9, "disguis": 8, "disord": [3, 9], "displai": [2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 20, 23], "displaystyl": [2, 7, 23], "disregard": [2, 23], "dissimilar": [13, 16], "dist": 16, "distanc": [2, 10, 11, 13, 16, 20], "distance_list": 11, "distinct": [5, 9, 10, 11, 12, 16], "distinctli": 10, "distinguish": [2, 6, 9, 10, 20, 23], "distplot": 2, "distribut": [2, 3, 6, 8, 9, 12, 13, 15, 16, 17, 18, 23], "distrubut": [2, 17, 23], "dive": [2, 10, 18, 23], "diverg": [3, 15], "divid": [2, 3, 5, 7, 8, 9, 10, 11, 13, 14, 20, 23], "divis": [8, 10, 11, 15, 18, 20], "dna": 9, "dnn": [2, 3, 4, 6, 14, 23], "dnn1": 6, "dnn2_gru2": 6, "dnn_kera": 3, "dnn_model": 3, "dnn_numpi": 3, "dnn_scikit": [2, 3, 23], "do": [0, 1, 2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 18, 23], "doc": [0, 1, 2, 17, 19, 21, 22, 23], "document": [0, 6, 15], "doe": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 18, 20, 23], "doesn": [5, 11, 14, 23], "dog": [3, 5, 6], "domain": [7, 10, 15], "domin": [2, 23], "don": [0, 1, 2, 3, 5, 7, 8, 10, 13, 15, 17, 23], "done": [1, 2, 4, 5, 6, 7, 8, 11, 12, 13, 15, 18, 23], "dot": [2, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "doubl": [1, 5, 6, 18, 23], "doubli": 3, "down": [2, 5, 8, 11, 13, 14, 15], "download": [0, 2, 3, 5, 7, 8, 18, 22, 23], "downsampl": 5, "dozen": 3, "dq": 8, "drag": 15, "dramat": 13, "drastic": 6, "draw": [6, 8, 12, 15], "drawback": [2, 3, 5, 15], "drawn": [3, 6, 8, 9, 13, 20, 23], "drive": [5, 6], "driven": 5, "drop": [2, 3, 7, 8, 13, 15, 20, 23], "dropna": [2, 8, 23], "dropout": 6, "dt": [4, 5, 15, 20], "dtype": [2, 3, 5, 6, 16, 18, 23], "dub": [2, 23], "due": [3, 4, 7, 8, 10, 12, 14, 15, 21, 23], "dummi": 2, "dure": [2, 3, 5, 6, 10, 11, 13, 17, 23], "dwell": 2, "dwh": 3, "dwo": 3, "dx": [4, 5, 10, 20], "dx_1": 20, "dx_1p": 8, "dx_2p": 8, "dx_mp": 8, "dx_n": 20, "dxp": 8, "dy": [3, 10, 20], "dynam": 6, "dz": 10, "e": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 21, 23], "e_": [2, 4, 23], "each": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23], "eapprox": [2, 23], "earli": [3, 15], "earlier": [2, 7, 9, 10, 11, 13, 14, 15, 23], "earthexplor": 8, "eas": [8, 11, 16], "easi": [0, 2, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 23], "easier": [0, 7, 8, 10, 11, 15, 20, 23], "easiest": 15, "easili": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 23], "eastern": [21, 23], "ebind": [2, 23], "eblock": 11, "econometr": 23, "economi": 7, "ecosystem": [17, 23], "ect": 19, "edg": 5, "edgecolor": 8, "edu": 15, "educ": [2, 23], "eff": 20, "effect": [1, 3, 6, 12, 15, 20], "effic": 3, "effici": [2, 5, 12, 15, 17, 18, 20, 23], "efron": 8, "egrad": 15, "eig": [7, 13, 15, 18, 20, 23], "eigen": 20, "eigenpair": [7, 13], "eigenvalu": [2, 7, 10, 13, 15, 18, 23], "eigenvector": [7, 13, 15], "eight": [18, 23], "eigval": [18, 20, 23], "eigvalu": [13, 15], "eigvec": [18, 20, 23], "eigvector": [13, 15], "eir": [21, 23], "eispack": [18, 23], "either": [3, 7, 8, 9, 10, 11, 12, 13, 15, 20, 23], "eivind": 21, "eivinsto": 21, "ekstr\u00f8m": 6, "elabor": 20, "elarn": 5, "electr": [2, 5, 14, 23], "electron": 23, "eleg": 13, "element": [3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 22], "elementari": [12, 15, 18], "elementwis": [5, 15], "elementwise_grad": [4, 15], "elessar": 23, "elif": 16, "elim": 18, "elimin": [5, 10], "elin": [21, 23], "els": [1, 3, 5, 6, 9, 11, 14, 15, 18], "elu": 3, "elus": [2, 23], "email": [19, 21, 23], "embed": [2, 13], "embodi": 8, "emit": 20, "emner": 22, "emphas": [2, 12, 17, 23], "emphasi": [2, 17, 22, 23], "empir": [3, 13, 20], "emploi": [2, 3, 7, 8, 13, 15, 20, 23], "employ": 2, "empti": [0, 8, 12], "emul": 14, "en": [17, 22], "enabl": 13, "enbodi": 8, "encod": [2, 5, 7, 11, 13, 16, 23], "encompass": [2, 20], "encount": [0, 2, 3, 7, 9, 15, 20, 23], "encourag": 0, "end": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23], "endpoint": [5, 8], "energi": [2, 6, 8], "enforc": 14, "eng": 22, "engin": [2, 3, 5, 6, 17, 23], "enorm": 5, "enough": [2, 8, 15, 23], "ensembl": [3, 11, 23], "ensur": [2, 3, 4, 5, 7, 8, 13, 15, 20], "entail": 23, "enter": [7, 8], "enthought": [2, 17, 23], "entir": [3, 5, 9, 11, 17, 20, 23], "entiti": [11, 14, 18, 23], "entri": [2, 7, 10, 13, 14, 18, 23], "entropi": [3, 5, 9, 12, 15, 23], "enumer": [2, 3, 4, 5, 6, 8, 10, 23], "env": [20, 23], "environ": [4, 17, 23], "environemnt": 0, "eo": [2, 8], "eol": 2, "eosfit": 2, "epoch": [2, 3, 5, 6, 14, 15, 23], "epsilon": [2, 7, 8, 9, 15, 23], "epsilon_": [2, 23], "epsilon_0": [2, 23], "epsilon_1": [2, 23], "epsilon_2": [2, 23], "epsilon_i": [2, 23], "eq": [5, 15, 16, 18, 20], "eqnarrai": [5, 7, 8], "equal": [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 15, 16, 18, 20, 23], "equat": [3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23], "equilibrium": [4, 14], "equiv": [5, 15, 18, 20], "equival": [2, 3, 7, 9, 10, 13, 15, 17, 18, 23], "erf": 20, "eriador": 23, "err": [2, 12], "err_": 8, "err_sqr": 4, "errat": 15, "erron": 4, "error": [0, 1, 3, 4, 6, 7, 8, 9, 11, 13, 14, 15, 17, 18, 20], "error_estimate_corr_tim": 20, "error_hidden": 3, "error_output": 3, "escap": 15, "especi": [0, 3, 5, 11, 14, 15], "essenti": [0, 2, 7, 8, 11, 12, 14, 16, 20], "establish": [1, 2, 8, 12, 13], "estim": [2, 3, 7, 8, 9, 12, 13, 15, 17, 20, 23], "estimated_mse_fold": 8, "estimated_mse_kfold": 8, "estimated_mse_sklearn": 8, "et": [1, 2, 4, 6, 22, 23], "eta": [2, 3, 5, 10, 14, 15, 23], "eta0": [10, 15], "eta_": 15, "eta_t": 15, "eta_v": [2, 3, 5, 23], "etc": [2, 3, 5, 7, 9, 10, 11, 13, 14, 15, 16, 17, 18, 20], "ethic": 17, "euclidean": [2, 16], "evalu": [0, 1, 2, 4, 5, 6, 7, 8, 11, 15, 20, 23], "evalut": 15, "even": [2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "evenli": 6, "event": [7, 9, 12, 20], "eventu": [2, 7, 8, 13, 14, 15, 21], "everi": [0, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 17, 20, 21, 23], "everyth": [1, 6, 14], "everywher": [6, 15], "evolv": 2, "exact": [2, 7, 13, 14, 15, 18, 20, 23], "exactli": [2, 5, 6, 8, 14, 17], "exam": 23, "examin": 8, "exampl": [0, 1, 7, 13, 14, 15, 17, 18, 20, 22], "exce": [3, 14, 15], "excel": [2, 3, 6, 7, 12, 23], "except": [5, 6, 8, 10, 11, 18], "excess": [2, 23], "excit": 2, "exclud": [3, 8, 14], "exclus": [2, 3, 5, 8, 20, 23], "execut": [0, 4, 7, 15], "exemplifi": 15, "exercic": [21, 23], "exercis": [7, 17, 19, 21, 23], "exhaust": 8, "exhibit": [2, 7, 8, 10, 23], "exist": [2, 3, 4, 5, 7, 8, 9, 10, 11, 15, 18, 23], "exit": [7, 18], "exp": [1, 2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 20], "exp_term": 3, "expand": [7, 9, 13, 15], "expans": [2, 5, 7, 10, 12, 14, 15, 23], "expect": [0, 2, 3, 7, 8, 9, 13, 14, 15, 17, 23], "expectation_value_of_h_wrt_p": 20, "expens": [1, 8, 12, 15], "experi": [0, 2, 3, 8, 10, 15, 17, 23], "experiment": [2, 6, 8, 11, 20, 23], "expert": [3, 11], "explain": [1, 2, 8, 11, 12, 13, 15, 23], "explained_variance_ratio_": 13, "explanatori": [2, 23], "explicit": [2, 5, 8, 15, 18, 23], "explicitli": [2, 6], "explod": 3, "exploit": [2, 5, 14, 15, 23], "explor": [3, 6, 8, 10, 15, 17, 23], "expon": 3, "exponenti": [2, 3, 7, 8, 12, 15, 20, 23], "export": [0, 1, 11], "export_graphviz": 11, "export_text": 11, "exporttext": 11, "expos": 17, "express": [2, 4, 5, 7, 8, 9, 12, 14, 15, 18, 20, 23], "exptmean": 20, "exptvari": 20, "extend": [2, 4, 9, 13, 15, 17, 23], "extens": [0, 2, 14, 17, 23], "extent": [2, 3, 8, 22], "extern": [5, 8, 11], "extra": [0, 3, 5, 7, 21, 23], "extract": [1, 2, 5, 7, 8, 9, 10, 13, 15, 18, 23], "extrapol": [2, 23], "extrem": [0, 1, 2, 3, 6, 7, 8, 9, 10, 11, 15, 18], "extremum": 15, "extrins": 13, "ey": [2, 7, 8, 15, 16, 18, 23], "f": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 15, 16, 18, 20, 21, 23], "f1": 15, "f11": [2, 23], "f12": [2, 23], "f13": [2, 23], "f1_grad": 15, "f1d": 15, "f2": 15, "f2_grad_x1": 15, "f2_grad_x1_analyt": 15, "f2_grad_x2": 15, "f2_grad_x2_analyt": 15, "f3": 15, "f3_grad": 15, "f3_grad_analyt": 15, "f4": 15, "f4_grad": 15, "f4_grad_analyt": 15, "f5": 15, "f5_grad": 15, "f6": 15, "f6_for": 15, "f6_for_grad": 15, "f6_grad_analyt": 15, "f6_while": 15, "f6_while_grad": 15, "f7": 15, "f7_grad": 15, "f7_grad_analyt": 15, "f8": 15, "f8_grad": 15, "f9": [2, 15, 23], "f9_altern": 15, "f9_alternative_grad": 15, "f9_grad": 15, "f_": 12, "f_0": [5, 12], "f_1": [12, 15], "f_2": [14, 15], "f_3": 14, "f_d": 20, "f_grad": 15, "f_grad_analyt": 15, "f_i": [1, 2, 8, 14], "f_m": [5, 12], "f_n": 5, "f_vec": 4, "face": [15, 23], "facecolor": [8, 10, 20], "facil": [2, 17], "facilit": 14, "fact": [2, 3, 5, 7, 11, 13, 14, 15, 23], "factor": [2, 3, 5, 7, 8, 11, 12, 13, 15, 18, 20, 23], "factori": 15, "fade": 8, "fafab0": [11, 12], "fail": [2, 8, 15, 21, 23], "failur": 9, "fairli": [3, 4, 20], "faisal": 1, "fake": 6, "fake_loss": 6, "fake_output": 6, "fall": [10, 11, 19], "fals": [1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 16, 18, 23], "famili": [2, 9, 10, 20], "familiar": [0, 2, 5, 7, 8, 10, 17, 18, 20, 23], "famou": [8, 14], "far": [1, 2, 5, 6, 7, 8, 10, 13, 14, 15, 16, 23], "fashion": [2, 11, 12, 23], "fast": [3, 5, 8, 12, 14, 15, 17, 20, 23], "faster": [3, 13, 15], "fastest": [15, 18], "favor": 9, "favorit": 20, "fc": 5, "featur": [0, 2, 3, 5, 7, 8, 9, 10, 12, 13, 14, 15, 17, 20, 23], "feature_nam": [2, 3, 9, 11], "feautur": 11, "fed": 3, "feed": [2, 4, 5, 13, 17, 23], "feed_forward": 3, "feed_forward_out": 3, "feed_forward_train": 3, "feedback": [6, 23], "feeddorward": 6, "feedforward": [3, 6, 14], "feel": [0, 1, 2, 7, 8, 13, 15, 17, 21, 23], "feet": 2, "fetch": [0, 8], "few": [3, 5, 6, 7, 11, 20, 23], "fewer": [2, 11, 13, 23], "ffnn": [3, 14], "field": [2, 5, 8, 14, 17], "fifth": [2, 8, 23], "fig": [2, 3, 4, 5, 6, 8, 9, 14, 15, 16, 23], "fig_id": [2, 8, 9, 11, 23], "figaxi": 20, "figsiz": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 23], "figur": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 23], "figure_id": [2, 8, 9, 11, 23], "figurefil": [2, 8, 9, 11, 23], "file": [0, 2, 6, 7, 8, 9, 11, 23], "file_prefix": 6, "filenam": 23, "fill": [7, 11], "filter": [5, 6], "final": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 19, 20, 21, 23], "financ": 2, "find": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23], "fine": [2, 16], "finish": 4, "finit": [5, 7, 8, 14, 15, 20], "finnicki": 0, "first": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 22], "firsteigvector": 13, "fit": [3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 20], "fit_beta": 8, "fit_intercept": [1, 2, 7, 8], "fit_mod": 11, "fit_transform": [0, 2, 8, 10, 11, 13], "fiti": [2, 23], "five": [2, 11, 23], "fix": [2, 5, 6, 8, 12, 13, 14, 15, 23], "flag": 6, "flat": [14, 15], "flatten": [3, 5, 6, 7, 18], "flexibl": [3, 8, 10, 12, 14, 23], "flip": [21, 23], "float": [2, 5, 6, 7, 11, 13, 15, 16, 18, 23], "float32": [6, 11], "float64": [6, 18, 23], "flop": [7, 18], "flow": [3, 6, 14], "fluctuat": 7, "fly": 13, "fm": 2, "fmax": 5, "fmesh": 15, "fn": 9, "focu": [0, 2, 5, 6, 7, 8, 17, 22, 23], "focus": [3, 8, 9, 18], "fold": [8, 11], "folder": [0, 2, 6, 8, 23], "follow": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 23], "font": [2, 9, 20, 23], "fontdict": 20, "fontsiz": [3, 8, 10, 11, 12, 20], "fontweight": 3, "footprint": 5, "foral": 10, "forc": [2, 7, 8, 12, 13], "forcast": 6, "forecast": [6, 14], "forest": [2, 3, 11, 17, 23], "forget": 13, "form": [0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23], "formal": [5, 6, 16, 20], "format": [2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 17, 20, 22], "format_data": 6, "formatstrformatt": [8, 15], "formul": [6, 8, 13, 16], "formula": [5, 15, 20], "forth": [6, 14], "fortran": [2, 17, 18, 23], "fortran2003": [17, 23], "fortran90": 20, "fortun": [2, 13], "forward": [2, 5, 8, 17, 18, 23], "found": [3, 4, 6, 7, 8, 14, 15, 23], "foundat": [17, 23], "four": [6, 7, 8, 10, 14, 18, 19, 21, 23], "fourier": [2, 23], "fourierdef1": 5, "fourierdef2": 5, "fourierseriessign": 5, "fourth": [14, 23], "fp": 9, "frac": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "fraction": 11, "frame": 9, "framework": [3, 10, 12, 20], "frank": [7, 13], "frankefunct": [7, 8, 13], "fredli": [21, 23], "free": [0, 1, 2, 8, 13, 15, 17, 18, 20, 21, 22, 23], "freecodecamp": 17, "freedom": 7, "freeli": 2, "freez": 0, "frequenc": [5, 8, 9, 20], "frequent": [2, 10, 11, 15], "frequentist": 17, "fresh": 12, "fridai": [0, 21, 23], "friedman": [8, 22, 23], "friendli": 6, "frodo": 23, "frog": 5, "from": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 15, 16, 17, 18, 20, 21, 22], "from_cod": 11, "from_logit": [5, 6], "from_tensor_slic": 6, "front": [2, 6, 7, 23], "frustrat": 0, "fulfil": [4, 7, 14], "full": [2, 3, 5, 7, 9, 11, 12, 15, 20, 23], "full_matric": 7, "fulli": [5, 8, 14, 20], "fun": [17, 23], "func": 4, "function": [0, 1, 4, 5, 6, 7, 11, 16, 17, 18], "functionali": 13, "fundament": [2, 8, 17, 23], "funtion": 4, "further": [4, 9, 11, 23], "furthermor": [2, 5, 7, 8, 9, 13, 14, 15, 17, 23], "futur": [2, 6, 10, 11, 23], "fy": [0, 19, 21, 22, 23], "fys5419": [22, 23], "fys5429": [22, 23], "f\u00f8470": [21, 23], "g": [0, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 20, 23], "g0": 4, "g_": [4, 11, 12], "g_0": 4, "g_1": [4, 12], "g_2": [4, 12], "g_analyt": 4, "g_dnn_ag": 4, "g_euler": 4, "g_i": 4, "g_m": [5, 12], "g_n": 5, "g_re": 4, "g_t": 4, "g_t_d2t": 4, "g_t_d2x": 4, "g_t_dt": 4, "g_t_hessian": 4, "g_t_hessian_func": 4, "g_t_jacobian": 4, "g_t_jacobian_func": 4, "g_trial": 4, "g_trial_deep": 4, "g_vec": 4, "gain": [3, 7, 9, 11, 12, 15], "galleri": [2, 23], "game": 6, "gamge": 23, "gamma": [2, 4, 10, 11, 12, 13, 15, 23], "gamma1": 10, "gamma2": 10, "gamma_": [2, 23], "gamma_0": 12, "gamma_1": 12, "gamma_1x": 12, "gamma_i": [2, 10, 20, 23], "gamma_j": 15, "gamma_k": 15, "gamma_m": 12, "gamma_x": [2, 23], "gap": 10, "gate": [6, 14], "gather": [2, 3, 14], "gaug": 14, "gaussbacksub": 18, "gaussian": [6, 7, 8, 10, 16, 20, 23], "gaussian_point": 16, "gaussian_rbf": 10, "gave": 15, "gavra": 23, "gbc": 23, "gca": [4, 8, 10, 15], "gd": 3, "gd_clf": 12, "gdclassiffiercgain": 12, "gdclassiffierconfus": 12, "gdclassiffierroc": 12, "gdm": 15, "gdregress": 12, "ge": [3, 7, 9, 20], "gen_loss": 6, "gen_tap": 6, "gender": [2, 23], "genener": 6, "gener": [0, 1, 2, 3, 4, 5, 7, 8, 10, 12, 13, 14, 15, 16, 18, 20, 22], "generaliz": 1, "generallay": 14, "generate_and_save_imag": 6, "generate_imag": 6, "generate_latent_point": 6, "generate_simple_clustering_dataset": 16, "generated_imag": 6, "generator_loss": 6, "generator_loss_list": 6, "generator_model": 6, "generator_optim": 6, "genom": 17, "geodes": 13, "geometr": [2, 15, 23], "geometri": 7, "georg": 22, "geotif": 8, "geq": [4, 7, 10, 11, 15], "geron": [2, 22, 23], "get": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 17, 18, 20, 21, 23], "get_dummi": 11, "get_paramet": 4, "get_split": 11, "get_yaxi": 10, "get_yticklabel": 8, "gh": 0, "gibb": [17, 23], "gif": 6, "gini": 12, "gini_index": 11, "ginvers": 15, "git": [0, 2, 17, 23], "giter": 15, "github": [2, 17, 19, 21, 22, 23], "gitignor": 0, "gitlab": [2, 17, 23], "give": [2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 20, 23], "given": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "global": [8, 9, 15], "glorot": 3, "gnew": 15, "go": [0, 1, 2, 3, 5, 7, 8, 10, 11, 13, 14, 15, 23], "goal": [2, 9, 11, 23], "goe": [0, 2, 3, 4, 7, 8, 15, 16, 18, 23], "golden": 15, "gone": 7, "gong": 3, "good": [0, 3, 5, 6, 7, 8, 11, 12, 13, 15, 17, 20, 22], "goodfellow": [6, 22, 23], "googl": [3, 6, 17, 23], "got": [3, 8], "gotten": 23, "gov": 8, "govern": 23, "gp": 22, "gpu": [3, 15, 17, 23], "grad": [4, 15], "grad_analyt": 15, "grade": 19, "gradient": [2, 5, 6, 9, 10, 11, 14, 17, 23], "gradientboostingclassifi": 12, "gradientboostingregressor": 12, "gradients_of_discrimin": 6, "gradients_of_gener": 6, "gradienttap": 6, "gradual": [3, 16], "grai": [6, 8], "graph": [1, 3, 11, 13, 14, 15], "graph_from_dot_data": 11, "graphic": [2, 3, 11, 23], "grasp": 2, "gray_r": [3, 5], "grayscal": 5, "great": [0, 7, 15], "greater": [3, 9, 20], "greatli": 15, "greedi": 11, "green": [2, 5, 11, 20], "grei": 6, "grid": [3, 5, 8, 9, 10, 14, 20], "grossli": 15, "ground": [2, 23], "group": [0, 2, 8, 9, 11, 16, 17, 19, 21, 23], "groupbi": [2, 23], "grow": [3, 5, 11, 12], "growth": [2, 23], "gru": 6, "guarante": [2, 6, 15, 20, 23], "guess": [3, 6, 12, 15, 16], "guestrin": 12, "guid": 3, "h": [0, 2, 3, 7, 8, 10, 15, 20, 21, 22, 23], "h1": 4, "h_": [2, 15, 23], "h_1": [4, 15], "h_2": [4, 15], "h_m": 12, "ha": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "haanen": [21, 23], "habit": 2, "had": [2, 3, 8, 9, 15, 23], "hadamard": [3, 14, 15], "half": [3, 10, 11], "halv": 12, "hand": [2, 3, 4, 5, 7, 13, 14, 15, 17, 18, 20, 21, 22, 23], "handi": 5, "handl": [0, 2, 3, 4, 7, 11, 13, 17], "handle_unknown": 11, "handsid": 14, "handwrit": 14, "handwritten": [3, 7], "happen": [3, 4, 5, 6, 7, 8, 12, 15, 20], "hard": [3, 9, 10, 12, 15], "hardcopi": [17, 23], "harder": [2, 3], "harmon": 5, "hasn": 23, "hassl": [2, 17, 23], "hast": [17, 23], "hasti": [1, 2, 8, 22, 23], "hat": [1, 2, 3, 7, 8, 9, 11, 12, 13, 14, 15, 18], "have": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "haven": 3, "he": [9, 23], "head": [2, 6, 12, 20], "header": [2, 23], "heads_proba": 12, "health": 2, "hear": [2, 15, 23], "heart": [2, 9, 23], "heatmap": [2, 3, 5, 9, 23], "heavili": 2, "heavisid": 3, "height": [3, 5, 8], "held": 15, "help": [0, 1, 2, 3, 6, 14, 15, 23], "helper": [6, 16], "henc": [2, 7, 8, 10, 11, 12, 14, 15, 23], "henrik": [21, 23], "her": 9, "here": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "hereaft": [2, 10, 14, 23], "hermitian": 18, "hessenberg": 18, "hessian": [2, 4, 7, 15], "heterogen": [11, 12], "hi": 9, "hidden": [3, 5, 6, 14], "hidden_bia": 3, "hidden_bias_gradi": 3, "hidden_layer_s": [2, 3, 23], "hidden_neuron": 6, "hidden_weight": 3, "hidden_weights_gradi": 3, "hierarch": 7, "high": [2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 15, 16, 17, 18, 23], "higher": [2, 3, 5, 7, 8, 10, 15, 23], "highest": [3, 4], "highli": [2, 5, 6, 12, 17, 18, 22, 23], "highwai": 2, "hing": 10, "hint": [0, 1, 15], "hip": 17, "hire": 2, "hist": [6, 8, 9, 20], "histogram": [2, 8, 9, 20], "histor": [9, 13], "histori": [0, 5, 6, 14], "hitherto": 7, "hjorth": [21, 23], "hobbi": 20, "hoc": 7, "hoff": 22, "hold": [3, 5, 8, 15, 16], "holder": [2, 23], "home": 2, "homepag": 23, "homework": [8, 15], "homogen": [3, 5, 11, 12, 15], "honchar": 4, "hopefulli": [0, 2, 13, 20, 23], "horizont": 13, "horlyk": [21, 23], "hors": [5, 9, 23], "hot": [3, 11], "hour": [3, 17, 19, 20, 21, 23], "how": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "howev": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "hspace": [2, 6, 10, 12, 20, 23], "hstack": 3, "htf": 23, "html": [1, 2, 17, 19, 21, 22, 23], "http": [0, 1, 2, 5, 6, 8, 15, 17, 18, 19, 21, 22, 23], "huang": [2, 23], "huber": [2, 23], "huge": [3, 5, 6, 17], "human": [2, 3, 5, 8, 11, 14], "humid": 11, "hundr": 3, "hungri": 3, "hybrid": 19, "hydrogen": [2, 23], "hyperbol": [3, 6, 14], "hyperparam": 10, "hyperparamet": [5, 6, 7, 8, 11, 15], "hyperplan": 13, "h\u00f8rlyk": [21, 23], "i": [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22], "i0": [2, 23], "i1": [2, 8, 10, 14, 23], "i2": [2, 10, 14, 23], "i3": [2, 14, 23], "i5": [2, 23], "i_": 15, "i_1": [7, 8], "i_2": [7, 8], "ian": 22, "ic": 3, "id": [9, 15], "ida": [21, 23], "idea": [2, 3, 4, 5, 6, 8, 11, 12, 14, 15, 18], "ideal": [2, 4, 8, 10, 15, 20, 23], "idem": 8, "ident": [7, 8, 14, 15, 18], "identifi": [2, 3, 9, 11, 13, 14, 15, 16, 23], "ieor": 20, "ifi": 22, "ifs": [17, 23], "ignor": [0, 2, 3, 5, 11], "ii": [18, 20], "iii": [18, 23], "ij": [1, 2, 3, 5, 8, 10, 14, 16, 18, 20, 23], "ik": [2, 18, 23], "illustr": [7, 9, 12, 14, 15, 16, 17, 23], "im": 8, "imag": [3, 5, 6, 8, 11, 13, 14, 16, 22, 23], "image_at_epoch_": 6, "image_batch": 6, "image_height": 5, "image_path": [2, 8, 9, 11, 23], "image_width": 5, "imageio": 8, "images_from_seed_imag": 6, "imagin": 3, "immedi": [2, 5, 6, 8, 17, 23], "implement": [2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 20, 23], "impli": [5, 7, 8, 9, 15, 18], "implicit": 5, "implicitli": [13, 20], "import": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20], "importantli": 5, "impos": [2, 8, 13, 14, 23], "imposs": [2, 7, 23], "impress": [2, 14, 23], "improv": [0, 2, 6, 7, 11, 12, 13, 15], "impur": 11, "imread": 8, "imshow": [3, 5, 6, 8], "in3050": [22, 23], "in3310": 23, "in4080": [22, 23], "in4300": [22, 23], "in4310": 22, "in5400": 5, "in5550": 22, "in_out_neuron": 6, "inaccur": 15, "inact": 14, "inadequ": [2, 23], "inch": 8, "includ": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 17, 20, 21, 22, 23], "include_bia": [8, 11], "incom": [1, 14], "incorrect": 3, "incoveni": 10, "increas": [2, 3, 5, 6, 7, 8, 11, 14, 15, 20, 23], "increasingli": 20, "ind": 8, "inde": [2, 4, 6, 7, 8, 15, 23], "indefinit": 6, "independ": [2, 7, 8, 9, 10, 14, 15, 20, 23], "index": [2, 3, 5, 6, 12, 16, 17, 18, 20, 22, 23], "index_col": [2, 23], "indic": [1, 2, 3, 5, 6, 7, 8, 11, 12, 13, 15, 23], "indispens": 8, "individu": [3, 8, 9, 12, 14, 20, 23], "indu": 2, "indx": 18, "indx1": 4, "indx2": 4, "indx3": 4, "ineffici": [5, 15], "inequ": [10, 15], "inertia": 15, "inf1000": [17, 23], "inf1100": [17, 23], "inf1100l": [17, 23], "inf1110": [17, 23], "inf3000": 23, "infeas": 11, "infer": [2, 3, 6, 8, 22, 23], "inferenc": 3, "infil": [2, 8, 9, 11, 23], "infin": [7, 8, 9, 13], "infinit": 5, "infinitesim": 20, "influenc": [8, 12], "influenti": 3, "info": 23, "inform": [2, 3, 5, 6, 8, 11, 13, 14, 15, 16, 18, 22, 23], "inforom": 0, "infti": [5, 8, 15, 20], "ingeni": 15, "ingredi": [2, 11, 23], "inher": 8, "inherit": [18, 23], "initi": [2, 3, 4, 8, 12, 15, 16, 18, 20, 23], "inject": 16, "inlin": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "inner": [2, 15], "inp": 6, "inplac": 15, "input": [1, 2, 3, 5, 6, 7, 8, 9, 10, 14, 15, 16, 20, 23], "input_dim": 3, "input_shap": [5, 6], "inputs": 3, "inputs_shuffl": [2, 3], "insert": [5, 7, 8, 10, 12, 20], "insid": [2, 6, 9], "insight": [2, 3, 7, 17, 23], "insist": [8, 15], "inspir": [2, 3, 14, 23], "instabl": 4, "instal": [0, 2, 3, 7, 8, 11], "instanc": [1, 2, 3, 4, 6, 8, 11, 13, 15, 23], "instanti": 12, "instead": [2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 15, 16, 18, 20, 23], "institut": 3, "instruct": [2, 3], "int": [2, 3, 4, 5, 6, 7, 8, 13, 15, 16, 18, 20], "int32": 12, "int_": [5, 8, 20], "int_0": 20, "int_a": 20, "intak": 2, "integ": [3, 4, 15, 16, 18, 20, 23], "integer_vector": 3, "integr": [5, 8, 20, 23], "intellig": [2, 16, 22, 23], "intend": 12, "intens": 3, "intention": 16, "interact": [2, 8, 11, 14, 17, 23], "intercept": [1, 2, 8, 10, 13, 15, 23], "intercept_": [2, 8, 10, 11, 15, 23], "interchang": [7, 14, 18], "interconnect": 3, "interest": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 17, 20, 23], "interfac": [2, 3, 18], "interior": [2, 11, 23], "intermedi": 18, "intern": [3, 12, 14], "interpol": [3, 5, 6, 8, 14], "interpr": 7, "interpret": [0, 1, 2, 3, 8, 11, 12, 14, 15, 18, 20], "interv": [2, 5, 7, 8, 9, 15, 20, 23], "intial": 15, "intract": [2, 6], "intrins": [5, 13, 18, 20, 23], "intro": [17, 22, 23], "introduc": [2, 3, 7, 8, 10, 12, 14, 18, 20, 23], "introduct": [3, 4, 6, 15, 22], "introductori": [2, 6, 18, 22, 23], "intuit": [2, 7, 8, 10, 14, 15, 23], "inv": [2, 7, 15, 23], "invalu": [2, 15, 17, 23], "invari": 3, "invd": 7, "inver": 10, "invers": [2, 5, 8, 15, 23], "inverse_transform": 10, "invert": [1, 2, 7, 9, 12, 15, 23], "invh": 15, "invok": [2, 10], "involv": [2, 4, 8, 9, 13, 14, 23], "io": [2, 17, 19, 21, 22, 23], "ip": [2, 10, 20, 23], "ipca": 13, "ipynb": [17, 23], "ipython": [2, 7, 9, 11, 13, 16, 17, 23], "iq": 8, "iri": [10, 11], "irreduc": 8, "irrelev": 7, "irrespect": [2, 23], "isn": 7, "isnul": 2, "isomap": 13, "issu": [0, 3, 11, 18], "it_arrai": 15, "item": [2, 15, 23], "items": [18, 23], "iter": [3, 4, 6, 8, 10, 15, 16, 20, 23], "its": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "itself": [7, 8, 14, 20, 23], "j": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 15, 16, 18, 20, 22, 23], "j1": 18, "j_": 8, "j_lasso_sk": 8, "j_ridge_sk": 8, "j_sk": 8, "jackknif": [8, 17, 23], "jacobian": [4, 15], "jason": 6, "jax": [17, 23], "jensen": [21, 23], "jerom": 22, "ji": [14, 18], "jit": 15, "jj": [2, 7, 8, 23], "jk": [2, 3, 8, 14, 18, 23], "jl": [2, 23], "jm": 18, "jnp": 15, "job": [0, 4, 10, 12], "join": [2, 6, 8, 9, 11, 23], "joint": [6, 7], "judg": 15, "judgement": 8, "julia": [17, 18], "jump": 20, "junk": 6, "jupit": 23, "jupyt": [0, 1, 2, 17, 22, 23], "just": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23], "justif": 2, "justifi": [5, 12], "k": [2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "k0": 9, "k1": 9, "kaggl": 8, "kappa_d": 20, "karl": [21, 23], "karush": 10, "katrin": [21, 23], "keep": [0, 2, 3, 6, 7, 8, 13, 15, 16, 18, 23], "keepdim": [3, 8, 12, 18], "kei": [2, 3, 5, 8, 14], "kept": [6, 8, 16], "kera": [2, 6, 17, 23], "kernel": [2, 3, 5, 17, 23], "kernel_regular": [3, 5], "kernel_s": 6, "kernelpca": 13, "kev": [2, 23], "kevin": [22, 23], "keyword": [18, 23], "kfold": 8, "kg": 3, "ki": 18, "kick": [3, 15], "kiener": 4, "kilomet": 8, "kind": [2, 4, 5, 6, 10, 14, 15, 16, 23], "kj": [8, 14, 18], "kjm": [17, 23], "kkt": 10, "kl": 20, "km": [14, 23], "kmean": 16, "kmeanspoint": 16, "kn_k": 16, "know": [0, 1, 2, 3, 4, 7, 8, 10, 15, 17, 23], "knowledg": [2, 17, 23], "known": [3, 5, 6, 7, 8, 9, 10, 11, 14, 18, 20, 22], "kondev": [2, 23], "kp": 20, "kpca": 13, "kroneck": 16, "kuhn": 10, "kvalsund": [21, 23], "kwown": [2, 23], "l": [2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 23], "l0": 9, "l1": [2, 3, 5, 9, 23], "l1_l2": [3, 5], "l1regl": 7, "l2": [3, 5], "l_": 18, "l_1": 9, "l_2": [9, 15], "l_j": 14, "la": 15, "la_i": 14, "la_k": 14, "lab": [17, 23], "label": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 20, 23], "labelencod": [9, 12], "labels": [8, 10, 11], "labels_shuffl": [2, 3], "laboratori": 19, "lack": [2, 23], "lagari": 4, "lagrang": [10, 13], "lambda": [2, 3, 4, 5, 7, 8, 9, 10, 12, 14, 15, 20, 23], "lambda_": 13, "lambda_0": 13, "lambda_1": [7, 10, 13], "lambda_2": [10, 13], "lambda_i": [10, 13], "lambda_iy_i": 10, "lambda_jy_iy_j": 10, "lambda_k": 10, "lambda_n": [7, 10], "lamda": 3, "land": [2, 10], "landmark": 10, "landscap": 15, "langl": [2, 8, 13, 20, 23], "languag": [2, 3, 6, 10, 17, 18, 22, 23], "lapack": [18, 23], "laplac": 7, "laptop": [0, 17], "larg": [2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 20, 22, 23], "larger": [2, 5, 7, 8, 10, 12, 13, 15, 20, 23], "largest": [6, 10, 13], "lasso": [2, 9, 17, 23], "lasso_sk": 8, "last": [1, 2, 3, 5, 6, 7, 8, 9, 10, 14, 18, 20, 21, 23], "latent": 6, "latent_dim": 6, "latent_point": 6, "latent_space_value_rang": 6, "later": [0, 2, 3, 6, 9, 10, 14, 15, 16, 17, 23], "latest": [0, 6, 17], "latest_checkpoint": 6, "latex": 23, "latter": [2, 5, 8, 9, 10, 13, 15, 18, 20, 23], "lattic": 14, "law": 2, "layer": [2, 6, 15, 23], "lbfg": [9, 11, 12], "lcc": [7, 8], "lda": 13, "ldot": [2, 8, 13, 23], "le": [7, 9, 12, 15, 20], "lead": [1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "leaf": 11, "leaki": 3, "leakyrelu": 6, "lear": 15, "learn": [5, 6, 7, 8, 9, 10, 11, 12, 14, 18, 21, 22], "learnabl": 5, "learner": 12, "learnig": 23, "learning_r": [10, 12], "learning_rate_init": [2, 3, 23], "learning_schedul": 15, "least": [2, 9, 10, 12, 13, 17, 18, 20], "leat": 15, "leav": [2, 3, 5, 7, 8, 11, 13, 23], "lectur": [2, 3, 7, 12, 13, 14, 15, 17, 18, 19, 21, 22], "lecturenot": [2, 17, 22, 23], "left": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "leftarrow": [10, 14], "legend": [0, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 23], "leinonen": 23, "len": [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 18, 23], "length": [1, 2, 3, 5, 6, 10, 11, 15, 17, 23], "length_of_sequ": 6, "leq": [2, 7, 9, 10, 15, 16, 20, 23], "less": [2, 3, 5, 6, 7, 8, 10, 11, 15, 17, 20, 23], "lessen": 3, "let": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "letter": [1, 2, 18, 20, 23], "level": [2, 3, 7, 8, 11, 17, 18, 19, 21, 23], "li": [10, 13, 23], "lib": 23, "liblinear": 12, "librari": [2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 18, 20, 22], "licens": [2, 3, 17, 23], "lie": [2, 8, 13, 20, 23], "life": [2, 3, 10, 14, 23], "lifetim": 15, "like": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 17, 18, 20, 23], "likelihood": [2, 3, 7, 11, 23], "lim_": 20, "limit": [2, 7, 8, 10, 14, 18, 23], "lin_clf": 10, "lin_model": 2, "lin_reg": 11, "linalg": [2, 4, 7, 8, 10, 13, 15, 18, 20, 23], "line": [0, 1, 2, 5, 8, 10, 13, 15, 23], "line1": 10, "line2": 10, "line3": 10, "line_model": 0, "line_ms": 0, "line_predict": 0, "linear": [1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 17, 20], "linear_model": [0, 1, 2, 7, 8, 9, 10, 11, 12, 13, 15, 23], "linear_regress": 8, "linearli": 7, "linearloc": [8, 15], "linearregress": [0, 1, 2, 8, 9, 11, 23], "linearsvc": 10, "liner": [3, 5], "linerar": 12, "linewidth": [2, 4, 6, 8, 10, 11, 12], "link": [0, 2, 6, 11, 14, 17, 19, 21, 23], "linlag": 7, "linpack": [18, 23], "linreg": [2, 23], "linspac": [1, 2, 4, 5, 6, 8, 10, 11, 12, 15, 18, 20, 23], "linu": 6, "linux": [2, 3, 17, 23], "liquid": [2, 23], "list": [0, 2, 3, 4, 5, 6, 11, 17, 23], "listedcolormap": [11, 12], "literatur": [3, 9, 16, 22], "littl": [3, 5, 11, 14], "live": [1, 10], "ll": [2, 20, 23], "lle": 2, "lloyd": [6, 16], "lmb": [2, 4, 7, 8], "lmbd": [2, 3, 5, 23], "lmbd_val": [2, 3, 5, 23], "lmbda": 15, "ln": [3, 15], "load": [2, 3, 6, 8, 9, 11, 12], "load_boston": 2, "load_breast_canc": [3, 9, 11, 12, 13], "load_data": [5, 6], "load_digit": [3, 5], "load_iri": [10, 11], "loc": [2, 5, 8, 9, 10, 11, 12, 23], "local": [0, 2, 3, 5, 9, 14, 15], "locat": [0, 4, 5, 10], "log": [0, 2, 3, 4, 6, 7, 8, 9, 11, 12, 13, 15, 18, 23], "log10": [2, 7, 8], "log_": [2, 23], "log_clf": 12, "logarithm": [2, 7, 9, 18, 23], "logic": [2, 3, 11, 23], "login": 0, "logist": [2, 3, 4, 10, 11, 12, 13, 14, 15, 17], "logisticregress": [9, 11, 12, 13], "logit": 9, "logreg": [9, 11, 12, 13], "logspac": [2, 3, 5, 7, 8, 23], "long": [2, 3, 5, 6, 14, 15, 23], "longer": [4, 5, 10, 12, 16, 18, 20, 23], "loocv": 8, "look": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 23], "loop": [1, 3, 6, 8, 12, 14, 16, 17, 18, 23], "lose": 3, "loss": [2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15, 18, 23], "loss_fil": 6, "lossfil": 6, "lost": 6, "lot": [1, 2, 3, 6, 8], "low": [2, 8, 11, 12, 13, 23], "lower": [1, 2, 3, 5, 8, 11, 12, 18], "lowercas": [18, 23], "lowest": [11, 15, 20], "lr": [3, 5, 6, 12], "lstat": 2, "lstm": 6, "lstm_2layer": 6, "lstsq": [2, 23], "lt": 8, "lu": [2, 7, 23], "lubksb": 18, "luckili": 4, "ludcmp": 18, "lux": 18, "lvert": 3, "lw": [2, 23], "m": [0, 2, 3, 4, 5, 7, 8, 10, 11, 12, 13, 14, 15, 18, 20, 21, 22, 23], "m_": [11, 14], "m_1": 16, "m_h": [2, 23], "m_k": 16, "m_l": 14, "m_n": [2, 23], "m_p": [2, 23], "m_t": 15, "ma": 13, "machin": [0, 1, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 18, 22], "machinelearn": [1, 2, 8, 17, 19, 21, 22, 23], "mackai": 22, "made": [2, 3, 5, 6, 7, 8, 9, 11, 13, 14, 23], "mae": [2, 23], "magic": 6, "magnitud": [3, 8, 9, 15], "mai": [2, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23], "mail": [19, 21], "main": [2, 3, 5, 6, 7, 8, 9, 11, 18, 22], "mainli": [2, 7, 8, 9, 11, 23], "maintain": 8, "major": [3, 8, 11, 12, 15, 18, 23], "make": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 20, 22, 23], "make_axes_locat": 8, "make_moon": [10, 11, 12], "make_pipelin": [2, 8, 12], "makedir": [2, 8, 9, 11, 23], "makeplot": 2, "malcondit": 18, "malign": [3, 9, 11], "mammographi": 7, "manag": [0, 2, 4, 5, 17, 23], "mandatori": [21, 23], "mani": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 17, 18, 20, 22, 23], "manifold": 13, "manner": 5, "manual": 8, "map": [2, 3, 4, 8, 9, 10, 13, 14, 16, 20, 23], "margin": [2, 7, 10], "marit": [2, 23], "mark": 23, "marker": [2, 9, 18, 23], "markov": [17, 23], "marsaglia": 20, "mass": [2, 3, 7, 15], "massag": [2, 23], "masses2016": [2, 23], "masses2016ol": [2, 23], "masses2016tre": 2, "masseval2016": [2, 23], "master": [19, 21], "mat": [17, 23], "mat1100": [17, 23], "mat1110": [17, 23], "mat1120": [17, 23], "match": [0, 3, 6, 7, 15, 16], "materi": [0, 6, 7, 9, 15, 18, 19, 21], "math": [5, 9, 14, 15, 18, 20, 22, 23], "mathbb": [2, 6, 7, 8, 9, 10, 13, 14, 15, 16, 18, 20, 23], "mathbf": [2, 7, 8, 9, 10, 15, 18, 23], "mathcal": [3, 7, 8, 9, 15], "matheemat": 5, "mathemat": [2, 8, 13, 14, 15, 17, 18, 20, 22, 23], "mathemati": 23, "mathrm": [2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 23], "matmul": [3, 4, 7], "matnat": 22, "matplotlib": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "matric": [1, 2, 3, 5, 6, 8, 9, 10, 13, 15, 17], "matrix": [2, 4, 5, 6, 8, 9, 10, 12, 15, 20], "matshow": 3, "matter": [4, 5, 15], "max": [2, 3, 4, 5, 6, 11, 12, 14, 15, 21, 23], "max_depth": [2, 11, 12], "max_diff": 4, "max_diff1": 4, "max_diff2": 4, "max_it": [2, 3, 10, 15, 23], "max_iter": 16, "max_leaf_nod": 12, "max_sampl": 12, "maxdegre": [2, 8, 12], "maxdepth": 12, "maxim": [3, 6, 7, 9, 10, 13], "maximum": [2, 4, 5, 7, 9, 10, 11, 12, 15, 16, 23], "maxpolydegre": [7, 8], "maxpooling2d": 5, "mbox": [7, 8], "mcculloch": 14, "md": 13, "mdoel": 6, "mean": [0, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "mean_absolute_error": [2, 23], "mean_divisor": 16, "mean_i": 20, "mean_matrix": 16, "mean_squared_error": [0, 2, 6, 8, 9, 12, 23], "mean_squared_log_error": [2, 23], "mean_vector": 16, "mean_x": 20, "meaning": [2, 6, 9, 23], "meansquarederror": [2, 23], "meant": [5, 9, 12, 15], "measur": [1, 2, 3, 4, 7, 8, 11, 13, 14, 16, 20, 23], "mechan": [2, 6, 20, 23], "median": [2, 23], "medicin": 14, "medium": [6, 10, 15], "medv": 2, "meet": [2, 21], "mehta": [2, 23], "memori": [5, 6, 13, 14, 15, 18], "mention": [2, 14, 15, 20, 23], "mere": 2, "meshgrid": [4, 7, 8, 10, 11, 12, 13], "mess": 0, "messag": [7, 15], "messi": 4, "met": [2, 5, 10], "meteorolog": 11, "meter": 8, "method": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 13, 14, 16, 17, 18, 20, 22], "metion": 8, "metric": [0, 2, 3, 5, 8, 9, 11, 12, 16, 23], "metropoli": [17, 23], "mev": [2, 20, 23], "mgd": 15, "mglearn": [17, 23], "mgrid": 15, "mhjensen": 23, "mi": 12, "mia": [21, 23], "microsoft": 22, "mid": 3, "midel": 6, "midnight": 0, "midpoint": 11, "might": [0, 2, 3, 4, 6, 8, 11, 15], "mild": 11, "millimet": 8, "million": [2, 23], "mimic": 14, "min": [2, 4, 7, 10, 11], "min_": [2, 4, 7, 16, 23], "min_samples_leaf": 11, "mind": [0, 2, 8, 15, 23], "mindboard": 6, "mine": [17, 23], "mini": [3, 13, 14, 15], "minibatch": [3, 13, 15], "minibathc": 15, "miniforge3": 23, "minim": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16], "minima": [2, 3, 9, 15, 23], "minimum": [2, 3, 4, 8, 10, 11, 13, 15], "minmaxscal": 2, "minor": 20, "minst": 3, "minu": 9, "mirjalili": 23, "mirror": 11, "misc": 8, "misclassif": [10, 11, 12], "misclassifi": [10, 12], "miser": 2, "mismatch": 3, "miss": [2, 9, 12], "mistak": 6, "mit": 22, "mix": [3, 4, 23], "mixtur": 15, "mk": [11, 18], "mkdir": [2, 8, 9, 11, 23], "ml": [2, 3, 12, 15, 18], "mlab": 20, "mle": [7, 9], "mlp": 3, "mlpclassifi": 3, "mlpregressor": [2, 23], "mm": 18, "mn": [14, 20], "mnist": [3, 13], "mod": 20, "mode": [19, 21, 23], "model": [1, 4, 5, 7, 9, 10, 11, 12, 13, 15, 16, 17, 20, 22], "model_select": [0, 1, 2, 3, 5, 7, 8, 9, 11, 12, 13, 23], "moder": 12, "modern": [2, 8, 9, 17, 23], "modif": [4, 14, 15], "modifi": [2, 3, 5, 7, 9, 10, 12, 14, 15, 23], "modul": [1, 2, 18, 23], "modular": 20, "modulo": 20, "moe": 13, "moment": [7, 8, 15, 20], "mondai": [21, 23], "monitor": 15, "monoton": [7, 14, 20], "mont": [2, 8, 17, 20, 22, 23], "montli": 1, "moor": [7, 8], "more": [1, 2, 3, 4, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20], "moreov": [2, 5], "morten": [21, 23], "mortenhj": 23, "most": [0, 1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23], "mostli": [3, 13], "motion": [2, 15], "motiv": [3, 6], "move": [0, 1, 2, 6, 7, 8, 9, 11, 14, 15, 16, 20], "mpl": [2, 9, 23], "mpl_toolkit": [4, 8, 15], "mplot3d": [4, 8, 15], "mplregressor": 3, "mse": [0, 1, 2, 6, 7, 8, 11, 12, 23], "mse_simpletre": 12, "mselassopredict": 7, "mselassotrain": 7, "mseownridgepredict": 8, "msepredict": 7, "mseridgepredict": [2, 7, 8], "msetrain": 7, "msle": [2, 23], "mt": [9, 14], "mu": [2, 8, 13, 15, 20, 23], "mu0": 20, "mu1": 20, "mu2": 20, "mu_": [8, 20], "mu_i": 8, "mu_n": 13, "mu_x": 20, "much": [0, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 18, 20, 23], "multi": [2, 3, 5, 9, 17, 23], "multiclass": [3, 9], "multidimension": [13, 14, 23], "multilay": 3, "multinomi": 9, "multipl": [0, 4, 6, 7, 8, 9, 14, 15, 20], "multipli": [5, 7, 8, 13, 15, 18, 20], "multiplum": 10, "multivari": [2, 4, 12, 13, 17, 20, 23], "multivariate_norm": [13, 16], "multpli": 1, "murphi": [13, 22, 23], "must": [0, 3, 4, 7, 8, 10, 12, 14, 15, 16, 20], "mutat": 9, "mutual": [3, 5, 8, 15], "mx_": 20, "my": 23, "myenv": 23, "myriad": [2, 17, 23], "mz1": 20, "mz2": 20, "m\u00f8svatn": 8, "n": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 23], "n1": 18, "n2": 18, "n_": [3, 4, 5, 10, 14, 20], "n_0": [14, 20], "n_boostrap": [8, 12], "n_bootstrap": 8, "n_categori": [3, 5], "n_cluster": 16, "n_compon": 13, "n_epoch": 15, "n_estim": 12, "n_examples_to_gener": 6, "n_featur": 3, "n_filter": 5, "n_hidden": 4, "n_hidden_neuron": [2, 3, 23], "n_i": 20, "n_input": [2, 3, 5], "n_instanc": 11, "n_job": 12, "n_k": 16, "n_l": [14, 20], "n_layer": 3, "n_m": 11, "n_neuron": 3, "n_neurons_connect": 5, "n_neurons_layer1": 3, "n_neurons_layer2": 3, "n_point": 16, "n_sampl": [8, 10, 11, 12, 16], "n_split": 8, "n_step": 6, "n_t": 4, "n_x": 4, "nabla": [3, 15], "nabla_": [4, 15], "nabla_w": 15, "nag": 15, "naimi": [2, 23], "naiv": 9, "naive_kmean": 16, "name": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 20, 21, 23], "narrow": 15, "nation": [3, 7], "nativ": [17, 23], "natur": [2, 3, 6, 10, 11, 14, 15, 20, 22, 23], "navier": 14, "navig": 0, "nb": 20, "nb_": 18, "nbconvert": 23, "nd": 16, "ndarrai": 8, "ne": [11, 12, 18, 20], "nearest": [3, 5, 8, 13], "nearli": 15, "neat": 23, "neccesari": 8, "necess": 4, "necessari": [2, 3, 5, 6, 10, 16, 23], "necessarili": [2, 6, 13, 20, 23], "necesserali": 7, "neck": 9, "need": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20], "neg": [2, 3, 5, 7, 8, 9, 12, 15, 18, 20, 23], "neg_mean_squared_error": 8, "neglect": 20, "neglig": 20, "neighbor": [5, 8, 13], "neither": [6, 15], "neq": [15, 16, 20], "nervou": 14, "nest": [11, 14], "nesterov": 15, "net": [4, 6, 14], "netlib": [18, 23], "network": [2, 11, 15, 17, 22], "neural": [2, 15, 17, 22], "neural_network": [2, 3, 4, 23], "neuralnetwork": 3, "neuron": [3, 4, 5, 6, 14], "neutral": [2, 23], "neutron": [2, 23], "never": [3, 6, 8, 11, 20], "new": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 23], "new_chang": 15, "new_hobbit": 23, "newaxi": [2, 5, 8, 11], "newli": [2, 23], "newton": [3, 9, 10, 15, 20], "next": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 15, 16, 23], "next_guess": 15, "next_input": 6, "ng": 3, "ni": 16, "nice": [2, 3, 7, 13, 23], "niter": 15, "nitric": 2, "nlambda": [2, 7, 8], "nlp": 22, "nm": 20, "nm_n": [2, 23], "nmse": 8, "nn": [4, 7, 8, 14, 18, 23], "nn_model": 3, "nnmin": 4, "node": [3, 5, 11, 12, 14], "nois": [2, 6, 7, 8, 10, 11, 12, 15, 23], "noise_dimens": 6, "noisi": [3, 8], "non": [2, 3, 5, 7, 8, 9, 11, 12, 13, 14, 15, 16, 18, 20, 23], "none": [2, 3, 4, 6, 7, 11, 12, 15, 20, 23], "nonlinear": [5, 8, 10, 11, 13, 14], "nonneg": [8, 11, 15], "nonparametr": 8, "nonsens": 20, "nonsingular": 18, "nonumb": [5, 9, 10, 15, 18], "nor": [3, 6, 15], "norm": [2, 3, 7, 8, 10, 13, 15, 23], "normal": [1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23], "normali": [18, 23], "norwai": [8, 23], "notat": [2, 4, 7, 8, 15, 16, 20, 23], "note": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 16, 17, 18, 20, 22, 23], "notebook": [0, 1, 2, 3, 5, 11, 17, 23], "noth": [3, 4, 7, 10, 14, 16, 20], "notic": [6, 7, 14, 15, 18, 20, 23], "notion": 5, "novel": [5, 8, 12, 23], "novemb": [3, 21, 23], "now": [0, 1, 2, 4, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20, 23], "nowadai": [2, 3, 5, 11, 17, 23], "nox": 2, "np": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23], "npr": 4, "nsampl": 8, "nt": 4, "nu": 20, "nuclear": 7, "nuclei": [2, 20, 23], "nucleon": [2, 23], "nucleu": [2, 23], "num": 6, "num_coordin": 4, "num_hidden_neuron": 4, "num_it": 4, "num_neuron": 4, "num_neurons_hidden": 4, "num_point": 4, "num_tre": 12, "num_valu": 4, "number": [1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 23], "numberid": 9, "numberparamet": 5, "numer": [2, 7, 8, 11, 12, 13, 14, 15, 17, 18, 22, 23], "numpi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20], "nunmpi": 7, "nx": 4, "ny": 20, "o": [2, 3, 6, 7, 8, 9, 10, 11, 13, 18, 21, 22, 23], "obei": [8, 13, 15], "object": [0, 2, 3, 6, 10, 12, 18, 23], "obliqu": 7, "observ": [2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 23], "obtain": [2, 3, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 23], "obviou": [7, 8, 13, 20], "obviouli": 23, "obvious": [2, 6, 7, 8, 18, 23], "occupi": 2, "occur": [2, 8, 10, 11, 18, 20, 23], "octob": [21, 23], "od": 2, "odd": [2, 5, 9, 23], "odenum": 4, "odesi": 4, "oen": 2, "off": [3, 5, 6, 7, 11, 15, 20], "offer": [8, 13, 17, 18, 19, 21, 23], "offic": [21, 23], "offici": [19, 23], "often": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "ofter": [18, 23], "ol": [2, 15], "old": [0, 3, 7, 12, 15], "ols_paramet": 1, "ols_sk": 8, "ols_svd": 8, "olsbeta": [2, 7], "omega": [4, 5, 8], "omega_0": 5, "omit": [2, 7, 23], "onc": [3, 8, 11, 13, 15], "one": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 20, 21, 23], "onehot": 3, "onehot_vector": 3, "onehotencod": 11, "ones": [1, 2, 4, 7, 8, 10, 11, 12, 13, 15, 18, 23], "ones_lik": 6, "onl": 5, "onli": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "onlin": [0, 13, 19], "onto": [7, 13], "open": [0, 2, 3, 6, 8, 9, 11, 17, 19, 21, 23], "oper": [0, 1, 2, 3, 5, 7, 8, 12, 13, 14, 15, 17, 20, 23], "operation": 20, "oplu": 20, "opmiz": 15, "opportun": 2, "oppos": [8, 15], "opposit": [3, 7, 10], "opt": [3, 7, 23], "optim": [1, 2, 4, 5, 6, 7, 8, 9, 11, 12, 13, 16], "optimis": [3, 5], "option": [0, 2, 3, 5, 7, 8, 10, 13, 18], "optmiz": [3, 10, 15], "oral": 23, "orang": 2, "order": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 18, 20, 23], "ordinari": [2, 4, 5, 9, 13, 15, 17], "oreilli": [22, 23], "org": [1, 2, 5, 6, 17, 18, 22, 23], "organ": [8, 9, 12, 18], "orient": [3, 7, 20], "origin": [0, 2, 5, 7, 8, 10, 13, 14, 15, 18, 23], "orthogn": 7, "orthogon": [2, 7, 8, 10, 13, 15, 18, 23], "orthonorm": 7, "os": [21, 23], "oscar": 3, "oscil": [5, 15], "oskar": 23, "oskarlei": 23, "oslo": [2, 17, 19, 21, 23], "osx": [2, 17, 23], "other": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 15, 16, 17, 19, 20, 21, 22], "otherwis": [2, 3, 6, 9, 15, 18, 23], "ouput": [7, 9, 14], "our": [0, 1, 3, 4, 5, 8, 9, 10, 11, 12, 14, 16, 17, 18, 20], "ourmodel": 2, "ourselv": [2, 7, 8, 10, 13, 15, 23], "out": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23], "out_fil": 11, "outcom": [2, 9, 11, 12, 14, 20], "outdoor": 11, "outer": [8, 14, 15], "outfil": 6, "outlier": [2, 10, 23], "outlin": [8, 12, 13], "outlook": 11, "outperform": 12, "output": [2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 23], "output_bia": 3, "output_bias_gradi": 3, "output_shap": 6, "output_weight": 3, "output_weights_gradi": 3, "outputlayer1": 14, "outputlayer2": 14, "outsid": 6, "over": [0, 1, 2, 3, 5, 6, 7, 8, 11, 12, 14, 15, 18, 23], "over1": 15, "overal": [3, 12], "overcast": 11, "overcom": [14, 15], "overdetermin": [2, 23], "overfit": [2, 3, 5, 8, 11, 12, 15], "overflow": 7, "overhead": 14, "overlap": [5, 9, 10, 11], "overlin": [2, 7, 8, 11, 12, 13, 16, 18, 23], "overst": 2, "overtrain": 6, "overview": 5, "own": [1, 6, 7, 8, 10, 14, 15, 17, 18], "owner": 2, "ownmsepredict": 2, "ownmsetrain": 2, "ownridgebeta": [2, 8], "ownypredictridg": 2, "ownytilderidg": 2, "oxid": 2, "p": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23], "p0": 4, "p1": 4, "p_": [4, 6, 10, 11], "p_hidden": 4, "p_i": [7, 20], "p_j": 20, "p_n": 20, "p_output": 4, "p_x": 20, "pack": [2, 23], "packag": [0, 2, 3, 5, 6, 7, 10, 13, 15, 17, 20], "packtpub": 23, "packtpublish": 23, "pad": [5, 6], "page": [2, 17, 23], "pai": [0, 2, 3, 11, 15], "pair": [2, 4, 5, 11, 17, 20, 23], "paltform": 0, "panda": [2, 6, 7, 8, 9, 11, 13, 17], "panel": 23, "paper": 3, "paradigm": [2, 23], "parallel": [12, 15, 17, 18, 23], "param": 4, "paramat": 4, "paramet": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 20], "parameter": [2, 8, 12, 23], "parametr": [2, 8, 23], "paramt": [5, 7], "part": [2, 3, 5, 7, 8, 12, 18, 19, 20, 21, 23], "partial": [1, 2, 3, 7, 8, 9, 10, 12, 13, 14, 15, 20, 23], "particip": [0, 17, 19, 21, 23], "particl": [2, 6, 15, 20, 23], "particular": [1, 2, 3, 4, 5, 7, 8, 11, 12, 13, 14, 15, 20, 22, 23], "particularli": [7, 8, 10, 13, 15, 20], "partit": [3, 6, 11], "partli": [8, 23], "partner": 0, "pass": [4, 5, 14, 16], "past": [12, 20], "patch": [8, 20], "path": [2, 6, 8, 9, 11, 17, 23], "patient": 9, "patter": 6, "pattern": [2, 5, 6, 14, 22, 23], "pauli": [2, 23], "pc": [13, 17], "pca": [2, 9, 17, 23], "pd": [2, 6, 7, 8, 9, 11, 13, 23], "pde": 4, "pdf": [0, 1, 2, 5, 6, 7, 8, 11, 22, 23], "pedagog": [2, 23], "penal": 8, "penalti": [8, 15], "penros": [7, 8], "pentagon": 15, "peopl": [2, 3, 11, 15, 17], "per": [2, 3, 8, 19, 21, 23], "percentag": [2, 12, 13, 21], "perceptron": [2, 3, 9, 23], "peregrin": 23, "perfect": [2, 3, 15, 23], "perfectli": [6, 8], "perform": [1, 2, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 16, 17, 18, 20, 23], "performac": 6, "perhap": [2, 7, 15, 23], "perimet": 3, "period": [3, 6, 20], "permiss": 0, "permut": 13, "persist": 15, "person": [1, 7, 8, 9, 19, 21, 23], "perspect": 22, "pertin": [14, 23], "petal": [10, 11], "peter": 22, "phantom": 20, "phase": [8, 14], "phenomena": 20, "phi": 10, "phi_k": 10, "philosophi": 15, "phone": [21, 23], "photo": [6, 23], "phrase": [2, 23], "physic": [2, 3, 6, 9, 14, 15, 20, 21, 22, 23], "pi": [4, 5, 7, 8, 9, 11, 14, 15, 20], "pick": [3, 11, 12, 13, 15, 16], "pickl": 3, "pictur": [2, 23], "pie": [17, 23], "piec": [13, 16], "pillow": [2, 17, 23], "pinv": [7, 8, 15], "pip": [0, 2, 3, 17, 23], "pip3": [2, 3, 23], "pipelin": [2, 8, 10, 12], "pippin": 23, "pit": 6, "pitfal": 8, "pitt": 14, "pixel": [3, 5, 6, 23], "pixel_height": [3, 5], "pixel_width": [3, 5], "place": [0, 2, 6, 8, 10, 15, 18, 23], "plai": [2, 5, 6, 7, 8, 10, 13, 17, 23], "plain": [10, 12, 14, 15, 16], "plan": [8, 11, 21, 22, 23], "plane": [10, 11], "plateau": 7, "platform": [17, 23], "plausibl": 14, "pleas": [15, 21, 23], "plenti": 3, "plethora": [5, 14], "plot": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "plot_confusion_matrix": [9, 12], "plot_count": 8, "plot_cumulative_gain": [9, 12], "plot_data": 3, "plot_dataset": 10, "plot_decision_boundari": [11, 12], "plot_import": 12, "plot_max": 6, "plot_min": 6, "plot_model": 6, "plot_numb": 6, "plot_predict": 10, "plot_regression_predict": 11, "plot_result": 6, "plot_roc": [9, 12], "plot_surfac": [4, 8, 15], "plot_train": 11, "plot_tre": [11, 12], "plt": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "plu": [2, 5, 7, 9, 23], "pm": 10, "pmatrix": 4, "pml": 22, "pn": 5, "png": [2, 6, 8, 9, 11, 23], "point": [2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 23], "point_1": 6, "point_2": 6, "poisson": [17, 20, 23], "poli": [8, 10], "poly100_kernel_svm_clf": 10, "poly3": 2, "poly3_plot": 2, "poly_featur": [0, 10, 11], "poly_features10": 11, "poly_fit": 11, "poly_fit10": 11, "poly_kernel_svm_clf": 10, "poly_model": 0, "poly_ms": 0, "poly_predict": 0, "polydegre": [2, 7, 8, 12], "polygon": 15, "polym": 14, "polynomi": [0, 2, 7, 8, 9, 10, 11, 12, 13, 23], "polynomial_featur": [0, 1, 8], "polynomial_svm_clf": 10, "polynomialfeatur": [0, 1, 2, 8, 10, 11], "polytrop": [2, 8], "pool": 5, "pool_siz": 5, "poor": [3, 15], "poorli": 2, "popul": [2, 7, 23], "popular": [0, 2, 3, 5, 8, 9, 10, 11, 13, 14, 17, 18, 20], "popularli": [2, 23], "portabl": 12, "portion": [13, 15], "pose": [2, 6, 7, 8, 13, 20, 23], "posit": [2, 3, 4, 5, 7, 9, 10, 12, 13, 15, 16, 18, 20, 23], "possibl": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 21, 23], "possibli": [8, 10, 15], "posterior": 7, "postpon": 2, "postul": 7, "potenti": [2, 5, 7, 8, 14, 15], "pott": 14, "power": [2, 3, 7, 8, 10, 11, 14, 15, 23], "pp": [7, 8], "practic": [1, 2, 7, 8, 9, 10, 20], "practition": [2, 3, 5, 23], "pre": 23, "preced": [3, 13, 14, 20], "preceed": 6, "preceq": 10, "precis": [2, 4, 7, 13, 15, 18, 20, 23], "pred": 8, "predicit": 2, "predict": [0, 1, 2, 3, 7, 8, 9, 10, 11, 12, 17, 22, 23], "predict_prob": 3, "predict_proba": [9, 12], "predictor": [2, 7, 8, 9, 11, 12, 13, 23], "prefer": [2, 3, 8, 10, 11, 13, 15, 17, 23], "prepar": [2, 8, 18, 23], "preprocess": [0, 1, 2, 6, 8, 9, 10, 11, 12, 13], "prerequisit": 2, "presenc": 15, "present": [2, 7, 8, 9, 11, 14, 15, 18, 20, 23], "preserv": [5, 13, 18], "press": [0, 15, 22], "pretrain": [3, 6], "pretti": [2, 6, 10, 11, 17, 23], "prev_centroid": 16, "prevent": [15, 20], "previou": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 18, 20], "previous": [4, 5, 11, 12, 20], "price": [2, 6, 11, 15], "primal": 10, "primari": [2, 9, 23], "prime": 20, "princip": [2, 7, 9, 17, 23], "principl": [2, 8, 9, 10, 16, 23], "print": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23], "print_funct": [10, 11], "printout": [2, 23], "prior": [2, 7, 8, 23], "privat": 2, "prob": [3, 20], "probabilist": [2, 22, 23], "probabl": [2, 3, 5, 6, 8, 9, 12, 15, 17, 23], "problem": [2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 20], "probml": 22, "proce": [2, 7, 8, 9, 10, 11, 12, 13, 15, 18, 23], "procedur": [4, 6, 7, 8, 10, 12, 13, 15], "proceed": 18, "process": [2, 4, 6, 8, 11, 12, 14, 15, 17, 18, 20, 22, 23], "prod": 22, "prod_": [3, 7, 9], "produc": [2, 5, 6, 7, 8, 11, 12, 13, 14, 15, 17, 18, 20, 23], "product": [1, 2, 3, 5, 7, 8, 9, 10, 14, 15, 17, 18, 23], "profess": [2, 23], "program": [0, 2, 3, 6, 7, 8, 10, 14, 16, 17, 18, 19, 20, 21, 23], "programm": 18, "progress": [3, 6, 16], "prohibit": 8, "project": [0, 2, 3, 4, 5, 7, 13, 15, 17, 19], "project_root_dir": [2, 8, 9, 11, 23], "promin": 14, "promis": 10, "promot": [21, 23], "prone": [0, 11], "pronounc": [15, 17, 23], "proof": [2, 13, 14, 15, 23], "propag": [4, 5, 15], "proper": [2, 4, 8, 9], "properli": [3, 8, 10, 12, 15], "properti": [1, 2, 3, 5, 14, 15, 18, 23], "proport": [2, 3, 7, 11, 13, 15, 20, 23], "propos": [3, 6, 8, 12, 23], "propto": [7, 15], "proton": [2, 23], "prove": [5, 15], "provid": [2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 15, 17, 18, 20, 23], "proxi": [3, 15], "prune": 11, "pseudo": [18, 20], "pseudoinv": 7, "pseudoinvers": [7, 8], "pseudorandom": [8, 20], "psychologi": [2, 23], "pt": 15, "public": [0, 2, 17, 23], "pull": 0, "punish": [2, 3, 23], "pure": [5, 11, 20], "purest": 11, "puriti": 11, "purpos": [2, 5, 12, 14, 16, 23], "push": 0, "put": 3, "py": [7, 23], "pycod": 23, "pydata": 17, "pydot": 11, "pyhton2": 23, "pylab": [2, 9, 23], "pypi": 17, "pyplot": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "pythagora": 7, "python": [3, 4, 5, 7, 8, 10, 13, 14, 15, 16, 20], "python2": 2, "python3": [2, 17, 23], "pytorch": [2, 17, 23], "q": [7, 8, 10, 13, 20], "qp": 10, "qquad": [4, 13, 15, 18], "qr": [7, 8, 18], "quad": [3, 15, 18], "quadrat": [2, 10, 11, 15, 23], "qualit": [6, 11, 20], "qualiti": [2, 11, 17, 23], "quantifi": 3, "quantil": 12, "quantit": [2, 8, 11, 23], "quantiti": [1, 2, 4, 7, 8, 9, 11, 12, 13, 14, 16, 18, 20, 23], "quantum": [6, 14, 22, 23], "quartil": 2, "quench": 7, "queri": 11, "question": [2, 7, 8, 11, 13, 14, 15, 21, 23], "qugan": 6, "quick": [6, 20], "quickli": [3, 5, 11, 13, 15], "quit": [0, 3, 7, 8, 11, 12, 14], "quot": 6, "r": [0, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20], "r2": [2, 7, 8, 23], "r2_score": [2, 23], "r2score": [2, 23], "r_1": 11, "r_2": 11, "r_j": 11, "r_m": 11, "rad": 2, "radial": [2, 10, 14], "radioact": 20, "radiu": [2, 3], "rain": 11, "ramp": 3, "ran0": 20, "ran1": 20, "ran2": 20, "ran3": 20, "rand": [0, 2, 6, 7, 8, 11, 12, 15, 18, 23], "randint": [8, 11, 15], "randn": [0, 2, 3, 4, 7, 8, 11, 13, 15, 23], "random": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 15, 16, 17, 18, 23], "random_forest_model": 12, "random_index": 15, "random_indic": [3, 5], "random_st": [2, 9, 10, 11, 12, 13], "randomforestclassifi": 12, "randomli": [3, 8, 11, 15, 16], "rang": [2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 18, 20, 23], "rangl": [2, 8, 13, 20, 23], "rangle_x": 20, "rank": 7, "rankdir": 6, "raphson": [3, 10, 15], "rapidli": 2, "rare": [3, 15], "raschka": 23, "rasckha": 23, "rate": [2, 3, 4, 5, 6, 10, 11, 12, 14, 15], "rather": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "ratio": [6, 9, 11, 12, 13], "rational": [2, 23], "ravel": [7, 8, 9, 10, 11, 12, 13, 15, 18], "raw": 5, "rbf": [10, 13, 14], "rbf_kernel_svm_clf": 10, "rbf_pca": 13, "rc": [2, 20], "rcond": [2, 23], "rcparam": [2, 3, 5, 9, 10, 11, 12, 20, 23], "re": [0, 4, 6, 15], "reach": [3, 6, 7, 8, 11, 12, 14, 15, 16, 23], "read": [1, 2, 4, 5, 6, 7, 8, 9, 10, 13, 14, 18, 20, 22], "read_csv": [2, 8, 9, 11], "read_fwf": [2, 23], "reader": [2, 8, 18, 20, 23], "readi": [2, 3, 7, 8, 10, 12, 13, 14, 18, 23], "readili": 3, "readm": 0, "readthedoc": 17, "real": [1, 2, 3, 6, 9, 12, 13, 14, 18], "real_loss": 6, "real_output": 6, "realist": [10, 23], "realiti": 20, "realiz": [3, 14], "realli": [2, 3, 23], "rearrang": 15, "reason": [2, 3, 5, 6, 12, 15, 22, 23], "reassign": 3, "recal": [7, 8, 11, 12, 13, 14, 18, 20, 23], "recast": 5, "receiv": [3, 5, 12, 14, 20], "recent": [2, 8, 15, 22], "recept": [5, 14], "receptive_field": 5, "recip": [2, 8, 9, 18, 23], "reciproc": 7, "recogn": [2, 6, 7, 12, 23], "recognit": [2, 3, 5, 14, 22, 23], "recommen": 23, "recommend": [0, 2, 4, 5, 6, 7, 8, 10, 15, 17, 18, 22], "reconsid": 11, "reconstruct": 13, "record": [12, 19, 21, 23], "recreat": 0, "rectangl": [11, 15], "rectangular": 7, "rectifi": [3, 5, 14], "recur": [2, 17, 23], "recurr": [2, 3, 17, 23], "recurs": [11, 17, 18, 23], "red": [2, 5, 6, 8, 10, 11], "redefin": [2, 12, 23], "reduc": [3, 5, 7, 8, 11, 12, 13, 15, 23], "reduct": [2, 12, 13, 17, 20, 23], "refer": [2, 3, 4, 5, 7, 8, 13, 14, 15, 16, 18, 22, 23], "referenc": 4, "refin": 14, "refit": 8, "reflect": [2, 3, 6, 7, 20, 23], "refresh": [17, 23], "refreshprogrammingskil": 23, "reg": [12, 13], "regard": [3, 11, 15], "regardless": [1, 14], "region": [5, 6, 8, 11, 14], "regist": [8, 20], "reglasso": 7, "regr_1": [2, 11], "regr_2": [2, 11], "regr_3": [2, 11], "regress": [1, 3, 10, 13, 14, 17, 18], "regressor": [2, 9, 12], "regridg": [2, 7, 8], "regular": [2, 5, 6, 7, 8, 9, 11, 15, 21, 23], "regularli": 0, "reilli": [2, 22, 23], "reinforc": [2, 10, 17, 23], "reiter": 3, "reject": 9, "rel": [2, 6, 8, 9, 11, 14, 15, 20, 23], "relat": [2, 3, 5, 6, 7, 13, 15, 16, 18, 20, 23], "relationship": [2, 6, 11, 23], "relativeerror": [2, 23], "releas": [3, 17, 23], "relev": [2, 3, 7, 9, 13, 17, 20, 23], "reli": [2, 8, 10], "reliabl": [9, 20], "relu": [5, 6, 23], "remain": [3, 4, 6, 8, 14, 18, 20], "remaind": 20, "reman": 4, "remark": 3, "rememb": [2, 10, 15, 18, 23], "remind": [2, 7, 13, 15, 18, 20], "remot": 0, "remov": [2, 6, 7, 8], "renam": 0, "render": [2, 23], "reorder": [7, 9], "reorgan": [2, 23], "repeat": [2, 3, 5, 6, 7, 8, 11, 12, 13, 15, 16, 18, 20, 23], "repeated": 23, "repeatedli": [2, 8, 12, 15], "repet": 5, "repetit": [8, 23], "rephras": 15, "replac": [2, 3, 5, 6, 7, 8, 12, 14, 16, 17, 23], "replica": 8, "repo": 0, "report": 23, "repositori": [2, 6, 23], "repres": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 20, 23], "represent": [2, 3, 5, 8, 20, 23], "representd": 5, "reproduc": [0, 1, 2, 7, 8, 11, 14, 17, 20, 23], "repuls": [2, 23], "request": [2, 15], "requir": [0, 2, 3, 5, 6, 7, 8, 10, 11, 13, 14, 15, 18, 23], "res1": 4, "res2": 4, "res3": 4, "res_analyt": 4, "res_analytical1": 4, "res_analytical2": 4, "res_analytical3": 4, "resaml": 8, "resampl": [2, 9, 12, 17, 23], "rescal": [2, 13, 14], "rescu": 7, "reseach": 8, "research": [2, 6, 15, 17, 22, 23], "resembl": [8, 20], "reserv": [3, 7, 8, 20], "reshap": [2, 3, 4, 5, 6, 8, 10, 11, 12, 18, 23], "residenti": 2, "residu": [2, 7, 15, 23], "resiz": 7, "resourc": 23, "respect": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 16, 20, 23], "respond": 14, "respons": [2, 9, 11, 14, 23], "rest": [2, 7], "restat": [2, 14, 23], "restor": 6, "restored_discrimin": 6, "restored_gener": 6, "restrict": [2, 5, 11, 14, 23], "result": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "retail": 2, "retain": [7, 8], "return": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 18, 20, 23], "return_data": 16, "return_sequ": 6, "return_x_i": 11, "reus": [3, 5, 8], "reveal": [2, 14, 23], "revers": [3, 18], "review": [17, 18], "revisit": 16, "revolut": 23, "reward": [2, 6, 23], "rewrit": [1, 2, 5, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20], "rewritten": [4, 8, 10, 12, 20], "rewrot": 15, "rf": 12, "rgb": 5, "rgoj5yh7evk": 17, "rh": 8, "rho": [2, 12, 15], "rho_1": 12, "rho_2": 12, "rho_m": 12, "rich": [2, 23], "ride": 11, "rideclass": 11, "ridedata": 11, "ridg": [9, 13, 15, 17, 23], "ridge_sk": 8, "ridgebeta": 7, "right": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 23], "right_sid": 4, "rightarrow": [2, 3, 7, 8, 10, 13, 14, 15, 20, 23], "rigor": [2, 23], "ring": 8, "rise": [2, 23], "risk": [2, 15, 23], "rival": 6, "river": 2, "rlm": 23, "rm": [2, 20], "rmse": 2, "rmsporp": 15, "rmsprop": [3, 5, 6, 15], "rnd_clf": 12, "rng": 20, "rnn": [6, 14], "rnn1": 6, "rnn2": 6, "rnn_2layer": 6, "rnn_input": 6, "rnn_output": 6, "rnn_train": 6, "rntrick1": 20, "rntrick2": 20, "rntrick3": 20, "rntrick4": 20, "ro": [2, 15, 23], "robert": 22, "robust": [2, 23], "robustscal": 2, "roc": [9, 12], "role": [2, 4, 7, 8, 10, 17, 23], "roll": 8, "room": [2, 21, 23], "root": [0, 2, 7, 11, 15, 20], "rot": 23, "rotat": [3, 10, 11, 12], "rotation_matrix": 11, "roughli": [3, 5], "round": [2, 9, 11, 15], "routin": [15, 18, 23], "row": [1, 2, 3, 4, 7, 8, 11, 13, 18, 23], "rr": 7, "rrr": 7, "rug": 15, "rule": [2, 3, 7, 8, 15, 23], "run": [0, 2, 3, 4, 6, 7, 8, 10, 11, 13, 15, 17, 23], "runtim": [0, 3, 8, 16], "rust": [2, 17, 18, 23], "rvert": 3, "rvert_2": 3, "s_": [5, 8], "s_1": 8, "s_i": [8, 9], "s_j": 8, "s_k": 8, "saddl": 15, "sai": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 20, 23], "said": [8, 11, 15], "sake": [2, 7, 9, 13, 23], "sale": [2, 23], "sam": 23, "same": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 16, 18, 20, 23], "samm": 12, "sampl": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 16, 17, 18, 20, 23], "sample_vari": 16, "sampleexptvari": 20, "samwis": 23, "sastri": 13, "satisfactori": [2, 23], "satisfi": [3, 4, 5, 8, 10, 15, 18, 20], "satur": [3, 8], "save": [2, 6, 8, 9, 11, 15, 23], "save_fig": [2, 8, 9, 11, 12, 23], "savefig": [2, 6, 8, 9, 11, 20, 23], "savetxt": 6, "saw": 7, "scalabl": 12, "scalar": [4, 7, 8, 12], "scale": [2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 21, 23], "scale_mean": 6, "scale_std": 6, "scaler": [2, 9, 10, 11, 12, 13], "scan": [7, 9], "scari": 7, "scatter": [0, 2, 3, 8, 9, 10, 11, 16, 23], "scenario": [8, 15], "schedul": 15, "scheme": [3, 15], "schrage": 20, "sch\u00f8yen": 8, "scienc": [2, 3, 12, 14, 15, 17, 19, 20, 21, 22], "scientif": [2, 17, 23], "scientist": [2, 23], "scikit": [0, 1, 5, 7, 8, 10, 11, 12, 15, 17, 18, 22], "scikit_learn": 2, "scikitlearn": 23, "scikitplot": [9, 12], "scipi": [2, 5, 7, 8, 15, 17, 18, 23], "scl": 8, "scm": 0, "score": [0, 1, 2, 3, 5, 8, 9, 11, 12, 13, 21, 23], "scores_kfold": 8, "scratch": [1, 3, 15], "sdg": 15, "seaborn": [2, 3, 5, 8, 9, 23], "seamless": [2, 17, 23], "search": [0, 2, 3, 5, 7, 11, 15, 23], "sebastian": 23, "sebastianraschka": 23, "sec": 8, "second": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 16, 17, 18, 20, 21, 23], "secondeigvector": 13, "secondli": 14, "section": [1, 6, 13, 18, 20], "sector": 2, "see": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 18, 20, 23], "seed": [2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 15, 16, 20, 23], "seed_imag": 6, "seek": [3, 4, 10], "seem": [3, 5, 6], "seemingli": [2, 23], "seen": [2, 3, 5, 7, 12, 14, 20], "segment": 15, "seismic": 8, "seldomli": [2, 23], "select": [0, 3, 7, 8, 10, 11, 12, 13, 19, 20, 21, 22, 23], "selevet": 0, "self": [3, 7], "sell": 6, "semest": [9, 19], "semi": [10, 15], "semilogx": 8, "send": [7, 14, 15, 21, 23], "senior": [19, 21], "sens": [2, 6, 8, 10, 23], "sensibl": 5, "sensit": [2, 7, 8, 11, 15, 23], "sent": 4, "sentenc": [6, 14], "separ": [2, 3, 4, 6, 8, 10, 11, 14, 16, 17, 20, 23], "septemb": 23, "sequenc": [5, 6, 9, 11, 12, 14, 15, 17, 18, 20, 23], "sequenti": [3, 5, 6, 12, 14, 20], "seri": [2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 18, 23], "serif": [2, 9, 20, 23], "serv": [2, 3, 4, 5, 7, 9, 15, 22, 23], "session": [0, 3, 19, 21, 23], "set": [1, 3, 6, 7, 8, 9, 10, 12, 13, 15, 16, 17, 18, 20, 21], "set_major_formatt": 8, "set_major_loc": 8, "set_tick": [3, 10], "set_ticklabel": 3, "set_titl": [2, 3, 4, 5, 9, 14, 16, 23], "set_xlabel": [2, 3, 4, 5, 9, 14, 23], "set_xlim": [9, 14], "set_xticklabel": 3, "set_ylabel": [2, 3, 4, 5, 9, 23], "set_ylim": [9, 14], "set_ytick": 9, "set_yticklabel": [3, 8], "set_zlim": 8, "seth": 6, "setminu": 8, "setosa": [10, 11], "setosa_or_versicolor": 10, "setp": 8, "setup": [3, 6, 8, 10, 17, 23], "sever": [1, 2, 5, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23], "sgd": [3, 5], "sgd_clf": 10, "sgdclassifi": 10, "sgdreg": 15, "sgdregressor": 15, "sgn": 7, "shallow": 15, "shape": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 23], "share": [0, 3, 5, 23], "shareabl": 0, "she": 9, "shift": [0, 3, 8, 14, 20], "ship": 5, "shire": 23, "short": [6, 7], "shortcom": 15, "shorten": 6, "shorter": 20, "shorthand": 23, "shortli": [18, 23], "should": [0, 2, 4, 5, 7, 8, 10, 11, 13, 14, 18, 20, 23], "show": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "show_shap": 6, "shown": [2, 6, 7, 10, 14, 15, 18], "shrink": [5, 7, 8, 10, 13], "shrinkag": [7, 8], "shrunk": 13, "shuffl": [2, 3, 6, 8, 15], "side": [2, 4, 7, 10, 14, 15, 18, 23], "sigh": [17, 23], "sigma": [2, 3, 7, 8, 9, 12, 13, 14, 15, 18, 20, 23], "sigma0": 20, "sigma1": 20, "sigma2": 20, "sigma_": [7, 18, 23], "sigma_0": 7, "sigma_1": 7, "sigma_2": 7, "sigma_fn": [9, 14], "sigma_i": [2, 7, 23], "sigma_j": 7, "sigma_m": [8, 20], "sigma_n": [13, 20], "sigma_t": 15, "sigma_x": 20, "sigmoid": [3, 4, 6, 9, 10, 12, 14], "sigmundson": 8, "sign": [3, 4, 9, 10, 12, 20, 21], "signal": [3, 5, 12, 14], "signifi": 6, "signific": 3, "significantli": [3, 15, 20], "sim": [6, 7, 8, 15, 20], "similar": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 23], "similarli": [2, 3, 5, 7, 10, 12, 15, 20, 23], "simpl": [1, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20], "simplepredict": 12, "simpler": [1, 2, 3, 7, 8, 9, 15, 17, 23], "simplernn": 6, "simplest": [2, 3, 5, 6, 11, 12, 14, 16, 23], "simpletre": 12, "simpli": [2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 17, 18, 20, 23], "simplic": [4, 7, 8, 9, 10, 11, 12, 13, 14, 16], "simplicti": 7, "simplifi": [2, 8, 11, 17, 23], "simplist": [5, 8, 20], "simul": 8, "simultan": 8, "sin": [2, 3, 4, 5, 6, 11, 14, 15, 18, 23], "sinc": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 22, 23], "sine": [5, 14], "singl": [2, 3, 4, 5, 7, 8, 9, 10, 11, 14, 15, 18, 20, 23], "singular": [2, 8, 15, 18, 23], "sinusoid": 5, "site": [2, 19, 23], "situat": [2, 6, 7, 9, 15, 20, 23], "six": [5, 20], "size": [2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 18, 20, 23], "sketch": 12, "ski": 11, "skill": 2, "skip": 13, "skl": [2, 8, 23], "sklearn": [0, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 23], "skplt": [9, 12], "sl": 8, "slack": 10, "slice": [4, 18, 23], "slide": [1, 2, 5, 20, 23], "slight": [8, 15], "slightli": [3, 4, 5, 7, 8, 9, 12, 20], "slope": [10, 13, 14], "slow": [2, 4, 10, 15], "slower": [7, 18, 23], "slowest": 18, "slowli": 14, "slp": 3, "small": [2, 3, 4, 5, 7, 8, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23], "smaller": [2, 3, 4, 7, 8, 10, 11, 13, 15, 20, 23], "smallest": [2, 6, 16, 23], "smallest_row_index": 16, "smooth": [2, 5, 8, 15, 23], "sn": [2, 3, 5, 8, 9, 23], "sne": 13, "so": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "soar": 8, "social": 2, "soft": [3, 9, 12, 14], "soften": 10, "softmax": [5, 9], "softwar": [2, 10, 17, 18], "sol": 10, "sole": [2, 8, 23], "solid": [2, 9], "solut": [2, 3, 4, 5, 7, 8, 10, 12, 13, 15, 18, 20, 23], "soluton": 4, "solv": [1, 2, 3, 5, 7, 8, 10, 12, 13, 14, 15, 18, 23], "solve_expdec": 4, "solve_ode_deep_neural_network": 4, "solve_ode_neural_network": 4, "solve_pde_deep_neural_network": 4, "solveod": 4, "solveode_popul": 4, "solver": [4, 9, 10, 11, 12, 18, 23], "some": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 20, 23], "some_model": 8, "somehow": 6, "someon": 1, "someth": [0, 2, 3, 5, 6, 9, 11, 13, 20, 23], "sometim": [2, 3, 13, 14, 15, 16], "soon": [18, 21], "sophist": [2, 23], "sopt": 15, "sort": [7, 8, 11, 13, 20], "sound": [5, 7], "sourc": [2, 3, 5, 8, 17, 18, 20, 23], "space": [2, 3, 6, 7, 10, 11, 13, 14, 15, 16, 20], "span": [2, 5, 7, 11, 13, 18, 23], "spare": 3, "spars": [5, 8, 18, 23], "sparse_mtx": [18, 23], "sparsecategoricalcrossentropi": 5, "sparsiti": 12, "spatial": [3, 4, 5, 14], "speak": 20, "special": [8, 9, 12, 14, 15, 18, 20, 23], "specif": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 17, 18, 20, 22, 23], "specifi": [2, 5, 7, 8, 9, 11, 13, 15, 16, 20, 23], "specifici": [2, 12, 23], "spectacular": 5, "spectral": 3, "speech": [2, 3, 5, 6, 14], "speed": [3, 4, 6, 15], "spend": [1, 20], "sphere": 2, "spin": 8, "spite": 2, "spline": 10, "split": [1, 3, 5, 6, 7, 8, 10, 11, 12, 13, 16, 20, 23], "splite": 2, "splitter": [3, 12], "spontan": 20, "spot": 5, "spread": [2, 13, 20, 23], "springer": [22, 23], "spuriou": 15, "sqrsignal": 5, "sqrt": [2, 5, 6, 7, 8, 10, 12, 13, 15, 20], "squar": [0, 3, 4, 5, 6, 9, 10, 11, 13, 15, 16, 17, 18, 20], "squarederror": 12, "squaredeuclidean": 16, "squash": 14, "srtm": 8, "srtm_data_norway_1": 8, "stabil": 7, "stabl": [1, 2, 6, 7, 8, 11, 17, 23], "stack": [5, 6], "stage": [0, 7, 15], "stai": [2, 4, 6, 7, 13, 23], "stand": [2, 7, 11, 14, 23], "standard": [2, 3, 6, 7, 8, 9, 10, 12, 14, 18, 20, 23], "standardscal": [2, 8, 9, 10, 11, 12, 13], "stanford": 15, "start": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23], "start_tim": 16, "stat": 8, "state": [3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 20, 23], "statement": [2, 9, 18, 23], "statist": [2, 3, 5, 6, 9, 11, 12, 13, 14, 15, 16, 18, 22], "statu": [0, 2, 9, 23], "stavang": 8, "std": [2, 6, 8, 23], "steep": 15, "step": [0, 2, 3, 4, 6, 8, 9, 11, 12, 13, 14, 15, 16, 18, 23], "step_fn": [9, 14], "step_length": 15, "steps_list": 11, "stereo": 5, "still": [2, 4, 5, 7, 8, 13, 15, 20], "stimuli": 14, "stk": [22, 23], "stk2100": [22, 23], "stk3155": [0, 19, 21], "stk4021": [22, 23], "stk4051": [22, 23], "stk4155": [19, 21], "stk5000": 22, "stochast": [2, 3, 7, 8, 10, 13, 14, 23], "stock": 6, "stoke": 14, "stone": [2, 9], "stop": [3, 6, 11, 15, 16], "storag": 7, "store": [2, 3, 4, 5, 8, 13, 15, 20, 23], "storehaug": [21, 23], "str": [3, 5, 6], "straight": [2, 8, 10, 15, 23], "straightforward": [2, 4, 5, 7, 8, 10, 11, 12, 15, 18, 23], "strategi": [2, 3, 11, 23], "stratifi": 8, "strength": [2, 7, 16], "stretch": 13, "strict": [10, 15], "strictli": [10, 15], "stride": [6, 18], "strike": 8, "string": 3, "stroke": 9, "strong": [5, 8, 11, 12, 14, 18, 20], "strongli": [0, 2, 10, 17, 18], "stronli": 2, "structur": [2, 3, 4, 5, 8, 11, 12, 14, 17, 23], "stuck": [3, 15], "student": [0, 2, 19, 21, 22, 23], "studi": [2, 5, 6, 7, 8, 9, 10, 13, 14, 15, 17, 22, 23], "studier": 22, "style": [2, 9, 11, 18, 23], "st\u00f8land": 21, "sub": [11, 14], "subdivid": [2, 18, 23], "subfield": 2, "subject": [8, 10, 20], "submit": 23, "subplot": [2, 3, 5, 6, 8, 9, 10, 11, 12, 16, 23], "subplots_adjust": [10, 20], "subprogram": [18, 23], "subract": 2, "subroutin": [2, 23], "subscript": 3, "subsequ": [3, 6, 7, 8, 14, 18, 20], "subset": [3, 8, 11, 14, 15, 17, 23], "subspac": [2, 10, 13], "substanti": [11, 12], "substep": 13, "substitut": [1, 5, 8, 14, 18], "subsubset": 11, "subtask": 8, "subtl": 3, "subtract": [2, 6, 7, 8, 13, 15, 18, 20], "subtre": 11, "succeed": [2, 6, 23], "success": [5, 9, 11, 15, 20], "successfulli": [6, 11], "sudo": [2, 17, 23], "suffer": [2, 3, 4, 7, 12, 23], "suffici": [3, 8, 10, 13, 15], "suggest": [3, 15, 22], "suit": [10, 14], "suitabl": [0, 2, 20], "sum": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "sum_": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "sum_i": [2, 4, 7, 8, 10, 15], "sum_j": 8, "sum_ja_": 2, "sum_k": [8, 10, 14, 18], "sum_logist": 15, "sum_m": 5, "sum_n": 5, "sum_nx_": 5, "summar": [7, 8, 11], "summari": [3, 5, 6, 12, 19], "summat": [1, 2, 5], "sunni": 11, "super": 7, "superfici": 5, "superscript": [3, 14], "supervis": [2, 7, 8, 9, 11, 14, 17, 23], "supplement": 9, "support": [2, 3, 11, 12, 13, 15, 17, 23], "suppos": [2, 7, 8, 9, 10, 12, 13, 14, 15, 18, 23], "suppress": [7, 15], "sure": [1, 2, 3, 6, 8], "surf": 8, "surfac": [2, 8, 23], "surpass": 8, "surpris": [2, 23], "surround": [5, 17], "survei": [2, 7, 8, 23], "svc": [10, 11, 12], "svd": [2, 8, 13, 23], "svdinv": 7, "svm": [10, 11, 12, 13], "svm_clf": [10, 12], "swath": 7, "switch": 2, "sy": 15, "symbol": [3, 7, 13, 15, 17, 20, 23], "symmeteri": 3, "symmetr": [2, 7, 10, 13, 14, 15, 18, 23], "symmetri": 8, "sympi": [2, 17, 23], "synonim": 20, "syntax": 15, "system": [0, 2, 3, 5, 6, 8, 9, 11, 12, 14, 15, 17, 18, 23], "systemat": [6, 8], "t": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "t0": [5, 8, 15], "t1": [4, 15], "t2": 4, "t3": 4, "t_": 4, "t_0": [4, 11, 15], "t_1": 15, "t_b": 12, "t_i": [3, 4, 7, 14], "t_j": 14, "t_k": 11, "tabl": [11, 20, 21, 23], "tabul": [2, 23], "tabular": 23, "tackl": 6, "tag": [4, 5, 6, 7, 8, 9, 14, 15, 16, 18, 20], "taht": [2, 23], "tail": 20, "tailor": [4, 10, 13, 23], "taiwan": [2, 23], "take": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23], "taken": [2, 3, 5, 8, 12, 15, 18], "tan": 5, "tangent": [3, 6, 14, 15], "tanh": [3, 6, 9, 10, 14], "target": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 23], "target_nam": 11, "task": [2, 3, 5, 8, 11, 13, 14, 16, 23], "tau": [5, 7, 20], "taught": 23, "tax": 2, "taylor": [4, 15], "taylornr": 15, "tc": 10, "teach": [0, 19, 23], "team": 3, "teaser": 2, "technic": [2, 7, 8, 15], "techniqu": [2, 3, 10, 12, 15, 17, 20, 22, 23], "technologi": [2, 3], "tell": [1, 2, 6, 8, 12, 13, 15, 20], "temp": 3, "temp1": 3, "temp2": 3, "temperatur": [2, 11, 23], "temporarili": 3, "ten": [5, 23], "tend": [5, 7, 8, 10, 11, 12, 14, 15, 16], "tendenc": [2, 23], "tension": 8, "tensor": 5, "tensorflow": [2, 4, 6, 10, 16, 17, 18, 22, 23], "term": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 23], "term1": [7, 8, 13], "term2": [7, 8, 13], "term3": [7, 8, 13], "term4": [7, 8, 13], "termin": [0, 2, 6, 7, 11, 12, 15], "terrain": 8, "terrain1": 8, "test": [1, 5, 6, 7, 8, 9, 10, 11, 12, 15, 18, 20, 23], "test_acc": 5, "test_accuraci": [3, 5], "test_error": 8, "test_imag": [5, 6], "test_ind": 8, "test_input": 6, "test_label": [5, 6], "test_loss": 5, "test_pr": 3, "test_predict": 3, "test_rnn": 6, "test_scor": [9, 12], "test_siz": [0, 2, 3, 5, 7, 8, 12], "test_split": 11, "testerror": [2, 8], "testi": 6, "testpredict": 6, "testx": 6, "text": [2, 3, 4, 6, 7, 10, 11, 13, 15, 18, 20, 22], "textbook": 1, "textual": 11, "textur": 3, "tf": [3, 5, 6, 15, 16], "th": [2, 3, 4, 7, 8, 9, 11, 14, 15, 16, 18, 20, 23], "than": [2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 17, 20, 23], "thank": [6, 8], "theano": [3, 17, 23], "thei": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 23], "them": [2, 3, 5, 6, 8, 10, 11, 12, 13, 14, 15, 18, 23], "theme": [0, 2, 23], "themselv": [2, 20, 23], "thenc": 8, "theorem": [4, 8, 9], "theoret": [2, 6, 12], "theori": [2, 3, 5, 10, 11, 14, 15, 17, 22, 23], "thereaft": [2, 7, 8, 13, 14, 18, 23], "therebi": [2, 7, 9, 13, 23], "therefor": [2, 3, 4, 5, 6, 8, 9, 10, 13, 15, 20, 23], "therein": 13, "thereof": [2, 8, 15, 23], "theta": [3, 6, 15, 20, 23], "theta_": [3, 15, 23], "theta_0": 23, "theta_0x_": 23, "theta_1": 23, "theta_1x_": 23, "theta_1x_0": 23, "theta_1x_1": 23, "theta_1x_2": 23, "theta_2": 23, "theta_2x_": 23, "theta_2x_0": 23, "theta_2x_1": 23, "theta_2x_2": 23, "theta_i": [3, 23], "theta_j": 23, "theta_linreg": 15, "theta_t": 15, "thi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22], "thing": [0, 1, 2, 3, 4, 6, 7, 9, 11, 20, 23], "think": [2, 3, 5, 6, 8, 11, 14, 15, 16, 20, 23], "third": [2, 5, 8, 15, 21, 23], "thirti": 9, "thorughout": 23, "those": [2, 5, 7, 8, 10, 11, 12, 13, 18, 23], "though": [1, 3, 4, 5, 6, 15, 18, 20], "thought": [8, 16, 20], "thousand": [2, 3], "three": [2, 3, 5, 7, 8, 10, 11, 14, 18, 19, 20, 21, 23], "threshold": [3, 5, 11, 12, 13, 14, 15], "through": [0, 2, 3, 4, 5, 6, 7, 8, 10, 13, 14, 15, 16, 17, 18, 20, 23], "throughout": [0, 2, 6, 7, 16, 17, 18, 20, 23], "throw": [5, 8, 20], "thu": [2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 21, 23], "thumb": [2, 8], "tibshirani": [8, 22, 23], "tick_param": 8, "ticker": [8, 15, 20], "tif": 8, "tight_layout": [3, 9], "tightli": 13, "tild": [2, 7, 8, 9, 13, 20, 23], "till": [2, 6, 9, 10, 11, 12, 14, 18, 23], "time": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "timeit": 6, "timer": 6, "tini": 3, "tip": 5, "titl": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 15, 20, 23], "tmp": 15, "tn": [4, 5, 9], "to_categor": [3, 5, 6], "to_categorical_numpi": 3, "to_numer": [2, 8, 23], "todai": 5, "togeth": [2, 5, 8, 10, 13, 15, 17, 23], "toi": 16, "told": 15, "toler": [4, 16], "tolist": 6, "tomographi": 14, "too": [2, 4, 6, 7, 8, 11, 13, 15, 20, 22], "took": [10, 23], "tool": [0, 2, 3, 5, 8, 15, 17], "toolbox": 10, "top": [2, 5, 7, 8, 11, 12, 17, 23], "topic": [2, 7, 8, 9, 10, 17], "topolog": [5, 14], "topologi": [3, 14], "torkjellsdatt": [21, 23], "toss": [12, 20], "total": [2, 3, 4, 5, 6, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 21, 23], "total_loss": 6, "totalclustervari": 16, "totalscatt": 16, "toward": [0, 3, 4, 9, 14, 15], "town": 2, "tp": [6, 9], "tpng": 11, "tpu": [15, 17, 23], "tqdm": 8, "track": [0, 5, 15, 16, 18], "tract": 2, "tractabl": [2, 23], "trade": [7, 11], "tradeoff": [2, 7, 23], "tradit": [2, 3, 6, 8, 23], "train": [1, 4, 5, 7, 8, 10, 11, 12, 13, 14, 15], "train_accuraci": [2, 3, 5, 23], "train_dataset": 6, "train_end": [2, 3], "train_error": 8, "train_imag": [5, 6], "train_ind": 8, "train_label": [5, 6], "train_pr": 3, "train_siz": [2, 3, 5], "train_step": 6, "train_test_split": [0, 1, 2, 3, 5, 7, 8, 9, 11, 12, 13, 23], "train_test_split_numpi": [2, 3], "trainable_vari": 6, "trained_model": 8, "trainerror": 2, "traini": 6, "training_checkpoint": 6, "training_dataset": 6, "training_gradi": 15, "trainingerror": 8, "trainpredict": 6, "trainscor": 6, "trainx": 6, "trait": [2, 23], "trajectori": 6, "transfer": [11, 23], "transform": [2, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 23], "transit": [8, 14], "translat": [3, 6, 8, 12, 23], "transpos": [3, 7, 13, 18], "travers": [2, 7], "treat": [2, 3, 5, 8, 14, 15, 20, 23], "tree": [2, 3, 17, 23], "tree_clf": [11, 12], "tree_clf_": 11, "tree_clf_sr": 11, "tree_reg": 11, "tree_reg1": 11, "tree_reg2": 11, "trend": 20, "treue": 9, "trevor": 22, "tri": [1, 4, 5, 6, 11, 15], "triain": 2, "trial": [2, 4, 6, 8, 15, 20, 23], "triangl": 15, "triangular": 18, "trick": [5, 6, 10, 13, 15, 20], "trickier": 20, "tridiagon": 18, "trillion": 17, "trivial": [2, 3, 7, 13, 20, 23], "troubl": [0, 2, 10, 14], "truck": 5, "true": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 23], "true_beta": 8, "true_fun": 8, "truli": 23, "try": [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 20, 23], "tucker": 10, "tuesdai": [21, 23], "tumor": [9, 11], "tumour": 9, "tunabl": 3, "tune": [6, 11, 15, 18, 23], "turn": [2, 3, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "tutori": [3, 6], "tv": 4, "tveito": 4, "tweak": [3, 6, 12, 20], "twice": 15, "twist": 13, "two": [0, 2, 3, 4, 6, 7, 8, 9, 11, 12, 13, 14, 15, 18, 19, 20, 22, 23], "tx": 15, "tx_1": 15, "txt": [0, 6], "ty": 15, "type": [2, 3, 5, 8, 10, 12, 15, 18, 20], "typic": [0, 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 14, 15, 20, 23], "u": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 22, 23], "u_": 18, "u_i": 14, "u_m": 12, "ua": [2, 23], "ubuntu": [2, 17, 23], "uci": 2, "uio": [0, 21, 22], "un": 16, "unabl": 0, "unari": [18, 23], "unbalanc": [8, 11], "unbias": [2, 7, 8, 23], "uncent": 8, "uncertainti": [2, 7, 23], "uncertitud": 20, "unchang": [3, 5], "uncorrel": [12, 20], "undefin": 7, "under": [2, 3, 7, 8, 12, 15, 17, 23], "underdetermin": [2, 23], "underfit": [3, 8], "underflowproblem": 7, "undergo": 7, "undergradu": [19, 21], "underli": [2, 3, 11, 15, 20, 23], "underset": [6, 16], "understand": [0, 2, 3, 5, 7, 8, 12, 15, 16, 17, 23], "understood": [10, 15], "undesir": 10, "undetermin": [7, 10], "undo": 6, "unexpect": 8, "unexpected": 20, "unfair": 8, "unfortun": [3, 10, 11, 12], "unicode_liter": [10, 11], "uniform": [2, 3, 7, 8, 13, 15, 20, 23], "uniformli": [15, 20], "unifrompdf": 20, "unimport": 15, "union": [7, 8], "uniqu": [2, 4, 8, 15, 16, 18, 23], "unique_cluster_label": 16, "unit": [2, 3, 5, 6, 7, 12, 14, 20, 23], "unitari": [7, 8, 18], "unitarili": [18, 23], "uniti": 20, "univari": 20, "univers": [2, 3, 4, 15, 17, 19, 21, 23], "unix": 3, "unknow": [2, 18, 23], "unknown": [2, 3, 5, 6, 7, 8, 10, 12, 15, 18, 23], "unknowwn": 14, "unlabel": 3, "unless": [2, 5, 8, 13, 15, 23], "unlik": [3, 5, 10, 15], "unnecessarili": 11, "unord": 5, "unravel": 3, "unrol": [5, 13], "unseen": [0, 2, 9, 11], "unstabl": 3, "unsupervis": [2, 3, 6, 14, 17, 23], "unsymmetr": [18, 23], "until": [3, 4, 6, 11, 14, 15, 16], "untouch": 2, "unusu": 14, "up": [1, 3, 5, 6, 7, 8, 10, 12, 13, 15, 16, 17, 18, 20, 21], "updat": [0, 3, 4, 12, 14, 15, 16], "uploa": 23, "upload": [0, 17, 22], "upon": [2, 3, 8, 9, 13, 18], "upper": [1, 2, 10, 11, 18], "uppercas": [18, 23], "upsampl": 6, "upscal": 6, "url": 23, "us": [0, 6, 7, 8, 10, 11, 12, 13, 14, 16, 18, 20, 22], "usag": [2, 10, 17, 23], "usd": 2, "usd10000": 2, "use_bia": 6, "usecol": [2, 23], "useless": 3, "user": [2, 3, 4, 6, 8, 9, 17, 18, 23], "usernam": 0, "usetex": 20, "usg": 8, "usr": 20, "usual": [2, 5, 6, 9, 14, 15, 16, 23], "ut": 7, "util": [3, 5, 6, 8, 9, 12, 16, 23], "ux": 18, "v": [0, 2, 4, 6, 7, 8, 13, 15, 17], "v0": 20, "v1": 20, "v2": 20, "v_0": 13, "va": 3, "vahid": 23, "val": 15, "val_accuraci": 5, "val_loss": 6, "vale": 4, "valid": [2, 3, 6, 9, 11, 12, 15, 17, 20, 23], "validation_data": 5, "validation_split": 6, "valu": [1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 23], "valuat": 11, "valy": 6, "van": [2, 23], "vandenbergh": [10, 15], "vandermond": [2, 23], "vanilla": [2, 8, 13, 16], "vanish": [3, 6, 15, 20], "var": [7, 8, 12, 13, 20], "var_x": 20, "varabl": 10, "varepsilon": [7, 8], "varepsilon_": [7, 8], "varepsilon_i": [7, 8], "vari": [2, 3, 5, 7, 8, 12, 23], "variabl": [2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 23], "varianc": [2, 3, 7, 9, 11, 12, 13, 15, 16, 17, 18, 20, 23], "variance_i": [7, 13], "variance_x": [7, 13], "variant": [2, 3, 8, 10, 14, 15, 23], "variat": [5, 6, 13, 23], "varieti": [2, 5, 14, 17, 23], "variou": [1, 3, 5, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23], "varydimens": 6, "vastli": 5, "vaue": 3, "vault": 2, "vdot": [4, 15], "vec": 8, "vector": [2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 17], "vector_mean": 16, "ventur": [2, 10, 17, 23], "venv": 0, "verbos": [3, 5, 6], "veri": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 22, 23], "verifi": [5, 13, 18, 23], "versatil": [10, 23], "versicolor": [10, 11], "version": [0, 2, 5, 12, 15, 16, 17, 18, 20, 23], "versu": 3, "vert": [1, 2, 3, 7, 8, 9, 10, 11, 13, 15, 23], "vert_1": [7, 8], "vert_2": [7, 8, 13], "via": [2, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 20, 21, 23], "vidal": 13, "video": [2, 3, 14, 17, 19, 21, 23], "view": [3, 5, 7, 8, 14, 15, 20, 22, 23], "violat": 10, "virginica": 11, "viridi": [2, 3, 4, 5, 23], "virtual": 3, "viscos": 15, "viscou": 15, "visibl": 0, "vision": [2, 5], "visual": [2, 5, 13, 14, 17, 23], "visualis": 3, "visualstudio": [0, 1], "viz": [8, 10, 20], "vmap": 15, "vmax": [3, 8], "vmin": [3, 8], "voic": 5, "volum": [2, 5, 23], "vote": [12, 23], "voting_clf": 12, "votingclassifi": 12, "votingsimpl": 12, "vstack": [7, 13, 18, 20, 23], "vt": 7, "w": [2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 23], "w1": 10, "w2": [10, 13], "w3": 10, "w_": [3, 14], "w_1": [10, 18], "w_1x_": 10, "w_1x_1": 10, "w_2": [10, 18], "w_2x_": 10, "w_2x_2": 10, "w_3": 18, "w_4": 18, "w_hidden": 4, "w_i": [3, 4, 12], "w_ix_i": 14, "w_j": 18, "w_m": 18, "w_output": 4, "w_px_": 10, "w_px_p": 10, "wa": [2, 3, 5, 6, 7, 8, 9, 12, 13, 14, 16, 18, 23], "wai": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 23], "walk": 11, "walker": 20, "wang": [2, 23], "want": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23], "warn": [6, 23], "warrant": 8, "wast": 5, "watch": 17, "wave": 5, "wavelet": 10, "we": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22], "weak": [11, 12, 16], "weather": [3, 14], "web": [17, 19, 21, 23], "webpag": 23, "websit": [8, 18, 19, 23], "wedg": [10, 20], "wednesdai": [21, 23], "wee": 13, "week": [2, 7, 8, 9, 19, 21], "weekli": [0, 1, 17, 19, 21, 22, 23], "weekss": [], "weight": [2, 3, 4, 5, 8, 9, 11, 12, 14, 15, 20], "weigth": 4, "welcom": [0, 10, 17], "well": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18, 20, 22, 23], "went": 10, "were": [2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 20, 23], "wessel": [2, 23], "what": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20], "whatev": 5, "when": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "whenev": [0, 15, 20], "where": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "wherea": [8, 20], "wherein": [3, 14], "whether": [2, 5, 7, 9, 11, 20, 23], "which": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23], "whichev": [3, 5], "while": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 20, 23], "white": 11, "who": [0, 2], "whole": [3, 5, 6, 7, 11, 13, 15], "whose": [2, 8, 12, 20], "whow": 13, "why": [0, 1, 2, 3, 5, 8, 15], "wide": [2, 3, 5, 8, 9, 14, 17, 18, 23], "widehat": 8, "width": [2, 5, 10, 11, 23], "wieringen": [2, 23], "win": 12, "wind": 11, "wing": [21, 23], "winther": 4, "wiothout": 8, "wiscons": 9, "wisconsin": 12, "wisdom": 8, "wise": [2, 3, 7, 14, 15], "wish": [2, 4, 7, 9, 10, 13, 15, 16, 18, 23], "with_std": 2, "wither": 8, "within": [2, 4, 5, 6, 9, 11, 14, 15, 16, 20, 22, 23], "withinclust": 16, "without": [0, 2, 3, 7, 8, 10, 11, 13, 14, 15, 23], "won": [0, 2, 23], "wonder": 10, "word": [2, 3, 5, 6, 7, 8, 9, 16, 20, 23], "work": [0, 1, 2, 3, 6, 8, 9, 10, 11, 15, 17, 19, 20, 21, 23], "workshop": 23, "world": [1, 2, 10], "worldwid": [2, 23], "worri": 0, "wors": [2, 3, 5, 6, 8, 23], "worth": 11, "would": [1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "wrap": [8, 18, 23], "write": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 14, 15, 18, 23], "written": [1, 2, 4, 5, 7, 13, 14, 15, 17, 18, 20, 23], "wrong": [0, 3, 10], "wrongli": 12, "wrote": [7, 13], "wrt": [12, 15], "wth": [12, 15], "www": [17, 18, 22, 23], "wx_1": 10, "x": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "x0": 10, "x1": [6, 10, 11, 12, 15], "x1_exampl": 10, "x1d": 10, "x2": [10, 11, 12, 15], "x2d": [10, 13], "x2d_train": 13, "x2dsl": 13, "x3": 10, "x_": [2, 4, 5, 7, 8, 10, 12, 13, 15, 16, 18, 20, 23], "x_0": [2, 7, 13, 18, 23], "x_1": [2, 4, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 23], "x_2": [2, 4, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 23], "x_3": [10, 18, 20], "x_4": 18, "x_center": 13, "x_data": 3, "x_data_ful": 3, "x_hidden": 4, "x_i": [2, 3, 4, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "x_input": 4, "x_ix_": [2, 23], "x_iy_i": 10, "x_j": [1, 2, 4, 10, 11, 14, 20], "x_jy_j": 10, "x_k": [14, 16, 18, 20], "x_l": 20, "x_m": [8, 14, 18, 20], "x_n": [2, 4, 5, 8, 10, 13, 14, 15, 18, 20, 23], "x_new": [11, 12], "x_offset": 8, "x_output": 4, "x_p": [5, 9, 11], "x_poli": 11, "x_poly10": 11, "x_pred": 6, "x_prev": 4, "x_reduc": 13, "x_scale": 10, "x_small": 15, "x_test": [0, 1, 2, 3, 5, 7, 8, 9, 11, 12, 13], "x_test_own": 8, "x_test_scal": [2, 8, 9, 11, 12, 13], "x_tot": 6, "x_train": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 23], "x_train_mean": 8, "x_train_own": 8, "x_train_scal": [2, 8, 9, 11, 12, 13], "x_val": 3, "xarrai": [17, 23], "xavier": 3, "xbnew": 15, "xcode": [2, 17, 23], "xdclassiffierconfus": 12, "xdclassiffierroc": 12, "xg_clf": 12, "xgb": 12, "xgbclassifi": 12, "xgboost": 11, "xgboot": 12, "xgbregressor": 12, "xgparam": 12, "xgtree": 12, "xi": [10, 15], "xi_": 10, "xi_1": 10, "xi_i": 10, "xk": 10, "xla": [15, 17, 23], "xlabel": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 20, 23], "xlim": [8, 12], "xm": 11, "xmesh": 15, "xnew": [2, 15, 23], "xp": 20, "xpanda": 2, "xpd": [7, 13], "xplot": 2, "xscale": 2, "xsr": 11, "xt_x": 15, "xtest": 8, "xtick": [5, 8, 10, 11], "xtrain": 8, "xu": [2, 23], "xx": [2, 18, 23], "xy": [2, 8, 10, 18, 23], "xytext": 10, "xz": [18, 23], "y": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "y1": 6, "y2": 6, "y3": 6, "y_": [2, 3, 7, 8, 12, 13, 18, 23], "y_0": [2, 7, 13, 18, 23], "y_1": [2, 7, 10, 11, 13, 15, 18, 23], "y_1y_1": 10, "y_1y_1k": 10, "y_1y_2": 10, "y_1y_2k": 10, "y_1y_n": 10, "y_1y_nk": 10, "y_2": [2, 7, 10, 11, 13, 18, 23], "y_2y_1": 10, "y_2y_1k": 10, "y_2y_2": 10, "y_2y_2k": 10, "y_3": [2, 11, 18], "y_4": 18, "y_data": [2, 3, 7, 8, 23], "y_data_ful": 3, "y_decis": 10, "y_fit": 2, "y_i": [2, 3, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 23], "y_if_": 12, "y_ix_": [2, 23], "y_ix_i": [9, 10, 15], "y_iy_jk": 10, "y_j": [8, 10, 14], "y_k": 14, "y_m": 18, "y_model": [2, 6, 7, 8, 23], "y_n": [10, 15], "y_ny_1": 10, "y_ny_1k": 10, "y_ny_2": 10, "y_ny_2k": 10, "y_ny_n": 10, "y_ny_nk": 10, "y_offset": 8, "y_plot": 11, "y_pred": [2, 3, 6, 8, 9, 10, 11, 12], "y_pred1": 11, "y_pred2": 11, "y_pred_rf": 12, "y_pred_tre": 12, "y_proba": [9, 12], "y_scaler": 8, "y_test": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13], "y_test_onehot": 3, "y_test_predict": 2, "y_tot": 6, "y_train": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 23], "y_train_mean": 8, "y_train_onehot": 3, "y_train_predict": 2, "y_train_scal": 8, "y_val": 3, "ye": [5, 8, 9], "year": [2, 17, 23], "yet": [2, 3, 8, 10, 13, 15, 23], "yi": 15, "yield": [2, 4, 7, 8, 10, 12, 14, 15, 16, 18, 20, 23], "yk": 10, "ylabel": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 20, 23], "ylim": [5, 8], "ym": 11, "ymesh": 15, "yn": 2, "yo": [10, 11, 12], "yoshua": [3, 22], "you": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 20, 21, 22, 23], "young": 2, "your": [0, 3, 4, 6, 7, 8, 10, 13, 15, 17, 18, 23], "your_model_object": 1, "yourself": [13, 15, 23], "youtub": 17, "ypred": 8, "ypredict": [2, 15, 23], "ypredict2": 15, "ypredictlasso": 7, "ypredictol": [2, 7], "ypredictown": 8, "ypredictownridg": 8, "ypredictridg": [2, 7, 8], "ypredictskl": 8, "ytest": 8, "ytick": [5, 8, 10, 11], "ytild": [2, 8, 23], "ytildelasso": 7, "ytildenp": [2, 23], "ytildeol": [2, 7], "ytildeownridg": 8, "ytilderidg": [7, 8], "ytrain": 8, "yuxi": 23, "yx": [18, 23], "yy": [18, 23], "yz": [18, 23], "z": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 23], "z_": [3, 4, 14, 18, 23], "z_0": [18, 23], "z_1": [18, 23], "z_2": [18, 23], "z_c": 3, "z_h": 3, "z_hidden": 4, "z_i": [3, 14], "z_j": [3, 14], "z_k": 14, "z_m": 3, "z_mod": 11, "z_o": 3, "z_output": 4, "zaman": 20, "zaxi": 8, "zero": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "zeros_lik": 6, "zfill": 6, "zip": [6, 8], "zm_h": [2, 23], "zn": 2, "zone": 2, "zoom": 23, "zx": [18, 23], "zy": [18, 23], "zz": [18, 23], "\u00f8yvind": 8}, "titles": ["Exercises week 34", "Exercises week 35", "3. Linear Regression", "14. Building a Feed Forward Neural Network", "15. Solving Differential Equations  with Deep Learning", "16. Convolutional Neural Networks", "17. Recurrent neural networks: Overarching view", "4. Ridge and Lasso Regression", "5. Resampling Methods", "6. Logistic Regression", "8. Support Vector Machines, overarching aims", "9. Decision trees, overarching aims", "10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods", "11. Basic ideas of the Principal Component Analysis (PCA)", "13. Neural networks", "7. Optimization, the central part of any Machine Learning algortithm", "12. Clustering and Unsupervised Learning", "Applied Data Analysis and Machine Learning", "2. Linear Algebra, Handling of Arrays and more Python Features", "Course setting", "1. Elements of Probability Theory and Statistical Data Analysis", "Teachers and Grading", "Textbooks", "Week 34: Introduction to the course, Logistics and Practicalities"], "titleterms": {"": [10, 12], "1": [0, 1, 2], "2": [0, 1, 2, 23], "2023": 21, "3": [0, 1, 2], "34": [0, 23], "35": 1, "4": [0, 1, 2], "5": [1, 2], "A": [2, 3, 6, 10, 11, 23], "And": 23, "In": 21, "Ising": 8, "The": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 17, 23], "To": 23, "With": 6, "about": 23, "activ": [3, 14], "ad": [2, 8, 23], "adaboost": 12, "adagrad": 15, "adam": 15, "adapt": 12, "adjust": 3, "adversari": 6, "again": [5, 11], "ai": 23, "aim": [10, 11, 23], "aka": 23, "algebra": [18, 23], "algorithm": [11, 12, 13, 14, 23], "algortithm": 15, "all": 10, "an": [0, 2, 6, 12, 23], "analys": 7, "analysi": [2, 7, 8, 13, 17, 20, 23], "analyt": [1, 2], "ani": 15, "anoth": 11, "appli": 17, "approach": [2, 10, 16, 23], "approxim": 14, "architectur": 3, "arrai": [18, 23], "assist": 21, "autocorrel": 20, "autograd": [4, 15], "automat": 15, "back": [3, 13, 14], "background": 17, "bag": 12, "base": 15, "basic": [2, 7, 9, 11, 12, 13, 18], "batch": 3, "bay": 7, "befor": 13, "better": 10, "bia": 8, "binari": 3, "bind": 23, "bird": 12, "boost": 12, "bootstrap": [8, 12], "boston": 2, "breast": 3, "brief": 23, "bring": 14, "build": [3, 5, 11], "c": 23, "can": 23, "cancer": [3, 9, 11, 13], "cart": 11, "case": [10, 12, 20], "central": [15, 17, 20], "chain": 14, "chang": 12, "channel": 23, "chi": [2, 23], "choos": 3, "cifar01": 5, "classic": 13, "classif": [3, 11, 12], "classifi": 10, "clip": 3, "cluster": 16, "cnn": 5, "code": [0, 1, 2, 3, 4, 7, 11, 13, 14, 15, 16, 23], "collect": [3, 5], "commun": 23, "compar": [1, 4, 12], "complex": [2, 8], "complic": 8, "compon": 13, "comput": 11, "computerlab": 23, "con": 11, "concept": 20, "conjug": 15, "contn": 23, "convex": [10, 15], "convolut": [5, 14], "correl": 13, "cost": [3, 12], "cours": [17, 19, 22, 23], "covari": [7, 13, 20], "cover": 23, "creat": 1, "cross": 8, "cython": 23, "data": [0, 2, 3, 5, 8, 9, 11, 13, 17, 20, 23], "dataset": [3, 5], "david": 23, "deadlin": 23, "deadllin": 21, "decai": 4, "decis": [11, 12], "decomposit": [7, 13, 18], "deeep": 23, "deep": [3, 4, 23], "defin": [3, 23], "degre": 2, "deliver": [0, 1], "dens": 2, "deriv": [1, 7, 14], "descent": [4, 12, 15], "detail": [5, 23], "develop": 3, "diagon": 13, "differ": 10, "differenti": [4, 15], "diffus": 4, "dimension": [4, 5, 10], "disadvantag": 11, "discret": 20, "discrimin": 23, "distribut": [7, 20], "do": 3, "domain": 20, "down": 3, "dropout": 3, "element": [2, 20, 23], "elimin": 18, "energi": 23, "ensembl": 12, "entropi": 11, "environ": [0, 2], "equat": [2, 4, 14], "error": [2, 12, 23], "essenti": 23, "etc": 23, "euler": 4, "evalu": 3, "exampl": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 23], "exercis": [0, 1, 2, 8], "expect": 20, "experi": 20, "explor": 2, "exponenti": 4, "express": 1, "extrapol": 6, "extrem": [12, 23], "ey": 12, "fall": 21, "famili": [3, 23], "famou": 18, "featur": [1, 11, 18], "feed": [3, 14], "final": 14, "find": 1, "fine": 3, "first": [6, 14, 23], "fit": [0, 1, 2, 12, 23], "forc": 5, "forest": 12, "format": 23, "forward": [3, 4, 14], "foster": 23, "fourier": 5, "frank": 8, "freedom": 2, "frequentist": [2, 23], "from": [7, 12, 14, 23], "full": 4, "function": [2, 3, 8, 9, 10, 12, 13, 14, 15, 20, 23], "further": [5, 7], "gan": 6, "gaussian": 18, "gd": 15, "gener": [6, 11, 23], "geometr": 13, "gini": 11, "github": 0, "goal": [0, 1], "good": [2, 23], "grade": [21, 23], "gradient": [3, 4, 12, 15], "growth": 4, "ha": 17, "handl": [18, 23], "hidden": 4, "hous": 2, "how": 1, "hyperparamet": 3, "hyperplan": 10, "i": [2, 3, 23], "id3": 11, "idea": 13, "ii": 23, "implement": [1, 3], "implic": 7, "import": [7, 18, 23], "improv": 3, "includ": 15, "increment": 13, "index": 11, "inform": 21, "input": 4, "instal": [17, 23], "instructor": 21, "interpret": [7, 13, 23], "introduc": [13, 15], "introduct": [2, 8, 17, 18, 23], "invers": [7, 18], "iter": 12, "jax": 15, "julia": 23, "jungl": 12, "kera": [3, 5], "kernel": [10, 13], "lagrangian": 10, "lasso": [7, 8], "later": 7, "layer": [3, 4, 5, 14], "learn": [0, 1, 2, 3, 4, 13, 15, 16, 17, 23], "least": [1, 7, 8, 23], "lectur": 23, "level": 12, "librari": [17, 23], "likelihood": 9, "limit": [3, 15, 20], "linear": [0, 2, 10, 15, 18, 23], "link": [7, 13, 22], "logist": [9, 23], "lu": 18, "machin": [2, 10, 15, 17, 23], "main": [20, 23], "make": [2, 11, 12], "mani": [12, 14], "mass": 23, "materi": 23, "math": 7, "mathemat": [5, 7, 10], "matric": [7, 18, 23], "matrix": [1, 3, 7, 13, 14, 18, 23], "matter": 2, "mean": 2, "meet": [7, 12, 20, 23], "mercer": 10, "method": [8, 11, 12, 15, 23], "minim": 23, "ml": 23, "mlp": 14, "mnist": [5, 6], "model": [0, 2, 3, 6, 8, 14, 23], "momentum": 15, "moon": [10, 11], "more": [5, 8, 18, 23], "multilay": 14, "multipl": [3, 5], "multipli": 10, "need": 23, "network": [3, 4, 5, 6, 9, 14, 23], "neural": [3, 4, 5, 6, 9, 14, 23], "new": 6, "non": 10, "normal": [2, 3], "notat": 14, "now": [3, 11, 15], "nuclear": [2, 23], "numba": 23, "number": [2, 4, 20], "numer": [4, 20], "numpi": [18, 23], "object": 5, "obtain": 13, "od": 4, "off": 8, "ol": [0, 1, 7, 8], "one": [4, 14], "oper": 18, "optim": [3, 10, 15, 17, 23], "order": 15, "ordinari": [1, 7, 8, 23], "organ": [2, 23], "oslo": 22, "other": [6, 11, 13, 14, 18, 23], "our": [2, 6, 7, 13, 15, 23], "outcom": [17, 23], "output": 4, "overarch": [2, 6, 10, 11, 23], "overview": [12, 23], "own": [2, 12, 13, 23], "packag": [18, 23], "panda": 23, "paramet": 23, "part": [15, 17], "partial": 4, "pass": 3, "pca": 13, "pdf": 20, "perceptron": 14, "perform": [3, 11], "period": 5, "perspect": 3, "plethora": 23, "point": 6, "poisson": 4, "polynomi": [1, 5], "popul": 4, "popular": 23, "practic": [15, 21, 23], "pre": [3, 5], "predict": 6, "prerequisit": [5, 17, 23], "princip": 13, "principl": 5, "pro": 11, "probabl": [7, 20], "problem": [3, 4, 15, 23], "procedur": [11, 23], "process": [3, 5], "program": [4, 15], "project": [8, 21, 23], "prop": 15, "propag": [3, 14], "properti": [7, 20], "python": [0, 2, 11, 17, 18, 23], "quick": 10, "r": 23, "random": [12, 13, 20], "read": [11, 23], "real": [8, 23], "recommend": 23, "recurr": [6, 14], "reduc": 2, "reduct": 5, "reformul": 4, "regress": [0, 2, 7, 8, 9, 11, 12, 15, 23], "regular": 3, "relev": 22, "relu": 3, "remark": 5, "remind": [8, 10, 23], "replac": 15, "repositori": 0, "requir": [4, 17], "resampl": 8, "rescal": 8, "resourc": 4, "revisit": 15, "rewrit": 23, "ridg": [2, 7, 8], "rm": 15, "rule": 14, "same": 15, "sampl": 13, "schedul": 23, "schemat": 11, "scheme": 4, "scienc": 23, "scikit": [2, 3, 13, 23], "second": 15, "semest": 21, "set": [0, 2, 4, 5, 11, 14, 19, 23], "setup": 0, "sgd": 15, "should": 3, "similar": 15, "simpl": [2, 6, 11, 15, 23], "singl": 12, "singular": [7, 13], "sklearn": 1, "soft": 10, "softmax": 3, "softwar": 23, "solv": 4, "solver": 15, "some": [15, 18], "specifi": 4, "split": [0, 2], "squar": [1, 2, 7, 8, 12, 23], "standard": 15, "state": 2, "statist": [7, 8, 17, 20, 23], "steepest": [12, 15], "stochast": [15, 20], "strongli": 23, "suggest": 23, "summari": [21, 23], "superposit": 5, "supervis": 3, "support": 10, "svd": 7, "systemat": 5, "take": 1, "taken": 23, "teach": 21, "teacher": [21, 23], "techniqu": [8, 13], "technologi": 17, "tensorflow": [3, 5], "tent": [21, 23], "test": [0, 2, 3], "text": 23, "textbook": [22, 23], "theorem": [7, 10, 13, 14, 20], "theori": 20, "thi": 23, "tip": 15, "togeth": 14, "tool": 23, "top": 3, "topic": 23, "toward": 13, "trade": 8, "tradeoff": 8, "train": [0, 2, 3, 6, 23], "transform": 5, "tree": [11, 12], "tune": 3, "two": [5, 10, 17], "type": [4, 6, 14, 23], "uio": 23, "univers": [14, 22], "unsupervis": 16, "up": [0, 2, 4, 11, 14, 23], "us": [1, 2, 3, 4, 5, 9, 15, 17, 23], "v": 5, "valid": 8, "valu": [7, 13, 20], "variabl": 20, "varianc": 8, "variou": 2, "vector": [1, 10, 14, 18, 23], "versu": 23, "view": [2, 6, 12], "virtual": 0, "visual": [3, 11], "wai": 11, "wave": 4, "we": 23, "week": [0, 1, 23], "what": [2, 23], "which": 3, "why": 23, "wisconsin": 9, "write": [6, 13], "xgboost": 12, "your": [1, 2, 12]}})
\ No newline at end of file
+Search.setIndex({"alltitles": {"A Classification Tree": [[11, "a-classification-tree"]], "A Frequentist approach to data analysis": [[2, "a-frequentist-approach-to-data-analysis"], [23, "a-frequentist-approach-to-data-analysis"]], "A better approach": [[10, "a-better-approach"]], "A first summary": [[23, "a-first-summary"]], "A quick Reminder on Lagrangian Multipliers": [[10, "a-quick-reminder-on-lagrangian-multipliers"]], "A simple example": [[6, "a-simple-example"]], "A soft classifier": [[10, "a-soft-classifier"]], "A top-down perspective on Neural networks": [[3, "a-top-down-perspective-on-neural-networks"]], "ADAM optimizer": [[15, "adam-optimizer"]], "Activation functions": [[14, "activation-functions"]], "Adaptive boosting: AdaBoost, Basic Algorithm": [[12, "adaptive-boosting-adaboost-basic-algorithm"]], "Adding error analysis and training set up": [[23, "adding-error-analysis-and-training-set-up"]], "Adjust hyperparameters": [[3, "adjust-hyperparameters"]], "Algorithms for Setting up Decision Trees": [[11, "algorithms-for-setting-up-decision-trees"]], "An Overview of Ensemble Methods": [[12, "an-overview-of-ensemble-methods"]], "An extrapolation example": [[6, "an-extrapolation-example"]], "An optimization/minimization problem": [[23, "an-optimization-minimization-problem"]], "And what about using neural networks?": [[23, "and-what-about-using-neural-networks"]], "Another example, the moons again": [[11, "another-example-the-moons-again"]], "Applied Data Analysis and Machine Learning": [[17, null]], "Autocorrelation function": [[20, "autocorrelation-function"]], "Automatic differentiation": [[15, "automatic-differentiation"]], "Back to the Cancer Data": [[13, "back-to-the-cancer-data"]], "Bagging": [[12, "bagging"]], "Bagging Examples": [[12, "bagging-examples"]], "Basic Matrix Features": [[18, "basic-matrix-features"]], "Basic ideas of the Principal Component Analysis (PCA)": [[13, null]], "Basic math of the SVD": [[7, "basic-math-of-the-svd"]], "Basics": [[9, "basics"]], "Basics of a tree": [[11, "basics-of-a-tree"]], "Batch Normalization": [[3, "batch-normalization"]], "Bayes\u2019 Theorem and Ridge and Lasso Regression": [[7, "bayes-theorem-and-ridge-and-lasso-regression"]], "Boosting, a Bird\u2019s Eye View": [[12, "boosting-a-bird-s-eye-view"]], "Bootstrap": [[8, "bootstrap"]], "Bringing it together, first back propagation equation": [[14, "bringing-it-together-first-back-propagation-equation"]], "Building a Feed Forward Neural Network": [[3, null]], "Building a tree, regression": [[11, "building-a-tree-regression"]], "Building neural networks in Tensorflow and Keras": [[3, "building-neural-networks-in-tensorflow-and-keras"]], "CNNs in more detail, building convolutional neural networks in Tensorflow and Keras": [[5, "cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras"]], "Cancer Data again now with Decision Trees and other Methods": [[11, "cancer-data-again-now-with-decision-trees-and-other-methods"]], "Choose cost function and optimizer": [[3, "choose-cost-function-and-optimizer"]], "Classical PCA Theorem": [[13, "classical-pca-theorem"]], "Clustering and Unsupervised Learning": [[16, null]], "Code for SVD and Inversion of Matrices": [[7, "code-for-svd-and-inversion-of-matrices"]], "Codes and Approaches": [[16, "codes-and-approaches"]], "Codes for the SVD": [[7, "codes-for-the-svd"]], "Coding Setup and Linear Regression": [[0, "coding-setup-and-linear-regression"]], "Collect and pre-process data": [[3, "collect-and-pre-process-data"]], "Communication channels": [[23, "communication-channels"]], "Compare  Bagging on Trees with Random Forests": [[12, "compare-bagging-on-trees-with-random-forests"]], "Comparing with a numerical scheme": [[4, "comparing-with-a-numerical-scheme"]], "Computing the Gini index": [[11, "computing-the-gini-index"]], "Conjugate gradient method": [[15, "conjugate-gradient-method"]], "Convex functions": [[15, "convex-functions"]], "Convolution Examples: Polynomial multiplication": [[5, "convolution-examples-polynomial-multiplication"]], "Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)": [[5, "convolution-examples-principle-of-superposition-and-periodic-forces-fourier-transforms"]], "Convolutional Neural Network": [[14, "convolutional-neural-network"]], "Convolutional Neural Networks": [[5, null]], "Correlation Matrix": [[13, "correlation-matrix"]], "Course Format": [[23, "course-format"]], "Course setting": [[19, null]], "Cross-validation": [[8, "cross-validation"]], "Deadlines for projects (tentative)": [[23, "deadlines-for-projects-tentative"]], "Decision trees, overarching aims": [[11, null]], "Deep learning methods": [[23, "deep-learning-methods"]], "Define model and architecture": [[3, "define-model-and-architecture"]], "Defining the cost function": [[3, "defining-the-cost-function"]], "Deliverables": [[0, "deliverables"], [1, "deliverables"]], "Derivatives and the chain rule": [[14, "derivatives-and-the-chain-rule"]], "Deriving OLS from a probability distribution": [[7, "deriving-ols-from-a-probability-distribution"]], "Deriving and Implementing Ordinary Least Squares": [[1, "deriving-and-implementing-ordinary-least-squares"]], "Deriving the back propagation code for a multilayer perceptron model": [[14, "deriving-the-back-propagation-code-for-a-multilayer-perceptron-model"]], "Developing a code for doing neural networks with back propagation": [[3, "developing-a-code-for-doing-neural-networks-with-back-propagation"]], "Diagonalize the sample covariance matrix to obtain the principal components": [[13, "diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components"]], "Different kernels and Mercer\u2019s theorem": [[10, "different-kernels-and-mercer-s-theorem"]], "Disadvantages": [[11, "disadvantages"]], "Discriminative Modeling": [[23, "discriminative-modeling"]], "Domains and probabilities": [[20, "domains-and-probabilities"]], "Dropout": [[3, "dropout"]], "Elements of Probability Theory and Statistical Data Analysis": [[20, null]], "Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods": [[12, null]], "Entropy and the ID3 algorithm": [[11, "entropy-and-the-id3-algorithm"]], "Essential elements of ML": [[23, "essential-elements-of-ml"]], "Evaluate model performance on test data": [[3, "evaluate-model-performance-on-test-data"]], "Example of discriminative modeling, taken from Generative Deeep Learning by David Foster": [[23, "example-of-discriminative-modeling-taken-from-generative-deeep-learning-by-david-foster"]], "Example of generative modeling, taken from Generative Deep Learning by David Foster": [[23, "example-of-generative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example: Exponential decay": [[4, "example-exponential-decay"]], "Example: Population growth": [[4, "example-population-growth"]], "Example: The diffusion equation": [[4, "example-the-diffusion-equation"]], "Example: binary classification problem": [[3, "example-binary-classification-problem"]], "Examples": [[23, "examples"]], "Examples of likelihood functions used in logistic regression and neural networks": [[9, "examples-of-likelihood-functions-used-in-logistic-regression-and-neural-networks"]], "Exercise 1 - Finding the derivative of Matrix-Vector expressions": [[1, "exercise-1-finding-the-derivative-of-matrix-vector-expressions"]], "Exercise 1 - Github Setup": [[0, "exercise-1-github-setup"]], "Exercise 1: Setting up various Python environments": [[2, "exercise-1-setting-up-various-python-environments"]], "Exercise 2 - Deriving the expression for OLS": [[1, "exercise-2-deriving-the-expression-for-ols"]], "Exercise 2 - Setting up a Github repository": [[0, "exercise-2-setting-up-a-github-repository"]], "Exercise 2: making your own data and exploring scikit-learn": [[2, "exercise-2-making-your-own-data-and-exploring-scikit-learn"]], "Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression": [[1, "exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression"]], "Exercise 3 - Fitting an OLS model to data": [[0, "exercise-3-fitting-an-ols-model-to-data"]], "Exercise 3 - Setting up a Python virtual environment": [[0, "exercise-3-setting-up-a-python-virtual-environment"]], "Exercise 3: Normalizing our data": [[2, "exercise-3-normalizing-our-data"]], "Exercise 4 - Fitting a polynomial": [[1, "exercise-4-fitting-a-polynomial"]], "Exercise 4 - The train-test split": [[0, "exercise-4-the-train-test-split"]], "Exercise 4: Adding Ridge Regression": [[2, "exercise-4-adding-ridge-regression"]], "Exercise 5 - Comparing your code with sklearn": [[1, "exercise-5-comparing-your-code-with-sklearn"]], "Exercise 5: Analytical exercises": [[2, "exercise-5-analytical-exercises"]], "Exercise:  Cross-validation as resampling techniques, adding more complexity": [[8, "exercise-cross-validation-as-resampling-techniques-adding-more-complexity"]], "Exercise: Analysis of real data": [[8, "exercise-analysis-of-real-data"]], "Exercise: Bias-variance trade-off and resampling techniques": [[8, "exercise-bias-variance-trade-off-and-resampling-techniques"]], "Exercise: Lasso Regression on the Franke function  with resampling": [[8, "exercise-lasso-regression-on-the-franke-function-with-resampling"]], "Exercise: Ordinary Least Square (OLS) on the Franke function": [[8, "exercise-ordinary-least-square-ols-on-the-franke-function"]], "Exercise: Ridge Regression on the Franke function  with resampling": [[8, "exercise-ridge-regression-on-the-franke-function-with-resampling"]], "Exercises": [[2, "exercises"]], "Exercises and Projects": [[8, "exercises-and-projects"]], "Exercises week 34": [[0, null]], "Exercises week 35": [[1, null]], "Expectation values": [[20, "expectation-values"]], "Extremely useful tools, strongly recommended": [[23, "extremely-useful-tools-strongly-recommended"]], "Feed-forward neural networks": [[14, "feed-forward-neural-networks"]], "Feed-forward pass": [[3, "feed-forward-pass"]], "Final back propagating equation": [[14, "final-back-propagating-equation"]], "Fine-tuning neural network hyperparameters": [[3, "fine-tuning-neural-network-hyperparameters"]], "Fitting an Equation of State for Dense Nuclear Matter": [[2, "fitting-an-equation-of-state-for-dense-nuclear-matter"]], "From one to many layers, the universal approximation theorem": [[14, "from-one-to-many-layers-the-universal-approximation-theorem"]], "Further Dimensionality Remarks": [[5, "further-dimensionality-remarks"]], "Further properties (important for our analyses later)": [[7, "further-properties-important-for-our-analyses-later"]], "Gaussian Elimination": [[18, "gaussian-elimination"]], "General Features": [[11, "general-features"]], "General linear models and linear algebra": [[23, "general-linear-models-and-linear-algebra"]], "Generalizing the fitting procedure as a linear algebra problem": [[23, "generalizing-the-fitting-procedure-as-a-linear-algebra-problem"], [23, "id1"]], "Generative Adversarial Networks": [[6, "generative-adversarial-networks"]], "Generative Models": [[6, "generative-models"]], "Generative Versus Discriminative Modeling": [[23, "generative-versus-discriminative-modeling"]], "Geometric Interpretation and link with Singular Value Decomposition": [[13, "geometric-interpretation-and-link-with-singular-value-decomposition"]], "Gradient Boosting, Classification Example": [[12, "gradient-boosting-classification-example"]], "Gradient Boosting, Examples of Regression": [[12, "gradient-boosting-examples-of-regression"]], "Gradient Clipping": [[3, "gradient-clipping"]], "Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent": [[12, "gradient-boosting-basics-with-steepest-descent-functional-gradient-descent"]], "Gradient descent": [[4, "gradient-descent"]], "Grading": [[21, "grading"], [21, "id2"], [23, "grading"]], "Housing data, the code": [[2, "housing-data-the-code"]], "How to take derivatives of Matrix-Vector expressions": [[1, "how-to-take-derivatives-of-matrix-vector-expressions"]], "Hyperplanes and all that": [[10, "hyperplanes-and-all-that"]], "Important Matrix and vector handling packages": [[18, "important-matrix-and-vector-handling-packages"]], "Improving performance": [[3, "improving-performance"]], "In summary": [[21, "in-summary"]], "Including Stochastic Gradient Descent with Autograd": [[15, "including-stochastic-gradient-descent-with-autograd"]], "Incremental PCA": [[13, "incremental-pca"]], "Installing R, C++, cython or Julia": [[23, "installing-r-c-cython-or-julia"]], "Installing R, C++, cython, Numba etc": [[23, "installing-r-c-cython-numba-etc"]], "Instructor information": [[21, "instructor-information"]], "Interpretations and optimizing our parameters": [[23, "interpretations-and-optimizing-our-parameters"], [23, "id2"], [23, "id3"]], "Introducing JAX": [[15, "introducing-jax"]], "Introducing the Covariance and Correlation functions": [[13, "introducing-the-covariance-and-correlation-functions"]], "Introduction": [[2, "introduction"], [8, "introduction"], [17, "introduction"], [18, "introduction"]], "Iterative Fitting, Classification and AdaBoost": [[12, "iterative-fitting-classification-and-adaboost"]], "Iterative Fitting, Regression and Squared-error Cost Function": [[12, "iterative-fitting-regression-and-squared-error-cost-function"]], "Kernel PCA": [[13, "kernel-pca"]], "Kernels and non-linearity": [[10, "kernels-and-non-linearity"]], "LU Decomposition, the inverse of a matrix": [[18, "lu-decomposition-the-inverse-of-a-matrix"]], "Layers": [[3, "layers"]], "Layers used to build CNNs": [[5, "layers-used-to-build-cnns"]], "Learning goals": [[0, "learning-goals"], [1, "learning-goals"]], "Learning outcomes": [[17, "learning-outcomes"], [23, "learning-outcomes"]], "Lectures and ComputerLab": [[23, "lectures-and-computerlab"]], "Limitations of supervised learning with deep networks": [[3, "limitations-of-supervised-learning-with-deep-networks"]], "Linear Algebra, Handling of Arrays and more Python Features": [[18, null]], "Linear Regression": [[2, null]], "Linear Regression, basic elements": [[2, "linear-regression-basic-elements"]], "Linking Bayes\u2019 Theorem with Ridge and Lasso Regression": [[7, "linking-bayes-theorem-with-ridge-and-lasso-regression"]], "Linking the regression analysis with a statistical interpretation": [[7, "linking-the-regression-analysis-with-a-statistical-interpretation"]], "Linking with the SVD": [[7, "linking-with-the-svd"]], "Links to relevant courses at the University of Oslo": [[22, "links-to-relevant-courses-at-the-university-of-oslo"]], "Logistic Regression": [[9, null], [9, "id1"]], "MNIST and GANs": [[6, "mnist-and-gans"]], "Machine Learning": [[23, "machine-learning"]], "Machine learning": [[17, "machine-learning"]], "Main textbooks": [[23, "main-textbooks"]], "Making a tree": [[11, "making-a-tree"]], "Making your own Bootstrap: Changing the Level of the Decision Tree": [[12, "making-your-own-bootstrap-changing-the-level-of-the-decision-tree"]], "Mathematical Interpretation of Ordinary Least Squares": [[7, "mathematical-interpretation-of-ordinary-least-squares"]], "Mathematical optimization of convex functions": [[10, "mathematical-optimization-of-convex-functions"]], "Mathematics of CNNs": [[5, "mathematics-of-cnns"]], "Mathematics of the SVD and implications": [[7, "mathematics-of-the-svd-and-implications"]], "Matrices in Python": [[23, "matrices-in-python"]], "Matrix  multiplication": [[3, "matrix-multiplication"]], "Matrix-vector notation  and activation": [[14, "matrix-vector-notation-and-activation"]], "Meet the  covariance!": [[20, "meet-the-covariance"]], "Meet the Covariance Matrix": [[7, "meet-the-covariance-matrix"]], "Meet the Pandas": [[23, "meet-the-pandas"]], "Momentum based GD": [[15, "momentum-based-gd"]], "More complicated Example: The Ising model": [[8, "more-complicated-example-the-ising-model"]], "More on Dimensionalities": [[5, "more-on-dimensionalities"]], "More on Rescaling data": [[8, "more-on-rescaling-data"]], "Multilayer perceptrons": [[14, "multilayer-perceptrons"]], "Network requirements": [[4, "network-requirements"]], "Neural Networks vs CNNs": [[5, "neural-networks-vs-cnns"]], "Neural networks": [[14, null]], "Numerical experiments and the covariance, central limit theorem": [[20, "numerical-experiments-and-the-covariance-central-limit-theorem"]], "Numpy and arrays": [[18, "numpy-and-arrays"], [23, "numpy-and-arrays"]], "Numpy examples and Important Matrix and vector handling packages": [[23, "numpy-examples-and-important-matrix-and-vector-handling-packages"]], "Optimization, the central part of any Machine Learning algortithm": [[15, null]], "Optimizing our parameters": [[23, "optimizing-our-parameters"]], "Optimizing our parameters, more details": [[23, "optimizing-our-parameters-more-details"]], "Optimizing the cost function": [[3, "optimizing-the-cost-function"]], "Organizing our data": [[2, "organizing-our-data"], [23, "organizing-our-data"]], "Other Matrix and Vector Operations": [[18, "other-matrix-and-vector-operations"]], "Other Types of Recurrent Neural Networks": [[6, "other-types-of-recurrent-neural-networks"]], "Other courses on Data science and Machine Learning  at UiO": [[23, "other-courses-on-data-science-and-machine-learning-at-uio"]], "Other courses on Data science and Machine Learning  at UiO, contn": [[23, "other-courses-on-data-science-and-machine-learning-at-uio-contn"]], "Other popular texts": [[23, "other-popular-texts"]], "Other techniques": [[13, "other-techniques"]], "Other types of networks": [[14, "other-types-of-networks"]], "Other ways of visualizing the trees": [[11, "other-ways-of-visualizing-the-trees"]], "Our model for the nuclear binding energies": [[23, "our-model-for-the-nuclear-binding-energies"]], "Overview of first week": [[23, "overview-of-first-week"]], "Own code for Ordinary Least Squares": [[23, "own-code-for-ordinary-least-squares"]], "PCA and scikit-learn": [[13, "pca-and-scikit-learn"]], "Pandas AI": [[23, "pandas-ai"]], "Partial Differential Equations": [[4, "partial-differential-equations"]], "Practical tips": [[15, "practical-tips"]], "Practicalities": [[21, "practicalities"], [21, "id1"]], "Predicting New Points With A Trained Recurrent Neural Network": [[6, "predicting-new-points-with-a-trained-recurrent-neural-network"]], "Prerequisites": [[23, "prerequisites"]], "Prerequisites and background": [[17, "prerequisites-and-background"]], "Prerequisites: Collect and pre-process data": [[5, "prerequisites-collect-and-pre-process-data"]], "Probability Distribution Functions": [[20, "probability-distribution-functions"]], "Program for stochastic gradient": [[15, "program-for-stochastic-gradient"]], "Properties of PDFs": [[20, "properties-of-pdfs"]], "Pros and cons of trees, pros": [[11, "pros-and-cons-of-trees-pros"]], "Python installers": [[17, "python-installers"], [23, "python-installers"]], "RMS prop": [[15, "rms-prop"]], "Random Numbers": [[20, "random-numbers"]], "Random forests": [[12, "random-forests"]], "Randomized PCA": [[13, "randomized-pca"]], "Reading material": [[23, "reading-material"]], "Reading suggestions week 34": [[23, "reading-suggestions-week-34"]], "Recurrent neural networks": [[14, "recurrent-neural-networks"]], "Recurrent neural networks: Overarching view": [[6, null]], "Reducing the number of degrees of freedom, overarching view": [[2, "reducing-the-number-of-degrees-of-freedom-overarching-view"]], "Reformulating the problem": [[4, "reformulating-the-problem"]], "Regression Case": [[12, "regression-case"]], "Regression analysis, overarching aims": [[23, "regression-analysis-overarching-aims"]], "Regression analysis, overarching aims II": [[23, "regression-analysis-overarching-aims-ii"]], "Regularization": [[3, "regularization"]], "Reminder on Statistics": [[8, "reminder-on-statistics"]], "Replace or not": [[15, "replace-or-not"]], "Required Technologies": [[17, "required-technologies"]], "Resampling Methods": [[8, null]], "Resampling methods": [[8, "id1"]], "Resources on differential equations and deep learning": [[4, "resources-on-differential-equations-and-deep-learning"]], "Revisiting our Linear Regression Solvers": [[15, "revisiting-our-linear-regression-solvers"]], "Rewriting the fitting procedure as a linear algebra problem": [[23, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem"]], "Rewriting the fitting procedure as a linear algebra problem, more details": [[23, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem-more-details"]], "Ridge and Lasso Regression": [[7, null], [7, "id1"]], "Same code but now with momentum gradient descent": [[15, "same-code-but-now-with-momentum-gradient-descent"]], "Schedule first week": [[23, "schedule-first-week"]], "Schematic Regression Procedure": [[11, "schematic-regression-procedure"]], "Setting up the Back propagation algorithm": [[14, "setting-up-the-back-propagation-algorithm"]], "Setting up the network using Autograd; The full program": [[4, "setting-up-the-network-using-autograd-the-full-program"]], "Similar (second order function now) problem but now with AdaGrad": [[15, "similar-second-order-function-now-problem-but-now-with-adagrad"]], "Simple Python Code to read in Data and perform Classification": [[11, "simple-python-code-to-read-in-data-and-perform-classification"]], "Simple linear regression model using scikit-learn": [[2, "simple-linear-regression-model-using-scikit-learn"], [23, "simple-linear-regression-model-using-scikit-learn"]], "Software and needed installations": [[23, "software-and-needed-installations"]], "Solving Differential Equations  with Deep Learning": [[4, null]], "Solving the one dimensional Poisson equation": [[4, "solving-the-one-dimensional-poisson-equation"]], "Solving the wave equation with Neural Networks": [[4, "solving-the-wave-equation-with-neural-networks"]], "Some famous Matrices": [[18, "some-famous-matrices"]], "Some simple problems": [[15, "some-simple-problems"]], "Splitting our Data in Training and Test data": [[2, "splitting-our-data-in-training-and-test-data"]], "Standard steepest descent": [[15, "standard-steepest-descent"]], "Statistical analysis and optimization of data": [[17, "statistical-analysis-and-optimization-of-data"], [23, "statistical-analysis-and-optimization-of-data"]], "Steepest descent": [[15, "steepest-descent"]], "Stochastic Gradient Descent (SGD)": [[15, "stochastic-gradient-descent-sgd"]], "Stochastic variables and the main concepts, the discrete case": [[20, "stochastic-variables-and-the-main-concepts-the-discrete-case"]], "Support Vector Machines, overarching aims": [[10, null]], "Systematic reduction": [[5, "systematic-reduction"]], "Teachers": [[23, "teachers"]], "Teachers and Grading": [[21, null]], "Teaching Assistants Fall semester 2023": [[21, "teaching-assistants-fall-semester-2023"]], "Tentative deadllines for projects": [[21, "tentative-deadllines-for-projects"]], "Testing the Means Squared Error as function of Complexity": [[2, "testing-the-means-squared-error-as-function-of-complexity"]], "Textbooks": [[22, null]], "The Algorithm before theorem": [[13, "the-algorithm-before-theorem"]], "The Boston housing data example": [[2, "the-boston-housing-data-example"]], "The Breast Cancer Data, now with Keras": [[3, "the-breast-cancer-data-now-with-keras"]], "The CART algorithm for Classification": [[11, "the-cart-algorithm-for-classification"]], "The CART algorithm for Regression": [[11, "the-cart-algorithm-for-regression"]], "The CIFAR01 data set": [[5, "the-cifar01-data-set"]], "The MNIST dataset again": [[5, "the-mnist-dataset-again"]], "The RELU function family": [[3, "the-relu-function-family"]], "The Softmax function": [[3, "the-softmax-function"]], "The \\chi^2 function": [[2, "the-chi-2-function"], [23, "the-chi-2-function"], [23, "id4"], [23, "id5"], [23, "id6"], [23, "id7"], [23, "id8"]], "The bias-variance tradeoff": [[8, "the-bias-variance-tradeoff"]], "The code for solving the ODE": [[4, "the-code-for-solving-the-ode"]], "The course has two central parts": [[17, "the-course-has-two-central-parts"]], "The logistic function": [[9, "the-logistic-function"]], "The moons example": [[10, "the-moons-example"]], "The multilayer  perceptron (MLP)": [[14, "the-multilayer-perceptron-mlp"]], "The network with one input layer, specified number of hidden layers, and one output layer": [[4, "the-network-with-one-input-layer-specified-number-of-hidden-layers-and-one-output-layer"]], "The plethora  of machine learning algorithms/methods": [[23, "the-plethora-of-machine-learning-algorithms-methods"]], "The singular value decomposition": [[7, "the-singular-value-decomposition"]], "The two-dimensional case": [[10, "the-two-dimensional-case"]], "To our real data: nuclear binding energies. Brief reminder on masses and binding energies": [[23, "to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies"]], "Topics covered in this course: Statistical analysis and optimization of data": [[23, "topics-covered-in-this-course-statistical-analysis-and-optimization-of-data"]], "Towards the PCA theorem": [[13, "towards-the-pca-theorem"]], "Train and test datasets": [[3, "train-and-test-datasets"]], "Two-dimensional Objects": [[5, "two-dimensional-objects"]], "Type of problem": [[4, "type-of-problem"]], "Types of Machine Learning": [[23, "types-of-machine-learning"]], "Useful Python libraries": [[17, "useful-python-libraries"], [23, "useful-python-libraries"]], "Using Autograd": [[15, "using-autograd"]], "Using forward Euler to solve the ODE": [[4, "using-forward-euler-to-solve-the-ode"]], "Using gradient descent methods, limitations": [[15, "using-gradient-descent-methods-limitations"]], "Visualization": [[3, "visualization"], [3, "id1"]], "Visualizing the Tree, Classification": [[11, "visualizing-the-tree-classification"]], "Week 34: Introduction to the course, Logistics and Practicalities": [[23, null]], "What Is Generative Modeling?": [[23, "what-is-generative-modeling"]], "What is Machine Learning?": [[2, "what-is-machine-learning"]], "What is a good model?": [[2, "what-is-a-good-model"], [23, "what-is-a-good-model"]], "What is a good model? Can we define it?": [[23, "what-is-a-good-model-can-we-define-it"]], "Which activation function should I use?": [[3, "which-activation-function-should-i-use"]], "Why Linear Regression (aka Ordinary Least Squares and family)": [[23, "why-linear-regression-aka-ordinary-least-squares-and-family"]], "Wisconsin Cancer Data": [[9, "wisconsin-cancer-data"]], "Writing Our First Generative Adversarial Network": [[6, "writing-our-first-generative-adversarial-network"]], "Writing our own PCA code": [[13, "writing-our-own-pca-code"]], "XGBoost: Extreme Gradient Boosting": [[12, "xgboost-extreme-gradient-boosting"]], "scikit-learn implementation": [[3, "scikit-learn-implementation"]]}, "docnames": ["E1", "E2", "chapter1", "chapter10", "chapter11", "chapter12", "chapter13", "chapter2", "chapter3", "chapter4", "chapter5", "chapter6", "chapter7", "chapter8", "chapter9", "chapteroptimization", "clustering", "intro", "linalg", "schedule", "statistics", "teachers", "textbooks", "week34"], "envversion": {"sphinx": 62, "sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1}, "filenames": ["E1.ipynb", "E2.ipynb", "chapter1.ipynb", "chapter10.ipynb", "chapter11.ipynb", "chapter12.ipynb", "chapter13.ipynb", "chapter2.ipynb", "chapter3.ipynb", "chapter4.ipynb", "chapter5.ipynb", "chapter6.ipynb", "chapter7.ipynb", "chapter8.ipynb", "chapter9.ipynb", "chapteroptimization.ipynb", "clustering.ipynb", "intro.md", "linalg.ipynb", "schedule.md", "statistics.ipynb", "teachers.md", "textbooks.md", "week34.ipynb"], "indexentries": {}, "objects": {}, "objnames": {}, "objtypes": {}, "terms": {"": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 15, 17, 18, 20, 21, 23], "0": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23], "00": [2, 3, 7, 13, 23], "000": [3, 5], "00000000e": 23, "001": [4, 10, 15], "004": 7, "00727646693": [2, 23], "0086649156": [2, 23], "01": [2, 3, 4, 7, 11, 13, 15, 22, 23], "0110": 20, "01719003e": 23, "02": [2, 6, 9, 14, 23], "02334824": 23, "02857": 6, "02f": 8, "03077640549": 6, "03097597e": 23, "031": 7, "04": 13, "0458": 11, "05": [6, 8, 23], "062292565": 6, "062435": 23, "06730814": 23, "07": 23, "0713": [2, 23], "07285": 5, "08": 20, "08078025e": 23, "08336233266": 6, "0917": 11, "0n": [2, 23], "1": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23], "10": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 19, 20, 21, 23], "100": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 18, 20, 21, 23], "1000": [2, 3, 4, 6, 7, 10, 13, 15, 16, 17, 20, 23], "10000": [4, 7, 8, 12, 13, 15, 20], "100000": 10, "10001": 12, "1001": 20, "1002": 20, "1003": 20, "1005": 20, "1009": 20, "101": 1, "1011": 20, "1013": 20, "1013904243": 20, "1015": 20, "102": 1, "1023": 20, "1024": 5, "1026": 20, "1027": 20, "103": 3, "1030": 20, "1037": 20, "1038": 20, "1040": 20, "1047": 20, "107": 1, "108": 23, "10th": 11, "10x": [2, 23], "11": [1, 2, 4, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 22, 23], "110": 23, "1100": 20, "1101": 20, "111": [3, 9, 14], "112": 1, "11340253": 23, "11590451": 23, "116": 1, "117": 1, "118": 1, "12": [2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 18, 20, 22, 23], "120": 5, "121": [1, 10, 11, 12], "1215pm": [21, 23], "122": [10, 11, 12, 23], "124": [2, 23], "125": 1, "127": [1, 6], "128": [5, 6, 15], "129": 1, "1298": 11, "12pm": [21, 23], "13": [2, 4, 11, 14, 18, 20, 23], "131": 1, "133": 9, "135": 1, "136": 1, "14": [2, 4, 6, 8, 10, 11, 12, 14, 18, 20, 22], "141": 1, "143": 1, "1446729567": 6, "149": 1, "14g": 8, "15": [2, 4, 6, 8, 9, 10, 11, 14, 15, 20, 23], "150": [6, 10], "152": [1, 23], "153760": 23, "156": [1, 23], "157": 23, "158": 23, "159": [1, 23], "15g": 8, "15pm": 23, "16": [3, 4, 5, 6, 7, 10, 11, 12, 20, 23], "160": [1, 23], "1603": 5, "161": 1, "162": 1, "16231451": 6, "163": 1, "16384": 5, "164": 1, "167": 1, "17": [3, 4, 10, 20, 23], "172": 1, "173": 1, "176": 1, "178": 1, "179": 1, "1797": 3, "18": [4, 8, 9, 10, 11, 12, 20, 23], "1807": 6, "18392847": 23, "19": [4, 20, 23], "1940": 2, "1943": 14, "1970": [18, 23], "1973": 11, "1979": 8, "1_1": 14, "1_2": 14, "1_3": 14, "1cm": [2, 10, 12, 20, 23], "1d": [3, 4, 5], "1e": [4, 6, 15, 16], "1e10": 16, "1e4": 8, "1f": 3, "1k": 18, "1n": [2, 23], "1x": [2, 23], "2": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22], "20": [1, 2, 3, 4, 8, 9, 10, 20, 21, 23], "200": [2, 4, 5, 6, 10, 11, 12], "2000": 2, "2004": 15, "2006": 22, "20072279": 23, "2008": 23, "2010": 3, "2011": 3, "2014": 6, "2015": 3, "2016": [2, 23], "2018": [2, 8], "2021": [8, 16], "2022": 23, "2025": 23, "21": [2, 3, 7, 9, 11, 14, 18, 23], "2116753732": 6, "215pm": [21, 23], "2167072": 23, "22": [2, 3, 7, 14, 15, 18, 23], "221": 10, "225": 6, "22948497": 23, "23": [3, 14, 18, 23], "24": [2, 3, 18, 23], "25": [4, 5, 6, 7, 8, 10, 11, 13], "250": [4, 6, 9, 11], "25000": 2, "250154": 23, "253775": 23, "255": 5, "256": 6, "26": 23, "26303845": 23, "264": 23, "265": 23, "265109911": 6, "266": 23, "269": 23, "27": [2, 3], "270": 23, "27n_": 20, "28": [3, 5, 6], "2830637392": 6, "2861": 20, "2873": 11, "2882": 20, "2886": 20, "2890": [2, 23], "2892": 20, "29": 23, "2915": 20, "2931": 23, "29364655": 23, "294399745619595": 23, "296247": 23, "2968": 23, "2980": 23, "298273": 23, "298375": 23, "2990": 23, "2_": 14, "2_1": 14, "2_2": 14, "2_3": 14, "2_i": 14, "2_m": [8, 20], "2_t": 15, "2_x": 20, "2b": 20, "2cm": 10, "2d": [3, 5, 13, 14, 17, 23], "2e": 8, "2f": [2, 9, 11, 12, 13, 14, 23], "2g": 4, "2g_i": 4, "2k": 5, "2m": 8, "2n": [2, 4, 5, 23], "2nd": 11, "2p": 20, "2pt": 6, "2x": [2, 5, 10, 15, 23], "2x_ix_jy_iy_j": 10, "2x_j": 10, "2y_i": 12, "2y_j": 10, "3": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23], "30": [2, 3, 6, 8, 9, 12, 15, 21], "30000": [2, 23], "3072": 5, "31": [14, 18, 20], "315": 8, "3155": [2, 7, 8], "32": [5, 6, 8, 14, 15, 18, 20], "3200": 3, "3250": 3, "3297": 23, "33": [14, 18, 21], "3303": 23, "3310": 23, "332331": 23, "333": 9, "3331": 23, "3337": 23, "34": 18, "3436": [2, 23], "3437": [2, 23], "35": [2, 8, 23], "3581341341": 6, "359": 7, "36": [2, 7, 8, 20], "370782966": 6, "38": 20, "39": [2, 21, 23], "3d": [1, 4, 5, 6, 8, 15], "3f": [3, 5, 11], "3n": 18, "3x": [4, 10], "3x_i": 4, "3y": 10, "4": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "40": [3, 8, 21, 23], "400": 6, "4000": 23, "4050": [22, 23], "41": 18, "4155": [0, 4], "41589548": 23, "42": [3, 6, 10, 11, 12, 18, 23], "43": [2, 9, 18], "4310": 23, "436462435": 6, "44": [2, 18], "45": [21, 23], "46": [21, 23], "462": 9, "47": [21, 23], "479465113": 6, "47958494": 23, "48": 23, "48257387": [21, 23], "49": [7, 8, 13], "49152": 5, "4940954": [2, 23], "4990": 20, "4992": 20, "4997": 20, "4c4c7f": [11, 12], "4d": 5, "4f": 8, "4pm": [21, 23], "4y": 10, "4y_i": 12, "5": [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "50": [3, 4, 5, 6, 8, 9, 10, 12, 15, 23], "500": [3, 5, 6, 8, 11, 12, 15], "5018": 20, "506": 2, "507d50": [11, 12], "50j": 15, "50x10": 3, "51": [12, 23], "510": 3, "512132": 23, "5177783846": 6, "53": 11, "54": [8, 20, 23], "5411205": 23, "54894451": 23, "55": [3, 23], "56": 3, "56536": [2, 23], "569": 3, "57": [2, 10, 21, 23], "571": 7, "58": [12, 21, 23], "591317992": 6, "5cm": 20, "5f": 10, "5x": 10, "5y": 10, "6": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 21, 23], "60": [3, 5], "60000": 6, "6019067271": 6, "606439": 23, "625": 9, "63": [2, 3], "64": [3, 5, 6, 15, 18, 23], "64x50": 3, "65": [3, 10, 11], "6887363571": 6, "69": [1, 20], "69069n_": 20, "691": 23, "6n_": 20, "7": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 22, 23], "70": [3, 9], "70653767": 6, "71": 3, "724": 5, "73": 23, "7304881": 23, "75": [7, 8, 10, 13], "76": [21, 23], "765": 9, "77": [21, 23], "7718": 11, "7782028952": 6, "77893972": 23, "78": 23, "7d7d58": [11, 12], "8": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 21, 23], "80": [2, 3, 7, 10], "800": [6, 9], "81": 3, "815am": [21, 23], "85": 3, "8702784034": 6, "88": 23, "8f": 8, "8g": 8, "8n": 18, "8x8": 3, "9": [2, 3, 4, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 23], "90": 3, "9040": 11, "91": [21, 23], "92": [21, 23], "93": 1, "931": [2, 23], "933": 7, "937": 20, "938": 20, "939": [2, 20, 23], "94": 20, "95": [3, 13, 23], "954": 20, "955820c21e8b": 6, "96": 8, "960": 20, "961": 20, "962": 20, "9649652536": 6, "96611194e": 23, "9780387310732": 22, "9780387848570": 22, "9781098134174": 23, "9781492032632": 22, "9781801819312": 23, "98": [1, 2, 3], "985": 20, "986": 20, "989": 20, "9898ff": [11, 12], "99": [1, 15], "991": 20, "992": 20, "993": 20, "996": 7, "999": [11, 20], "9x": 8, "9y": 8, "A": [0, 1, 4, 5, 7, 8, 9, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22], "AND": 4, "And": [2, 5, 6, 7, 8, 11, 15, 17, 20], "As": [1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 15, 18, 20, 23], "At": [2, 6, 8, 15, 23], "BE": [2, 23], "Be": [4, 17, 23], "Being": 15, "But": [1, 2, 3, 4, 5, 7, 8, 11, 12, 20], "By": [2, 5, 7, 8, 14, 15, 18, 23], "For": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23], "IF": 8, "IN": 22, "If": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23], "In": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23], "Ising": [7, 14], "It": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "Its": [3, 4, 6, 13], "No": [8, 11, 23], "Not": [2, 3, 7, 8], "OR": 20, "Of": 20, "On": [2, 5, 20, 21, 22, 23], "One": [2, 3, 5, 6, 7, 8, 9, 10, 13, 14, 15, 20], "Or": [2, 3, 8, 23], "Such": [1, 2, 8, 14, 20], "That": [2, 7, 9, 12, 13, 14, 16, 20, 23], "The": [1, 6, 12, 15, 16, 18, 19, 20, 21, 22], "Then": [0, 1, 2, 3, 8, 10, 11, 12, 13, 14, 15, 16, 18, 23], "There": [0, 2, 5, 6, 7, 8, 10, 11, 13, 14, 16, 18, 20, 21, 23], "These": [2, 5, 6, 7, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23], "To": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20], "With": [1, 2, 7, 8, 10, 11, 12, 13, 14, 16, 18, 20, 23], "_": [1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 23], "_0": [7, 10, 12, 13, 15], "_1": [4, 7, 8, 10, 12, 13, 14, 15, 16, 18], "_2": [4, 7, 10, 13, 14, 15, 18], "_3": 18, "_4": 18, "_9": 15, "__class__": 12, "__doc__": 8, "__future__": [10, 11], "__init__": 3, "__main__": 4, "__name__": [4, 12], "_auto1": [4, 5, 6, 7, 8, 9, 14, 15, 18, 20], "_auto10": [8, 14], "_auto11": 8, "_auto12": 8, "_auto2": [4, 5, 6, 7, 8, 14, 15, 18, 20], "_auto3": [5, 6, 7, 8, 14, 15, 18], "_auto4": [6, 8, 14, 15, 18], "_auto5": [6, 8, 14, 15, 18], "_auto6": [6, 8, 14, 18], "_auto7": [6, 8, 14, 18], "_auto8": [8, 14], "_auto9": [8, 14], "_build": [2, 17, 22, 23], "_c": 3, "_compon": 13, "_depth": 11, "_export": [0, 1], "_fraction": 11, "_i": [2, 3, 4, 7, 8, 9, 10, 13, 14, 15, 23], "_j": [2, 3, 4, 5, 7, 8, 10, 15], "_k": 15, "_l": 14, "_lambda": 8, "_leaf": 11, "_m": 12, "_multilayer_perceptron": 23, "_n": [4, 7, 10, 13, 15], "_node": 11, "_p": [7, 10], "_ratio": 13, "_sampl": 11, "_split": [8, 11], "_t": 15, "_test": 8, "_varianc": 13, "_weight": 11, "a0": 5, "a0faa0": [11, 12], "a1": [2, 23], "a2": [2, 23], "a3": [2, 23], "a4": [2, 23], "a_": [1, 2, 3, 18, 23], "a_0": [2, 23], "a_1a": [2, 23], "a_2a": [2, 23], "a_3": [2, 23], "a_3a": [2, 23], "a_4": [2, 23], "a_4a": [2, 23], "a_h": 3, "a_i": [2, 3, 4, 14, 23], "a_j": [3, 14], "a_k": [2, 3, 14], "aaron": 22, "ab": [2, 4, 7, 15, 16, 23], "ab_channel": 17, "abandon": 3, "abid": 20, "abil": [2, 12], "abl": [1, 2, 3, 6, 7, 8, 9, 12, 14, 15], "about": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 21], "abov": [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 22, 23], "abovement": [8, 23], "abscissa": 15, "absolut": [2, 4, 7, 8, 15, 23], "abstract": 3, "acceler": 15, "accept": [2, 5, 8, 11], "access": [2, 5, 13, 20, 23], "accid": [6, 8], "accompani": [2, 23], "accomplish": [10, 11, 15], "accord": [2, 3, 4, 7, 8, 11, 14, 15, 16, 20, 23], "accordingli": 13, "account": [0, 1, 2, 5, 7, 15, 20, 23], "accumul": [14, 15, 20], "accur": [2, 5, 6, 8, 12, 15], "accuraci": [2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 23], "accuracy_scor": [2, 3, 12, 23], "accuracy_score_numpi": 3, "achiev": [2, 3, 7, 8, 10, 14, 18, 23], "aco": 20, "acquaint": 17, "acquir": [3, 17, 23], "acr": 2, "across": [3, 5, 8, 11, 17, 23], "act": [3, 5, 18], "action": 20, "activ": [0, 2, 4, 5, 6, 11, 19, 21, 23], "actual": [0, 1, 2, 3, 6, 7, 8, 10, 13, 18, 20, 23], "ad": [0, 1, 3, 5, 6, 7, 10, 15, 18], "ada_clf": 12, "adaboostclassifi": 12, "adadelta": 15, "adam": [3, 5, 6, 23], "adapt": [6, 8, 15, 22], "add": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 20, 21, 23], "add_subplot": [3, 9, 14, 16], "addendum": 7, "addit": [0, 2, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18, 20, 21, 22, 23], "addition": [14, 15], "address": [3, 11, 13, 15, 23], "adjac": [5, 14], "adjoint": 7, "adjust": [2, 7, 14, 15], "admir": [2, 23], "advanc": [6, 8, 14, 22, 23], "advantag": [3, 5, 7, 8, 12, 15, 18], "adversari": 23, "afecionado": 23, "affect": [0, 5], "affin": [2, 5, 10, 13], "afford": 5, "aficionado": 23, "aforement": 16, "african": 2, "after": [0, 1, 2, 3, 4, 6, 7, 8, 11, 13, 14, 15, 17, 18, 20, 23], "afterward": [2, 23], "ag": [2, 9, 23], "ag_0": 4, "again": [2, 3, 6, 7, 8, 9, 10, 12, 13, 14, 15, 20, 23], "against": [3, 6, 9, 12], "agegroup": 9, "agegroupmean": 9, "aggreg": [11, 12], "agorithm": 12, "agre": [7, 8, 20], "agreement": 15, "ahead": 11, "ai": [2, 22], "aid": 13, "aim": [1, 2, 3, 6, 8, 9, 13, 16, 17, 18], "ainv": 7, "airplan": 5, "aka": 7, "al": [1, 2, 4, 6, 22, 23], "alarm": [7, 9], "algebra": [2, 5, 7, 15, 17], "algorithm": [1, 2, 3, 4, 6, 7, 8, 9, 10, 15, 16, 17, 18, 20, 22], "align": [2, 4, 7, 8, 9, 10, 15, 20, 23], "all": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23], "allevi": [3, 15], "alloc": [5, 18], "allow": [2, 3, 4, 5, 7, 8, 10, 12, 15, 17, 18, 23], "almost": [2, 3, 8, 10, 13, 15, 20], "alon": [4, 11], "along": [0, 4, 5, 6, 7, 8, 11, 12, 13, 17, 18, 23], "alpha": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 15, 16, 20, 23], "alpha_": 12, "alpha_0": 5, "alpha_1": 5, "alpha_2": 5, "alpha_i": [5, 15], "alpha_k": 15, "alpha_m": 12, "alpha_n": 5, "alpha_opt": 15, "alreadi": [0, 4, 5, 6, 7, 8, 12, 14, 17, 18, 20, 23], "also": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "alter": 3, "altern": [2, 3, 6, 7, 8, 10, 11, 13, 15, 18, 23], "although": [1, 2, 3, 7, 8, 10, 12, 15, 23], "alwai": [1, 2, 5, 7, 8, 14, 15, 20, 23], "am": 6, "ame2016": [2, 23], "american": 2, "among": [2, 5, 7, 11, 12, 14, 18, 23], "amongst": 7, "amount": [2, 3, 5, 6, 8, 10, 12, 16, 17], "an": [1, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 20, 21, 22], "an_": 20, "anaconda": [2, 3, 17, 23], "analogi": 15, "analys": 8, "analysi": [3, 5, 6, 9, 16, 18, 22], "analyt": [4, 5, 7, 8, 9, 14, 15, 17, 23], "analyz": [1, 2, 3, 5, 6, 7, 8, 20], "andrew": 3, "angl": [2, 5, 11], "anharmon": 5, "ani": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 20, 23], "anim": [6, 14], "ann": 14, "annot": [2, 3, 5, 9, 10, 23], "announc": 23, "anoth": [0, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 23], "ansatz": [2, 23], "answer": [2, 3, 5, 7, 8, 18, 21, 23], "antialias": [4, 8], "anticip": 6, "anymor": [3, 10], "anyon": [0, 6, 10], "anyth": [0, 1, 3, 20], "anytim": [21, 23], "apach": 3, "apart": [13, 15], "api": [3, 17, 23], "appar": 4, "appear": [2, 3, 5, 15, 18, 20], "append": [3, 5, 6, 10, 11, 15, 23], "appli": [2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 20, 22, 23], "applic": [1, 2, 3, 5, 6, 7, 8, 9, 11, 14, 15, 18, 20, 22, 23], "apply_gradi": 6, "approach": [0, 1, 3, 4, 6, 7, 8, 11, 12, 13, 14, 15, 17, 20, 22], "appropri": [4, 8, 11, 14, 15, 17, 20], "approv": 23, "approx": [2, 4, 5, 8, 12, 13, 15, 20, 23], "approxim": [2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 15, 20, 23], "apt": [2, 17, 23], "aq": 20, "ar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23], "aragorn": 23, "arang": [3, 5, 6, 8, 9, 11, 12, 14, 15, 23], "arbitrari": [3, 6, 8, 10, 14, 15, 20], "arbitrarili": [2, 3, 13, 23], "arc": 8, "architectur": [5, 6, 14], "area": [2, 5, 8, 22, 23], "argmax": [3, 13], "argmin": [6, 12, 16], "argsort": 13, "argu": [3, 15], "argument": [2, 4, 5, 7, 13, 14, 15, 23], "aris": [2, 8, 14, 15, 20, 23], "arithmet": [2, 15, 18, 23], "arm": 8, "armadillo": 18, "around": [2, 3, 6, 7, 8, 13, 20, 23], "arrai": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 15, 16, 17, 20], "arrang": [5, 23], "arraybox": 15, "arriv": [2, 8, 11, 13, 18, 20, 23], "arrow": 14, "arrowprop": 10, "art": [2, 3, 17], "articl": [2, 5, 6, 8, 12, 23], "artifici": [2, 4, 9, 14, 22, 23], "artificialneuron": 14, "arug": 15, "arxiv": [5, 6], "asarrai": [2, 8, 11], "ask": [0, 7, 8, 13, 14], "aspect": [2, 8, 17, 23], "assembl": 5, "assembli": [2, 23], "assert": 6, "assess": [2, 8, 23], "assici": 6, "assign": [0, 2, 9, 10, 11, 14, 15, 16, 19, 21, 22, 23], "associ": [2, 8, 11, 14, 16, 20, 23], "assum": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "assumpt": [2, 5, 7, 8, 11, 13, 20, 23], "ast": [2, 7, 8, 23], "astyp": [6, 11, 12], "asymmetri": [2, 23], "asymptot": [6, 8], "atom": [2, 23], "attempt": [2, 6, 8, 9, 10, 12, 23], "attend": 23, "attent": [2, 18, 23], "attract": [2, 12, 23], "attribut": [2, 11, 23], "audi": [2, 23], "audio": [5, 6], "august": 23, "aurelien": [2, 22, 23], "austfjel": 8, "auth": 0, "authent": 0, "author": [2, 3, 12, 20], "authour": 23, "auto": [11, 12, 20], "autocor": 20, "autocorrelation_tim": 20, "autocorrelform": 20, "autocovari": 20, "autoencod": [6, 17, 23], "autoencond": 17, "autograd": [17, 23], "autom": [2, 17, 22, 23], "automac": 18, "automag": 23, "automat": [1, 2, 3, 4, 5, 6, 13, 17, 18, 23], "automobil": 5, "autonom": 6, "avail": [2, 3, 6, 8, 12, 13, 17, 18, 19, 21, 22, 23], "averag": [2, 3, 5, 8, 11, 12, 15, 16, 20, 21, 23], "avoid": [2, 6, 7, 8, 11, 13, 15, 18], "awai": [4, 5, 8], "awar": [4, 12], "award": [21, 23], "ax": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 23], "axes3d": [4, 8, 15], "axes_grid1": 8, "axhlin": 10, "axi": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "axiom": 7, "axvlin": [6, 10], "axvspan": 6, "b": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 15, 16, 20, 21, 23], "b1": 10, "b2": 10, "b3": 10, "b_": [2, 3, 18], "b_0": 2, "b_1": [2, 4, 14, 15], "b_2": [2, 15], "b_5": 15, "b_group": 11, "b_i": [2, 3, 4, 14, 23], "b_ia_": [2, 23], "b_ia_i": 2, "b_index": 11, "b_j": [3, 14], "b_k": [2, 3, 14, 15], "b_m": 14, "b_score": 11, "b_valu": 11, "babcock": 23, "bachelor": [19, 21], "back": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 18, 20, 23], "backbon": 18, "backend": [3, 6], "background": [22, 23], "backpropag": 3, "backtrack": 11, "backup": 18, "backward": [3, 4, 6, 14, 18], "bad": 8, "badli": 20, "bag": [11, 17, 23], "bag_clf": 12, "baggin": 23, "baggingboot": 12, "baggingclassifi": 12, "baggingtre": 12, "balanc": 8, "band": 18, "bandwidth": 18, "bar": [2, 8, 13, 23], "barber": 22, "bare": [6, 12], "base": [1, 2, 3, 5, 6, 7, 9, 10, 11, 12, 16, 17, 20, 21, 22, 23], "basi": [7, 9, 10, 12, 13, 14, 15, 18], "basic": [0, 8, 10, 14, 15, 16, 17, 20, 23], "batch": [5, 6, 13, 14, 15], "batch_shap": 6, "batch_siz": [3, 5, 6], "batchnorm": 6, "bay": 9, "bayesian": [7, 17, 22, 23], "becaus": [2, 3, 4, 5, 6, 7, 8, 10, 11, 14, 15, 16, 23], "becom": [2, 3, 4, 7, 8, 9, 11, 14, 15, 20, 23], "been": [2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 17, 18, 23], "befor": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 18, 20, 23], "beforehand": [2, 20, 23], "begin": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 20, 21, 23], "behav": [3, 8, 15], "behavior": [2, 3, 15, 23], "behaviour": 14, "behind": [2, 3, 8, 10, 15, 23], "being": [2, 3, 4, 5, 6, 7, 9, 10, 12, 13, 14, 15, 20, 23], "believ": [11, 18], "belong": [9, 10, 11, 15, 16], "below": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "benchmark": 12, "benefici": [3, 15], "benefit": [2, 3, 6, 13, 15, 17, 23], "bengio": [3, 22, 23], "benign": [3, 9], "besid": [6, 7], "bessel": 7, "best": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 21, 23], "beta": [1, 2, 3, 5, 12, 13, 15, 23], "beta_": [5, 15], "beta_0": [3, 5, 15], "beta_1": [3, 5, 12, 15], "beta_1x_i": 15, "beta_2": [5, 15], "beta_3": 5, "beta_i": 5, "beta_j": 15, "beta_k": 15, "beta_linreg": 15, "beta_m": 12, "beta_mg_m": 12, "beta_n": 5, "better": [2, 3, 4, 5, 6, 8, 11, 12, 13, 14, 15, 23], "between": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 20, 23], "beyond": [2, 3, 7, 8, 10, 15, 23], "bf": [15, 16, 18, 20], "bg": 23, "bgd": 15, "bia": [2, 3, 4, 5, 7, 10, 11, 12, 14, 15, 23], "bias": [3, 4, 5, 7, 8, 11, 14], "big": [2, 3, 4, 7, 8, 16], "bigger": [3, 8], "bigr": 14, "bike": 11, "bilbo": 23, "billion": [5, 14, 17], "bin": [2, 9, 20], "binari": [2, 5, 7, 9, 11, 12, 14, 23], "binarycrossentropi": 6, "bind": 2, "binomi": [17, 20, 23], "binsboot": 8, "bioinformat": 2, "biolog": [3, 14], "bios1100": [17, 23], "bird": [2, 5], "birth": 23, "bishop": [22, 23], "bit": [3, 6, 18, 20, 23], "bitwis": 20, "bivari": 4, "bk": [2, 15], "bla": [18, 23], "black": [10, 11, 16], "block": [8, 12, 17, 18, 20, 23], "blog": 23, "blogpost": 6, "blue": [2, 5], "bmatrix": [2, 3, 5, 7, 9, 10, 13, 15, 18, 23], "bmi": 3, "bodi": [2, 3, 6, 14], "bold": 3, "boldfac": [1, 2, 7], "boldsymbol": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 15, 16, 23], "boltzmann": [14, 17, 23], "book": [22, 23], "book1": 22, "boolean": 6, "boost": [3, 11, 17, 23], "boostrap": 12, "bootstrap": [3, 15, 17, 23], "borrow": 23, "boston_dataset": 2, "bot": 10, "both": [0, 1, 2, 3, 6, 7, 8, 10, 11, 12, 15, 16, 17, 18, 20, 21, 23], "bottl": 9, "bound": [2, 10, 14], "boundari": [4, 6, 10, 13, 14], "box": [6, 11], "boyd": [10, 15], "bracket": [6, 20], "brain": [3, 9, 14], "branch": [11, 23], "break": [2, 6, 8, 13, 16, 23], "breast": [7, 9, 13], "breviti": 15, "brew": [2, 17, 23], "brg": 10, "briefli": [1, 2, 23], "bring": [2, 7, 8, 12], "britt": [21, 23], "broad": 2, "broadli": 23, "brought": [15, 17, 23], "brownle": 6, "browser": [0, 23], "brute": [5, 7, 13], "buffer_s": 6, "bui": 6, "build": [1, 2, 6, 7, 8, 12, 18, 20, 23], "built": [2, 3, 5, 6, 8], "bunch": 13, "busi": 2, "byte": [18, 23], "c": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22], "c1": [10, 13], "c2": [10, 13], "c_": [2, 10, 11, 12, 15, 20], "c_0": 20, "c_1": 14, "c_2": 14, "c_3": 14, "c_4": 14, "c_i": [14, 15], "c_k": 20, "ca": [3, 23], "cach": 12, "cal": [2, 10, 12, 14, 15], "calcul": [1, 2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "call": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "calor": 2, "cambridg": [15, 22], "can": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22], "cancel": [2, 15, 23], "cancer": [7, 12], "cancerpd": 9, "candid": [10, 11, 12], "cannot": [2, 3, 6, 7, 8, 9, 10, 11, 20], "canopi": [2, 17, 23], "canva": [0, 1, 23], "cap": 7, "capabl": [2, 3, 10, 15, 17, 23], "capac": [4, 21], "capita": 2, "captur": [6, 13, 14, 23], "car": [5, 6], "card": [2, 9, 23], "cardin": 3, "care": [0, 13], "carefulli": 15, "carlo": [2, 8, 17, 20, 22, 23], "carri": [4, 8, 9], "cart": 12, "case": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15, 16, 17, 18, 23], "casella": 22, "cast": 3, "cat": [5, 6], "catch": 2, "categor": [2, 3, 5, 11, 13, 23], "categori": [2, 3, 5, 9, 12, 14, 16, 23], "categorical_crossentropi": [3, 5], "caus": [2, 7, 8, 20, 23], "causal": 2, "causat": [2, 23], "cax": 3, "cb": [8, 23], "cbar": 3, "cc": [2, 3, 7, 15, 23], "ccc": [7, 14], "cdf": 20, "cdot": [2, 4, 8, 14, 15, 16, 18, 20, 23], "celebr": 15, "cell": 6, "center": [2, 3, 8, 9, 10, 11, 13, 16, 20, 23], "central": [1, 2, 5, 7, 8, 10, 18, 23], "centroid": [16, 20], "centroid_differ": 16, "centuri": 5, "certain": [2, 5, 8, 9, 11, 20, 23], "cg": 15, "cha": 2, "chain": [2, 3, 15, 17, 20, 23], "challeng": 0, "chanc": [3, 7, 15, 20], "chang": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 15, 16, 18, 20, 23], "channel": 5, "chapter": [1, 2, 8, 12, 13, 18, 22, 23], "chapter3": 2, "charact": [2, 5, 7, 23], "character": [10, 11, 12, 14, 20], "characterist": [2, 3, 5, 12, 15, 23], "charg": [2, 23], "charl": 2, "chase": 6, "chatgpt": 0, "chd": 9, "chddata": 9, "cheap": 7, "cheaper": [3, 15], "check": [0, 1, 2, 3, 5, 6, 7, 13, 15, 18, 23], "checkmark": 5, "checkpoint": 6, "checkpoint_dir": 6, "checkpoint_prefix": 6, "chen": 12, "chiaramont": 4, "childcar": 1, "children": 1, "choic": [2, 3, 4, 5, 6, 8, 11, 14, 15, 16, 18, 23], "choleski": [7, 18], "choos": [0, 4, 5, 8, 11, 12, 13, 15, 16], "chosen": [1, 2, 3, 4, 8, 10, 11, 12, 15, 20, 23], "chosen_datapoint": 3, "christian": 22, "christoph": [22, 23], "cifar": 5, "cifar10": 5, "circ": [3, 14], "circl": [2, 10, 14], "circuit": 5, "circumfer": 11, "circumv": [3, 7, 15], "ckpt": 6, "clariti": 20, "class": [2, 3, 5, 6, 8, 9, 10, 11, 13, 14, 15, 20, 23], "class_nam": [5, 11], "class_val": 11, "class_valu": 11, "classic": [9, 11, 15], "classif": [2, 5, 7, 8, 9, 10, 13, 14, 17, 22, 23], "classifi": [2, 3, 6, 9, 11, 12, 13, 23], "classificaton": 3, "classifii": 12, "clean": 3, "clear": [3, 7, 12, 14, 15], "clearli": [2, 5, 7, 8, 9, 10, 20], "clever": [3, 12], "clf": [2, 8, 10, 11, 12, 23], "clf3": 2, "clf_lasso": 8, "clf_ridg": 8, "cli": 0, "clip": [5, 20], "clone": [0, 21], "close": [2, 3, 4, 6, 8, 10, 11, 13, 14, 15, 16, 20, 22, 23], "closer": [5, 7, 15], "closest": [10, 13, 15, 16], "closur": [17, 23], "cloud": [17, 23], "cluster": [2, 3, 6, 8, 13, 17, 23], "cluster_label": 16, "cm": [3, 4, 5, 8, 10, 15], "cmap": [2, 3, 4, 5, 6, 8, 10, 11, 12, 23], "cmap_arg": 8, "cmd": [0, 11], "cn_": 20, "cnn": 14, "cnn_kera": 5, "cntk": [17, 23], "co": [2, 4, 5, 8, 11, 15, 23], "code": [5, 6, 8, 9, 10, 17, 18, 20, 22], "coef": [2, 23], "coef0": 10, "coef_": [1, 2, 7, 8, 10, 11, 15, 23], "coeff": 7, "coeffici": [2, 5, 7, 8, 9, 10, 11, 15, 18, 23], "coerc": [2, 8, 23], "coin": [12, 20], "coin_toss": 12, "col": [2, 13, 23], "colab": [17, 23], "cold": 11, "colinear": 2, "collaps": 10, "collect": [4, 8, 12, 13, 17, 20, 22, 23], "collinear": 7, "color": [2, 5, 6, 8, 10, 11, 12, 20], "color_channel": 5, "color_cod": 8, "colorbar": [3, 8], "colsample_bytre": 12, "colsaobject": 12, "column": [1, 2, 3, 4, 7, 8, 9, 10, 11, 13, 14, 18, 23], "columntransform": 11, "com": [0, 1, 6, 8, 17, 22, 23], "combin": [3, 4, 7, 8, 9, 12, 20], "come": [0, 2, 3, 5, 6, 7, 14, 15, 16, 23], "command": [0, 2, 3], "comment": [2, 6, 7, 8], "commerci": [2, 17, 23], "commit": 0, "commod": [2, 23], "common": [1, 2, 3, 5, 7, 8, 9, 11, 13, 15, 16, 20, 23], "commonli": [2, 3, 6, 8, 9, 11, 15, 16], "commun": [2, 14], "commut": 5, "commutatitav": 5, "compact": [2, 3, 5, 7, 8, 9, 11, 13, 14, 15, 16, 23], "compair": 2, "compar": [2, 5, 6, 7, 8, 13, 15, 18, 23], "comparison": [4, 6, 15], "compat": 9, "compet": 2, "competit": 12, "compil": [2, 3, 5, 6, 15, 17, 18, 23], "complet": [0, 1, 2, 4, 5, 6, 11, 14, 23], "completenn": 14, "complex": [1, 3, 7, 10, 11, 13, 14, 15, 23], "complic": [2, 3, 11, 15, 23], "compon": [1, 2, 3, 5, 6, 7, 8, 9, 11, 16, 17, 23], "components_": 13, "compos": [11, 14, 15, 16, 17, 23], "compphys": [1, 2, 8, 17, 19, 21, 22, 23], "compress": [2, 23], "compris": 8, "compromis": 7, "compulsori": [17, 23], "comput": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 18, 19, 20, 22, 23], "computation": [2, 5, 8, 11, 15, 20, 23], "computationalscienceuio": 23, "concaten": [4, 6, 8, 16], "concav": [3, 15], "concentr": [2, 12], "concept": [2, 4, 17, 23], "conceptu": [14, 15], "concern": [2, 3, 6, 9, 23], "concic": 23, "conclud": [2, 7, 15], "conclus": 3, "cond": 4, "conda": [2, 3, 17, 23], "condit": [2, 4, 6, 7, 8, 10, 11, 13, 15, 20, 23], "conduct": 17, "condwav": 4, "confid": [2, 7, 8, 9, 10, 23], "configur": 5, "confirm": [7, 14], "confus": [7, 8, 9, 12, 18], "confusion_matrix": 11, "congruenti": 20, "conjug": [6, 10], "conjugaci": 15, "conjunct": 5, "connect": [2, 3, 5, 6, 11, 13, 14, 15, 18, 23], "consequ": [7, 8, 10, 12, 14, 15], "conserv": [7, 16], "consid": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 23], "consider": [2, 3, 7, 15, 23], "consist": [2, 3, 4, 5, 6, 8, 14, 15, 20], "constant": [1, 2, 4, 6, 7, 8, 10, 14, 15, 20, 23], "constitu": [2, 23], "constitut": [4, 8], "constrain": [3, 5, 7, 9, 13], "constraint": [7, 8, 10, 15], "construct": [2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 18, 20, 23], "contact": [2, 23], "contain": [0, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 22, 23], "contemporari": 23, "content": [0, 3, 17, 18, 23], "context": [8, 12, 15], "contigu": 18, "continu": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 23], "contour": [11, 12, 15], "contourf": [10, 11, 12], "contrast": [3, 6, 11, 12, 14, 23], "contribut": [2, 5, 7, 15, 20, 23], "contributor": 2, "control": [0, 2, 3, 5, 11, 15, 17, 23], "conv": [5, 6], "conv2d": [5, 6], "conv2dtranspos": 6, "convei": 23, "conveni": [7, 8, 14, 15, 18, 23], "convent": 14, "converg": [3, 4, 6, 7, 10, 15, 16, 23], "convergencewarn": 23, "convert": [2, 3, 6, 7, 11, 13, 15, 18, 23], "converttomatrix": 6, "convex": [6, 7, 9], "convinc": 15, "convolut": [3, 6, 17, 23], "cool": [6, 11], "coolwarm": 8, "coordin": [7, 14, 16], "coorel": 2, "copi": [0, 2, 3, 16], "core": 12, "corel": 23, "coronari": 9, "corr": [2, 7, 9, 13], "correalt": [13, 17], "correct": [0, 2, 3, 4, 5, 6, 7, 9, 15, 18, 20, 23], "correctli": [3, 4, 8, 9, 12], "correl": [2, 3, 5, 7, 8, 9, 12, 14, 15, 17, 20, 23], "correlation_matrix": [2, 7, 9, 13], "correspond": [2, 5, 7, 8, 10, 11, 13, 14, 17, 18, 20, 23], "cortex": 14, "cosin": [5, 8], "cost": [1, 2, 4, 5, 7, 8, 9, 10, 11, 14, 15, 23], "cost_deep_grad": 4, "cost_funct": 4, "cost_function_deep": 4, "cost_function_deep_grad": 4, "cost_function_grad": 4, "cost_grad": 4, "cost_sum": 4, "costol": 15, "could": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "coulomb": [2, 23], "count": [0, 2, 11, 19, 20, 21, 23], "counterpart": 23, "countor": 15, "coupl": [6, 7, 8], "cours": [0, 1, 2, 3, 5, 7, 13, 21], "coursework": 0, "courvil": [22, 23], "cov": [7, 8, 13, 18, 20, 23], "cov_xi": [7, 13], "cov_xx": [7, 13], "cov_yi": [7, 13], "covari": [2, 9, 17, 18, 23], "covariance_matrix": [7, 13, 16], "cover": [2, 7, 17, 21, 22], "covert": [2, 23], "covxi": 20, "covxx": 20, "covxz": 20, "covyi": 20, "covyz": 20, "covzz": 20, "cpu": 3, "craft": 5, "creat": [0, 3, 5, 6, 7, 11, 12, 13, 14, 17, 23], "create_biases_and_weight": 3, "create_convolutional_neural_network_kera": 5, "create_neural_network_kera": 3, "create_x": [7, 13], "credit": [2, 9, 21, 23], "crim": 2, "crime": 2, "criteria": [2, 6, 11, 12, 16, 20, 23], "criterion": [11, 12, 15], "critic": 8, "cross": [0, 2, 3, 5, 9, 11, 12, 15, 17, 20, 23], "cross_entropi": 6, "cross_val_scor": 8, "cross_valid": [9, 12], "crossvalid": 8, "crucial": [3, 20], "cs231": 5, "csr_matrix": [18, 23], "csv": [2, 6, 8, 9, 11], "ctnk": 3, "cubic": 2, "cumbersom": 7, "cumsum": [12, 13, 23], "cumul": [9, 12, 20], "cumulative_heads_ratio": 12, "cup": 7, "current": [0, 1, 3, 4, 5, 6, 15, 16, 22], "curs": 2, "curv": [8, 9, 12, 14], "curvatur": 15, "custom": [8, 16], "custom_cmap": [11, 12], "custom_cmap2": [11, 12], "cutpoint": 11, "cv": [8, 9, 12], "cvxbook": 15, "cvxopt": [7, 10], "cycl": [3, 14], "d": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 23], "d2_g_t": 4, "d_f": 15, "d_g_t": 4, "d_net_out": 4, "da": 5, "dagger": [7, 18], "dai": [3, 11, 17], "damp": 5, "darget": 11, "darkr": 20, "dat": [2, 23], "dat_id": [2, 8, 9, 11, 23], "data": [1, 4, 6, 7, 10, 12, 14, 15, 16, 18, 22], "data1": 16, "data2": 16, "data3": 16, "data4": 16, "data_id": [2, 8, 9, 11, 23], "data_indic": 3, "data_panda": 23, "data_path": [2, 8, 9, 11, 23], "databas": 3, "datafil": [2, 8, 9, 11, 23], "datafram": [2, 6, 7, 9, 11, 13, 23], "datapoint": [1, 3, 7, 8, 9, 13, 15], "datasci": [0, 1], "dataset": [1, 2, 6, 8, 9, 10, 11, 12, 13, 15, 16, 23], "datatyp": 6, "date": [0, 23], "daughter": 12, "david": 22, "dbh": 3, "dbo": 3, "dcomposit": 18, "ddot": 4, "dead": 3, "deadlin": 0, "deal": [2, 3, 5, 7, 8, 10, 13, 15, 16, 18, 20, 23], "dealt": 2, "debt": 9, "debug": [2, 7, 8], "decad": [2, 5], "decai": [2, 15, 20, 23], "decemb": [21, 23], "decent": 12, "decid": [2, 4, 5, 7, 8, 11], "decim": [2, 23], "decis": [2, 3, 10, 13, 17, 22, 23], "decision_funct": 10, "decision_tre": 11, "decisiontreeclassifi": [11, 12], "decisiontreeregressor": [2, 11, 12], "declar": [2, 6, 18, 23], "decompos": [7, 8, 18], "decomposit": [2, 8, 14, 23], "decompost": 7, "deconvolut": 5, "decorrel": [12, 15], "decreas": [3, 4, 6, 7, 8, 12, 13, 15], "deduc": [2, 23], "deep": [5, 9, 14, 15, 17, 22], "deep_neural_network": 4, "deep_param": 4, "deep_tree_clf": [11, 12], "deep_tree_clf1": 11, "deep_tree_clf2": 11, "deepen": [7, 17, 23], "deeper": [2, 5, 6, 23], "deeplearningbook": [22, 23], "deer": 5, "def": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 20, 23], "def_covari": 20, "default": [2, 3, 4, 6, 8, 9, 18, 23], "default_tim": 6, "defect": 7, "defici": 7, "defin": [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20], "definit": [3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20], "defint": 20, "degre": [0, 1, 5, 7, 8, 10, 11, 12, 13, 20, 23], "del": 3, "delet": [0, 8], "delimit": 6, "deliv": [0, 19, 23], "delta": [2, 4, 5, 8, 10, 14, 15, 16, 23], "delta_": [3, 18], "delta_0": 5, "delta_1": 5, "delta_2": 5, "delta_3": 5, "delta_4": 5, "delta_5": 5, "delta_h": [2, 3, 23], "delta_j": [5, 14], "delta_k": 14, "delta_l": [3, 5], "delta_momentum": 15, "delta_n": [2, 5, 23], "delug": 17, "delv": 2, "demand": 15, "demonstr": [2, 5, 7, 8, 9, 13, 14, 17, 23], "den": 6, "denomin": [3, 7], "denot": [3, 4, 8, 9, 15, 20], "dens": [3, 5, 6], "densiti": [2, 4, 8, 20], "depart": [21, 23], "depend": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 20, 23], "depict": 20, "deploy": [2, 17, 23], "depth": [2, 5, 11, 12, 18], "deriv": [2, 3, 4, 8, 9, 10, 12, 13, 15, 17, 23], "derivati": 15, "derivative_fn": 15, "descend": [7, 11, 13], "descent": [2, 3, 5, 9, 10, 14, 23], "describ": [2, 4, 6, 7, 8, 10, 12, 13, 14, 15, 18, 23], "descript": [2, 10, 11, 23], "design": [2, 3, 5, 6, 7, 8, 9, 12, 13, 14, 15, 23], "designmatrix": [2, 23], "desir": [2, 4, 6, 7, 15, 16, 23], "desktop": 0, "despit": [3, 14], "destroi": 18, "det": [7, 18], "detail": [2, 8, 13, 15, 16, 18], "detect": [5, 10, 14], "determin": [2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 18, 20, 23], "determinist": [9, 15, 20], "dev": 3, "develop": [2, 5, 7, 10, 12, 13, 14, 17, 18, 23], "deviat": [2, 3, 4, 6, 7, 8, 20, 23], "devis": 14, "df": [6, 10, 13, 15, 23], "df1": 23, "di": 2, "diag": [7, 10], "diagnost": [3, 12], "diagon": [2, 7, 9, 15, 18, 20, 23], "diagonaliz": 7, "diagram": 12, "diagsvd": 8, "dice": [8, 20], "dict": [8, 10], "dictionari": 2, "did": [1, 2, 3, 7, 8, 9, 12, 13, 16, 23], "die": 3, "diff": 4, "diff1": 4, "diff2": 4, "diff_ag": 4, "diffeent": 10, "differ": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23], "differenti": [1, 2, 5, 17, 18, 23], "difficult": [2, 3, 8, 12, 15, 20, 23], "difficulti": [2, 3, 15, 23], "diffonedim": 4, "digit": [2, 3, 5, 6, 8, 21, 23], "dilemma": 15, "dilut": 3, "dim": [6, 13, 16, 18], "dimens": [1, 2, 3, 4, 5, 6, 7, 10, 13, 16, 18, 23], "dimension": [2, 6, 7, 8, 11, 13, 15, 16, 17, 18, 23], "dimensionless": [2, 5, 23], "diment": 18, "dimnsion": 6, "diod": 5, "direct": [2, 3, 4, 6, 13, 14, 15, 16, 23], "directli": [3, 6, 7, 8, 20], "disadvantag": [2, 23], "disappear": [5, 8], "disc_loss": 6, "disc_tap": 6, "discard": [8, 13], "disciplin": [2, 5, 14], "disclaim": 20, "discord": 23, "discourag": [0, 15], "discov": [2, 23], "discover": 7, "discret": [3, 5, 7, 9, 15], "discrimin": [6, 9, 12, 13], "discriminator_loss": 6, "discriminator_loss_list": 6, "discriminator_model": 6, "discriminator_optim": 6, "discuss": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23], "diseas": 9, "disguis": 8, "disord": [3, 9], "displai": [2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 20, 23], "displaystyl": [2, 7, 23], "disregard": [2, 23], "dissimilar": [13, 16], "dist": 16, "distanc": [2, 10, 11, 13, 16, 20], "distance_list": 11, "distinct": [5, 9, 10, 11, 12, 16], "distinctli": 10, "distinguish": [2, 6, 9, 10, 20, 23], "distplot": 2, "distribut": [2, 3, 6, 8, 9, 12, 13, 15, 16, 17, 18, 23], "distrubut": [2, 17, 23], "dive": [2, 10, 18, 23], "diverg": [3, 15], "divid": [2, 3, 5, 7, 8, 9, 10, 11, 13, 14, 20, 23], "divis": [8, 10, 11, 15, 18, 20], "dna": 9, "dnn": [2, 3, 4, 6, 14, 23], "dnn1": 6, "dnn2_gru2": 6, "dnn_kera": 3, "dnn_model": 3, "dnn_numpi": 3, "dnn_scikit": [2, 3, 23], "do": [0, 1, 2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 18, 23], "doc": [0, 1, 2, 17, 19, 21, 22, 23], "document": [0, 6, 15], "doe": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 18, 20, 23], "doesn": [5, 11, 14, 23], "dog": [3, 5, 6], "domain": [7, 10, 15], "domin": [2, 23], "don": [0, 1, 2, 3, 5, 7, 8, 10, 13, 15, 17, 23], "done": [1, 2, 4, 5, 6, 7, 8, 11, 12, 13, 15, 18, 23], "dot": [2, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "doubl": [1, 5, 6, 18, 23], "doubli": 3, "down": [2, 5, 8, 11, 13, 14, 15], "download": [0, 2, 3, 5, 7, 8, 18, 22, 23], "downsampl": 5, "dozen": 3, "dq": 8, "drag": 15, "dramat": 13, "drastic": 6, "draw": [6, 8, 12, 15], "drawback": [2, 3, 5, 15], "drawn": [3, 6, 8, 9, 13, 20, 23], "drive": [5, 6], "driven": 5, "drop": [2, 3, 7, 8, 13, 15, 20, 23], "dropna": [2, 8, 23], "dropout": 6, "dt": [4, 5, 15, 20], "dtype": [2, 3, 5, 6, 16, 18, 23], "dub": [2, 23], "due": [3, 4, 7, 8, 10, 12, 14, 15, 21, 23], "dummi": 2, "dure": [2, 3, 5, 6, 10, 11, 13, 17, 23], "dwell": 2, "dwh": 3, "dwo": 3, "dx": [4, 5, 10, 20], "dx_1": 20, "dx_1p": 8, "dx_2p": 8, "dx_mp": 8, "dx_n": 20, "dxp": 8, "dy": [3, 10, 20], "dynam": 6, "dz": 10, "e": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 21, 23], "e_": [2, 4, 23], "each": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23], "eapprox": [2, 23], "earli": [3, 15], "earlier": [2, 7, 9, 10, 11, 13, 14, 15, 23], "earthexplor": 8, "eas": [8, 11, 16], "easi": [0, 2, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 23], "easier": [0, 7, 8, 10, 11, 15, 20, 23], "easiest": 15, "easili": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 23], "eastern": [21, 23], "ebind": [2, 23], "eblock": 11, "econometr": 23, "economi": 7, "ecosystem": [17, 23], "ect": 19, "edg": 5, "edgecolor": 8, "edu": 15, "educ": [2, 23], "eff": 20, "effect": [1, 3, 6, 12, 15, 20], "effic": 3, "effici": [2, 5, 12, 15, 17, 18, 20, 23], "efron": 8, "egrad": 15, "eig": [7, 13, 15, 18, 20, 23], "eigen": 20, "eigenpair": [7, 13], "eigenvalu": [2, 7, 10, 13, 15, 18, 23], "eigenvector": [7, 13, 15], "eight": [18, 23], "eigval": [18, 20, 23], "eigvalu": [13, 15], "eigvec": [18, 20, 23], "eigvector": [13, 15], "eir": [21, 23], "eispack": [18, 23], "either": [3, 7, 8, 9, 10, 11, 12, 13, 15, 20, 23], "eivind": 21, "eivinsto": 21, "ekstr\u00f8m": 6, "elabor": 20, "elarn": 5, "electr": [2, 5, 14, 23], "electron": 23, "eleg": 13, "element": [3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 22], "elementari": [12, 15, 18], "elementwis": [5, 15], "elementwise_grad": [4, 15], "elessar": 23, "elif": 16, "elim": 18, "elimin": [5, 10], "elin": [21, 23], "ellipsi": 1, "els": [1, 3, 5, 6, 9, 11, 14, 15, 18], "elu": 3, "elus": [2, 23], "email": [19, 21, 23], "embed": [2, 13], "embodi": 8, "emit": 20, "emner": 22, "emphas": [2, 12, 17, 23], "emphasi": [2, 17, 22, 23], "empir": [3, 13, 20], "emploi": [2, 3, 7, 8, 13, 15, 20, 23], "employ": 2, "empti": [0, 8, 12], "emul": 14, "en": [17, 22], "enabl": 13, "enbodi": 8, "encod": [2, 5, 7, 11, 13, 16, 23], "encompass": [2, 20], "encount": [0, 2, 3, 7, 9, 15, 20, 23], "encourag": 0, "end": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23], "endpoint": [5, 8], "energi": [2, 6, 8], "enforc": 14, "eng": 22, "engin": [2, 3, 5, 6, 17, 23], "enorm": 5, "enough": [2, 8, 15, 23], "ensembl": [3, 11, 23], "ensur": [2, 3, 4, 5, 7, 8, 13, 15, 20], "entail": 23, "enter": [7, 8], "enthought": [2, 17, 23], "entir": [3, 5, 9, 11, 17, 20, 23], "entiti": [11, 14, 18, 23], "entri": [2, 7, 10, 13, 14, 18, 23], "entropi": [3, 5, 9, 12, 15, 23], "enumer": [2, 3, 4, 5, 6, 8, 10, 23], "env": [20, 23], "environ": [4, 17, 23], "environemnt": 0, "eo": [2, 8], "eol": 2, "eosfit": 2, "epoch": [2, 3, 5, 6, 14, 15, 23], "epsilon": [2, 7, 8, 9, 15, 23], "epsilon_": [2, 23], "epsilon_0": [2, 23], "epsilon_1": [2, 23], "epsilon_2": [2, 23], "epsilon_i": [2, 23], "eq": [5, 15, 16, 18, 20], "eqnarrai": [5, 7, 8], "equal": [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 15, 16, 18, 20, 23], "equat": [3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23], "equilibrium": [4, 14], "equiv": [5, 15, 18, 20], "equival": [2, 3, 7, 9, 10, 13, 15, 17, 18, 23], "erf": 20, "eriador": 23, "err": [2, 12], "err_": 8, "err_sqr": 4, "errat": 15, "erron": 4, "error": [0, 1, 3, 4, 6, 7, 8, 9, 11, 13, 14, 15, 17, 18, 20], "error_estimate_corr_tim": 20, "error_hidden": 3, "error_output": 3, "escap": 15, "especi": [0, 3, 5, 11, 14, 15], "essenti": [0, 2, 7, 8, 11, 12, 14, 16, 20], "establish": [1, 2, 8, 12, 13], "estim": [2, 3, 7, 8, 9, 12, 13, 15, 17, 20, 23], "estimated_mse_fold": 8, "estimated_mse_kfold": 8, "estimated_mse_sklearn": 8, "et": [1, 2, 4, 6, 22, 23], "eta": [2, 3, 5, 10, 14, 15, 23], "eta0": [10, 15], "eta_": 15, "eta_t": 15, "eta_v": [2, 3, 5, 23], "etc": [2, 3, 5, 7, 9, 10, 11, 13, 14, 15, 16, 17, 18, 20], "ethic": 17, "euclidean": [2, 16], "evalu": [0, 1, 2, 4, 5, 6, 7, 8, 11, 15, 20, 23], "evalut": 15, "even": [2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "evenli": 6, "event": [7, 9, 12, 20], "eventu": [2, 7, 8, 13, 14, 15, 21], "everi": [0, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 17, 20, 21, 23], "everyth": [1, 6, 14], "everywher": [6, 15], "evolv": 2, "exact": [2, 7, 13, 14, 15, 18, 20, 23], "exactli": [2, 5, 6, 8, 14, 17], "exam": 23, "examin": 8, "exampl": [0, 1, 7, 13, 14, 15, 17, 18, 20, 22], "exce": [3, 14, 15], "excel": [2, 3, 6, 7, 12, 23], "except": [5, 6, 8, 10, 11, 18], "excess": [2, 23], "excit": 2, "exclud": [3, 8, 14], "exclus": [2, 3, 5, 8, 20, 23], "execut": [0, 4, 7, 15], "exemplifi": 15, "exercic": [21, 23], "exercis": [7, 17, 19, 21, 23], "exhaust": 8, "exhibit": [2, 7, 8, 10, 23], "exist": [2, 3, 4, 5, 7, 8, 9, 10, 11, 15, 18, 23], "exit": [7, 18], "exp": [1, 2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 20], "exp_term": 3, "expand": [7, 9, 13, 15], "expans": [2, 5, 7, 10, 12, 14, 15, 23], "expect": [0, 2, 3, 7, 8, 9, 13, 14, 15, 17, 23], "expectation_value_of_h_wrt_p": 20, "expens": [1, 8, 12, 15], "experi": [0, 2, 3, 8, 10, 15, 17, 23], "experiment": [2, 6, 8, 11, 20, 23], "expert": [3, 11], "explain": [1, 2, 8, 11, 12, 13, 15, 23], "explained_variance_ratio_": 13, "explanatori": [2, 23], "explicit": [2, 5, 8, 15, 18, 23], "explicitli": [2, 6], "explod": 3, "exploit": [2, 5, 14, 15, 23], "explor": [3, 6, 8, 10, 15, 17, 23], "expon": 3, "exponenti": [2, 3, 7, 8, 12, 15, 20, 23], "export": [0, 1, 11], "export_graphviz": 11, "export_text": 11, "exporttext": 11, "expos": 17, "express": [2, 4, 5, 7, 8, 9, 12, 14, 15, 18, 20, 23], "exptmean": 20, "exptvari": 20, "extend": [2, 4, 9, 13, 15, 17, 23], "extens": [0, 2, 14, 17, 23], "extent": [2, 3, 8, 22], "extern": [5, 8, 11], "extra": [0, 3, 5, 7, 21, 23], "extract": [1, 2, 5, 7, 8, 9, 10, 13, 15, 18, 23], "extrapol": [2, 23], "extrem": [0, 1, 2, 3, 6, 7, 8, 9, 10, 11, 15, 18], "extremum": 15, "extrins": 13, "ey": [2, 7, 8, 15, 16, 18, 23], "f": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 15, 16, 18, 20, 21, 23], "f1": 15, "f11": [2, 23], "f12": [2, 23], "f13": [2, 23], "f1_grad": 15, "f1d": 15, "f2": 15, "f2_grad_x1": 15, "f2_grad_x1_analyt": 15, "f2_grad_x2": 15, "f2_grad_x2_analyt": 15, "f3": 15, "f3_grad": 15, "f3_grad_analyt": 15, "f4": 15, "f4_grad": 15, "f4_grad_analyt": 15, "f5": 15, "f5_grad": 15, "f6": 15, "f6_for": 15, "f6_for_grad": 15, "f6_grad_analyt": 15, "f6_while": 15, "f6_while_grad": 15, "f7": 15, "f7_grad": 15, "f7_grad_analyt": 15, "f8": 15, "f8_grad": 15, "f9": [2, 15, 23], "f9_altern": 15, "f9_alternative_grad": 15, "f9_grad": 15, "f_": 12, "f_0": [5, 12], "f_1": [12, 15], "f_2": [14, 15], "f_3": 14, "f_d": 20, "f_grad": 15, "f_grad_analyt": 15, "f_i": [1, 2, 8, 14], "f_m": [5, 12], "f_n": 5, "f_vec": 4, "face": [15, 23], "facecolor": [8, 10, 20], "facil": [2, 17], "facilit": 14, "fact": [2, 3, 5, 7, 11, 13, 14, 15, 23], "factor": [2, 3, 5, 7, 8, 11, 12, 13, 15, 18, 20, 23], "factori": 15, "fade": 8, "fafab0": [11, 12], "fail": [2, 8, 15, 21, 23], "failur": 9, "fairli": [3, 4, 20], "faisal": 1, "fake": 6, "fake_loss": 6, "fake_output": 6, "fall": [10, 11, 19], "fals": [1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 16, 18, 23], "famili": [2, 9, 10, 20], "familiar": [0, 2, 5, 7, 8, 10, 17, 18, 20, 23], "famou": [8, 14], "far": [1, 2, 5, 6, 7, 8, 10, 13, 14, 15, 16, 23], "fashion": [2, 11, 12, 23], "fast": [3, 5, 8, 12, 14, 15, 17, 20, 23], "faster": [3, 13, 15], "fastest": [15, 18], "favor": 9, "favorit": 20, "fc": 5, "featur": [0, 2, 3, 5, 7, 8, 9, 10, 12, 13, 14, 15, 17, 20, 23], "feature_nam": [2, 3, 9, 11], "feautur": 11, "fed": 3, "feed": [2, 4, 5, 13, 17, 23], "feed_forward": 3, "feed_forward_out": 3, "feed_forward_train": 3, "feedback": [6, 23], "feeddorward": 6, "feedforward": [3, 6, 14], "feel": [0, 1, 2, 7, 8, 13, 15, 17, 21, 23], "feet": 2, "fetch": [0, 8], "few": [3, 5, 6, 7, 11, 20, 23], "fewer": [2, 11, 13, 23], "ffnn": [3, 14], "field": [2, 5, 8, 14, 17], "fifth": [2, 8, 23], "fig": [2, 3, 4, 5, 6, 8, 9, 14, 15, 16, 23], "fig_id": [2, 8, 9, 11, 23], "figaxi": 20, "figsiz": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 23], "figur": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 23], "figure_id": [2, 8, 9, 11, 23], "figurefil": [2, 8, 9, 11, 23], "file": [0, 2, 6, 7, 8, 9, 11, 23], "file_prefix": 6, "filenam": 23, "fill": [7, 11], "filter": [5, 6], "final": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 19, 20, 21, 23], "financ": 2, "find": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23], "fine": [2, 16], "finish": 4, "finit": [5, 7, 8, 14, 15, 20], "finnicki": 0, "first": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 22], "firsteigvector": 13, "fit": [3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 20], "fit_intercept": [1, 2, 7, 8], "fit_mod": 11, "fit_theta": 8, "fit_transform": [0, 2, 8, 10, 11, 13], "fiti": [2, 23], "five": [2, 11, 23], "fix": [2, 5, 6, 8, 12, 13, 14, 15, 23], "flag": 6, "flat": [14, 15], "flatten": [3, 5, 6, 7, 18], "flexibl": [3, 8, 10, 12, 14, 23], "flip": [21, 23], "float": [2, 5, 6, 7, 11, 13, 15, 16, 18, 23], "float32": [6, 11], "float64": [6, 18, 23], "flop": [7, 18], "flow": [3, 6, 14], "fluctuat": 7, "fly": 13, "fm": 2, "fmax": 5, "fmesh": 15, "fn": 9, "focu": [0, 2, 5, 6, 7, 8, 17, 22, 23], "focus": [3, 8, 9, 18], "fold": [8, 11], "folder": [0, 2, 6, 8, 23], "follow": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 23], "font": [9, 20, 23], "fontdict": 20, "fontsiz": [3, 8, 10, 11, 12, 20], "fontweight": 3, "footprint": 5, "foral": 10, "forc": [2, 7, 8, 12, 13], "forcast": 6, "forecast": [6, 14], "forest": [2, 3, 11, 17, 23], "forget": 13, "form": [0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23], "formal": [5, 6, 16, 20], "format": [2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 17, 20, 22], "format_data": 6, "formatstrformatt": [8, 15], "formul": [6, 8, 13, 16], "formula": [5, 15, 20], "forth": [6, 14], "fortran": [2, 17, 18, 23], "fortran2003": [17, 23], "fortran90": 20, "fortun": [2, 13], "forward": [2, 5, 8, 17, 18, 23], "found": [3, 4, 6, 7, 8, 14, 15, 23], "foundat": [17, 23], "four": [6, 7, 8, 10, 14, 18, 19, 21, 23], "fourier": [2, 23], "fourierdef1": 5, "fourierdef2": 5, "fourierseriessign": 5, "fourth": [14, 23], "fp": 9, "frac": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "fraction": 11, "frame": 9, "framework": [3, 10, 12, 20], "frank": [7, 13], "frankefunct": [7, 8, 13], "fredli": [21, 23], "free": [0, 1, 2, 8, 13, 15, 17, 18, 20, 21, 22, 23], "freecodecamp": 17, "freedom": 7, "freeli": 2, "freez": 0, "frequenc": [5, 8, 9, 20], "frequent": [2, 10, 11, 15], "frequentist": 17, "fresh": 12, "fridai": [0, 21, 23], "friedman": [8, 22, 23], "friendli": 6, "frodo": 23, "frog": 5, "from": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 15, 16, 17, 18, 20, 21, 22], "from_cod": 11, "from_logit": [5, 6], "from_tensor_slic": 6, "front": [2, 6, 7, 23], "frustrat": 0, "fulfil": [4, 7, 14], "full": [2, 3, 5, 7, 9, 11, 12, 15, 20, 23], "full_matric": 7, "fulli": [5, 8, 14, 20], "fun": [17, 23], "func": 4, "function": [0, 1, 4, 5, 6, 7, 11, 16, 17, 18], "functionali": 13, "fundament": [2, 8, 17, 23], "funtion": 4, "further": [4, 9, 11, 23], "furthermor": [2, 5, 7, 8, 9, 13, 14, 15, 17, 23], "futur": [2, 6, 10, 11, 23], "fy": [0, 19, 21, 22, 23], "fys5419": [22, 23], "fys5429": [22, 23], "f\u00f8470": [21, 23], "g": [0, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 20, 23], "g0": 4, "g_": [4, 11, 12], "g_0": 4, "g_1": [4, 12], "g_2": [4, 12], "g_analyt": 4, "g_dnn_ag": 4, "g_euler": 4, "g_i": 4, "g_m": [5, 12], "g_n": 5, "g_re": 4, "g_t": 4, "g_t_d2t": 4, "g_t_d2x": 4, "g_t_dt": 4, "g_t_hessian": 4, "g_t_hessian_func": 4, "g_t_jacobian": 4, "g_t_jacobian_func": 4, "g_trial": 4, "g_trial_deep": 4, "g_vec": 4, "gain": [3, 7, 9, 11, 12, 15], "galleri": [2, 23], "game": 6, "gamge": 23, "gamma": [2, 4, 10, 11, 12, 13, 15, 23], "gamma1": 10, "gamma2": 10, "gamma_": [2, 23], "gamma_0": 12, "gamma_1": 12, "gamma_1x": 12, "gamma_i": [2, 10, 20, 23], "gamma_j": 15, "gamma_k": 15, "gamma_m": 12, "gamma_x": [2, 23], "gap": 10, "gate": [6, 14], "gather": [2, 3, 14], "gaug": 14, "gaussbacksub": 18, "gaussian": [6, 7, 8, 10, 16, 20, 23], "gaussian_point": 16, "gaussian_rbf": 10, "gave": 15, "gavra": 23, "gbc": 23, "gca": [4, 8, 10, 15], "gd": 3, "gd_clf": 12, "gdclassiffiercgain": 12, "gdclassiffierconfus": 12, "gdclassiffierroc": 12, "gdm": 15, "gdregress": 12, "ge": [3, 7, 9, 20], "gen_loss": 6, "gen_tap": 6, "gender": [2, 23], "genener": 6, "gener": [0, 1, 2, 3, 4, 5, 7, 8, 10, 12, 13, 14, 15, 16, 18, 20, 22], "generaliz": 1, "generallay": 14, "generate_and_save_imag": 6, "generate_imag": 6, "generate_latent_point": 6, "generate_simple_clustering_dataset": 16, "generated_imag": 6, "generator_loss": 6, "generator_loss_list": 6, "generator_model": 6, "generator_optim": 6, "genom": 17, "geodes": 13, "geometr": [2, 15, 23], "geometri": 7, "georg": 22, "geotif": 8, "geq": [4, 7, 10, 11, 15], "geron": [2, 22, 23], "get": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 17, 18, 20, 21, 23], "get_dummi": 11, "get_paramet": 4, "get_split": 11, "get_yaxi": 10, "get_yticklabel": 8, "gh": 0, "gibb": [17, 23], "gif": 6, "gini": 12, "gini_index": 11, "ginvers": 15, "git": [0, 2, 17, 23], "giter": 15, "github": [2, 17, 19, 21, 22, 23], "gitignor": 0, "gitlab": [2, 17, 23], "give": [2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 20, 23], "given": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "global": [8, 9, 15], "glorot": 3, "gnew": 15, "go": [0, 1, 2, 3, 5, 7, 8, 10, 11, 13, 14, 15, 23], "goal": [2, 9, 11, 23], "goe": [0, 2, 3, 4, 7, 8, 15, 16, 18, 23], "golden": 15, "gone": 7, "gong": 3, "good": [0, 3, 5, 6, 7, 8, 11, 12, 13, 15, 17, 20, 22], "goodfellow": [6, 22, 23], "googl": [3, 6, 17, 23], "got": [3, 8], "gotten": 23, "gov": 8, "govern": 23, "gp": 22, "gpu": [3, 15, 17, 23], "grad": [4, 15], "grad_analyt": 15, "grade": 19, "gradient": [2, 5, 6, 9, 10, 11, 14, 17, 23], "gradientboostingclassifi": 12, "gradientboostingregressor": 12, "gradients_of_discrimin": 6, "gradients_of_gener": 6, "gradienttap": 6, "gradual": [3, 16], "grai": [6, 8], "graph": [1, 3, 11, 13, 14, 15], "graph_from_dot_data": 11, "graphic": [2, 3, 11, 23], "grasp": 2, "gray_r": [3, 5], "grayscal": 5, "great": [0, 7, 15], "greater": [3, 9, 20], "greatli": 15, "greedi": 11, "green": [2, 5, 11, 20], "grei": 6, "grid": [3, 5, 8, 9, 10, 14, 20], "grossli": 15, "ground": [2, 23], "group": [0, 2, 8, 9, 11, 16, 17, 19, 21, 23], "groupbi": [2, 23], "grow": [3, 5, 11, 12], "growth": [2, 23], "gru": 6, "guarante": [2, 6, 15, 20, 23], "guess": [3, 6, 12, 15, 16], "guestrin": 12, "guid": 3, "h": [0, 2, 3, 7, 8, 10, 15, 20, 21, 22, 23], "h1": 4, "h_": [2, 15, 23], "h_1": [4, 15], "h_2": [4, 15], "h_m": 12, "ha": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "haanen": [21, 23], "habit": 2, "had": [2, 3, 8, 9, 15, 23], "hadamard": [3, 14, 15], "half": [3, 10, 11], "halv": 12, "hand": [2, 3, 4, 5, 7, 13, 14, 15, 17, 18, 20, 21, 22, 23], "handi": 5, "handl": [0, 2, 3, 4, 7, 11, 13, 17], "handle_unknown": 11, "handsid": 14, "handwrit": 14, "handwritten": [3, 7], "happen": [3, 4, 5, 6, 7, 8, 12, 15, 20], "hard": [3, 9, 10, 12, 15], "hardcopi": [17, 23], "harder": [2, 3], "harmon": 5, "hasn": 23, "hassl": [2, 17, 23], "hast": [17, 23], "hasti": [1, 2, 8, 22, 23], "hat": [1, 2, 3, 7, 8, 9, 11, 12, 13, 14, 15, 18], "have": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "haven": 3, "he": [9, 23], "head": [2, 6, 12, 20], "header": [2, 23], "heads_proba": 12, "health": 2, "hear": [2, 15, 23], "heart": [2, 9, 23], "heatmap": [2, 3, 5, 9, 23], "heavili": 2, "heavisid": 3, "height": [3, 5, 8], "held": 15, "help": [0, 1, 2, 3, 6, 14, 15, 23], "helper": [6, 16], "henc": [2, 7, 8, 10, 11, 12, 14, 15, 23], "henrik": [21, 23], "her": 9, "here": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "hereaft": [2, 10, 14, 23], "hermitian": 18, "hessenberg": 18, "hessian": [2, 4, 7, 15], "heterogen": [11, 12], "hi": 9, "hidden": [3, 5, 6, 14], "hidden_bia": 3, "hidden_bias_gradi": 3, "hidden_layer_s": [2, 3, 23], "hidden_neuron": 6, "hidden_weight": 3, "hidden_weights_gradi": 3, "hierarch": 7, "high": [2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 15, 16, 17, 18, 23], "higher": [2, 3, 5, 7, 8, 10, 15, 23], "highest": [3, 4], "highli": [2, 5, 6, 12, 17, 18, 22, 23], "highwai": 2, "hing": 10, "hint": [0, 1, 15], "hip": 17, "hire": 2, "hist": [6, 8, 9, 20], "histogram": [2, 8, 9, 20], "histor": [9, 13], "histori": [0, 5, 6, 14], "hitherto": 7, "hjorth": [21, 23], "hobbi": 20, "hoc": 7, "hoff": 22, "hold": [3, 5, 8, 15, 16], "holder": [2, 23], "home": 2, "homepag": 23, "homework": [8, 15], "homogen": [3, 5, 11, 12, 15], "honchar": 4, "hopefulli": [0, 2, 13, 20, 23], "horizont": 13, "horlyk": [21, 23], "hors": [5, 9, 23], "hot": [3, 11], "hour": [3, 17, 19, 20, 21, 23], "how": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "howev": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "hspace": [2, 6, 10, 12, 20, 23], "hstack": 3, "htf": 23, "html": [1, 2, 17, 19, 21, 22, 23], "http": [0, 1, 2, 5, 6, 8, 15, 17, 18, 19, 21, 22, 23], "huang": [2, 23], "huber": [2, 23], "huge": [3, 5, 6, 17], "human": [2, 3, 5, 8, 11, 14], "humid": 11, "hundr": 3, "hungri": 3, "hybrid": 19, "hydrogen": [2, 23], "hyperbol": [3, 6, 14], "hyperparam": 10, "hyperparamet": [5, 6, 7, 8, 11, 15], "hyperplan": 13, "h\u00f8rlyk": [21, 23], "i": [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22], "i0": [2, 23], "i1": [2, 8, 10, 14, 23], "i2": [2, 10, 14, 23], "i3": [2, 14, 23], "i5": [2, 23], "i_": 15, "i_1": [7, 8], "i_2": [7, 8], "ian": 22, "ic": 3, "id": [9, 15], "ida": [21, 23], "idea": [2, 3, 4, 5, 6, 8, 11, 12, 14, 15, 18], "ideal": [2, 4, 8, 10, 15, 20, 23], "idem": 8, "ident": [7, 8, 14, 15, 18], "identifi": [2, 3, 9, 11, 13, 14, 15, 16, 23], "ieor": 20, "ifi": 22, "ifs": [17, 23], "ignor": [0, 2, 3, 5, 11], "ii": [18, 20], "iii": [18, 23], "ij": [1, 2, 3, 5, 8, 10, 14, 16, 18, 20, 23], "ik": [2, 18, 23], "illustr": [7, 9, 12, 14, 15, 16, 17, 23], "im": 8, "imag": [3, 5, 6, 8, 11, 13, 14, 16, 22, 23], "image_at_epoch_": 6, "image_batch": 6, "image_height": 5, "image_path": [2, 8, 9, 11, 23], "image_width": 5, "imageio": 8, "images_from_seed_imag": 6, "imagin": 3, "immedi": [2, 5, 6, 8, 17, 23], "implement": [2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 20, 23], "impli": [5, 7, 8, 9, 15, 18], "implicit": 5, "implicitli": [13, 20], "import": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20], "importantli": 5, "impos": [2, 8, 13, 14, 23], "imposs": [2, 7, 23], "impress": [2, 14, 23], "improv": [0, 2, 6, 7, 11, 12, 13, 15], "impur": 11, "imread": 8, "imshow": [3, 5, 6, 8], "in3050": [22, 23], "in3310": 23, "in4080": [22, 23], "in4300": [22, 23], "in4310": 22, "in5400": 5, "in5550": 22, "in_out_neuron": 6, "inaccur": 15, "inact": 14, "inadequ": [2, 23], "inch": 8, "includ": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 17, 20, 21, 22, 23], "include_bia": [8, 11], "incom": [1, 14], "incorrect": 3, "incoveni": 10, "increas": [2, 3, 5, 6, 7, 8, 11, 14, 15, 20, 23], "increasingli": 20, "ind": 8, "inde": [2, 4, 6, 7, 8, 15, 23], "indefinit": 6, "independ": [2, 7, 8, 9, 10, 14, 15, 20, 23], "index": [2, 3, 5, 6, 12, 16, 17, 18, 20, 22, 23], "index_col": [2, 23], "indic": [1, 2, 3, 5, 6, 7, 8, 11, 12, 13, 15, 23], "indispens": 8, "individu": [3, 8, 9, 12, 14, 20, 23], "indu": 2, "indx": 18, "indx1": 4, "indx2": 4, "indx3": 4, "ineffici": [5, 15], "inequ": [10, 15], "inertia": 15, "inf1000": [17, 23], "inf1100": [17, 23], "inf1100l": [17, 23], "inf1110": [17, 23], "inf3000": 23, "infeas": 11, "infer": [2, 3, 6, 8, 22, 23], "inferenc": 3, "infil": [2, 8, 9, 11, 23], "infin": [7, 8, 9, 13], "infinit": 5, "infinitesim": 20, "influenc": [8, 12], "influenti": 3, "info": 23, "inform": [2, 3, 5, 6, 8, 11, 13, 14, 15, 16, 18, 22, 23], "inforom": 0, "infti": [5, 8, 15, 20], "ingeni": 15, "ingredi": [2, 11, 23], "inher": 8, "inherit": [18, 23], "initi": [2, 3, 4, 8, 12, 15, 16, 18, 20, 23], "inject": 16, "inlin": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "inner": [2, 15], "inp": 6, "inplac": 15, "input": [1, 2, 3, 5, 6, 7, 8, 9, 10, 14, 15, 16, 20, 23], "input_dim": 3, "input_shap": [5, 6], "inputs": 3, "inputs_shuffl": [2, 3], "insert": [5, 7, 8, 10, 12, 20], "insid": [2, 6, 9], "insight": [2, 3, 7, 17, 23], "insist": [8, 15], "inspir": [2, 3, 14, 23], "instabl": 4, "instal": [0, 2, 3, 7, 8, 11], "instanc": [1, 2, 3, 4, 6, 8, 11, 13, 15, 23], "instanti": 12, "instead": [2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 15, 16, 18, 20, 23], "institut": 3, "instruct": [2, 3], "int": [2, 3, 4, 5, 6, 7, 8, 13, 15, 16, 18, 20], "int32": 12, "int_": [5, 8, 20], "int_0": 20, "int_a": 20, "intak": 2, "integ": [3, 4, 15, 16, 18, 20, 23], "integer_vector": 3, "integr": [5, 8, 20, 23], "intellig": [2, 16, 22, 23], "intend": 12, "intens": 3, "intention": 16, "interact": [2, 8, 11, 14, 17, 23], "intercept": [1, 2, 8, 10, 13, 15, 23], "intercept_": [2, 8, 10, 11, 15, 23], "interchang": [7, 14, 18], "interconnect": 3, "interest": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 17, 20, 23], "interfac": [2, 3, 18], "interior": [2, 11, 23], "intermedi": 18, "intern": [3, 12, 14], "interpol": [3, 5, 6, 8, 14], "interpr": 7, "interpret": [0, 1, 2, 3, 8, 11, 12, 14, 15, 18, 20], "interv": [2, 5, 7, 8, 9, 15, 20, 23], "intial": 15, "intract": [2, 6], "intrins": [5, 13, 18, 20, 23], "intro": [17, 22, 23], "introduc": [2, 3, 7, 8, 10, 12, 14, 18, 20, 23], "introduct": [3, 4, 6, 15, 22], "introductori": [2, 6, 18, 22, 23], "intuit": [2, 7, 8, 10, 14, 15, 23], "inv": [2, 7, 15, 23], "invalu": [2, 15, 17, 23], "invari": 3, "invd": 7, "inver": 10, "invers": [2, 5, 8, 15, 23], "inverse_transform": 10, "invert": [1, 2, 7, 9, 12, 15, 23], "invh": 15, "invok": [2, 10], "involv": [2, 4, 8, 9, 13, 14, 23], "io": [2, 17, 19, 21, 22, 23], "ip": [2, 10, 20, 23], "ipca": 13, "ipynb": [17, 23], "ipython": [2, 7, 9, 11, 13, 16, 17, 23], "iq": 8, "iri": [10, 11], "irreduc": 8, "irrelev": 7, "irrespect": [2, 23], "isn": 7, "isnul": 2, "isomap": 13, "issu": [0, 3, 11, 18], "it_arrai": 15, "item": [2, 15, 23], "items": [18, 23], "iter": [3, 4, 6, 8, 10, 15, 16, 20, 23], "its": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "itself": [7, 8, 14, 20, 23], "j": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 15, 16, 18, 20, 22, 23], "j1": 18, "j_": 8, "j_lasso_sk": 8, "j_ridge_sk": 8, "j_sk": 8, "jackknif": [8, 17, 23], "jacobian": [4, 15], "jason": 6, "jax": [17, 23], "jensen": [21, 23], "jerom": 22, "ji": [14, 18], "jit": 15, "jj": [2, 7, 8, 23], "jk": [2, 3, 8, 14, 18, 23], "jl": [2, 23], "jm": 18, "jnp": 15, "job": [0, 4, 10, 12], "join": [2, 6, 8, 9, 11, 23], "joint": [6, 7], "judg": 15, "judgement": 8, "julia": [17, 18], "jump": 20, "junk": 6, "jupit": 23, "jupyt": [0, 1, 2, 17, 22, 23], "just": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23], "justif": 2, "justifi": [5, 12], "k": [2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "k0": 9, "k1": 9, "kaggl": 8, "kappa_d": 20, "karl": [21, 23], "karush": 10, "katrin": [21, 23], "keep": [0, 2, 3, 6, 7, 8, 13, 15, 16, 18, 23], "keepdim": [3, 8, 12, 18], "kei": [2, 3, 5, 8, 14], "kept": [6, 8, 16], "kera": [2, 6, 17, 23], "kernel": [2, 3, 5, 17, 23], "kernel_regular": [3, 5], "kernel_s": 6, "kernelpca": 13, "kev": [2, 23], "kevin": [22, 23], "keyword": [18, 23], "kfold": 8, "kg": 3, "ki": 18, "kick": [3, 15], "kiener": 4, "kilomet": 8, "kind": [2, 4, 5, 6, 10, 14, 15, 16, 23], "kj": [8, 14, 18], "kjm": [17, 23], "kkt": 10, "kl": 20, "km": [14, 23], "kmean": 16, "kmeanspoint": 16, "kn_k": 16, "know": [0, 1, 2, 3, 4, 7, 8, 10, 15, 17, 23], "knowledg": [2, 17, 23], "known": [3, 5, 6, 7, 8, 9, 10, 11, 14, 18, 20, 22], "kondev": [2, 23], "kp": 20, "kpca": 13, "kroneck": 16, "kuhn": 10, "kvalsund": [21, 23], "kwown": [2, 23], "l": [2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 23], "l0": 9, "l1": [2, 3, 5, 9, 23], "l1_l2": [3, 5], "l1regl": 7, "l2": [3, 5], "l_": 18, "l_1": 9, "l_2": [9, 15], "l_j": 14, "la": 15, "la_i": 14, "la_k": 14, "lab": [17, 23], "label": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 20, 23], "labelencod": [9, 12], "labels": [8, 10, 11], "labels_shuffl": [2, 3], "laboratori": 19, "lack": [2, 23], "lagari": 4, "lagrang": [10, 13], "lambda": [2, 3, 4, 5, 7, 8, 9, 10, 12, 14, 15, 20, 23], "lambda_": 13, "lambda_0": 13, "lambda_1": [7, 10, 13], "lambda_2": [10, 13], "lambda_i": [10, 13], "lambda_iy_i": 10, "lambda_jy_iy_j": 10, "lambda_k": 10, "lambda_n": [7, 10], "lamda": 3, "land": [2, 10], "landmark": 10, "landscap": 15, "langl": [2, 8, 13, 20, 23], "languag": [2, 3, 6, 10, 17, 18, 22, 23], "lapack": [18, 23], "laplac": 7, "laptop": [0, 17], "larg": [2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 20, 22, 23], "larger": [2, 5, 7, 8, 10, 12, 13, 15, 20, 23], "largest": [6, 10, 13], "lasso": [2, 9, 17, 23], "lasso_sk": 8, "last": [1, 2, 3, 5, 6, 7, 8, 9, 10, 14, 18, 20, 21, 23], "latent": 6, "latent_dim": 6, "latent_point": 6, "latent_space_value_rang": 6, "later": [0, 2, 3, 6, 9, 10, 14, 15, 16, 17, 23], "latest": [0, 6, 17], "latest_checkpoint": 6, "latex": 23, "latter": [2, 5, 8, 9, 10, 13, 15, 18, 20, 23], "lattic": 14, "law": 2, "layer": [2, 6, 15, 23], "lbfg": [9, 11, 12], "lcc": [7, 8], "lda": 13, "ldot": [2, 8, 13, 23], "le": [7, 9, 12, 15, 20], "lead": [1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "leaf": 11, "leaki": 3, "leakyrelu": 6, "lear": 15, "learn": [5, 6, 7, 8, 9, 10, 11, 12, 14, 18, 21, 22], "learnabl": 5, "learner": 12, "learnig": 23, "learning_r": [10, 12], "learning_rate_init": [2, 3, 23], "learning_schedul": 15, "least": [2, 9, 10, 12, 13, 17, 18, 20], "leat": 15, "leav": [2, 3, 5, 7, 8, 11, 13, 23], "lectur": [2, 3, 7, 12, 13, 14, 15, 17, 18, 19, 21, 22], "lecturenot": [2, 17, 22, 23], "left": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "leftarrow": [10, 14], "legend": [0, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 23], "leinonen": 23, "len": [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 18, 23], "length": [1, 2, 3, 5, 6, 10, 11, 15, 17, 23], "length_of_sequ": 6, "leq": [2, 7, 9, 10, 15, 16, 20, 23], "less": [2, 3, 5, 6, 7, 8, 10, 11, 15, 17, 20, 23], "lessen": 3, "let": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "letter": [1, 2, 18, 20, 23], "level": [2, 3, 7, 8, 11, 17, 18, 19, 21, 23], "li": [10, 13, 23], "lib": 23, "liblinear": 12, "librari": [2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 18, 20, 22], "licens": [2, 3, 17, 23], "lie": [2, 8, 13, 20, 23], "life": [2, 3, 10, 14, 23], "lifetim": 15, "like": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 17, 18, 20, 23], "likelihood": [2, 3, 7, 11, 23], "lim_": 20, "limit": [2, 7, 8, 10, 14, 18, 23], "lin_clf": 10, "lin_model": 2, "lin_reg": 11, "linalg": [2, 4, 7, 8, 10, 13, 15, 18, 20, 23], "line": [0, 1, 2, 5, 8, 10, 13, 15, 23], "line1": 10, "line2": 10, "line3": 10, "line_model": 0, "line_ms": 0, "line_predict": 0, "linear": [1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 17, 20], "linear_model": [0, 1, 2, 7, 8, 9, 10, 11, 12, 13, 15, 23], "linear_regress": 8, "linearli": 7, "linearloc": [8, 15], "linearregress": [0, 1, 2, 8, 9, 11, 23], "linearsvc": 10, "liner": [3, 5], "linerar": 12, "linewidth": [2, 4, 6, 8, 10, 11, 12], "link": [0, 2, 6, 11, 14, 17, 19, 21, 23], "linlag": 7, "linpack": [18, 23], "linreg": [2, 23], "linspac": [1, 2, 4, 5, 6, 8, 10, 11, 12, 15, 18, 20, 23], "linu": 6, "linux": [2, 3, 17, 23], "liquid": [2, 23], "list": [0, 2, 3, 4, 5, 6, 11, 17, 23], "listedcolormap": [11, 12], "literatur": [3, 9, 16, 22], "littl": [3, 5, 11, 14], "live": [1, 10], "ll": [2, 20, 23], "lle": 2, "lloyd": [6, 16], "lmb": [2, 4, 7, 8], "lmbd": [2, 3, 5, 23], "lmbd_val": [2, 3, 5, 23], "lmbda": 15, "ln": [3, 15], "load": [2, 3, 6, 8, 9, 11, 12], "load_boston": 2, "load_breast_canc": [3, 9, 11, 12, 13], "load_data": [5, 6], "load_digit": [3, 5], "load_iri": [10, 11], "loc": [5, 8, 9, 10, 11, 12, 23], "local": [0, 2, 3, 5, 9, 14, 15], "locat": [0, 4, 5, 10], "log": [0, 2, 3, 4, 6, 7, 8, 9, 11, 12, 13, 15, 18, 23], "log10": [2, 7, 8], "log_": [2, 23], "log_clf": 12, "logarithm": [2, 7, 9, 18, 23], "logic": [2, 3, 11, 23], "login": 0, "logist": [2, 3, 4, 10, 11, 12, 13, 14, 15, 17], "logisticregress": [9, 11, 12, 13], "logit": 9, "logreg": [9, 11, 12, 13], "logspac": [2, 3, 5, 7, 8, 23], "long": [2, 3, 5, 6, 14, 15, 23], "longer": [4, 5, 10, 12, 16, 18, 20, 23], "loocv": 8, "look": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 23], "loop": [1, 3, 6, 8, 12, 14, 16, 17, 18, 23], "lose": 3, "loss": [2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15, 18, 23], "loss_fil": 6, "lossfil": 6, "lost": 6, "lot": [1, 2, 3, 6, 8], "low": [2, 8, 11, 12, 13, 23], "lower": [1, 2, 3, 5, 8, 11, 12, 18], "lowercas": [18, 23], "lowest": [11, 15, 20], "lr": [3, 5, 6, 12], "lstat": 2, "lstm": 6, "lstm_2layer": 6, "lstsq": [2, 23], "lt": 8, "lu": [2, 7, 23], "lubksb": 18, "luckili": 4, "ludcmp": 18, "lux": 18, "lvert": 3, "lw": [2, 23], "m": [0, 2, 3, 4, 5, 7, 8, 10, 11, 12, 13, 14, 15, 18, 20, 21, 22, 23], "m_": [11, 14], "m_1": 16, "m_h": [2, 23], "m_k": 16, "m_l": 14, "m_n": [2, 23], "m_p": [2, 23], "m_t": 15, "ma": 13, "machin": [0, 1, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 18, 22], "machinelearn": [1, 2, 8, 17, 19, 21, 22, 23], "mackai": 22, "made": [2, 3, 5, 6, 7, 8, 9, 11, 13, 14, 23], "mae": [2, 23], "magic": 6, "magnitud": [3, 8, 9, 15], "mai": [2, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23], "mail": [19, 21], "main": [2, 3, 5, 6, 7, 8, 9, 11, 18, 22], "mainli": [2, 7, 8, 9, 11, 23], "maintain": 8, "major": [3, 8, 11, 12, 15, 18, 23], "make": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 20, 22, 23], "make_axes_locat": 8, "make_moon": [10, 11, 12], "make_pipelin": [2, 8, 12], "makedir": [2, 8, 9, 11, 23], "malcondit": 18, "malign": [3, 9, 11], "mammographi": 7, "manag": [0, 2, 4, 5, 17, 23], "mandatori": [21, 23], "mani": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 17, 18, 20, 22, 23], "manifold": 13, "manner": 5, "manual": 8, "map": [2, 3, 4, 8, 9, 10, 13, 14, 16, 20, 23], "margin": [2, 7, 10], "marit": [2, 23], "mark": 23, "marker": [2, 9, 18, 23], "markov": [17, 23], "marsaglia": 20, "mass": [2, 3, 7, 15], "massag": [2, 23], "masses2016": [2, 23], "masses2016ol": [2, 23], "masses2016tre": 2, "masseval2016": [2, 23], "master": [19, 21], "mat": [17, 23], "mat1100": [17, 23], "mat1110": [17, 23], "mat1120": [17, 23], "match": [0, 3, 6, 7, 15, 16], "materi": [0, 6, 7, 9, 15, 18, 19, 21], "math": [5, 9, 14, 15, 18, 20, 22, 23], "mathbb": [2, 6, 7, 8, 9, 10, 13, 14, 15, 16, 18, 20, 23], "mathbf": [2, 7, 8, 9, 10, 15, 18, 23], "mathcal": [3, 7, 8, 9, 15], "matheemat": 5, "mathemat": [2, 8, 13, 14, 15, 17, 18, 20, 22, 23], "mathemati": 23, "mathrm": [2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 23], "matmul": [3, 4, 7], "matnat": 22, "matplotlib": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "matric": [1, 2, 3, 5, 6, 8, 9, 10, 13, 15, 17], "matrix": [2, 4, 5, 6, 8, 9, 10, 12, 15, 20], "matshow": 3, "matter": [4, 5, 15], "max": [2, 3, 4, 5, 6, 11, 12, 14, 15, 21, 23], "max_depth": [2, 11, 12], "max_diff": 4, "max_diff1": 4, "max_diff2": 4, "max_it": [2, 3, 10, 15, 23], "max_iter": 16, "max_leaf_nod": 12, "max_sampl": 12, "maxdegre": [2, 8, 12], "maxdepth": 12, "maxim": [3, 6, 7, 9, 10, 13], "maximum": [2, 4, 5, 7, 9, 10, 11, 12, 15, 16, 23], "maxpolydegre": [7, 8], "maxpooling2d": 5, "mbox": [7, 8], "mcculloch": 14, "md": 13, "mdoel": 6, "mean": [0, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "mean_absolute_error": [2, 23], "mean_divisor": 16, "mean_i": 20, "mean_matrix": 16, "mean_squared_error": [0, 2, 6, 8, 9, 12, 23], "mean_squared_log_error": [2, 23], "mean_vector": 16, "mean_x": 20, "meaning": [2, 6, 9, 23], "meansquarederror": [2, 23], "meant": [5, 9, 12, 15], "measur": [1, 2, 3, 4, 7, 8, 11, 13, 14, 16, 20, 23], "mechan": [2, 6, 20, 23], "median": [2, 23], "medicin": 14, "medium": [6, 10, 15], "medv": 2, "meet": [2, 21], "mehta": [2, 23], "memori": [5, 6, 13, 14, 15, 18], "mention": [2, 14, 15, 20, 23], "mere": 2, "meshgrid": [4, 7, 8, 10, 11, 12, 13], "mess": 0, "messag": [7, 15], "messi": 4, "met": [2, 5, 10], "meteorolog": 11, "meter": 8, "method": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 13, 14, 16, 17, 18, 20, 22], "metion": 8, "metric": [0, 2, 3, 5, 8, 9, 11, 12, 16, 23], "metropoli": [17, 23], "mev": [2, 20, 23], "mgd": 15, "mglearn": [17, 23], "mgrid": 15, "mhjensen": 23, "mi": 12, "mia": [21, 23], "microsoft": 22, "mid": 3, "midel": 6, "midnight": 0, "midpoint": 11, "might": [0, 2, 3, 4, 6, 8, 11, 15], "mild": 11, "millimet": 8, "million": [2, 23], "mimic": 14, "min": [2, 4, 7, 10, 11], "min_": [2, 4, 7, 16, 23], "min_samples_leaf": 11, "mind": [0, 2, 8, 15, 23], "mindboard": 6, "mine": [17, 23], "mini": [3, 13, 14, 15], "minibatch": [3, 13, 15], "minibathc": 15, "miniforge3": 23, "minim": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16], "minima": [2, 3, 9, 15, 23], "minimum": [2, 3, 4, 8, 10, 11, 13, 15], "minmaxscal": 2, "minor": 20, "minst": 3, "minu": 9, "mirjalili": 23, "mirror": 11, "misc": 8, "misclassif": [10, 11, 12], "misclassifi": [10, 12], "miser": 2, "mismatch": 3, "miss": [2, 9, 12], "mistak": 6, "mit": 22, "mix": [3, 4, 23], "mixtur": 15, "mk": [11, 18], "mkdir": [2, 8, 9, 11, 23], "ml": [2, 3, 12, 15, 18], "mlab": 20, "mle": [7, 9], "mlp": 3, "mlpclassifi": 3, "mlpregressor": [2, 23], "mm": 18, "mn": [14, 20], "mnist": [3, 13], "mod": 20, "mode": [19, 21, 23], "model": [1, 4, 5, 7, 9, 10, 11, 12, 13, 15, 16, 17, 20, 22], "model_select": [0, 1, 2, 3, 5, 7, 8, 9, 11, 12, 13, 23], "moder": 12, "modern": [2, 8, 9, 17, 23], "modif": [4, 14, 15], "modifi": [2, 3, 5, 7, 9, 10, 12, 14, 15, 23], "modul": [1, 2, 18, 23], "modular": 20, "modulo": 20, "moe": 13, "moment": [7, 8, 15, 20], "mondai": [21, 23], "monitor": 15, "monoton": [7, 14, 20], "mont": [2, 8, 17, 20, 22, 23], "montli": 1, "moor": [7, 8], "more": [1, 2, 3, 4, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20], "moreov": [2, 5], "morten": [21, 23], "mortenhj": 23, "most": [0, 1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23], "mostli": [3, 13], "motion": [2, 15], "motiv": [3, 6], "move": [0, 1, 2, 6, 7, 8, 9, 11, 14, 15, 16, 20], "mpl": [9, 23], "mpl_toolkit": [4, 8, 15], "mplot3d": [4, 8, 15], "mplregressor": 3, "mse": [0, 1, 2, 6, 7, 8, 11, 12, 23], "mse_simpletre": 12, "mselassopredict": 7, "mselassotrain": 7, "mseownridgepredict": 8, "msepredict": 7, "mseridgepredict": [2, 7, 8], "msetrain": 7, "msle": [2, 23], "mt": [9, 14], "mu": [2, 8, 13, 15, 20, 23], "mu0": 20, "mu1": 20, "mu2": 20, "mu_": [8, 20], "mu_i": 8, "mu_n": 13, "mu_x": 20, "much": [0, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 18, 20, 23], "multi": [2, 3, 5, 9, 17, 23], "multiclass": [3, 9], "multidimension": [13, 14, 23], "multilay": 3, "multinomi": 9, "multipl": [0, 4, 6, 7, 8, 9, 14, 15, 20], "multipli": [5, 7, 8, 13, 15, 18, 20], "multiplum": 10, "multivari": [2, 4, 12, 13, 17, 20, 23], "multivariate_norm": [13, 16], "multpli": 1, "murphi": [13, 22, 23], "must": [0, 3, 4, 7, 8, 10, 12, 14, 15, 16, 20], "mutat": 9, "mutual": [3, 5, 8, 15], "mx_": 20, "my": 23, "myenv": 23, "myriad": [2, 17, 23], "mz1": 20, "mz2": 20, "m\u00f8svatn": 8, "n": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 23], "n1": 18, "n2": 18, "n_": [3, 4, 5, 10, 14, 20], "n_0": [14, 20], "n_boostrap": [8, 12], "n_bootstrap": 8, "n_categori": [3, 5], "n_cluster": 16, "n_compon": 13, "n_epoch": 15, "n_estim": 12, "n_examples_to_gener": 6, "n_featur": 3, "n_filter": 5, "n_hidden": 4, "n_hidden_neuron": [2, 3, 23], "n_i": 20, "n_input": [2, 3, 5], "n_instanc": 11, "n_job": 12, "n_k": 16, "n_l": [14, 20], "n_layer": 3, "n_m": 11, "n_neuron": 3, "n_neurons_connect": 5, "n_neurons_layer1": 3, "n_neurons_layer2": 3, "n_point": 16, "n_sampl": [8, 10, 11, 12, 16], "n_split": 8, "n_step": 6, "n_t": 4, "n_x": 4, "nabla": [3, 15], "nabla_": [4, 15], "nabla_w": 15, "nag": 15, "naimi": [2, 23], "naiv": 9, "naive_kmean": 16, "name": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 20, 21, 23], "narrow": 15, "nation": [3, 7], "nativ": [17, 23], "natur": [2, 3, 6, 10, 11, 14, 15, 20, 22, 23], "navier": 14, "navig": 0, "nb": 20, "nb_": 18, "nbconvert": 23, "nd": 16, "ndarrai": 8, "ne": [11, 12, 18, 20], "nearest": [3, 5, 8, 13], "nearli": 15, "neat": 23, "neccesari": 8, "necess": 4, "necessari": [2, 3, 5, 6, 10, 16, 23], "necessarili": [2, 6, 13, 20, 23], "necesserali": 7, "neck": 9, "need": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20], "neg": [2, 3, 5, 7, 8, 9, 12, 15, 18, 20, 23], "neg_mean_squared_error": 8, "neglect": 20, "neglig": 20, "neighbor": [5, 8, 13], "neither": [6, 15], "neq": [15, 16, 20], "nervou": 14, "nest": [11, 14], "nesterov": 15, "net": [4, 6, 14], "netlib": [18, 23], "network": [2, 11, 15, 17, 22], "neural": [2, 15, 17, 22], "neural_network": [2, 3, 4, 23], "neuralnetwork": 3, "neuron": [3, 4, 5, 6, 14], "neutral": [2, 23], "neutron": [2, 23], "never": [3, 6, 8, 11, 20], "new": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 23], "new_chang": 15, "new_hobbit": 23, "newaxi": [2, 5, 8, 11], "newli": [2, 23], "newton": [3, 9, 10, 15, 20], "next": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 15, 16, 23], "next_guess": 15, "next_input": 6, "ng": 3, "ni": 16, "nice": [2, 3, 7, 13, 23], "niter": 15, "nitric": 2, "nlambda": [2, 7, 8], "nlp": 22, "nm": 20, "nm_n": [2, 23], "nmse": 8, "nn": [4, 7, 8, 14, 18, 23], "nn_model": 3, "nnmin": 4, "node": [3, 5, 11, 12, 14], "nois": [2, 6, 7, 8, 10, 11, 12, 15, 23], "noise_dimens": 6, "noisi": [3, 8], "non": [2, 3, 5, 7, 8, 9, 11, 12, 13, 14, 15, 16, 18, 20, 23], "none": [2, 3, 4, 6, 7, 11, 12, 15, 20, 23], "nonlinear": [5, 8, 10, 11, 13, 14], "nonneg": [8, 11, 15], "nonparametr": 8, "nonsens": 20, "nonsingular": 18, "nonumb": [5, 9, 10, 15, 18], "nor": [3, 6, 15], "norm": [2, 3, 7, 8, 10, 13, 15, 23], "normal": [1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23], "normali": [18, 23], "norwai": [8, 23], "notat": [2, 4, 7, 8, 15, 16, 20, 23], "note": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 16, 17, 18, 20, 22, 23], "notebook": [0, 1, 2, 3, 5, 11, 17, 23], "noth": [3, 4, 7, 10, 14, 16, 20], "notic": [6, 7, 14, 15, 18, 20, 23], "notion": 5, "novel": [5, 8, 12, 23], "novemb": [3, 21, 23], "now": [0, 1, 2, 4, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20, 23], "nowadai": [2, 3, 5, 11, 17, 23], "nox": 2, "np": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23], "npr": 4, "nsampl": 8, "nt": 4, "nu": 20, "nuclear": 7, "nuclei": [2, 20, 23], "nucleon": [2, 23], "nucleu": [2, 23], "num": 6, "num_coordin": 4, "num_hidden_neuron": 4, "num_it": 4, "num_neuron": 4, "num_neurons_hidden": 4, "num_point": 4, "num_tre": 12, "num_valu": 4, "number": [1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 23], "numberid": 9, "numberparamet": 5, "numer": [2, 7, 8, 11, 12, 13, 14, 15, 17, 18, 22, 23], "numpi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20], "nunmpi": 7, "nx": 4, "ny": 20, "o": [2, 3, 6, 7, 8, 9, 10, 11, 13, 18, 21, 22, 23], "obei": [8, 13, 15], "object": [0, 2, 3, 6, 10, 12, 18, 23], "obliqu": 7, "observ": [2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 23], "obtain": [2, 3, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 23], "obviou": [7, 8, 13, 20], "obviouli": 23, "obvious": [2, 6, 7, 8, 18, 23], "occupi": 2, "occur": [2, 8, 10, 11, 18, 20, 23], "octob": [21, 23], "od": 2, "odd": [2, 5, 9, 23], "odenum": 4, "odesi": 4, "oen": 2, "off": [3, 5, 6, 7, 11, 15, 20], "offer": [8, 13, 17, 18, 19, 21, 23], "offic": [21, 23], "offici": [19, 23], "often": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "ofter": [18, 23], "ol": [2, 15], "old": [0, 3, 7, 12, 15], "ols_paramet": 1, "ols_sk": 8, "ols_svd": 8, "olstheta": [2, 7], "omega": [4, 5, 8], "omega_0": 5, "omit": [2, 7, 23], "onc": [3, 8, 11, 13, 15], "one": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 20, 21, 23], "onehot": 3, "onehot_vector": 3, "onehotencod": 11, "ones": [1, 2, 4, 7, 8, 10, 11, 12, 13, 15, 18, 23], "ones_lik": 6, "onl": 5, "onli": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "onlin": [0, 13, 19], "onto": [7, 13], "open": [0, 2, 3, 6, 8, 9, 11, 17, 19, 21, 23], "oper": [0, 1, 2, 3, 5, 7, 8, 12, 13, 14, 15, 17, 20, 23], "operation": 20, "oplu": 20, "opmiz": 15, "opportun": 2, "oppos": [8, 15], "opposit": [3, 7, 10], "opt": [3, 7, 23], "optim": [1, 2, 4, 5, 6, 7, 8, 9, 11, 12, 13, 16], "optimis": [3, 5], "option": [0, 2, 3, 5, 7, 8, 10, 13, 18], "optmiz": [3, 10, 15], "oral": 23, "orang": 2, "order": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 18, 20, 23], "ordinari": [2, 4, 5, 9, 13, 15, 17], "oreilli": [22, 23], "org": [1, 2, 5, 6, 17, 18, 22, 23], "organ": [8, 9, 12, 18], "orient": [3, 7, 20], "origin": [0, 2, 5, 7, 8, 10, 13, 14, 15, 18, 23], "orthogn": 7, "orthogon": [2, 7, 8, 10, 13, 15, 18, 23], "orthonorm": 7, "os": [21, 23], "oscar": 3, "oscil": [5, 15], "oskar": 23, "oskarlei": 23, "oslo": [2, 17, 19, 21, 23], "osx": [2, 17, 23], "other": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 15, 16, 17, 19, 20, 21, 22], "otherwis": [2, 3, 6, 9, 15, 18, 23], "ouput": [7, 9, 14], "our": [0, 1, 3, 4, 5, 8, 9, 10, 11, 12, 14, 16, 17, 18, 20], "ourmodel": 2, "ourselv": [2, 7, 8, 10, 13, 15, 23], "out": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23], "out_fil": 11, "outcom": [2, 9, 11, 12, 14, 20], "outdoor": 11, "outer": [8, 14, 15], "outfil": 6, "outlier": [2, 10, 23], "outlin": [8, 12, 13], "outlook": 11, "outperform": 12, "output": [2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 23], "output_bia": 3, "output_bias_gradi": 3, "output_shap": 6, "output_weight": 3, "output_weights_gradi": 3, "outputlayer1": 14, "outputlayer2": 14, "outsid": 6, "over": [0, 1, 2, 3, 5, 6, 7, 8, 11, 12, 14, 15, 18, 23], "over1": 15, "overal": [3, 12], "overcast": 11, "overcom": [14, 15], "overdetermin": [2, 23], "overfit": [2, 3, 5, 8, 11, 12, 15], "overflow": 7, "overhead": 14, "overlap": [5, 9, 10, 11], "overlin": [2, 7, 8, 11, 12, 13, 16, 18, 23], "overst": 2, "overtrain": 6, "overview": 5, "own": [1, 6, 7, 8, 10, 14, 15, 17, 18], "owner": 2, "ownmsepredict": 2, "ownmsetrain": 2, "ownridgetheta": [2, 8], "ownypredictridg": 2, "ownytilderidg": 2, "oxid": 2, "p": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23], "p0": 4, "p1": 4, "p_": [4, 6, 10, 11], "p_hidden": 4, "p_i": [7, 20], "p_j": 20, "p_n": 20, "p_output": 4, "p_x": 20, "pack": [2, 23], "packag": [0, 2, 3, 5, 6, 7, 10, 13, 15, 17, 20], "packtpub": 23, "packtpublish": 23, "pad": [5, 6], "page": [2, 17, 23], "pai": [0, 2, 3, 11, 15], "pair": [2, 4, 5, 11, 17, 20, 23], "paltform": 0, "panda": [2, 6, 7, 8, 9, 11, 13, 17], "panel": 23, "paper": 3, "paradigm": [2, 23], "parallel": [12, 15, 17, 18, 23], "param": 4, "paramat": 4, "paramet": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 20], "parameter": [2, 8, 12, 23], "parametr": [2, 8, 23], "paramt": [5, 7], "part": [2, 3, 5, 7, 8, 12, 18, 19, 20, 21, 23], "partial": [1, 2, 3, 7, 8, 9, 10, 12, 13, 14, 15, 20, 23], "particip": [0, 17, 19, 21, 23], "particl": [2, 6, 15, 20, 23], "particular": [1, 2, 3, 4, 5, 7, 8, 11, 12, 13, 14, 15, 20, 22, 23], "particularli": [7, 8, 10, 13, 15, 20], "partit": [3, 6, 11], "partli": [8, 23], "partner": 0, "pass": [4, 5, 14, 16], "past": [12, 20], "patch": [8, 20], "path": [2, 6, 8, 9, 11, 17, 23], "patient": 9, "patter": 6, "pattern": [2, 5, 6, 14, 22, 23], "pauli": [2, 23], "pc": [13, 17], "pca": [2, 9, 17, 23], "pd": [2, 6, 7, 8, 9, 11, 13, 23], "pde": 4, "pdf": [0, 1, 2, 5, 6, 7, 8, 11, 22, 23], "pedagog": [2, 23], "penal": 8, "penalti": [8, 15], "penros": [7, 8], "pentagon": 15, "peopl": [2, 3, 11, 15, 17], "per": [2, 3, 8, 19, 21, 23], "percentag": [2, 12, 13, 21], "perceptron": [2, 3, 9, 23], "peregrin": 23, "perfect": [2, 3, 15, 23], "perfectli": [6, 8], "perform": [1, 2, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 16, 17, 18, 20, 23], "performac": 6, "perhap": [2, 7, 15, 23], "perimet": 3, "period": [3, 6, 20], "permiss": 0, "permut": 13, "persist": 15, "person": [1, 7, 8, 9, 19, 21, 23], "perspect": 22, "pertin": [14, 23], "petal": [10, 11], "peter": 22, "phantom": 20, "phase": [8, 14], "phenomena": 20, "phi": 10, "phi_k": 10, "philosophi": 15, "phone": [21, 23], "photo": [6, 23], "phrase": [2, 23], "physic": [2, 3, 6, 9, 14, 15, 20, 21, 22, 23], "pi": [4, 5, 7, 8, 9, 11, 14, 15, 20], "pick": [3, 11, 12, 13, 15, 16], "pickl": 3, "pictur": [2, 23], "pie": [17, 23], "piec": [13, 16], "pillow": [2, 17, 23], "pinv": [7, 8, 15], "pip": [0, 2, 3, 17, 23], "pip3": [2, 3, 23], "pipelin": [2, 8, 10, 12], "pippin": 23, "pit": 6, "pitfal": 8, "pitt": 14, "pixel": [3, 5, 6, 23], "pixel_height": [3, 5], "pixel_width": [3, 5], "place": [0, 2, 6, 8, 10, 15, 18, 23], "plai": [2, 5, 6, 7, 8, 10, 13, 17, 23], "plain": [10, 12, 14, 15, 16], "plan": [8, 11, 21, 22, 23], "plane": [10, 11], "plateau": 7, "platform": [17, 23], "plausibl": 14, "pleas": [15, 21, 23], "plenti": 3, "plethora": [5, 14], "plot": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "plot_confusion_matrix": [9, 12], "plot_count": 8, "plot_cumulative_gain": [9, 12], "plot_data": 3, "plot_dataset": 10, "plot_decision_boundari": [11, 12], "plot_import": 12, "plot_max": 6, "plot_min": 6, "plot_model": 6, "plot_numb": 6, "plot_predict": 10, "plot_regression_predict": 11, "plot_result": 6, "plot_roc": [9, 12], "plot_surfac": [4, 8, 15], "plot_train": 11, "plot_tre": [11, 12], "plt": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "plu": [2, 5, 7, 9, 23], "pm": 10, "pmatrix": 4, "pml": 22, "pn": 5, "png": [2, 6, 8, 9, 11, 23], "point": [2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 23], "point_1": 6, "point_2": 6, "poisson": [17, 20, 23], "poli": [8, 10], "poly100_kernel_svm_clf": 10, "poly3": 2, "poly3_plot": 2, "poly_featur": [0, 10, 11], "poly_features10": 11, "poly_fit": 11, "poly_fit10": 11, "poly_kernel_svm_clf": 10, "poly_model": 0, "poly_ms": 0, "poly_predict": 0, "polydegre": [2, 7, 8, 12], "polygon": 15, "polym": 14, "polynomi": [0, 2, 7, 8, 9, 10, 11, 12, 13, 23], "polynomial_featur": [0, 1, 8], "polynomial_svm_clf": 10, "polynomialfeatur": [0, 1, 2, 8, 10, 11], "polytrop": [2, 8], "pool": 5, "pool_siz": 5, "poor": [3, 15], "poorli": 2, "popul": [2, 7, 23], "popular": [0, 2, 3, 5, 8, 9, 10, 11, 13, 14, 17, 18, 20], "popularli": [2, 23], "portabl": 12, "portion": [13, 15], "pose": [2, 6, 7, 8, 13, 20, 23], "posit": [2, 3, 4, 5, 7, 9, 10, 12, 13, 15, 16, 18, 20, 23], "possibl": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 21, 23], "possibli": [8, 10, 15], "posterior": 7, "postpon": 2, "postul": 7, "potenti": [2, 5, 7, 8, 14, 15], "pott": 14, "power": [2, 3, 7, 8, 10, 11, 14, 15, 23], "pp": [7, 8], "practic": [1, 2, 7, 8, 9, 10, 20], "practition": [2, 3, 5, 23], "pre": 23, "preced": [3, 13, 14, 20], "preceed": 6, "preceq": 10, "precis": [2, 4, 7, 13, 15, 18, 20, 23], "pred": 8, "predicit": 2, "predict": [0, 1, 2, 3, 7, 8, 9, 10, 11, 12, 17, 22, 23], "predict_prob": 3, "predict_proba": [9, 12], "predictor": [2, 7, 8, 9, 11, 12, 13, 23], "prefer": [2, 3, 8, 10, 11, 13, 15, 17, 23], "prepar": [2, 8, 18, 23], "preprocess": [0, 1, 2, 6, 8, 9, 10, 11, 12, 13], "prerequisit": 2, "presenc": 15, "present": [2, 7, 8, 9, 11, 14, 15, 18, 20, 23], "preserv": [5, 13, 18], "press": [0, 15, 22], "pretrain": [3, 6], "pretti": [2, 6, 10, 11, 17, 23], "prev_centroid": 16, "prevent": [15, 20], "previou": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 18, 20], "previous": [4, 5, 11, 12, 20], "price": [2, 6, 11, 15], "primal": 10, "primari": [2, 9, 23], "prime": 20, "princip": [2, 7, 9, 17, 23], "principl": [2, 8, 9, 10, 16, 23], "print": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23], "print_funct": [10, 11], "printout": [2, 23], "prior": [2, 7, 8, 23], "privat": 2, "prob": [3, 20], "probabilist": [2, 22, 23], "probabl": [2, 3, 5, 6, 8, 9, 12, 15, 17, 23], "problem": [2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 20], "probml": 22, "proce": [2, 7, 8, 9, 10, 11, 12, 13, 15, 18, 23], "procedur": [4, 6, 7, 8, 10, 12, 13, 15], "proceed": 18, "process": [2, 4, 6, 8, 11, 12, 14, 15, 17, 18, 20, 22, 23], "prod": 22, "prod_": [3, 7, 9], "produc": [2, 5, 6, 7, 8, 11, 12, 13, 14, 15, 17, 18, 20, 23], "product": [1, 2, 3, 5, 7, 8, 9, 10, 14, 15, 17, 18, 23], "profess": [2, 23], "program": [0, 2, 3, 6, 7, 8, 10, 14, 16, 17, 18, 19, 20, 21, 23], "programm": 18, "progress": [3, 6, 16], "prohibit": 8, "project": [0, 2, 3, 4, 5, 7, 13, 15, 17, 19], "project_root_dir": [2, 8, 9, 11, 23], "promin": 14, "promis": 10, "promot": [21, 23], "prone": [0, 11], "pronounc": [15, 17, 23], "proof": [2, 13, 14, 15, 23], "propag": [4, 5, 15], "proper": [2, 4, 8, 9], "properli": [3, 8, 10, 12, 15], "properti": [1, 2, 3, 5, 14, 15, 18, 23], "proport": [2, 3, 7, 11, 13, 15, 20, 23], "propos": [3, 6, 8, 12, 23], "propto": [7, 15], "proton": [2, 23], "prove": [5, 15], "provid": [2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 15, 17, 18, 20, 23], "proxi": [3, 15], "prune": 11, "pseudo": [18, 20], "pseudoinv": 7, "pseudoinvers": [7, 8], "pseudorandom": [8, 20], "psychologi": [2, 23], "pt": 15, "public": [0, 2, 17, 23], "pull": 0, "punish": [2, 3, 23], "pure": [5, 11, 20], "purest": 11, "puriti": 11, "purpos": [2, 5, 12, 14, 16, 23], "push": 0, "put": 3, "py": [7, 23], "pycod": 23, "pydata": 17, "pydot": 11, "pyhton2": 23, "pylab": [9, 23], "pypi": 17, "pyplot": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "pythagora": 7, "python": [3, 4, 5, 7, 8, 10, 13, 14, 15, 16, 20], "python2": 2, "python3": [2, 17, 23], "pytorch": [2, 17, 23], "q": [7, 8, 10, 13, 20], "qp": 10, "qquad": [4, 13, 15, 18], "qr": [7, 8, 18], "quad": [3, 15, 18], "quadrat": [2, 10, 11, 15, 23], "qualit": [6, 11, 20], "qualiti": [2, 11, 17, 23], "quantifi": 3, "quantil": 12, "quantit": [2, 8, 11, 23], "quantiti": [1, 2, 4, 7, 8, 9, 11, 12, 13, 14, 16, 18, 20, 23], "quantum": [6, 14, 22, 23], "quartil": 2, "quench": 7, "queri": 11, "question": [2, 7, 8, 11, 13, 14, 15, 21, 23], "qugan": 6, "quick": [6, 20], "quickli": [3, 5, 11, 13, 15], "quit": [0, 3, 7, 8, 11, 12, 14], "quot": 6, "r": [0, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20], "r2": [2, 7, 8, 23], "r2_score": [2, 23], "r2score": [2, 23], "r_1": 11, "r_2": 11, "r_j": 11, "r_m": 11, "rad": 2, "radial": [2, 10, 14], "radioact": 20, "radiu": [2, 3], "rain": 11, "ramp": 3, "ran0": 20, "ran1": 20, "ran2": 20, "ran3": 20, "rand": [0, 2, 6, 7, 8, 11, 12, 15, 18, 23], "randint": [8, 11, 15], "randn": [0, 2, 3, 4, 7, 8, 11, 13, 15, 23], "random": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 15, 16, 17, 18, 23], "random_forest_model": 12, "random_index": 15, "random_indic": [3, 5], "random_st": [2, 9, 10, 11, 12, 13], "randomforestclassifi": 12, "randomli": [3, 8, 11, 15, 16], "rang": [2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 18, 20, 23], "rangl": [2, 8, 13, 20, 23], "rangle_x": 20, "rank": 7, "rankdir": 6, "raphson": [3, 10, 15], "rapidli": 2, "rare": [3, 15], "raschka": 23, "rasckha": 23, "rate": [2, 3, 4, 5, 6, 10, 11, 12, 14, 15], "rather": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "ratio": [6, 9, 11, 12, 13], "rational": [2, 23], "ravel": [7, 8, 9, 10, 11, 12, 13, 15, 18], "raw": 5, "rbf": [10, 13, 14], "rbf_kernel_svm_clf": 10, "rbf_pca": 13, "rc": [2, 20], "rcond": [2, 23], "rcparam": [3, 5, 9, 10, 11, 12, 20, 23], "re": [0, 4, 6, 15], "reach": [3, 6, 7, 8, 11, 12, 14, 15, 16, 23], "read": [1, 2, 4, 5, 6, 7, 8, 9, 10, 13, 14, 18, 20, 22], "read_csv": [2, 8, 9, 11], "read_fwf": [2, 23], "reader": [2, 8, 18, 20, 23], "readi": [2, 3, 7, 8, 10, 12, 13, 14, 18, 23], "readili": 3, "readm": 0, "readthedoc": 17, "real": [1, 2, 3, 6, 9, 12, 13, 14, 18], "real_loss": 6, "real_output": 6, "realist": [10, 23], "realiti": 20, "realiz": [3, 14], "realli": [2, 3, 23], "rearrang": 15, "reason": [2, 3, 5, 6, 12, 15, 22, 23], "reassign": 3, "recal": [7, 8, 11, 12, 13, 14, 18, 20, 23], "recast": 5, "receiv": [3, 5, 12, 14, 20], "recent": [2, 8, 15, 22], "recept": [5, 14], "receptive_field": 5, "recip": [2, 8, 9, 18, 23], "reciproc": 7, "recogn": [2, 6, 7, 12, 23], "recognit": [2, 3, 5, 14, 22, 23], "recommen": 23, "recommend": [0, 2, 4, 5, 6, 7, 8, 10, 15, 17, 18, 22], "reconsid": 11, "reconstruct": 13, "record": [12, 19, 21, 23], "recreat": 0, "rectangl": [11, 15], "rectangular": 7, "rectifi": [3, 5, 14], "recur": [2, 17, 23], "recurr": [2, 3, 17, 23], "recurs": [11, 17, 18, 23], "red": [2, 5, 6, 8, 10, 11], "redefin": [2, 12, 23], "reduc": [3, 5, 7, 8, 11, 12, 13, 15, 23], "reduct": [2, 12, 13, 17, 20, 23], "refer": [2, 3, 4, 5, 7, 8, 13, 14, 15, 16, 18, 22, 23], "referenc": 4, "refin": 14, "refit": 8, "reflect": [2, 3, 6, 7, 20, 23], "refresh": [17, 23], "refreshprogrammingskil": 23, "reg": [12, 13], "regard": [3, 11, 15], "regardless": [1, 14], "region": [5, 6, 8, 11, 14], "regist": [8, 20], "reglasso": 7, "regr_1": [2, 11], "regr_2": [2, 11], "regr_3": [2, 11], "regress": [1, 3, 10, 13, 14, 17, 18], "regressor": [2, 9, 12], "regridg": [2, 7, 8], "regular": [2, 5, 6, 7, 8, 9, 11, 15, 21, 23], "regularli": 0, "reilli": [2, 22, 23], "reinforc": [2, 10, 17, 23], "reiter": 3, "reject": 9, "rel": [2, 6, 8, 9, 11, 14, 15, 20, 23], "relat": [2, 3, 5, 6, 7, 13, 15, 16, 18, 20, 23], "relationship": [2, 6, 11, 23], "relativeerror": [2, 23], "releas": [3, 17, 23], "relev": [2, 3, 7, 9, 13, 17, 20, 23], "reli": [2, 8, 10], "reliabl": [9, 20], "relu": [5, 6, 23], "remain": [3, 4, 6, 8, 14, 18, 20], "remaind": 20, "reman": 4, "remark": 3, "rememb": [2, 10, 15, 18, 23], "remind": [2, 7, 13, 15, 18, 20], "remot": 0, "remov": [2, 6, 7, 8], "renam": 0, "render": [2, 23], "reorder": [7, 9], "reorgan": [2, 23], "repeat": [2, 3, 5, 6, 7, 8, 11, 12, 13, 15, 16, 18, 20, 23], "repeated": 23, "repeatedli": [2, 8, 12, 15], "repet": 5, "repetit": [8, 23], "rephras": 15, "replac": [2, 3, 5, 6, 7, 8, 12, 14, 16, 17, 23], "replica": 8, "repo": 0, "report": 23, "repositori": [2, 6, 23], "repres": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 20, 23], "represent": [2, 3, 5, 8, 20, 23], "representd": 5, "reproduc": [0, 1, 2, 7, 8, 11, 14, 17, 20, 23], "repuls": [2, 23], "request": [2, 15], "requir": [0, 2, 3, 5, 6, 7, 8, 10, 11, 13, 14, 15, 18, 23], "res1": 4, "res2": 4, "res3": 4, "res_analyt": 4, "res_analytical1": 4, "res_analytical2": 4, "res_analytical3": 4, "resaml": 8, "resampl": [2, 9, 12, 17, 23], "rescal": [2, 13, 14], "rescu": 7, "reseach": 8, "research": [2, 6, 15, 17, 22, 23], "resembl": [8, 20], "reserv": [3, 7, 8, 20], "reshap": [2, 3, 4, 5, 6, 8, 10, 11, 12, 18, 23], "residenti": 2, "residu": [2, 7, 15, 23], "resiz": 7, "resourc": 23, "respect": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 16, 20, 23], "respond": 14, "respons": [2, 9, 11, 14, 23], "rest": [2, 7], "restat": [2, 14, 23], "restor": 6, "restored_discrimin": 6, "restored_gener": 6, "restrict": [2, 5, 11, 14, 23], "result": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "retail": 2, "retain": [7, 8], "return": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 18, 20, 23], "return_data": 16, "return_sequ": 6, "return_x_i": 11, "reus": [3, 5, 8], "reveal": [2, 14, 23], "revers": [3, 18], "review": [17, 18], "revisit": 16, "revolut": 23, "reward": [2, 6, 23], "rewrit": [1, 2, 5, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20], "rewritten": [4, 8, 10, 12, 20], "rewrot": 15, "rf": 12, "rgb": 5, "rgoj5yh7evk": 17, "rh": 8, "rho": [2, 12, 15], "rho_1": 12, "rho_2": 12, "rho_m": 12, "rich": [2, 23], "ride": 11, "rideclass": 11, "ridedata": 11, "ridg": [9, 13, 15, 17, 23], "ridge_sk": 8, "ridgetheta": 7, "right": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 23], "right_sid": 4, "rightarrow": [2, 3, 7, 8, 10, 13, 14, 15, 20, 23], "rigor": [2, 23], "ring": 8, "rise": [2, 23], "risk": [2, 15, 23], "rival": 6, "river": 2, "rlm": 23, "rm": [2, 20], "rmse": 2, "rmsporp": 15, "rmsprop": [3, 5, 6, 15], "rnd_clf": 12, "rng": 20, "rnn": [6, 14], "rnn1": 6, "rnn2": 6, "rnn_2layer": 6, "rnn_input": 6, "rnn_output": 6, "rnn_train": 6, "rntrick1": 20, "rntrick2": 20, "rntrick3": 20, "rntrick4": 20, "ro": [2, 15, 23], "robert": 22, "robust": [2, 23], "robustscal": 2, "roc": [9, 12], "role": [2, 4, 7, 8, 10, 17, 23], "roll": 8, "room": [2, 21, 23], "root": [0, 2, 7, 11, 15, 20], "rot": 23, "rotat": [3, 10, 11, 12], "rotation_matrix": 11, "roughli": [3, 5], "round": [2, 9, 11, 15], "routin": [15, 18, 23], "row": [1, 2, 3, 4, 7, 8, 11, 13, 18, 23], "rr": 7, "rrr": 7, "rug": 15, "rule": [2, 3, 7, 8, 15, 23], "run": [0, 2, 3, 4, 6, 7, 8, 10, 11, 13, 15, 17, 23], "runtim": [0, 3, 8, 16], "rust": [2, 17, 18, 23], "rvert": 3, "rvert_2": 3, "s_": [5, 8], "s_1": 8, "s_i": [8, 9], "s_j": 8, "s_k": 8, "saddl": 15, "sai": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 20, 23], "said": [8, 11, 15], "sake": [2, 7, 9, 13, 23], "sale": [2, 23], "sam": 23, "same": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 16, 18, 20, 23], "samm": 12, "sampl": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 16, 17, 18, 20, 23], "sample_vari": 16, "sampleexptvari": 20, "samwis": 23, "sastri": 13, "satisfactori": [2, 23], "satisfi": [3, 4, 5, 8, 10, 15, 18, 20], "satur": [3, 8], "save": [2, 6, 8, 9, 11, 15, 23], "save_fig": [2, 8, 9, 11, 12, 23], "savefig": [2, 6, 8, 9, 11, 20, 23], "savetxt": 6, "saw": 7, "scalabl": 12, "scalar": [4, 7, 8, 12], "scale": [2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 21, 23], "scale_mean": 6, "scale_std": 6, "scaler": [2, 9, 10, 11, 12, 13], "scan": [7, 9], "scari": 7, "scatter": [0, 2, 3, 8, 9, 10, 11, 16, 23], "scenario": [8, 15], "schedul": 15, "scheme": [3, 15], "schrage": 20, "sch\u00f8yen": 8, "scienc": [2, 3, 12, 14, 15, 17, 19, 20, 21, 22], "scientif": [2, 17, 23], "scientist": [2, 23], "scikit": [0, 1, 5, 7, 8, 10, 11, 12, 15, 17, 18, 22], "scikit_learn": 2, "scikitlearn": 23, "scikitplot": [9, 12], "scipi": [2, 5, 7, 8, 15, 17, 18, 23], "scl": 8, "scm": 0, "score": [0, 1, 2, 3, 5, 8, 9, 11, 12, 13, 21, 23], "scores_kfold": 8, "scratch": [1, 3, 15], "sdg": 15, "seaborn": [2, 3, 5, 8, 9, 23], "seamless": [2, 17, 23], "search": [0, 2, 3, 5, 7, 11, 15, 23], "sebastian": 23, "sebastianraschka": 23, "sec": 8, "second": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 16, 17, 18, 20, 21, 23], "secondeigvector": 13, "secondli": 14, "section": [1, 6, 13, 18, 20], "sector": 2, "see": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 18, 20, 23], "seed": [2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 15, 16, 20, 23], "seed_imag": 6, "seek": [3, 4, 10], "seem": [3, 5, 6], "seemingli": [2, 23], "seen": [2, 3, 5, 7, 12, 14, 20], "segment": 15, "seismic": 8, "seldomli": [2, 23], "select": [0, 3, 7, 8, 10, 11, 12, 13, 19, 20, 21, 22, 23], "selevet": 0, "self": [3, 7], "sell": 6, "semest": [9, 19], "semi": [10, 15], "semilogx": 8, "send": [7, 14, 15, 21, 23], "senior": [19, 21], "sens": [2, 6, 8, 10, 23], "sensibl": 5, "sensit": [2, 7, 8, 11, 15, 23], "sent": 4, "sentenc": [6, 14], "separ": [2, 3, 4, 6, 8, 10, 11, 14, 16, 17, 20, 23], "septemb": 23, "sequenc": [5, 6, 9, 11, 12, 14, 15, 17, 18, 20, 23], "sequenti": [3, 5, 6, 12, 14, 20], "seri": [2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 18, 23], "serif": [9, 20, 23], "serv": [2, 3, 4, 5, 7, 9, 15, 22, 23], "session": [0, 3, 19, 21, 23], "set": [1, 3, 6, 7, 8, 9, 10, 12, 13, 15, 16, 17, 18, 20, 21], "set_major_formatt": 8, "set_major_loc": 8, "set_tick": [3, 10], "set_ticklabel": 3, "set_titl": [2, 3, 4, 5, 9, 14, 16, 23], "set_xlabel": [2, 3, 4, 5, 9, 14, 23], "set_xlim": [9, 14], "set_xticklabel": 3, "set_ylabel": [2, 3, 4, 5, 9, 23], "set_ylim": [9, 14], "set_ytick": 9, "set_yticklabel": [3, 8], "set_zlim": 8, "seth": 6, "setminu": 8, "setosa": [10, 11], "setosa_or_versicolor": 10, "setp": 8, "setup": [3, 6, 8, 10, 17, 23], "sever": [1, 2, 5, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23], "sgd": [3, 5], "sgd_clf": 10, "sgdclassifi": 10, "sgdreg": 15, "sgdregressor": 15, "sgn": 7, "shallow": 15, "shape": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 23], "share": [0, 3, 5, 23], "shareabl": 0, "she": 9, "shift": [0, 3, 8, 14, 20], "ship": 5, "shire": 23, "short": [6, 7], "shortcom": 15, "shorten": 6, "shorter": 20, "shorthand": 23, "shortli": [18, 23], "should": [0, 2, 4, 5, 7, 8, 10, 11, 13, 14, 18, 20, 23], "show": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "show_shap": 6, "shown": [2, 6, 7, 10, 14, 15, 18], "shrink": [5, 7, 8, 10, 13], "shrinkag": [7, 8], "shrunk": 13, "shuffl": [2, 3, 6, 8, 15], "side": [2, 4, 7, 10, 14, 15, 18, 23], "sigh": [17, 23], "sigma": [2, 3, 7, 8, 9, 12, 13, 14, 15, 18, 20, 23], "sigma0": 20, "sigma1": 20, "sigma2": 20, "sigma_": [7, 18, 23], "sigma_0": 7, "sigma_1": 7, "sigma_2": 7, "sigma_fn": [9, 14], "sigma_i": [2, 7, 23], "sigma_j": 7, "sigma_m": [8, 20], "sigma_n": [13, 20], "sigma_t": 15, "sigma_x": 20, "sigmoid": [3, 4, 6, 9, 10, 12, 14], "sigmundson": 8, "sign": [3, 4, 9, 10, 12, 20, 21], "signal": [3, 5, 12, 14], "signifi": 6, "signific": 3, "significantli": [3, 15, 20], "sim": [6, 7, 8, 15, 20], "similar": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 23], "similarli": [2, 3, 5, 7, 10, 12, 15, 20, 23], "simpl": [1, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20], "simplepredict": 12, "simpler": [1, 2, 3, 7, 8, 9, 15, 17, 23], "simplernn": 6, "simplest": [2, 3, 5, 6, 11, 12, 14, 16, 23], "simpletre": 12, "simpli": [2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 17, 18, 20, 23], "simplic": [4, 7, 8, 9, 10, 11, 12, 13, 14, 16], "simplicti": 7, "simplifi": [2, 8, 11, 17, 23], "simplist": [5, 8, 20], "simul": 8, "simultan": 8, "sin": [2, 3, 4, 5, 6, 11, 14, 15, 18, 23], "sinc": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 22, 23], "sine": [5, 14], "singl": [2, 3, 4, 5, 7, 8, 9, 10, 11, 14, 15, 18, 20, 23], "singular": [2, 8, 15, 18, 23], "sinusoid": 5, "site": [2, 19, 23], "situat": [2, 6, 7, 9, 15, 20, 23], "six": [5, 20], "size": [2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 18, 20, 23], "sketch": 12, "ski": 11, "skill": 2, "skip": 13, "skl": [2, 8, 23], "sklearn": [0, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 23], "skplt": [9, 12], "sl": 8, "slack": 10, "slice": [4, 18, 23], "slide": [1, 2, 5, 20, 23], "slight": [8, 15], "slightli": [3, 4, 5, 7, 8, 9, 12, 20], "slope": [10, 13, 14], "slow": [2, 4, 10, 15], "slower": [7, 18, 23], "slowest": 18, "slowli": 14, "slp": 3, "small": [2, 3, 4, 5, 7, 8, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23], "smaller": [2, 3, 4, 7, 8, 10, 11, 13, 15, 20, 23], "smallest": [2, 6, 16, 23], "smallest_row_index": 16, "smooth": [2, 5, 8, 15, 23], "sn": [2, 3, 5, 8, 9, 23], "sne": 13, "so": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "soar": 8, "social": 2, "soft": [3, 9, 12, 14], "soften": 10, "softmax": [5, 9], "softwar": [2, 10, 17, 18], "sol": 10, "sole": [2, 8, 23], "solid": [2, 9], "solut": [2, 3, 4, 5, 7, 8, 10, 12, 13, 15, 18, 20, 23], "soluton": 4, "solv": [1, 2, 3, 5, 7, 8, 10, 12, 13, 14, 15, 18, 23], "solve_expdec": 4, "solve_ode_deep_neural_network": 4, "solve_ode_neural_network": 4, "solve_pde_deep_neural_network": 4, "solveod": 4, "solveode_popul": 4, "solver": [4, 9, 10, 11, 12, 18, 23], "some": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 20, 23], "some_model": 8, "somehow": 6, "someon": 1, "someth": [0, 2, 3, 5, 6, 9, 11, 13, 20, 23], "sometim": [2, 3, 13, 14, 15, 16], "soon": [18, 21], "sophist": [2, 23], "sopt": 15, "sort": [7, 8, 11, 13, 20], "sound": [5, 7], "sourc": [2, 3, 5, 8, 17, 18, 20, 23], "space": [2, 3, 6, 7, 10, 11, 13, 14, 15, 16, 20], "span": [2, 5, 7, 11, 13, 18, 23], "spare": 3, "spars": [5, 8, 18, 23], "sparse_mtx": [18, 23], "sparsecategoricalcrossentropi": 5, "sparsiti": 12, "spatial": [3, 4, 5, 14], "speak": 20, "special": [8, 9, 12, 14, 15, 18, 20, 23], "specif": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 17, 18, 20, 22, 23], "specifi": [2, 5, 7, 8, 9, 11, 13, 15, 16, 20, 23], "specifici": [2, 12, 23], "spectacular": 5, "spectral": 3, "speech": [2, 3, 5, 6, 14], "speed": [3, 4, 6, 15], "spend": [1, 20], "sphere": 2, "spin": 8, "spite": 2, "spline": 10, "split": [1, 3, 5, 6, 7, 8, 10, 11, 12, 13, 16, 20, 23], "splite": 2, "splitter": [3, 12], "spontan": 20, "spot": 5, "spread": [2, 13, 20, 23], "springer": [22, 23], "spuriou": 15, "sqrsignal": 5, "sqrt": [2, 5, 6, 7, 8, 10, 12, 13, 15, 20], "squar": [0, 3, 4, 5, 6, 9, 10, 11, 13, 15, 16, 17, 18, 20], "squarederror": 12, "squaredeuclidean": 16, "squash": 14, "srtm": 8, "srtm_data_norway_1": 8, "stabil": 7, "stabl": [1, 2, 6, 7, 8, 11, 17, 23], "stack": [5, 6], "stage": [0, 7, 15], "stai": [2, 4, 6, 7, 13, 23], "stand": [2, 7, 11, 14, 23], "standard": [2, 3, 6, 7, 8, 9, 10, 12, 14, 18, 20, 23], "standardscal": [2, 8, 9, 10, 11, 12, 13], "stanford": 15, "start": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23], "start_tim": 16, "stat": 8, "state": [3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 20, 23], "statement": [2, 9, 18, 23], "statist": [2, 3, 5, 6, 9, 11, 12, 13, 14, 15, 16, 18, 22], "statu": [0, 2, 9, 23], "stavang": 8, "std": [2, 6, 8, 23], "steep": 15, "step": [0, 2, 3, 4, 6, 8, 9, 11, 12, 13, 14, 15, 16, 18, 23], "step_fn": [9, 14], "step_length": 15, "steps_list": 11, "stereo": 5, "still": [2, 4, 5, 7, 8, 13, 15, 20], "stimuli": 14, "stk": [22, 23], "stk2100": [22, 23], "stk3155": [0, 19, 21], "stk4021": [22, 23], "stk4051": [22, 23], "stk4155": [19, 21], "stk5000": 22, "stochast": [2, 3, 7, 8, 10, 13, 14, 23], "stock": 6, "stoke": 14, "stone": [2, 9], "stop": [3, 6, 11, 15, 16], "storag": 7, "store": [2, 3, 4, 5, 8, 13, 15, 20, 23], "storehaug": [21, 23], "str": [3, 5, 6], "straight": [2, 8, 10, 15, 23], "straightforward": [2, 4, 5, 7, 8, 10, 11, 12, 15, 18, 23], "strategi": [2, 3, 11, 23], "stratifi": 8, "strength": [2, 7, 16], "stretch": 13, "strict": [10, 15], "strictli": [10, 15], "stride": [6, 18], "strike": 8, "string": 3, "stroke": 9, "strong": [5, 8, 11, 12, 14, 18, 20], "strongli": [0, 2, 10, 17, 18], "stronli": 2, "structur": [2, 3, 4, 5, 8, 11, 12, 14, 17, 23], "stuck": [3, 15], "student": [0, 2, 19, 21, 22, 23], "studi": [2, 5, 6, 7, 8, 9, 10, 13, 14, 15, 17, 22, 23], "studier": 22, "style": [9, 11, 18, 23], "st\u00f8land": 21, "sub": [11, 14], "subdivid": [2, 18, 23], "subfield": 2, "subject": [8, 10, 20], "submit": 23, "subplot": [2, 3, 5, 6, 8, 9, 10, 11, 12, 16, 23], "subplots_adjust": [10, 20], "subprogram": [18, 23], "subract": 2, "subroutin": [2, 23], "subscript": 3, "subsequ": [3, 6, 7, 8, 14, 18, 20], "subset": [3, 8, 11, 14, 15, 17, 23], "subspac": [2, 10, 13], "substanti": [11, 12], "substep": 13, "substitut": [1, 5, 8, 14, 18], "subsubset": 11, "subtask": 8, "subtl": 3, "subtract": [2, 6, 7, 8, 13, 15, 18, 20], "subtre": 11, "succeed": [2, 6, 23], "success": [5, 9, 11, 15, 20], "successfulli": [6, 11], "sudo": [2, 17, 23], "suffer": [2, 3, 4, 7, 12, 23], "suffici": [3, 8, 10, 13, 15], "suggest": [3, 15, 22], "suit": [10, 14], "suitabl": [0, 2, 20], "sum": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "sum_": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "sum_i": [2, 4, 7, 8, 10, 15], "sum_j": 8, "sum_ja_": 2, "sum_k": [8, 10, 14, 18], "sum_logist": 15, "sum_m": 5, "sum_n": 5, "sum_nx_": 5, "summar": [7, 8, 11], "summari": [3, 5, 6, 12, 19], "summat": [1, 2, 5], "sunni": 11, "super": 7, "superfici": 5, "superscript": [3, 14], "supervis": [2, 7, 8, 9, 11, 14, 17, 23], "supplement": 9, "support": [2, 3, 11, 12, 13, 15, 17, 23], "suppos": [2, 7, 8, 9, 10, 12, 13, 14, 15, 18, 23], "suppress": [7, 15], "sure": [1, 2, 3, 6, 8], "surf": 8, "surfac": [2, 8, 23], "surpass": 8, "surpris": [2, 23], "surround": [5, 17], "survei": [2, 7, 8, 23], "svc": [10, 11, 12], "svd": [2, 8, 13, 23], "svdinv": 7, "svm": [10, 11, 12, 13], "svm_clf": [10, 12], "swath": 7, "switch": 2, "sy": 15, "symbol": [3, 7, 13, 15, 17, 20, 23], "symmeteri": 3, "symmetr": [2, 7, 10, 13, 14, 15, 18, 23], "symmetri": 8, "sympi": [2, 17, 23], "synonim": 20, "syntax": 15, "system": [0, 2, 3, 5, 6, 8, 9, 11, 12, 14, 15, 17, 18, 23], "systemat": [6, 8], "t": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "t0": [5, 8, 15], "t1": [4, 15], "t2": 4, "t3": 4, "t_": 4, "t_0": [4, 11, 15], "t_1": 15, "t_b": 12, "t_i": [3, 4, 7, 14], "t_j": 14, "t_k": 11, "tabl": [11, 20, 21, 23], "tabul": [2, 23], "tabular": 23, "tackl": 6, "tag": [4, 5, 6, 7, 8, 9, 14, 15, 16, 18, 20], "taht": [2, 23], "tail": 20, "tailor": [4, 10, 13, 23], "taiwan": [2, 23], "take": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23], "taken": [2, 3, 5, 8, 12, 15, 18], "tan": 5, "tangent": [3, 6, 14, 15], "tanh": [3, 6, 9, 10, 14], "target": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 23], "target_nam": 11, "task": [2, 3, 5, 8, 11, 13, 14, 16, 23], "tau": [5, 7, 20], "taught": 23, "tax": 2, "taylor": [4, 15], "taylornr": 15, "tc": 10, "teach": [0, 19, 23], "team": 3, "teaser": 2, "technic": [2, 7, 8, 15], "techniqu": [2, 3, 10, 12, 15, 17, 20, 22, 23], "technologi": [2, 3], "tell": [1, 2, 6, 8, 12, 13, 15, 20], "temp": 3, "temp1": 3, "temp2": 3, "temperatur": [2, 11, 23], "temporarili": 3, "ten": [5, 23], "tend": [5, 7, 8, 10, 11, 12, 14, 15, 16], "tendenc": [2, 23], "tension": 8, "tensor": 5, "tensorflow": [2, 4, 6, 10, 16, 17, 18, 22, 23], "term": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 23], "term1": [7, 8, 13], "term2": [7, 8, 13], "term3": [7, 8, 13], "term4": [7, 8, 13], "termin": [0, 2, 6, 7, 11, 12, 15], "terrain": 8, "terrain1": 8, "test": [1, 5, 6, 7, 8, 9, 10, 11, 12, 15, 18, 20, 23], "test_acc": 5, "test_accuraci": [3, 5], "test_error": 8, "test_imag": [5, 6], "test_ind": 8, "test_input": 6, "test_label": [5, 6], "test_loss": 5, "test_pr": 3, "test_predict": 3, "test_rnn": 6, "test_scor": [9, 12], "test_siz": [0, 2, 3, 5, 7, 8, 12], "test_split": 11, "testerror": [2, 8], "testi": 6, "testpredict": 6, "testx": 6, "text": [2, 3, 4, 6, 7, 10, 11, 13, 15, 18, 20, 22], "textbook": 1, "textual": 11, "textur": 3, "tf": [3, 5, 6, 15, 16], "th": [2, 3, 4, 7, 8, 9, 11, 14, 15, 16, 18, 20, 23], "than": [2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 17, 20, 23], "thank": [6, 8], "theano": [3, 17, 23], "thei": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 23], "them": [2, 3, 5, 6, 8, 10, 11, 12, 13, 14, 15, 18, 23], "theme": [0, 2, 23], "themselv": [2, 20, 23], "thenc": 8, "theorem": [4, 8, 9], "theoret": [2, 6, 12], "theori": [2, 3, 5, 10, 11, 14, 15, 17, 22, 23], "thereaft": [2, 7, 8, 13, 14, 18, 23], "therebi": [2, 7, 9, 13, 23], "therefor": [2, 3, 4, 5, 6, 8, 9, 10, 13, 15, 20, 23], "therein": 13, "thereof": [2, 8, 15, 23], "theta": [1, 2, 3, 6, 7, 8, 9, 15, 20, 23], "theta_": [2, 3, 8, 9, 15, 23], "theta_0": [1, 2, 7, 8, 9, 23], "theta_0x_": [2, 23], "theta_1": [2, 7, 8, 9, 23], "theta_1x_": [2, 23], "theta_1x_0": [2, 23], "theta_1x_1": [2, 9, 23], "theta_1x_2": [2, 23], "theta_1x_i": 9, "theta_2": [2, 23], "theta_2x_": [2, 23], "theta_2x_0": [2, 23], "theta_2x_1": [2, 23], "theta_2x_2": [2, 9, 23], "theta_i": [2, 3, 7, 23], "theta_j": [2, 7, 8, 23], "theta_linreg": 15, "theta_p": 9, "theta_px_p": 9, "theta_t": 15, "thetavalu": 7, "thi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22], "thing": [0, 1, 2, 3, 4, 6, 7, 9, 11, 20, 23], "think": [2, 3, 5, 6, 8, 11, 14, 15, 16, 20, 23], "third": [2, 5, 8, 15, 21, 23], "thirti": 9, "thorughout": 23, "those": [2, 5, 7, 8, 10, 11, 12, 13, 18, 23], "though": [1, 3, 4, 5, 6, 15, 18, 20], "thought": [8, 16, 20], "thousand": [2, 3], "three": [2, 3, 5, 7, 8, 10, 11, 14, 18, 19, 20, 21, 23], "threshold": [3, 5, 11, 12, 13, 14, 15], "through": [0, 2, 3, 4, 5, 6, 7, 8, 10, 13, 14, 15, 16, 17, 18, 20, 23], "throughout": [0, 2, 6, 7, 16, 17, 18, 20, 23], "throw": [5, 8, 20], "thu": [2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 21, 23], "thumb": [2, 8], "tibshirani": [8, 22, 23], "tick_param": 8, "ticker": [8, 15, 20], "tif": 8, "tight_layout": [3, 9], "tightli": 13, "tild": [2, 7, 8, 9, 13, 20, 23], "till": [2, 6, 9, 10, 11, 12, 14, 18, 23], "time": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "timeit": 6, "timer": 6, "tini": 3, "tip": 5, "titl": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 15, 20, 23], "tmp": 15, "tn": [4, 5, 9], "to_categor": [3, 5, 6], "to_categorical_numpi": 3, "to_numer": [2, 8, 23], "todai": 5, "togeth": [2, 5, 8, 10, 13, 15, 17, 23], "toi": 16, "told": 15, "toler": [4, 16], "tolist": 6, "tomographi": 14, "too": [2, 4, 6, 7, 8, 11, 13, 15, 20, 22], "took": [10, 23], "tool": [0, 2, 3, 5, 8, 15, 17], "toolbox": 10, "top": [2, 5, 7, 8, 11, 12, 17, 23], "topic": [2, 7, 8, 9, 10, 17], "topolog": [5, 14], "topologi": [3, 14], "torkjellsdatt": [21, 23], "toss": [12, 20], "total": [2, 3, 4, 5, 6, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 21, 23], "total_loss": 6, "totalclustervari": 16, "totalscatt": 16, "toward": [0, 3, 4, 9, 14, 15], "town": 2, "tp": [6, 9], "tpng": 11, "tpu": [15, 17, 23], "tqdm": 8, "track": [0, 5, 15, 16, 18], "tract": 2, "tractabl": [2, 23], "trade": [7, 11], "tradeoff": [2, 7, 23], "tradit": [2, 3, 6, 8, 23], "train": [1, 4, 5, 7, 8, 10, 11, 12, 13, 14, 15], "train_accuraci": [2, 3, 5, 23], "train_dataset": 6, "train_end": [2, 3], "train_error": 8, "train_imag": [5, 6], "train_ind": 8, "train_label": [5, 6], "train_pr": 3, "train_siz": [2, 3, 5], "train_step": 6, "train_test_split": [0, 1, 2, 3, 5, 7, 8, 9, 11, 12, 13, 23], "train_test_split_numpi": [2, 3], "trainable_vari": 6, "trained_model": 8, "trainerror": 2, "traini": 6, "training_checkpoint": 6, "training_dataset": 6, "training_gradi": 15, "trainingerror": 8, "trainpredict": 6, "trainscor": 6, "trainx": 6, "trait": [2, 23], "trajectori": 6, "transfer": [11, 23], "transform": [2, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 23], "transit": [8, 14], "translat": [3, 6, 8, 12, 23], "transpos": [3, 7, 13, 18], "travers": [2, 7], "treat": [2, 3, 5, 8, 14, 15, 20, 23], "tree": [2, 3, 17, 23], "tree_clf": [11, 12], "tree_clf_": 11, "tree_clf_sr": 11, "tree_reg": 11, "tree_reg1": 11, "tree_reg2": 11, "trend": 20, "treue": 9, "trevor": 22, "tri": [1, 4, 5, 6, 11, 15], "triain": 2, "trial": [2, 4, 6, 8, 15, 20, 23], "triangl": 15, "triangular": 18, "trick": [5, 6, 10, 13, 15, 20], "trickier": 20, "tridiagon": 18, "trillion": 17, "trivial": [2, 3, 7, 13, 20, 23], "troubl": [0, 2, 10, 14], "truck": 5, "true": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 23], "true_fun": 8, "true_theta": 8, "truli": 23, "try": [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 20, 23], "tucker": 10, "tuesdai": [21, 23], "tumor": [9, 11], "tumour": 9, "tunabl": 3, "tune": [6, 11, 15, 18, 23], "turn": [2, 3, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "tutori": [3, 6], "tv": 4, "tveito": 4, "tweak": [3, 6, 12, 20], "twice": 15, "twist": 13, "two": [0, 2, 3, 4, 6, 7, 8, 9, 11, 12, 13, 14, 15, 18, 19, 20, 22, 23], "tx": 15, "tx_1": 15, "txt": [0, 6], "ty": 15, "type": [2, 3, 5, 8, 10, 12, 15, 18, 20], "typic": [0, 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 14, 15, 20, 23], "u": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 22, 23], "u_": 18, "u_i": 14, "u_m": 12, "ua": [2, 23], "ubuntu": [2, 17, 23], "uci": 2, "uio": [0, 21, 22], "un": 16, "unabl": 0, "unari": [18, 23], "unbalanc": [8, 11], "unbias": [2, 7, 8, 23], "uncent": 8, "uncertainti": [2, 7, 23], "uncertitud": 20, "unchang": [3, 5], "uncorrel": [12, 20], "undefin": 7, "under": [2, 3, 7, 8, 12, 15, 17, 23], "underdetermin": [2, 23], "underfit": [3, 8], "underflowproblem": 7, "undergo": 7, "undergradu": [19, 21], "underli": [2, 3, 11, 15, 20, 23], "underset": [6, 16], "understand": [0, 2, 3, 5, 7, 8, 12, 15, 16, 17, 23], "understood": [10, 15], "undesir": 10, "undetermin": [7, 10], "undo": 6, "unexpect": 8, "unexpected": 20, "unfair": 8, "unfortun": [3, 10, 11, 12], "unicode_liter": [10, 11], "uniform": [2, 3, 7, 8, 13, 15, 20, 23], "uniformli": [15, 20], "unifrompdf": 20, "unimport": 15, "union": [7, 8], "uniqu": [2, 4, 8, 15, 16, 18, 23], "unique_cluster_label": 16, "unit": [2, 3, 5, 6, 7, 12, 14, 20, 23], "unitari": [7, 8, 18], "unitarili": [18, 23], "uniti": 20, "univari": 20, "univers": [2, 3, 4, 15, 17, 19, 21, 23], "unix": 3, "unknow": [2, 18, 23], "unknown": [2, 3, 5, 6, 7, 8, 10, 12, 15, 18, 23], "unknowwn": 14, "unlabel": 3, "unless": [2, 5, 8, 13, 15, 23], "unlik": [3, 5, 10, 15], "unnecessarili": 11, "unord": 5, "unravel": 3, "unrol": [5, 13], "unseen": [0, 2, 9, 11], "unstabl": 3, "unsupervis": [2, 3, 6, 14, 17, 23], "unsymmetr": [18, 23], "until": [3, 4, 6, 11, 14, 15, 16], "untouch": 2, "unusu": 14, "up": [1, 3, 5, 6, 7, 8, 10, 12, 13, 15, 16, 17, 18, 20, 21], "updat": [0, 3, 4, 12, 14, 15, 16], "uploa": 23, "upload": [0, 17, 22], "upon": [2, 3, 8, 9, 13, 18], "upper": [1, 2, 10, 11, 18], "uppercas": [18, 23], "upsampl": 6, "upscal": 6, "url": 23, "us": [0, 6, 7, 8, 10, 11, 12, 13, 14, 16, 18, 20, 22], "usag": [2, 10, 17, 23], "usd": 2, "usd10000": 2, "use_bia": 6, "usecol": [2, 23], "useless": 3, "user": [2, 3, 4, 6, 8, 9, 17, 18, 23], "usernam": 0, "usetex": 20, "usg": 8, "usr": 20, "usual": [2, 5, 6, 9, 14, 15, 16, 23], "ut": 7, "util": [3, 5, 6, 8, 9, 12, 16, 23], "ux": 18, "v": [0, 2, 4, 6, 7, 8, 13, 15, 17], "v0": 20, "v1": 20, "v2": 20, "v_0": 13, "va": 3, "vahid": 23, "val": 15, "val_accuraci": 5, "val_loss": 6, "vale": 4, "valid": [2, 3, 6, 9, 11, 12, 15, 17, 20, 23], "validation_data": 5, "validation_split": 6, "valu": [1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 23], "valuat": 11, "valy": 6, "van": [2, 23], "vandenbergh": [10, 15], "vandermond": [2, 23], "vanilla": [2, 8, 13, 16], "vanish": [3, 6, 15, 20], "var": [7, 8, 12, 13, 20], "var_x": 20, "varabl": 10, "varepsilon": [7, 8], "varepsilon_": [7, 8], "varepsilon_i": [7, 8], "vari": [2, 3, 5, 7, 8, 12, 23], "variabl": [2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 23], "varianc": [2, 3, 7, 9, 11, 12, 13, 15, 16, 17, 18, 20, 23], "variance_i": [7, 13], "variance_x": [7, 13], "variant": [2, 3, 8, 10, 14, 15, 23], "variat": [5, 6, 13, 23], "varieti": [2, 5, 14, 17, 23], "variou": [1, 3, 5, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23], "varydimens": 6, "vastli": 5, "vaue": 3, "vault": 2, "vdot": [4, 15], "vec": 8, "vector": [2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 17], "vector_mean": 16, "ventur": [2, 10, 17, 23], "venv": 0, "verbos": [3, 5, 6], "veri": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 22, 23], "verifi": [5, 13, 18, 23], "versatil": [10, 23], "versicolor": [10, 11], "version": [0, 2, 5, 12, 15, 16, 17, 18, 20, 23], "versu": 3, "vert": [1, 2, 3, 7, 8, 9, 10, 11, 13, 15, 23], "vert_1": [7, 8], "vert_2": [7, 8, 13], "via": [2, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 20, 21, 23], "vidal": 13, "video": [2, 3, 14, 17, 19, 21, 23], "view": [3, 5, 7, 8, 14, 15, 20, 22, 23], "violat": 10, "virginica": 11, "viridi": [2, 3, 4, 5, 23], "virtual": 3, "viscos": 15, "viscou": 15, "visibl": 0, "vision": [2, 5], "visual": [2, 5, 13, 14, 17, 23], "visualis": 3, "visualstudio": [0, 1], "viz": [8, 10, 20], "vmap": 15, "vmax": [3, 8], "vmin": [3, 8], "voic": 5, "volum": [2, 5, 23], "vote": [12, 23], "voting_clf": 12, "votingclassifi": 12, "votingsimpl": 12, "vstack": [7, 13, 18, 20, 23], "vt": 7, "w": [2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 23], "w1": 10, "w2": [10, 13], "w3": 10, "w_": [3, 14], "w_1": [10, 18], "w_1x_": 10, "w_1x_1": 10, "w_2": [10, 18], "w_2x_": 10, "w_2x_2": 10, "w_3": 18, "w_4": 18, "w_hidden": 4, "w_i": [3, 4, 12], "w_ix_i": 14, "w_j": 18, "w_m": 18, "w_output": 4, "w_px_": 10, "w_px_p": 10, "wa": [2, 3, 5, 6, 7, 8, 9, 12, 13, 14, 16, 18, 23], "wai": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 23], "walk": 11, "walker": 20, "wang": [2, 23], "want": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23], "warn": [6, 23], "warrant": 8, "wast": 5, "watch": 17, "wave": 5, "wavelet": 10, "we": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22], "weak": [11, 12, 16], "weather": [3, 14], "web": [17, 19, 21, 23], "webpag": 23, "websit": [8, 18, 19, 23], "wedg": [10, 20], "wednesdai": [21, 23], "wee": 13, "week": [2, 7, 8, 9, 19, 21], "weekli": [0, 1, 17, 19, 21, 22, 23], "weight": [2, 3, 4, 5, 8, 9, 11, 12, 14, 15, 20], "weigth": 4, "welcom": [0, 10, 17], "well": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18, 20, 22, 23], "went": 10, "were": [2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 20, 23], "wessel": [2, 23], "what": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20], "whatev": 5, "when": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "whenev": [0, 15, 20], "where": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "wherea": [8, 20], "wherein": [3, 14], "whether": [2, 5, 7, 9, 11, 20, 23], "which": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23], "whichev": [3, 5], "while": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 20, 23], "white": 11, "who": [0, 2], "whole": [3, 5, 6, 7, 11, 13, 15], "whose": [2, 8, 12, 20], "whow": 13, "why": [0, 1, 2, 3, 5, 8, 15], "wide": [2, 3, 5, 8, 9, 14, 17, 18, 23], "widehat": 8, "width": [2, 5, 10, 11, 23], "wieringen": [2, 23], "win": 12, "wind": 11, "wing": [21, 23], "winther": 4, "wiothout": 8, "wiscons": 9, "wisconsin": 12, "wisdom": 8, "wise": [2, 3, 7, 14, 15], "wish": [2, 4, 7, 9, 10, 13, 15, 16, 18, 23], "with_std": 2, "wither": 8, "within": [2, 4, 5, 6, 9, 11, 14, 15, 16, 20, 22, 23], "withinclust": 16, "without": [0, 2, 3, 7, 8, 10, 11, 13, 14, 15, 23], "won": [0, 2, 23], "wonder": 10, "word": [2, 3, 5, 6, 7, 8, 9, 16, 20, 23], "work": [0, 1, 2, 3, 6, 8, 9, 10, 11, 15, 17, 19, 20, 21, 23], "workshop": 23, "world": [1, 2, 10], "worldwid": [2, 23], "worri": 0, "wors": [2, 3, 5, 6, 8, 23], "worth": 11, "would": [1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23], "wrap": [8, 18, 23], "write": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 14, 15, 18, 23], "written": [1, 2, 4, 5, 7, 13, 14, 15, 17, 18, 20, 23], "wrong": [0, 3, 10], "wrongli": 12, "wrote": [7, 13], "wrt": [12, 15], "wth": [12, 15], "www": [17, 18, 22, 23], "wx_1": 10, "x": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "x0": 10, "x1": [6, 10, 11, 12, 15], "x1_exampl": 10, "x1d": 10, "x2": [10, 11, 12, 15], "x2d": [10, 13], "x2d_train": 13, "x2dsl": 13, "x3": 10, "x_": [2, 4, 5, 7, 8, 10, 12, 13, 15, 16, 18, 20, 23], "x_0": [2, 7, 13, 18, 23], "x_1": [2, 4, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 23], "x_2": [2, 4, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 23], "x_3": [10, 18, 20], "x_4": 18, "x_center": 13, "x_data": 3, "x_data_ful": 3, "x_hidden": 4, "x_i": [2, 3, 4, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "x_input": 4, "x_ix_": [2, 23], "x_iy_i": 10, "x_j": [1, 2, 4, 10, 11, 14, 20], "x_jy_j": 10, "x_k": [14, 16, 18, 20], "x_l": 20, "x_m": [8, 14, 18, 20], "x_n": [2, 4, 5, 8, 10, 13, 14, 15, 18, 20, 23], "x_new": [11, 12], "x_offset": 8, "x_output": 4, "x_p": [5, 9, 11], "x_poli": 11, "x_poly10": 11, "x_pred": 6, "x_prev": 4, "x_reduc": 13, "x_scale": 10, "x_small": 15, "x_test": [0, 1, 2, 3, 5, 7, 8, 9, 11, 12, 13], "x_test_own": 8, "x_test_scal": [2, 8, 9, 11, 12, 13], "x_tot": 6, "x_train": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 23], "x_train_mean": 8, "x_train_own": 8, "x_train_scal": [2, 8, 9, 11, 12, 13], "x_val": 3, "xarrai": [17, 23], "xavier": 3, "xbnew": 15, "xcode": [2, 17, 23], "xdclassiffierconfus": 12, "xdclassiffierroc": 12, "xg_clf": 12, "xgb": 12, "xgbclassifi": 12, "xgboost": 11, "xgboot": 12, "xgbregressor": 12, "xgparam": 12, "xgtree": 12, "xi": [10, 15], "xi_": 10, "xi_1": 10, "xi_i": 10, "xk": 10, "xla": [15, 17, 23], "xlabel": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 20, 23], "xlim": [8, 12], "xm": 11, "xmesh": 15, "xnew": [2, 15, 23], "xp": 20, "xpanda": 2, "xpd": [7, 13], "xplot": 2, "xscale": 2, "xsr": 11, "xt_x": 15, "xtest": 8, "xtick": [5, 8, 10, 11], "xtrain": 8, "xu": [2, 23], "xx": [2, 18, 23], "xy": [2, 8, 10, 18, 23], "xytext": 10, "xz": [18, 23], "y": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "y1": 6, "y2": 6, "y3": 6, "y_": [2, 3, 7, 8, 12, 13, 18, 23], "y_0": [2, 7, 13, 18, 23], "y_1": [2, 7, 10, 11, 13, 15, 18, 23], "y_1y_1": 10, "y_1y_1k": 10, "y_1y_2": 10, "y_1y_2k": 10, "y_1y_n": 10, "y_1y_nk": 10, "y_2": [2, 7, 10, 11, 13, 18, 23], "y_2y_1": 10, "y_2y_1k": 10, "y_2y_2": 10, "y_2y_2k": 10, "y_3": [2, 11, 18], "y_4": 18, "y_data": [2, 3, 7, 8, 23], "y_data_ful": 3, "y_decis": 10, "y_fit": 2, "y_i": [2, 3, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 23], "y_if_": 12, "y_ix_": [2, 23], "y_ix_i": [9, 10, 15], "y_iy_jk": 10, "y_j": [8, 10, 14], "y_k": 14, "y_m": 18, "y_model": [2, 6, 7, 8, 23], "y_n": [10, 15], "y_ny_1": 10, "y_ny_1k": 10, "y_ny_2": 10, "y_ny_2k": 10, "y_ny_n": 10, "y_ny_nk": 10, "y_offset": 8, "y_plot": 11, "y_pred": [2, 3, 6, 8, 9, 10, 11, 12], "y_pred1": 11, "y_pred2": 11, "y_pred_rf": 12, "y_pred_tre": 12, "y_proba": [9, 12], "y_scaler": 8, "y_test": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13], "y_test_onehot": 3, "y_test_predict": 2, "y_tot": 6, "y_train": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 23], "y_train_mean": 8, "y_train_onehot": 3, "y_train_predict": 2, "y_train_scal": 8, "y_val": 3, "ye": [5, 8, 9], "year": [2, 17, 23], "yet": [2, 3, 8, 10, 13, 15, 23], "yi": 15, "yield": [2, 4, 7, 8, 10, 12, 14, 15, 16, 18, 20, 23], "yk": 10, "ylabel": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 20, 23], "ylim": [5, 8], "ym": 11, "ymesh": 15, "yn": 2, "yo": [10, 11, 12], "yoshua": [3, 22], "you": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 20, 21, 22, 23], "young": 2, "your": [0, 3, 4, 6, 7, 8, 10, 13, 15, 17, 18, 23], "your_model_object": 1, "yourself": [13, 15, 23], "youtub": 17, "ypred": 8, "ypredict": [2, 15, 23], "ypredict2": 15, "ypredictlasso": 7, "ypredictol": [2, 7], "ypredictown": 8, "ypredictownridg": 8, "ypredictridg": [2, 7, 8], "ypredictskl": 8, "ytest": 8, "ytick": [5, 8, 10, 11], "ytild": [2, 8, 23], "ytildelasso": 7, "ytildenp": [2, 23], "ytildeol": [2, 7], "ytildeownridg": 8, "ytilderidg": [7, 8], "ytrain": 8, "yuxi": 23, "yx": [18, 23], "yy": [18, 23], "yz": [18, 23], "z": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 23], "z_": [3, 4, 14, 18, 23], "z_0": [18, 23], "z_1": [18, 23], "z_2": [18, 23], "z_c": 3, "z_h": 3, "z_hidden": 4, "z_i": [3, 14], "z_j": [3, 14], "z_k": 14, "z_m": 3, "z_mod": 11, "z_o": 3, "z_output": 4, "zaman": 20, "zaxi": 8, "zero": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "zeros_lik": 6, "zfill": 6, "zip": [6, 8], "zm_h": [2, 23], "zn": 2, "zone": 2, "zoom": 23, "zx": [18, 23], "zy": [18, 23], "zz": [18, 23], "\u00f8yvind": 8}, "titles": ["Exercises week 34", "Exercises week 35", "3. Linear Regression", "14. Building a Feed Forward Neural Network", "15. Solving Differential Equations  with Deep Learning", "16. Convolutional Neural Networks", "17. Recurrent neural networks: Overarching view", "4. Ridge and Lasso Regression", "5. Resampling Methods", "6. Logistic Regression", "8. Support Vector Machines, overarching aims", "9. Decision trees, overarching aims", "10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods", "11. Basic ideas of the Principal Component Analysis (PCA)", "13. Neural networks", "7. Optimization, the central part of any Machine Learning algortithm", "12. Clustering and Unsupervised Learning", "Applied Data Analysis and Machine Learning", "2. Linear Algebra, Handling of Arrays and more Python Features", "Course setting", "1. Elements of Probability Theory and Statistical Data Analysis", "Teachers and Grading", "Textbooks", "Week 34: Introduction to the course, Logistics and Practicalities"], "titleterms": {"": [10, 12], "1": [0, 1, 2], "2": [0, 1, 2, 23], "2023": 21, "3": [0, 1, 2], "34": [0, 23], "35": 1, "4": [0, 1, 2], "5": [1, 2], "A": [2, 3, 6, 10, 11, 23], "And": 23, "In": 21, "Ising": 8, "The": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 17, 23], "To": 23, "With": 6, "about": 23, "activ": [3, 14], "ad": [2, 8, 23], "adaboost": 12, "adagrad": 15, "adam": 15, "adapt": 12, "adjust": 3, "adversari": 6, "again": [5, 11], "ai": 23, "aim": [10, 11, 23], "aka": 23, "algebra": [18, 23], "algorithm": [11, 12, 13, 14, 23], "algortithm": 15, "all": 10, "an": [0, 2, 6, 12, 23], "analys": 7, "analysi": [2, 7, 8, 13, 17, 20, 23], "analyt": [1, 2], "ani": 15, "anoth": 11, "appli": 17, "approach": [2, 10, 16, 23], "approxim": 14, "architectur": 3, "arrai": [18, 23], "assist": 21, "autocorrel": 20, "autograd": [4, 15], "automat": 15, "back": [3, 13, 14], "background": 17, "bag": 12, "base": 15, "basic": [2, 7, 9, 11, 12, 13, 18], "batch": 3, "bay": 7, "befor": 13, "better": 10, "bia": 8, "binari": 3, "bind": 23, "bird": 12, "boost": 12, "bootstrap": [8, 12], "boston": 2, "breast": 3, "brief": 23, "bring": 14, "build": [3, 5, 11], "c": 23, "can": 23, "cancer": [3, 9, 11, 13], "cart": 11, "case": [10, 12, 20], "central": [15, 17, 20], "chain": 14, "chang": 12, "channel": 23, "chi": [2, 23], "choos": 3, "cifar01": 5, "classic": 13, "classif": [3, 11, 12], "classifi": 10, "clip": 3, "cluster": 16, "cnn": 5, "code": [0, 1, 2, 3, 4, 7, 11, 13, 14, 15, 16, 23], "collect": [3, 5], "commun": 23, "compar": [1, 4, 12], "complex": [2, 8], "complic": 8, "compon": 13, "comput": 11, "computerlab": 23, "con": 11, "concept": 20, "conjug": 15, "contn": 23, "convex": [10, 15], "convolut": [5, 14], "correl": 13, "cost": [3, 12], "cours": [17, 19, 22, 23], "covari": [7, 13, 20], "cover": 23, "creat": 1, "cross": 8, "cython": 23, "data": [0, 2, 3, 5, 8, 9, 11, 13, 17, 20, 23], "dataset": [3, 5], "david": 23, "deadlin": 23, "deadllin": 21, "decai": 4, "decis": [11, 12], "decomposit": [7, 13, 18], "deeep": 23, "deep": [3, 4, 23], "defin": [3, 23], "degre": 2, "deliver": [0, 1], "dens": 2, "deriv": [1, 7, 14], "descent": [4, 12, 15], "detail": [5, 23], "develop": 3, "diagon": 13, "differ": 10, "differenti": [4, 15], "diffus": 4, "dimension": [4, 5, 10], "disadvantag": 11, "discret": 20, "discrimin": 23, "distribut": [7, 20], "do": 3, "domain": 20, "down": 3, "dropout": 3, "element": [2, 20, 23], "elimin": 18, "energi": 23, "ensembl": 12, "entropi": 11, "environ": [0, 2], "equat": [2, 4, 14], "error": [2, 12, 23], "essenti": 23, "etc": 23, "euler": 4, "evalu": 3, "exampl": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 23], "exercis": [0, 1, 2, 8], "expect": 20, "experi": 20, "explor": 2, "exponenti": 4, "express": 1, "extrapol": 6, "extrem": [12, 23], "ey": 12, "fall": 21, "famili": [3, 23], "famou": 18, "featur": [1, 11, 18], "feed": [3, 14], "final": 14, "find": 1, "fine": 3, "first": [6, 14, 23], "fit": [0, 1, 2, 12, 23], "forc": 5, "forest": 12, "format": 23, "forward": [3, 4, 14], "foster": 23, "fourier": 5, "frank": 8, "freedom": 2, "frequentist": [2, 23], "from": [7, 12, 14, 23], "full": 4, "function": [2, 3, 8, 9, 10, 12, 13, 14, 15, 20, 23], "further": [5, 7], "gan": 6, "gaussian": 18, "gd": 15, "gener": [6, 11, 23], "geometr": 13, "gini": 11, "github": 0, "goal": [0, 1], "good": [2, 23], "grade": [21, 23], "gradient": [3, 4, 12, 15], "growth": 4, "ha": 17, "handl": [18, 23], "hidden": 4, "hous": 2, "how": 1, "hyperparamet": 3, "hyperplan": 10, "i": [2, 3, 23], "id3": 11, "idea": 13, "ii": 23, "implement": [1, 3], "implic": 7, "import": [7, 18, 23], "improv": 3, "includ": 15, "increment": 13, "index": 11, "inform": 21, "input": 4, "instal": [17, 23], "instructor": 21, "interpret": [7, 13, 23], "introduc": [13, 15], "introduct": [2, 8, 17, 18, 23], "invers": [7, 18], "iter": 12, "jax": 15, "julia": 23, "jungl": 12, "kera": [3, 5], "kernel": [10, 13], "lagrangian": 10, "lasso": [7, 8], "later": 7, "layer": [3, 4, 5, 14], "learn": [0, 1, 2, 3, 4, 13, 15, 16, 17, 23], "least": [1, 7, 8, 23], "lectur": 23, "level": 12, "librari": [17, 23], "likelihood": 9, "limit": [3, 15, 20], "linear": [0, 2, 10, 15, 18, 23], "link": [7, 13, 22], "logist": [9, 23], "lu": 18, "machin": [2, 10, 15, 17, 23], "main": [20, 23], "make": [2, 11, 12], "mani": [12, 14], "mass": 23, "materi": 23, "math": 7, "mathemat": [5, 7, 10], "matric": [7, 18, 23], "matrix": [1, 3, 7, 13, 14, 18, 23], "matter": 2, "mean": 2, "meet": [7, 12, 20, 23], "mercer": 10, "method": [8, 11, 12, 15, 23], "minim": 23, "ml": 23, "mlp": 14, "mnist": [5, 6], "model": [0, 2, 3, 6, 8, 14, 23], "momentum": 15, "moon": [10, 11], "more": [5, 8, 18, 23], "multilay": 14, "multipl": [3, 5], "multipli": 10, "need": 23, "network": [3, 4, 5, 6, 9, 14, 23], "neural": [3, 4, 5, 6, 9, 14, 23], "new": 6, "non": 10, "normal": [2, 3], "notat": 14, "now": [3, 11, 15], "nuclear": [2, 23], "numba": 23, "number": [2, 4, 20], "numer": [4, 20], "numpi": [18, 23], "object": 5, "obtain": 13, "od": 4, "off": 8, "ol": [0, 1, 7, 8], "one": [4, 14], "oper": 18, "optim": [3, 10, 15, 17, 23], "order": 15, "ordinari": [1, 7, 8, 23], "organ": [2, 23], "oslo": 22, "other": [6, 11, 13, 14, 18, 23], "our": [2, 6, 7, 13, 15, 23], "outcom": [17, 23], "output": 4, "overarch": [2, 6, 10, 11, 23], "overview": [12, 23], "own": [2, 12, 13, 23], "packag": [18, 23], "panda": 23, "paramet": 23, "part": [15, 17], "partial": 4, "pass": 3, "pca": 13, "pdf": 20, "perceptron": 14, "perform": [3, 11], "period": 5, "perspect": 3, "plethora": 23, "point": 6, "poisson": 4, "polynomi": [1, 5], "popul": 4, "popular": 23, "practic": [15, 21, 23], "pre": [3, 5], "predict": 6, "prerequisit": [5, 17, 23], "princip": 13, "principl": 5, "pro": 11, "probabl": [7, 20], "problem": [3, 4, 15, 23], "procedur": [11, 23], "process": [3, 5], "program": [4, 15], "project": [8, 21, 23], "prop": 15, "propag": [3, 14], "properti": [7, 20], "python": [0, 2, 11, 17, 18, 23], "quick": 10, "r": 23, "random": [12, 13, 20], "read": [11, 23], "real": [8, 23], "recommend": 23, "recurr": [6, 14], "reduc": 2, "reduct": 5, "reformul": 4, "regress": [0, 2, 7, 8, 9, 11, 12, 15, 23], "regular": 3, "relev": 22, "relu": 3, "remark": 5, "remind": [8, 10, 23], "replac": 15, "repositori": 0, "requir": [4, 17], "resampl": 8, "rescal": 8, "resourc": 4, "revisit": 15, "rewrit": 23, "ridg": [2, 7, 8], "rm": 15, "rule": 14, "same": 15, "sampl": 13, "schedul": 23, "schemat": 11, "scheme": 4, "scienc": 23, "scikit": [2, 3, 13, 23], "second": 15, "semest": 21, "set": [0, 2, 4, 5, 11, 14, 19, 23], "setup": 0, "sgd": 15, "should": 3, "similar": 15, "simpl": [2, 6, 11, 15, 23], "singl": 12, "singular": [7, 13], "sklearn": 1, "soft": 10, "softmax": 3, "softwar": 23, "solv": 4, "solver": 15, "some": [15, 18], "specifi": 4, "split": [0, 2], "squar": [1, 2, 7, 8, 12, 23], "standard": 15, "state": 2, "statist": [7, 8, 17, 20, 23], "steepest": [12, 15], "stochast": [15, 20], "strongli": 23, "suggest": 23, "summari": [21, 23], "superposit": 5, "supervis": 3, "support": 10, "svd": 7, "systemat": 5, "take": 1, "taken": 23, "teach": 21, "teacher": [21, 23], "techniqu": [8, 13], "technologi": 17, "tensorflow": [3, 5], "tent": [21, 23], "test": [0, 2, 3], "text": 23, "textbook": [22, 23], "theorem": [7, 10, 13, 14, 20], "theori": 20, "thi": 23, "tip": 15, "togeth": 14, "tool": 23, "top": 3, "topic": 23, "toward": 13, "trade": 8, "tradeoff": 8, "train": [0, 2, 3, 6, 23], "transform": 5, "tree": [11, 12], "tune": 3, "two": [5, 10, 17], "type": [4, 6, 14, 23], "uio": 23, "univers": [14, 22], "unsupervis": 16, "up": [0, 2, 4, 11, 14, 23], "us": [1, 2, 3, 4, 5, 9, 15, 17, 23], "v": 5, "valid": 8, "valu": [7, 13, 20], "variabl": 20, "varianc": 8, "variou": 2, "vector": [1, 10, 14, 18, 23], "versu": 23, "view": [2, 6, 12], "virtual": 0, "visual": [3, 11], "wai": 11, "wave": 4, "we": 23, "week": [0, 1, 23], "what": [2, 23], "which": 3, "why": 23, "wisconsin": 9, "write": [6, 13], "xgboost": 12, "your": [1, 2, 12]}})
\ No newline at end of file
diff --git a/doc/LectureNotes/_build/jupyter_execute/E2.ipynb b/doc/LectureNotes/_build/jupyter_execute/E2.ipynb
index 1512be049..6b559d91e 100644
--- a/doc/LectureNotes/_build/jupyter_execute/E2.ipynb
+++ b/doc/LectureNotes/_build/jupyter_execute/E2.ipynb
@@ -146,9 +146,9 @@
    "id": "f3f771de",
    "metadata": {},
    "source": [
-    "The ordinary least squares method finds the parameters $\\boldsymbol{\\beta}$ which minimizes the squared error between our model $\\boldsymbol{X\\beta}$ and the true values $\\boldsymbol{y}$.\n",
+    "The ordinary least squares method finds the parameters $\\boldsymbol{\\theta}$ which minimizes the squared error between our model $\\boldsymbol{X\\theta}$ and the true values $\\boldsymbol{y}$.\n",
     "\n",
-    "To find the parameters $\\boldsymbol{\\beta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\\boldsymbol{\\beta}$, and set it equal to 0."
+    "To find the parameters $\\boldsymbol{\\theta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\\boldsymbol{\\theta}$, and set it equal to 0."
    ]
   },
   {
@@ -156,7 +156,7 @@
    "id": "49690237",
    "metadata": {},
    "source": [
-    "**a)** Very briefly explain why the approach above finds the parameters $\\boldsymbol{\\beta}$ which minimizes this error."
+    "**a)** Very briefly explain why the approach above finds the parameters $\\boldsymbol{\\theta}$ which minimizes this error."
    ]
   },
   {
@@ -167,13 +167,13 @@
     "We typically write the squared error as\n",
     "\n",
     "$$\n",
-    "\\vert\\vert\\boldsymbol{y} - \\boldsymbol{X\\beta}\\vert\\vert^2\n",
+    "\\vert\\vert\\boldsymbol{y} - \\boldsymbol{X\\theta}\\vert\\vert^2\n",
     "$$\n",
     "\n",
     "which we can rewrite in matrix-vector form as\n",
     "\n",
     "$$\n",
-    "\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\n",
+    "\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\n",
     "$$"
    ]
   },
@@ -182,7 +182,7 @@
    "id": "8fbecf74",
    "metadata": {},
    "source": [
-    "**b)** If $\\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\\boldsymbol{\\beta}$? (**Hint:** Don't compute any derivatives, but solve $\\boldsymbol{X\\beta}=\\boldsymbol{y}$ for $\\boldsymbol{\\beta}$)"
+    "**b)** If $\\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\\boldsymbol{\\theta}$? (**Hint:** Don't compute any derivatives, but solve $\\boldsymbol{X\\theta}=\\boldsymbol{y}$ for $\\boldsymbol{\\theta}$)"
    ]
   },
   {
@@ -202,10 +202,10 @@
    "id": "869fca4d",
    "metadata": {},
    "source": [
-    "**d)** Using the expression from **c)**, but substituting back in $\\boldsymbol{\\beta}$, $\\boldsymbol{y}$ and $\\boldsymbol{X}$, find the expression for the optimal parameters $\\boldsymbol{\\beta}$ in the case that $\\boldsymbol{X}$ is not invertible, but $\\boldsymbol{X^T X}$ is, which is most often the case.\n",
+    "**d)** Using the expression from **c)**, but substituting back in $\\boldsymbol{\\theta}$, $\\boldsymbol{y}$ and $\\boldsymbol{X}$, find the expression for the optimal parameters $\\boldsymbol{\\theta}$ in the case that $\\boldsymbol{X}$ is not invertible, but $\\boldsymbol{X^T X}$ is, which is most often the case.\n",
     "\n",
     "$$\n",
-    "\\boldsymbol{\\hat{\\beta}_{OLS}} = ...\n",
+    "\\boldsymbol{\\hat{\\theta}_{OLS}} = ...\n",
     "$$"
    ]
   },
@@ -222,18 +222,18 @@
    "id": "5dc179f7",
    "metadata": {},
    "source": [
-    "With the expression for $\\boldsymbol{\\hat{\\beta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\\boldsymbol{X}$.\n",
+    "With the expression for $\\boldsymbol{\\hat{\\theta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\\boldsymbol{X}$.\n",
     "\n",
     "In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, with the montly income and the number of children as columns.\n",
     "\n",
     "We typically also include an intercept in our models. The intercept is a value that is added to our prediction regardless of the value of the other features. The intercept tries to account for constant effects in our data that are not dependant on anything else. In our current example, the intercept could account for living expenses which are typical regardless of income or childcare expenses.\n",
     "\n",
-    "We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter $\\beta_0$ from the OLS method into the optimal intercept value (which will be $\\beta_0$). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix."
+    "We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter $\\theta_0$ from the OLS method into the optimal intercept value (which will be $\\theta_0$). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "id": "e5ff2a69",
    "metadata": {},
    "outputs": [],
@@ -243,7 +243,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
    "id": "a3cf2792",
    "metadata": {},
    "outputs": [],
@@ -264,7 +264,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
    "id": "5ad87a65",
    "metadata": {},
    "outputs": [],
@@ -285,7 +285,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 4,
    "id": "8f3f68aa",
    "metadata": {},
    "outputs": [],
@@ -316,7 +316,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 5,
    "id": "d7476c84",
    "metadata": {},
    "outputs": [],
@@ -336,7 +336,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 6,
    "id": "91496e40",
    "metadata": {},
    "outputs": [],
@@ -362,7 +362,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 7,
    "id": "034f502c",
    "metadata": {},
    "outputs": [],
@@ -380,7 +380,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 8,
    "id": "29171358",
    "metadata": {},
    "outputs": [],
@@ -400,10 +400,21 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 9,
    "id": "1e346f4c",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Ellipsis"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
     "..."
    ]
@@ -418,10 +429,21 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 10,
    "id": "ceb57457",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Ellipsis"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
     "..."
    ]
@@ -495,7 +517,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": ".venv",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -509,7 +531,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.13.0"
+   "version": "3.9.15"
   }
  },
  "nbformat": 4,
diff --git a/doc/LectureNotes/_build/jupyter_execute/chapter1.ipynb b/doc/LectureNotes/_build/jupyter_execute/chapter1.ipynb
index 92702e774..4b22b16cd 100644
--- a/doc/LectureNotes/_build/jupyter_execute/chapter1.ipynb
+++ b/doc/LectureNotes/_build/jupyter_execute/chapter1.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "d66e8057",
+   "id": "a453b968",
    "metadata": {
     "editable": true
    },
@@ -13,7 +13,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3f6464d0",
+   "id": "499b2ddb",
    "metadata": {
     "editable": true
    },
@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fab3a8f9",
+   "id": "c84cce7e",
    "metadata": {
     "editable": true
    },
@@ -65,7 +65,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "66513db3",
+   "id": "8419208e",
    "metadata": {
     "editable": true
    },
@@ -167,7 +167,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0d1f19f9",
+   "id": "ceb7a805",
    "metadata": {
     "editable": true
    },
@@ -202,7 +202,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0479c66e",
+   "id": "d6c1062f",
    "metadata": {
     "editable": true
    },
@@ -253,7 +253,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "609d8174",
+   "id": "50c1b706",
    "metadata": {
     "editable": true
    },
@@ -286,7 +286,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e580e18",
+   "id": "12e3ac84",
    "metadata": {
     "editable": true
    },
@@ -298,7 +298,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bb8f332d",
+   "id": "10b4c333",
    "metadata": {
     "editable": true
    },
@@ -335,7 +335,7 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "9e479f11",
+   "id": "3e216ef0",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -368,7 +368,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8c7b818b",
+   "id": "af765a12",
    "metadata": {
     "editable": true
    },
@@ -385,7 +385,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "201888d1",
+   "id": "2f6fd730",
    "metadata": {
     "editable": true
    },
@@ -397,7 +397,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e6a16258",
+   "id": "1770d85d",
    "metadata": {
     "editable": true
    },
@@ -418,7 +418,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c3aab7c4",
+   "id": "684b72a0",
    "metadata": {
     "editable": true
    },
@@ -431,7 +431,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f2957240",
+   "id": "e9fa8fd5",
    "metadata": {
     "editable": true
    },
@@ -462,7 +462,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2a15d7ec",
+   "id": "15a4642f",
    "metadata": {
     "editable": true
    },
@@ -474,7 +474,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12af21ef",
+   "id": "589ec9cb",
    "metadata": {
     "editable": true
    },
@@ -492,7 +492,7 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "16f86276",
+   "id": "cf21ae2b",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -520,7 +520,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "87da55cc",
+   "id": "41c55cbb",
    "metadata": {
     "editable": true
    },
@@ -542,7 +542,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "221ebafe",
+   "id": "d62c9412",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -580,7 +580,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "aa05a791",
+   "id": "ae0c6c2a",
    "metadata": {
     "editable": true
    },
@@ -591,7 +591,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bac50889",
+   "id": "64cf13de",
    "metadata": {
     "editable": true
    },
@@ -604,7 +604,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "825fa9b8",
+   "id": "18439dae",
    "metadata": {
     "editable": true
    },
@@ -625,7 +625,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a3fa2a46",
+   "id": "3e9fb291",
    "metadata": {
     "editable": true
    },
@@ -637,7 +637,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7627c75a",
+   "id": "46d8744e",
    "metadata": {
     "editable": true
    },
@@ -647,7 +647,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "52686744",
+   "id": "38a29b65",
    "metadata": {
     "editable": true
    },
@@ -659,7 +659,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f6cb9efa",
+   "id": "438e73b8",
    "metadata": {
     "editable": true
    },
@@ -671,7 +671,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4822580d",
+   "id": "1403cc6a",
    "metadata": {
     "editable": true
    },
@@ -683,7 +683,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fb61578b",
+   "id": "9fbd1c1b",
    "metadata": {
     "editable": true
    },
@@ -694,7 +694,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6cb0d520",
+   "id": "d25da1d3",
    "metadata": {
     "editable": true
    },
@@ -706,7 +706,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "87835e2a",
+   "id": "c43c7452",
    "metadata": {
     "editable": true
    },
@@ -728,7 +728,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "69fc895e",
+   "id": "ad0d5004",
    "metadata": {
     "editable": true
    },
@@ -740,7 +740,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "debfac94",
+   "id": "eb9e2efb",
    "metadata": {
     "editable": true
    },
@@ -755,7 +755,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "96164cb3",
+   "id": "963f4f84",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -796,7 +796,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7cac7e62",
+   "id": "d530f60a",
    "metadata": {
     "editable": true
    },
@@ -811,7 +811,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8b9b62fe",
+   "id": "f8951307",
    "metadata": {
     "editable": true
    },
@@ -823,7 +823,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "81afb7da",
+   "id": "913bb6a6",
    "metadata": {
     "editable": true
    },
@@ -833,7 +833,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "266b9b6d",
+   "id": "25cb82ce",
    "metadata": {
     "editable": true
    },
@@ -845,7 +845,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "26bfbb29",
+   "id": "d04f788d",
    "metadata": {
     "editable": true
    },
@@ -855,7 +855,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e4ee0e4",
+   "id": "866bf69f",
    "metadata": {
     "editable": true
    },
@@ -867,7 +867,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "40627d20",
+   "id": "a0d3421f",
    "metadata": {
     "editable": true
    },
@@ -877,7 +877,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b4fbd650",
+   "id": "680eefd0",
    "metadata": {
     "editable": true
    },
@@ -889,7 +889,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "40eecb7a",
+   "id": "de97e875",
    "metadata": {
     "editable": true
    },
@@ -905,7 +905,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "97f2b230",
+   "id": "5d7dbc49",
    "metadata": {
     "editable": true
    },
@@ -917,7 +917,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "34eed0f8",
+   "id": "5a9a6d4d",
    "metadata": {
     "editable": true
    },
@@ -928,7 +928,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "37ca2a5b",
+   "id": "fef247ce",
    "metadata": {
     "editable": true
    },
@@ -940,7 +940,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "36071972",
+   "id": "3ed61266",
    "metadata": {
     "editable": true
    },
@@ -954,7 +954,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "32a6cbaf",
+   "id": "9882fb4e",
    "metadata": {
     "editable": true
    },
@@ -966,7 +966,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "68d9be7d",
+   "id": "992b3ae7",
    "metadata": {
     "editable": true
    },
@@ -991,7 +991,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7ec5f81c",
+   "id": "6d772376",
    "metadata": {
     "editable": true
    },
@@ -1008,7 +1008,7 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "fb39ace0",
+   "id": "f861cd3f",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1052,40 +1052,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fc71e4e1",
-   "metadata": {
-    "editable": true
-   },
-   "source": [
-    "Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various **matplotlib** commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "30e59ca4",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
-   "outputs": [],
-   "source": [
-    "from pylab import plt, mpl\n",
-    "plt.style.use('seaborn')\n",
-    "mpl.rcParams['font.family'] = 'serif'\n",
-    "\n",
-    "def MakePlot(x,y, styles, labels, axlabels):\n",
-    "    plt.figure(figsize=(10,6))\n",
-    "    for i in range(len(x)):\n",
-    "        plt.plot(x[i], y[i], styles[i], label = labels[i])\n",
-    "        plt.xlabel(axlabels[0])\n",
-    "        plt.ylabel(axlabels[1])\n",
-    "    plt.legend(loc=0)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e58a020b",
+   "id": "da27a328",
    "metadata": {
     "editable": true
    },
@@ -1101,8 +1068,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
-   "id": "de0f27ba",
+   "execution_count": 6,
+   "id": "b4ef8b5a",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1123,7 +1090,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5da52984",
+   "id": "1352f7d2",
    "metadata": {
     "editable": true
    },
@@ -1136,8 +1103,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
-   "id": "ba67d882",
+   "execution_count": 7,
+   "id": "c5b3333d",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1166,7 +1133,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b75681d2",
+   "id": "6652db3a",
    "metadata": {
     "editable": true
    },
@@ -1185,8 +1152,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
-   "id": "d9760898",
+   "execution_count": 8,
+   "id": "715c03ae",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1203,7 +1170,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5199cc2b",
+   "id": "eaff6065",
    "metadata": {
     "editable": true
    },
@@ -1214,8 +1181,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
-   "id": "d48c42f2",
+   "execution_count": 9,
+   "id": "4053e205",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1233,7 +1200,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "91ca751e",
+   "id": "587289d6",
    "metadata": {
     "editable": true
    },
@@ -1248,8 +1215,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
-   "id": "ea82f2ad",
+   "execution_count": 10,
+   "id": "52f1c9dc",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1262,7 +1229,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0fb84108",
+   "id": "25241a4e",
    "metadata": {
     "editable": true
    },
@@ -1273,8 +1240,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
-   "id": "3b24153e",
+   "execution_count": 11,
+   "id": "06020cef",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1304,7 +1271,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d9acc526",
+   "id": "8d07d616",
    "metadata": {
     "editable": true
    },
@@ -1314,8 +1281,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
-   "id": "942b4d36",
+   "execution_count": 12,
+   "id": "1c4a2fd6",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1356,7 +1323,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8fe6a29e",
+   "id": "b9728189",
    "metadata": {
     "editable": true
    },
@@ -1377,8 +1344,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
-   "id": "5dc4b33c",
+   "execution_count": 13,
+   "id": "8b5b6d13",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1418,7 +1385,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8dd1a395",
+   "id": "ec654f7a",
    "metadata": {
     "editable": true
    },
@@ -1427,12 +1394,12 @@
     "\n",
     "[Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureAug27.mp4?vrtx=view-as-webpage).\n",
     "\n",
-    "Fitting a continuous function with linear parameterization in terms of the parameters  $\\boldsymbol{\\beta}$.\n",
+    "Fitting a continuous function with linear parameterization in terms of the parameters  $\\boldsymbol{\\theta}$.\n",
     "* Method of choice for fitting a continuous function!\n",
     "\n",
     "* Gives an excellent introduction to central Machine Learning features with **understandable pedagogical** links to other methods like **Neural Networks**, **Support Vector Machines** etc\n",
     "\n",
-    "* Analytical expression for the fitting parameters $\\boldsymbol{\\beta}$\n",
+    "* Analytical expression for the fitting parameters $\\boldsymbol{\\theta}$\n",
     "\n",
     "* Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more\n",
     "\n",
@@ -1450,14 +1417,14 @@
     "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n",
     "\n",
     "Regression modeling deals with the description of  the sampling distribution of a given random variable $y$ and how it varies as function of another variable or a set of such variables $\\boldsymbol{x} =[x_0, x_1,\\dots, x_{n-1}]^T$. \n",
-    "The first variable is called the **dependent**, the **outcome** or the **response** variable while the set of variables $\\boldsymbol{x}$ is called the independent variable, or the predictor variable or the explanatory variable. \n",
+    "The first variable is called the **dependent**, the **outcome** or the **response** or just the output variable while the set of variables $\\boldsymbol{x}$ is called the independent variable, or the predictor variable or the explanatory variable, or just the input variable. We will hereafter call $\\boldsymbol{y}$ and $\\boldsymbol{x}x$ for the output and input variables, respectively. \n",
     "\n",
-    "A regression model aims at finding a likelihood function $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$, that is the conditional distribution for $\\boldsymbol{y}$ with a given $\\boldsymbol{x}$. The estimation of  $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$ is made using a data set with \n",
+    "A regression model aims at finding a likelihood function $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$ (or just a function $f(\\boldsymbol{x}$) , that is the conditional distribution for $\\boldsymbol{y}$ with a given $\\boldsymbol{x}$. The estimation of  $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$ is made using a data set with \n",
     "* $n$ cases $i = 0, 1, 2, \\dots, n-1$ \n",
     "\n",
-    "* Response (target, dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \\dots, n-1$ \n",
+    "* Response/output (target, dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \\dots, n-1$ \n",
     "\n",
-    "* $p$ so-called explanatory (independent or predictor) variables $\\boldsymbol{x}_i=[x_{i0}, x_{i1}, \\dots, x_{ip-1}]$ with $i = 0, 1, 2, \\dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples.   \n",
+    "* $p$ so-called explanatory/input (independent or predictor) variables $\\boldsymbol{x}_i=[x_{i0}, x_{i1}, \\dots, x_{ip-1}]$ with $i = 0, 1, 2, \\dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples.   \n",
     "\n",
     " The goal of the regression analysis is to extract/exploit relationship between $\\boldsymbol{y}$ and $\\boldsymbol{x}$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things.\n",
     "\n",
@@ -1474,10 +1441,10 @@
     "f(\\mathbf{X}_{i,\\ast})$. When no prior knowledge on the form of\n",
     "$f(\\cdot)$ is available, it is common to assume a linear relationship\n",
     "between $\\boldsymbol{X}$ and $\\boldsymbol{y}$. This assumption gives rise to\n",
-    "the *linear regression model* where $\\boldsymbol{\\beta} = [\\beta_0, \\ldots,\n",
-    "\\beta_{p-1}]^{T}$ are the *regression parameters*. \n",
+    "the *linear regression model* where $\\boldsymbol{\\theta} = [\\theta_0, \\ldots,\n",
+    "\\theta_{p-1}]^{T}$ are the *regression parameters*. \n",
     "\n",
-    "Linear regression gives us a set of analytical equations for the parameters $\\beta_j$.\n",
+    "Linear regression gives us a set of analytical equations for the parameters $\\theta_j$.\n",
     "\n",
     "In order to understand the relation among the predictors $p$, the set of data $n$ and the target (outcome, output etc) $\\boldsymbol{y}$,\n",
     "consider the model we discussed for describing nuclear binding energies. \n",
@@ -1488,7 +1455,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9845070b",
+   "id": "08d4a54f",
    "metadata": {
     "editable": true
    },
@@ -1500,7 +1467,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fc1de086",
+   "id": "df21cf23",
    "metadata": {
     "editable": true
    },
@@ -1519,19 +1486,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e9b27dfb",
+   "id": "6359332f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "y=y(x) \\rightarrow y(x_i)=\\tilde{y}_i+\\epsilon_i=\\sum_{j=0}^{n-1} \\beta_j x_i^j+\\epsilon_i,\n",
+    "y=y(x) \\rightarrow y(x_i)=\\tilde{y}_i+\\epsilon_i=\\sum_{j=0}^{n-1} \\theta_j x_i^j+\\epsilon_i,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d80ac08b",
+   "id": "6c8e31ed",
    "metadata": {
     "editable": true
    },
@@ -1543,25 +1510,25 @@
   },
   {
    "cell_type": "markdown",
-   "id": "59de586a",
+   "id": "0cc24030",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "y_0&=\\beta_0+\\beta_1x_0^1+\\beta_2x_0^2+\\dots+\\beta_{n-1}x_0^{n-1}+\\epsilon_0\\\\\n",
-    "y_1&=\\beta_0+\\beta_1x_1^1+\\beta_2x_1^2+\\dots+\\beta_{n-1}x_1^{n-1}+\\epsilon_1\\\\\n",
-    "y_2&=\\beta_0+\\beta_1x_2^1+\\beta_2x_2^2+\\dots+\\beta_{n-1}x_2^{n-1}+\\epsilon_2\\\\\n",
+    "y_0&=\\theta_0+\\theta_1x_0^1+\\theta_2x_0^2+\\dots+\\theta_{n-1}x_0^{n-1}+\\epsilon_0\\\\\n",
+    "y_1&=\\theta_0+\\theta_1x_1^1+\\theta_2x_1^2+\\dots+\\theta_{n-1}x_1^{n-1}+\\epsilon_1\\\\\n",
+    "y_2&=\\theta_0+\\theta_1x_2^1+\\theta_2x_2^2+\\dots+\\theta_{n-1}x_2^{n-1}+\\epsilon_2\\\\\n",
     "\\dots & \\dots \\\\\n",
-    "y_{n-1}&=\\beta_0+\\beta_1x_{n-1}^1+\\beta_2x_{n-1}^2+\\dots+\\beta_{n-1}x_{n-1}^{n-1}+\\epsilon_{n-1}.\\\\\n",
+    "y_{n-1}&=\\theta_0+\\theta_1x_{n-1}^1+\\theta_2x_{n-1}^2+\\dots+\\theta_{n-1}x_{n-1}^{n-1}+\\epsilon_{n-1}.\\\\\n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "3a0a6d11",
+   "id": "146809cb",
    "metadata": {
     "editable": true
    },
@@ -1571,7 +1538,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f545dd3f",
+   "id": "4796a596",
    "metadata": {
     "editable": true
    },
@@ -1583,7 +1550,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a4484e53",
+   "id": "edfabcdf",
    "metadata": {
     "editable": true
    },
@@ -1593,19 +1560,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a1e5488",
+   "id": "0b4a6062",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta} = [\\beta_0,\\beta_1, \\beta_2,\\dots, \\beta_{n-1}]^T,\n",
+    "\\boldsymbol{\\theta} = [\\theta_0,\\theta_1, \\theta_2,\\dots, \\theta_{n-1}]^T,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "65f8b020",
+   "id": "e419ff68",
    "metadata": {
     "editable": true
    },
@@ -1615,7 +1582,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0ded825d",
+   "id": "dd55bff1",
    "metadata": {
     "editable": true
    },
@@ -1627,7 +1594,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4747b482",
+   "id": "df571b73",
    "metadata": {
     "editable": true
    },
@@ -1637,7 +1604,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1a50e6b1",
+   "id": "c1c6ffaa",
    "metadata": {
     "editable": true
    },
@@ -1656,7 +1623,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2404cda8",
+   "id": "592bff5d",
    "metadata": {
     "editable": true
    },
@@ -1666,19 +1633,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "23bc5fc5",
+   "id": "9b776152",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta}+\\boldsymbol{\\epsilon}.\n",
+    "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\theta}+\\boldsymbol{\\epsilon}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ae3d3214",
+   "id": "29c95a7a",
    "metadata": {
     "editable": true
    },
@@ -1694,27 +1661,27 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bd07fd5c",
+   "id": "b53b0001",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
-    "y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
-    "y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_2\\\\\n",
+    "y_0&=\\theta_0x_{00}+\\theta_1x_{01}+\\theta_2x_{02}+\\dots+\\theta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
+    "y_1&=\\theta_0x_{10}+\\theta_1x_{11}+\\theta_2x_{12}+\\dots+\\theta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
+    "y_2&=\\theta_0x_{20}+\\theta_1x_{21}+\\theta_2x_{22}+\\dots+\\theta_{n-1}x_{2n-1}+\\epsilon_2\\\\\n",
     "\\dots & \\dots \\\\\n",
-    "y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_i\\\\\n",
+    "y_{i}&=\\theta_0x_{i0}+\\theta_1x_{i1}+\\theta_2x_{i2}+\\dots+\\theta_{n-1}x_{in-1}+\\epsilon_i\\\\\n",
     "\\dots & \\dots \\\\\n",
-    "y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
+    "y_{n-1}&=\\theta_0x_{n-1,0}+\\theta_1x_{n-1,2}+\\theta_2x_{n-1,2}+\\dots+\\theta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "cd7bdfc0",
+   "id": "f2fea940",
    "metadata": {
     "editable": true
    },
@@ -1726,7 +1693,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "acb43d02",
+   "id": "26411342",
    "metadata": {
     "editable": true
    },
@@ -1745,7 +1712,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f1e9d7fc",
+   "id": "d432e651",
    "metadata": {
     "editable": true
    },
@@ -1755,51 +1722,51 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c41dcd08",
+   "id": "e536d0b3",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta}+\\boldsymbol{\\epsilon}.\n",
+    "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\theta}+\\boldsymbol{\\epsilon}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "89ebc548",
+   "id": "bbe0dd39",
    "metadata": {
     "editable": true
    },
    "source": [
-    "The left-hand side of this equation is kwown. Our error vector $\\boldsymbol{\\epsilon}$ and the parameter vector $\\boldsymbol{\\beta}$ are our unknow quantities. How can we obtain the optimal set of $\\beta_i$ values? \n",
+    "The left-hand side of this equation is kwown. Our error vector $\\boldsymbol{\\epsilon}$ and the parameter vector $\\boldsymbol{\\theta}$ are our unknow quantities. How can we obtain the optimal set of $\\theta_i$ values? \n",
     "\n",
     "We have defined the matrix $\\boldsymbol{X}$ via the equations"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8e0a376a",
+   "id": "51d53970",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
-    "y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
-    "y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_1\\\\\n",
+    "y_0&=\\theta_0x_{00}+\\theta_1x_{01}+\\theta_2x_{02}+\\dots+\\theta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
+    "y_1&=\\theta_0x_{10}+\\theta_1x_{11}+\\theta_2x_{12}+\\dots+\\theta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
+    "y_2&=\\theta_0x_{20}+\\theta_1x_{21}+\\theta_2x_{22}+\\dots+\\theta_{n-1}x_{2n-1}+\\epsilon_1\\\\\n",
     "\\dots & \\dots \\\\\n",
-    "y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_1\\\\\n",
+    "y_{i}&=\\theta_0x_{i0}+\\theta_1x_{i1}+\\theta_2x_{i2}+\\dots+\\theta_{n-1}x_{in-1}+\\epsilon_1\\\\\n",
     "\\dots & \\dots \\\\\n",
-    "y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
+    "y_{n-1}&=\\theta_0x_{n-1,0}+\\theta_1x_{n-1,2}+\\theta_2x_{n-1,2}+\\dots+\\theta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b2c90704",
+   "id": "4778eaaf",
    "metadata": {
     "editable": true
    },
@@ -1815,8 +1782,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
-   "id": "e89031dc",
+   "execution_count": 14,
+   "id": "87ed061d",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1896,75 +1863,75 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a1e8f92",
+   "id": "42b8f7c4",
    "metadata": {
     "editable": true
    },
    "source": [
-    "With $\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p\\times 1}$, it means that we will hereafter write our equations for the approximation as"
+    "With $\\boldsymbol{\\theta}\\in {\\mathbb{R}}^{p\\times 1}$, it means that we will hereafter write our equations for the approximation as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "84bf3673",
+   "id": "73ea9a01",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n",
+    "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\theta},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e10c4fc5",
+   "id": "253891dd",
    "metadata": {
     "editable": true
    },
    "source": [
     "throughout these lectures. \n",
     "\n",
-    "With the above we use the design matrix to define the approximation $\\boldsymbol{\\tilde{y}}$ via the unknown quantity $\\boldsymbol{\\beta}$ as"
+    "With the above we use the design matrix to define the approximation $\\boldsymbol{\\tilde{y}}$ via the unknown quantity $\\boldsymbol{\\theta}$ as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "20866df8",
+   "id": "d2d20886",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n",
+    "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\theta},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a075809a",
+   "id": "480c4c58",
    "metadata": {
     "editable": true
    },
    "source": [
-    "and in order to find the optimal parameters $\\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\\tilde{y}_i$, namely"
+    "and in order to find the optimal parameters $\\theta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\\tilde{y}_i$, namely"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ba0e251b",
+   "id": "7a782da9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n",
+    "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "93a04fb7",
+   "id": "c1c60d77",
    "metadata": {
     "editable": true
    },
@@ -1974,19 +1941,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "56c4fcf1",
+   "id": "60d26064",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
+    "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e8fc405e",
+   "id": "8a19c134",
    "metadata": {
     "editable": true
    },
@@ -1999,43 +1966,43 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fefb3fd3",
+   "id": "f21d525f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{1}{2n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2,\n",
+    "C(\\boldsymbol{\\theta})=\\frac{1}{2n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d0eeb2dc",
+   "id": "f41c45c4",
    "metadata": {
     "editable": true
    },
    "source": [
-    "since when taking the first derivative with respect to the unknown parameters $\\beta$, the factor of $2$ cancels out. \n",
+    "since when taking the first derivative with respect to the unknown parameters $\\theta$, the factor of $2$ cancels out. \n",
     "\n",
     "The function"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1f408b23",
+   "id": "b356426a",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\},\n",
+    "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "dee2a248",
+   "id": "5169a2dd",
    "metadata": {
     "editable": true
    },
@@ -2046,19 +2013,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f1d1d292",
+   "id": "19839cdf",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "y_{i}=\\langle y_i \\rangle = \\beta_0x_{i,0}+\\beta_1x_{i,1}+\\beta_2x_{i,2}+\\dots+\\beta_{n-1}x_{i,n-1}+\\epsilon_i,\n",
+    "y_{i}=\\langle y_i \\rangle = \\theta_0x_{i,0}+\\theta_1x_{i,1}+\\theta_2x_{i,2}+\\dots+\\theta_{n-1}x_{i,n-1}+\\epsilon_i,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8b2075d1",
+   "id": "e664d67f",
    "metadata": {
     "editable": true
    },
@@ -2072,25 +2039,25 @@
     "the standard deviation discussed earlier. In the discussion here we\n",
     "will treat $y_i$ as our exact value for the response variable.\n",
     "\n",
-    "In order to find the parameters $\\beta_i$ we will then minimize the spread of $C(\\boldsymbol{\\beta})$, that is we are going to solve the problem"
+    "In order to find the parameters $\\theta_i$ we will then minimize the spread of $C(\\boldsymbol{\\theta})$, that is we are going to solve the problem"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "0c3145a6",
+   "id": "a14a28ab",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
-    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
+    "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n",
+    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e72f34b6",
+   "id": "e2f643ef",
    "metadata": {
     "editable": true
    },
@@ -2100,19 +2067,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f2f9ee3f",
+   "id": "869ceba9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n",
+    "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_j} = \\frac{\\partial }{\\partial \\theta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f15f85e7",
+   "id": "371f2221",
    "metadata": {
     "editable": true
    },
@@ -2122,19 +2089,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ded06fd1",
+   "id": "b4bf3615",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)\\right]=0,\n",
+    "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}\\right)\\right]=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "dc9c099b",
+   "id": "3baf0d9a",
    "metadata": {
     "editable": true
    },
@@ -2144,19 +2111,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "40bc212d",
+   "id": "5e98ea5f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right).\n",
+    "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7bc3483f",
+   "id": "6e515735",
    "metadata": {
     "editable": true
    },
@@ -2166,19 +2133,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "22662e2a",
+   "id": "8363a7ce",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right),\n",
+    "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right),\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6798151b",
+   "id": "77eb2d81",
    "metadata": {
     "editable": true
    },
@@ -2188,19 +2155,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e63dc09e",
+   "id": "a94cb0f2",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta},\n",
+    "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a2f541a1",
+   "id": "09bdeb45",
    "metadata": {
     "editable": true
    },
@@ -2210,19 +2177,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fd8bba77",
+   "id": "d44f06e9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+    "\\boldsymbol{\\theta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2163c93e",
+   "id": "1a83dceb",
    "metadata": {
     "editable": true
    },
@@ -2246,7 +2213,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "48634f87",
+   "id": "efa701a0",
    "metadata": {
     "editable": true
    },
@@ -2258,7 +2225,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9117c119",
+   "id": "84be9b76",
    "metadata": {
     "editable": true
    },
@@ -2268,7 +2235,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c7cd5b11",
+   "id": "194914da",
    "metadata": {
     "editable": true
    },
@@ -2280,7 +2247,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1159b476",
+   "id": "da2cf79d",
    "metadata": {
     "editable": true
    },
@@ -2290,7 +2257,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "75098194",
+   "id": "7f331eee",
    "metadata": {
     "editable": true
    },
@@ -2302,7 +2269,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3bd3d52b",
+   "id": "ad13d0dc",
    "metadata": {
     "editable": true
    },
@@ -2315,19 +2282,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b086e3f6",
+   "id": "d00b722b",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial^2 C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
+    "\\frac{\\partial^2 C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}^T\\partial \\boldsymbol{\\theta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6fe2d631",
+   "id": "8e486d44",
    "metadata": {
     "editable": true
    },
@@ -2339,7 +2306,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "26bd4db6",
+   "id": "29dcd5db",
    "metadata": {
     "editable": true
    },
@@ -2351,7 +2318,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "141de40c",
+   "id": "1a45048f",
    "metadata": {
     "editable": true
    },
@@ -2366,19 +2333,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d44398a3",
+   "id": "3c8c91cc",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\epsilon} = \\boldsymbol{y}-\\boldsymbol{\\tilde{y}} = \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta},\n",
+    "\\boldsymbol{\\epsilon} = \\boldsymbol{y}-\\boldsymbol{\\tilde{y}} = \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "fb4c7d58",
+   "id": "4adc4243",
    "metadata": {
     "editable": true
    },
@@ -2388,19 +2355,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3c1a0371",
+   "id": "9c998ca0",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n",
+    "\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)= 0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "84a0c456",
+   "id": "4894c857",
    "metadata": {
     "editable": true
    },
@@ -2410,50 +2377,50 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1c5edf48",
+   "id": "80ea22e9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{X}^T\\boldsymbol{\\epsilon}=\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n",
+    "\\boldsymbol{X}^T\\boldsymbol{\\epsilon}=\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)= 0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "332a3b12",
+   "id": "e398b8f7",
    "metadata": {
     "editable": true
    },
    "source": [
-    "meaning that the solution for $\\boldsymbol{\\beta}$ is the one which minimizes the residuals.  Later we will link this with the maximum likelihood approach.\n",
+    "meaning that the solution for $\\boldsymbol{\\theta}$ is the one which minimizes the residuals.  Later we will link this with the maximum likelihood approach.\n",
     "\n",
     "Let us now return to our nuclear binding energies and simply code the above equations. \n",
     "\n",
-    "It is rather straightforward to implement the matrix inversion and obtain the parameters $\\boldsymbol{\\beta}$. After having defined the matrix $\\boldsymbol{X}$ we simply need to \n",
+    "It is rather straightforward to implement the matrix inversion and obtain the parameters $\\boldsymbol{\\theta}$. After having defined the matrix $\\boldsymbol{X}$ we simply need to \n",
     "write"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
-   "id": "f52e02a6",
+   "execution_count": 15,
+   "id": "27d5b629",
    "metadata": {
     "collapsed": false,
     "editable": true
    },
    "outputs": [],
    "source": [
-    "# matrix inversion to find beta\n",
-    "beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)\n",
+    "# matrix inversion to find theta\n",
+    "theta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)\n",
     "# and then make the prediction\n",
-    "ytilde = X @ beta"
+    "ytilde = X @ theta"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "36ae01f6",
+   "id": "a95d91a9",
    "metadata": {
     "editable": true
    },
@@ -2463,8 +2430,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
-   "id": "142d2f33",
+   "execution_count": 16,
+   "id": "4d42fedc",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2477,7 +2444,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "46296414",
+   "id": "d15ca047",
    "metadata": {
     "editable": true
    },
@@ -2487,8 +2454,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
-   "id": "86ed7e92",
+   "execution_count": 17,
+   "id": "414ab312",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2511,7 +2478,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "942647c0",
+   "id": "11bc6803",
    "metadata": {
     "editable": true
    },
@@ -2522,8 +2489,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
-   "id": "e26cee7f",
+   "execution_count": 18,
+   "id": "aaa2eeb3",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2536,7 +2503,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "462de17c",
+   "id": "b9eb34c0",
    "metadata": {
     "editable": true
    },
@@ -2546,8 +2513,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
-   "id": "e50560cf",
+   "execution_count": 19,
+   "id": "1a22333a",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2559,7 +2526,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ad325590",
+   "id": "067660ad",
    "metadata": {
     "editable": true
    },
@@ -2569,8 +2536,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
-   "id": "682dcbff",
+   "execution_count": 20,
+   "id": "25d43913",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2586,7 +2553,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a3c40b6",
+   "id": "73c9fe40",
    "metadata": {
     "editable": true
    },
@@ -2596,8 +2563,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
-   "id": "1cc29e48",
+   "execution_count": 21,
+   "id": "0ecca752",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2611,7 +2578,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1ddbddcd",
+   "id": "da2c5259",
    "metadata": {
     "editable": true
    },
@@ -2633,43 +2600,43 @@
   },
   {
    "cell_type": "markdown",
-   "id": "90a8edfe",
+   "id": "acd25436",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\chi^2(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\frac{\\left(y_i-\\tilde{y}_i\\right)^2}{\\sigma_i^2}=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\frac{1}{\\boldsymbol{\\Sigma^2}}\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n",
+    "\\chi^2(\\boldsymbol{\\theta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\frac{\\left(y_i-\\tilde{y}_i\\right)^2}{\\sigma_i^2}=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\frac{1}{\\boldsymbol{\\Sigma^2}}\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4ce63e21",
+   "id": "c3333906",
    "metadata": {
     "editable": true
    },
    "source": [
     "where the matrix $\\boldsymbol{\\Sigma}$ is a diagonal matrix with $\\sigma_i$ as matrix elements. \n",
     "\n",
-    "In order to find the parameters $\\beta_i$ we will then minimize the spread of $\\chi^2(\\boldsymbol{\\beta})$ by requiring"
+    "In order to find the parameters $\\theta_i$ we will then minimize the spread of $\\chi^2(\\boldsymbol{\\theta})$ by requiring"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d4c3d110",
+   "id": "d9069e54",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)^2\\right]=0,\n",
+    "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\theta_j} = \\frac{\\partial }{\\partial \\theta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)^2\\right]=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "cf9ad9c2",
+   "id": "2d4b51c7",
    "metadata": {
     "editable": true
    },
@@ -2679,19 +2646,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3b1f5fc1",
+   "id": "365750e2",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}\\frac{x_{ij}}{\\sigma_i}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)\\right]=0,\n",
+    "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\theta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}\\frac{x_{ij}}{\\sigma_i}\\left(\\frac{y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)\\right]=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "559f6a04",
+   "id": "16a7ed8d",
    "metadata": {
     "editable": true
    },
@@ -2701,19 +2668,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bba9e13a",
+   "id": "ce115915",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\beta}\\right).\n",
+    "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\theta}\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c97cf725",
+   "id": "10c88bf7",
    "metadata": {
     "editable": true
    },
@@ -2725,19 +2692,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6256d393",
+   "id": "c13f349e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\beta}\\right),\n",
+    "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\theta}\\right),\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4274f29f",
+   "id": "e1cfa827",
    "metadata": {
     "editable": true
    },
@@ -2747,19 +2714,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6424e32f",
+   "id": "228294e8",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{A}^T\\boldsymbol{b} = \\boldsymbol{A}^T\\boldsymbol{A}\\boldsymbol{\\beta},\n",
+    "\\boldsymbol{A}^T\\boldsymbol{b} = \\boldsymbol{A}^T\\boldsymbol{A}\\boldsymbol{\\theta},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2babfc95",
+   "id": "dd1aa581",
    "metadata": {
     "editable": true
    },
@@ -2769,19 +2736,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1a25daa2",
+   "id": "4e631fe5",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta} =\\left(\\boldsymbol{A}^T\\boldsymbol{A}\\right)^{-1}\\boldsymbol{A}^T\\boldsymbol{b}.\n",
+    "\\boldsymbol{\\theta} =\\left(\\boldsymbol{A}^T\\boldsymbol{A}\\right)^{-1}\\boldsymbol{A}^T\\boldsymbol{b}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d2a18a90",
+   "id": "a1c651fa",
    "metadata": {
     "editable": true
    },
@@ -2791,7 +2758,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c2730e09",
+   "id": "0ba91c05",
    "metadata": {
     "editable": true
    },
@@ -2803,51 +2770,51 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dab1abb8",
+   "id": "9c7d45d8",
    "metadata": {
     "editable": true
    },
    "source": [
-    "we have then the following expression for the parameters $\\beta_j$ (the matrix elements of $\\boldsymbol{H}$ are $h_{ij}$)"
+    "we have then the following expression for the parameters $\\theta_j$ (the matrix elements of $\\boldsymbol{H}$ are $h_{ij}$)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "be9b74c2",
+   "id": "1148dda1",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_j = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}\\frac{y_i}{\\sigma_i}\\frac{x_{ik}}{\\sigma_i} = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}b_ia_{ik}\n",
+    "\\theta_j = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}\\frac{y_i}{\\sigma_i}\\frac{x_{ik}}{\\sigma_i} = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}b_ia_{ik}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "38e7188c",
+   "id": "899ad725",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We state without proof the expression for the uncertainty  in the parameters $\\beta_j$ as (we leave this as an exercise)"
+    "We state without proof the expression for the uncertainty  in the parameters $\\theta_j$ as (we leave this as an exercise)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c568814d",
+   "id": "442cefde",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\sigma^2(\\beta_j) = \\sum_{i=0}^{n-1}\\sigma_i^2\\left( \\frac{\\partial \\beta_j}{\\partial y_i}\\right)^2,\n",
+    "\\sigma^2(\\theta_j) = \\sum_{i=0}^{n-1}\\sigma_i^2\\left( \\frac{\\partial \\theta_j}{\\partial y_i}\\right)^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "37fb03f1",
+   "id": "303d558b",
    "metadata": {
     "editable": true
    },
@@ -2857,19 +2824,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "753cb5d9",
+   "id": "c8361a08",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\sigma^2(\\beta_j) = \\left(\\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}a_{ik}\\right)\\left(\\sum_{l=0}^{p-1}h_{jl}\\sum_{m=0}^{n-1}a_{ml}\\right) = h_{jj}!\n",
+    "\\sigma^2(\\theta_j) = \\left(\\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}a_{ik}\\right)\\left(\\sum_{l=0}^{p-1}h_{jl}\\sum_{m=0}^{n-1}a_{ml}\\right) = h_{jj}!\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a887be4f",
+   "id": "fad61a6c",
    "metadata": {
     "editable": true
    },
@@ -2879,41 +2846,41 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c99c1576",
+   "id": "29731faf",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "y=y(x) \\rightarrow y(x_i) \\approx \\beta_0+\\beta_1 x_i.\n",
+    "y=y(x) \\rightarrow y(x_i) \\approx \\theta_0+\\theta_1 x_i.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ddb752ed",
+   "id": "ecb0789a",
    "metadata": {
     "editable": true
    },
    "source": [
-    "By computing the derivatives of $\\chi^2$ with respect to $\\beta_0$ and $\\beta_1$ show that these are given by"
+    "By computing the derivatives of $\\chi^2$ with respect to $\\theta_0$ and $\\theta_1$ show that these are given by"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d5d9bc16",
+   "id": "bf04c0eb",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -2\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0,\n",
+    "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\theta_0} = -2\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\theta_0-\\theta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "971e3cbe",
+   "id": "7fba1a68",
    "metadata": {
     "editable": true
    },
@@ -2923,19 +2890,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "14a81f3f",
+   "id": "cb8528d5",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_i\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0.\n",
+    "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\theta_1} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_i\\left(\\frac{y_i-\\theta_0-\\theta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "daa744f2",
+   "id": "2771f01a",
    "metadata": {
     "editable": true
    },
@@ -2946,7 +2913,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bcdd91b0",
+   "id": "dcf74fcf",
    "metadata": {
     "editable": true
    },
@@ -2958,7 +2925,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "78646001",
+   "id": "a7e95721",
    "metadata": {
     "editable": true
    },
@@ -2970,7 +2937,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8eb6b0de",
+   "id": "a636424d",
    "metadata": {
     "editable": true
    },
@@ -2982,7 +2949,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f5641168",
+   "id": "294e5692",
    "metadata": {
     "editable": true
    },
@@ -2994,7 +2961,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c14fcf77",
+   "id": "923ebdc5",
    "metadata": {
     "editable": true
    },
@@ -3006,7 +2973,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9b374023",
+   "id": "d605b9e0",
    "metadata": {
     "editable": true
    },
@@ -3016,45 +2983,45 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5f34ff5d",
+   "id": "d9e8c477",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0 = \\frac{\\gamma_{xx}\\gamma_y-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2},\n",
+    "\\theta_0 = \\frac{\\gamma_{xx}\\gamma_y-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "349a1654",
+   "id": "2831c92f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_1 = \\frac{\\gamma_{xy}\\gamma-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2}.\n",
+    "\\theta_1 = \\frac{\\gamma_{xy}\\gamma-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "bdc8cc84",
+   "id": "36edba71",
    "metadata": {
     "editable": true
    },
    "source": [
     "This approach (different linear and non-linear regression) suffers\n",
     "often from both being underdetermined and overdetermined in the\n",
-    "unknown coefficients $\\beta_i$.  A better approach is to use the\n",
+    "unknown coefficients $\\theta_i$.  A better approach is to use the\n",
     "Singular Value Decomposition (SVD) method discussed below. Or using\n",
     "Lasso and Ridge regression. See below."
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8e31d846",
+   "id": "f209ca1f",
    "metadata": {
     "editable": true
    },
@@ -3079,8 +3046,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 23,
-   "id": "ba56fb3a",
+   "execution_count": 22,
+   "id": "5d230f93",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3162,7 +3129,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "982e0c69",
+   "id": "b3d6f5a2",
    "metadata": {
     "editable": true
    },
@@ -3173,7 +3140,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "acb30447",
+   "id": "4b61d1b0",
    "metadata": {
     "editable": true
    },
@@ -3200,8 +3167,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
-   "id": "301a7664",
+   "execution_count": 23,
+   "id": "8c25e039",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3232,16 +3199,16 @@
     "X[:,2] = x**2\n",
     "# We split the data in test and training data\n",
     "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
-    "# matrix inversion to find beta\n",
-    "beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
-    "print(beta)\n",
+    "# matrix inversion to find theta\n",
+    "theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
+    "print(theta)\n",
     "# and then make the prediction\n",
-    "ytilde = X_train @ beta\n",
+    "ytilde = X_train @ theta\n",
     "print(\"Training R2\")\n",
     "print(R2(y_train,ytilde))\n",
     "print(\"Training MSE\")\n",
     "print(MSE(y_train,ytilde))\n",
-    "ypredict = X_test @ beta\n",
+    "ypredict = X_test @ theta\n",
     "print(\"Test R2\")\n",
     "print(R2(y_test,ypredict))\n",
     "print(\"Test MSE\")\n",
@@ -3250,7 +3217,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "655f046e",
+   "id": "3c09d70c",
    "metadata": {
     "editable": true
    },
@@ -3260,8 +3227,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 25,
-   "id": "413acb4d",
+   "execution_count": 24,
+   "id": "3e9b7ad8",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3286,7 +3253,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f46a080f",
+   "id": "18804e9f",
    "metadata": {
     "editable": true
    },
@@ -3302,8 +3269,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 26,
-   "id": "96391e08",
+   "execution_count": 25,
+   "id": "90f453cd",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3361,15 +3328,15 @@
     "X[:,4] = Density**(5.0/3.0)\n",
     "# We split the data in test and training data\n",
     "X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)\n",
-    "# matrix inversion to find beta\n",
-    "beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)\n",
+    "# matrix inversion to find theta\n",
+    "theta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)\n",
     "# and then make the prediction\n",
-    "ytilde = X_train @ beta\n",
+    "ytilde = X_train @ theta\n",
     "print(\"Training R2\")\n",
     "print(R2(y_train,ytilde))\n",
     "print(\"Training MSE\")\n",
     "print(MSE(y_train,ytilde))\n",
-    "ypredict = X_test @ beta\n",
+    "ypredict = X_test @ theta\n",
     "print(\"Test R2\")\n",
     "print(R2(y_test,ypredict))\n",
     "print(\"Test MSE\")\n",
@@ -3378,7 +3345,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "46a1885f",
+   "id": "3d672d7a",
    "metadata": {
     "editable": true
    },
@@ -3422,7 +3389,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a2041293",
+   "id": "e2426f64",
    "metadata": {
     "editable": true
    },
@@ -3433,8 +3400,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 27,
-   "id": "f813a977",
+   "execution_count": 26,
+   "id": "519e0c09",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3450,7 +3417,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5fc086ec",
+   "id": "7b6fd188",
    "metadata": {
     "editable": true
    },
@@ -3460,8 +3427,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 28,
-   "id": "bf72511a",
+   "execution_count": 27,
+   "id": "3665e1b0",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3479,7 +3446,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "70823078",
+   "id": "96f2fe7b",
    "metadata": {
     "editable": true
    },
@@ -3489,8 +3456,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 29,
-   "id": "85e192bf",
+   "execution_count": 28,
+   "id": "df2c84ca",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3504,7 +3471,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "13fd362f",
+   "id": "0c4588d5",
    "metadata": {
     "editable": true
    },
@@ -3514,8 +3481,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 30,
-   "id": "8f379d9b",
+   "execution_count": 29,
+   "id": "209438db",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3528,7 +3495,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c5f9dc18",
+   "id": "1b67c46a",
    "metadata": {
     "editable": true
    },
@@ -3538,8 +3505,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 31,
-   "id": "cb9f3445",
+   "execution_count": 30,
+   "id": "236343bf",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3556,7 +3523,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a16f238",
+   "id": "c8726877",
    "metadata": {
     "editable": true
    },
@@ -3566,8 +3533,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 32,
-   "id": "a5386de5",
+   "execution_count": 31,
+   "id": "aaa47b00",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3583,7 +3550,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1d2e90ee",
+   "id": "0283f3f1",
    "metadata": {
     "editable": true
    },
@@ -3593,8 +3560,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 33,
-   "id": "b494903c",
+   "execution_count": 32,
+   "id": "c0823ed1",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3618,7 +3585,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c580e7c5",
+   "id": "dc2cf448",
    "metadata": {
     "editable": true
    },
@@ -3628,8 +3595,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 34,
-   "id": "45830e09",
+   "execution_count": 33,
+   "id": "ee945b00",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3642,7 +3609,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fe0e34ba",
+   "id": "288a1417",
    "metadata": {
     "editable": true
    },
@@ -3652,8 +3619,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 35,
-   "id": "f73e5b0f",
+   "execution_count": 34,
+   "id": "0b640ff9",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3673,7 +3640,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a99e4f96",
+   "id": "32cfad5b",
    "metadata": {
     "editable": true
    },
@@ -3683,8 +3650,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 36,
-   "id": "8c972941",
+   "execution_count": 35,
+   "id": "792df674",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3726,8 +3693,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 37,
-   "id": "91ef994e",
+   "execution_count": 36,
+   "id": "6f9196e5",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3742,7 +3709,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b2302654",
+   "id": "b72d5080",
    "metadata": {
     "editable": true
    },
@@ -3819,7 +3786,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9f2998ff",
+   "id": "009b948b",
    "metadata": {
     "editable": true
    },
@@ -3831,7 +3798,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bc8c9873",
+   "id": "4b02a26a",
    "metadata": {
     "editable": true
    },
@@ -3850,8 +3817,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 38,
-   "id": "810f6b81",
+   "execution_count": 37,
+   "id": "efd083fc",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3885,7 +3852,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "565a1cd1",
+   "id": "b3590aea",
    "metadata": {
     "editable": true
    },
@@ -3900,7 +3867,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0a65c309",
+   "id": "9592b7bf",
    "metadata": {
     "editable": true
    },
@@ -3912,7 +3879,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8b0d4688",
+   "id": "0941f045",
    "metadata": {
     "editable": true
    },
@@ -3922,7 +3889,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d1f9ca89",
+   "id": "d47fc8d3",
    "metadata": {
     "editable": true
    },
@@ -3945,8 +3912,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 39,
-   "id": "28d4c3a0",
+   "execution_count": 38,
+   "id": "1ddb9cb5",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3990,7 +3957,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dfe532c5",
+   "id": "725b78e9",
    "metadata": {
     "editable": true
    },
@@ -4000,7 +3967,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "86f0a026",
+   "id": "b08f94e5",
    "metadata": {
     "editable": true
    },
@@ -4069,7 +4036,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fd6857da",
+   "id": "95e72a9e",
    "metadata": {
     "editable": true
    },
@@ -4082,8 +4049,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 40,
-   "id": "c84d4e1f",
+   "execution_count": 39,
+   "id": "fb3ad8e5",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4096,7 +4063,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "10579805",
+   "id": "02fe1db6",
    "metadata": {
     "editable": true
    },
@@ -4110,7 +4077,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "30c31580",
+   "id": "33a4aed5",
    "metadata": {
     "editable": true
    },
@@ -4123,7 +4090,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "af797776",
+   "id": "78a3bc86",
    "metadata": {
     "editable": true
    },
@@ -4134,7 +4101,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9c6eebe7",
+   "id": "38c3a27d",
    "metadata": {
     "editable": true
    },
@@ -4146,7 +4113,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "06a2f8be",
+   "id": "7eb5c51b",
    "metadata": {
     "editable": true
    },
@@ -4156,7 +4123,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0b5633b3",
+   "id": "3597b20a",
    "metadata": {
     "editable": true
    },
@@ -4168,7 +4135,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "108886bb",
+   "id": "82922e13",
    "metadata": {
     "editable": true
    },
@@ -4178,13 +4145,13 @@
     "\n",
     "\n",
     "**Solution.**\n",
-    "The code here is an example of where we define our own design matrix and fit parameters $\\beta$."
+    "The code here is an example of where we define our own design matrix and fit parameters $\\theta$."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 41,
-   "id": "94716007",
+   "execution_count": 40,
+   "id": "61cd693e",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4217,16 +4184,16 @@
     "X[:,2] = x**2\n",
     "# We split the data in test and training data\n",
     "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
-    "# matrix inversion to find beta\n",
-    "beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
-    "print(beta)\n",
+    "# matrix inversion to find theta\n",
+    "theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
+    "print(theta)\n",
     "# and then make the prediction\n",
-    "ytilde = X_train @ beta\n",
+    "ytilde = X_train @ theta\n",
     "print(\"Training R2\")\n",
     "print(R2(y_train,ytilde))\n",
     "print(\"Training MSE\")\n",
     "print(MSE(y_train,ytilde))\n",
-    "ypredict = X_test @ beta\n",
+    "ypredict = X_test @ theta\n",
     "print(\"Test R2\")\n",
     "print(R2(y_test,ypredict))\n",
     "print(\"Test MSE\")\n",
@@ -4235,7 +4202,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3aa5be35",
+   "id": "2f8d2e6e",
    "metadata": {
     "editable": true
    },
@@ -4245,7 +4212,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1af6d7cd",
+   "id": "c81f5caf",
    "metadata": {
     "editable": true
    },
@@ -4290,8 +4257,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 42,
-   "id": "1f4f0115",
+   "execution_count": 41,
+   "id": "37ca3335",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4304,7 +4271,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8904ba25",
+   "id": "e6925e8c",
    "metadata": {
     "editable": true
    },
@@ -4314,8 +4281,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 43,
-   "id": "04fdf05f",
+   "execution_count": 42,
+   "id": "caecb70a",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4330,7 +4297,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a22f1ef9",
+   "id": "deabdf0c",
    "metadata": {
     "editable": true
    },
@@ -4347,8 +4314,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 44,
-   "id": "ad96211c",
+   "execution_count": 43,
+   "id": "b083bb84",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4365,7 +4332,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a103eff1",
+   "id": "21102c44",
    "metadata": {
     "editable": true
    },
@@ -4380,8 +4347,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 45,
-   "id": "d9b5eed5",
+   "execution_count": 44,
+   "id": "b9108dab",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4425,7 +4392,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9decc489",
+   "id": "cd19b575",
    "metadata": {
     "editable": true
    },
@@ -4435,7 +4402,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f2ffecd1",
+   "id": "eb2f6352",
    "metadata": {
     "editable": true
    },
@@ -4446,7 +4413,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "356683e0",
+   "id": "6e09ea94",
    "metadata": {
     "editable": true
    },
@@ -4457,7 +4424,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "01abcbae",
+   "id": "38857156",
    "metadata": {
     "editable": true
    },
@@ -4468,7 +4435,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8b2a42d8",
+   "id": "6b9115a6",
    "metadata": {
     "editable": true
    },
@@ -4490,8 +4457,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 46,
-   "id": "4ae34751",
+   "execution_count": 45,
+   "id": "ae7a71c1",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4504,7 +4471,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "559e2cff",
+   "id": "ca615d39",
    "metadata": {
     "editable": true
    },
@@ -4521,7 +4488,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "03b05ffa",
+   "id": "d83c8354",
    "metadata": {
     "editable": true
    },
@@ -4534,7 +4501,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1ff2cf84",
+   "id": "74f6e912",
    "metadata": {
     "editable": true
    },
@@ -4545,7 +4512,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fe18f837",
+   "id": "93761664",
    "metadata": {
     "editable": true
    },
@@ -4557,7 +4524,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "49eff4b4",
+   "id": "be729d32",
    "metadata": {
     "editable": true
    },
@@ -4567,7 +4534,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "acc25706",
+   "id": "08549523",
    "metadata": {
     "editable": true
    },
@@ -4579,7 +4546,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5f945b64",
+   "id": "03cc0ca0",
    "metadata": {
     "editable": true
    },
@@ -4595,8 +4562,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 47,
-   "id": "f9578f4e",
+   "execution_count": 46,
+   "id": "9b1b9378",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4635,16 +4602,16 @@
     "# We split the data in test and training data\n",
     "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
     "\n",
-    "# matrix inversion to find beta\n",
-    "OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
-    "print(OLSbeta)\n",
+    "# matrix inversion to find theta\n",
+    "OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
+    "print(OLStheta)\n",
     "# and then make the prediction\n",
-    "ytildeOLS = X_train @ OLSbeta\n",
+    "ytildeOLS = X_train @ OLStheta\n",
     "print(\"Training R2 for OLS\")\n",
     "print(R2(y_train,ytildeOLS))\n",
     "print(\"Training MSE for OLS\")\n",
     "print(MSE(y_train,ytildeOLS))\n",
-    "ypredictOLS = X_test @ OLSbeta\n",
+    "ypredictOLS = X_test @ OLStheta\n",
     "print(\"Test R2 for OLS\")\n",
     "print(R2(y_test,ypredictOLS))\n",
     "print(\"Test MSE OLS\")\n",
@@ -4661,10 +4628,10 @@
     "lambdas = np.logspace(-4, 1, nlambdas)\n",
     "for i in range(nlambdas):\n",
     "    lmb = lambdas[i]\n",
-    "    OwnRidgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
+    "    OwnRidgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
     "    # and then make the prediction\n",
-    "    OwnytildeRidge = X_train @ OwnRidgebeta\n",
-    "    OwnypredictRidge = X_test @ OwnRidgebeta\n",
+    "    OwnytildeRidge = X_train @ OwnRidgetheta\n",
+    "    OwnypredictRidge = X_test @ OwnRidgetheta\n",
     "    OwnMSEPredict[i] = MSE(y_test,OwnypredictRidge)\n",
     "    OwnMSETrain[i] = MSE(y_train,OwnytildeRidge)\n",
     "    # Make the fit using Ridge from Sklearn\n",
@@ -4688,7 +4655,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "88be64bc",
+   "id": "341e9820",
    "metadata": {
     "editable": true
    },
@@ -4698,7 +4665,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3a0e09e7",
+   "id": "6feb372f",
    "metadata": {
     "editable": true
    },
@@ -4720,7 +4687,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fd41f7f2",
+   "id": "3c12062d",
    "metadata": {
     "editable": true
    },
@@ -4732,7 +4699,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "49d2bdba",
+   "id": "edfcd8ff",
    "metadata": {
     "editable": true
    },
@@ -4742,7 +4709,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fd5a139b",
+   "id": "e8689eb2",
    "metadata": {
     "editable": true
    },
@@ -4754,7 +4721,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "af22b4d2",
+   "id": "62145d09",
    "metadata": {
     "editable": true
    },
@@ -4764,7 +4731,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b120000f",
+   "id": "ec992c25",
    "metadata": {
     "editable": true
    },
@@ -4776,7 +4743,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0e7df31e",
+   "id": "4a922090",
    "metadata": {
     "editable": true
    },
@@ -4791,7 +4758,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "faa1b4d0",
+   "id": "10f294a3",
    "metadata": {
     "editable": true
    },
@@ -4803,7 +4770,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "02f78802",
+   "id": "1dfa508a",
    "metadata": {
     "editable": true
    },
@@ -4813,7 +4780,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e85b6015",
+   "id": "b5474b9f",
    "metadata": {
     "editable": true
    },
@@ -4825,7 +4792,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12ae8696",
+   "id": "4dee1baf",
    "metadata": {
     "editable": true
    },
@@ -4835,7 +4802,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d90de117",
+   "id": "9949ee71",
    "metadata": {
     "editable": true
    },
@@ -4847,7 +4814,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "15a6c139",
+   "id": "73ce2a98",
    "metadata": {
     "editable": true
    },
@@ -4857,7 +4824,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f5e7aa62",
+   "id": "41e1008d",
    "metadata": {
     "editable": true
    },
@@ -4869,7 +4836,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5a1f86bc",
+   "id": "597c4ebb",
    "metadata": {
     "editable": true
    },
@@ -4879,7 +4846,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "57018aa2",
+   "id": "e9163f17",
    "metadata": {
     "editable": true
    },
@@ -4891,7 +4858,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6a9c70a8",
+   "id": "fc3d2fdb",
    "metadata": {
     "editable": true
    },
@@ -4901,7 +4868,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "75aaf0ec",
+   "id": "e7cec690",
    "metadata": {
     "editable": true
    },
@@ -4913,7 +4880,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "31a142ea",
+   "id": "1a74bac9",
    "metadata": {
     "editable": true
    },
@@ -4923,7 +4890,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2552b361",
+   "id": "7087bc76",
    "metadata": {
     "editable": true
    },
@@ -4935,7 +4902,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "91b53dc5",
+   "id": "1e5a21fd",
    "metadata": {
     "editable": true
    },
@@ -4945,7 +4912,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bf83b147",
+   "id": "624d40e0",
    "metadata": {
     "editable": true
    },
@@ -4957,7 +4924,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0c8668a6",
+   "id": "d871f62f",
    "metadata": {
     "editable": true
    },
@@ -4967,7 +4934,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "89704701",
+   "id": "675aea37",
    "metadata": {
     "editable": true
    },
@@ -4979,7 +4946,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0054bb58",
+   "id": "746b74e2",
    "metadata": {
     "editable": true
    },
@@ -4989,7 +4956,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "24f3ee5f",
+   "id": "6381d6ee",
    "metadata": {
     "editable": true
    },
@@ -5001,7 +4968,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0827a002",
+   "id": "4331c20e",
    "metadata": {
     "editable": true
    },
diff --git a/doc/LectureNotes/_build/jupyter_execute/chapter2.ipynb b/doc/LectureNotes/_build/jupyter_execute/chapter2.ipynb
index 41c0490a1..3b65028c6 100644
--- a/doc/LectureNotes/_build/jupyter_execute/chapter2.ipynb
+++ b/doc/LectureNotes/_build/jupyter_execute/chapter2.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "74d0d498",
+   "id": "d5cb084b",
    "metadata": {
     "editable": true
    },
@@ -13,7 +13,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "acd544cd",
+   "id": "10ec5810",
    "metadata": {
     "editable": true
    },
@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0acaa875",
+   "id": "607f5658",
    "metadata": {
     "editable": true
    },
@@ -32,48 +32,48 @@
     "\n",
     "What is presented here is a mathematical analysis of various regression algorithms (ordinary least  squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). \n",
     "\n",
-    "We have shown that in ordinary least squares (OLS) the optimal parameters $\\beta$ are given by"
+    "We have shown that in ordinary least squares (OLS) the optimal parameters $\\theta$ are given by"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "0924b488",
+   "id": "d5cf7dd7",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+    "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b7307eb7",
+   "id": "4bccceb2",
    "metadata": {
     "editable": true
    },
    "source": [
-    "The **hat** over $\\boldsymbol{\\beta}$ means we have the optimal parameters after minimization of the cost function.\n",
+    "The **hat** over $\\boldsymbol{\\theta}$ means we have the optimal parameters after minimization of the cost function.\n",
     "\n",
     "This means that our best model is defined as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8096b1f0",
+   "id": "a51f281a",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+    "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d64adaf2",
+   "id": "66dd8fd1",
    "metadata": {
     "editable": true
    },
@@ -83,7 +83,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "78b35483",
+   "id": "a4914b87",
    "metadata": {
     "editable": true
    },
@@ -95,7 +95,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f4a3ff68",
+   "id": "12e868a7",
    "metadata": {
     "editable": true
    },
@@ -105,19 +105,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5704d260",
+   "id": "a76003c4",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{A}\\boldsymbol{y}.\n",
+    "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}} = \\boldsymbol{A}\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "12fa775a",
+   "id": "f894a9be",
    "metadata": {
     "editable": true
    },
@@ -130,7 +130,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a694774d",
+   "id": "1c7cf524",
    "metadata": {
     "editable": true
    },
@@ -142,7 +142,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e0775433",
+   "id": "de9e3352",
    "metadata": {
     "editable": true
    },
@@ -154,7 +154,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ec16ae7d",
+   "id": "4ece2993",
    "metadata": {
     "editable": true
    },
@@ -166,7 +166,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "258fbf6a",
+   "id": "f525eb65",
    "metadata": {
     "editable": true
    },
@@ -176,7 +176,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "05ea3605",
+   "id": "7d2849f0",
    "metadata": {
     "editable": true
    },
@@ -188,7 +188,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "db5ed2d4",
+   "id": "b214cf3a",
    "metadata": {
     "editable": true
    },
@@ -198,7 +198,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e99ad8e8",
+   "id": "d566c8b4",
    "metadata": {
     "editable": true
    },
@@ -210,7 +210,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1cc7938a",
+   "id": "cac5aeb2",
    "metadata": {
     "editable": true
    },
@@ -220,7 +220,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0e3fc9d5",
+   "id": "b349c4ef",
    "metadata": {
     "editable": true
    },
@@ -267,7 +267,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6d2bc570",
+   "id": "b5c5014d",
    "metadata": {
     "editable": true
    },
@@ -290,7 +290,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fdda638f",
+   "id": "2b07f548",
    "metadata": {
     "editable": true
    },
@@ -307,7 +307,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "29a3a620",
+   "id": "f8223e8b",
    "metadata": {
     "editable": true
    },
@@ -326,7 +326,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3278b038",
+   "id": "c1c28b92",
    "metadata": {
     "editable": true
    },
@@ -339,7 +339,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7b66f623",
+   "id": "8cebf32c",
    "metadata": {
     "editable": true
    },
@@ -349,7 +349,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "\\boldsymbol{\\beta}  =  (\\boldsymbol{X}^{T} \\boldsymbol{X})^{-1} \\boldsymbol{X}^{T} \\boldsymbol{y},\n",
+    "\\boldsymbol{\\theta}  =  (\\boldsymbol{X}^{T} \\boldsymbol{X})^{-1} \\boldsymbol{X}^{T} \\boldsymbol{y},\n",
     "\\label{_auto1} \\tag{1}\n",
     "\\end{equation}\n",
     "$$"
@@ -357,23 +357,23 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c97ae6c5",
+   "id": "4c2f125e",
    "metadata": {
     "editable": true
    },
    "source": [
     "has linearly dependent column vectors, we will not be able to compute the inverse\n",
-    "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\beta_i$. \n",
+    "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\theta_i$. \n",
     "The estimators are only well-defined if $(\\boldsymbol{X}^{T}\\boldsymbol{X})$ can be inverted. \n",
     "This is more likely to happen when the matrix $\\boldsymbol{X}$ is high-dimensional. In this case it is likely to encounter a situation where \n",
-    "the regression parameters $\\beta_i$ cannot be estimated.\n",
+    "the regression parameters $\\theta_i$ cannot be estimated.\n",
     "\n",
     "A cheap  *ad hoc* approach is  simply to add a small diagonal component to the matrix to invert, that is we change"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "af0bfc59",
+   "id": "28b01b73",
    "metadata": {
     "editable": true
    },
@@ -385,7 +385,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6469bfe8",
+   "id": "df72f01b",
    "metadata": {
     "editable": true
    },
@@ -395,7 +395,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "75d45f30",
+   "id": "62fc7dce",
    "metadata": {
     "editable": true
    },
@@ -410,7 +410,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "28b4fac2",
+   "id": "5b0639a7",
    "metadata": {
     "editable": true
    },
@@ -422,7 +422,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "085b2de7",
+   "id": "397cb734",
    "metadata": {
     "editable": true
    },
@@ -432,7 +432,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1fb42fdc",
+   "id": "05ac3f20",
    "metadata": {
     "editable": true
    },
@@ -444,7 +444,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0f7b8c0d",
+   "id": "ee59ca38",
    "metadata": {
     "editable": true
    },
@@ -454,7 +454,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "057f3ed2",
+   "id": "6c60f584",
    "metadata": {
     "editable": true
    },
@@ -466,7 +466,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "628031b9",
+   "id": "b7031193",
    "metadata": {
     "editable": true
    },
@@ -478,7 +478,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3177dcfe",
+   "id": "4c0dce45",
    "metadata": {
     "editable": true
    },
@@ -493,7 +493,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a392a54d",
+   "id": "ce330467",
    "metadata": {
     "editable": true
    },
@@ -514,7 +514,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3496079e",
+   "id": "e42d67ce",
    "metadata": {
     "editable": true
    },
@@ -526,7 +526,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "77b61c68",
+   "id": "d43e350b",
    "metadata": {
     "editable": true
    },
@@ -536,7 +536,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dbb84089",
+   "id": "4ad05081",
    "metadata": {
     "editable": true
    },
@@ -548,7 +548,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "04fc9eec",
+   "id": "7831a051",
    "metadata": {
     "editable": true
    },
@@ -594,7 +594,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4f1e3bb4",
+   "id": "706c9e36",
    "metadata": {
     "editable": true
    },
@@ -605,7 +605,7 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "a39cbeb0",
+   "id": "9605f17b",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -645,7 +645,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dca36481",
+   "id": "d6dc015b",
    "metadata": {
     "editable": true
    },
@@ -675,7 +675,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b9b3c7db",
+   "id": "2b38c905",
    "metadata": {
     "editable": true
    },
@@ -689,7 +689,7 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "6aa5981e",
+   "id": "3b82ddeb",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -701,7 +701,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae58132b",
+   "id": "8bdcf281",
    "metadata": {
     "editable": true
    },
@@ -712,7 +712,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "761d48f5",
+   "id": "8090ac1c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -751,7 +751,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6297c4ec",
+   "id": "35ee1cc9",
    "metadata": {
     "editable": true
    },
@@ -769,7 +769,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2bb1f4b8",
+   "id": "13c4c73e",
    "metadata": {
     "editable": true
    },
@@ -781,7 +781,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "832a6404",
+   "id": "392f8342",
    "metadata": {
     "editable": true
    },
@@ -792,7 +792,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "d393c8c7",
+   "id": "3d64803c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -826,7 +826,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3d0570c2",
+   "id": "a48cdbe7",
    "metadata": {
     "editable": true
    },
@@ -836,7 +836,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "071ff3b6",
+   "id": "22f55640",
    "metadata": {
     "editable": true
    },
@@ -850,7 +850,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "113247f9",
+   "id": "88772ea4",
    "metadata": {
     "editable": true
    },
@@ -869,7 +869,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "05603c38",
+   "id": "4f144230",
    "metadata": {
     "editable": true
    },
@@ -879,7 +879,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8848e319",
+   "id": "c1a77d69",
    "metadata": {
     "editable": true
    },
@@ -891,7 +891,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5b4146e5",
+   "id": "c77977a5",
    "metadata": {
     "editable": true
    },
@@ -905,7 +905,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "43701d21",
+   "id": "b3bac40c",
    "metadata": {
     "editable": true
    },
@@ -917,7 +917,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7cdca00d",
+   "id": "a5803728",
    "metadata": {
     "editable": true
    },
@@ -929,7 +929,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "add83821",
+   "id": "21edee39",
    "metadata": {
     "editable": true
    },
@@ -946,7 +946,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3746dd8d",
+   "id": "3e23d609",
    "metadata": {
     "editable": true
    },
@@ -956,7 +956,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae591c49",
+   "id": "64ec1fc5",
    "metadata": {
     "editable": true
    },
@@ -972,7 +972,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "63009724",
+   "id": "56d47d93",
    "metadata": {
     "editable": true
    },
@@ -982,7 +982,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c674e8da",
+   "id": "2fda48ae",
    "metadata": {
     "editable": true
    },
@@ -998,7 +998,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9f60659a",
+   "id": "824ffed2",
    "metadata": {
     "editable": true
    },
@@ -1008,7 +1008,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "27e17584",
+   "id": "0ac7a41e",
    "metadata": {
     "editable": true
    },
@@ -1024,7 +1024,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3efc62c4",
+   "id": "8d0dd964",
    "metadata": {
     "editable": true
    },
@@ -1034,7 +1034,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "195d30c4",
+   "id": "30282069",
    "metadata": {
     "editable": true
    },
@@ -1051,7 +1051,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a00d95e8",
+   "id": "064c0e6b",
    "metadata": {
     "editable": true
    },
@@ -1065,7 +1065,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "985428ec",
+   "id": "273056c4",
    "metadata": {
     "editable": true
    },
@@ -1077,7 +1077,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0affd014",
+   "id": "e38034e4",
    "metadata": {
     "editable": true
    },
@@ -1087,7 +1087,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "146dcd49",
+   "id": "8cc943db",
    "metadata": {
     "editable": true
    },
@@ -1099,7 +1099,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "63bc8186",
+   "id": "9dfc2966",
    "metadata": {
     "editable": true
    },
@@ -1111,7 +1111,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4bea8a7d",
+   "id": "8ca54207",
    "metadata": {
     "editable": true
    },
@@ -1123,7 +1123,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5321e8fc",
+   "id": "a986b780",
    "metadata": {
     "editable": true
    },
@@ -1133,7 +1133,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c16d0109",
+   "id": "2c31d500",
    "metadata": {
     "editable": true
    },
@@ -1145,7 +1145,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4ca7bc79",
+   "id": "9e4536a0",
    "metadata": {
     "editable": true
    },
@@ -1155,7 +1155,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "795969aa",
+   "id": "92636687",
    "metadata": {
     "editable": true
    },
@@ -1167,7 +1167,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "36fd11ec",
+   "id": "a665d593",
    "metadata": {
     "editable": true
    },
@@ -1177,7 +1177,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8b59e361",
+   "id": "769bb394",
    "metadata": {
     "editable": true
    },
@@ -1189,7 +1189,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4212728f",
+   "id": "baf742bb",
    "metadata": {
     "editable": true
    },
@@ -1201,7 +1201,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8d6940eb",
+   "id": "7fd2be6b",
    "metadata": {
     "editable": true
    },
@@ -1211,7 +1211,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eabfdb9c",
+   "id": "e488ea83",
    "metadata": {
     "editable": true
    },
@@ -1223,7 +1223,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bbc85eab",
+   "id": "76ac57c1",
    "metadata": {
     "editable": true
    },
@@ -1234,7 +1234,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f5b1bf52",
+   "id": "b789d545",
    "metadata": {
     "editable": true
    },
@@ -1246,7 +1246,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "efcc256a",
+   "id": "fd7008f9",
    "metadata": {
     "editable": true
    },
@@ -1256,7 +1256,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0650a070",
+   "id": "9d37f0bf",
    "metadata": {
     "editable": true
    },
@@ -1268,7 +1268,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c168d425",
+   "id": "0fd2b28c",
    "metadata": {
     "editable": true
    },
@@ -1278,7 +1278,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "30fb2230",
+   "id": "98432443",
    "metadata": {
     "editable": true
    },
@@ -1290,7 +1290,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d46b1fd5",
+   "id": "29174a30",
    "metadata": {
     "editable": true
    },
@@ -1301,7 +1301,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "16a93afd",
+   "id": "d07b5e5c",
    "metadata": {
     "editable": true
    },
@@ -1313,7 +1313,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "80e47710",
+   "id": "21892e7e",
    "metadata": {
     "editable": true
    },
@@ -1331,7 +1331,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1dc0f98b",
+   "id": "a4ed7dd4",
    "metadata": {
     "editable": true
    },
@@ -1347,19 +1347,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c5f9ce8b",
+   "id": "c7aafbc8",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial^2 C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
+    "\\frac{\\partial^2 C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}^T\\partial \\boldsymbol{\\theta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c803fdb7",
+   "id": "f80de4f9",
    "metadata": {
     "editable": true
    },
@@ -1371,7 +1371,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eef3c89e",
+   "id": "8ec45bf8",
    "metadata": {
     "editable": true
    },
@@ -1383,7 +1383,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e886f303",
+   "id": "4554f86d",
    "metadata": {
     "editable": true
    },
@@ -1402,7 +1402,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "efcb2b9e",
+   "id": "c5c46053",
    "metadata": {
     "editable": true
    },
@@ -1416,7 +1416,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "caf380d7",
+   "id": "0d367725",
    "metadata": {
     "editable": true
    },
@@ -1426,7 +1426,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9d676665",
+   "id": "ca783893",
    "metadata": {
     "editable": true
    },
@@ -1438,7 +1438,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "301b0d53",
+   "id": "81dacef8",
    "metadata": {
     "editable": true
    },
@@ -1448,7 +1448,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7c134c39",
+   "id": "10790e67",
    "metadata": {
     "editable": true
    },
@@ -1460,7 +1460,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12aec852",
+   "id": "0df66dcf",
    "metadata": {
     "editable": true
    },
@@ -1470,7 +1470,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "54a5de6b",
+   "id": "021fd105",
    "metadata": {
     "editable": true
    },
@@ -1484,7 +1484,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3edc1dd1",
+   "id": "9cee5e0c",
    "metadata": {
     "editable": true
    },
@@ -1507,7 +1507,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9d2d7b4e",
+   "id": "30ea5811",
    "metadata": {
     "editable": true
    },
@@ -1519,7 +1519,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dcee4258",
+   "id": "a129d192",
    "metadata": {
     "editable": true
    },
@@ -1532,7 +1532,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "25100476",
+   "id": "6d5c0a69",
    "metadata": {
     "editable": true
    },
@@ -1546,7 +1546,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5c46fdef",
+   "id": "79a613b9",
    "metadata": {
     "editable": true
    },
@@ -1559,7 +1559,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dcfa6d43",
+   "id": "2d0dabde",
    "metadata": {
     "editable": true
    },
@@ -1578,7 +1578,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "89dd5e4e",
+   "id": "2731cd83",
    "metadata": {
     "editable": true
    },
@@ -1590,7 +1590,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7edacd6c",
+   "id": "82de2dcc",
    "metadata": {
     "editable": true
    },
@@ -1602,7 +1602,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "94e8929f",
+   "id": "e8bf54be",
    "metadata": {
     "editable": true
    },
@@ -1612,7 +1612,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3f10861a",
+   "id": "a0597d71",
    "metadata": {
     "editable": true
    },
@@ -1624,7 +1624,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "173fbc84",
+   "id": "d74275c9",
    "metadata": {
     "editable": true
    },
@@ -1637,7 +1637,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0913dc51",
+   "id": "df0589bd",
    "metadata": {
     "editable": true
    },
@@ -1656,7 +1656,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "70346587",
+   "id": "c23bd593",
    "metadata": {
     "editable": true
    },
@@ -1666,7 +1666,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "86feec9f",
+   "id": "8806c741",
    "metadata": {
     "editable": true
    },
@@ -1685,7 +1685,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "25ed76e9",
+   "id": "ae1e6181",
    "metadata": {
     "editable": true
    },
@@ -1701,7 +1701,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cea26675",
+   "id": "9a4af97a",
    "metadata": {
     "editable": true
    },
@@ -1715,7 +1715,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1fa1423f",
+   "id": "607da328",
    "metadata": {
     "editable": true
    },
@@ -1730,7 +1730,7 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "f277b044",
+   "id": "afa9a0d9",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1751,7 +1751,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "02205795",
+   "id": "8acdd66a",
    "metadata": {
     "editable": true
    },
@@ -1766,7 +1766,7 @@
   {
    "cell_type": "code",
    "execution_count": 6,
-   "id": "6c182a7d",
+   "id": "4886841b",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1798,7 +1798,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "16ff9454",
+   "id": "90b67d9e",
    "metadata": {
     "editable": true
    },
@@ -1815,7 +1815,7 @@
   {
    "cell_type": "code",
    "execution_count": 7,
-   "id": "903635fb",
+   "id": "2dc76e27",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1840,7 +1840,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "034c38ef",
+   "id": "3b9c94f0",
    "metadata": {
     "editable": true
    },
@@ -1851,7 +1851,7 @@
   {
    "cell_type": "code",
    "execution_count": 8,
-   "id": "91afb8cb",
+   "id": "f12e4764",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1877,7 +1877,7 @@
     "\t\ty = np.ravel(y)\n",
     "\n",
     "\tN = len(x)\n",
-    "\tl = int((n+1)*(n+2)/2)\t\t# Number of elements in beta\n",
+    "\tl = int((n+1)*(n+2)/2)\t\t# Number of elements in theta\n",
     "\tX = np.ones((N,l))\n",
     "\n",
     "\tfor i in range(1,n+1):\n",
@@ -1905,7 +1905,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9d8768f8",
+   "id": "6d8aa859",
    "metadata": {
     "editable": true
    },
@@ -1924,7 +1924,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a7de38a0",
+   "id": "d4715f97",
    "metadata": {
     "editable": true
    },
@@ -1936,7 +1936,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4855248a",
+   "id": "1800f79e",
    "metadata": {
     "editable": true
    },
@@ -1946,7 +1946,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3a150f73",
+   "id": "dac62d99",
    "metadata": {
     "editable": true
    },
@@ -1963,7 +1963,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6ab3f1f7",
+   "id": "97c7d1d3",
    "metadata": {
     "editable": true
    },
@@ -1973,7 +1973,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae35f475",
+   "id": "5645cf35",
    "metadata": {
     "editable": true
    },
@@ -1988,7 +1988,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9f6b4b6b",
+   "id": "ae8afe59",
    "metadata": {
     "editable": true
    },
@@ -1998,7 +1998,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "af0c59a3",
+   "id": "3bae9bfc",
    "metadata": {
     "editable": true
    },
@@ -2012,7 +2012,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "824ae63b",
+   "id": "b60c8bb6",
    "metadata": {
     "editable": true
    },
@@ -2026,7 +2026,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5b480160",
+   "id": "d8c6fa53",
    "metadata": {
     "editable": true
    },
@@ -2038,7 +2038,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "506a98da",
+   "id": "074c5673",
    "metadata": {
     "editable": true
    },
@@ -2050,7 +2050,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "72b1e665",
+   "id": "8320630a",
    "metadata": {
     "editable": true
    },
@@ -2060,7 +2060,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6ea9123f",
+   "id": "2eebcb57",
    "metadata": {
     "editable": true
    },
@@ -2072,7 +2072,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ab471475",
+   "id": "7a7c1b9c",
    "metadata": {
     "editable": true
    },
@@ -2082,7 +2082,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ff364443",
+   "id": "c89ccbc8",
    "metadata": {
     "editable": true
    },
@@ -2099,7 +2099,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4619ec8f",
+   "id": "b4fc87f6",
    "metadata": {
     "editable": true
    },
@@ -2109,7 +2109,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "529d5ed0",
+   "id": "15d7da17",
    "metadata": {
     "editable": true
    },
@@ -2121,7 +2121,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "762e07ea",
+   "id": "ee6e0550",
    "metadata": {
     "editable": true
    },
@@ -2131,7 +2131,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f45c2ce1",
+   "id": "2789c2b4",
    "metadata": {
     "editable": true
    },
@@ -2143,7 +2143,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "35835939",
+   "id": "1b0a4cce",
    "metadata": {
     "editable": true
    },
@@ -2155,7 +2155,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "901c3505",
+   "id": "ad73966d",
    "metadata": {
     "editable": true
    },
@@ -2167,7 +2167,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "63486657",
+   "id": "107fdb0a",
    "metadata": {
     "editable": true
    },
@@ -2189,7 +2189,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b8cb7b04",
+   "id": "8d070a5b",
    "metadata": {
     "editable": true
    },
@@ -2201,7 +2201,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8962eeb3",
+   "id": "356bded4",
    "metadata": {
     "editable": true
    },
@@ -2218,7 +2218,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ff83bf5a",
+   "id": "2b232f6c",
    "metadata": {
     "editable": true
    },
@@ -2230,7 +2230,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "67c405ee",
+   "id": "9ec41351",
    "metadata": {
     "editable": true
    },
@@ -2240,7 +2240,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d2279b06",
+   "id": "5fd89419",
    "metadata": {
     "editable": true
    },
@@ -2252,7 +2252,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c08bd913",
+   "id": "22a2cb66",
    "metadata": {
     "editable": true
    },
@@ -2262,7 +2262,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "585fcec4",
+   "id": "ac6721a2",
    "metadata": {
     "editable": true
    },
@@ -2274,7 +2274,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c5ee7c18",
+   "id": "da33b69e",
    "metadata": {
     "editable": true
    },
@@ -2284,7 +2284,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "250d6a7b",
+   "id": "44110c4b",
    "metadata": {
     "editable": true
    },
@@ -2296,7 +2296,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "26dca3eb",
+   "id": "de0ee363",
    "metadata": {
     "editable": true
    },
@@ -2313,7 +2313,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "43ae52cf",
+   "id": "3c2b4cb7",
    "metadata": {
     "editable": true
    },
@@ -2326,19 +2326,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fa5c2da2",
+   "id": "b8979ebf",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
+    "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ef125654",
+   "id": "f635713d",
    "metadata": {
     "editable": true
    },
@@ -2348,20 +2348,20 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3e398c3d",
+   "id": "e335573e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
-    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2,\n",
+    "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n",
+    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6e4285d1",
+   "id": "197171c0",
    "metadata": {
     "editable": true
    },
@@ -2371,7 +2371,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c1d8fde6",
+   "id": "494553da",
    "metadata": {
     "editable": true
    },
@@ -2383,57 +2383,57 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5b082cd2",
+   "id": "d131f255",
    "metadata": {
     "editable": true
    },
    "source": [
     "By minimizing the above equation with respect to the parameters\n",
-    "$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n",
-    "parameters $\\boldsymbol{\\beta}$.  We can add a regularization parameter $\\lambda$ by\n",
+    "$\\boldsymbol{\\theta}$ we could then obtain an analytical expression for the\n",
+    "parameters $\\boldsymbol{\\theta}$.  We can add a regularization parameter $\\lambda$ by\n",
     "defining a new cost function to be optimized, that is"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "306b6062",
+   "id": "3992861b",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
-    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\n",
+    "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n",
+    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f9c22f99",
+   "id": "e210e429",
    "metadata": {
     "editable": true
    },
    "source": [
     "which leads to the Ridge regression minimization problem where we\n",
-    "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n",
+    "require that $\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\\le t$, where $t$ is\n",
     "a finite number larger than zero. By defining"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8fa998f9",
+   "id": "960bf071",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n",
+    "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "954b3b63",
+   "id": "69ff9b18",
    "metadata": {
     "editable": true
    },
@@ -2443,20 +2443,20 @@
   },
   {
    "cell_type": "markdown",
-   "id": "40d226a6",
+   "id": "3b619870",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
-    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1\n",
+    "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n",
+    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "571654e9",
+   "id": "020995b7",
    "metadata": {
     "editable": true
    },
@@ -2468,7 +2468,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4b96f90f",
+   "id": "dc41832a",
    "metadata": {
     "editable": true
    },
@@ -2480,7 +2480,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6af80b33",
+   "id": "e4a20e48",
    "metadata": {
     "editable": true
    },
@@ -2490,25 +2490,25 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ed4e7687",
+   "id": "03ea4d0d",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n",
+    "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\boldsymbol{\\theta}^T\\boldsymbol{\\theta},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2eb319cc",
+   "id": "6734e20a",
    "metadata": {
     "editable": true
    },
    "source": [
     "and \n",
-    "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n",
+    "taking the derivatives with respect to $\\boldsymbol{\\theta}$ we obtain then\n",
     "a slightly modified matrix inversion problem which for finite values\n",
     "of $\\lambda$ does not suffer from singularity problems. We obtain\n",
     "the optimal parameters"
@@ -2516,19 +2516,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8528934c",
+   "id": "502e8acf",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
+    "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "22a7e810",
+   "id": "fcb7eeca",
    "metadata": {
     "editable": true
    },
@@ -2538,19 +2538,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "01d9c1c9",
+   "id": "f9789400",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n",
+    "\\sum_{i=0}^{p-1} \\theta_i^2 \\leq t,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5d910fd1",
+   "id": "cce5edcd",
    "metadata": {
     "editable": true
    },
@@ -2562,19 +2562,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b8bbfa39",
+   "id": "8b0e4f12",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
+    "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "51ac6ef1",
+   "id": "c59da228",
    "metadata": {
     "editable": true
    },
@@ -2585,7 +2585,7 @@
     "modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n",
     "particular for our discussion of the bias-variance tradeoff are rather\n",
     "interesting. We will see that for specific values of $\\lambda$, we may\n",
-    "even reduce the variance of the optimal parameters $\\boldsymbol{\\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n",
+    "even reduce the variance of the optimal parameters $\\boldsymbol{\\theta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n",
     "\n",
     "Using our insights about the SVD of the design matrix $\\boldsymbol{X}$ \n",
     "We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\\boldsymbol{U}$ as"
@@ -2593,19 +2593,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12ee6646",
+   "id": "96960524",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta}  =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n",
+    "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\theta}  =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "22852ccf",
+   "id": "ef0014d1",
    "metadata": {
     "editable": true
    },
@@ -2615,19 +2615,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "68d87eac",
+   "id": "57ba3f2e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n",
+    "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\theta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "0219a1a8",
+   "id": "86313c14",
    "metadata": {
     "editable": true
    },
@@ -2639,7 +2639,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "be1f339c",
+   "id": "c5c6f72d",
    "metadata": {
     "editable": true
    },
@@ -2651,7 +2651,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e1d5221",
+   "id": "4f780155",
    "metadata": {
     "editable": true
    },
@@ -2669,7 +2669,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0b4bc87c",
+   "id": "08413d65",
    "metadata": {
     "editable": true
    },
@@ -2681,7 +2681,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "de7e0986",
+   "id": "10f8be87",
    "metadata": {
     "editable": true
    },
@@ -2691,19 +2691,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0ae7dd26",
+   "id": "9d2b90ba",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n",
+    "\\boldsymbol{\\theta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5caa1086",
+   "id": "367f8b0e",
    "metadata": {
     "editable": true
    },
@@ -2713,19 +2713,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c737b14f",
+   "id": "cd4334f5",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n",
+    "\\boldsymbol{\\theta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\theta}^{\\mathrm{OLS}},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "60570cf0",
+   "id": "06f7db12",
    "metadata": {
     "editable": true
    },
@@ -2741,41 +2741,41 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f0c2e386",
+   "id": "b81118eb",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
+    "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5f3e91d2",
+   "id": "7d2b22a9",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)"
+    "Taking the derivative with respect to $\\boldsymbol{\\theta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "66b637e8",
+   "id": "2648d39f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{d \\vert \\beta\\vert}{d \\boldsymbol{\\beta}}=\\mathrm{sgn}(\\boldsymbol{\\beta})=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\-1 & \\beta < 0, \\end{array}\\right.\n",
+    "\\frac{d \\vert \\theta\\vert}{d \\boldsymbol{\\theta}}=\\mathrm{sgn}(\\boldsymbol{\\theta})=\\left\\{\\begin{array}{cc} 1 & \\theta > 0 \\\\-1 & \\theta < 0, \\end{array}\\right.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4082c969",
+   "id": "ea227065",
    "metadata": {
     "editable": true
    },
@@ -2785,19 +2785,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2c0b085b",
+   "id": "ce21fc5d",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n",
+    "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})+\\lambda sgn(\\boldsymbol{\\theta})=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "bf355533",
+   "id": "f4d13288",
    "metadata": {
     "editable": true
    },
@@ -2807,19 +2807,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "928d6cb3",
+   "id": "116661fd",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+    "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta}+\\lambda sgn(\\boldsymbol{\\theta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "39a40fd9",
+   "id": "dc54e1a7",
    "metadata": {
     "editable": true
    },
@@ -2829,24 +2829,24 @@
     "Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the\n",
     "diagonal. In this case we have an equal number of rows and columns $n=p$.\n",
     "\n",
-    "Our model approximation is just $\\tilde{\\boldsymbol{y}}=\\boldsymbol{\\beta}$ and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term $1/n$)"
+    "Our model approximation is just $\\tilde{\\boldsymbol{y}}=\\boldsymbol{\\theta}$ and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term $1/n$)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5d891e5c",
+   "id": "87c27103",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2,\n",
+    "C(\\boldsymbol{\\theta})=\\sum_{i=0}^{p-1}(y_i-\\theta_i)^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f9ea4d16",
+   "id": "530dada7",
    "metadata": {
     "editable": true
    },
@@ -2856,19 +2856,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "59b56446",
+   "id": "2e70789b",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\beta}_i^{\\mathrm{OLS}} = y_i.\n",
+    "\\hat{\\theta}_i^{\\mathrm{OLS}} = y_i.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b178aedb",
+   "id": "a69b1fda",
    "metadata": {
     "editable": true
    },
@@ -2878,19 +2878,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3c7c60e4",
+   "id": "6dbb9f75",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,\n",
+    "C(\\boldsymbol{\\theta})=\\sum_{i=0}^{p-1}(y_i-\\theta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\theta_i^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2f5d3c17",
+   "id": "fe4b2167",
    "metadata": {
     "editable": true
    },
@@ -2900,19 +2900,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2a612078",
+   "id": "c7ad140a",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\beta}_i^{\\mathrm{Ridge}} = \\frac{y_i}{1+\\lambda}.\n",
+    "\\hat{\\theta}_i^{\\mathrm{Ridge}} = \\frac{y_i}{1+\\lambda}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "dfb11d07",
+   "id": "6dab591b",
    "metadata": {
     "editable": true
    },
@@ -2922,19 +2922,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9331f4a6",
+   "id": "bfa5b9f9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\beta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\beta_i^2},\n",
+    "C(\\boldsymbol{\\theta})=\\sum_{i=0}^{p-1}(y_i-\\theta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\theta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\theta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\theta_i^2},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b4661ad6",
+   "id": "3125ab9c",
    "metadata": {
     "editable": true
    },
@@ -2944,19 +2944,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "60adb333",
+   "id": "309aed47",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "-2\\sum_{i=0}^{p-1}(y_i-\\beta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\beta_i)}{\\vert\\beta_i\\vert}=0,\n",
+    "-2\\sum_{i=0}^{p-1}(y_i-\\theta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\theta_i)}{\\vert\\theta_i\\vert}=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d5fa48d0",
+   "id": "b863d5f9",
    "metadata": {
     "editable": true
    },
@@ -2966,13 +2966,13 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bee441a6",
+   "id": "ffaed247",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n",
+    "\\hat{\\boldsymbol{\\theta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n",
     "                                                          y_i+\\frac{\\lambda}{2} &\\mathrm{if} & y_i< -\\frac{\\lambda}{2}\\\\\n",
     "\t\t\t\t\t\t\t  0 &\\mathrm{if} & \\vert y_i\\vert\\le  \\frac{\\lambda}{2}\\end{array}\\right.\\\\.\n",
     "$$"
@@ -2980,12 +2980,12 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dc6929bb",
+   "id": "6b89a6c5",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$.\n",
+    "Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\\theta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\theta_i$ as function of $\\lambda$.\n",
     "\n",
     "As another example, \n",
     "let us assume we have a data set with outputs/targets given by the vector"
@@ -2993,7 +2993,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "954061f9",
+   "id": "adadb0c3",
    "metadata": {
     "editable": true
    },
@@ -3005,7 +3005,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "97efa82b",
+   "id": "63968be0",
    "metadata": {
     "editable": true
    },
@@ -3015,7 +3015,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f2ed5f9b",
+   "id": "48e3a74e",
    "metadata": {
     "editable": true
    },
@@ -3027,31 +3027,31 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1c424fc1",
+   "id": "5948f0d4",
    "metadata": {
     "editable": true
    },
    "source": [
-    "meaning that we have two features and two unknown parameters $\\beta_0$ and $\\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression.\n",
+    "meaning that we have two features and two unknown parameters $\\theta_0$ and $\\theta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression.\n",
     "\n",
     "For ordinary least squares (OLS) we know that the optimal solution is"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "95467447",
+   "id": "55c9c8c1",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+    "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5da0739a",
+   "id": "42fd5011",
    "metadata": {
     "editable": true
    },
@@ -3061,19 +3061,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "47e7e72f",
+   "id": "cf6a68ad",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n",
+    "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "974570c8",
+   "id": "cf208d17",
    "metadata": {
     "editable": true
    },
@@ -3085,19 +3085,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dd9e2341",
+   "id": "60413571",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+    "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "70f08735",
+   "id": "8ff207f9",
    "metadata": {
     "editable": true
    },
@@ -3107,25 +3107,25 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c0a08b76",
+   "id": "846f76e7",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n",
+    "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1d8ed72a",
+   "id": "c3d07bb7",
    "metadata": {
     "editable": true
    },
    "source": [
-    "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2$ via the parameter $\\lambda$.\n",
-    "Let us for simplicity assume that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\beta$ and $\\lambda$.\n",
+    "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2$ via the parameter $\\lambda$.\n",
+    "Let us for simplicity assume that $\\theta_0^2+\\theta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\theta$ and $\\lambda$.\n",
     "\n",
     "To see this, let us write the cost function for Ridge regression.  \n",
     "\n",
@@ -3134,85 +3134,85 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9d4ef6fd",
+   "id": "ec07f63c",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{X}\\boldsymbol{\\beta}=\\begin{bmatrix} 2\\beta_0 \\\\ \\beta_1 \\\\0 \\end{bmatrix},\n",
+    "\\boldsymbol{X}\\boldsymbol{\\theta}=\\begin{bmatrix} 2\\theta_0 \\\\ \\theta_1 \\\\0 \\end{bmatrix},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "3ba1fd07",
+   "id": "40bd6afb",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\beta_0^2+\\beta_1^2),\n",
+    "C(\\boldsymbol{\\theta})=(4-2\\theta_0)^2+(2-\\theta_1)^2+\\lambda(\\theta_0^2+\\theta_1^2),\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2c6a0484",
+   "id": "b1d4b933",
    "metadata": {
     "editable": true
    },
    "source": [
-    "and taking the derivative with respect to $\\beta_0$ we get"
+    "and taking the derivative with respect to $\\theta_0$ we get"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7f86e745",
+   "id": "63371364",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0=\\frac{8}{4+\\lambda},\n",
+    "\\theta_0=\\frac{8}{4+\\lambda},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "46c8a1af",
+   "id": "71fb4eb4",
    "metadata": {
     "editable": true
    },
    "source": [
-    "and for $\\beta_1$ we obtain"
+    "and for $\\theta_1$ we obtain"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "33d6b311",
+   "id": "cacea391",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_1=\\frac{2}{1+\\lambda},\n",
+    "\\theta_1=\\frac{2}{1+\\lambda},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1e4eae3a",
+   "id": "a65f9b33",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Using the constraint for $\\beta_0^2+\\beta_1^2=1$ we can constrain $\\lambda$ by solving"
+    "Using the constraint for $\\theta_0^2+\\theta_1^2=1$ we can constrain $\\lambda$ by solving"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7a68a399",
+   "id": "96471d24",
    "metadata": {
     "editable": true
    },
@@ -3224,44 +3224,44 @@
   },
   {
    "cell_type": "markdown",
-   "id": "89874fbd",
+   "id": "ff4a5368",
    "metadata": {
     "editable": true
    },
    "source": [
-    "which gives $\\lambda=4.571$ and $\\beta_0=0.933$ and $\\beta_1=0.359$.\n",
+    "which gives $\\lambda=4.571$ and $\\theta_0=0.933$ and $\\theta_1=0.359$.\n",
     "\n",
-    "For Lasso we need now, keeping a  constraint on $\\vert\\beta_0\\vert+\\vert\\beta_1\\vert=1$,  to take the derivative of the absolute values of $\\beta_0$\n",
-    "and $\\beta_1$. This gives us the following derivatives of the cost function"
+    "For Lasso we need now, keeping a  constraint on $\\vert\\theta_0\\vert+\\vert\\theta_1\\vert=1$,  to take the derivative of the absolute values of $\\theta_0$\n",
+    "and $\\theta_1$. This gives us the following derivatives of the cost function"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a849fabc",
+   "id": "cbaab2b3",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\vert\\beta_0\\vert+\\vert\\beta_1\\vert),\n",
+    "C(\\boldsymbol{\\theta})=(4-2\\theta_0)^2+(2-\\theta_1)^2+\\lambda(\\vert\\theta_0\\vert+\\vert\\theta_1\\vert),\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "445f0b08",
+   "id": "8074c920",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-4(4-2\\beta_0)+\\lambda\\mathrm{sgn}(\\beta_0)=0,\n",
+    "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_0}=-4(4-2\\theta_0)+\\lambda\\mathrm{sgn}(\\theta_0)=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "669385c5",
+   "id": "339ebabc",
    "metadata": {
     "editable": true
    },
@@ -3271,50 +3271,50 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d5d14c30",
+   "id": "8803be3d",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}(\\beta_1)=0.\n",
+    "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_1}=-2(2-\\theta_1)+\\lambda\\mathrm{sgn}(\\theta_1)=0.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1db05ce9",
+   "id": "a3af4674",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We have now four cases to solve besides the trivial cases $\\beta_0$ and/or $\\beta_1$ are zero, namely\n",
-    "1. $\\beta_0 > 0$ and $\\beta_1 > 0$,\n",
+    "We have now four cases to solve besides the trivial cases $\\theta_0$ and/or $\\theta_1$ are zero, namely\n",
+    "1. $\\theta_0 > 0$ and $\\theta_1 > 0$,\n",
     "\n",
-    "2. $\\beta_0 > 0$ and $\\beta_1 < 0$,\n",
+    "2. $\\theta_0 > 0$ and $\\theta_1 < 0$,\n",
     "\n",
-    "3. $\\beta_0 < 0$ and $\\beta_1 > 0$,\n",
+    "3. $\\theta_0 < 0$ and $\\theta_1 > 0$,\n",
     "\n",
-    "4. $\\beta_0 < 0$ and $\\beta_1 < 0$.\n",
+    "4. $\\theta_0 < 0$ and $\\theta_1 < 0$.\n",
     "\n",
     "If we consider the first case, we have then"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "48598bde",
+   "id": "c77ef35e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "-4(4-2\\beta_0)+\\lambda=0,\n",
+    "-4(4-2\\theta_0)+\\lambda=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "458cc863",
+   "id": "69c8185d",
    "metadata": {
     "editable": true
    },
@@ -3324,19 +3324,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ff948eca",
+   "id": "304a2df1",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "-2(2-\\beta_1)+\\lambda=0.\n",
+    "-2(2-\\theta_1)+\\lambda=0.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "38dfba54",
+   "id": "55dd00fc",
    "metadata": {
     "editable": true
    },
@@ -3346,19 +3346,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bc715180",
+   "id": "d8f3cb2f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0=\\frac{16+\\lambda}{8},\n",
+    "\\theta_0=\\frac{16+\\lambda}{8},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "62e9a17c",
+   "id": "2db2c43e",
    "metadata": {
     "editable": true
    },
@@ -3368,24 +3368,24 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e564b775",
+   "id": "d875947f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_1=\\frac{4+\\lambda}{2}.\n",
+    "\\theta_1=\\frac{4+\\lambda}{2}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2bdbcac1",
+   "id": "a4ebed7b",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Using the constraint on $\\beta_0$ and $\\beta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you.\n",
+    "Using the constraint on $\\theta_0$ and $\\theta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you.\n",
     "\n",
     "Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of $\\lambda$, meaning that we need to perform a search in order to find the optimal values.\n",
     "\n",
@@ -3396,7 +3396,7 @@
   {
    "cell_type": "code",
    "execution_count": 9,
-   "id": "a1e3dee1",
+   "id": "ce6cdf5c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3424,14 +3424,14 @@
     "y = np.array( [4, 2, 3])\n",
     "\n",
     "\n",
-    "# matrix inversion to find beta\n",
-    "OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y\n",
-    "print(OLSbeta)\n",
+    "# matrix inversion to find theta\n",
+    "OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y\n",
+    "print(OLStheta)\n",
     "# and then make the prediction\n",
-    "ytildeOLS = X @ OLSbeta\n",
+    "ytildeOLS = X @ OLStheta\n",
     "print(\"Training MSE for OLS\")\n",
     "print(MSE(y,ytildeOLS))\n",
-    "ypredictOLS = X @ OLSbeta\n",
+    "ypredictOLS = X @ OLStheta\n",
     "\n",
     "# Repeat now for Ridge regression and various values of the regularization parameter\n",
     "I = np.eye(2,2)\n",
@@ -3441,10 +3441,10 @@
     "lambdas = np.logspace(-4, 4, nlambdas)\n",
     "for i in range(nlambdas):\n",
     "    lmb = lambdas[i]\n",
-    "    Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n",
-    "#    print(Ridgebeta)\n",
+    "    Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n",
+    "#    print(Ridgetheta)\n",
     "    # and then make the prediction\n",
-    "    ypredictRidge = X @ Ridgebeta\n",
+    "    ypredictRidge = X @ Ridgetheta\n",
     "    MSEPredict[i] = MSE(y,ypredictRidge)\n",
     "#    print(MSEPredict[i])\n",
     "    # Now plot the results\n",
@@ -3458,15 +3458,15 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5973148f",
+   "id": "79720acf",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\\boldsymbol{\\beta}$, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of $\\tilde{\\boldsymbol{y}}$ approach zero.\n",
+    "We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\\boldsymbol{\\theta}$, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of $\\tilde{\\boldsymbol{y}}$ approach zero.\n",
     "\n",
     "This happens also for Lasso regression, as seen from the next code\n",
-    "output. The difference is that Lasso shrinks the values of $\\beta$ to\n",
+    "output. The difference is that Lasso shrinks the values of $\\theta$ to\n",
     "zero at a much earlier stage and the results flatten out. We see that\n",
     "Lasso gives also an excellent fit for small values of $\\lambda$ and\n",
     "shows the best performance of the three regression methods."
@@ -3475,7 +3475,7 @@
   {
    "cell_type": "code",
    "execution_count": 10,
-   "id": "a961f69c",
+   "id": "ac1089e2",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3502,14 +3502,14 @@
     "y = np.array( [4, 2, 3])\n",
     "\n",
     "\n",
-    "# matrix inversion to find beta\n",
-    "OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y\n",
-    "print(OLSbeta)\n",
+    "# matrix inversion to find theta\n",
+    "OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y\n",
+    "print(OLStheta)\n",
     "# and then make the prediction\n",
-    "ytildeOLS = X @ OLSbeta\n",
+    "ytildeOLS = X @ OLStheta\n",
     "print(\"Training MSE for OLS\")\n",
     "print(MSE(y,ytildeOLS))\n",
-    "ypredictOLS = X @ OLSbeta\n",
+    "ypredictOLS = X @ OLStheta\n",
     "\n",
     "# Repeat now for Ridge regression and various values of the regularization parameter\n",
     "I = np.eye(2,2)\n",
@@ -3520,10 +3520,10 @@
     "lambdas = np.logspace(-4, 4, nlambdas)\n",
     "for i in range(nlambdas):\n",
     "    lmb = lambdas[i]\n",
-    "    Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n",
-    "    print(Ridgebeta)\n",
+    "    Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n",
+    "    print(Ridgetheta)\n",
     "    # and then make the prediction\n",
-    "    ypredictRidge = X @ Ridgebeta\n",
+    "    ypredictRidge = X @ Ridgetheta\n",
     "    MSERidgePredict[i] = MSE(y,ypredictRidge)\n",
     "    RegLasso = linear_model.Lasso(lmb)\n",
     "    RegLasso.fit(X,y)\n",
@@ -3542,7 +3542,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "60cfd641",
+   "id": "fbc293e8",
    "metadata": {
     "editable": true
    },
@@ -3552,7 +3552,7 @@
     "that for small values of the hyperparameter $\\lambda$ all three\n",
     "methods produce the same mean squared error. Again, Lasso shrinks the\n",
     "parameter values to zero much earlier than Ridge regression and the\n",
-    "Lasso results flatten out much earlier since all $\\beta_j=0$ (check\n",
+    "Lasso results flatten out much earlier since all $\\theta_j=0$ (check\n",
     "this by printing the values). This case is an example of where OLS\n",
     "performs best. Lasso and Ridge reproduce the OLS results for a limited\n",
     "set of $\\lambda$ values."
@@ -3561,7 +3561,7 @@
   {
    "cell_type": "code",
    "execution_count": 11,
-   "id": "171876b3",
+   "id": "b8d962a8",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3599,14 +3599,14 @@
     "# We split the data in test and training data\n",
     "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
     "\n",
-    "# matrix inversion to find beta\n",
-    "OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
-    "print(OLSbeta)\n",
+    "# matrix inversion to find theta\n",
+    "OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
+    "print(OLStheta)\n",
     "# and then make the prediction\n",
-    "ytildeOLS = X_train @ OLSbeta\n",
+    "ytildeOLS = X_train @ OLStheta\n",
     "print(\"Training MSE for OLS\")\n",
     "print(MSE(y_train,ytildeOLS))\n",
-    "ypredictOLS = X_test @ OLSbeta\n",
+    "ypredictOLS = X_test @ OLStheta\n",
     "print(\"Test MSE OLS\")\n",
     "print(MSE(y_test,ypredictOLS))\n",
     "\n",
@@ -3621,13 +3621,13 @@
     "lambdas = np.logspace(-4, 4, nlambdas)\n",
     "for i in range(nlambdas):\n",
     "    lmb = lambdas[i]\n",
-    "    Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
+    "    Ridgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
     "    # include lasso using Scikit-Learn\n",
     "    RegLasso = linear_model.Lasso(lmb)\n",
     "    RegLasso.fit(X_train,y_train)\n",
     "    # and then make the prediction\n",
-    "    ytildeRidge = X_train @ Ridgebeta\n",
-    "    ypredictRidge = X_test @ Ridgebeta\n",
+    "    ytildeRidge = X_train @ Ridgetheta\n",
+    "    ypredictRidge = X_test @ Ridgetheta\n",
     "    ytildeLasso = RegLasso.predict(X_train)\n",
     "    ypredictLasso = RegLasso.predict(X_test)\n",
     "    MSEPredict[i] = MSE(y_test,ypredictRidge)\n",
@@ -3650,7 +3650,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "947928e7",
+   "id": "083abcdc",
    "metadata": {
     "editable": true
    },
@@ -3665,7 +3665,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9559d0a8",
+   "id": "b23e5726",
    "metadata": {
     "editable": true
    },
@@ -3678,7 +3678,7 @@
     "particular, we will focus on what the regularization terms can result\n",
     "in.  We will amongst other things show that the regularization\n",
     "parameter can reduce considerably the variance of the parameters\n",
-    "$\\beta$.\n",
+    "$\\theta$.\n",
     "\n",
     "The\n",
     "advantage of doing linear regression is that we actually end up with\n",
@@ -3694,7 +3694,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6810eb7d",
+   "id": "2ea9005f",
    "metadata": {
     "editable": true
    },
@@ -3710,7 +3710,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bd997167",
+   "id": "dda3af52",
    "metadata": {
     "editable": true
    },
@@ -3718,7 +3718,7 @@
     "The randomness of $\\varepsilon_i$ implies that\n",
     "$\\mathbf{y}_i$ is also a random variable. In particular,\n",
     "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n",
-    "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n",
+    "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\theta}$ is a\n",
     "non-random scalar. To specify the parameters of the distribution of\n",
     "$\\mathbf{y}_i$ we need to calculate its first two moments. \n",
     "\n",
@@ -3733,7 +3733,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4ff740b9",
+   "id": "094aef6f",
    "metadata": {
     "editable": true
    },
@@ -3745,7 +3745,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fe0b2250",
+   "id": "677a8764",
    "metadata": {
     "editable": true
    },
@@ -3756,19 +3756,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "87e6b9b3",
+   "id": "396b620b",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
+    "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\theta}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "68d0de57",
+   "id": "9c73587c",
    "metadata": {
     "editable": true
    },
@@ -3778,7 +3778,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2f239890",
+   "id": "1a2d8f88",
    "metadata": {
     "editable": true
    },
@@ -3786,15 +3786,15 @@
     "$$\n",
     "\\begin{align*} \n",
     "\\mathbb{E}(y_i) & =\n",
-    "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n",
-    "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n",
+    "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta}) + \\mathbb{E}(\\varepsilon_i)\n",
+    "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\theta, \n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2a724679",
+   "id": "f9c84479",
    "metadata": {
     "editable": true
    },
@@ -3805,7 +3805,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2d710e45",
+   "id": "7b39b6a3",
    "metadata": {
     "editable": true
    },
@@ -3814,12 +3814,12 @@
     "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n",
     "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n",
     "[\\mathbb{E}(y_i)]^2  \\\\  & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n",
-    "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n",
-    "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n",
-    "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
-    "\\ast} \\, \\beta)^2 \\\\  & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n",
-    "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n",
-    "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n",
+    "\\theta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 \\\\ &\n",
+    "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 + 2 \\varepsilon_i\n",
+    "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
+    "\\ast} \\, \\theta)^2 \\\\  & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 + 2\n",
+    "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta} +\n",
+    "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 \n",
     "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n",
     "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2.  \n",
     "\\end{align*}\n",
@@ -3828,32 +3828,32 @@
   },
   {
    "cell_type": "markdown",
-   "id": "488a73d8",
+   "id": "b89a5103",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
-    "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n",
+    "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
+    "mean value $\\boldsymbol{X}\\boldsymbol{\\theta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n",
     "\n",
-    "With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ we can evaluate the expectation value"
+    "With the OLS expressions for the parameters $\\boldsymbol{\\theta}$ we can evaluate the expectation value"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "9b1dca9a",
+   "id": "311e8043",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathbb{E}(\\boldsymbol{\\beta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n",
+    "\\mathbb{E}(\\boldsymbol{\\theta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\theta}=\\boldsymbol{\\theta}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "07089a59",
+   "id": "2b32d6f7",
    "metadata": {
     "editable": true
    },
@@ -3862,35 +3862,35 @@
     "\n",
     "We can also calculate the variance\n",
     "\n",
-    "The variance of $\\boldsymbol{\\beta}$ is"
+    "The variance of $\\boldsymbol{\\theta}$ is"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "690bd104",
+   "id": "f5a7fb75",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{eqnarray*}\n",
-    "\\mbox{Var}(\\boldsymbol{\\beta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n",
+    "\\mbox{Var}(\\boldsymbol{\\theta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\theta} - \\mathbb{E}(\\boldsymbol{\\theta})] [\\boldsymbol{\\theta} - \\mathbb{E}(\\boldsymbol{\\theta})]^{T} \\}\n",
     "\\\\\n",
-    "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}]^{T} \\}\n",
+    "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\theta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\theta}]^{T} \\}\n",
     "\\\\\n",
-    "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "% \\\\\n",
-    "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}  \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}  \\} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "% \\\\\n",
-    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "\\\\\n",
-    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\,  \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} \\,  \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "% \\\\\n",
-    "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\,  \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
+    "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^T \\,  \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
     "% \\\\\n",
-    "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T  \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n",
+    "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T  \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\boldsymbol{\\theta}^T\n",
     "\\\\\n",
-    "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}  + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "& = & \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}  + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n",
     "\\end{eqnarray*}\n",
     "$$"
@@ -3898,21 +3898,21 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6a9132ce",
+   "id": "6944f707",
    "metadata": {
     "editable": true
    },
    "source": [
     "where we have used  that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n",
-    "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n",
-    "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n",
+    "\\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} \\, \\mathbf{X}^{T} +\n",
+    "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\theta}) = \\sigma^2\n",
     "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n",
     "variance of the estimate of the $j$-th regression coefficient:\n",
-    "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n",
+    "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\theta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n",
     "construct a confidence interval for the estimates.\n",
     "\n",
     "In a similar way, we can obtain analytical expressions for say the\n",
-    "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n",
+    "expectation values of the parameters $\\boldsymbol{\\theta}$ and their variance\n",
     "when we employ Ridge regression, allowing us again to define a confidence interval. \n",
     "\n",
     "It is rather straightforward to show that"
@@ -3920,80 +3920,80 @@
   },
   {
    "cell_type": "markdown",
-   "id": "68cce775",
+   "id": "9c8d4134",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n",
+    "\\mathbb{E} \\big[ \\boldsymbol{\\theta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\theta}^{\\mathrm{OLS}}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a9c3f89a",
+   "id": "6d47f172",
    "metadata": {
     "editable": true
    },
    "source": [
     "We see clearly that \n",
-    "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n",
+    "$\\mathbb{E} \\big[ \\boldsymbol{\\theta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\theta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n",
     "\n",
     "We can also compute the variance as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f9e2f9d7",
+   "id": "e41732d4",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}  \\mathbf{X}^{T} \\mathbf{X} \\{ [  \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
+    "\\mbox{Var}[\\boldsymbol{\\theta}^{\\mathrm{Ridge}}]=\\sigma^2[  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}  \\mathbf{X}^{T} \\mathbf{X} \\{ [  \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "58443fe8",
+   "id": "2e3123fe",
    "metadata": {
     "editable": true
    },
    "source": [
-    "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n",
+    "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\theta}$ goes to zero. \n",
     "\n",
     "With this, we can compute the difference"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "cc34c059",
+   "id": "9ba7ba08",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
+    "\\mbox{Var}[\\boldsymbol{\\theta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\theta}^{\\mathrm{Ridge}})=\\sigma^2 [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6ad9c8e3",
+   "id": "f4077e50",
    "metadata": {
     "editable": true
    },
    "source": [
     "The difference is non-negative definite since each component of the\n",
     "matrix product is non-negative definite. \n",
-    "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below."
+    "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\theta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below."
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7c09657d",
+   "id": "2945f567",
    "metadata": {
     "editable": true
    },
@@ -4007,28 +4007,28 @@
     "$\\sigma^2$.\n",
     "\n",
     "We found above that the outputs $\\boldsymbol{y}$ have a mean value given by\n",
-    "$\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}}$ and variance $\\sigma^2$. Since the entries to\n",
+    "$\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$ and variance $\\sigma^2$. Since the entries to\n",
     "the design matrix are not stochastic variables, we can assume that the\n",
     "probability distribution of our targets is also a normal distribution\n",
-    "but now with mean value $\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}}$. This means that a\n",
+    "but now with mean value $\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$. This means that a\n",
     "single output $y_i$ is given by the Gaussian distribution"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "abe9915b",
+   "id": "1324fb2e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
+    "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "326e0c75",
+   "id": "f19032d0",
    "metadata": {
     "editable": true
    },
@@ -4039,43 +4039,43 @@
   },
   {
    "cell_type": "markdown",
-   "id": "567fb1b1",
+   "id": "a18bd251",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n",
+    "p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\theta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "107abe1c",
+   "id": "9f5be910",
    "metadata": {
     "editable": true
    },
    "source": [
-    "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\beta}$.\n",
+    "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\theta}$.\n",
     "\n",
     "Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event $\\boldsymbol{y}$ as the product of the single events, that is we have"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f11ddf78",
+   "id": "d92545ca",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta}).\n",
+    "p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\theta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\theta}).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2abd6e3b",
+   "id": "d30bf273",
    "metadata": {
     "editable": true
    },
@@ -4086,7 +4086,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "caddb652",
+   "id": "70795118",
    "metadata": {
     "editable": true
    },
@@ -4098,7 +4098,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "291e1dd6",
+   "id": "ecfb478e",
    "metadata": {
     "editable": true
    },
@@ -4109,26 +4109,26 @@
   },
   {
    "cell_type": "markdown",
-   "id": "73ac95c1",
+   "id": "89c5df6b",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
+    "p(\\boldsymbol{D}\\vert\\boldsymbol{\\theta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ce49493b",
+   "id": "082b8841",
    "metadata": {
     "editable": true
    },
    "source": [
     "It is a conditional probability (see below) and reads as the\n",
     "likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters\n",
-    "$\\boldsymbol{\\beta}$.\n",
+    "$\\boldsymbol{\\theta}$.\n",
     "\n",
     "In statistics, maximum likelihood estimation (MLE) is a method of\n",
     "estimating the parameters of an assumed probability distribution,\n",
@@ -4137,7 +4137,7 @@
     "data is the most probable. \n",
     "\n",
     "We will assume here that our events are given by the above Gaussian\n",
-    "distribution and we will determine the optimal parameters $\\beta$ by\n",
+    "distribution and we will determine the optimal parameters $\\theta$ by\n",
     "maximizing the above PDF. However, computing the derivatives of a\n",
     "product function is cumbersome and can easily lead to overflow and/or\n",
     "underflowproblems, with potentials for loss of numerical precision.\n",
@@ -4156,19 +4156,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1ff54861",
+   "id": "59d1eaca",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta}=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})},\n",
+    "C(\\boldsymbol{\\theta}=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\theta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\theta})},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e8cdd425",
+   "id": "7142a2d9",
    "metadata": {
     "editable": true
    },
@@ -4178,63 +4178,63 @@
   },
   {
    "cell_type": "markdown",
-   "id": "95d54be7",
+   "id": "3e111f5b",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}.\n",
+    "C(\\boldsymbol{\\theta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "731e3e2a",
+   "id": "de59021f",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Taking the derivative of the *new* cost function with respect to the parameters $\\beta$ we recognize our familiar OLS equation, namely"
+    "Taking the derivative of the *new* cost function with respect to the parameters $\\theta$ we recognize our familiar OLS equation, namely"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8c40a24c",
+   "id": "5e78bfd8",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right) =0,\n",
+    "\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right) =0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1e298a02",
+   "id": "09836e12",
    "metadata": {
     "editable": true
    },
    "source": [
-    "which leads to the well-known OLS equation for the optimal paramters $\\beta$"
+    "which leads to the well-known OLS equation for the optimal paramters $\\theta$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "fd5c3e4f",
+   "id": "2f3404a5",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n",
+    "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d3aab131",
+   "id": "0bc175f7",
    "metadata": {
     "editable": true
    },
@@ -4254,7 +4254,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "64646b7c",
+   "id": "2d0434aa",
    "metadata": {
     "editable": true
    },
@@ -4266,7 +4266,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0ad4cc29",
+   "id": "42f4b84f",
    "metadata": {
     "editable": true
    },
@@ -4276,7 +4276,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8dfd2150",
+   "id": "3677eed8",
    "metadata": {
     "editable": true
    },
@@ -4288,7 +4288,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9c0313b7",
+   "id": "134f8f1a",
    "metadata": {
     "editable": true
    },
@@ -4302,7 +4302,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "af94800f",
+   "id": "9f6349f1",
    "metadata": {
     "editable": true
    },
@@ -4314,7 +4314,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3ed2ccef",
+   "id": "557eae67",
    "metadata": {
     "editable": true
    },
@@ -4324,7 +4324,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a74ee19",
+   "id": "0859c412",
    "metadata": {
     "editable": true
    },
@@ -4336,7 +4336,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5191a71e",
+   "id": "ef82304a",
    "metadata": {
     "editable": true
    },
@@ -4346,7 +4346,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5d5de8f7",
+   "id": "9f2500ca",
    "metadata": {
     "editable": true
    },
@@ -4358,7 +4358,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cca75f59",
+   "id": "b7b9d2d8",
    "metadata": {
     "editable": true
    },
@@ -4368,7 +4368,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9113e675",
+   "id": "f79df283",
    "metadata": {
     "editable": true
    },
@@ -4380,7 +4380,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a21d13da",
+   "id": "ca93f0cf",
    "metadata": {
     "editable": true
    },
@@ -4416,7 +4416,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0c7abec6",
+   "id": "9092d777",
    "metadata": {
     "editable": true
    },
@@ -4428,7 +4428,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2eccb1af",
+   "id": "017ca9fb",
    "metadata": {
     "editable": true
    },
@@ -4441,7 +4441,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3c6635e5",
+   "id": "529014b2",
    "metadata": {
     "editable": true
    },
@@ -4453,7 +4453,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ecf0a0b6",
+   "id": "23629d2d",
    "metadata": {
     "editable": true
    },
@@ -4467,7 +4467,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "166345a1",
+   "id": "80c6a09b",
    "metadata": {
     "editable": true
    },
@@ -4479,7 +4479,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8a73e80e",
+   "id": "fe371d69",
    "metadata": {
     "editable": true
    },
@@ -4490,7 +4490,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "01441388",
+   "id": "ed9320dd",
    "metadata": {
     "editable": true
    },
@@ -4502,7 +4502,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "93fe1e0a",
+   "id": "710d72a1",
    "metadata": {
     "editable": true
    },
@@ -4514,7 +4514,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d6860415",
+   "id": "331424a5",
    "metadata": {
     "editable": true
    },
@@ -4532,7 +4532,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9710dd92",
+   "id": "683874d3",
    "metadata": {
     "editable": true
    },
@@ -4550,7 +4550,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "23e3912e",
+   "id": "2b32fd95",
    "metadata": {
     "editable": true
    },
@@ -4560,7 +4560,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cf500f71",
+   "id": "9ed06a74",
    "metadata": {
     "editable": true
    },
@@ -4575,9 +4575,9 @@
     "Before we proceed let us perform a Ridge, Lasso  and OLS analysis of a polynomial fit. \n",
     "\n",
     "We will play around with a study of the values for the optimal\n",
-    "parameters $\\boldsymbol{\\beta}$ using OLS, Ridge and Lasso regression.  For\n",
+    "parameters $\\boldsymbol{\\theta}$ using OLS, Ridge and Lasso regression.  For\n",
     "OLS, you will notice as function of the noise and polynomial degree,\n",
-    "that the parameters $\\beta$ will fluctuate from order to order in the\n",
+    "that the parameters $\\theta$ will fluctuate from order to order in the\n",
     "polynomial fit and that for larger and larger polynomial degrees of\n",
     "freedom, the parameters will tend to increase in value for OLS.\n",
     "\n",
@@ -4589,7 +4589,7 @@
   {
    "cell_type": "code",
    "execution_count": 12,
-   "id": "134d0a22",
+   "id": "9a336ec4",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4624,10 +4624,10 @@
     "# We split the data in test and training data\n",
     "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
     "\n",
-    "# matrix inversion to find beta\n",
-    "OLSbeta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train\n",
-    "print(OLSbeta)\n",
-    "ypredictOLS = X_test @ OLSbeta\n",
+    "# matrix inversion to find theta\n",
+    "OLStheta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train\n",
+    "print(OLStheta)\n",
+    "ypredictOLS = X_test @ OLStheta\n",
     "print(\"Test MSE OLS\")\n",
     "print(MSE(y_test,ypredictOLS))\n",
     "# Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn\n",
@@ -4663,19 +4663,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7dcfe550",
+   "id": "87b78389",
    "metadata": {
     "editable": true
    },
    "source": [
     "How can we understand this?\n",
     "\n",
-    "Let us write out the values of the coefficients $\\beta_i$ as functions\n",
+    "Let us write out the values of the coefficients $\\theta_i$ as functions\n",
     "of the polynomial degree and noise. We will focus only on the Ridge\n",
     "results and some few selected values of the hyperparameter $\\lambda$.\n",
     "\n",
     "If we don't include any noise and run this code for different values\n",
-    "of the polynomial degree, we notice that the results for $\\beta_i$ do\n",
+    "of the polynomial degree, we notice that the results for $\\theta_i$ do\n",
     "not show great changes from one order to the next. This is an\n",
     "indication that for higher polynomial orders, our parameters become\n",
     "less important.\n",
@@ -4683,17 +4683,17 @@
     "If we however add noise, what happens is that the polynomial fit is\n",
     "trying to adjust the fit to traverse in the best possible way all data\n",
     "points. This can lead to large fluctuations in the parameters\n",
-    "$\\beta_i$ as functions of polynomial order. It will also be reflected\n",
-    "in a larger value of the variance of each parameter $\\beta_i$.  What\n",
+    "$\\theta_i$ as functions of polynomial order. It will also be reflected\n",
+    "in a larger value of the variance of each parameter $\\theta_i$.  What\n",
     "Ridge regression (and Lasso as well) are doing then is to try to\n",
-    "quench the fluctuations in the parameters of $\\beta_i$ which have a\n",
+    "quench the fluctuations in the parameters of $\\theta_i$ which have a\n",
     "large variance (normally for higher orders in the polynomial)."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 13,
-   "id": "5ff5d7c2",
+   "id": "e2c403e8",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4735,30 +4735,30 @@
     "    # and then make the prediction\n",
     "    ypredictRidge = RegRidge.predict(X_test)\n",
     "    Coeffs = np.array(RegRidge.coef_)\n",
-    "    BetaValues = pd.DataFrame(Coeffs)\n",
-    "    BetaValues.columns = ['beta']\n",
-    "    display(BetaValues)"
+    "    ThetaValues = pd.DataFrame(Coeffs)\n",
+    "    ThetaValues.columns = ['theta']\n",
+    "    display(ThetaValues)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f1680928",
+   "id": "4f4dd369",
    "metadata": {
     "editable": true
    },
    "source": [
     "As an exercise, repeat these calculations with ordinary least squares\n",
     "only with and without noise. Calculate thereafter the variance of the\n",
-    "parameters $\\beta_j$ as function of polynomial order and of the added\n",
+    "parameters $\\theta_j$ as function of polynomial order and of the added\n",
     "noise. Here we recommend to use $\\sigma^2=1$ as variance for the\n",
     "added noise (which follows a normal distribution with mean value zero).\n",
-    "Comment your results. If you have a large noise term, do the parameters $\\beta_j$ vary more as function\n",
+    "Comment your results. If you have a large noise term, do the parameters $\\theta_j$ vary more as function\n",
     "of model complexity? And what about their variance?"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5b458403",
+   "id": "67adfefc",
    "metadata": {
     "editable": true
    },
@@ -4767,7 +4767,7 @@
     "\n",
     "We have seen that Ridge regression suppresses those features which\n",
     "have a small singular value. This corresponds to a feature which exhibits\n",
-    "a large variance in the parameters $\\beta_j$.\n",
+    "a large variance in the parameters $\\theta_j$.\n",
     "Our analysis hitherto has been based on linear algebra. To add to our intuition, we will use\n",
     "Bayes' theorem in order to deepen our understanding of  Ridge and Lasso regression. \n",
     "\n",
@@ -4776,7 +4776,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6e2d6bf6",
+   "id": "ab781318",
    "metadata": {
     "editable": true
    },
@@ -4788,7 +4788,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "098b6cbd",
+   "id": "ccf25024",
    "metadata": {
     "editable": true
    },
@@ -4798,41 +4798,41 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8a02d0aa",
+   "id": "9c8bc496",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
+    "p(\\boldsymbol{D}\\vert\\boldsymbol{\\theta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "97f22408",
+   "id": "7dcf96d4",
    "metadata": {
     "editable": true
    },
    "source": [
-    "In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\beta}$ given a domain of events $\\boldsymbol{D}$?  That is, how can we define the posterior probability"
+    "In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\theta}$ given a domain of events $\\boldsymbol{D}$?  That is, how can we define the posterior probability"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "27038459",
+   "id": "83af5cfb",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D}).\n",
+    "p(\\boldsymbol{\\theta}\\vert\\boldsymbol{D}).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f682a8c3",
+   "id": "baf02e7e",
    "metadata": {
     "editable": true
    },
@@ -4842,49 +4842,49 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7fdaa748",
+   "id": "1ec2d48a",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})p(\\boldsymbol{\\beta}).\n",
+    "p(\\boldsymbol{\\theta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\theta})p(\\boldsymbol{\\theta}).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1e7fa52c",
+   "id": "e77ae0c0",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta})$!   \n",
+    "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\theta})$ but need one for the **prior** $p(\\boldsymbol{\\theta})$!   \n",
     "\n",
     "With the posterior probability defined by a likelihood which we have\n",
     "already modeled and an unknown prior, we are now ready to make\n",
     "additional models for the prior.\n",
     "\n",
-    "We can, based on our discussions of the variance of $\\boldsymbol{\\beta}$ and\n",
-    "the mean value, assume that the prior for the values $\\boldsymbol{\\beta}$ is\n",
+    "We can, based on our discussions of the variance of $\\boldsymbol{\\theta}$ and\n",
+    "the mean value, assume that the prior for the values $\\boldsymbol{\\theta}$ is\n",
     "given by a Gaussian with mean value zero and variance $\\tau^2$, that"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "501d66f4",
+   "id": "5ee70503",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n",
+    "p(\\boldsymbol{\\theta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\theta_j^2}{2\\tau^2}\\right)}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f029c143",
+   "id": "81b5ea11",
    "metadata": {
     "editable": true
    },
@@ -4894,44 +4894,44 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7f7c3e11",
+   "id": "a15cac35",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n",
+    "p(\\boldsymbol{\\theta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\theta_j^2}{2\\tau^2}\\right)}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1f39114c",
+   "id": "76f66371",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We can now optimize this quantity with respect to $\\boldsymbol{\\beta}$. As we\n",
+    "We can now optimize this quantity with respect to $\\boldsymbol{\\theta}$. As we\n",
     "did for OLS, this is most conveniently done by taking the negative\n",
     "logarithm of the posterior probability. Doing so and leaving out the\n",
-    "terms that do not depend on $\\beta$, we have"
+    "terms that do not depend on $\\theta$, we have"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "81cc7b03",
+   "id": "deaecaf5",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n",
+    "C(\\boldsymbol{\\theta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_2^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1e614b9b",
+   "id": "da1297cc",
    "metadata": {
     "editable": true
    },
@@ -4941,19 +4941,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "77252afc",
+   "id": "4c429a58",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n",
+    "C(\\boldsymbol{\\theta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_2^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "14953579",
+   "id": "2c9fd875",
    "metadata": {
     "editable": true
    },
@@ -4965,19 +4965,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "36f1f63d",
+   "id": "350a62eb",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n",
+    "p(\\boldsymbol{\\theta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\theta_j\\vert}{\\tau}\\right)}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "50dd90a5",
+   "id": "9bb76e79",
    "metadata": {
     "editable": true
    },
@@ -4987,43 +4987,43 @@
   },
   {
    "cell_type": "markdown",
-   "id": "de39cb12",
+   "id": "e08c07dd",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n",
+    "p(\\boldsymbol{\\theta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\theta_j\\vert}{\\tau}\\right)}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ad1fc46e",
+   "id": "e4bd6b2c",
    "metadata": {
     "editable": true
    },
    "source": [
     "Taking the negative\n",
     "logarithm of the posterior probability and leaving out the\n",
-    "constants terms that do not depend on $\\beta$, we have"
+    "constants terms that do not depend on $\\theta$, we have"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ff8695d4",
+   "id": "7c1bc34c",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
+    "C(\\boldsymbol{\\theta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "0de8080e",
+   "id": "5f24477c",
    "metadata": {
     "editable": true
    },
@@ -5033,19 +5033,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3965e5ef",
+   "id": "d9948bc0",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
+    "C(\\boldsymbol{\\theta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5c978cdf",
+   "id": "d690afa3",
    "metadata": {
     "editable": true
    },
@@ -5054,7 +5054,7 @@
     "\n",
     "Plotting these prior functions shows us that we can use the parameter\n",
     "$\\lambda$ to shrink or increase the role of a given parameter\n",
-    "$\\beta_j$.  The variance for the Laplace distribution is\n",
+    "$\\theta_j$.  The variance for the Laplace distribution is\n",
     "$2\\tau^2=1/\\lambda$ while for the Gaussian distribution it is\n",
     "$\\sigma^2=1/(2\\lambda)$. Thus, increasing the variance means\n",
     "decreasing $\\lambda$ and shrinking the variance means increasing\n",
diff --git a/doc/LectureNotes/_build/jupyter_execute/chapter3.ipynb b/doc/LectureNotes/_build/jupyter_execute/chapter3.ipynb
index 4e9aadce0..66adc6b17 100644
--- a/doc/LectureNotes/_build/jupyter_execute/chapter3.ipynb
+++ b/doc/LectureNotes/_build/jupyter_execute/chapter3.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "51186d57",
+   "id": "663483da",
    "metadata": {
     "editable": true
    },
@@ -13,7 +13,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "79a63fb4",
+   "id": "05ca2993",
    "metadata": {
     "editable": true
    },
@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f3d916b4",
+   "id": "cea49fb9",
    "metadata": {
     "editable": true
    },
@@ -74,7 +74,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "46cb3279",
+   "id": "85adbc81",
    "metadata": {
     "editable": true
    },
@@ -105,7 +105,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0fe38e07",
+   "id": "4d010e03",
    "metadata": {
     "editable": true
    },
@@ -121,7 +121,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d9d6955b",
+   "id": "54c13e4f",
    "metadata": {
     "editable": true
    },
@@ -129,7 +129,7 @@
     "The randomness of $\\varepsilon_i$ implies that\n",
     "$\\mathbf{y}_i$ is also a random variable. In particular,\n",
     "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n",
-    "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n",
+    "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\theta}$ is a\n",
     "non-random scalar. To specify the parameters of the distribution of\n",
     "$\\mathbf{y}_i$ we need to calculate its first two moments. \n",
     "\n",
@@ -144,7 +144,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f41f7049",
+   "id": "1705258c",
    "metadata": {
     "editable": true
    },
@@ -156,7 +156,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2ee172f4",
+   "id": "f96c2ee1",
    "metadata": {
     "editable": true
    },
@@ -167,19 +167,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1aecc768",
+   "id": "84fc8fc9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
+    "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\theta}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "12e9bdea",
+   "id": "c846eb35",
    "metadata": {
     "editable": true
    },
@@ -189,7 +189,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8f9db9db",
+   "id": "a7c728c5",
    "metadata": {
     "editable": true
    },
@@ -197,15 +197,15 @@
     "$$\n",
     "\\begin{align*} \n",
     "\\mathbb{E}(y_i) & =\n",
-    "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n",
-    "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n",
+    "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta}) + \\mathbb{E}(\\varepsilon_i)\n",
+    "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\theta, \n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b10abe89",
+   "id": "d7d34864",
    "metadata": {
     "editable": true
    },
@@ -216,7 +216,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bec51521",
+   "id": "f235a710",
    "metadata": {
     "editable": true
    },
@@ -225,12 +225,12 @@
     "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n",
     "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n",
     "[\\mathbb{E}(y_i)]^2  \\\\  & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n",
-    "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n",
-    "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n",
-    "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
-    "\\ast} \\, \\beta)^2 \\\\  & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n",
-    "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n",
-    "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n",
+    "\\theta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 \\\\ &\n",
+    "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 + 2 \\varepsilon_i\n",
+    "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
+    "\\ast} \\, \\theta)^2 \\\\  & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 + 2\n",
+    "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta} +\n",
+    "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 \n",
     "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n",
     "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2.  \n",
     "\\end{align*}\n",
@@ -239,32 +239,32 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4859640c",
+   "id": "faa4158d",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
-    "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n",
+    "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
+    "mean value $\\boldsymbol{X}\\boldsymbol{\\theta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n",
     "\n",
-    "With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ we can evaluate the expectation value"
+    "With the OLS expressions for the parameters $\\boldsymbol{\\theta}$ we can evaluate the expectation value"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "69978823",
+   "id": "da189fae",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathbb{E}(\\boldsymbol{\\beta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n",
+    "\\mathbb{E}(\\boldsymbol{\\theta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\theta}=\\boldsymbol{\\theta}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5caee9d9",
+   "id": "640c53e9",
    "metadata": {
     "editable": true
    },
@@ -273,35 +273,35 @@
     "\n",
     "We can also calculate the variance\n",
     "\n",
-    "The variance of $\\boldsymbol{\\beta}$ is"
+    "The variance of $\\boldsymbol{\\theta}$ is"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6791e5b4",
+   "id": "e2eee7fb",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{eqnarray*}\n",
-    "\\mbox{Var}(\\boldsymbol{\\beta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n",
+    "\\mbox{Var}(\\boldsymbol{\\theta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\theta} - \\mathbb{E}(\\boldsymbol{\\theta})] [\\boldsymbol{\\theta} - \\mathbb{E}(\\boldsymbol{\\theta})]^{T} \\}\n",
     "\\\\\n",
-    "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}]^{T} \\}\n",
+    "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\theta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\theta}]^{T} \\}\n",
     "\\\\\n",
-    "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "% \\\\\n",
-    "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}  \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}  \\} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "% \\\\\n",
-    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "\\\\\n",
-    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\,  \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} \\,  \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "% \\\\\n",
-    "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\,  \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
+    "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^T \\,  \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
     "% \\\\\n",
-    "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T  \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n",
+    "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T  \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\boldsymbol{\\theta}^T\n",
     "\\\\\n",
-    "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}  + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "& = & \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}  + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n",
     "\\end{eqnarray*}\n",
     "$$"
@@ -309,22 +309,22 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4f747992",
+   "id": "69660ffd",
    "metadata": {
     "editable": true
    },
    "source": [
     "where we have used  that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n",
-    "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n",
-    "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n",
+    "\\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} \\, \\mathbf{X}^{T} +\n",
+    "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\theta}) = \\sigma^2\n",
     "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n",
     "variance of the estimate of the $j$-th regression coefficient:\n",
-    "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 \\sqrt{\n",
+    "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\theta}_j ) = \\boldsymbol{\\sigma}^2 \\sqrt{\n",
     "[(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} }$. This may be used to\n",
     "construct a confidence interval for the estimates.\n",
     "\n",
     "In a similar way, we can obtain analytical expressions for say the\n",
-    "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n",
+    "expectation values of the parameters $\\boldsymbol{\\theta}$ and their variance\n",
     "when we employ Ridge regression, allowing us again to define a confidence interval. \n",
     "\n",
     "It is rather straightforward to show that"
@@ -332,80 +332,80 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d4679496",
+   "id": "166fe7a6",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n",
+    "\\mathbb{E} \\big[ \\boldsymbol{\\theta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\theta}^{\\mathrm{OLS}}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "bd3534e1",
+   "id": "9b9e1336",
    "metadata": {
     "editable": true
    },
    "source": [
     "We see clearly that \n",
-    "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n",
+    "$\\mathbb{E} \\big[ \\boldsymbol{\\theta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\theta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n",
     "\n",
     "We can also compute the variance as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "125b139e",
+   "id": "d4548ed8",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}  \\mathbf{X}^{T} \\mathbf{X} \\{ [  \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
+    "\\mbox{Var}[\\boldsymbol{\\theta}^{\\mathrm{Ridge}}]=\\sigma^2[  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}  \\mathbf{X}^{T} \\mathbf{X} \\{ [  \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "98b59ea0",
+   "id": "642f4b4a",
    "metadata": {
     "editable": true
    },
    "source": [
-    "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n",
+    "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\theta}$ goes to zero. \n",
     "\n",
     "With this, we can compute the difference"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "dcfd7a6c",
+   "id": "e8a1528a",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
+    "\\mbox{Var}[\\boldsymbol{\\theta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\theta}^{\\mathrm{Ridge}})=\\sigma^2 [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "30f43be3",
+   "id": "1bd2f713",
    "metadata": {
     "editable": true
    },
    "source": [
     "The difference is non-negative definite since each component of the\n",
     "matrix product is non-negative definite. \n",
-    "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below."
+    "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\theta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below."
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e3c1bf3c",
+   "id": "8019b9c6",
    "metadata": {
     "editable": true
    },
@@ -446,14 +446,14 @@
     "variance of $\\overline{X}$ (which often is the case), then there is no\n",
     "need for bootstrapping. \n",
     "\n",
-    "The Jackknife works by making many replicas of the estimator $\\widehat{\\beta}$. \n",
+    "The Jackknife works by making many replicas of the estimator $\\widehat{\\theta}$. \n",
     "The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\\boldsymbol{x} = (x_1,x_2,\\cdots,X_n)$. \n",
     "Let $\\boldsymbol{x}_i$ denote the vector"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "3925f435",
+   "id": "38e804f2",
    "metadata": {
     "editable": true
    },
@@ -465,21 +465,21 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cc6328ee",
+   "id": "35eb791c",
    "metadata": {
     "editable": true
    },
    "source": [
     "which equals the vector $\\boldsymbol{x}$ with the exception that observation\n",
     "number $i$ is left out. Using this notation, define\n",
-    "$\\widehat{\\beta}_i$ to be the estimator\n",
-    "$\\widehat{\\beta}$ computed using $\\vec{X}_i$."
+    "$\\widehat{\\theta}_i$ to be the estimator\n",
+    "$\\widehat{\\theta}$ computed using $\\vec{X}_i$."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "b751a941",
+   "id": "3261f074",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -518,7 +518,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "25ff562a",
+   "id": "9d117c52",
    "metadata": {
     "editable": true
    },
@@ -537,28 +537,28 @@
     "\n",
     "4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).\n",
     "\n",
-    "Since $\\widehat{\\beta} = \\widehat{\\beta}(\\boldsymbol{X})$ is a function of random variables,\n",
-    "$\\widehat{\\beta}$ itself must be a random variable. Thus it has\n",
+    "Since $\\widehat{\\theta} = \\widehat{\\theta}(\\boldsymbol{X})$ is a function of random variables,\n",
+    "$\\widehat{\\theta}$ itself must be a random variable. Thus it has\n",
     "a pdf, call this function $p(\\boldsymbol{t})$. The aim of the bootstrap is to\n",
     "estimate $p(\\boldsymbol{t})$ by the relative frequency of\n",
-    "$\\widehat{\\beta}$. You can think of this as using a histogram\n",
+    "$\\widehat{\\theta}$. You can think of this as using a histogram\n",
     "in the place of $p(\\boldsymbol{t})$. If the relative frequency closely\n",
     "resembles $p(\\vec{t})$, then using numerics, it is straight forward to\n",
     "estimate all the interesting parameters of $p(\\boldsymbol{t})$ using point\n",
     "estimators.  \n",
     "\n",
-    "In the case that $\\widehat{\\beta}$ has\n",
+    "In the case that $\\widehat{\\theta}$ has\n",
     "more than one component, and the components are independent, we use the\n",
     "same estimator on each component separately.  If the probability\n",
     "density function of $X_i$, $p(x)$, had been known, then it would have\n",
     "been straight forward to do this by: \n",
     "1. Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \\cdots, X_n^*)$. \n",
     "\n",
-    "2. Then using these numbers, we could compute a replica of $\\widehat{\\beta}$ called $\\widehat{\\beta}^*$. \n",
+    "2. Then using these numbers, we could compute a replica of $\\widehat{\\theta}$ called $\\widehat{\\theta}^*$. \n",
     "\n",
     "By repeated use of (1) and (2), many\n",
-    "estimates of $\\widehat{\\beta}$ could have been obtained. The\n",
-    "idea is to use the relative frequency of $\\widehat{\\beta}^*$\n",
+    "estimates of $\\widehat{\\theta}$ could have been obtained. The\n",
+    "idea is to use the relative frequency of $\\widehat{\\theta}^*$\n",
     "(think of a histogram) as an estimate of $p(\\boldsymbol{t})$.\n",
     "\n",
     "But\n",
@@ -581,19 +581,19 @@
     "\n",
     "2. Define a vector $\\boldsymbol{x}^*$ containing the values which were drawn from $\\boldsymbol{x}$. \n",
     "\n",
-    "3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\beta}^*$ by evaluating $\\widehat \\beta$ under the observations $\\boldsymbol{x}^*$. \n",
+    "3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\theta}^*$ by evaluating $\\widehat \\theta$ under the observations $\\boldsymbol{x}^*$. \n",
     "\n",
     "4. Repeat this process $k$ times. \n",
     "\n",
     "When you are done, you can draw a histogram of the relative frequency\n",
-    "of $\\widehat \\beta^*$. This is your estimate of the probability\n",
+    "of $\\widehat \\theta^*$. This is your estimate of the probability\n",
     "distribution $p(t)$. Using this probability distribution you can\n",
     "estimate any statistics thereof. In principle you never draw the\n",
-    "histogram of the relative frequency of $\\widehat{\\beta}^*$. Instead\n",
+    "histogram of the relative frequency of $\\widehat{\\theta}^*$. Instead\n",
     "you use the estimators corresponding to the statistic of interest. For\n",
     "example, if you are interested in estimating the variance of $\\widehat\n",
-    "\\beta$, apply the estimator $\\widehat \\sigma^2$ to the values\n",
-    "$\\widehat \\beta^*$.\n",
+    "\\theta$, apply the estimator $\\widehat \\sigma^2$ to the values\n",
+    "$\\widehat \\theta^*$.\n",
     "\n",
     "Before we proceed however, we need to remind ourselves about a central\n",
     "theorem in statistics, namely the so-called **central limit theorem**.\n",
@@ -614,7 +614,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fa55ab5a",
+   "id": "d864122f",
    "metadata": {
     "editable": true
    },
@@ -626,7 +626,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "697c0c94",
+   "id": "6750e014",
    "metadata": {
     "editable": true
    },
@@ -641,7 +641,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bd26bbd1",
+   "id": "d5a74550",
    "metadata": {
     "editable": true
    },
@@ -654,7 +654,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "68664e4f",
+   "id": "c44dc3f1",
    "metadata": {
     "editable": true
    },
@@ -669,7 +669,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "742a107a",
+   "id": "202b1bba",
    "metadata": {
     "editable": true
    },
@@ -682,7 +682,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b6d624c1",
+   "id": "da055544",
    "metadata": {
     "editable": true
    },
@@ -693,7 +693,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "46458586",
+   "id": "7773cf22",
    "metadata": {
     "editable": true
    },
@@ -707,7 +707,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ca119461",
+   "id": "8f5e06ca",
    "metadata": {
     "editable": true
    },
@@ -717,7 +717,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f75b40fc",
+   "id": "82157124",
    "metadata": {
     "editable": true
    },
@@ -731,7 +731,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a4b64e20",
+   "id": "2af358ec",
    "metadata": {
     "editable": true
    },
@@ -742,7 +742,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fc76951f",
+   "id": "4d4bafce",
    "metadata": {
     "editable": true
    },
@@ -755,7 +755,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "76979572",
+   "id": "628ec1b1",
    "metadata": {
     "editable": true
    },
@@ -765,7 +765,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3edff7d2",
+   "id": "eb27434b",
    "metadata": {
     "editable": true
    },
@@ -778,7 +778,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "73a9341f",
+   "id": "8dbfd377",
    "metadata": {
     "editable": true
    },
@@ -788,7 +788,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "88f1cc30",
+   "id": "9eda7c0f",
    "metadata": {
     "editable": true
    },
@@ -801,7 +801,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1aec913e",
+   "id": "6c97b3dc",
    "metadata": {
     "editable": true
    },
@@ -822,7 +822,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "42b317e7",
+   "id": "6d316b42",
    "metadata": {
     "editable": true
    },
@@ -835,7 +835,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e5baf71d",
+   "id": "426cae9c",
    "metadata": {
     "editable": true
    },
@@ -847,7 +847,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8e2b39e5",
+   "id": "3445fa66",
    "metadata": {
     "editable": true
    },
@@ -860,7 +860,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "472e7c2c",
+   "id": "fee204e8",
    "metadata": {
     "editable": true
    },
@@ -894,7 +894,7 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "0ff7b796",
+   "id": "b98fbaba",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -933,7 +933,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "22680159",
+   "id": "bc56e30f",
    "metadata": {
     "editable": true
    },
@@ -946,7 +946,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "fe018bb3",
+   "id": "817ebac7",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -966,7 +966,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5bbd613c",
+   "id": "454fba01",
    "metadata": {
     "editable": true
    },
@@ -984,7 +984,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ca3fde4a",
+   "id": "1716dfb2",
    "metadata": {
     "editable": true
    },
@@ -996,7 +996,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8e11f84b",
+   "id": "5295a516",
    "metadata": {
     "editable": true
    },
@@ -1005,27 +1005,27 @@
     "\n",
     "In our derivation of the ordinary least squares method we defined then\n",
     "an approximation to the function $f$ in terms of the parameters\n",
-    "$\\boldsymbol{\\beta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n",
-    "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\beta}$. \n",
+    "$\\boldsymbol{\\theta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n",
+    "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\theta}$. \n",
     "\n",
-    "Thereafter we found the parameters $\\boldsymbol{\\beta}$ by optimizing the means squared error via the so-called cost function"
+    "Thereafter we found the parameters $\\boldsymbol{\\theta}$ by optimizing the means squared error via the so-called cost function"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "026a65c8",
+   "id": "b5953a4f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n",
+    "C(\\boldsymbol{X},\\boldsymbol{\\theta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e59918c7",
+   "id": "ea8014cf",
    "metadata": {
     "editable": true
    },
@@ -1035,7 +1035,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2fd3f73c",
+   "id": "4159a3e3",
    "metadata": {
     "editable": true
    },
@@ -1047,7 +1047,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7daf46c9",
+   "id": "3c18f501",
    "metadata": {
     "editable": true
    },
@@ -1064,7 +1064,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6094266b",
+   "id": "09b0f8b1",
    "metadata": {
     "editable": true
    },
@@ -1076,7 +1076,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "643e0047",
+   "id": "80679145",
    "metadata": {
     "editable": true
    },
@@ -1086,7 +1086,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1319bde5",
+   "id": "1f39a627",
    "metadata": {
     "editable": true
    },
@@ -1098,7 +1098,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9c6d6da1",
+   "id": "b045c821",
    "metadata": {
     "editable": true
    },
@@ -1108,7 +1108,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "855756ef",
+   "id": "15cb1a46",
    "metadata": {
     "editable": true
    },
@@ -1120,7 +1120,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "34d24717",
+   "id": "d0c484eb",
    "metadata": {
     "editable": true
    },
@@ -1131,7 +1131,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "d51b6100",
+   "id": "7940a9b2",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1197,7 +1197,7 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "bd636def",
+   "id": "e9698e1e",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1254,7 +1254,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8cf88b3a",
+   "id": "3edbafad",
    "metadata": {
     "editable": true
    },
@@ -1291,7 +1291,7 @@
   {
    "cell_type": "code",
    "execution_count": 6,
-   "id": "86bfc49a",
+   "id": "5aa48d95",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1374,7 +1374,7 @@
   {
    "cell_type": "code",
    "execution_count": 7,
-   "id": "0c2a183a",
+   "id": "50c5c640",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1463,7 +1463,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2c6c9e89",
+   "id": "947db26f",
    "metadata": {
     "editable": true
    },
@@ -1494,14 +1494,14 @@
   },
   {
    "cell_type": "markdown",
-   "id": "71738b2a",
+   "id": "9974dc92",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "\\boldsymbol{\\beta}_{-i}(\\lambda) & =  ( \\boldsymbol{X}_{-i, \\ast}^{T}\n",
+    "\\boldsymbol{\\theta}_{-i}(\\lambda) & =  ( \\boldsymbol{X}_{-i, \\ast}^{T}\n",
     "\\boldsymbol{X}_{-i, \\ast} + \\lambda \\boldsymbol{I}_{pp})^{-1}\n",
     "\\boldsymbol{X}_{-i, \\ast}^{T} \\boldsymbol{y}_{-i}\n",
     "\\end{align*}\n",
@@ -1510,12 +1510,12 @@
   },
   {
    "cell_type": "markdown",
-   "id": "14db46b6",
+   "id": "45b8c96c",
    "metadata": {
     "editable": true
    },
    "source": [
-    "* Evaluate the prediction performance of these models on the test set by $\\log\\{L[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\beta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n",
+    "* Evaluate the prediction performance of these models on the test set by $\\log\\{L[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\theta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\theta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n",
     "\n",
     "* Repeat the first three steps  such that each sample plays the role of the test set once.\n",
     "\n",
@@ -1524,21 +1524,21 @@
   },
   {
    "cell_type": "markdown",
-   "id": "95e5c8e4",
+   "id": "25b8400a",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "\\frac{1}{n} \\sum_{i = 1}^n \\log\\{L[y_i, \\mathbf{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}.\n",
+    "\\frac{1}{n} \\sum_{i = 1}^n \\log\\{L[y_i, \\mathbf{X}_{i, \\ast}; \\boldsymbol{\\theta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}.\n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7e60f51d",
+   "id": "91b1405d",
    "metadata": {
     "editable": true
    },
@@ -1567,7 +1567,7 @@
   {
    "cell_type": "code",
    "execution_count": 8,
-   "id": "2cef0eb7",
+   "id": "48621261",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1667,7 +1667,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f501c9cf",
+   "id": "2213dc0e",
    "metadata": {
     "editable": true
    },
@@ -1678,7 +1678,7 @@
   {
    "cell_type": "code",
    "execution_count": 9,
-   "id": "30e1e320",
+   "id": "6e7f3e92",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1756,7 +1756,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "383e5c2a",
+   "id": "9c98e1e5",
    "metadata": {
     "editable": true
    },
@@ -1766,7 +1766,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1f9cd409",
+   "id": "b00dd83d",
    "metadata": {
     "editable": true
    },
@@ -1789,8 +1789,8 @@
     "when all predictors are zero (the columns in the design matrix), it\n",
     "may be a bad idea to implement a model which penalizes the intercept.\n",
     "Furthermore, in for example Ridge and Lasso regression, the default solutions\n",
-    "from the library **Scikit-Learn** (when not shrinking $\\beta_0$) for the unknown parameters\n",
-    "$\\boldsymbol{\\beta}$, are derived under the assumption that both $\\boldsymbol{y}$ and\n",
+    "from the library **Scikit-Learn** (when not shrinking $\\theta_0$) for the unknown parameters\n",
+    "$\\boldsymbol{\\theta}$, are derived under the assumption that both $\\boldsymbol{y}$ and\n",
     "$\\boldsymbol{X}$ are zero centered, that is we subtract the mean values.\n",
     "\n",
     "If our predictors represent different scales, then it is important to\n",
@@ -1825,7 +1825,7 @@
   {
    "cell_type": "code",
    "execution_count": 10,
-   "id": "eab81633",
+   "id": "bae38eac",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1852,7 +1852,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0ca51a54",
+   "id": "61bd7f35",
    "metadata": {
     "editable": true
    },
@@ -1866,19 +1866,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fa3a4110",
+   "id": "1a0f8db3",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n",
+    "C(\\theta_0, \\theta_1, ... , \\theta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\theta_0 - \\sum_{j=1}^{p-1} X_{ij}\\theta_j\\right)^2,.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "9f499302",
+   "id": "4b0c41b0",
    "metadata": {
     "editable": true
    },
@@ -1887,49 +1887,49 @@
     "increased penalty for higher differences between predicted and\n",
     "output/target values.\n",
     "\n",
-    "What we have done is to single out the $\\beta_0$ term in the\n",
+    "What we have done is to single out the $\\theta_0$ term in the\n",
     "definition of the mean squared error (MSE).  The design matrix $X$\n",
     "does in this case not contain any intercept column.  When we take the\n",
-    "derivative with respect to $\\beta_0$, we want the derivative to obey"
+    "derivative with respect to $\\theta_0$, we want the derivative to obey"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b1f7ba52",
+   "id": "0aa213f5",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n",
+    "\\frac{\\partial C}{\\partial \\theta_j} = 0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7ff75b7f",
+   "id": "d0808572",
    "metadata": {
     "editable": true
    },
    "source": [
-    "for all $j$. For $\\beta_0$ we have"
+    "for all $j$. For $\\theta_0$ we have"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "eac37d3c",
+   "id": "cf1692d9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n",
+    "\\frac{\\partial C}{\\partial \\theta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\theta_0 - \\sum_{j=1}^{p-1} X_{ij} \\theta_j\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6e653464",
+   "id": "cc8156be",
    "metadata": {
     "editable": true
    },
@@ -1939,42 +1939,42 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8afa60a1",
+   "id": "cd70e62c",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n",
+    "\\sum_{i=0}^{n-1} \\theta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\theta_j.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "055b7975",
+   "id": "92c99c16",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Let us specialize first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n",
-    "Our result for $\\beta_0$ simplifies then to"
+    "Let us specialize first to the case where we have only two parameters $\\theta_0$ and $\\theta_1$.\n",
+    "Our result for $\\theta_0$ simplifies then to"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a2bf6e5d",
+   "id": "cf522258",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n",
+    "n\\theta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\theta_1.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e15afbd5",
+   "id": "cc86b9e1",
    "metadata": {
     "editable": true
    },
@@ -1984,19 +1984,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d2f9a064",
+   "id": "58fd6599",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n",
+    "\\theta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\theta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8e8035d8",
+   "id": "86b6ed8e",
    "metadata": {
     "editable": true
    },
@@ -2006,7 +2006,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0bbcef44",
+   "id": "fac0cd1f",
    "metadata": {
     "editable": true
    },
@@ -2018,7 +2018,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "919462c2",
+   "id": "904fde54",
    "metadata": {
     "editable": true
    },
@@ -2028,7 +2028,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3a0899ec",
+   "id": "21e40e22",
    "metadata": {
     "editable": true
    },
@@ -2040,7 +2040,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9b255ec5",
+   "id": "1fd599b0",
    "metadata": {
     "editable": true
    },
@@ -2050,41 +2050,41 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7716aca1",
+   "id": "69e81981",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0 = \\mu_y - \\beta_1\\mu_{\\boldsymbol{x}_1}.\n",
+    "\\theta_0 = \\mu_y - \\theta_1\\mu_{\\boldsymbol{x}_1}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "24863f48",
+   "id": "b95e7089",
    "metadata": {
     "editable": true
    },
    "source": [
-    "In the general case with more parameters than $\\beta_0$ and $\\beta_1$, we have"
+    "In the general case with more parameters than $\\theta_0$ and $\\theta_1$, we have"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6c10f940",
+   "id": "29fb77c2",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n",
+    "\\theta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\theta_j.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7afefe95",
+   "id": "6c5d2f58",
    "metadata": {
     "editable": true
    },
@@ -2094,19 +2094,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a2e7bc1a",
+   "id": "cbd7ab47",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\beta_j,\n",
+    "\\theta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\theta_j,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5ded7d78",
+   "id": "e1c475ee",
    "metadata": {
     "editable": true
    },
@@ -2116,7 +2116,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bcac7366",
+   "id": "54ae8c6e",
    "metadata": {
     "editable": true
    },
@@ -2128,7 +2128,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bafe8156",
+   "id": "c63feefb",
    "metadata": {
     "editable": true
    },
@@ -2140,41 +2140,41 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7c91d280",
+   "id": "12498a19",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n",
+    "C(\\boldsymbol{\\theta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\theta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\theta}).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1b81d6de",
+   "id": "75c5b8bb",
    "metadata": {
     "editable": true
    },
    "source": [
-    "If we minimize with respect to $\\boldsymbol{\\beta}$ we have then"
+    "If we minimize with respect to $\\boldsymbol{\\theta}$ we have then"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7d6d497e",
+   "id": "da602e6e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n",
+    "\\hat{\\boldsymbol{\\theta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "772593b2",
+   "id": "3cbb6060",
    "metadata": {
     "editable": true
    },
@@ -2182,24 +2182,24 @@
     "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n",
     "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n",
     "\n",
-    "For Ridge regression we need to add $\\lambda \\boldsymbol{\\beta}^T\\boldsymbol{\\beta}$ to the cost function and get then"
+    "For Ridge regression we need to add $\\lambda \\boldsymbol{\\theta}^T\\boldsymbol{\\theta}$ to the cost function and get then"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "3edbdb56",
+   "id": "139ef52e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n",
+    "\\hat{\\boldsymbol{\\theta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7de9723e",
+   "id": "afd4d249",
    "metadata": {
     "editable": true
    },
@@ -2213,7 +2213,7 @@
   {
    "cell_type": "code",
    "execution_count": 11,
-   "id": "2e4a0363",
+   "id": "99a09833",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2233,15 +2233,15 @@
     "    return np.sum((y_data-y_model)**2)/n\n",
     "\n",
     "\n",
-    "def fit_beta(X, y):\n",
+    "def fit_theta(X, y):\n",
     "    return np.linalg.pinv(X.T @ X) @ X.T @ y\n",
     "\n",
     "\n",
-    "true_beta = [2, 0.5, 3.7]\n",
+    "true_theta = [2, 0.5, 3.7]\n",
     "\n",
     "x = np.linspace(0, 1, 11)\n",
     "y = np.sum(\n",
-    "    np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0\n",
+    "    np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0\n",
     ") + 0.1 * np.random.normal(size=len(x))\n",
     "\n",
     "degree = 3\n",
@@ -2251,15 +2251,15 @@
     "for p in range(degree):\n",
     "    X[:, p] = x ** p\n",
     "\n",
-    "beta = fit_beta(X, y)\n",
+    "theta = fit_theta(X, y)\n",
     "\n",
     "# Intercept is included in the design matrix\n",
     "skl = LinearRegression(fit_intercept=False).fit(X, y)\n",
     "\n",
-    "print(f\"True beta: {true_beta}\")\n",
-    "print(f\"Fitted beta: {beta}\")\n",
-    "print(f\"Sklearn fitted beta: {skl.coef_}\")\n",
-    "ypredictOwn = X @ beta\n",
+    "print(f\"True theta: {true_theta}\")\n",
+    "print(f\"Fitted theta: {theta}\")\n",
+    "print(f\"Sklearn fitted theta: {skl.coef_}\")\n",
+    "ypredictOwn = X @ theta\n",
     "ypredictSKL = skl.predict(X)\n",
     "print(f\"MSE with intercept column\")\n",
     "print(MSE(y,ypredictOwn))\n",
@@ -2269,7 +2269,7 @@
     "\n",
     "plt.figure()\n",
     "plt.scatter(x, y, label=\"Data\")\n",
-    "plt.plot(x, X @ beta, label=\"Fit\")\n",
+    "plt.plot(x, X @ theta, label=\"Fit\")\n",
     "plt.plot(x, skl.predict(X), label=\"Sklearn (fit_intercept=False)\")\n",
     "\n",
     "\n",
@@ -2286,21 +2286,21 @@
     "y_offset = np.average(y, axis=0)\n",
     "X_offset = np.average(X, axis=0)\n",
     "\n",
-    "beta = fit_beta(X - X_offset, y - y_offset)\n",
-    "intercept = np.mean(y_offset - X_offset @ beta)\n",
+    "theta = fit_theta(X - X_offset, y - y_offset)\n",
+    "intercept = np.mean(y_offset - X_offset @ theta)\n",
     "\n",
     "print(f\"Manual intercept: {intercept}\")\n",
-    "print(f\"Fitted beta (wiothout intercept): {beta}\")\n",
+    "print(f\"Fitted theta (wiothout intercept): {theta}\")\n",
     "print(f\"Sklearn intercept: {skl.intercept_}\")\n",
-    "print(f\"Sklearn fitted beta (without intercept): {skl.coef_}\")\n",
-    "ypredictOwn = X @ beta\n",
+    "print(f\"Sklearn fitted theta (without intercept): {skl.coef_}\")\n",
+    "ypredictOwn = X @ theta\n",
     "ypredictSKL = skl.predict(X)\n",
     "print(f\"MSE with Manual intercept\")\n",
     "print(MSE(y,ypredictOwn+intercept))\n",
     "print(f\"MSE with Sklearn intercept\")\n",
     "print(MSE(y,ypredictSKL))\n",
     "\n",
-    "plt.plot(x, X @ beta + intercept, \"--\", label=\"Fit (manual intercept)\")\n",
+    "plt.plot(x, X @ theta + intercept, \"--\", label=\"Fit (manual intercept)\")\n",
     "plt.plot(x, skl.predict(X), \"--\", label=\"Sklearn (fit_intercept=True)\")\n",
     "plt.grid()\n",
     "plt.legend()\n",
@@ -2310,7 +2310,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c6d3a071",
+   "id": "1bacfcef",
    "metadata": {
     "editable": true
    },
@@ -2323,7 +2323,7 @@
     "the way we treat the intercept may give a larger or smaller MSE,\n",
     "meaning that the MSE can be penalized by the value of the\n",
     "intercept. Not including the intercept in the fit, means that the\n",
-    "regularization term does not include $\\beta_0$. For different values\n",
+    "regularization term does not include $\\theta_0$. For different values\n",
     "of $\\lambda$, this may lead to different MSE values. \n",
     "\n",
     "To remind the reader, the regularization term, with the intercept in Ridge regression, is given by"
@@ -2331,19 +2331,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "db803b77",
+   "id": "8061073e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\beta_j^2,\n",
+    "\\lambda \\vert\\vert \\boldsymbol{\\theta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\theta_j^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d7a9c9d8",
+   "id": "fcea892f",
    "metadata": {
     "editable": true
    },
@@ -2353,19 +2353,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e08af763",
+   "id": "af6d400e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\beta_j^2.\n",
+    "\\lambda \\vert\\vert \\boldsymbol{\\theta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\theta_j^2.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "10fe24e4",
+   "id": "1bfb0c9e",
    "metadata": {
     "editable": true
    },
@@ -2375,19 +2375,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6feb3bdd",
+   "id": "3b87c68e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\beta_j\\vert.\n",
+    "\\lambda \\vert\\vert \\boldsymbol{\\theta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\theta_j\\vert.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "657a0777",
+   "id": "eae8012d",
    "metadata": {
     "editable": true
    },
@@ -2405,7 +2405,7 @@
   {
    "cell_type": "code",
    "execution_count": 12,
-   "id": "9f21317d",
+   "id": "e02c5a51",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2448,20 +2448,20 @@
     "lambdas = np.logspace(-4, 2, nlambdas)\n",
     "for i in range(nlambdas):\n",
     "    lmb = lambdas[i]\n",
-    "    OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
+    "    OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
     "    # Note: we include the intercept column and no scaling\n",
     "    RegRidge = linear_model.Ridge(lmb,fit_intercept=False)\n",
     "    RegRidge.fit(X_train,y_train)\n",
     "    # and then make the prediction\n",
-    "    ytildeOwnRidge = X_train @ OwnRidgeBeta\n",
-    "    ypredictOwnRidge = X_test @ OwnRidgeBeta\n",
+    "    ytildeOwnRidge = X_train @ OwnRidgeTheta\n",
+    "    ypredictOwnRidge = X_test @ OwnRidgeTheta\n",
     "    ytildeRidge = RegRidge.predict(X_train)\n",
     "    ypredictRidge = RegRidge.predict(X_test)\n",
     "    MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)\n",
     "    MSERidgePredict[i] = MSE(y_test,ypredictRidge)\n",
-    "    print(\"Beta values for own Ridge implementation\")\n",
-    "    print(OwnRidgeBeta)\n",
-    "    print(\"Beta values for Scikit-Learn Ridge implementation\")\n",
+    "    print(\"Theta values for own Ridge implementation\")\n",
+    "    print(OwnRidgeTheta)\n",
+    "    print(\"Theta values for Scikit-Learn Ridge implementation\")\n",
     "    print(RegRidge.coef_)\n",
     "    print(\"MSE values for own Ridge implementation\")\n",
     "    print(MSEOwnRidgePredict[i])\n",
@@ -2481,7 +2481,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "20de82fd",
+   "id": "df4ec33b",
    "metadata": {
     "editable": true
    },
@@ -2495,7 +2495,7 @@
   {
    "cell_type": "code",
    "execution_count": 13,
-   "id": "4138ed50",
+   "id": "019f59bd",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2549,18 +2549,18 @@
     "lambdas = np.logspace(-4, 2, nlambdas)\n",
     "for i in range(nlambdas):\n",
     "    lmb = lambdas[i]\n",
-    "    OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)\n",
-    "    intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data\n",
+    "    OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)\n",
+    "    intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data\n",
     "    #Add intercept to prediction\n",
-    "    ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler \n",
+    "    ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler \n",
     "    RegRidge = linear_model.Ridge(lmb)\n",
     "    RegRidge.fit(X_train,y_train)\n",
     "    ypredictRidge = RegRidge.predict(X_test)\n",
     "    MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)\n",
     "    MSERidgePredict[i] = MSE(y_test,ypredictRidge)\n",
-    "    print(\"Beta values for own Ridge implementation\")\n",
-    "    print(OwnRidgeBeta) #Intercept is given by mean of target variable\n",
-    "    print(\"Beta values for Scikit-Learn Ridge implementation\")\n",
+    "    print(\"Theta values for own Ridge implementation\")\n",
+    "    print(OwnRidgeTheta) #Intercept is given by mean of target variable\n",
+    "    print(\"Theta values for Scikit-Learn Ridge implementation\")\n",
     "    print(RegRidge.coef_)\n",
     "    print('Intercept from own implementation:')\n",
     "    print(intercept_)\n",
@@ -2584,7 +2584,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e9dfda5",
+   "id": "11a0c8f8",
    "metadata": {
     "editable": true
    },
@@ -2592,7 +2592,7 @@
     "We see here, when compared to the code which includes explicitely the\n",
     "intercept column, that our MSE value is actually smaller. This is\n",
     "because the regularization term does not include the intercept value\n",
-    "$\\beta_0$ in the fitting.  This applies to Lasso regularization as\n",
+    "$\\theta_0$ in the fitting.  This applies to Lasso regularization as\n",
     "well.  It means that our optimization is now done only with the\n",
     "centered matrix and/or vector that enter the fitting procedure. Note\n",
     "also that the problem with the intercept occurs mainly in these type\n",
@@ -2603,7 +2603,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a7eb252c",
+   "id": "fca71b1a",
    "metadata": {
     "editable": true
    },
@@ -2616,7 +2616,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "990ff280",
+   "id": "bea82853",
    "metadata": {
     "editable": true
    },
@@ -2634,7 +2634,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5ea46b31",
+   "id": "d92e0742",
    "metadata": {
     "editable": true
    },
@@ -2651,7 +2651,7 @@
   {
    "cell_type": "code",
    "execution_count": 14,
-   "id": "fa60aaab",
+   "id": "039e1f8c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2682,7 +2682,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b9e8214e",
+   "id": "94a8e6b6",
    "metadata": {
     "editable": true
    },
@@ -2698,7 +2698,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a578cd63",
+   "id": "0ba92480",
    "metadata": {
     "editable": true
    },
@@ -2716,7 +2716,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d90dcfee",
+   "id": "313c4354",
    "metadata": {
     "editable": true
    },
@@ -2728,7 +2728,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c4eec69a",
+   "id": "da5f96ae",
    "metadata": {
     "editable": true
    },
@@ -2746,7 +2746,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae46a11b",
+   "id": "14375be5",
    "metadata": {
     "editable": true
    },
@@ -2758,7 +2758,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "87ebb9b8",
+   "id": "58f91570",
    "metadata": {
     "editable": true
    },
@@ -2768,7 +2768,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "    \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon},\n",
+    "    \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\theta} + \\boldsymbol{\\epsilon},\n",
     "\\label{_auto4} \\tag{4}\n",
     "\\end{equation}\n",
     "$$"
@@ -2776,7 +2776,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "856efd3f",
+   "id": "820108ea",
    "metadata": {
     "editable": true
    },
@@ -2787,7 +2787,7 @@
   {
    "cell_type": "code",
    "execution_count": 15,
-   "id": "5ae2828f",
+   "id": "ec4efa81",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2803,7 +2803,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0d6201d9",
+   "id": "aa3656f3",
    "metadata": {
     "editable": true
    },
@@ -2813,7 +2813,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4af380a7",
+   "id": "0c1bbe7c",
    "metadata": {
     "editable": true
    },
@@ -2823,7 +2823,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "    C(\\boldsymbol{X}, \\boldsymbol{\\beta})= \\frac{1}{n}\\left\\{(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})\\right\\}.\n",
+    "    C(\\boldsymbol{X}, \\boldsymbol{\\theta})= \\frac{1}{n}\\left\\{(\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y})\\right\\}.\n",
     "\\label{_auto5} \\tag{5}\n",
     "\\end{equation}\n",
     "$$"
@@ -2831,30 +2831,30 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8e8eefa3",
+   "id": "784bc4e4",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We then find the extremal point of $C$ by taking the derivative with respect to $\\boldsymbol{\\beta}$ as discussed above.\n",
-    "This yields the expression for $\\boldsymbol{\\beta}$ to be"
+    "We then find the extremal point of $C$ by taking the derivative with respect to $\\boldsymbol{\\theta}$ as discussed above.\n",
+    "This yields the expression for $\\boldsymbol{\\theta}$ to be"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a1c0765c",
+   "id": "db1ec5fe",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta} = \\frac{\\boldsymbol{X}^T \\boldsymbol{y}}{\\boldsymbol{X}^T \\boldsymbol{X}},\n",
+    "\\boldsymbol{\\theta} = \\frac{\\boldsymbol{X}^T \\boldsymbol{y}}{\\boldsymbol{X}^T \\boldsymbol{X}},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c754e28b",
+   "id": "5d2e6b17",
    "metadata": {
     "editable": true
    },
@@ -2868,7 +2868,7 @@
   {
    "cell_type": "code",
    "execution_count": 16,
-   "id": "ff056cf9",
+   "id": "6a242f2e",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2887,7 +2887,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "524f259e",
+   "id": "329b50d2",
    "metadata": {
     "editable": true
    },
@@ -2895,24 +2895,24 @@
     "Doing the inversion directly turns out to be a bad idea since the matrix\n",
     "$\\boldsymbol{X}^T\\boldsymbol{X}$ is singular. An alternative approach is to use the **singular\n",
     "value decomposition**. Using the definition of the Moore-Penrose\n",
-    "pseudoinverse we can write the equation for $\\boldsymbol{\\beta}$ as"
+    "pseudoinverse we can write the equation for $\\boldsymbol{\\theta}$ as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "52be4553",
+   "id": "2b544864",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta} = \\boldsymbol{X}^{+}\\boldsymbol{y},\n",
+    "\\boldsymbol{\\theta} = \\boldsymbol{X}^{+}\\boldsymbol{y},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c37ff09a",
+   "id": "bf0d128d",
    "metadata": {
     "editable": true
    },
@@ -2922,7 +2922,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ea49a105",
+   "id": "67e527de",
    "metadata": {
     "editable": true
    },
@@ -2934,7 +2934,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bf31067a",
+   "id": "d4ef2ece",
    "metadata": {
     "editable": true
    },
@@ -2947,7 +2947,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5ef2c839",
+   "id": "c4c467f8",
    "metadata": {
     "editable": true
    },
@@ -2957,7 +2957,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "    \\boldsymbol{\\beta} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^{+} \\boldsymbol{U}^T \\boldsymbol{y}.\n",
+    "    \\boldsymbol{\\theta} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^{+} \\boldsymbol{U}^T \\boldsymbol{y}.\n",
     "\\label{_auto6} \\tag{6}\n",
     "\\end{equation}\n",
     "$$"
@@ -2965,7 +2965,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "96bbf134",
+   "id": "ffd1ba3d",
    "metadata": {
     "editable": true
    },
@@ -2981,7 +2981,7 @@
   {
    "cell_type": "code",
    "execution_count": 17,
-   "id": "5e729b1e",
+   "id": "9cdf3698",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2996,19 +2996,19 @@
   {
    "cell_type": "code",
    "execution_count": 18,
-   "id": "80b923d1",
+   "id": "164ccc0c",
    "metadata": {
     "collapsed": false,
     "editable": true
    },
    "outputs": [],
    "source": [
-    "beta = ols_svd(X_train_own,y_train)"
+    "theta = ols_svd(X_train_own,y_train)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b4cf2f1a",
+   "id": "a23f2826",
    "metadata": {
     "editable": true
    },
@@ -3019,19 +3019,19 @@
   {
    "cell_type": "code",
    "execution_count": 19,
-   "id": "93be2c0d",
+   "id": "a164aca2",
    "metadata": {
     "collapsed": false,
     "editable": true
    },
    "outputs": [],
    "source": [
-    "J = beta[1:].reshape(L, L)"
+    "J = theta[1:].reshape(L, L)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ae869089",
+   "id": "3bb35ae1",
    "metadata": {
     "editable": true
    },
@@ -3042,7 +3042,7 @@
   {
    "cell_type": "code",
    "execution_count": 20,
-   "id": "8d6d8152",
+   "id": "93445ba4",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3061,7 +3061,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8092ac84",
+   "id": "f602ed47",
    "metadata": {
     "editable": true
    },
@@ -3084,7 +3084,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d9251cde",
+   "id": "86d25ab0",
    "metadata": {
     "editable": true
    },
@@ -3102,7 +3102,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "abe454c9",
+   "id": "2008a25b",
    "metadata": {
     "editable": true
    },
@@ -3115,7 +3115,7 @@
   {
    "cell_type": "code",
    "execution_count": 21,
-   "id": "89f8fbd0",
+   "id": "01b2b75a",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3147,7 +3147,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e06639d4",
+   "id": "d6b5a067",
    "metadata": {
     "editable": true
    },
@@ -3157,7 +3157,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "43ef0f40",
+   "id": "497d84f1",
    "metadata": {
     "editable": true
    },
@@ -3175,7 +3175,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "523dc03b",
+   "id": "38cc642a",
    "metadata": {
     "editable": true
    },
@@ -3187,7 +3187,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0f389eef",
+   "id": "bec7f36d",
    "metadata": {
     "editable": true
    },
@@ -3205,7 +3205,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b6e00f24",
+   "id": "14ad8f62",
    "metadata": {
     "editable": true
    },
@@ -3217,7 +3217,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5ea1084d",
+   "id": "4ccac4ea",
    "metadata": {
     "editable": true
    },
@@ -3227,7 +3227,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "    \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon}.\n",
+    "    \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\theta} + \\boldsymbol{\\epsilon}.\n",
     "\\label{_auto10} \\tag{10}\n",
     "\\end{equation}\n",
     "$$"
@@ -3235,7 +3235,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5551bb05",
+   "id": "cabb6ced",
    "metadata": {
     "editable": true
    },
@@ -3246,7 +3246,7 @@
   {
    "cell_type": "code",
    "execution_count": 22,
-   "id": "f5dd7795",
+   "id": "ce90ccbe",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3272,7 +3272,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f36c807b",
+   "id": "6f6d672f",
    "metadata": {
     "editable": true
    },
@@ -3283,7 +3283,7 @@
   {
    "cell_type": "code",
    "execution_count": 23,
-   "id": "eb701dd4",
+   "id": "945777f9",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3295,7 +3295,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bd356ada",
+   "id": "ffc8317f",
    "metadata": {
     "editable": true
    },
@@ -3306,7 +3306,7 @@
   {
    "cell_type": "code",
    "execution_count": 24,
-   "id": "ae737db8",
+   "id": "e17c598f",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3318,7 +3318,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d152a32a",
+   "id": "3396f010",
    "metadata": {
     "editable": true
    },
@@ -3329,7 +3329,7 @@
   {
    "cell_type": "code",
    "execution_count": 25,
-   "id": "8d713977",
+   "id": "311d3cea",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3348,7 +3348,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ede30c18",
+   "id": "13705eed",
    "metadata": {
     "editable": true
    },
@@ -3358,13 +3358,13 @@
     "Having explored the ordinary least squares we move on to ridge\n",
     "regression. In ridge regression we include a **regularizer**. This\n",
     "involves a new cost function which leads to a new estimate for the\n",
-    "weights $\\boldsymbol{\\beta}$. This results in a penalized regression problem. The\n",
+    "weights $\\boldsymbol{\\theta}$. This results in a penalized regression problem. The\n",
     "cost function is given by"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "eac772e9",
+   "id": "4cd9530c",
    "metadata": {
     "editable": true
    },
@@ -3374,7 +3374,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "    C(\\boldsymbol{X}, \\boldsymbol{\\beta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y}) + \\lambda \\boldsymbol{\\beta}^T\\boldsymbol{\\beta}.\n",
+    "    C(\\boldsymbol{X}, \\boldsymbol{\\theta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y}) + \\lambda \\boldsymbol{\\theta}^T\\boldsymbol{\\theta}.\n",
     "\\label{_auto11} \\tag{11}\n",
     "\\end{equation}\n",
     "$$"
@@ -3383,7 +3383,7 @@
   {
    "cell_type": "code",
    "execution_count": 26,
-   "id": "b66675f0",
+   "id": "d1c9a462",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3406,7 +3406,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "71aa37b5",
+   "id": "97e6303c",
    "metadata": {
     "editable": true
    },
@@ -3416,7 +3416,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8b70ef66",
+   "id": "8631d307",
    "metadata": {
     "editable": true
    },
@@ -3426,7 +3426,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "    C(\\boldsymbol{X}, \\boldsymbol{\\beta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y}) + \\lambda \\sqrt{\\boldsymbol{\\beta}^T\\boldsymbol{\\beta}}.\n",
+    "    C(\\boldsymbol{X}, \\boldsymbol{\\theta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y}) + \\lambda \\sqrt{\\boldsymbol{\\theta}^T\\boldsymbol{\\theta}}.\n",
     "\\label{_auto12} \\tag{12}\n",
     "\\end{equation}\n",
     "$$"
@@ -3434,7 +3434,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "079981ff",
+   "id": "dc2a06fb",
    "metadata": {
     "editable": true
    },
@@ -3445,7 +3445,7 @@
   {
    "cell_type": "code",
    "execution_count": 27,
-   "id": "c8d4a180",
+   "id": "2f3333c6",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3467,7 +3467,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "57839941",
+   "id": "692844ac",
    "metadata": {
     "editable": true
    },
@@ -3482,7 +3482,7 @@
   {
    "cell_type": "code",
    "execution_count": 28,
-   "id": "0e3bbb4e",
+   "id": "d40c4d10",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3529,7 +3529,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "14711abd",
+   "id": "965b51d2",
    "metadata": {
     "editable": true
    },
@@ -3550,7 +3550,7 @@
   {
    "cell_type": "code",
    "execution_count": 29,
-   "id": "9ff29a72",
+   "id": "04fe5f87",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3591,7 +3591,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9c10be56",
+   "id": "845d9279",
    "metadata": {
     "editable": true
    },
@@ -3603,7 +3603,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "efce3b63",
+   "id": "6d8674ba",
    "metadata": {
     "editable": true
    },
@@ -3629,7 +3629,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ef5a4df0",
+   "id": "b26b187d",
    "metadata": {
     "editable": true
    },
@@ -3644,7 +3644,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b085eff4",
+   "id": "76e8040d",
    "metadata": {
     "editable": true
    },
@@ -3674,7 +3674,7 @@
   {
    "cell_type": "code",
    "execution_count": 30,
-   "id": "5056dccb",
+   "id": "7608edfe",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3724,7 +3724,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f5e72aef",
+   "id": "c0d0b45f",
    "metadata": {
     "editable": true
    },
@@ -3741,13 +3741,13 @@
     "value decomposition from e.g., **numpy** ) or use your code from\n",
     "homeworks 1 and 2 and perform a standard least square regression\n",
     "analysis using polynomials in $x$ and $y$ up to fifth order. Find the\n",
-    "[confidence intervals](https://en.wikipedia.org/wiki/Confidence_interval) of the parameters (estimators) $\\beta$ by computing their\n",
+    "[confidence intervals](https://en.wikipedia.org/wiki/Confidence_interval) of the parameters (estimators) $\\theta$ by computing their\n",
     "variances, evaluate the Mean Squared error (MSE)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "0ba33237",
+   "id": "76861250",
    "metadata": {
     "editable": true
    },
@@ -3760,7 +3760,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1f9af2ef",
+   "id": "90074ad8",
    "metadata": {
     "editable": true
    },
@@ -3772,7 +3772,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "77d5b0f2",
+   "id": "31d7a9ad",
    "metadata": {
     "editable": true
    },
@@ -3784,7 +3784,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6c1f9df9",
+   "id": "b6bcc7b7",
    "metadata": {
     "editable": true
    },
@@ -3794,7 +3794,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1d06102c",
+   "id": "45f3893f",
    "metadata": {
     "editable": true
    },
@@ -3806,7 +3806,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8fb40b08",
+   "id": "8fea6a0a",
    "metadata": {
     "editable": true
    },
@@ -3831,7 +3831,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1701de47",
+   "id": "6283b437",
    "metadata": {
     "editable": true
    },
@@ -3863,7 +3863,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "15110cdf",
+   "id": "62f77e00",
    "metadata": {
     "editable": true
    },
@@ -3875,7 +3875,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "33046595",
+   "id": "7380e759",
    "metadata": {
     "editable": true
    },
@@ -3885,28 +3885,28 @@
     "\n",
     "In our derivation of the ordinary least squares method we defined then\n",
     "an approximation to the function $f$ in terms of the parameters\n",
-    "$\\boldsymbol{\\beta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n",
-    "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\beta}$.\n",
+    "$\\boldsymbol{\\theta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n",
+    "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\theta}$.\n",
     "\n",
-    "The parameters $\\boldsymbol{\\beta}$ are in turn found by optimizing the means\n",
+    "The parameters $\\boldsymbol{\\theta}$ are in turn found by optimizing the means\n",
     "squared error via the so-called cost function"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "84527747",
+   "id": "b5c040e0",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n",
+    "C(\\boldsymbol{X},\\boldsymbol{\\theta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "da2f876d",
+   "id": "2cf7f08f",
    "metadata": {
     "editable": true
    },
@@ -3918,7 +3918,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1845a7dc",
+   "id": "8cf351f9",
    "metadata": {
     "editable": true
    },
@@ -3930,7 +3930,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d2f1d7d7",
+   "id": "53fff2ad",
    "metadata": {
     "editable": true
    },
@@ -3950,7 +3950,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e898c902",
+   "id": "89dec1fb",
    "metadata": {
     "editable": true
    },
@@ -3975,7 +3975,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d23dc734",
+   "id": "2d5f7364",
    "metadata": {
     "editable": true
    },
@@ -3995,7 +3995,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "39a35330",
+   "id": "d56e6dc8",
    "metadata": {
     "editable": true
    },
@@ -4012,7 +4012,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "44d9e821",
+   "id": "608453fd",
    "metadata": {
     "editable": true
    },
@@ -4040,7 +4040,7 @@
   {
    "cell_type": "code",
    "execution_count": 31,
-   "id": "0a4e6d7e",
+   "id": "9399d087",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4052,7 +4052,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "16a73292",
+   "id": "ae1d9c8f",
    "metadata": {
     "editable": true
    },
@@ -4064,7 +4064,7 @@
   {
    "cell_type": "code",
    "execution_count": 32,
-   "id": "168356a4",
+   "id": "381e9da0",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4092,7 +4092,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e9ec3e68",
+   "id": "6e1c4af1",
    "metadata": {
     "editable": true
    },
diff --git a/doc/LectureNotes/_build/jupyter_execute/chapter4.ipynb b/doc/LectureNotes/_build/jupyter_execute/chapter4.ipynb
index 33b1e76fa..51a85e51d 100644
--- a/doc/LectureNotes/_build/jupyter_execute/chapter4.ipynb
+++ b/doc/LectureNotes/_build/jupyter_execute/chapter4.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "f63b8c1a",
+   "id": "225f0ca7",
    "metadata": {
     "editable": true
    },
@@ -13,7 +13,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9a284cf5",
+   "id": "a48a2920",
    "metadata": {
     "editable": true
    },
@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d78b73e2",
+   "id": "90b6e822",
    "metadata": {
     "editable": true
    },
@@ -38,7 +38,7 @@
     "analytical expressions for standard ordinary Least Squares or Ridge\n",
     "regression (in terms of matrices to invert) for several quantities,\n",
     "ranging from the variance and thereby the confidence intervals of the\n",
-    "optimal parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n",
+    "optimal parameters $\\hat{\\theta}$ to the mean squared error. If we can invert\n",
     "the product of the design matrices, linear regression gives then a\n",
     "simple recipe for fitting our data.\n",
     "\n",
@@ -61,7 +61,7 @@
     "Logistic regression will also serve as our stepping stone towards\n",
     "neural network algorithms and supervised deep learning. For logistic\n",
     "learning, the minimization of the cost function leads to a non-linear\n",
-    "equation in the parameters $\\hat{\\beta}$. The optimization of the\n",
+    "equation in the parameters $\\hat{\\theta}$. The optimization of the\n",
     "problem calls therefore for minimization algorithms. This forms the\n",
     "bottle neck of all machine learning algorithms, namely how to find\n",
     "reliable minima of a multi-variable function. This leads us to the\n",
@@ -75,7 +75,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d0c829b4",
+   "id": "08ebdf8e",
    "metadata": {
     "editable": true
    },
@@ -100,7 +100,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f1f4d5f2",
+   "id": "f08eca95",
    "metadata": {
     "editable": true
    },
@@ -112,7 +112,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8e2f836b",
+   "id": "d82a2a29",
    "metadata": {
     "editable": true
    },
@@ -128,7 +128,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6b58f1ba",
+   "id": "467f1d23",
    "metadata": {
     "editable": true
    },
@@ -138,7 +138,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\beta} +  \\boldsymbol{\\epsilon},\n",
+    "\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\theta} +  \\boldsymbol{\\epsilon},\n",
     "\\label{_auto1} \\tag{1}\n",
     "\\end{equation}\n",
     "$$"
@@ -146,13 +146,13 @@
   },
   {
    "cell_type": "markdown",
-   "id": "54e3604b",
+   "id": "0f2560d0",
    "metadata": {
     "editable": true
    },
    "source": [
     "where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n",
-    "$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors.\n",
+    "$n\\times p$ design matrix and $\\boldsymbol{\\theta}$ represents our estimators/predictors.\n",
     "\n",
     "The main problem with our function is that it takes values on the\n",
     "entire real axis. In the case of logistic regression, however, the\n",
@@ -175,7 +175,7 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "cd1dd1c5",
+   "id": "3916f494",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -242,7 +242,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2e06ee82",
+   "id": "df52e736",
    "metadata": {
     "editable": true
    },
@@ -253,7 +253,7 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "a774f001",
+   "id": "07b5c0a7",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -272,7 +272,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e102fa56",
+   "id": "25ce17fc",
    "metadata": {
     "editable": true
    },
@@ -283,19 +283,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "75d995ba",
+   "id": "818e2f55",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "f(y_i\\vert x_i)=\\beta_0+\\beta_1 x_i.\n",
+    "f(y_i\\vert x_i)=\\theta_0+\\theta_1 x_i.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e2cacca5",
+   "id": "b2616b20",
    "metadata": {
     "editable": true
    },
@@ -314,7 +314,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "af9ac586",
+   "id": "ab85db3c",
    "metadata": {
     "editable": true
    },
@@ -336,7 +336,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "80aa3cbe",
+   "id": "efbd4772",
    "metadata": {
     "editable": true
    },
@@ -348,7 +348,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "28d49915",
+   "id": "1b9231ea",
    "metadata": {
     "editable": true
    },
@@ -358,7 +358,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eeff6cd9",
+   "id": "ad6acb90",
    "metadata": {
     "editable": true
    },
@@ -371,7 +371,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "5ecec05e",
+   "id": "bf66d0dd",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -436,56 +436,56 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5802efe3",
+   "id": "363760d3",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\beta$ in our fitting of the Sigmoid function, that is we define probabilities"
+    "We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\theta$ in our fitting of the Sigmoid function, that is we define probabilities"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f85d14ea",
+   "id": "eefe7b8d",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "p(y_i=1|x_i,\\boldsymbol{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
-    "p(y_i=0|x_i,\\boldsymbol{\\beta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\beta}),\n",
+    "p(y_i=1|x_i,\\boldsymbol{\\theta}) &= \\frac{\\exp{(\\theta_0+\\theta_1x_i)}}{1+\\exp{(\\theta_0+\\theta_1x_i)}},\\nonumber\\\\\n",
+    "p(y_i=0|x_i,\\boldsymbol{\\theta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\theta}),\n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6c4e0334",
+   "id": "102fb347",
    "metadata": {
     "editable": true
    },
    "source": [
-    "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
+    "where $\\boldsymbol{\\theta}$ are the weights we wish to extract from data, in our case $\\theta_0$ and $\\theta_1$. \n",
     "\n",
     "Note that we used"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "df7facc9",
+   "id": "0b230504",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n",
+    "p(y_i=0\\vert x_i, \\boldsymbol{\\theta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\theta}).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a920e6d8",
+   "id": "669b4c9f",
    "metadata": {
     "editable": true
    },
@@ -500,21 +500,21 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a780bfe1",
+   "id": "5dc36460",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "P(\\mathcal{D}|\\boldsymbol{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n",
+    "P(\\mathcal{D}|\\boldsymbol{\\theta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\theta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\theta}))\\right]^{1-y_i}\\nonumber \\\\\n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "9f810a44",
+   "id": "0874a9c6",
    "metadata": {
     "editable": true
    },
@@ -524,19 +524,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3fac4ef4",
+   "id": "212397d3",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n",
+    "\\mathcal{C}(\\boldsymbol{\\theta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\theta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\theta}))\\right]\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "75aa2c14",
+   "id": "f99396bf",
    "metadata": {
     "editable": true
    },
@@ -546,42 +546,42 @@
   },
   {
    "cell_type": "markdown",
-   "id": "780f2038",
+   "id": "fd5e7b5f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n  \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
+    "\\mathcal{C}(\\boldsymbol{\\theta}) = \\sum_{i=1}^n  \\left(y_i(\\theta_0+\\theta_1x_i) -\\log{(1+\\exp{(\\theta_0+\\theta_1x_i)})}\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c8c940aa",
+   "id": "80dbca63",
    "metadata": {
     "editable": true
    },
    "source": [
-    "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n",
+    "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\theta$.\n",
     "Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "9d4a527b",
+   "id": "f5d6c58f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n  \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
+    "\\mathcal{C}(\\boldsymbol{\\theta})=-\\sum_{i=1}^n  \\left(y_i(\\theta_0+\\theta_1x_i) -\\log{(1+\\exp{(\\theta_0+\\theta_1x_i)})}\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c1706606",
+   "id": "57e33c81",
    "metadata": {
     "editable": true
    },
@@ -589,28 +589,28 @@
     "This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n",
     "in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.\n",
     "\n",
-    "The cross entropy is a convex function of the weights $\\boldsymbol{\\beta}$ and,\n",
+    "The cross entropy is a convex function of the weights $\\boldsymbol{\\theta}$ and,\n",
     "therefore, any local minimizer is a global minimizer. \n",
     "\n",
     "Minimizing this\n",
-    "cost function with respect to the two parameters $\\beta_0$ and $\\beta_1$ we obtain"
+    "cost function with respect to the two parameters $\\theta_0$ and $\\theta_1$ we obtain"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "fe8fb387",
+   "id": "1833cc5d",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n  \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
+    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\theta_0} = -\\sum_{i=1}^n  \\left(y_i -\\frac{\\exp{(\\theta_0+\\theta_1x_i)}}{1+\\exp{(\\theta_0+\\theta_1x_i)}}\\right),\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "bb564e81",
+   "id": "ea34e8e6",
    "metadata": {
     "editable": true
    },
@@ -620,67 +620,67 @@
   },
   {
    "cell_type": "markdown",
-   "id": "700c5443",
+   "id": "f1bc373e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n  \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
+    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\theta_1} = -\\sum_{i=1}^n  \\left(y_ix_i -x_i\\frac{\\exp{(\\theta_0+\\theta_1x_i)}}{1+\\exp{(\\theta_0+\\theta_1x_i)}}\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f442c956",
+   "id": "617c6d86",
    "metadata": {
     "editable": true
    },
    "source": [
     "Let us now define a vector $\\boldsymbol{y}$ with $n$ elements $y_i$, an\n",
     "$n\\times p$ matrix $\\boldsymbol{X}$ which contains the $x_i$ values and a\n",
-    "vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\beta})$. We can rewrite in a more compact form the first\n",
+    "vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\theta})$. We can rewrite in a more compact form the first\n",
     "derivative of cost function as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "83937f6f",
+   "id": "e79426c6",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
+    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "71819c1d",
+   "id": "b1dd9451",
    "metadata": {
     "editable": true
    },
    "source": [
     "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
-    "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
+    "$p(y_i\\vert x_i,\\boldsymbol{\\theta})(1-p(y_i\\vert x_i,\\boldsymbol{\\theta})$, we can obtain a compact expression of the second derivative as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "23dfd975",
+   "id": "ac6500cc",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
+    "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}\\partial \\boldsymbol{\\theta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6ca9f74b",
+   "id": "12f4d5d0",
    "metadata": {
     "editable": true
    },
@@ -690,41 +690,41 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ddb3d93a",
+   "id": "b4733616",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
+    "\\log{ \\frac{p(\\boldsymbol{\\theta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\theta}\\boldsymbol{x})}} = \\theta_0+\\theta_1x_1+\\theta_2x_2+\\dots+\\theta_px_p.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "0cfae560",
+   "id": "eec7d85e",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
+    "Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\theta}=[\\theta_0, \\theta_1, \\dots, \\theta_p]$ leading to"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7c427fc3",
+   "id": "152e6fa3",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
+    "p(\\boldsymbol{\\theta}\\boldsymbol{x})=\\frac{ \\exp{(\\theta_0+\\theta_1x_1+\\theta_2x_2+\\dots+\\theta_px_p)}}{1+\\exp{(\\theta_0+\\theta_1x_1+\\theta_2x_2+\\dots+\\theta_px_p)}}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6a6814bf",
+   "id": "b8168ae0",
    "metadata": {
     "editable": true
    },
@@ -736,19 +736,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "85282137",
+   "id": "3af519ee",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\beta_{10}+\\beta_{11}x_1,\n",
+    "\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\theta_{10}+\\theta_{11}x_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4864f76d",
+   "id": "fa02c958",
    "metadata": {
     "editable": true
    },
@@ -758,19 +758,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "390e9a55",
+   "id": "c550e015",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n",
+    "\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\theta_{20}+\\theta_{21}x_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c3f39c44",
+   "id": "8398ec7a",
    "metadata": {
     "editable": true
    },
@@ -780,19 +780,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "92f32a03",
+   "id": "6be24150",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n",
+    "\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\theta_{(K-1)0}+\\theta_{(K-1)1}x_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "677a5c4d",
+   "id": "2153c8e3",
    "metadata": {
     "editable": true
    },
@@ -810,25 +810,25 @@
     "multinomial logistic regression and linear discriminant analysis, the\n",
     "input to the function is the result of $K$ distinct linear functions,\n",
     "and the predicted probability for the $k$-th class given a sample\n",
-    "vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\beta}$ is (with two\n",
+    "vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\theta}$ is (with two\n",
     "predictors):"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "fa70e3bf",
+   "id": "ab540777",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n",
+    "p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\theta_{k0}+\\theta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\theta_{l0}+\\theta_{l1}x_1)}}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4c59400c",
+   "id": "b7bcd7e8",
    "metadata": {
     "editable": true
    },
@@ -838,19 +838,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f4073652",
+   "id": "c9605782",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n",
+    "p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\theta_{l0}+\\theta_{l1}x_1)}},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "810e872a",
+   "id": "856dd896",
    "metadata": {
     "editable": true
    },
@@ -867,7 +867,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f4ebc1da",
+   "id": "73029fad",
    "metadata": {
     "editable": true
    },
@@ -882,7 +882,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "b7a8cbac",
+   "id": "d81a44e3",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -918,7 +918,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "10f74b93",
+   "id": "83a2e903",
    "metadata": {
     "editable": true
    },
@@ -930,7 +930,7 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "d0b8025d",
+   "id": "976fc321",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -975,7 +975,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9d84a2aa",
+   "id": "ea10d953",
    "metadata": {
     "editable": true
    },
@@ -998,7 +998,7 @@
   {
    "cell_type": "code",
    "execution_count": 6,
-   "id": "6144ea0a",
+   "id": "42204f9c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1010,7 +1010,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5627f5ae",
+   "id": "a56de206",
    "metadata": {
     "editable": true
    },
@@ -1021,7 +1021,7 @@
   {
    "cell_type": "code",
    "execution_count": 7,
-   "id": "a7c8662a",
+   "id": "0f2ff030",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1033,7 +1033,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d03ee2a7",
+   "id": "68abe92d",
    "metadata": {
     "editable": true
    },
@@ -1059,7 +1059,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "44ce5095",
+   "id": "3cea1c5a",
    "metadata": {
     "editable": true
    },
@@ -1072,7 +1072,7 @@
   {
    "cell_type": "code",
    "execution_count": 8,
-   "id": "24ff3dd3",
+   "id": "f42e1d87",
    "metadata": {
     "collapsed": false,
     "editable": true
diff --git a/doc/LectureNotes/chapter1.ipynb b/doc/LectureNotes/chapter1.ipynb
index 4d16c5ab1..231068020 100644
--- a/doc/LectureNotes/chapter1.ipynb
+++ b/doc/LectureNotes/chapter1.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "d66e8057",
+   "id": "a453b968",
    "metadata": {
     "editable": true
    },
@@ -13,7 +13,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3f6464d0",
+   "id": "499b2ddb",
    "metadata": {
     "editable": true
    },
@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fab3a8f9",
+   "id": "c84cce7e",
    "metadata": {
     "editable": true
    },
@@ -65,7 +65,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "66513db3",
+   "id": "8419208e",
    "metadata": {
     "editable": true
    },
@@ -167,7 +167,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0d1f19f9",
+   "id": "ceb7a805",
    "metadata": {
     "editable": true
    },
@@ -202,7 +202,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0479c66e",
+   "id": "d6c1062f",
    "metadata": {
     "editable": true
    },
@@ -253,7 +253,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "609d8174",
+   "id": "50c1b706",
    "metadata": {
     "editable": true
    },
@@ -286,7 +286,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e580e18",
+   "id": "12e3ac84",
    "metadata": {
     "editable": true
    },
@@ -298,7 +298,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bb8f332d",
+   "id": "10b4c333",
    "metadata": {
     "editable": true
    },
@@ -335,7 +335,7 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "9e479f11",
+   "id": "3e216ef0",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -368,7 +368,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8c7b818b",
+   "id": "af765a12",
    "metadata": {
     "editable": true
    },
@@ -385,7 +385,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "201888d1",
+   "id": "2f6fd730",
    "metadata": {
     "editable": true
    },
@@ -397,7 +397,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e6a16258",
+   "id": "1770d85d",
    "metadata": {
     "editable": true
    },
@@ -418,7 +418,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c3aab7c4",
+   "id": "684b72a0",
    "metadata": {
     "editable": true
    },
@@ -431,7 +431,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f2957240",
+   "id": "e9fa8fd5",
    "metadata": {
     "editable": true
    },
@@ -462,7 +462,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2a15d7ec",
+   "id": "15a4642f",
    "metadata": {
     "editable": true
    },
@@ -474,7 +474,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12af21ef",
+   "id": "589ec9cb",
    "metadata": {
     "editable": true
    },
@@ -492,7 +492,7 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "16f86276",
+   "id": "cf21ae2b",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -520,7 +520,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "87da55cc",
+   "id": "41c55cbb",
    "metadata": {
     "editable": true
    },
@@ -542,7 +542,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "221ebafe",
+   "id": "d62c9412",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -580,7 +580,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "aa05a791",
+   "id": "ae0c6c2a",
    "metadata": {
     "editable": true
    },
@@ -591,7 +591,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bac50889",
+   "id": "64cf13de",
    "metadata": {
     "editable": true
    },
@@ -604,7 +604,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "825fa9b8",
+   "id": "18439dae",
    "metadata": {
     "editable": true
    },
@@ -625,7 +625,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a3fa2a46",
+   "id": "3e9fb291",
    "metadata": {
     "editable": true
    },
@@ -637,7 +637,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7627c75a",
+   "id": "46d8744e",
    "metadata": {
     "editable": true
    },
@@ -647,7 +647,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "52686744",
+   "id": "38a29b65",
    "metadata": {
     "editable": true
    },
@@ -659,7 +659,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f6cb9efa",
+   "id": "438e73b8",
    "metadata": {
     "editable": true
    },
@@ -671,7 +671,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4822580d",
+   "id": "1403cc6a",
    "metadata": {
     "editable": true
    },
@@ -683,7 +683,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fb61578b",
+   "id": "9fbd1c1b",
    "metadata": {
     "editable": true
    },
@@ -694,7 +694,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6cb0d520",
+   "id": "d25da1d3",
    "metadata": {
     "editable": true
    },
@@ -706,7 +706,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "87835e2a",
+   "id": "c43c7452",
    "metadata": {
     "editable": true
    },
@@ -728,7 +728,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "69fc895e",
+   "id": "ad0d5004",
    "metadata": {
     "editable": true
    },
@@ -740,7 +740,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "debfac94",
+   "id": "eb9e2efb",
    "metadata": {
     "editable": true
    },
@@ -755,7 +755,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "96164cb3",
+   "id": "963f4f84",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -796,7 +796,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7cac7e62",
+   "id": "d530f60a",
    "metadata": {
     "editable": true
    },
@@ -811,7 +811,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8b9b62fe",
+   "id": "f8951307",
    "metadata": {
     "editable": true
    },
@@ -823,7 +823,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "81afb7da",
+   "id": "913bb6a6",
    "metadata": {
     "editable": true
    },
@@ -833,7 +833,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "266b9b6d",
+   "id": "25cb82ce",
    "metadata": {
     "editable": true
    },
@@ -845,7 +845,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "26bfbb29",
+   "id": "d04f788d",
    "metadata": {
     "editable": true
    },
@@ -855,7 +855,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e4ee0e4",
+   "id": "866bf69f",
    "metadata": {
     "editable": true
    },
@@ -867,7 +867,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "40627d20",
+   "id": "a0d3421f",
    "metadata": {
     "editable": true
    },
@@ -877,7 +877,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b4fbd650",
+   "id": "680eefd0",
    "metadata": {
     "editable": true
    },
@@ -889,7 +889,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "40eecb7a",
+   "id": "de97e875",
    "metadata": {
     "editable": true
    },
@@ -905,7 +905,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "97f2b230",
+   "id": "5d7dbc49",
    "metadata": {
     "editable": true
    },
@@ -917,7 +917,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "34eed0f8",
+   "id": "5a9a6d4d",
    "metadata": {
     "editable": true
    },
@@ -928,7 +928,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "37ca2a5b",
+   "id": "fef247ce",
    "metadata": {
     "editable": true
    },
@@ -940,7 +940,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "36071972",
+   "id": "3ed61266",
    "metadata": {
     "editable": true
    },
@@ -954,7 +954,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "32a6cbaf",
+   "id": "9882fb4e",
    "metadata": {
     "editable": true
    },
@@ -966,7 +966,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "68d9be7d",
+   "id": "992b3ae7",
    "metadata": {
     "editable": true
    },
@@ -991,7 +991,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7ec5f81c",
+   "id": "6d772376",
    "metadata": {
     "editable": true
    },
@@ -1008,7 +1008,7 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "fb39ace0",
+   "id": "f861cd3f",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1052,40 +1052,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fc71e4e1",
-   "metadata": {
-    "editable": true
-   },
-   "source": [
-    "Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various **matplotlib** commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "30e59ca4",
-   "metadata": {
-    "collapsed": false,
-    "editable": true
-   },
-   "outputs": [],
-   "source": [
-    "from pylab import plt, mpl\n",
-    "plt.style.use('seaborn')\n",
-    "mpl.rcParams['font.family'] = 'serif'\n",
-    "\n",
-    "def MakePlot(x,y, styles, labels, axlabels):\n",
-    "    plt.figure(figsize=(10,6))\n",
-    "    for i in range(len(x)):\n",
-    "        plt.plot(x[i], y[i], styles[i], label = labels[i])\n",
-    "        plt.xlabel(axlabels[0])\n",
-    "        plt.ylabel(axlabels[1])\n",
-    "    plt.legend(loc=0)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e58a020b",
+   "id": "da27a328",
    "metadata": {
     "editable": true
    },
@@ -1101,8 +1068,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
-   "id": "de0f27ba",
+   "execution_count": 6,
+   "id": "b4ef8b5a",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1123,7 +1090,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5da52984",
+   "id": "1352f7d2",
    "metadata": {
     "editable": true
    },
@@ -1136,8 +1103,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
-   "id": "ba67d882",
+   "execution_count": 7,
+   "id": "c5b3333d",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1166,7 +1133,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b75681d2",
+   "id": "6652db3a",
    "metadata": {
     "editable": true
    },
@@ -1185,8 +1152,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
-   "id": "d9760898",
+   "execution_count": 8,
+   "id": "715c03ae",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1203,7 +1170,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5199cc2b",
+   "id": "eaff6065",
    "metadata": {
     "editable": true
    },
@@ -1214,8 +1181,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
-   "id": "d48c42f2",
+   "execution_count": 9,
+   "id": "4053e205",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1233,7 +1200,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "91ca751e",
+   "id": "587289d6",
    "metadata": {
     "editable": true
    },
@@ -1248,8 +1215,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
-   "id": "ea82f2ad",
+   "execution_count": 10,
+   "id": "52f1c9dc",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1262,7 +1229,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0fb84108",
+   "id": "25241a4e",
    "metadata": {
     "editable": true
    },
@@ -1273,8 +1240,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
-   "id": "3b24153e",
+   "execution_count": 11,
+   "id": "06020cef",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1304,7 +1271,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d9acc526",
+   "id": "8d07d616",
    "metadata": {
     "editable": true
    },
@@ -1314,8 +1281,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
-   "id": "942b4d36",
+   "execution_count": 12,
+   "id": "1c4a2fd6",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1356,7 +1323,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8fe6a29e",
+   "id": "b9728189",
    "metadata": {
     "editable": true
    },
@@ -1377,8 +1344,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
-   "id": "5dc4b33c",
+   "execution_count": 13,
+   "id": "8b5b6d13",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1418,7 +1385,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8dd1a395",
+   "id": "ec654f7a",
    "metadata": {
     "editable": true
    },
@@ -1427,12 +1394,12 @@
     "\n",
     "[Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureAug27.mp4?vrtx=view-as-webpage).\n",
     "\n",
-    "Fitting a continuous function with linear parameterization in terms of the parameters  $\\boldsymbol{\\beta}$.\n",
+    "Fitting a continuous function with linear parameterization in terms of the parameters  $\\boldsymbol{\\theta}$.\n",
     "* Method of choice for fitting a continuous function!\n",
     "\n",
     "* Gives an excellent introduction to central Machine Learning features with **understandable pedagogical** links to other methods like **Neural Networks**, **Support Vector Machines** etc\n",
     "\n",
-    "* Analytical expression for the fitting parameters $\\boldsymbol{\\beta}$\n",
+    "* Analytical expression for the fitting parameters $\\boldsymbol{\\theta}$\n",
     "\n",
     "* Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more\n",
     "\n",
@@ -1450,14 +1417,14 @@
     "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n",
     "\n",
     "Regression modeling deals with the description of  the sampling distribution of a given random variable $y$ and how it varies as function of another variable or a set of such variables $\\boldsymbol{x} =[x_0, x_1,\\dots, x_{n-1}]^T$. \n",
-    "The first variable is called the **dependent**, the **outcome** or the **response** variable while the set of variables $\\boldsymbol{x}$ is called the independent variable, or the predictor variable or the explanatory variable. \n",
+    "The first variable is called the **dependent**, the **outcome** or the **response** or just the output variable while the set of variables $\\boldsymbol{x}$ is called the independent variable, or the predictor variable or the explanatory variable, or just the input variable. We will hereafter call $\\boldsymbol{y}$ and $\\boldsymbol{x}x$ for the output and input variables, respectively. \n",
     "\n",
-    "A regression model aims at finding a likelihood function $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$, that is the conditional distribution for $\\boldsymbol{y}$ with a given $\\boldsymbol{x}$. The estimation of  $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$ is made using a data set with \n",
+    "A regression model aims at finding a likelihood function $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$ (or just a function $f(\\boldsymbol{x}$) , that is the conditional distribution for $\\boldsymbol{y}$ with a given $\\boldsymbol{x}$. The estimation of  $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$ is made using a data set with \n",
     "* $n$ cases $i = 0, 1, 2, \\dots, n-1$ \n",
     "\n",
-    "* Response (target, dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \\dots, n-1$ \n",
+    "* Response/output (target, dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \\dots, n-1$ \n",
     "\n",
-    "* $p$ so-called explanatory (independent or predictor) variables $\\boldsymbol{x}_i=[x_{i0}, x_{i1}, \\dots, x_{ip-1}]$ with $i = 0, 1, 2, \\dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples.   \n",
+    "* $p$ so-called explanatory/input (independent or predictor) variables $\\boldsymbol{x}_i=[x_{i0}, x_{i1}, \\dots, x_{ip-1}]$ with $i = 0, 1, 2, \\dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples.   \n",
     "\n",
     " The goal of the regression analysis is to extract/exploit relationship between $\\boldsymbol{y}$ and $\\boldsymbol{x}$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things.\n",
     "\n",
@@ -1474,10 +1441,10 @@
     "f(\\mathbf{X}_{i,\\ast})$. When no prior knowledge on the form of\n",
     "$f(\\cdot)$ is available, it is common to assume a linear relationship\n",
     "between $\\boldsymbol{X}$ and $\\boldsymbol{y}$. This assumption gives rise to\n",
-    "the *linear regression model* where $\\boldsymbol{\\beta} = [\\beta_0, \\ldots,\n",
-    "\\beta_{p-1}]^{T}$ are the *regression parameters*. \n",
+    "the *linear regression model* where $\\boldsymbol{\\theta} = [\\theta_0, \\ldots,\n",
+    "\\theta_{p-1}]^{T}$ are the *regression parameters*. \n",
     "\n",
-    "Linear regression gives us a set of analytical equations for the parameters $\\beta_j$.\n",
+    "Linear regression gives us a set of analytical equations for the parameters $\\theta_j$.\n",
     "\n",
     "In order to understand the relation among the predictors $p$, the set of data $n$ and the target (outcome, output etc) $\\boldsymbol{y}$,\n",
     "consider the model we discussed for describing nuclear binding energies. \n",
@@ -1488,7 +1455,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9845070b",
+   "id": "08d4a54f",
    "metadata": {
     "editable": true
    },
@@ -1500,7 +1467,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fc1de086",
+   "id": "df21cf23",
    "metadata": {
     "editable": true
    },
@@ -1519,19 +1486,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e9b27dfb",
+   "id": "6359332f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "y=y(x) \\rightarrow y(x_i)=\\tilde{y}_i+\\epsilon_i=\\sum_{j=0}^{n-1} \\beta_j x_i^j+\\epsilon_i,\n",
+    "y=y(x) \\rightarrow y(x_i)=\\tilde{y}_i+\\epsilon_i=\\sum_{j=0}^{n-1} \\theta_j x_i^j+\\epsilon_i,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d80ac08b",
+   "id": "6c8e31ed",
    "metadata": {
     "editable": true
    },
@@ -1543,25 +1510,25 @@
   },
   {
    "cell_type": "markdown",
-   "id": "59de586a",
+   "id": "0cc24030",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "y_0&=\\beta_0+\\beta_1x_0^1+\\beta_2x_0^2+\\dots+\\beta_{n-1}x_0^{n-1}+\\epsilon_0\\\\\n",
-    "y_1&=\\beta_0+\\beta_1x_1^1+\\beta_2x_1^2+\\dots+\\beta_{n-1}x_1^{n-1}+\\epsilon_1\\\\\n",
-    "y_2&=\\beta_0+\\beta_1x_2^1+\\beta_2x_2^2+\\dots+\\beta_{n-1}x_2^{n-1}+\\epsilon_2\\\\\n",
+    "y_0&=\\theta_0+\\theta_1x_0^1+\\theta_2x_0^2+\\dots+\\theta_{n-1}x_0^{n-1}+\\epsilon_0\\\\\n",
+    "y_1&=\\theta_0+\\theta_1x_1^1+\\theta_2x_1^2+\\dots+\\theta_{n-1}x_1^{n-1}+\\epsilon_1\\\\\n",
+    "y_2&=\\theta_0+\\theta_1x_2^1+\\theta_2x_2^2+\\dots+\\theta_{n-1}x_2^{n-1}+\\epsilon_2\\\\\n",
     "\\dots & \\dots \\\\\n",
-    "y_{n-1}&=\\beta_0+\\beta_1x_{n-1}^1+\\beta_2x_{n-1}^2+\\dots+\\beta_{n-1}x_{n-1}^{n-1}+\\epsilon_{n-1}.\\\\\n",
+    "y_{n-1}&=\\theta_0+\\theta_1x_{n-1}^1+\\theta_2x_{n-1}^2+\\dots+\\theta_{n-1}x_{n-1}^{n-1}+\\epsilon_{n-1}.\\\\\n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "3a0a6d11",
+   "id": "146809cb",
    "metadata": {
     "editable": true
    },
@@ -1571,7 +1538,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f545dd3f",
+   "id": "4796a596",
    "metadata": {
     "editable": true
    },
@@ -1583,7 +1550,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a4484e53",
+   "id": "edfabcdf",
    "metadata": {
     "editable": true
    },
@@ -1593,19 +1560,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a1e5488",
+   "id": "0b4a6062",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta} = [\\beta_0,\\beta_1, \\beta_2,\\dots, \\beta_{n-1}]^T,\n",
+    "\\boldsymbol{\\theta} = [\\theta_0,\\theta_1, \\theta_2,\\dots, \\theta_{n-1}]^T,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "65f8b020",
+   "id": "e419ff68",
    "metadata": {
     "editable": true
    },
@@ -1615,7 +1582,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0ded825d",
+   "id": "dd55bff1",
    "metadata": {
     "editable": true
    },
@@ -1627,7 +1594,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4747b482",
+   "id": "df571b73",
    "metadata": {
     "editable": true
    },
@@ -1637,7 +1604,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1a50e6b1",
+   "id": "c1c6ffaa",
    "metadata": {
     "editable": true
    },
@@ -1656,7 +1623,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2404cda8",
+   "id": "592bff5d",
    "metadata": {
     "editable": true
    },
@@ -1666,19 +1633,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "23bc5fc5",
+   "id": "9b776152",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta}+\\boldsymbol{\\epsilon}.\n",
+    "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\theta}+\\boldsymbol{\\epsilon}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ae3d3214",
+   "id": "29c95a7a",
    "metadata": {
     "editable": true
    },
@@ -1694,27 +1661,27 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bd07fd5c",
+   "id": "b53b0001",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
-    "y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
-    "y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_2\\\\\n",
+    "y_0&=\\theta_0x_{00}+\\theta_1x_{01}+\\theta_2x_{02}+\\dots+\\theta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
+    "y_1&=\\theta_0x_{10}+\\theta_1x_{11}+\\theta_2x_{12}+\\dots+\\theta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
+    "y_2&=\\theta_0x_{20}+\\theta_1x_{21}+\\theta_2x_{22}+\\dots+\\theta_{n-1}x_{2n-1}+\\epsilon_2\\\\\n",
     "\\dots & \\dots \\\\\n",
-    "y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_i\\\\\n",
+    "y_{i}&=\\theta_0x_{i0}+\\theta_1x_{i1}+\\theta_2x_{i2}+\\dots+\\theta_{n-1}x_{in-1}+\\epsilon_i\\\\\n",
     "\\dots & \\dots \\\\\n",
-    "y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
+    "y_{n-1}&=\\theta_0x_{n-1,0}+\\theta_1x_{n-1,2}+\\theta_2x_{n-1,2}+\\dots+\\theta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "cd7bdfc0",
+   "id": "f2fea940",
    "metadata": {
     "editable": true
    },
@@ -1726,7 +1693,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "acb43d02",
+   "id": "26411342",
    "metadata": {
     "editable": true
    },
@@ -1745,7 +1712,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f1e9d7fc",
+   "id": "d432e651",
    "metadata": {
     "editable": true
    },
@@ -1755,51 +1722,51 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c41dcd08",
+   "id": "e536d0b3",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta}+\\boldsymbol{\\epsilon}.\n",
+    "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\theta}+\\boldsymbol{\\epsilon}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "89ebc548",
+   "id": "bbe0dd39",
    "metadata": {
     "editable": true
    },
    "source": [
-    "The left-hand side of this equation is kwown. Our error vector $\\boldsymbol{\\epsilon}$ and the parameter vector $\\boldsymbol{\\beta}$ are our unknow quantities. How can we obtain the optimal set of $\\beta_i$ values? \n",
+    "The left-hand side of this equation is kwown. Our error vector $\\boldsymbol{\\epsilon}$ and the parameter vector $\\boldsymbol{\\theta}$ are our unknow quantities. How can we obtain the optimal set of $\\theta_i$ values? \n",
     "\n",
     "We have defined the matrix $\\boldsymbol{X}$ via the equations"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8e0a376a",
+   "id": "51d53970",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
-    "y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
-    "y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_1\\\\\n",
+    "y_0&=\\theta_0x_{00}+\\theta_1x_{01}+\\theta_2x_{02}+\\dots+\\theta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
+    "y_1&=\\theta_0x_{10}+\\theta_1x_{11}+\\theta_2x_{12}+\\dots+\\theta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
+    "y_2&=\\theta_0x_{20}+\\theta_1x_{21}+\\theta_2x_{22}+\\dots+\\theta_{n-1}x_{2n-1}+\\epsilon_1\\\\\n",
     "\\dots & \\dots \\\\\n",
-    "y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_1\\\\\n",
+    "y_{i}&=\\theta_0x_{i0}+\\theta_1x_{i1}+\\theta_2x_{i2}+\\dots+\\theta_{n-1}x_{in-1}+\\epsilon_1\\\\\n",
     "\\dots & \\dots \\\\\n",
-    "y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
+    "y_{n-1}&=\\theta_0x_{n-1,0}+\\theta_1x_{n-1,2}+\\theta_2x_{n-1,2}+\\dots+\\theta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b2c90704",
+   "id": "4778eaaf",
    "metadata": {
     "editable": true
    },
@@ -1815,8 +1782,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
-   "id": "e89031dc",
+   "execution_count": 14,
+   "id": "87ed061d",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1896,75 +1863,75 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a1e8f92",
+   "id": "42b8f7c4",
    "metadata": {
     "editable": true
    },
    "source": [
-    "With $\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p\\times 1}$, it means that we will hereafter write our equations for the approximation as"
+    "With $\\boldsymbol{\\theta}\\in {\\mathbb{R}}^{p\\times 1}$, it means that we will hereafter write our equations for the approximation as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "84bf3673",
+   "id": "73ea9a01",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n",
+    "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\theta},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e10c4fc5",
+   "id": "253891dd",
    "metadata": {
     "editable": true
    },
    "source": [
     "throughout these lectures. \n",
     "\n",
-    "With the above we use the design matrix to define the approximation $\\boldsymbol{\\tilde{y}}$ via the unknown quantity $\\boldsymbol{\\beta}$ as"
+    "With the above we use the design matrix to define the approximation $\\boldsymbol{\\tilde{y}}$ via the unknown quantity $\\boldsymbol{\\theta}$ as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "20866df8",
+   "id": "d2d20886",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n",
+    "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\theta},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a075809a",
+   "id": "480c4c58",
    "metadata": {
     "editable": true
    },
    "source": [
-    "and in order to find the optimal parameters $\\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\\tilde{y}_i$, namely"
+    "and in order to find the optimal parameters $\\theta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\\tilde{y}_i$, namely"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ba0e251b",
+   "id": "7a782da9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n",
+    "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "93a04fb7",
+   "id": "c1c60d77",
    "metadata": {
     "editable": true
    },
@@ -1974,19 +1941,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "56c4fcf1",
+   "id": "60d26064",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
+    "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e8fc405e",
+   "id": "8a19c134",
    "metadata": {
     "editable": true
    },
@@ -1999,43 +1966,43 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fefb3fd3",
+   "id": "f21d525f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{1}{2n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2,\n",
+    "C(\\boldsymbol{\\theta})=\\frac{1}{2n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d0eeb2dc",
+   "id": "f41c45c4",
    "metadata": {
     "editable": true
    },
    "source": [
-    "since when taking the first derivative with respect to the unknown parameters $\\beta$, the factor of $2$ cancels out. \n",
+    "since when taking the first derivative with respect to the unknown parameters $\\theta$, the factor of $2$ cancels out. \n",
     "\n",
     "The function"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1f408b23",
+   "id": "b356426a",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\},\n",
+    "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "dee2a248",
+   "id": "5169a2dd",
    "metadata": {
     "editable": true
    },
@@ -2046,19 +2013,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f1d1d292",
+   "id": "19839cdf",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "y_{i}=\\langle y_i \\rangle = \\beta_0x_{i,0}+\\beta_1x_{i,1}+\\beta_2x_{i,2}+\\dots+\\beta_{n-1}x_{i,n-1}+\\epsilon_i,\n",
+    "y_{i}=\\langle y_i \\rangle = \\theta_0x_{i,0}+\\theta_1x_{i,1}+\\theta_2x_{i,2}+\\dots+\\theta_{n-1}x_{i,n-1}+\\epsilon_i,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8b2075d1",
+   "id": "e664d67f",
    "metadata": {
     "editable": true
    },
@@ -2072,25 +2039,25 @@
     "the standard deviation discussed earlier. In the discussion here we\n",
     "will treat $y_i$ as our exact value for the response variable.\n",
     "\n",
-    "In order to find the parameters $\\beta_i$ we will then minimize the spread of $C(\\boldsymbol{\\beta})$, that is we are going to solve the problem"
+    "In order to find the parameters $\\theta_i$ we will then minimize the spread of $C(\\boldsymbol{\\theta})$, that is we are going to solve the problem"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "0c3145a6",
+   "id": "a14a28ab",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
-    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
+    "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n",
+    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e72f34b6",
+   "id": "e2f643ef",
    "metadata": {
     "editable": true
    },
@@ -2100,19 +2067,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f2f9ee3f",
+   "id": "869ceba9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n",
+    "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_j} = \\frac{\\partial }{\\partial \\theta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f15f85e7",
+   "id": "371f2221",
    "metadata": {
     "editable": true
    },
@@ -2122,19 +2089,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ded06fd1",
+   "id": "b4bf3615",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)\\right]=0,\n",
+    "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}\\right)\\right]=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "dc9c099b",
+   "id": "3baf0d9a",
    "metadata": {
     "editable": true
    },
@@ -2144,19 +2111,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "40bc212d",
+   "id": "5e98ea5f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right).\n",
+    "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7bc3483f",
+   "id": "6e515735",
    "metadata": {
     "editable": true
    },
@@ -2166,19 +2133,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "22662e2a",
+   "id": "8363a7ce",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right),\n",
+    "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right),\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6798151b",
+   "id": "77eb2d81",
    "metadata": {
     "editable": true
    },
@@ -2188,19 +2155,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e63dc09e",
+   "id": "a94cb0f2",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta},\n",
+    "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a2f541a1",
+   "id": "09bdeb45",
    "metadata": {
     "editable": true
    },
@@ -2210,19 +2177,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fd8bba77",
+   "id": "d44f06e9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+    "\\boldsymbol{\\theta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2163c93e",
+   "id": "1a83dceb",
    "metadata": {
     "editable": true
    },
@@ -2246,7 +2213,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "48634f87",
+   "id": "efa701a0",
    "metadata": {
     "editable": true
    },
@@ -2258,7 +2225,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9117c119",
+   "id": "84be9b76",
    "metadata": {
     "editable": true
    },
@@ -2268,7 +2235,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c7cd5b11",
+   "id": "194914da",
    "metadata": {
     "editable": true
    },
@@ -2280,7 +2247,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1159b476",
+   "id": "da2cf79d",
    "metadata": {
     "editable": true
    },
@@ -2290,7 +2257,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "75098194",
+   "id": "7f331eee",
    "metadata": {
     "editable": true
    },
@@ -2302,7 +2269,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3bd3d52b",
+   "id": "ad13d0dc",
    "metadata": {
     "editable": true
    },
@@ -2315,19 +2282,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b086e3f6",
+   "id": "d00b722b",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial^2 C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
+    "\\frac{\\partial^2 C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}^T\\partial \\boldsymbol{\\theta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6fe2d631",
+   "id": "8e486d44",
    "metadata": {
     "editable": true
    },
@@ -2339,7 +2306,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "26bd4db6",
+   "id": "29dcd5db",
    "metadata": {
     "editable": true
    },
@@ -2351,7 +2318,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "141de40c",
+   "id": "1a45048f",
    "metadata": {
     "editable": true
    },
@@ -2366,19 +2333,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d44398a3",
+   "id": "3c8c91cc",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\epsilon} = \\boldsymbol{y}-\\boldsymbol{\\tilde{y}} = \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta},\n",
+    "\\boldsymbol{\\epsilon} = \\boldsymbol{y}-\\boldsymbol{\\tilde{y}} = \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "fb4c7d58",
+   "id": "4adc4243",
    "metadata": {
     "editable": true
    },
@@ -2388,19 +2355,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3c1a0371",
+   "id": "9c998ca0",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n",
+    "\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)= 0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "84a0c456",
+   "id": "4894c857",
    "metadata": {
     "editable": true
    },
@@ -2410,50 +2377,50 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1c5edf48",
+   "id": "80ea22e9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{X}^T\\boldsymbol{\\epsilon}=\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n",
+    "\\boldsymbol{X}^T\\boldsymbol{\\epsilon}=\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)= 0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "332a3b12",
+   "id": "e398b8f7",
    "metadata": {
     "editable": true
    },
    "source": [
-    "meaning that the solution for $\\boldsymbol{\\beta}$ is the one which minimizes the residuals.  Later we will link this with the maximum likelihood approach.\n",
+    "meaning that the solution for $\\boldsymbol{\\theta}$ is the one which minimizes the residuals.  Later we will link this with the maximum likelihood approach.\n",
     "\n",
     "Let us now return to our nuclear binding energies and simply code the above equations. \n",
     "\n",
-    "It is rather straightforward to implement the matrix inversion and obtain the parameters $\\boldsymbol{\\beta}$. After having defined the matrix $\\boldsymbol{X}$ we simply need to \n",
+    "It is rather straightforward to implement the matrix inversion and obtain the parameters $\\boldsymbol{\\theta}$. After having defined the matrix $\\boldsymbol{X}$ we simply need to \n",
     "write"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
-   "id": "f52e02a6",
+   "execution_count": 15,
+   "id": "27d5b629",
    "metadata": {
     "collapsed": false,
     "editable": true
    },
    "outputs": [],
    "source": [
-    "# matrix inversion to find beta\n",
-    "beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)\n",
+    "# matrix inversion to find theta\n",
+    "theta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)\n",
     "# and then make the prediction\n",
-    "ytilde = X @ beta"
+    "ytilde = X @ theta"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "36ae01f6",
+   "id": "a95d91a9",
    "metadata": {
     "editable": true
    },
@@ -2463,8 +2430,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
-   "id": "142d2f33",
+   "execution_count": 16,
+   "id": "4d42fedc",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2477,7 +2444,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "46296414",
+   "id": "d15ca047",
    "metadata": {
     "editable": true
    },
@@ -2487,8 +2454,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
-   "id": "86ed7e92",
+   "execution_count": 17,
+   "id": "414ab312",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2511,7 +2478,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "942647c0",
+   "id": "11bc6803",
    "metadata": {
     "editable": true
    },
@@ -2522,8 +2489,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
-   "id": "e26cee7f",
+   "execution_count": 18,
+   "id": "aaa2eeb3",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2536,7 +2503,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "462de17c",
+   "id": "b9eb34c0",
    "metadata": {
     "editable": true
    },
@@ -2546,8 +2513,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
-   "id": "e50560cf",
+   "execution_count": 19,
+   "id": "1a22333a",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2559,7 +2526,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ad325590",
+   "id": "067660ad",
    "metadata": {
     "editable": true
    },
@@ -2569,8 +2536,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
-   "id": "682dcbff",
+   "execution_count": 20,
+   "id": "25d43913",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2586,7 +2553,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a3c40b6",
+   "id": "73c9fe40",
    "metadata": {
     "editable": true
    },
@@ -2596,8 +2563,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
-   "id": "1cc29e48",
+   "execution_count": 21,
+   "id": "0ecca752",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2611,7 +2578,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1ddbddcd",
+   "id": "da2c5259",
    "metadata": {
     "editable": true
    },
@@ -2633,43 +2600,43 @@
   },
   {
    "cell_type": "markdown",
-   "id": "90a8edfe",
+   "id": "acd25436",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\chi^2(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\frac{\\left(y_i-\\tilde{y}_i\\right)^2}{\\sigma_i^2}=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\frac{1}{\\boldsymbol{\\Sigma^2}}\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n",
+    "\\chi^2(\\boldsymbol{\\theta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\frac{\\left(y_i-\\tilde{y}_i\\right)^2}{\\sigma_i^2}=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\frac{1}{\\boldsymbol{\\Sigma^2}}\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4ce63e21",
+   "id": "c3333906",
    "metadata": {
     "editable": true
    },
    "source": [
     "where the matrix $\\boldsymbol{\\Sigma}$ is a diagonal matrix with $\\sigma_i$ as matrix elements. \n",
     "\n",
-    "In order to find the parameters $\\beta_i$ we will then minimize the spread of $\\chi^2(\\boldsymbol{\\beta})$ by requiring"
+    "In order to find the parameters $\\theta_i$ we will then minimize the spread of $\\chi^2(\\boldsymbol{\\theta})$ by requiring"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d4c3d110",
+   "id": "d9069e54",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)^2\\right]=0,\n",
+    "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\theta_j} = \\frac{\\partial }{\\partial \\theta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)^2\\right]=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "cf9ad9c2",
+   "id": "2d4b51c7",
    "metadata": {
     "editable": true
    },
@@ -2679,19 +2646,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3b1f5fc1",
+   "id": "365750e2",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}\\frac{x_{ij}}{\\sigma_i}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)\\right]=0,\n",
+    "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\theta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}\\frac{x_{ij}}{\\sigma_i}\\left(\\frac{y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)\\right]=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "559f6a04",
+   "id": "16a7ed8d",
    "metadata": {
     "editable": true
    },
@@ -2701,19 +2668,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bba9e13a",
+   "id": "ce115915",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\beta}\\right).\n",
+    "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\theta}\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c97cf725",
+   "id": "10c88bf7",
    "metadata": {
     "editable": true
    },
@@ -2725,19 +2692,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6256d393",
+   "id": "c13f349e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\beta}\\right),\n",
+    "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\theta}\\right),\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4274f29f",
+   "id": "e1cfa827",
    "metadata": {
     "editable": true
    },
@@ -2747,19 +2714,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6424e32f",
+   "id": "228294e8",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{A}^T\\boldsymbol{b} = \\boldsymbol{A}^T\\boldsymbol{A}\\boldsymbol{\\beta},\n",
+    "\\boldsymbol{A}^T\\boldsymbol{b} = \\boldsymbol{A}^T\\boldsymbol{A}\\boldsymbol{\\theta},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2babfc95",
+   "id": "dd1aa581",
    "metadata": {
     "editable": true
    },
@@ -2769,19 +2736,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1a25daa2",
+   "id": "4e631fe5",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta} =\\left(\\boldsymbol{A}^T\\boldsymbol{A}\\right)^{-1}\\boldsymbol{A}^T\\boldsymbol{b}.\n",
+    "\\boldsymbol{\\theta} =\\left(\\boldsymbol{A}^T\\boldsymbol{A}\\right)^{-1}\\boldsymbol{A}^T\\boldsymbol{b}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d2a18a90",
+   "id": "a1c651fa",
    "metadata": {
     "editable": true
    },
@@ -2791,7 +2758,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c2730e09",
+   "id": "0ba91c05",
    "metadata": {
     "editable": true
    },
@@ -2803,51 +2770,51 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dab1abb8",
+   "id": "9c7d45d8",
    "metadata": {
     "editable": true
    },
    "source": [
-    "we have then the following expression for the parameters $\\beta_j$ (the matrix elements of $\\boldsymbol{H}$ are $h_{ij}$)"
+    "we have then the following expression for the parameters $\\theta_j$ (the matrix elements of $\\boldsymbol{H}$ are $h_{ij}$)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "be9b74c2",
+   "id": "1148dda1",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_j = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}\\frac{y_i}{\\sigma_i}\\frac{x_{ik}}{\\sigma_i} = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}b_ia_{ik}\n",
+    "\\theta_j = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}\\frac{y_i}{\\sigma_i}\\frac{x_{ik}}{\\sigma_i} = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}b_ia_{ik}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "38e7188c",
+   "id": "899ad725",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We state without proof the expression for the uncertainty  in the parameters $\\beta_j$ as (we leave this as an exercise)"
+    "We state without proof the expression for the uncertainty  in the parameters $\\theta_j$ as (we leave this as an exercise)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c568814d",
+   "id": "442cefde",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\sigma^2(\\beta_j) = \\sum_{i=0}^{n-1}\\sigma_i^2\\left( \\frac{\\partial \\beta_j}{\\partial y_i}\\right)^2,\n",
+    "\\sigma^2(\\theta_j) = \\sum_{i=0}^{n-1}\\sigma_i^2\\left( \\frac{\\partial \\theta_j}{\\partial y_i}\\right)^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "37fb03f1",
+   "id": "303d558b",
    "metadata": {
     "editable": true
    },
@@ -2857,19 +2824,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "753cb5d9",
+   "id": "c8361a08",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\sigma^2(\\beta_j) = \\left(\\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}a_{ik}\\right)\\left(\\sum_{l=0}^{p-1}h_{jl}\\sum_{m=0}^{n-1}a_{ml}\\right) = h_{jj}!\n",
+    "\\sigma^2(\\theta_j) = \\left(\\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}a_{ik}\\right)\\left(\\sum_{l=0}^{p-1}h_{jl}\\sum_{m=0}^{n-1}a_{ml}\\right) = h_{jj}!\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a887be4f",
+   "id": "fad61a6c",
    "metadata": {
     "editable": true
    },
@@ -2879,41 +2846,41 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c99c1576",
+   "id": "29731faf",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "y=y(x) \\rightarrow y(x_i) \\approx \\beta_0+\\beta_1 x_i.\n",
+    "y=y(x) \\rightarrow y(x_i) \\approx \\theta_0+\\theta_1 x_i.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ddb752ed",
+   "id": "ecb0789a",
    "metadata": {
     "editable": true
    },
    "source": [
-    "By computing the derivatives of $\\chi^2$ with respect to $\\beta_0$ and $\\beta_1$ show that these are given by"
+    "By computing the derivatives of $\\chi^2$ with respect to $\\theta_0$ and $\\theta_1$ show that these are given by"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d5d9bc16",
+   "id": "bf04c0eb",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -2\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0,\n",
+    "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\theta_0} = -2\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\theta_0-\\theta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "971e3cbe",
+   "id": "7fba1a68",
    "metadata": {
     "editable": true
    },
@@ -2923,19 +2890,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "14a81f3f",
+   "id": "cb8528d5",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_i\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0.\n",
+    "\\frac{\\partial \\chi^2(\\boldsymbol{\\theta})}{\\partial \\theta_1} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_i\\left(\\frac{y_i-\\theta_0-\\theta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "daa744f2",
+   "id": "2771f01a",
    "metadata": {
     "editable": true
    },
@@ -2946,7 +2913,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bcdd91b0",
+   "id": "dcf74fcf",
    "metadata": {
     "editable": true
    },
@@ -2958,7 +2925,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "78646001",
+   "id": "a7e95721",
    "metadata": {
     "editable": true
    },
@@ -2970,7 +2937,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8eb6b0de",
+   "id": "a636424d",
    "metadata": {
     "editable": true
    },
@@ -2982,7 +2949,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f5641168",
+   "id": "294e5692",
    "metadata": {
     "editable": true
    },
@@ -2994,7 +2961,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c14fcf77",
+   "id": "923ebdc5",
    "metadata": {
     "editable": true
    },
@@ -3006,7 +2973,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9b374023",
+   "id": "d605b9e0",
    "metadata": {
     "editable": true
    },
@@ -3016,45 +2983,45 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5f34ff5d",
+   "id": "d9e8c477",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0 = \\frac{\\gamma_{xx}\\gamma_y-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2},\n",
+    "\\theta_0 = \\frac{\\gamma_{xx}\\gamma_y-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "349a1654",
+   "id": "2831c92f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_1 = \\frac{\\gamma_{xy}\\gamma-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2}.\n",
+    "\\theta_1 = \\frac{\\gamma_{xy}\\gamma-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "bdc8cc84",
+   "id": "36edba71",
    "metadata": {
     "editable": true
    },
    "source": [
     "This approach (different linear and non-linear regression) suffers\n",
     "often from both being underdetermined and overdetermined in the\n",
-    "unknown coefficients $\\beta_i$.  A better approach is to use the\n",
+    "unknown coefficients $\\theta_i$.  A better approach is to use the\n",
     "Singular Value Decomposition (SVD) method discussed below. Or using\n",
     "Lasso and Ridge regression. See below."
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8e31d846",
+   "id": "f209ca1f",
    "metadata": {
     "editable": true
    },
@@ -3079,8 +3046,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 23,
-   "id": "ba56fb3a",
+   "execution_count": 22,
+   "id": "5d230f93",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3162,7 +3129,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "982e0c69",
+   "id": "b3d6f5a2",
    "metadata": {
     "editable": true
    },
@@ -3173,7 +3140,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "acb30447",
+   "id": "4b61d1b0",
    "metadata": {
     "editable": true
    },
@@ -3200,8 +3167,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
-   "id": "301a7664",
+   "execution_count": 23,
+   "id": "8c25e039",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3232,16 +3199,16 @@
     "X[:,2] = x**2\n",
     "# We split the data in test and training data\n",
     "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
-    "# matrix inversion to find beta\n",
-    "beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
-    "print(beta)\n",
+    "# matrix inversion to find theta\n",
+    "theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
+    "print(theta)\n",
     "# and then make the prediction\n",
-    "ytilde = X_train @ beta\n",
+    "ytilde = X_train @ theta\n",
     "print(\"Training R2\")\n",
     "print(R2(y_train,ytilde))\n",
     "print(\"Training MSE\")\n",
     "print(MSE(y_train,ytilde))\n",
-    "ypredict = X_test @ beta\n",
+    "ypredict = X_test @ theta\n",
     "print(\"Test R2\")\n",
     "print(R2(y_test,ypredict))\n",
     "print(\"Test MSE\")\n",
@@ -3250,7 +3217,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "655f046e",
+   "id": "3c09d70c",
    "metadata": {
     "editable": true
    },
@@ -3260,8 +3227,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 25,
-   "id": "413acb4d",
+   "execution_count": 24,
+   "id": "3e9b7ad8",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3286,7 +3253,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f46a080f",
+   "id": "18804e9f",
    "metadata": {
     "editable": true
    },
@@ -3302,8 +3269,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 26,
-   "id": "96391e08",
+   "execution_count": 25,
+   "id": "90f453cd",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3361,15 +3328,15 @@
     "X[:,4] = Density**(5.0/3.0)\n",
     "# We split the data in test and training data\n",
     "X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)\n",
-    "# matrix inversion to find beta\n",
-    "beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)\n",
+    "# matrix inversion to find theta\n",
+    "theta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)\n",
     "# and then make the prediction\n",
-    "ytilde = X_train @ beta\n",
+    "ytilde = X_train @ theta\n",
     "print(\"Training R2\")\n",
     "print(R2(y_train,ytilde))\n",
     "print(\"Training MSE\")\n",
     "print(MSE(y_train,ytilde))\n",
-    "ypredict = X_test @ beta\n",
+    "ypredict = X_test @ theta\n",
     "print(\"Test R2\")\n",
     "print(R2(y_test,ypredict))\n",
     "print(\"Test MSE\")\n",
@@ -3378,7 +3345,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "46a1885f",
+   "id": "3d672d7a",
    "metadata": {
     "editable": true
    },
@@ -3422,7 +3389,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a2041293",
+   "id": "e2426f64",
    "metadata": {
     "editable": true
    },
@@ -3433,8 +3400,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 27,
-   "id": "f813a977",
+   "execution_count": 26,
+   "id": "519e0c09",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3450,7 +3417,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5fc086ec",
+   "id": "7b6fd188",
    "metadata": {
     "editable": true
    },
@@ -3460,8 +3427,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 28,
-   "id": "bf72511a",
+   "execution_count": 27,
+   "id": "3665e1b0",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3479,7 +3446,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "70823078",
+   "id": "96f2fe7b",
    "metadata": {
     "editable": true
    },
@@ -3489,8 +3456,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 29,
-   "id": "85e192bf",
+   "execution_count": 28,
+   "id": "df2c84ca",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3504,7 +3471,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "13fd362f",
+   "id": "0c4588d5",
    "metadata": {
     "editable": true
    },
@@ -3514,8 +3481,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 30,
-   "id": "8f379d9b",
+   "execution_count": 29,
+   "id": "209438db",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3528,7 +3495,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c5f9dc18",
+   "id": "1b67c46a",
    "metadata": {
     "editable": true
    },
@@ -3538,8 +3505,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 31,
-   "id": "cb9f3445",
+   "execution_count": 30,
+   "id": "236343bf",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3556,7 +3523,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a16f238",
+   "id": "c8726877",
    "metadata": {
     "editable": true
    },
@@ -3566,8 +3533,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 32,
-   "id": "a5386de5",
+   "execution_count": 31,
+   "id": "aaa47b00",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3583,7 +3550,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1d2e90ee",
+   "id": "0283f3f1",
    "metadata": {
     "editable": true
    },
@@ -3593,8 +3560,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 33,
-   "id": "b494903c",
+   "execution_count": 32,
+   "id": "c0823ed1",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3618,7 +3585,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c580e7c5",
+   "id": "dc2cf448",
    "metadata": {
     "editable": true
    },
@@ -3628,8 +3595,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 34,
-   "id": "45830e09",
+   "execution_count": 33,
+   "id": "ee945b00",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3642,7 +3609,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fe0e34ba",
+   "id": "288a1417",
    "metadata": {
     "editable": true
    },
@@ -3652,8 +3619,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 35,
-   "id": "f73e5b0f",
+   "execution_count": 34,
+   "id": "0b640ff9",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3673,7 +3640,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a99e4f96",
+   "id": "32cfad5b",
    "metadata": {
     "editable": true
    },
@@ -3683,8 +3650,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 36,
-   "id": "8c972941",
+   "execution_count": 35,
+   "id": "792df674",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3726,8 +3693,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 37,
-   "id": "91ef994e",
+   "execution_count": 36,
+   "id": "6f9196e5",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3742,7 +3709,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b2302654",
+   "id": "b72d5080",
    "metadata": {
     "editable": true
    },
@@ -3819,7 +3786,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9f2998ff",
+   "id": "009b948b",
    "metadata": {
     "editable": true
    },
@@ -3831,7 +3798,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bc8c9873",
+   "id": "4b02a26a",
    "metadata": {
     "editable": true
    },
@@ -3850,8 +3817,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 38,
-   "id": "810f6b81",
+   "execution_count": 37,
+   "id": "efd083fc",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3885,7 +3852,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "565a1cd1",
+   "id": "b3590aea",
    "metadata": {
     "editable": true
    },
@@ -3900,7 +3867,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0a65c309",
+   "id": "9592b7bf",
    "metadata": {
     "editable": true
    },
@@ -3912,7 +3879,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8b0d4688",
+   "id": "0941f045",
    "metadata": {
     "editable": true
    },
@@ -3922,7 +3889,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d1f9ca89",
+   "id": "d47fc8d3",
    "metadata": {
     "editable": true
    },
@@ -3945,8 +3912,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 39,
-   "id": "28d4c3a0",
+   "execution_count": 38,
+   "id": "1ddb9cb5",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3990,7 +3957,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dfe532c5",
+   "id": "725b78e9",
    "metadata": {
     "editable": true
    },
@@ -4000,7 +3967,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "86f0a026",
+   "id": "b08f94e5",
    "metadata": {
     "editable": true
    },
@@ -4069,7 +4036,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fd6857da",
+   "id": "95e72a9e",
    "metadata": {
     "editable": true
    },
@@ -4082,8 +4049,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 40,
-   "id": "c84d4e1f",
+   "execution_count": 39,
+   "id": "fb3ad8e5",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4096,7 +4063,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "10579805",
+   "id": "02fe1db6",
    "metadata": {
     "editable": true
    },
@@ -4110,7 +4077,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "30c31580",
+   "id": "33a4aed5",
    "metadata": {
     "editable": true
    },
@@ -4123,7 +4090,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "af797776",
+   "id": "78a3bc86",
    "metadata": {
     "editable": true
    },
@@ -4134,7 +4101,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9c6eebe7",
+   "id": "38c3a27d",
    "metadata": {
     "editable": true
    },
@@ -4146,7 +4113,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "06a2f8be",
+   "id": "7eb5c51b",
    "metadata": {
     "editable": true
    },
@@ -4156,7 +4123,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0b5633b3",
+   "id": "3597b20a",
    "metadata": {
     "editable": true
    },
@@ -4168,7 +4135,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "108886bb",
+   "id": "82922e13",
    "metadata": {
     "editable": true
    },
@@ -4178,13 +4145,13 @@
     "\n",
     "\n",
     "**Solution.**\n",
-    "The code here is an example of where we define our own design matrix and fit parameters $\\beta$."
+    "The code here is an example of where we define our own design matrix and fit parameters $\\theta$."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 41,
-   "id": "94716007",
+   "execution_count": 40,
+   "id": "61cd693e",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4217,16 +4184,16 @@
     "X[:,2] = x**2\n",
     "# We split the data in test and training data\n",
     "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
-    "# matrix inversion to find beta\n",
-    "beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
-    "print(beta)\n",
+    "# matrix inversion to find theta\n",
+    "theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
+    "print(theta)\n",
     "# and then make the prediction\n",
-    "ytilde = X_train @ beta\n",
+    "ytilde = X_train @ theta\n",
     "print(\"Training R2\")\n",
     "print(R2(y_train,ytilde))\n",
     "print(\"Training MSE\")\n",
     "print(MSE(y_train,ytilde))\n",
-    "ypredict = X_test @ beta\n",
+    "ypredict = X_test @ theta\n",
     "print(\"Test R2\")\n",
     "print(R2(y_test,ypredict))\n",
     "print(\"Test MSE\")\n",
@@ -4235,7 +4202,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3aa5be35",
+   "id": "2f8d2e6e",
    "metadata": {
     "editable": true
    },
@@ -4245,7 +4212,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1af6d7cd",
+   "id": "c81f5caf",
    "metadata": {
     "editable": true
    },
@@ -4290,8 +4257,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 42,
-   "id": "1f4f0115",
+   "execution_count": 41,
+   "id": "37ca3335",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4304,7 +4271,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8904ba25",
+   "id": "e6925e8c",
    "metadata": {
     "editable": true
    },
@@ -4314,8 +4281,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 43,
-   "id": "04fdf05f",
+   "execution_count": 42,
+   "id": "caecb70a",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4330,7 +4297,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a22f1ef9",
+   "id": "deabdf0c",
    "metadata": {
     "editable": true
    },
@@ -4347,8 +4314,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 44,
-   "id": "ad96211c",
+   "execution_count": 43,
+   "id": "b083bb84",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4365,7 +4332,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a103eff1",
+   "id": "21102c44",
    "metadata": {
     "editable": true
    },
@@ -4380,8 +4347,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 45,
-   "id": "d9b5eed5",
+   "execution_count": 44,
+   "id": "b9108dab",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4425,7 +4392,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9decc489",
+   "id": "cd19b575",
    "metadata": {
     "editable": true
    },
@@ -4435,7 +4402,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f2ffecd1",
+   "id": "eb2f6352",
    "metadata": {
     "editable": true
    },
@@ -4446,7 +4413,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "356683e0",
+   "id": "6e09ea94",
    "metadata": {
     "editable": true
    },
@@ -4457,7 +4424,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "01abcbae",
+   "id": "38857156",
    "metadata": {
     "editable": true
    },
@@ -4468,7 +4435,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8b2a42d8",
+   "id": "6b9115a6",
    "metadata": {
     "editable": true
    },
@@ -4490,8 +4457,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 46,
-   "id": "4ae34751",
+   "execution_count": 45,
+   "id": "ae7a71c1",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4504,7 +4471,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "559e2cff",
+   "id": "ca615d39",
    "metadata": {
     "editable": true
    },
@@ -4521,7 +4488,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "03b05ffa",
+   "id": "d83c8354",
    "metadata": {
     "editable": true
    },
@@ -4534,7 +4501,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1ff2cf84",
+   "id": "74f6e912",
    "metadata": {
     "editable": true
    },
@@ -4545,7 +4512,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fe18f837",
+   "id": "93761664",
    "metadata": {
     "editable": true
    },
@@ -4557,7 +4524,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "49eff4b4",
+   "id": "be729d32",
    "metadata": {
     "editable": true
    },
@@ -4567,7 +4534,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "acc25706",
+   "id": "08549523",
    "metadata": {
     "editable": true
    },
@@ -4579,7 +4546,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5f945b64",
+   "id": "03cc0ca0",
    "metadata": {
     "editable": true
    },
@@ -4595,8 +4562,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 47,
-   "id": "f9578f4e",
+   "execution_count": 46,
+   "id": "9b1b9378",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4635,16 +4602,16 @@
     "# We split the data in test and training data\n",
     "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
     "\n",
-    "# matrix inversion to find beta\n",
-    "OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
-    "print(OLSbeta)\n",
+    "# matrix inversion to find theta\n",
+    "OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
+    "print(OLStheta)\n",
     "# and then make the prediction\n",
-    "ytildeOLS = X_train @ OLSbeta\n",
+    "ytildeOLS = X_train @ OLStheta\n",
     "print(\"Training R2 for OLS\")\n",
     "print(R2(y_train,ytildeOLS))\n",
     "print(\"Training MSE for OLS\")\n",
     "print(MSE(y_train,ytildeOLS))\n",
-    "ypredictOLS = X_test @ OLSbeta\n",
+    "ypredictOLS = X_test @ OLStheta\n",
     "print(\"Test R2 for OLS\")\n",
     "print(R2(y_test,ypredictOLS))\n",
     "print(\"Test MSE OLS\")\n",
@@ -4661,10 +4628,10 @@
     "lambdas = np.logspace(-4, 1, nlambdas)\n",
     "for i in range(nlambdas):\n",
     "    lmb = lambdas[i]\n",
-    "    OwnRidgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
+    "    OwnRidgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
     "    # and then make the prediction\n",
-    "    OwnytildeRidge = X_train @ OwnRidgebeta\n",
-    "    OwnypredictRidge = X_test @ OwnRidgebeta\n",
+    "    OwnytildeRidge = X_train @ OwnRidgetheta\n",
+    "    OwnypredictRidge = X_test @ OwnRidgetheta\n",
     "    OwnMSEPredict[i] = MSE(y_test,OwnypredictRidge)\n",
     "    OwnMSETrain[i] = MSE(y_train,OwnytildeRidge)\n",
     "    # Make the fit using Ridge from Sklearn\n",
@@ -4688,7 +4655,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "88be64bc",
+   "id": "341e9820",
    "metadata": {
     "editable": true
    },
@@ -4698,7 +4665,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3a0e09e7",
+   "id": "6feb372f",
    "metadata": {
     "editable": true
    },
@@ -4720,7 +4687,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fd41f7f2",
+   "id": "3c12062d",
    "metadata": {
     "editable": true
    },
@@ -4732,7 +4699,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "49d2bdba",
+   "id": "edfcd8ff",
    "metadata": {
     "editable": true
    },
@@ -4742,7 +4709,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fd5a139b",
+   "id": "e8689eb2",
    "metadata": {
     "editable": true
    },
@@ -4754,7 +4721,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "af22b4d2",
+   "id": "62145d09",
    "metadata": {
     "editable": true
    },
@@ -4764,7 +4731,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b120000f",
+   "id": "ec992c25",
    "metadata": {
     "editable": true
    },
@@ -4776,7 +4743,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0e7df31e",
+   "id": "4a922090",
    "metadata": {
     "editable": true
    },
@@ -4791,7 +4758,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "faa1b4d0",
+   "id": "10f294a3",
    "metadata": {
     "editable": true
    },
@@ -4803,7 +4770,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "02f78802",
+   "id": "1dfa508a",
    "metadata": {
     "editable": true
    },
@@ -4813,7 +4780,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e85b6015",
+   "id": "b5474b9f",
    "metadata": {
     "editable": true
    },
@@ -4825,7 +4792,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12ae8696",
+   "id": "4dee1baf",
    "metadata": {
     "editable": true
    },
@@ -4835,7 +4802,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d90de117",
+   "id": "9949ee71",
    "metadata": {
     "editable": true
    },
@@ -4847,7 +4814,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "15a6c139",
+   "id": "73ce2a98",
    "metadata": {
     "editable": true
    },
@@ -4857,7 +4824,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f5e7aa62",
+   "id": "41e1008d",
    "metadata": {
     "editable": true
    },
@@ -4869,7 +4836,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5a1f86bc",
+   "id": "597c4ebb",
    "metadata": {
     "editable": true
    },
@@ -4879,7 +4846,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "57018aa2",
+   "id": "e9163f17",
    "metadata": {
     "editable": true
    },
@@ -4891,7 +4858,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6a9c70a8",
+   "id": "fc3d2fdb",
    "metadata": {
     "editable": true
    },
@@ -4901,7 +4868,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "75aaf0ec",
+   "id": "e7cec690",
    "metadata": {
     "editable": true
    },
@@ -4913,7 +4880,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "31a142ea",
+   "id": "1a74bac9",
    "metadata": {
     "editable": true
    },
@@ -4923,7 +4890,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2552b361",
+   "id": "7087bc76",
    "metadata": {
     "editable": true
    },
@@ -4935,7 +4902,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "91b53dc5",
+   "id": "1e5a21fd",
    "metadata": {
     "editable": true
    },
@@ -4945,7 +4912,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bf83b147",
+   "id": "624d40e0",
    "metadata": {
     "editable": true
    },
@@ -4957,7 +4924,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0c8668a6",
+   "id": "d871f62f",
    "metadata": {
     "editable": true
    },
@@ -4967,7 +4934,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "89704701",
+   "id": "675aea37",
    "metadata": {
     "editable": true
    },
@@ -4979,7 +4946,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0054bb58",
+   "id": "746b74e2",
    "metadata": {
     "editable": true
    },
@@ -4989,7 +4956,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "24f3ee5f",
+   "id": "6381d6ee",
    "metadata": {
     "editable": true
    },
@@ -5001,7 +4968,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0827a002",
+   "id": "4331c20e",
    "metadata": {
     "editable": true
    },
diff --git a/doc/LectureNotes/chapter2.ipynb b/doc/LectureNotes/chapter2.ipynb
index 31a6d7135..d663cda6a 100644
--- a/doc/LectureNotes/chapter2.ipynb
+++ b/doc/LectureNotes/chapter2.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "74d0d498",
+   "id": "d5cb084b",
    "metadata": {
     "editable": true
    },
@@ -13,7 +13,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "acd544cd",
+   "id": "10ec5810",
    "metadata": {
     "editable": true
    },
@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0acaa875",
+   "id": "607f5658",
    "metadata": {
     "editable": true
    },
@@ -32,48 +32,48 @@
     "\n",
     "What is presented here is a mathematical analysis of various regression algorithms (ordinary least  squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). \n",
     "\n",
-    "We have shown that in ordinary least squares (OLS) the optimal parameters $\\beta$ are given by"
+    "We have shown that in ordinary least squares (OLS) the optimal parameters $\\theta$ are given by"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "0924b488",
+   "id": "d5cf7dd7",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+    "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b7307eb7",
+   "id": "4bccceb2",
    "metadata": {
     "editable": true
    },
    "source": [
-    "The **hat** over $\\boldsymbol{\\beta}$ means we have the optimal parameters after minimization of the cost function.\n",
+    "The **hat** over $\\boldsymbol{\\theta}$ means we have the optimal parameters after minimization of the cost function.\n",
     "\n",
     "This means that our best model is defined as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8096b1f0",
+   "id": "a51f281a",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+    "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d64adaf2",
+   "id": "66dd8fd1",
    "metadata": {
     "editable": true
    },
@@ -83,7 +83,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "78b35483",
+   "id": "a4914b87",
    "metadata": {
     "editable": true
    },
@@ -95,7 +95,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f4a3ff68",
+   "id": "12e868a7",
    "metadata": {
     "editable": true
    },
@@ -105,19 +105,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5704d260",
+   "id": "a76003c4",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{A}\\boldsymbol{y}.\n",
+    "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}} = \\boldsymbol{A}\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "12fa775a",
+   "id": "f894a9be",
    "metadata": {
     "editable": true
    },
@@ -130,7 +130,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a694774d",
+   "id": "1c7cf524",
    "metadata": {
     "editable": true
    },
@@ -142,7 +142,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e0775433",
+   "id": "de9e3352",
    "metadata": {
     "editable": true
    },
@@ -154,7 +154,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ec16ae7d",
+   "id": "4ece2993",
    "metadata": {
     "editable": true
    },
@@ -166,7 +166,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "258fbf6a",
+   "id": "f525eb65",
    "metadata": {
     "editable": true
    },
@@ -176,7 +176,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "05ea3605",
+   "id": "7d2849f0",
    "metadata": {
     "editable": true
    },
@@ -188,7 +188,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "db5ed2d4",
+   "id": "b214cf3a",
    "metadata": {
     "editable": true
    },
@@ -198,7 +198,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e99ad8e8",
+   "id": "d566c8b4",
    "metadata": {
     "editable": true
    },
@@ -210,7 +210,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1cc7938a",
+   "id": "cac5aeb2",
    "metadata": {
     "editable": true
    },
@@ -220,7 +220,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0e3fc9d5",
+   "id": "b349c4ef",
    "metadata": {
     "editable": true
    },
@@ -267,7 +267,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6d2bc570",
+   "id": "b5c5014d",
    "metadata": {
     "editable": true
    },
@@ -290,7 +290,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fdda638f",
+   "id": "2b07f548",
    "metadata": {
     "editable": true
    },
@@ -307,7 +307,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "29a3a620",
+   "id": "f8223e8b",
    "metadata": {
     "editable": true
    },
@@ -326,7 +326,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3278b038",
+   "id": "c1c28b92",
    "metadata": {
     "editable": true
    },
@@ -339,7 +339,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7b66f623",
+   "id": "8cebf32c",
    "metadata": {
     "editable": true
    },
@@ -349,7 +349,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "\\boldsymbol{\\beta}  =  (\\boldsymbol{X}^{T} \\boldsymbol{X})^{-1} \\boldsymbol{X}^{T} \\boldsymbol{y},\n",
+    "\\boldsymbol{\\theta}  =  (\\boldsymbol{X}^{T} \\boldsymbol{X})^{-1} \\boldsymbol{X}^{T} \\boldsymbol{y},\n",
     "\\label{_auto1} \\tag{1}\n",
     "\\end{equation}\n",
     "$$"
@@ -357,23 +357,23 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c97ae6c5",
+   "id": "4c2f125e",
    "metadata": {
     "editable": true
    },
    "source": [
     "has linearly dependent column vectors, we will not be able to compute the inverse\n",
-    "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\beta_i$. \n",
+    "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\theta_i$. \n",
     "The estimators are only well-defined if $(\\boldsymbol{X}^{T}\\boldsymbol{X})$ can be inverted. \n",
     "This is more likely to happen when the matrix $\\boldsymbol{X}$ is high-dimensional. In this case it is likely to encounter a situation where \n",
-    "the regression parameters $\\beta_i$ cannot be estimated.\n",
+    "the regression parameters $\\theta_i$ cannot be estimated.\n",
     "\n",
     "A cheap  *ad hoc* approach is  simply to add a small diagonal component to the matrix to invert, that is we change"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "af0bfc59",
+   "id": "28b01b73",
    "metadata": {
     "editable": true
    },
@@ -385,7 +385,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6469bfe8",
+   "id": "df72f01b",
    "metadata": {
     "editable": true
    },
@@ -395,7 +395,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "75d45f30",
+   "id": "62fc7dce",
    "metadata": {
     "editable": true
    },
@@ -410,7 +410,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "28b4fac2",
+   "id": "5b0639a7",
    "metadata": {
     "editable": true
    },
@@ -422,7 +422,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "085b2de7",
+   "id": "397cb734",
    "metadata": {
     "editable": true
    },
@@ -432,7 +432,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1fb42fdc",
+   "id": "05ac3f20",
    "metadata": {
     "editable": true
    },
@@ -444,7 +444,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0f7b8c0d",
+   "id": "ee59ca38",
    "metadata": {
     "editable": true
    },
@@ -454,7 +454,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "057f3ed2",
+   "id": "6c60f584",
    "metadata": {
     "editable": true
    },
@@ -466,7 +466,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "628031b9",
+   "id": "b7031193",
    "metadata": {
     "editable": true
    },
@@ -478,7 +478,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3177dcfe",
+   "id": "4c0dce45",
    "metadata": {
     "editable": true
    },
@@ -493,7 +493,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a392a54d",
+   "id": "ce330467",
    "metadata": {
     "editable": true
    },
@@ -514,7 +514,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3496079e",
+   "id": "e42d67ce",
    "metadata": {
     "editable": true
    },
@@ -526,7 +526,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "77b61c68",
+   "id": "d43e350b",
    "metadata": {
     "editable": true
    },
@@ -536,7 +536,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dbb84089",
+   "id": "4ad05081",
    "metadata": {
     "editable": true
    },
@@ -548,7 +548,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "04fc9eec",
+   "id": "7831a051",
    "metadata": {
     "editable": true
    },
@@ -594,7 +594,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4f1e3bb4",
+   "id": "706c9e36",
    "metadata": {
     "editable": true
    },
@@ -605,7 +605,7 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "a39cbeb0",
+   "id": "9605f17b",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -645,7 +645,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dca36481",
+   "id": "d6dc015b",
    "metadata": {
     "editable": true
    },
@@ -675,7 +675,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b9b3c7db",
+   "id": "2b38c905",
    "metadata": {
     "editable": true
    },
@@ -689,7 +689,7 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "6aa5981e",
+   "id": "3b82ddeb",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -701,7 +701,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae58132b",
+   "id": "8bdcf281",
    "metadata": {
     "editable": true
    },
@@ -712,7 +712,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "761d48f5",
+   "id": "8090ac1c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -751,7 +751,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6297c4ec",
+   "id": "35ee1cc9",
    "metadata": {
     "editable": true
    },
@@ -769,7 +769,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2bb1f4b8",
+   "id": "13c4c73e",
    "metadata": {
     "editable": true
    },
@@ -781,7 +781,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "832a6404",
+   "id": "392f8342",
    "metadata": {
     "editable": true
    },
@@ -792,7 +792,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "d393c8c7",
+   "id": "3d64803c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -826,7 +826,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3d0570c2",
+   "id": "a48cdbe7",
    "metadata": {
     "editable": true
    },
@@ -836,7 +836,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "071ff3b6",
+   "id": "22f55640",
    "metadata": {
     "editable": true
    },
@@ -850,7 +850,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "113247f9",
+   "id": "88772ea4",
    "metadata": {
     "editable": true
    },
@@ -869,7 +869,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "05603c38",
+   "id": "4f144230",
    "metadata": {
     "editable": true
    },
@@ -879,7 +879,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8848e319",
+   "id": "c1a77d69",
    "metadata": {
     "editable": true
    },
@@ -891,7 +891,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5b4146e5",
+   "id": "c77977a5",
    "metadata": {
     "editable": true
    },
@@ -905,7 +905,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "43701d21",
+   "id": "b3bac40c",
    "metadata": {
     "editable": true
    },
@@ -917,7 +917,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7cdca00d",
+   "id": "a5803728",
    "metadata": {
     "editable": true
    },
@@ -929,7 +929,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "add83821",
+   "id": "21edee39",
    "metadata": {
     "editable": true
    },
@@ -946,7 +946,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3746dd8d",
+   "id": "3e23d609",
    "metadata": {
     "editable": true
    },
@@ -956,7 +956,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae591c49",
+   "id": "64ec1fc5",
    "metadata": {
     "editable": true
    },
@@ -972,7 +972,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "63009724",
+   "id": "56d47d93",
    "metadata": {
     "editable": true
    },
@@ -982,7 +982,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c674e8da",
+   "id": "2fda48ae",
    "metadata": {
     "editable": true
    },
@@ -998,7 +998,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9f60659a",
+   "id": "824ffed2",
    "metadata": {
     "editable": true
    },
@@ -1008,7 +1008,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "27e17584",
+   "id": "0ac7a41e",
    "metadata": {
     "editable": true
    },
@@ -1024,7 +1024,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3efc62c4",
+   "id": "8d0dd964",
    "metadata": {
     "editable": true
    },
@@ -1034,7 +1034,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "195d30c4",
+   "id": "30282069",
    "metadata": {
     "editable": true
    },
@@ -1051,7 +1051,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a00d95e8",
+   "id": "064c0e6b",
    "metadata": {
     "editable": true
    },
@@ -1065,7 +1065,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "985428ec",
+   "id": "273056c4",
    "metadata": {
     "editable": true
    },
@@ -1077,7 +1077,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0affd014",
+   "id": "e38034e4",
    "metadata": {
     "editable": true
    },
@@ -1087,7 +1087,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "146dcd49",
+   "id": "8cc943db",
    "metadata": {
     "editable": true
    },
@@ -1099,7 +1099,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "63bc8186",
+   "id": "9dfc2966",
    "metadata": {
     "editable": true
    },
@@ -1111,7 +1111,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4bea8a7d",
+   "id": "8ca54207",
    "metadata": {
     "editable": true
    },
@@ -1123,7 +1123,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5321e8fc",
+   "id": "a986b780",
    "metadata": {
     "editable": true
    },
@@ -1133,7 +1133,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c16d0109",
+   "id": "2c31d500",
    "metadata": {
     "editable": true
    },
@@ -1145,7 +1145,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4ca7bc79",
+   "id": "9e4536a0",
    "metadata": {
     "editable": true
    },
@@ -1155,7 +1155,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "795969aa",
+   "id": "92636687",
    "metadata": {
     "editable": true
    },
@@ -1167,7 +1167,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "36fd11ec",
+   "id": "a665d593",
    "metadata": {
     "editable": true
    },
@@ -1177,7 +1177,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8b59e361",
+   "id": "769bb394",
    "metadata": {
     "editable": true
    },
@@ -1189,7 +1189,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4212728f",
+   "id": "baf742bb",
    "metadata": {
     "editable": true
    },
@@ -1201,7 +1201,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8d6940eb",
+   "id": "7fd2be6b",
    "metadata": {
     "editable": true
    },
@@ -1211,7 +1211,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eabfdb9c",
+   "id": "e488ea83",
    "metadata": {
     "editable": true
    },
@@ -1223,7 +1223,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bbc85eab",
+   "id": "76ac57c1",
    "metadata": {
     "editable": true
    },
@@ -1234,7 +1234,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f5b1bf52",
+   "id": "b789d545",
    "metadata": {
     "editable": true
    },
@@ -1246,7 +1246,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "efcc256a",
+   "id": "fd7008f9",
    "metadata": {
     "editable": true
    },
@@ -1256,7 +1256,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0650a070",
+   "id": "9d37f0bf",
    "metadata": {
     "editable": true
    },
@@ -1268,7 +1268,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c168d425",
+   "id": "0fd2b28c",
    "metadata": {
     "editable": true
    },
@@ -1278,7 +1278,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "30fb2230",
+   "id": "98432443",
    "metadata": {
     "editable": true
    },
@@ -1290,7 +1290,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d46b1fd5",
+   "id": "29174a30",
    "metadata": {
     "editable": true
    },
@@ -1301,7 +1301,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "16a93afd",
+   "id": "d07b5e5c",
    "metadata": {
     "editable": true
    },
@@ -1313,7 +1313,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "80e47710",
+   "id": "21892e7e",
    "metadata": {
     "editable": true
    },
@@ -1331,7 +1331,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1dc0f98b",
+   "id": "a4ed7dd4",
    "metadata": {
     "editable": true
    },
@@ -1347,19 +1347,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c5f9ce8b",
+   "id": "c7aafbc8",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial^2 C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
+    "\\frac{\\partial^2 C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}^T\\partial \\boldsymbol{\\theta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c803fdb7",
+   "id": "f80de4f9",
    "metadata": {
     "editable": true
    },
@@ -1371,7 +1371,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eef3c89e",
+   "id": "8ec45bf8",
    "metadata": {
     "editable": true
    },
@@ -1383,7 +1383,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e886f303",
+   "id": "4554f86d",
    "metadata": {
     "editable": true
    },
@@ -1402,7 +1402,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "efcb2b9e",
+   "id": "c5c46053",
    "metadata": {
     "editable": true
    },
@@ -1416,7 +1416,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "caf380d7",
+   "id": "0d367725",
    "metadata": {
     "editable": true
    },
@@ -1426,7 +1426,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9d676665",
+   "id": "ca783893",
    "metadata": {
     "editable": true
    },
@@ -1438,7 +1438,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "301b0d53",
+   "id": "81dacef8",
    "metadata": {
     "editable": true
    },
@@ -1448,7 +1448,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7c134c39",
+   "id": "10790e67",
    "metadata": {
     "editable": true
    },
@@ -1460,7 +1460,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12aec852",
+   "id": "0df66dcf",
    "metadata": {
     "editable": true
    },
@@ -1470,7 +1470,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "54a5de6b",
+   "id": "021fd105",
    "metadata": {
     "editable": true
    },
@@ -1484,7 +1484,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3edc1dd1",
+   "id": "9cee5e0c",
    "metadata": {
     "editable": true
    },
@@ -1507,7 +1507,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9d2d7b4e",
+   "id": "30ea5811",
    "metadata": {
     "editable": true
    },
@@ -1519,7 +1519,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dcee4258",
+   "id": "a129d192",
    "metadata": {
     "editable": true
    },
@@ -1532,7 +1532,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "25100476",
+   "id": "6d5c0a69",
    "metadata": {
     "editable": true
    },
@@ -1546,7 +1546,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5c46fdef",
+   "id": "79a613b9",
    "metadata": {
     "editable": true
    },
@@ -1559,7 +1559,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dcfa6d43",
+   "id": "2d0dabde",
    "metadata": {
     "editable": true
    },
@@ -1578,7 +1578,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "89dd5e4e",
+   "id": "2731cd83",
    "metadata": {
     "editable": true
    },
@@ -1590,7 +1590,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7edacd6c",
+   "id": "82de2dcc",
    "metadata": {
     "editable": true
    },
@@ -1602,7 +1602,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "94e8929f",
+   "id": "e8bf54be",
    "metadata": {
     "editable": true
    },
@@ -1612,7 +1612,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3f10861a",
+   "id": "a0597d71",
    "metadata": {
     "editable": true
    },
@@ -1624,7 +1624,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "173fbc84",
+   "id": "d74275c9",
    "metadata": {
     "editable": true
    },
@@ -1637,7 +1637,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0913dc51",
+   "id": "df0589bd",
    "metadata": {
     "editable": true
    },
@@ -1656,7 +1656,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "70346587",
+   "id": "c23bd593",
    "metadata": {
     "editable": true
    },
@@ -1666,7 +1666,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "86feec9f",
+   "id": "8806c741",
    "metadata": {
     "editable": true
    },
@@ -1685,7 +1685,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "25ed76e9",
+   "id": "ae1e6181",
    "metadata": {
     "editable": true
    },
@@ -1701,7 +1701,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cea26675",
+   "id": "9a4af97a",
    "metadata": {
     "editable": true
    },
@@ -1715,7 +1715,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1fa1423f",
+   "id": "607da328",
    "metadata": {
     "editable": true
    },
@@ -1730,7 +1730,7 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "f277b044",
+   "id": "afa9a0d9",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1751,7 +1751,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "02205795",
+   "id": "8acdd66a",
    "metadata": {
     "editable": true
    },
@@ -1766,7 +1766,7 @@
   {
    "cell_type": "code",
    "execution_count": 6,
-   "id": "6c182a7d",
+   "id": "4886841b",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1798,7 +1798,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "16ff9454",
+   "id": "90b67d9e",
    "metadata": {
     "editable": true
    },
@@ -1815,7 +1815,7 @@
   {
    "cell_type": "code",
    "execution_count": 7,
-   "id": "903635fb",
+   "id": "2dc76e27",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1840,7 +1840,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "034c38ef",
+   "id": "3b9c94f0",
    "metadata": {
     "editable": true
    },
@@ -1851,7 +1851,7 @@
   {
    "cell_type": "code",
    "execution_count": 8,
-   "id": "91afb8cb",
+   "id": "f12e4764",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1877,7 +1877,7 @@
     "\t\ty = np.ravel(y)\n",
     "\n",
     "\tN = len(x)\n",
-    "\tl = int((n+1)*(n+2)/2)\t\t# Number of elements in beta\n",
+    "\tl = int((n+1)*(n+2)/2)\t\t# Number of elements in theta\n",
     "\tX = np.ones((N,l))\n",
     "\n",
     "\tfor i in range(1,n+1):\n",
@@ -1905,7 +1905,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9d8768f8",
+   "id": "6d8aa859",
    "metadata": {
     "editable": true
    },
@@ -1924,7 +1924,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a7de38a0",
+   "id": "d4715f97",
    "metadata": {
     "editable": true
    },
@@ -1936,7 +1936,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4855248a",
+   "id": "1800f79e",
    "metadata": {
     "editable": true
    },
@@ -1946,7 +1946,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3a150f73",
+   "id": "dac62d99",
    "metadata": {
     "editable": true
    },
@@ -1963,7 +1963,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6ab3f1f7",
+   "id": "97c7d1d3",
    "metadata": {
     "editable": true
    },
@@ -1973,7 +1973,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae35f475",
+   "id": "5645cf35",
    "metadata": {
     "editable": true
    },
@@ -1988,7 +1988,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9f6b4b6b",
+   "id": "ae8afe59",
    "metadata": {
     "editable": true
    },
@@ -1998,7 +1998,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "af0c59a3",
+   "id": "3bae9bfc",
    "metadata": {
     "editable": true
    },
@@ -2012,7 +2012,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "824ae63b",
+   "id": "b60c8bb6",
    "metadata": {
     "editable": true
    },
@@ -2026,7 +2026,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5b480160",
+   "id": "d8c6fa53",
    "metadata": {
     "editable": true
    },
@@ -2038,7 +2038,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "506a98da",
+   "id": "074c5673",
    "metadata": {
     "editable": true
    },
@@ -2050,7 +2050,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "72b1e665",
+   "id": "8320630a",
    "metadata": {
     "editable": true
    },
@@ -2060,7 +2060,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6ea9123f",
+   "id": "2eebcb57",
    "metadata": {
     "editable": true
    },
@@ -2072,7 +2072,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ab471475",
+   "id": "7a7c1b9c",
    "metadata": {
     "editable": true
    },
@@ -2082,7 +2082,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ff364443",
+   "id": "c89ccbc8",
    "metadata": {
     "editable": true
    },
@@ -2099,7 +2099,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4619ec8f",
+   "id": "b4fc87f6",
    "metadata": {
     "editable": true
    },
@@ -2109,7 +2109,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "529d5ed0",
+   "id": "15d7da17",
    "metadata": {
     "editable": true
    },
@@ -2121,7 +2121,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "762e07ea",
+   "id": "ee6e0550",
    "metadata": {
     "editable": true
    },
@@ -2131,7 +2131,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f45c2ce1",
+   "id": "2789c2b4",
    "metadata": {
     "editable": true
    },
@@ -2143,7 +2143,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "35835939",
+   "id": "1b0a4cce",
    "metadata": {
     "editable": true
    },
@@ -2155,7 +2155,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "901c3505",
+   "id": "ad73966d",
    "metadata": {
     "editable": true
    },
@@ -2167,7 +2167,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "63486657",
+   "id": "107fdb0a",
    "metadata": {
     "editable": true
    },
@@ -2189,7 +2189,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b8cb7b04",
+   "id": "8d070a5b",
    "metadata": {
     "editable": true
    },
@@ -2201,7 +2201,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8962eeb3",
+   "id": "356bded4",
    "metadata": {
     "editable": true
    },
@@ -2218,7 +2218,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ff83bf5a",
+   "id": "2b232f6c",
    "metadata": {
     "editable": true
    },
@@ -2230,7 +2230,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "67c405ee",
+   "id": "9ec41351",
    "metadata": {
     "editable": true
    },
@@ -2240,7 +2240,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d2279b06",
+   "id": "5fd89419",
    "metadata": {
     "editable": true
    },
@@ -2252,7 +2252,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c08bd913",
+   "id": "22a2cb66",
    "metadata": {
     "editable": true
    },
@@ -2262,7 +2262,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "585fcec4",
+   "id": "ac6721a2",
    "metadata": {
     "editable": true
    },
@@ -2274,7 +2274,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c5ee7c18",
+   "id": "da33b69e",
    "metadata": {
     "editable": true
    },
@@ -2284,7 +2284,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "250d6a7b",
+   "id": "44110c4b",
    "metadata": {
     "editable": true
    },
@@ -2296,7 +2296,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "26dca3eb",
+   "id": "de0ee363",
    "metadata": {
     "editable": true
    },
@@ -2313,7 +2313,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "43ae52cf",
+   "id": "3c2b4cb7",
    "metadata": {
     "editable": true
    },
@@ -2326,19 +2326,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fa5c2da2",
+   "id": "b8979ebf",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
+    "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ef125654",
+   "id": "f635713d",
    "metadata": {
     "editable": true
    },
@@ -2348,20 +2348,20 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3e398c3d",
+   "id": "e335573e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
-    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2,\n",
+    "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n",
+    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6e4285d1",
+   "id": "197171c0",
    "metadata": {
     "editable": true
    },
@@ -2371,7 +2371,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c1d8fde6",
+   "id": "494553da",
    "metadata": {
     "editable": true
    },
@@ -2383,57 +2383,57 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5b082cd2",
+   "id": "d131f255",
    "metadata": {
     "editable": true
    },
    "source": [
     "By minimizing the above equation with respect to the parameters\n",
-    "$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n",
-    "parameters $\\boldsymbol{\\beta}$.  We can add a regularization parameter $\\lambda$ by\n",
+    "$\\boldsymbol{\\theta}$ we could then obtain an analytical expression for the\n",
+    "parameters $\\boldsymbol{\\theta}$.  We can add a regularization parameter $\\lambda$ by\n",
     "defining a new cost function to be optimized, that is"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "306b6062",
+   "id": "3992861b",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
-    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\n",
+    "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n",
+    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f9c22f99",
+   "id": "e210e429",
    "metadata": {
     "editable": true
    },
    "source": [
     "which leads to the Ridge regression minimization problem where we\n",
-    "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n",
+    "require that $\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\\le t$, where $t$ is\n",
     "a finite number larger than zero. By defining"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8fa998f9",
+   "id": "960bf071",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n",
+    "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "954b3b63",
+   "id": "69ff9b18",
    "metadata": {
     "editable": true
    },
@@ -2443,20 +2443,20 @@
   },
   {
    "cell_type": "markdown",
-   "id": "40d226a6",
+   "id": "3b619870",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
-    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1\n",
+    "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n",
+    "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "571654e9",
+   "id": "020995b7",
    "metadata": {
     "editable": true
    },
@@ -2468,7 +2468,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4b96f90f",
+   "id": "dc41832a",
    "metadata": {
     "editable": true
    },
@@ -2480,7 +2480,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6af80b33",
+   "id": "e4a20e48",
    "metadata": {
     "editable": true
    },
@@ -2490,25 +2490,25 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ed4e7687",
+   "id": "03ea4d0d",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n",
+    "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\boldsymbol{\\theta}^T\\boldsymbol{\\theta},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2eb319cc",
+   "id": "6734e20a",
    "metadata": {
     "editable": true
    },
    "source": [
     "and \n",
-    "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n",
+    "taking the derivatives with respect to $\\boldsymbol{\\theta}$ we obtain then\n",
     "a slightly modified matrix inversion problem which for finite values\n",
     "of $\\lambda$ does not suffer from singularity problems. We obtain\n",
     "the optimal parameters"
@@ -2516,19 +2516,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8528934c",
+   "id": "502e8acf",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
+    "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "22a7e810",
+   "id": "fcb7eeca",
    "metadata": {
     "editable": true
    },
@@ -2538,19 +2538,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "01d9c1c9",
+   "id": "f9789400",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n",
+    "\\sum_{i=0}^{p-1} \\theta_i^2 \\leq t,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5d910fd1",
+   "id": "cce5edcd",
    "metadata": {
     "editable": true
    },
@@ -2562,19 +2562,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b8bbfa39",
+   "id": "8b0e4f12",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
+    "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "51ac6ef1",
+   "id": "c59da228",
    "metadata": {
     "editable": true
    },
@@ -2585,7 +2585,7 @@
     "modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n",
     "particular for our discussion of the bias-variance tradeoff are rather\n",
     "interesting. We will see that for specific values of $\\lambda$, we may\n",
-    "even reduce the variance of the optimal parameters $\\boldsymbol{\\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n",
+    "even reduce the variance of the optimal parameters $\\boldsymbol{\\theta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n",
     "\n",
     "Using our insights about the SVD of the design matrix $\\boldsymbol{X}$ \n",
     "We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\\boldsymbol{U}$ as"
@@ -2593,19 +2593,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12ee6646",
+   "id": "96960524",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta}  =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n",
+    "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\theta}  =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "22852ccf",
+   "id": "ef0014d1",
    "metadata": {
     "editable": true
    },
@@ -2615,19 +2615,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "68d87eac",
+   "id": "57ba3f2e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n",
+    "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\theta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "0219a1a8",
+   "id": "86313c14",
    "metadata": {
     "editable": true
    },
@@ -2639,7 +2639,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "be1f339c",
+   "id": "c5c6f72d",
    "metadata": {
     "editable": true
    },
@@ -2651,7 +2651,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e1d5221",
+   "id": "4f780155",
    "metadata": {
     "editable": true
    },
@@ -2669,7 +2669,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0b4bc87c",
+   "id": "08413d65",
    "metadata": {
     "editable": true
    },
@@ -2681,7 +2681,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "de7e0986",
+   "id": "10f8be87",
    "metadata": {
     "editable": true
    },
@@ -2691,19 +2691,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0ae7dd26",
+   "id": "9d2b90ba",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n",
+    "\\boldsymbol{\\theta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5caa1086",
+   "id": "367f8b0e",
    "metadata": {
     "editable": true
    },
@@ -2713,19 +2713,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c737b14f",
+   "id": "cd4334f5",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n",
+    "\\boldsymbol{\\theta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\theta}^{\\mathrm{OLS}},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "60570cf0",
+   "id": "06f7db12",
    "metadata": {
     "editable": true
    },
@@ -2741,41 +2741,41 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f0c2e386",
+   "id": "b81118eb",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
+    "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5f3e91d2",
+   "id": "7d2b22a9",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)"
+    "Taking the derivative with respect to $\\boldsymbol{\\theta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "66b637e8",
+   "id": "2648d39f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{d \\vert \\beta\\vert}{d \\boldsymbol{\\beta}}=\\mathrm{sgn}(\\boldsymbol{\\beta})=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\-1 & \\beta < 0, \\end{array}\\right.\n",
+    "\\frac{d \\vert \\theta\\vert}{d \\boldsymbol{\\theta}}=\\mathrm{sgn}(\\boldsymbol{\\theta})=\\left\\{\\begin{array}{cc} 1 & \\theta > 0 \\\\-1 & \\theta < 0, \\end{array}\\right.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4082c969",
+   "id": "ea227065",
    "metadata": {
     "editable": true
    },
@@ -2785,19 +2785,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2c0b085b",
+   "id": "ce21fc5d",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n",
+    "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})+\\lambda sgn(\\boldsymbol{\\theta})=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "bf355533",
+   "id": "f4d13288",
    "metadata": {
     "editable": true
    },
@@ -2807,19 +2807,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "928d6cb3",
+   "id": "116661fd",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+    "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta}+\\lambda sgn(\\boldsymbol{\\theta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "39a40fd9",
+   "id": "dc54e1a7",
    "metadata": {
     "editable": true
    },
@@ -2829,24 +2829,24 @@
     "Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the\n",
     "diagonal. In this case we have an equal number of rows and columns $n=p$.\n",
     "\n",
-    "Our model approximation is just $\\tilde{\\boldsymbol{y}}=\\boldsymbol{\\beta}$ and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term $1/n$)"
+    "Our model approximation is just $\\tilde{\\boldsymbol{y}}=\\boldsymbol{\\theta}$ and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term $1/n$)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5d891e5c",
+   "id": "87c27103",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2,\n",
+    "C(\\boldsymbol{\\theta})=\\sum_{i=0}^{p-1}(y_i-\\theta_i)^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f9ea4d16",
+   "id": "530dada7",
    "metadata": {
     "editable": true
    },
@@ -2856,19 +2856,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "59b56446",
+   "id": "2e70789b",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\beta}_i^{\\mathrm{OLS}} = y_i.\n",
+    "\\hat{\\theta}_i^{\\mathrm{OLS}} = y_i.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b178aedb",
+   "id": "a69b1fda",
    "metadata": {
     "editable": true
    },
@@ -2878,19 +2878,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3c7c60e4",
+   "id": "6dbb9f75",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,\n",
+    "C(\\boldsymbol{\\theta})=\\sum_{i=0}^{p-1}(y_i-\\theta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\theta_i^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2f5d3c17",
+   "id": "fe4b2167",
    "metadata": {
     "editable": true
    },
@@ -2900,19 +2900,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2a612078",
+   "id": "c7ad140a",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\beta}_i^{\\mathrm{Ridge}} = \\frac{y_i}{1+\\lambda}.\n",
+    "\\hat{\\theta}_i^{\\mathrm{Ridge}} = \\frac{y_i}{1+\\lambda}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "dfb11d07",
+   "id": "6dab591b",
    "metadata": {
     "editable": true
    },
@@ -2922,19 +2922,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9331f4a6",
+   "id": "bfa5b9f9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\beta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\beta_i^2},\n",
+    "C(\\boldsymbol{\\theta})=\\sum_{i=0}^{p-1}(y_i-\\theta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\theta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\theta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\theta_i^2},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b4661ad6",
+   "id": "3125ab9c",
    "metadata": {
     "editable": true
    },
@@ -2944,19 +2944,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "60adb333",
+   "id": "309aed47",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "-2\\sum_{i=0}^{p-1}(y_i-\\beta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\beta_i)}{\\vert\\beta_i\\vert}=0,\n",
+    "-2\\sum_{i=0}^{p-1}(y_i-\\theta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\theta_i)}{\\vert\\theta_i\\vert}=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d5fa48d0",
+   "id": "b863d5f9",
    "metadata": {
     "editable": true
    },
@@ -2966,13 +2966,13 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bee441a6",
+   "id": "ffaed247",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n",
+    "\\hat{\\boldsymbol{\\theta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n",
     "                                                          y_i+\\frac{\\lambda}{2} &\\mathrm{if} & y_i< -\\frac{\\lambda}{2}\\\\\n",
     "\t\t\t\t\t\t\t  0 &\\mathrm{if} & \\vert y_i\\vert\\le  \\frac{\\lambda}{2}\\end{array}\\right.\\\\.\n",
     "$$"
@@ -2980,12 +2980,12 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dc6929bb",
+   "id": "6b89a6c5",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$.\n",
+    "Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\\theta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\theta_i$ as function of $\\lambda$.\n",
     "\n",
     "As another example, \n",
     "let us assume we have a data set with outputs/targets given by the vector"
@@ -2993,7 +2993,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "954061f9",
+   "id": "adadb0c3",
    "metadata": {
     "editable": true
    },
@@ -3005,7 +3005,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "97efa82b",
+   "id": "63968be0",
    "metadata": {
     "editable": true
    },
@@ -3015,7 +3015,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f2ed5f9b",
+   "id": "48e3a74e",
    "metadata": {
     "editable": true
    },
@@ -3027,31 +3027,31 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1c424fc1",
+   "id": "5948f0d4",
    "metadata": {
     "editable": true
    },
    "source": [
-    "meaning that we have two features and two unknown parameters $\\beta_0$ and $\\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression.\n",
+    "meaning that we have two features and two unknown parameters $\\theta_0$ and $\\theta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression.\n",
     "\n",
     "For ordinary least squares (OLS) we know that the optimal solution is"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "95467447",
+   "id": "55c9c8c1",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+    "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5da0739a",
+   "id": "42fd5011",
    "metadata": {
     "editable": true
    },
@@ -3061,19 +3061,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "47e7e72f",
+   "id": "cf6a68ad",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n",
+    "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "974570c8",
+   "id": "cf208d17",
    "metadata": {
     "editable": true
    },
@@ -3085,19 +3085,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dd9e2341",
+   "id": "60413571",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+    "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "70f08735",
+   "id": "8ff207f9",
    "metadata": {
     "editable": true
    },
@@ -3107,25 +3107,25 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c0a08b76",
+   "id": "846f76e7",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n",
+    "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1d8ed72a",
+   "id": "c3d07bb7",
    "metadata": {
     "editable": true
    },
    "source": [
-    "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2$ via the parameter $\\lambda$.\n",
-    "Let us for simplicity assume that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\beta$ and $\\lambda$.\n",
+    "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2$ via the parameter $\\lambda$.\n",
+    "Let us for simplicity assume that $\\theta_0^2+\\theta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\theta$ and $\\lambda$.\n",
     "\n",
     "To see this, let us write the cost function for Ridge regression.  \n",
     "\n",
@@ -3134,85 +3134,85 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9d4ef6fd",
+   "id": "ec07f63c",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{X}\\boldsymbol{\\beta}=\\begin{bmatrix} 2\\beta_0 \\\\ \\beta_1 \\\\0 \\end{bmatrix},\n",
+    "\\boldsymbol{X}\\boldsymbol{\\theta}=\\begin{bmatrix} 2\\theta_0 \\\\ \\theta_1 \\\\0 \\end{bmatrix},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "3ba1fd07",
+   "id": "40bd6afb",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\beta_0^2+\\beta_1^2),\n",
+    "C(\\boldsymbol{\\theta})=(4-2\\theta_0)^2+(2-\\theta_1)^2+\\lambda(\\theta_0^2+\\theta_1^2),\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2c6a0484",
+   "id": "b1d4b933",
    "metadata": {
     "editable": true
    },
    "source": [
-    "and taking the derivative with respect to $\\beta_0$ we get"
+    "and taking the derivative with respect to $\\theta_0$ we get"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7f86e745",
+   "id": "63371364",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0=\\frac{8}{4+\\lambda},\n",
+    "\\theta_0=\\frac{8}{4+\\lambda},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "46c8a1af",
+   "id": "71fb4eb4",
    "metadata": {
     "editable": true
    },
    "source": [
-    "and for $\\beta_1$ we obtain"
+    "and for $\\theta_1$ we obtain"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "33d6b311",
+   "id": "cacea391",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_1=\\frac{2}{1+\\lambda},\n",
+    "\\theta_1=\\frac{2}{1+\\lambda},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1e4eae3a",
+   "id": "a65f9b33",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Using the constraint for $\\beta_0^2+\\beta_1^2=1$ we can constrain $\\lambda$ by solving"
+    "Using the constraint for $\\theta_0^2+\\theta_1^2=1$ we can constrain $\\lambda$ by solving"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7a68a399",
+   "id": "96471d24",
    "metadata": {
     "editable": true
    },
@@ -3224,44 +3224,44 @@
   },
   {
    "cell_type": "markdown",
-   "id": "89874fbd",
+   "id": "ff4a5368",
    "metadata": {
     "editable": true
    },
    "source": [
-    "which gives $\\lambda=4.571$ and $\\beta_0=0.933$ and $\\beta_1=0.359$.\n",
+    "which gives $\\lambda=4.571$ and $\\theta_0=0.933$ and $\\theta_1=0.359$.\n",
     "\n",
-    "For Lasso we need now, keeping a  constraint on $\\vert\\beta_0\\vert+\\vert\\beta_1\\vert=1$,  to take the derivative of the absolute values of $\\beta_0$\n",
-    "and $\\beta_1$. This gives us the following derivatives of the cost function"
+    "For Lasso we need now, keeping a  constraint on $\\vert\\theta_0\\vert+\\vert\\theta_1\\vert=1$,  to take the derivative of the absolute values of $\\theta_0$\n",
+    "and $\\theta_1$. This gives us the following derivatives of the cost function"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a849fabc",
+   "id": "cbaab2b3",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\vert\\beta_0\\vert+\\vert\\beta_1\\vert),\n",
+    "C(\\boldsymbol{\\theta})=(4-2\\theta_0)^2+(2-\\theta_1)^2+\\lambda(\\vert\\theta_0\\vert+\\vert\\theta_1\\vert),\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "445f0b08",
+   "id": "8074c920",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-4(4-2\\beta_0)+\\lambda\\mathrm{sgn}(\\beta_0)=0,\n",
+    "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_0}=-4(4-2\\theta_0)+\\lambda\\mathrm{sgn}(\\theta_0)=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "669385c5",
+   "id": "339ebabc",
    "metadata": {
     "editable": true
    },
@@ -3271,50 +3271,50 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d5d14c30",
+   "id": "8803be3d",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}(\\beta_1)=0.\n",
+    "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_1}=-2(2-\\theta_1)+\\lambda\\mathrm{sgn}(\\theta_1)=0.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1db05ce9",
+   "id": "a3af4674",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We have now four cases to solve besides the trivial cases $\\beta_0$ and/or $\\beta_1$ are zero, namely\n",
-    "1. $\\beta_0 > 0$ and $\\beta_1 > 0$,\n",
+    "We have now four cases to solve besides the trivial cases $\\theta_0$ and/or $\\theta_1$ are zero, namely\n",
+    "1. $\\theta_0 > 0$ and $\\theta_1 > 0$,\n",
     "\n",
-    "2. $\\beta_0 > 0$ and $\\beta_1 < 0$,\n",
+    "2. $\\theta_0 > 0$ and $\\theta_1 < 0$,\n",
     "\n",
-    "3. $\\beta_0 < 0$ and $\\beta_1 > 0$,\n",
+    "3. $\\theta_0 < 0$ and $\\theta_1 > 0$,\n",
     "\n",
-    "4. $\\beta_0 < 0$ and $\\beta_1 < 0$.\n",
+    "4. $\\theta_0 < 0$ and $\\theta_1 < 0$.\n",
     "\n",
     "If we consider the first case, we have then"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "48598bde",
+   "id": "c77ef35e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "-4(4-2\\beta_0)+\\lambda=0,\n",
+    "-4(4-2\\theta_0)+\\lambda=0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "458cc863",
+   "id": "69c8185d",
    "metadata": {
     "editable": true
    },
@@ -3324,19 +3324,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ff948eca",
+   "id": "304a2df1",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "-2(2-\\beta_1)+\\lambda=0.\n",
+    "-2(2-\\theta_1)+\\lambda=0.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "38dfba54",
+   "id": "55dd00fc",
    "metadata": {
     "editable": true
    },
@@ -3346,19 +3346,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bc715180",
+   "id": "d8f3cb2f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0=\\frac{16+\\lambda}{8},\n",
+    "\\theta_0=\\frac{16+\\lambda}{8},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "62e9a17c",
+   "id": "2db2c43e",
    "metadata": {
     "editable": true
    },
@@ -3368,24 +3368,24 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e564b775",
+   "id": "d875947f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_1=\\frac{4+\\lambda}{2}.\n",
+    "\\theta_1=\\frac{4+\\lambda}{2}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2bdbcac1",
+   "id": "a4ebed7b",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Using the constraint on $\\beta_0$ and $\\beta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you.\n",
+    "Using the constraint on $\\theta_0$ and $\\theta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you.\n",
     "\n",
     "Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of $\\lambda$, meaning that we need to perform a search in order to find the optimal values.\n",
     "\n",
@@ -3396,7 +3396,7 @@
   {
    "cell_type": "code",
    "execution_count": 9,
-   "id": "a1e3dee1",
+   "id": "ce6cdf5c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3424,14 +3424,14 @@
     "y = np.array( [4, 2, 3])\n",
     "\n",
     "\n",
-    "# matrix inversion to find beta\n",
-    "OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y\n",
-    "print(OLSbeta)\n",
+    "# matrix inversion to find theta\n",
+    "OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y\n",
+    "print(OLStheta)\n",
     "# and then make the prediction\n",
-    "ytildeOLS = X @ OLSbeta\n",
+    "ytildeOLS = X @ OLStheta\n",
     "print(\"Training MSE for OLS\")\n",
     "print(MSE(y,ytildeOLS))\n",
-    "ypredictOLS = X @ OLSbeta\n",
+    "ypredictOLS = X @ OLStheta\n",
     "\n",
     "# Repeat now for Ridge regression and various values of the regularization parameter\n",
     "I = np.eye(2,2)\n",
@@ -3441,10 +3441,10 @@
     "lambdas = np.logspace(-4, 4, nlambdas)\n",
     "for i in range(nlambdas):\n",
     "    lmb = lambdas[i]\n",
-    "    Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n",
-    "#    print(Ridgebeta)\n",
+    "    Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n",
+    "#    print(Ridgetheta)\n",
     "    # and then make the prediction\n",
-    "    ypredictRidge = X @ Ridgebeta\n",
+    "    ypredictRidge = X @ Ridgetheta\n",
     "    MSEPredict[i] = MSE(y,ypredictRidge)\n",
     "#    print(MSEPredict[i])\n",
     "    # Now plot the results\n",
@@ -3458,15 +3458,15 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5973148f",
+   "id": "79720acf",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\\boldsymbol{\\beta}$, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of $\\tilde{\\boldsymbol{y}}$ approach zero.\n",
+    "We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\\boldsymbol{\\theta}$, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of $\\tilde{\\boldsymbol{y}}$ approach zero.\n",
     "\n",
     "This happens also for Lasso regression, as seen from the next code\n",
-    "output. The difference is that Lasso shrinks the values of $\\beta$ to\n",
+    "output. The difference is that Lasso shrinks the values of $\\theta$ to\n",
     "zero at a much earlier stage and the results flatten out. We see that\n",
     "Lasso gives also an excellent fit for small values of $\\lambda$ and\n",
     "shows the best performance of the three regression methods."
@@ -3475,7 +3475,7 @@
   {
    "cell_type": "code",
    "execution_count": 10,
-   "id": "a961f69c",
+   "id": "ac1089e2",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3502,14 +3502,14 @@
     "y = np.array( [4, 2, 3])\n",
     "\n",
     "\n",
-    "# matrix inversion to find beta\n",
-    "OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y\n",
-    "print(OLSbeta)\n",
+    "# matrix inversion to find theta\n",
+    "OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y\n",
+    "print(OLStheta)\n",
     "# and then make the prediction\n",
-    "ytildeOLS = X @ OLSbeta\n",
+    "ytildeOLS = X @ OLStheta\n",
     "print(\"Training MSE for OLS\")\n",
     "print(MSE(y,ytildeOLS))\n",
-    "ypredictOLS = X @ OLSbeta\n",
+    "ypredictOLS = X @ OLStheta\n",
     "\n",
     "# Repeat now for Ridge regression and various values of the regularization parameter\n",
     "I = np.eye(2,2)\n",
@@ -3520,10 +3520,10 @@
     "lambdas = np.logspace(-4, 4, nlambdas)\n",
     "for i in range(nlambdas):\n",
     "    lmb = lambdas[i]\n",
-    "    Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n",
-    "    print(Ridgebeta)\n",
+    "    Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n",
+    "    print(Ridgetheta)\n",
     "    # and then make the prediction\n",
-    "    ypredictRidge = X @ Ridgebeta\n",
+    "    ypredictRidge = X @ Ridgetheta\n",
     "    MSERidgePredict[i] = MSE(y,ypredictRidge)\n",
     "    RegLasso = linear_model.Lasso(lmb)\n",
     "    RegLasso.fit(X,y)\n",
@@ -3542,7 +3542,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "60cfd641",
+   "id": "fbc293e8",
    "metadata": {
     "editable": true
    },
@@ -3552,7 +3552,7 @@
     "that for small values of the hyperparameter $\\lambda$ all three\n",
     "methods produce the same mean squared error. Again, Lasso shrinks the\n",
     "parameter values to zero much earlier than Ridge regression and the\n",
-    "Lasso results flatten out much earlier since all $\\beta_j=0$ (check\n",
+    "Lasso results flatten out much earlier since all $\\theta_j=0$ (check\n",
     "this by printing the values). This case is an example of where OLS\n",
     "performs best. Lasso and Ridge reproduce the OLS results for a limited\n",
     "set of $\\lambda$ values."
@@ -3561,7 +3561,7 @@
   {
    "cell_type": "code",
    "execution_count": 11,
-   "id": "171876b3",
+   "id": "b8d962a8",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3599,14 +3599,14 @@
     "# We split the data in test and training data\n",
     "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
     "\n",
-    "# matrix inversion to find beta\n",
-    "OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
-    "print(OLSbeta)\n",
+    "# matrix inversion to find theta\n",
+    "OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
+    "print(OLStheta)\n",
     "# and then make the prediction\n",
-    "ytildeOLS = X_train @ OLSbeta\n",
+    "ytildeOLS = X_train @ OLStheta\n",
     "print(\"Training MSE for OLS\")\n",
     "print(MSE(y_train,ytildeOLS))\n",
-    "ypredictOLS = X_test @ OLSbeta\n",
+    "ypredictOLS = X_test @ OLStheta\n",
     "print(\"Test MSE OLS\")\n",
     "print(MSE(y_test,ypredictOLS))\n",
     "\n",
@@ -3621,13 +3621,13 @@
     "lambdas = np.logspace(-4, 4, nlambdas)\n",
     "for i in range(nlambdas):\n",
     "    lmb = lambdas[i]\n",
-    "    Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
+    "    Ridgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
     "    # include lasso using Scikit-Learn\n",
     "    RegLasso = linear_model.Lasso(lmb)\n",
     "    RegLasso.fit(X_train,y_train)\n",
     "    # and then make the prediction\n",
-    "    ytildeRidge = X_train @ Ridgebeta\n",
-    "    ypredictRidge = X_test @ Ridgebeta\n",
+    "    ytildeRidge = X_train @ Ridgetheta\n",
+    "    ypredictRidge = X_test @ Ridgetheta\n",
     "    ytildeLasso = RegLasso.predict(X_train)\n",
     "    ypredictLasso = RegLasso.predict(X_test)\n",
     "    MSEPredict[i] = MSE(y_test,ypredictRidge)\n",
@@ -3650,7 +3650,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "947928e7",
+   "id": "083abcdc",
    "metadata": {
     "editable": true
    },
@@ -3665,7 +3665,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9559d0a8",
+   "id": "b23e5726",
    "metadata": {
     "editable": true
    },
@@ -3678,7 +3678,7 @@
     "particular, we will focus on what the regularization terms can result\n",
     "in.  We will amongst other things show that the regularization\n",
     "parameter can reduce considerably the variance of the parameters\n",
-    "$\\beta$.\n",
+    "$\\theta$.\n",
     "\n",
     "The\n",
     "advantage of doing linear regression is that we actually end up with\n",
@@ -3694,7 +3694,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6810eb7d",
+   "id": "2ea9005f",
    "metadata": {
     "editable": true
    },
@@ -3710,7 +3710,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bd997167",
+   "id": "dda3af52",
    "metadata": {
     "editable": true
    },
@@ -3718,7 +3718,7 @@
     "The randomness of $\\varepsilon_i$ implies that\n",
     "$\\mathbf{y}_i$ is also a random variable. In particular,\n",
     "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n",
-    "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n",
+    "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\theta}$ is a\n",
     "non-random scalar. To specify the parameters of the distribution of\n",
     "$\\mathbf{y}_i$ we need to calculate its first two moments. \n",
     "\n",
@@ -3733,7 +3733,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4ff740b9",
+   "id": "094aef6f",
    "metadata": {
     "editable": true
    },
@@ -3745,7 +3745,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fe0b2250",
+   "id": "677a8764",
    "metadata": {
     "editable": true
    },
@@ -3756,19 +3756,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "87e6b9b3",
+   "id": "396b620b",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
+    "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\theta}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "68d0de57",
+   "id": "9c73587c",
    "metadata": {
     "editable": true
    },
@@ -3778,7 +3778,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2f239890",
+   "id": "1a2d8f88",
    "metadata": {
     "editable": true
    },
@@ -3786,15 +3786,15 @@
     "$$\n",
     "\\begin{align*} \n",
     "\\mathbb{E}(y_i) & =\n",
-    "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n",
-    "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n",
+    "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta}) + \\mathbb{E}(\\varepsilon_i)\n",
+    "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\theta, \n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2a724679",
+   "id": "f9c84479",
    "metadata": {
     "editable": true
    },
@@ -3805,7 +3805,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2d710e45",
+   "id": "7b39b6a3",
    "metadata": {
     "editable": true
    },
@@ -3814,12 +3814,12 @@
     "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n",
     "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n",
     "[\\mathbb{E}(y_i)]^2  \\\\  & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n",
-    "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n",
-    "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n",
-    "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
-    "\\ast} \\, \\beta)^2 \\\\  & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n",
-    "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n",
-    "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n",
+    "\\theta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 \\\\ &\n",
+    "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 + 2 \\varepsilon_i\n",
+    "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
+    "\\ast} \\, \\theta)^2 \\\\  & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 + 2\n",
+    "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta} +\n",
+    "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 \n",
     "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n",
     "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2.  \n",
     "\\end{align*}\n",
@@ -3828,32 +3828,32 @@
   },
   {
    "cell_type": "markdown",
-   "id": "488a73d8",
+   "id": "b89a5103",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
-    "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n",
+    "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
+    "mean value $\\boldsymbol{X}\\boldsymbol{\\theta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n",
     "\n",
-    "With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ we can evaluate the expectation value"
+    "With the OLS expressions for the parameters $\\boldsymbol{\\theta}$ we can evaluate the expectation value"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "9b1dca9a",
+   "id": "311e8043",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathbb{E}(\\boldsymbol{\\beta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n",
+    "\\mathbb{E}(\\boldsymbol{\\theta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\theta}=\\boldsymbol{\\theta}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "07089a59",
+   "id": "2b32d6f7",
    "metadata": {
     "editable": true
    },
@@ -3862,35 +3862,35 @@
     "\n",
     "We can also calculate the variance\n",
     "\n",
-    "The variance of $\\boldsymbol{\\beta}$ is"
+    "The variance of $\\boldsymbol{\\theta}$ is"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "690bd104",
+   "id": "f5a7fb75",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{eqnarray*}\n",
-    "\\mbox{Var}(\\boldsymbol{\\beta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n",
+    "\\mbox{Var}(\\boldsymbol{\\theta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\theta} - \\mathbb{E}(\\boldsymbol{\\theta})] [\\boldsymbol{\\theta} - \\mathbb{E}(\\boldsymbol{\\theta})]^{T} \\}\n",
     "\\\\\n",
-    "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}]^{T} \\}\n",
+    "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\theta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\theta}]^{T} \\}\n",
     "\\\\\n",
-    "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "% \\\\\n",
-    "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}  \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}  \\} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "% \\\\\n",
-    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "\\\\\n",
-    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\,  \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} \\,  \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "% \\\\\n",
-    "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\,  \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
+    "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^T \\,  \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
     "% \\\\\n",
-    "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T  \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n",
+    "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T  \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\boldsymbol{\\theta}^T\n",
     "\\\\\n",
-    "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}  + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "& = & \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}  + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n",
     "\\end{eqnarray*}\n",
     "$$"
@@ -3898,21 +3898,21 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6a9132ce",
+   "id": "6944f707",
    "metadata": {
     "editable": true
    },
    "source": [
     "where we have used  that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n",
-    "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n",
-    "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n",
+    "\\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} \\, \\mathbf{X}^{T} +\n",
+    "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\theta}) = \\sigma^2\n",
     "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n",
     "variance of the estimate of the $j$-th regression coefficient:\n",
-    "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n",
+    "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\theta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n",
     "construct a confidence interval for the estimates.\n",
     "\n",
     "In a similar way, we can obtain analytical expressions for say the\n",
-    "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n",
+    "expectation values of the parameters $\\boldsymbol{\\theta}$ and their variance\n",
     "when we employ Ridge regression, allowing us again to define a confidence interval. \n",
     "\n",
     "It is rather straightforward to show that"
@@ -3920,80 +3920,80 @@
   },
   {
    "cell_type": "markdown",
-   "id": "68cce775",
+   "id": "9c8d4134",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n",
+    "\\mathbb{E} \\big[ \\boldsymbol{\\theta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\theta}^{\\mathrm{OLS}}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a9c3f89a",
+   "id": "6d47f172",
    "metadata": {
     "editable": true
    },
    "source": [
     "We see clearly that \n",
-    "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n",
+    "$\\mathbb{E} \\big[ \\boldsymbol{\\theta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\theta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n",
     "\n",
     "We can also compute the variance as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f9e2f9d7",
+   "id": "e41732d4",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}  \\mathbf{X}^{T} \\mathbf{X} \\{ [  \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
+    "\\mbox{Var}[\\boldsymbol{\\theta}^{\\mathrm{Ridge}}]=\\sigma^2[  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}  \\mathbf{X}^{T} \\mathbf{X} \\{ [  \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "58443fe8",
+   "id": "2e3123fe",
    "metadata": {
     "editable": true
    },
    "source": [
-    "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n",
+    "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\theta}$ goes to zero. \n",
     "\n",
     "With this, we can compute the difference"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "cc34c059",
+   "id": "9ba7ba08",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
+    "\\mbox{Var}[\\boldsymbol{\\theta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\theta}^{\\mathrm{Ridge}})=\\sigma^2 [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6ad9c8e3",
+   "id": "f4077e50",
    "metadata": {
     "editable": true
    },
    "source": [
     "The difference is non-negative definite since each component of the\n",
     "matrix product is non-negative definite. \n",
-    "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below."
+    "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\theta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below."
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7c09657d",
+   "id": "2945f567",
    "metadata": {
     "editable": true
    },
@@ -4007,28 +4007,28 @@
     "$\\sigma^2$.\n",
     "\n",
     "We found above that the outputs $\\boldsymbol{y}$ have a mean value given by\n",
-    "$\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}}$ and variance $\\sigma^2$. Since the entries to\n",
+    "$\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$ and variance $\\sigma^2$. Since the entries to\n",
     "the design matrix are not stochastic variables, we can assume that the\n",
     "probability distribution of our targets is also a normal distribution\n",
-    "but now with mean value $\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}}$. This means that a\n",
+    "but now with mean value $\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}}$. This means that a\n",
     "single output $y_i$ is given by the Gaussian distribution"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "abe9915b",
+   "id": "1324fb2e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
+    "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "326e0c75",
+   "id": "f19032d0",
    "metadata": {
     "editable": true
    },
@@ -4039,43 +4039,43 @@
   },
   {
    "cell_type": "markdown",
-   "id": "567fb1b1",
+   "id": "a18bd251",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n",
+    "p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\theta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "107abe1c",
+   "id": "9f5be910",
    "metadata": {
     "editable": true
    },
    "source": [
-    "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\beta}$.\n",
+    "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\theta}$.\n",
     "\n",
     "Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event $\\boldsymbol{y}$ as the product of the single events, that is we have"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f11ddf78",
+   "id": "d92545ca",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta}).\n",
+    "p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\theta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\theta}).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2abd6e3b",
+   "id": "d30bf273",
    "metadata": {
     "editable": true
    },
@@ -4086,7 +4086,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "caddb652",
+   "id": "70795118",
    "metadata": {
     "editable": true
    },
@@ -4098,7 +4098,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "291e1dd6",
+   "id": "ecfb478e",
    "metadata": {
     "editable": true
    },
@@ -4109,26 +4109,26 @@
   },
   {
    "cell_type": "markdown",
-   "id": "73ac95c1",
+   "id": "89c5df6b",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
+    "p(\\boldsymbol{D}\\vert\\boldsymbol{\\theta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ce49493b",
+   "id": "082b8841",
    "metadata": {
     "editable": true
    },
    "source": [
     "It is a conditional probability (see below) and reads as the\n",
     "likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters\n",
-    "$\\boldsymbol{\\beta}$.\n",
+    "$\\boldsymbol{\\theta}$.\n",
     "\n",
     "In statistics, maximum likelihood estimation (MLE) is a method of\n",
     "estimating the parameters of an assumed probability distribution,\n",
@@ -4137,7 +4137,7 @@
     "data is the most probable. \n",
     "\n",
     "We will assume here that our events are given by the above Gaussian\n",
-    "distribution and we will determine the optimal parameters $\\beta$ by\n",
+    "distribution and we will determine the optimal parameters $\\theta$ by\n",
     "maximizing the above PDF. However, computing the derivatives of a\n",
     "product function is cumbersome and can easily lead to overflow and/or\n",
     "underflowproblems, with potentials for loss of numerical precision.\n",
@@ -4156,19 +4156,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1ff54861",
+   "id": "59d1eaca",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta}=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})},\n",
+    "C(\\boldsymbol{\\theta}=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\theta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\theta})},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e8cdd425",
+   "id": "7142a2d9",
    "metadata": {
     "editable": true
    },
@@ -4178,63 +4178,63 @@
   },
   {
    "cell_type": "markdown",
-   "id": "95d54be7",
+   "id": "3e111f5b",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}.\n",
+    "C(\\boldsymbol{\\theta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "731e3e2a",
+   "id": "de59021f",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Taking the derivative of the *new* cost function with respect to the parameters $\\beta$ we recognize our familiar OLS equation, namely"
+    "Taking the derivative of the *new* cost function with respect to the parameters $\\theta$ we recognize our familiar OLS equation, namely"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8c40a24c",
+   "id": "5e78bfd8",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right) =0,\n",
+    "\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right) =0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1e298a02",
+   "id": "09836e12",
    "metadata": {
     "editable": true
    },
    "source": [
-    "which leads to the well-known OLS equation for the optimal paramters $\\beta$"
+    "which leads to the well-known OLS equation for the optimal paramters $\\theta$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "fd5c3e4f",
+   "id": "2f3404a5",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n",
+    "\\hat{\\boldsymbol{\\theta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d3aab131",
+   "id": "0bc175f7",
    "metadata": {
     "editable": true
    },
@@ -4254,7 +4254,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "64646b7c",
+   "id": "2d0434aa",
    "metadata": {
     "editable": true
    },
@@ -4266,7 +4266,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0ad4cc29",
+   "id": "42f4b84f",
    "metadata": {
     "editable": true
    },
@@ -4276,7 +4276,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8dfd2150",
+   "id": "3677eed8",
    "metadata": {
     "editable": true
    },
@@ -4288,7 +4288,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9c0313b7",
+   "id": "134f8f1a",
    "metadata": {
     "editable": true
    },
@@ -4302,7 +4302,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "af94800f",
+   "id": "9f6349f1",
    "metadata": {
     "editable": true
    },
@@ -4314,7 +4314,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3ed2ccef",
+   "id": "557eae67",
    "metadata": {
     "editable": true
    },
@@ -4324,7 +4324,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a74ee19",
+   "id": "0859c412",
    "metadata": {
     "editable": true
    },
@@ -4336,7 +4336,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5191a71e",
+   "id": "ef82304a",
    "metadata": {
     "editable": true
    },
@@ -4346,7 +4346,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5d5de8f7",
+   "id": "9f2500ca",
    "metadata": {
     "editable": true
    },
@@ -4358,7 +4358,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cca75f59",
+   "id": "b7b9d2d8",
    "metadata": {
     "editable": true
    },
@@ -4368,7 +4368,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9113e675",
+   "id": "f79df283",
    "metadata": {
     "editable": true
    },
@@ -4380,7 +4380,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a21d13da",
+   "id": "ca93f0cf",
    "metadata": {
     "editable": true
    },
@@ -4416,7 +4416,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0c7abec6",
+   "id": "9092d777",
    "metadata": {
     "editable": true
    },
@@ -4428,7 +4428,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2eccb1af",
+   "id": "017ca9fb",
    "metadata": {
     "editable": true
    },
@@ -4441,7 +4441,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3c6635e5",
+   "id": "529014b2",
    "metadata": {
     "editable": true
    },
@@ -4453,7 +4453,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ecf0a0b6",
+   "id": "23629d2d",
    "metadata": {
     "editable": true
    },
@@ -4467,7 +4467,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "166345a1",
+   "id": "80c6a09b",
    "metadata": {
     "editable": true
    },
@@ -4479,7 +4479,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8a73e80e",
+   "id": "fe371d69",
    "metadata": {
     "editable": true
    },
@@ -4490,7 +4490,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "01441388",
+   "id": "ed9320dd",
    "metadata": {
     "editable": true
    },
@@ -4502,7 +4502,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "93fe1e0a",
+   "id": "710d72a1",
    "metadata": {
     "editable": true
    },
@@ -4514,7 +4514,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d6860415",
+   "id": "331424a5",
    "metadata": {
     "editable": true
    },
@@ -4532,7 +4532,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9710dd92",
+   "id": "683874d3",
    "metadata": {
     "editable": true
    },
@@ -4550,7 +4550,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "23e3912e",
+   "id": "2b32fd95",
    "metadata": {
     "editable": true
    },
@@ -4560,7 +4560,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cf500f71",
+   "id": "9ed06a74",
    "metadata": {
     "editable": true
    },
@@ -4575,9 +4575,9 @@
     "Before we proceed let us perform a Ridge, Lasso  and OLS analysis of a polynomial fit. \n",
     "\n",
     "We will play around with a study of the values for the optimal\n",
-    "parameters $\\boldsymbol{\\beta}$ using OLS, Ridge and Lasso regression.  For\n",
+    "parameters $\\boldsymbol{\\theta}$ using OLS, Ridge and Lasso regression.  For\n",
     "OLS, you will notice as function of the noise and polynomial degree,\n",
-    "that the parameters $\\beta$ will fluctuate from order to order in the\n",
+    "that the parameters $\\theta$ will fluctuate from order to order in the\n",
     "polynomial fit and that for larger and larger polynomial degrees of\n",
     "freedom, the parameters will tend to increase in value for OLS.\n",
     "\n",
@@ -4589,7 +4589,7 @@
   {
    "cell_type": "code",
    "execution_count": 12,
-   "id": "134d0a22",
+   "id": "9a336ec4",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4624,10 +4624,10 @@
     "# We split the data in test and training data\n",
     "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
     "\n",
-    "# matrix inversion to find beta\n",
-    "OLSbeta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train\n",
-    "print(OLSbeta)\n",
-    "ypredictOLS = X_test @ OLSbeta\n",
+    "# matrix inversion to find theta\n",
+    "OLStheta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train\n",
+    "print(OLStheta)\n",
+    "ypredictOLS = X_test @ OLStheta\n",
     "print(\"Test MSE OLS\")\n",
     "print(MSE(y_test,ypredictOLS))\n",
     "# Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn\n",
@@ -4663,19 +4663,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7dcfe550",
+   "id": "87b78389",
    "metadata": {
     "editable": true
    },
    "source": [
     "How can we understand this?\n",
     "\n",
-    "Let us write out the values of the coefficients $\\beta_i$ as functions\n",
+    "Let us write out the values of the coefficients $\\theta_i$ as functions\n",
     "of the polynomial degree and noise. We will focus only on the Ridge\n",
     "results and some few selected values of the hyperparameter $\\lambda$.\n",
     "\n",
     "If we don't include any noise and run this code for different values\n",
-    "of the polynomial degree, we notice that the results for $\\beta_i$ do\n",
+    "of the polynomial degree, we notice that the results for $\\theta_i$ do\n",
     "not show great changes from one order to the next. This is an\n",
     "indication that for higher polynomial orders, our parameters become\n",
     "less important.\n",
@@ -4683,17 +4683,17 @@
     "If we however add noise, what happens is that the polynomial fit is\n",
     "trying to adjust the fit to traverse in the best possible way all data\n",
     "points. This can lead to large fluctuations in the parameters\n",
-    "$\\beta_i$ as functions of polynomial order. It will also be reflected\n",
-    "in a larger value of the variance of each parameter $\\beta_i$.  What\n",
+    "$\\theta_i$ as functions of polynomial order. It will also be reflected\n",
+    "in a larger value of the variance of each parameter $\\theta_i$.  What\n",
     "Ridge regression (and Lasso as well) are doing then is to try to\n",
-    "quench the fluctuations in the parameters of $\\beta_i$ which have a\n",
+    "quench the fluctuations in the parameters of $\\theta_i$ which have a\n",
     "large variance (normally for higher orders in the polynomial)."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 13,
-   "id": "5ff5d7c2",
+   "id": "e2c403e8",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4735,30 +4735,30 @@
     "    # and then make the prediction\n",
     "    ypredictRidge = RegRidge.predict(X_test)\n",
     "    Coeffs = np.array(RegRidge.coef_)\n",
-    "    BetaValues = pd.DataFrame(Coeffs)\n",
-    "    BetaValues.columns = ['beta']\n",
-    "    display(BetaValues)"
+    "    ThetaValues = pd.DataFrame(Coeffs)\n",
+    "    ThetaValues.columns = ['theta']\n",
+    "    display(ThetaValues)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f1680928",
+   "id": "4f4dd369",
    "metadata": {
     "editable": true
    },
    "source": [
     "As an exercise, repeat these calculations with ordinary least squares\n",
     "only with and without noise. Calculate thereafter the variance of the\n",
-    "parameters $\\beta_j$ as function of polynomial order and of the added\n",
+    "parameters $\\theta_j$ as function of polynomial order and of the added\n",
     "noise. Here we recommend to use $\\sigma^2=1$ as variance for the\n",
     "added noise (which follows a normal distribution with mean value zero).\n",
-    "Comment your results. If you have a large noise term, do the parameters $\\beta_j$ vary more as function\n",
+    "Comment your results. If you have a large noise term, do the parameters $\\theta_j$ vary more as function\n",
     "of model complexity? And what about their variance?"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5b458403",
+   "id": "67adfefc",
    "metadata": {
     "editable": true
    },
@@ -4767,7 +4767,7 @@
     "\n",
     "We have seen that Ridge regression suppresses those features which\n",
     "have a small singular value. This corresponds to a feature which exhibits\n",
-    "a large variance in the parameters $\\beta_j$.\n",
+    "a large variance in the parameters $\\theta_j$.\n",
     "Our analysis hitherto has been based on linear algebra. To add to our intuition, we will use\n",
     "Bayes' theorem in order to deepen our understanding of  Ridge and Lasso regression. \n",
     "\n",
@@ -4776,7 +4776,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6e2d6bf6",
+   "id": "ab781318",
    "metadata": {
     "editable": true
    },
@@ -4788,7 +4788,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "098b6cbd",
+   "id": "ccf25024",
    "metadata": {
     "editable": true
    },
@@ -4798,41 +4798,41 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8a02d0aa",
+   "id": "9c8bc496",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
+    "p(\\boldsymbol{D}\\vert\\boldsymbol{\\theta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "97f22408",
+   "id": "7dcf96d4",
    "metadata": {
     "editable": true
    },
    "source": [
-    "In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\beta}$ given a domain of events $\\boldsymbol{D}$?  That is, how can we define the posterior probability"
+    "In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\theta}$ given a domain of events $\\boldsymbol{D}$?  That is, how can we define the posterior probability"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "27038459",
+   "id": "83af5cfb",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D}).\n",
+    "p(\\boldsymbol{\\theta}\\vert\\boldsymbol{D}).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f682a8c3",
+   "id": "baf02e7e",
    "metadata": {
     "editable": true
    },
@@ -4842,49 +4842,49 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7fdaa748",
+   "id": "1ec2d48a",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})p(\\boldsymbol{\\beta}).\n",
+    "p(\\boldsymbol{\\theta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\theta})p(\\boldsymbol{\\theta}).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1e7fa52c",
+   "id": "e77ae0c0",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta})$!   \n",
+    "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\theta})$ but need one for the **prior** $p(\\boldsymbol{\\theta})$!   \n",
     "\n",
     "With the posterior probability defined by a likelihood which we have\n",
     "already modeled and an unknown prior, we are now ready to make\n",
     "additional models for the prior.\n",
     "\n",
-    "We can, based on our discussions of the variance of $\\boldsymbol{\\beta}$ and\n",
-    "the mean value, assume that the prior for the values $\\boldsymbol{\\beta}$ is\n",
+    "We can, based on our discussions of the variance of $\\boldsymbol{\\theta}$ and\n",
+    "the mean value, assume that the prior for the values $\\boldsymbol{\\theta}$ is\n",
     "given by a Gaussian with mean value zero and variance $\\tau^2$, that"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "501d66f4",
+   "id": "5ee70503",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n",
+    "p(\\boldsymbol{\\theta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\theta_j^2}{2\\tau^2}\\right)}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f029c143",
+   "id": "81b5ea11",
    "metadata": {
     "editable": true
    },
@@ -4894,44 +4894,44 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7f7c3e11",
+   "id": "a15cac35",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n",
+    "p(\\boldsymbol{\\theta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\theta_j^2}{2\\tau^2}\\right)}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1f39114c",
+   "id": "76f66371",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We can now optimize this quantity with respect to $\\boldsymbol{\\beta}$. As we\n",
+    "We can now optimize this quantity with respect to $\\boldsymbol{\\theta}$. As we\n",
     "did for OLS, this is most conveniently done by taking the negative\n",
     "logarithm of the posterior probability. Doing so and leaving out the\n",
-    "terms that do not depend on $\\beta$, we have"
+    "terms that do not depend on $\\theta$, we have"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "81cc7b03",
+   "id": "deaecaf5",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n",
+    "C(\\boldsymbol{\\theta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_2^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1e614b9b",
+   "id": "da1297cc",
    "metadata": {
     "editable": true
    },
@@ -4941,19 +4941,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "77252afc",
+   "id": "4c429a58",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n",
+    "C(\\boldsymbol{\\theta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_2^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "14953579",
+   "id": "2c9fd875",
    "metadata": {
     "editable": true
    },
@@ -4965,19 +4965,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "36f1f63d",
+   "id": "350a62eb",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n",
+    "p(\\boldsymbol{\\theta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\theta_j\\vert}{\\tau}\\right)}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "50dd90a5",
+   "id": "9bb76e79",
    "metadata": {
     "editable": true
    },
@@ -4987,43 +4987,43 @@
   },
   {
    "cell_type": "markdown",
-   "id": "de39cb12",
+   "id": "e08c07dd",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n",
+    "p(\\boldsymbol{\\theta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\theta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\theta_j\\vert}{\\tau}\\right)}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ad1fc46e",
+   "id": "e4bd6b2c",
    "metadata": {
     "editable": true
    },
    "source": [
     "Taking the negative\n",
     "logarithm of the posterior probability and leaving out the\n",
-    "constants terms that do not depend on $\\beta$, we have"
+    "constants terms that do not depend on $\\theta$, we have"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ff8695d4",
+   "id": "7c1bc34c",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
+    "C(\\boldsymbol{\\theta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "0de8080e",
+   "id": "5f24477c",
    "metadata": {
     "editable": true
    },
@@ -5033,19 +5033,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3965e5ef",
+   "id": "d9948bc0",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
+    "C(\\boldsymbol{\\theta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5c978cdf",
+   "id": "d690afa3",
    "metadata": {
     "editable": true
    },
@@ -5054,7 +5054,7 @@
     "\n",
     "Plotting these prior functions shows us that we can use the parameter\n",
     "$\\lambda$ to shrink or increase the role of a given parameter\n",
-    "$\\beta_j$.  The variance for the Laplace distribution is\n",
+    "$\\theta_j$.  The variance for the Laplace distribution is\n",
     "$2\\tau^2=1/\\lambda$ while for the Gaussian distribution it is\n",
     "$\\sigma^2=1/(2\\lambda)$. Thus, increasing the variance means\n",
     "decreasing $\\lambda$ and shrinking the variance means increasing\n",
diff --git a/doc/LectureNotes/chapter3.ipynb b/doc/LectureNotes/chapter3.ipynb
index 6e1a008cf..a9b8c34f2 100644
--- a/doc/LectureNotes/chapter3.ipynb
+++ b/doc/LectureNotes/chapter3.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "51186d57",
+   "id": "663483da",
    "metadata": {
     "editable": true
    },
@@ -13,7 +13,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "79a63fb4",
+   "id": "05ca2993",
    "metadata": {
     "editable": true
    },
@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f3d916b4",
+   "id": "cea49fb9",
    "metadata": {
     "editable": true
    },
@@ -74,7 +74,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "46cb3279",
+   "id": "85adbc81",
    "metadata": {
     "editable": true
    },
@@ -105,7 +105,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0fe38e07",
+   "id": "4d010e03",
    "metadata": {
     "editable": true
    },
@@ -121,7 +121,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d9d6955b",
+   "id": "54c13e4f",
    "metadata": {
     "editable": true
    },
@@ -129,7 +129,7 @@
     "The randomness of $\\varepsilon_i$ implies that\n",
     "$\\mathbf{y}_i$ is also a random variable. In particular,\n",
     "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n",
-    "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n",
+    "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\theta}$ is a\n",
     "non-random scalar. To specify the parameters of the distribution of\n",
     "$\\mathbf{y}_i$ we need to calculate its first two moments. \n",
     "\n",
@@ -144,7 +144,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f41f7049",
+   "id": "1705258c",
    "metadata": {
     "editable": true
    },
@@ -156,7 +156,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2ee172f4",
+   "id": "f96c2ee1",
    "metadata": {
     "editable": true
    },
@@ -167,19 +167,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1aecc768",
+   "id": "84fc8fc9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
+    "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\theta}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "12e9bdea",
+   "id": "c846eb35",
    "metadata": {
     "editable": true
    },
@@ -189,7 +189,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8f9db9db",
+   "id": "a7c728c5",
    "metadata": {
     "editable": true
    },
@@ -197,15 +197,15 @@
     "$$\n",
     "\\begin{align*} \n",
     "\\mathbb{E}(y_i) & =\n",
-    "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n",
-    "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n",
+    "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta}) + \\mathbb{E}(\\varepsilon_i)\n",
+    "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\theta, \n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b10abe89",
+   "id": "d7d34864",
    "metadata": {
     "editable": true
    },
@@ -216,7 +216,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bec51521",
+   "id": "f235a710",
    "metadata": {
     "editable": true
    },
@@ -225,12 +225,12 @@
     "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n",
     "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n",
     "[\\mathbb{E}(y_i)]^2  \\\\  & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n",
-    "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n",
-    "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n",
-    "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
-    "\\ast} \\, \\beta)^2 \\\\  & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n",
-    "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n",
-    "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n",
+    "\\theta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 \\\\ &\n",
+    "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 + 2 \\varepsilon_i\n",
+    "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
+    "\\ast} \\, \\theta)^2 \\\\  & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 + 2\n",
+    "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta} +\n",
+    "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta})^2 \n",
     "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n",
     "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2.  \n",
     "\\end{align*}\n",
@@ -239,32 +239,32 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4859640c",
+   "id": "faa4158d",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
-    "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n",
+    "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\theta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
+    "mean value $\\boldsymbol{X}\\boldsymbol{\\theta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n",
     "\n",
-    "With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ we can evaluate the expectation value"
+    "With the OLS expressions for the parameters $\\boldsymbol{\\theta}$ we can evaluate the expectation value"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "69978823",
+   "id": "da189fae",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathbb{E}(\\boldsymbol{\\beta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n",
+    "\\mathbb{E}(\\boldsymbol{\\theta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\theta}=\\boldsymbol{\\theta}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5caee9d9",
+   "id": "640c53e9",
    "metadata": {
     "editable": true
    },
@@ -273,35 +273,35 @@
     "\n",
     "We can also calculate the variance\n",
     "\n",
-    "The variance of $\\boldsymbol{\\beta}$ is"
+    "The variance of $\\boldsymbol{\\theta}$ is"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6791e5b4",
+   "id": "e2eee7fb",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{eqnarray*}\n",
-    "\\mbox{Var}(\\boldsymbol{\\beta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n",
+    "\\mbox{Var}(\\boldsymbol{\\theta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\theta} - \\mathbb{E}(\\boldsymbol{\\theta})] [\\boldsymbol{\\theta} - \\mathbb{E}(\\boldsymbol{\\theta})]^{T} \\}\n",
     "\\\\\n",
-    "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}]^{T} \\}\n",
+    "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\theta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\theta}]^{T} \\}\n",
     "\\\\\n",
-    "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "% \\\\\n",
-    "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}  \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}  \\} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "% \\\\\n",
-    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "\\\\\n",
-    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\,  \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} \\,  \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "% \\\\\n",
-    "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\,  \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
+    "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^T \\,  \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n",
     "% \\\\\n",
-    "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T  \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n",
+    "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T  \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\boldsymbol{\\theta}^T\n",
     "\\\\\n",
-    "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}  + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n",
+    "& = & \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}  + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T}\n",
     "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n",
     "\\end{eqnarray*}\n",
     "$$"
@@ -309,22 +309,22 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4f747992",
+   "id": "69660ffd",
    "metadata": {
     "editable": true
    },
    "source": [
     "where we have used  that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n",
-    "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n",
-    "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n",
+    "\\mathbf{X} \\, \\boldsymbol{\\theta} \\, \\boldsymbol{\\theta}^{T} \\, \\mathbf{X}^{T} +\n",
+    "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\theta}) = \\sigma^2\n",
     "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n",
     "variance of the estimate of the $j$-th regression coefficient:\n",
-    "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 \\sqrt{\n",
+    "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\theta}_j ) = \\boldsymbol{\\sigma}^2 \\sqrt{\n",
     "[(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} }$. This may be used to\n",
     "construct a confidence interval for the estimates.\n",
     "\n",
     "In a similar way, we can obtain analytical expressions for say the\n",
-    "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n",
+    "expectation values of the parameters $\\boldsymbol{\\theta}$ and their variance\n",
     "when we employ Ridge regression, allowing us again to define a confidence interval. \n",
     "\n",
     "It is rather straightforward to show that"
@@ -332,80 +332,80 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d4679496",
+   "id": "166fe7a6",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n",
+    "\\mathbb{E} \\big[ \\boldsymbol{\\theta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\theta}^{\\mathrm{OLS}}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "bd3534e1",
+   "id": "9b9e1336",
    "metadata": {
     "editable": true
    },
    "source": [
     "We see clearly that \n",
-    "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n",
+    "$\\mathbb{E} \\big[ \\boldsymbol{\\theta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\theta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n",
     "\n",
     "We can also compute the variance as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "125b139e",
+   "id": "d4548ed8",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}  \\mathbf{X}^{T} \\mathbf{X} \\{ [  \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
+    "\\mbox{Var}[\\boldsymbol{\\theta}^{\\mathrm{Ridge}}]=\\sigma^2[  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}  \\mathbf{X}^{T} \\mathbf{X} \\{ [  \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "98b59ea0",
+   "id": "642f4b4a",
    "metadata": {
     "editable": true
    },
    "source": [
-    "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n",
+    "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\theta}$ goes to zero. \n",
     "\n",
     "With this, we can compute the difference"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "dcfd7a6c",
+   "id": "e8a1528a",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
+    "\\mbox{Var}[\\boldsymbol{\\theta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\theta}^{\\mathrm{Ridge}})=\\sigma^2 [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [  \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "30f43be3",
+   "id": "1bd2f713",
    "metadata": {
     "editable": true
    },
    "source": [
     "The difference is non-negative definite since each component of the\n",
     "matrix product is non-negative definite. \n",
-    "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below."
+    "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\theta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below."
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e3c1bf3c",
+   "id": "8019b9c6",
    "metadata": {
     "editable": true
    },
@@ -446,14 +446,14 @@
     "variance of $\\overline{X}$ (which often is the case), then there is no\n",
     "need for bootstrapping. \n",
     "\n",
-    "The Jackknife works by making many replicas of the estimator $\\widehat{\\beta}$. \n",
+    "The Jackknife works by making many replicas of the estimator $\\widehat{\\theta}$. \n",
     "The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\\boldsymbol{x} = (x_1,x_2,\\cdots,X_n)$. \n",
     "Let $\\boldsymbol{x}_i$ denote the vector"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "3925f435",
+   "id": "38e804f2",
    "metadata": {
     "editable": true
    },
@@ -465,21 +465,21 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cc6328ee",
+   "id": "35eb791c",
    "metadata": {
     "editable": true
    },
    "source": [
     "which equals the vector $\\boldsymbol{x}$ with the exception that observation\n",
     "number $i$ is left out. Using this notation, define\n",
-    "$\\widehat{\\beta}_i$ to be the estimator\n",
-    "$\\widehat{\\beta}$ computed using $\\vec{X}_i$."
+    "$\\widehat{\\theta}_i$ to be the estimator\n",
+    "$\\widehat{\\theta}$ computed using $\\vec{X}_i$."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "b751a941",
+   "id": "3261f074",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -518,7 +518,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "25ff562a",
+   "id": "9d117c52",
    "metadata": {
     "editable": true
    },
@@ -537,28 +537,28 @@
     "\n",
     "4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).\n",
     "\n",
-    "Since $\\widehat{\\beta} = \\widehat{\\beta}(\\boldsymbol{X})$ is a function of random variables,\n",
-    "$\\widehat{\\beta}$ itself must be a random variable. Thus it has\n",
+    "Since $\\widehat{\\theta} = \\widehat{\\theta}(\\boldsymbol{X})$ is a function of random variables,\n",
+    "$\\widehat{\\theta}$ itself must be a random variable. Thus it has\n",
     "a pdf, call this function $p(\\boldsymbol{t})$. The aim of the bootstrap is to\n",
     "estimate $p(\\boldsymbol{t})$ by the relative frequency of\n",
-    "$\\widehat{\\beta}$. You can think of this as using a histogram\n",
+    "$\\widehat{\\theta}$. You can think of this as using a histogram\n",
     "in the place of $p(\\boldsymbol{t})$. If the relative frequency closely\n",
     "resembles $p(\\vec{t})$, then using numerics, it is straight forward to\n",
     "estimate all the interesting parameters of $p(\\boldsymbol{t})$ using point\n",
     "estimators.  \n",
     "\n",
-    "In the case that $\\widehat{\\beta}$ has\n",
+    "In the case that $\\widehat{\\theta}$ has\n",
     "more than one component, and the components are independent, we use the\n",
     "same estimator on each component separately.  If the probability\n",
     "density function of $X_i$, $p(x)$, had been known, then it would have\n",
     "been straight forward to do this by: \n",
     "1. Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \\cdots, X_n^*)$. \n",
     "\n",
-    "2. Then using these numbers, we could compute a replica of $\\widehat{\\beta}$ called $\\widehat{\\beta}^*$. \n",
+    "2. Then using these numbers, we could compute a replica of $\\widehat{\\theta}$ called $\\widehat{\\theta}^*$. \n",
     "\n",
     "By repeated use of (1) and (2), many\n",
-    "estimates of $\\widehat{\\beta}$ could have been obtained. The\n",
-    "idea is to use the relative frequency of $\\widehat{\\beta}^*$\n",
+    "estimates of $\\widehat{\\theta}$ could have been obtained. The\n",
+    "idea is to use the relative frequency of $\\widehat{\\theta}^*$\n",
     "(think of a histogram) as an estimate of $p(\\boldsymbol{t})$.\n",
     "\n",
     "But\n",
@@ -581,19 +581,19 @@
     "\n",
     "2. Define a vector $\\boldsymbol{x}^*$ containing the values which were drawn from $\\boldsymbol{x}$. \n",
     "\n",
-    "3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\beta}^*$ by evaluating $\\widehat \\beta$ under the observations $\\boldsymbol{x}^*$. \n",
+    "3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\theta}^*$ by evaluating $\\widehat \\theta$ under the observations $\\boldsymbol{x}^*$. \n",
     "\n",
     "4. Repeat this process $k$ times. \n",
     "\n",
     "When you are done, you can draw a histogram of the relative frequency\n",
-    "of $\\widehat \\beta^*$. This is your estimate of the probability\n",
+    "of $\\widehat \\theta^*$. This is your estimate of the probability\n",
     "distribution $p(t)$. Using this probability distribution you can\n",
     "estimate any statistics thereof. In principle you never draw the\n",
-    "histogram of the relative frequency of $\\widehat{\\beta}^*$. Instead\n",
+    "histogram of the relative frequency of $\\widehat{\\theta}^*$. Instead\n",
     "you use the estimators corresponding to the statistic of interest. For\n",
     "example, if you are interested in estimating the variance of $\\widehat\n",
-    "\\beta$, apply the estimator $\\widehat \\sigma^2$ to the values\n",
-    "$\\widehat \\beta^*$.\n",
+    "\\theta$, apply the estimator $\\widehat \\sigma^2$ to the values\n",
+    "$\\widehat \\theta^*$.\n",
     "\n",
     "Before we proceed however, we need to remind ourselves about a central\n",
     "theorem in statistics, namely the so-called **central limit theorem**.\n",
@@ -614,7 +614,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fa55ab5a",
+   "id": "d864122f",
    "metadata": {
     "editable": true
    },
@@ -626,7 +626,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "697c0c94",
+   "id": "6750e014",
    "metadata": {
     "editable": true
    },
@@ -641,7 +641,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bd26bbd1",
+   "id": "d5a74550",
    "metadata": {
     "editable": true
    },
@@ -654,7 +654,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "68664e4f",
+   "id": "c44dc3f1",
    "metadata": {
     "editable": true
    },
@@ -669,7 +669,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "742a107a",
+   "id": "202b1bba",
    "metadata": {
     "editable": true
    },
@@ -682,7 +682,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b6d624c1",
+   "id": "da055544",
    "metadata": {
     "editable": true
    },
@@ -693,7 +693,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "46458586",
+   "id": "7773cf22",
    "metadata": {
     "editable": true
    },
@@ -707,7 +707,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ca119461",
+   "id": "8f5e06ca",
    "metadata": {
     "editable": true
    },
@@ -717,7 +717,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f75b40fc",
+   "id": "82157124",
    "metadata": {
     "editable": true
    },
@@ -731,7 +731,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a4b64e20",
+   "id": "2af358ec",
    "metadata": {
     "editable": true
    },
@@ -742,7 +742,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fc76951f",
+   "id": "4d4bafce",
    "metadata": {
     "editable": true
    },
@@ -755,7 +755,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "76979572",
+   "id": "628ec1b1",
    "metadata": {
     "editable": true
    },
@@ -765,7 +765,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3edff7d2",
+   "id": "eb27434b",
    "metadata": {
     "editable": true
    },
@@ -778,7 +778,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "73a9341f",
+   "id": "8dbfd377",
    "metadata": {
     "editable": true
    },
@@ -788,7 +788,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "88f1cc30",
+   "id": "9eda7c0f",
    "metadata": {
     "editable": true
    },
@@ -801,7 +801,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1aec913e",
+   "id": "6c97b3dc",
    "metadata": {
     "editable": true
    },
@@ -822,7 +822,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "42b317e7",
+   "id": "6d316b42",
    "metadata": {
     "editable": true
    },
@@ -835,7 +835,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e5baf71d",
+   "id": "426cae9c",
    "metadata": {
     "editable": true
    },
@@ -847,7 +847,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8e2b39e5",
+   "id": "3445fa66",
    "metadata": {
     "editable": true
    },
@@ -860,7 +860,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "472e7c2c",
+   "id": "fee204e8",
    "metadata": {
     "editable": true
    },
@@ -894,7 +894,7 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "0ff7b796",
+   "id": "b98fbaba",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -933,7 +933,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "22680159",
+   "id": "bc56e30f",
    "metadata": {
     "editable": true
    },
@@ -946,7 +946,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "fe018bb3",
+   "id": "817ebac7",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -966,7 +966,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5bbd613c",
+   "id": "454fba01",
    "metadata": {
     "editable": true
    },
@@ -984,7 +984,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ca3fde4a",
+   "id": "1716dfb2",
    "metadata": {
     "editable": true
    },
@@ -996,7 +996,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8e11f84b",
+   "id": "5295a516",
    "metadata": {
     "editable": true
    },
@@ -1005,27 +1005,27 @@
     "\n",
     "In our derivation of the ordinary least squares method we defined then\n",
     "an approximation to the function $f$ in terms of the parameters\n",
-    "$\\boldsymbol{\\beta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n",
-    "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\beta}$. \n",
+    "$\\boldsymbol{\\theta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n",
+    "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\theta}$. \n",
     "\n",
-    "Thereafter we found the parameters $\\boldsymbol{\\beta}$ by optimizing the means squared error via the so-called cost function"
+    "Thereafter we found the parameters $\\boldsymbol{\\theta}$ by optimizing the means squared error via the so-called cost function"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "026a65c8",
+   "id": "b5953a4f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n",
+    "C(\\boldsymbol{X},\\boldsymbol{\\theta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e59918c7",
+   "id": "ea8014cf",
    "metadata": {
     "editable": true
    },
@@ -1035,7 +1035,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2fd3f73c",
+   "id": "4159a3e3",
    "metadata": {
     "editable": true
    },
@@ -1047,7 +1047,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7daf46c9",
+   "id": "3c18f501",
    "metadata": {
     "editable": true
    },
@@ -1064,7 +1064,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6094266b",
+   "id": "09b0f8b1",
    "metadata": {
     "editable": true
    },
@@ -1076,7 +1076,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "643e0047",
+   "id": "80679145",
    "metadata": {
     "editable": true
    },
@@ -1086,7 +1086,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1319bde5",
+   "id": "1f39a627",
    "metadata": {
     "editable": true
    },
@@ -1098,7 +1098,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9c6d6da1",
+   "id": "b045c821",
    "metadata": {
     "editable": true
    },
@@ -1108,7 +1108,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "855756ef",
+   "id": "15cb1a46",
    "metadata": {
     "editable": true
    },
@@ -1120,7 +1120,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "34d24717",
+   "id": "d0c484eb",
    "metadata": {
     "editable": true
    },
@@ -1131,7 +1131,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "d51b6100",
+   "id": "7940a9b2",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1197,7 +1197,7 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "bd636def",
+   "id": "e9698e1e",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1254,7 +1254,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8cf88b3a",
+   "id": "3edbafad",
    "metadata": {
     "editable": true
    },
@@ -1291,7 +1291,7 @@
   {
    "cell_type": "code",
    "execution_count": 6,
-   "id": "86bfc49a",
+   "id": "5aa48d95",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1374,7 +1374,7 @@
   {
    "cell_type": "code",
    "execution_count": 7,
-   "id": "0c2a183a",
+   "id": "50c5c640",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1463,7 +1463,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2c6c9e89",
+   "id": "947db26f",
    "metadata": {
     "editable": true
    },
@@ -1494,14 +1494,14 @@
   },
   {
    "cell_type": "markdown",
-   "id": "71738b2a",
+   "id": "9974dc92",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "\\boldsymbol{\\beta}_{-i}(\\lambda) & =  ( \\boldsymbol{X}_{-i, \\ast}^{T}\n",
+    "\\boldsymbol{\\theta}_{-i}(\\lambda) & =  ( \\boldsymbol{X}_{-i, \\ast}^{T}\n",
     "\\boldsymbol{X}_{-i, \\ast} + \\lambda \\boldsymbol{I}_{pp})^{-1}\n",
     "\\boldsymbol{X}_{-i, \\ast}^{T} \\boldsymbol{y}_{-i}\n",
     "\\end{align*}\n",
@@ -1510,12 +1510,12 @@
   },
   {
    "cell_type": "markdown",
-   "id": "14db46b6",
+   "id": "45b8c96c",
    "metadata": {
     "editable": true
    },
    "source": [
-    "* Evaluate the prediction performance of these models on the test set by $\\log\\{L[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\beta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n",
+    "* Evaluate the prediction performance of these models on the test set by $\\log\\{L[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\theta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\theta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n",
     "\n",
     "* Repeat the first three steps  such that each sample plays the role of the test set once.\n",
     "\n",
@@ -1524,21 +1524,21 @@
   },
   {
    "cell_type": "markdown",
-   "id": "95e5c8e4",
+   "id": "25b8400a",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "\\frac{1}{n} \\sum_{i = 1}^n \\log\\{L[y_i, \\mathbf{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}.\n",
+    "\\frac{1}{n} \\sum_{i = 1}^n \\log\\{L[y_i, \\mathbf{X}_{i, \\ast}; \\boldsymbol{\\theta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}.\n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7e60f51d",
+   "id": "91b1405d",
    "metadata": {
     "editable": true
    },
@@ -1567,7 +1567,7 @@
   {
    "cell_type": "code",
    "execution_count": 8,
-   "id": "2cef0eb7",
+   "id": "48621261",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1667,7 +1667,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f501c9cf",
+   "id": "2213dc0e",
    "metadata": {
     "editable": true
    },
@@ -1678,7 +1678,7 @@
   {
    "cell_type": "code",
    "execution_count": 9,
-   "id": "30e1e320",
+   "id": "6e7f3e92",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1756,7 +1756,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "383e5c2a",
+   "id": "9c98e1e5",
    "metadata": {
     "editable": true
    },
@@ -1766,7 +1766,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1f9cd409",
+   "id": "b00dd83d",
    "metadata": {
     "editable": true
    },
@@ -1789,8 +1789,8 @@
     "when all predictors are zero (the columns in the design matrix), it\n",
     "may be a bad idea to implement a model which penalizes the intercept.\n",
     "Furthermore, in for example Ridge and Lasso regression, the default solutions\n",
-    "from the library **Scikit-Learn** (when not shrinking $\\beta_0$) for the unknown parameters\n",
-    "$\\boldsymbol{\\beta}$, are derived under the assumption that both $\\boldsymbol{y}$ and\n",
+    "from the library **Scikit-Learn** (when not shrinking $\\theta_0$) for the unknown parameters\n",
+    "$\\boldsymbol{\\theta}$, are derived under the assumption that both $\\boldsymbol{y}$ and\n",
     "$\\boldsymbol{X}$ are zero centered, that is we subtract the mean values.\n",
     "\n",
     "If our predictors represent different scales, then it is important to\n",
@@ -1825,7 +1825,7 @@
   {
    "cell_type": "code",
    "execution_count": 10,
-   "id": "eab81633",
+   "id": "bae38eac",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1852,7 +1852,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0ca51a54",
+   "id": "61bd7f35",
    "metadata": {
     "editable": true
    },
@@ -1866,19 +1866,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fa3a4110",
+   "id": "1a0f8db3",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n",
+    "C(\\theta_0, \\theta_1, ... , \\theta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\theta_0 - \\sum_{j=1}^{p-1} X_{ij}\\theta_j\\right)^2,.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "9f499302",
+   "id": "4b0c41b0",
    "metadata": {
     "editable": true
    },
@@ -1887,49 +1887,49 @@
     "increased penalty for higher differences between predicted and\n",
     "output/target values.\n",
     "\n",
-    "What we have done is to single out the $\\beta_0$ term in the\n",
+    "What we have done is to single out the $\\theta_0$ term in the\n",
     "definition of the mean squared error (MSE).  The design matrix $X$\n",
     "does in this case not contain any intercept column.  When we take the\n",
-    "derivative with respect to $\\beta_0$, we want the derivative to obey"
+    "derivative with respect to $\\theta_0$, we want the derivative to obey"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b1f7ba52",
+   "id": "0aa213f5",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n",
+    "\\frac{\\partial C}{\\partial \\theta_j} = 0,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7ff75b7f",
+   "id": "d0808572",
    "metadata": {
     "editable": true
    },
    "source": [
-    "for all $j$. For $\\beta_0$ we have"
+    "for all $j$. For $\\theta_0$ we have"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "eac37d3c",
+   "id": "cf1692d9",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n",
+    "\\frac{\\partial C}{\\partial \\theta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\theta_0 - \\sum_{j=1}^{p-1} X_{ij} \\theta_j\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6e653464",
+   "id": "cc8156be",
    "metadata": {
     "editable": true
    },
@@ -1939,42 +1939,42 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8afa60a1",
+   "id": "cd70e62c",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n",
+    "\\sum_{i=0}^{n-1} \\theta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\theta_j.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "055b7975",
+   "id": "92c99c16",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Let us specialize first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n",
-    "Our result for $\\beta_0$ simplifies then to"
+    "Let us specialize first to the case where we have only two parameters $\\theta_0$ and $\\theta_1$.\n",
+    "Our result for $\\theta_0$ simplifies then to"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a2bf6e5d",
+   "id": "cf522258",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n",
+    "n\\theta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\theta_1.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e15afbd5",
+   "id": "cc86b9e1",
    "metadata": {
     "editable": true
    },
@@ -1984,19 +1984,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d2f9a064",
+   "id": "58fd6599",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n",
+    "\\theta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\theta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "8e8035d8",
+   "id": "86b6ed8e",
    "metadata": {
     "editable": true
    },
@@ -2006,7 +2006,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0bbcef44",
+   "id": "fac0cd1f",
    "metadata": {
     "editable": true
    },
@@ -2018,7 +2018,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "919462c2",
+   "id": "904fde54",
    "metadata": {
     "editable": true
    },
@@ -2028,7 +2028,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3a0899ec",
+   "id": "21e40e22",
    "metadata": {
     "editable": true
    },
@@ -2040,7 +2040,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9b255ec5",
+   "id": "1fd599b0",
    "metadata": {
     "editable": true
    },
@@ -2050,41 +2050,41 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7716aca1",
+   "id": "69e81981",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0 = \\mu_y - \\beta_1\\mu_{\\boldsymbol{x}_1}.\n",
+    "\\theta_0 = \\mu_y - \\theta_1\\mu_{\\boldsymbol{x}_1}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "24863f48",
+   "id": "b95e7089",
    "metadata": {
     "editable": true
    },
    "source": [
-    "In the general case with more parameters than $\\beta_0$ and $\\beta_1$, we have"
+    "In the general case with more parameters than $\\theta_0$ and $\\theta_1$, we have"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6c10f940",
+   "id": "29fb77c2",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n",
+    "\\theta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\theta_j.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7afefe95",
+   "id": "6c5d2f58",
    "metadata": {
     "editable": true
    },
@@ -2094,19 +2094,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a2e7bc1a",
+   "id": "cbd7ab47",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\beta_j,\n",
+    "\\theta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\theta_j,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "5ded7d78",
+   "id": "e1c475ee",
    "metadata": {
     "editable": true
    },
@@ -2116,7 +2116,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bcac7366",
+   "id": "54ae8c6e",
    "metadata": {
     "editable": true
    },
@@ -2128,7 +2128,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bafe8156",
+   "id": "c63feefb",
    "metadata": {
     "editable": true
    },
@@ -2140,41 +2140,41 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7c91d280",
+   "id": "12498a19",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n",
+    "C(\\boldsymbol{\\theta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\theta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\theta}).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1b81d6de",
+   "id": "75c5b8bb",
    "metadata": {
     "editable": true
    },
    "source": [
-    "If we minimize with respect to $\\boldsymbol{\\beta}$ we have then"
+    "If we minimize with respect to $\\boldsymbol{\\theta}$ we have then"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7d6d497e",
+   "id": "da602e6e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n",
+    "\\hat{\\boldsymbol{\\theta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "772593b2",
+   "id": "3cbb6060",
    "metadata": {
     "editable": true
    },
@@ -2182,24 +2182,24 @@
     "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n",
     "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n",
     "\n",
-    "For Ridge regression we need to add $\\lambda \\boldsymbol{\\beta}^T\\boldsymbol{\\beta}$ to the cost function and get then"
+    "For Ridge regression we need to add $\\lambda \\boldsymbol{\\theta}^T\\boldsymbol{\\theta}$ to the cost function and get then"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "3edbdb56",
+   "id": "139ef52e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n",
+    "\\hat{\\boldsymbol{\\theta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7de9723e",
+   "id": "afd4d249",
    "metadata": {
     "editable": true
    },
@@ -2213,7 +2213,7 @@
   {
    "cell_type": "code",
    "execution_count": 11,
-   "id": "2e4a0363",
+   "id": "99a09833",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2233,15 +2233,15 @@
     "    return np.sum((y_data-y_model)**2)/n\n",
     "\n",
     "\n",
-    "def fit_beta(X, y):\n",
+    "def fit_theta(X, y):\n",
     "    return np.linalg.pinv(X.T @ X) @ X.T @ y\n",
     "\n",
     "\n",
-    "true_beta = [2, 0.5, 3.7]\n",
+    "true_theta = [2, 0.5, 3.7]\n",
     "\n",
     "x = np.linspace(0, 1, 11)\n",
     "y = np.sum(\n",
-    "    np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0\n",
+    "    np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0\n",
     ") + 0.1 * np.random.normal(size=len(x))\n",
     "\n",
     "degree = 3\n",
@@ -2251,15 +2251,15 @@
     "for p in range(degree):\n",
     "    X[:, p] = x ** p\n",
     "\n",
-    "beta = fit_beta(X, y)\n",
+    "theta = fit_theta(X, y)\n",
     "\n",
     "# Intercept is included in the design matrix\n",
     "skl = LinearRegression(fit_intercept=False).fit(X, y)\n",
     "\n",
-    "print(f\"True beta: {true_beta}\")\n",
-    "print(f\"Fitted beta: {beta}\")\n",
-    "print(f\"Sklearn fitted beta: {skl.coef_}\")\n",
-    "ypredictOwn = X @ beta\n",
+    "print(f\"True theta: {true_theta}\")\n",
+    "print(f\"Fitted theta: {theta}\")\n",
+    "print(f\"Sklearn fitted theta: {skl.coef_}\")\n",
+    "ypredictOwn = X @ theta\n",
     "ypredictSKL = skl.predict(X)\n",
     "print(f\"MSE with intercept column\")\n",
     "print(MSE(y,ypredictOwn))\n",
@@ -2269,7 +2269,7 @@
     "\n",
     "plt.figure()\n",
     "plt.scatter(x, y, label=\"Data\")\n",
-    "plt.plot(x, X @ beta, label=\"Fit\")\n",
+    "plt.plot(x, X @ theta, label=\"Fit\")\n",
     "plt.plot(x, skl.predict(X), label=\"Sklearn (fit_intercept=False)\")\n",
     "\n",
     "\n",
@@ -2286,21 +2286,21 @@
     "y_offset = np.average(y, axis=0)\n",
     "X_offset = np.average(X, axis=0)\n",
     "\n",
-    "beta = fit_beta(X - X_offset, y - y_offset)\n",
-    "intercept = np.mean(y_offset - X_offset @ beta)\n",
+    "theta = fit_theta(X - X_offset, y - y_offset)\n",
+    "intercept = np.mean(y_offset - X_offset @ theta)\n",
     "\n",
     "print(f\"Manual intercept: {intercept}\")\n",
-    "print(f\"Fitted beta (wiothout intercept): {beta}\")\n",
+    "print(f\"Fitted theta (wiothout intercept): {theta}\")\n",
     "print(f\"Sklearn intercept: {skl.intercept_}\")\n",
-    "print(f\"Sklearn fitted beta (without intercept): {skl.coef_}\")\n",
-    "ypredictOwn = X @ beta\n",
+    "print(f\"Sklearn fitted theta (without intercept): {skl.coef_}\")\n",
+    "ypredictOwn = X @ theta\n",
     "ypredictSKL = skl.predict(X)\n",
     "print(f\"MSE with Manual intercept\")\n",
     "print(MSE(y,ypredictOwn+intercept))\n",
     "print(f\"MSE with Sklearn intercept\")\n",
     "print(MSE(y,ypredictSKL))\n",
     "\n",
-    "plt.plot(x, X @ beta + intercept, \"--\", label=\"Fit (manual intercept)\")\n",
+    "plt.plot(x, X @ theta + intercept, \"--\", label=\"Fit (manual intercept)\")\n",
     "plt.plot(x, skl.predict(X), \"--\", label=\"Sklearn (fit_intercept=True)\")\n",
     "plt.grid()\n",
     "plt.legend()\n",
@@ -2310,7 +2310,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c6d3a071",
+   "id": "1bacfcef",
    "metadata": {
     "editable": true
    },
@@ -2323,7 +2323,7 @@
     "the way we treat the intercept may give a larger or smaller MSE,\n",
     "meaning that the MSE can be penalized by the value of the\n",
     "intercept. Not including the intercept in the fit, means that the\n",
-    "regularization term does not include $\\beta_0$. For different values\n",
+    "regularization term does not include $\\theta_0$. For different values\n",
     "of $\\lambda$, this may lead to different MSE values. \n",
     "\n",
     "To remind the reader, the regularization term, with the intercept in Ridge regression, is given by"
@@ -2331,19 +2331,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "db803b77",
+   "id": "8061073e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\beta_j^2,\n",
+    "\\lambda \\vert\\vert \\boldsymbol{\\theta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\theta_j^2,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d7a9c9d8",
+   "id": "fcea892f",
    "metadata": {
     "editable": true
    },
@@ -2353,19 +2353,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e08af763",
+   "id": "af6d400e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\beta_j^2.\n",
+    "\\lambda \\vert\\vert \\boldsymbol{\\theta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\theta_j^2.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "10fe24e4",
+   "id": "1bfb0c9e",
    "metadata": {
     "editable": true
    },
@@ -2375,19 +2375,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6feb3bdd",
+   "id": "3b87c68e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\beta_j\\vert.\n",
+    "\\lambda \\vert\\vert \\boldsymbol{\\theta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\theta_j\\vert.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "657a0777",
+   "id": "eae8012d",
    "metadata": {
     "editable": true
    },
@@ -2405,7 +2405,7 @@
   {
    "cell_type": "code",
    "execution_count": 12,
-   "id": "9f21317d",
+   "id": "e02c5a51",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2448,20 +2448,20 @@
     "lambdas = np.logspace(-4, 2, nlambdas)\n",
     "for i in range(nlambdas):\n",
     "    lmb = lambdas[i]\n",
-    "    OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
+    "    OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
     "    # Note: we include the intercept column and no scaling\n",
     "    RegRidge = linear_model.Ridge(lmb,fit_intercept=False)\n",
     "    RegRidge.fit(X_train,y_train)\n",
     "    # and then make the prediction\n",
-    "    ytildeOwnRidge = X_train @ OwnRidgeBeta\n",
-    "    ypredictOwnRidge = X_test @ OwnRidgeBeta\n",
+    "    ytildeOwnRidge = X_train @ OwnRidgeTheta\n",
+    "    ypredictOwnRidge = X_test @ OwnRidgeTheta\n",
     "    ytildeRidge = RegRidge.predict(X_train)\n",
     "    ypredictRidge = RegRidge.predict(X_test)\n",
     "    MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)\n",
     "    MSERidgePredict[i] = MSE(y_test,ypredictRidge)\n",
-    "    print(\"Beta values for own Ridge implementation\")\n",
-    "    print(OwnRidgeBeta)\n",
-    "    print(\"Beta values for Scikit-Learn Ridge implementation\")\n",
+    "    print(\"Theta values for own Ridge implementation\")\n",
+    "    print(OwnRidgeTheta)\n",
+    "    print(\"Theta values for Scikit-Learn Ridge implementation\")\n",
     "    print(RegRidge.coef_)\n",
     "    print(\"MSE values for own Ridge implementation\")\n",
     "    print(MSEOwnRidgePredict[i])\n",
@@ -2481,7 +2481,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "20de82fd",
+   "id": "df4ec33b",
    "metadata": {
     "editable": true
    },
@@ -2495,7 +2495,7 @@
   {
    "cell_type": "code",
    "execution_count": 13,
-   "id": "4138ed50",
+   "id": "019f59bd",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2549,18 +2549,18 @@
     "lambdas = np.logspace(-4, 2, nlambdas)\n",
     "for i in range(nlambdas):\n",
     "    lmb = lambdas[i]\n",
-    "    OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)\n",
-    "    intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data\n",
+    "    OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)\n",
+    "    intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data\n",
     "    #Add intercept to prediction\n",
-    "    ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler \n",
+    "    ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler \n",
     "    RegRidge = linear_model.Ridge(lmb)\n",
     "    RegRidge.fit(X_train,y_train)\n",
     "    ypredictRidge = RegRidge.predict(X_test)\n",
     "    MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)\n",
     "    MSERidgePredict[i] = MSE(y_test,ypredictRidge)\n",
-    "    print(\"Beta values for own Ridge implementation\")\n",
-    "    print(OwnRidgeBeta) #Intercept is given by mean of target variable\n",
-    "    print(\"Beta values for Scikit-Learn Ridge implementation\")\n",
+    "    print(\"Theta values for own Ridge implementation\")\n",
+    "    print(OwnRidgeTheta) #Intercept is given by mean of target variable\n",
+    "    print(\"Theta values for Scikit-Learn Ridge implementation\")\n",
     "    print(RegRidge.coef_)\n",
     "    print('Intercept from own implementation:')\n",
     "    print(intercept_)\n",
@@ -2584,7 +2584,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e9dfda5",
+   "id": "11a0c8f8",
    "metadata": {
     "editable": true
    },
@@ -2592,7 +2592,7 @@
     "We see here, when compared to the code which includes explicitely the\n",
     "intercept column, that our MSE value is actually smaller. This is\n",
     "because the regularization term does not include the intercept value\n",
-    "$\\beta_0$ in the fitting.  This applies to Lasso regularization as\n",
+    "$\\theta_0$ in the fitting.  This applies to Lasso regularization as\n",
     "well.  It means that our optimization is now done only with the\n",
     "centered matrix and/or vector that enter the fitting procedure. Note\n",
     "also that the problem with the intercept occurs mainly in these type\n",
@@ -2603,7 +2603,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a7eb252c",
+   "id": "fca71b1a",
    "metadata": {
     "editable": true
    },
@@ -2616,7 +2616,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "990ff280",
+   "id": "bea82853",
    "metadata": {
     "editable": true
    },
@@ -2634,7 +2634,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5ea46b31",
+   "id": "d92e0742",
    "metadata": {
     "editable": true
    },
@@ -2651,7 +2651,7 @@
   {
    "cell_type": "code",
    "execution_count": 14,
-   "id": "fa60aaab",
+   "id": "039e1f8c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2682,7 +2682,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b9e8214e",
+   "id": "94a8e6b6",
    "metadata": {
     "editable": true
    },
@@ -2698,7 +2698,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a578cd63",
+   "id": "0ba92480",
    "metadata": {
     "editable": true
    },
@@ -2716,7 +2716,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d90dcfee",
+   "id": "313c4354",
    "metadata": {
     "editable": true
    },
@@ -2728,7 +2728,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c4eec69a",
+   "id": "da5f96ae",
    "metadata": {
     "editable": true
    },
@@ -2746,7 +2746,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae46a11b",
+   "id": "14375be5",
    "metadata": {
     "editable": true
    },
@@ -2758,7 +2758,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "87ebb9b8",
+   "id": "58f91570",
    "metadata": {
     "editable": true
    },
@@ -2768,7 +2768,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "    \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon},\n",
+    "    \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\theta} + \\boldsymbol{\\epsilon},\n",
     "\\label{_auto4} \\tag{4}\n",
     "\\end{equation}\n",
     "$$"
@@ -2776,7 +2776,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "856efd3f",
+   "id": "820108ea",
    "metadata": {
     "editable": true
    },
@@ -2787,7 +2787,7 @@
   {
    "cell_type": "code",
    "execution_count": 15,
-   "id": "5ae2828f",
+   "id": "ec4efa81",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2803,7 +2803,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0d6201d9",
+   "id": "aa3656f3",
    "metadata": {
     "editable": true
    },
@@ -2813,7 +2813,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4af380a7",
+   "id": "0c1bbe7c",
    "metadata": {
     "editable": true
    },
@@ -2823,7 +2823,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "    C(\\boldsymbol{X}, \\boldsymbol{\\beta})= \\frac{1}{n}\\left\\{(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})\\right\\}.\n",
+    "    C(\\boldsymbol{X}, \\boldsymbol{\\theta})= \\frac{1}{n}\\left\\{(\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y})\\right\\}.\n",
     "\\label{_auto5} \\tag{5}\n",
     "\\end{equation}\n",
     "$$"
@@ -2831,30 +2831,30 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8e8eefa3",
+   "id": "784bc4e4",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We then find the extremal point of $C$ by taking the derivative with respect to $\\boldsymbol{\\beta}$ as discussed above.\n",
-    "This yields the expression for $\\boldsymbol{\\beta}$ to be"
+    "We then find the extremal point of $C$ by taking the derivative with respect to $\\boldsymbol{\\theta}$ as discussed above.\n",
+    "This yields the expression for $\\boldsymbol{\\theta}$ to be"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a1c0765c",
+   "id": "db1ec5fe",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta} = \\frac{\\boldsymbol{X}^T \\boldsymbol{y}}{\\boldsymbol{X}^T \\boldsymbol{X}},\n",
+    "\\boldsymbol{\\theta} = \\frac{\\boldsymbol{X}^T \\boldsymbol{y}}{\\boldsymbol{X}^T \\boldsymbol{X}},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c754e28b",
+   "id": "5d2e6b17",
    "metadata": {
     "editable": true
    },
@@ -2868,7 +2868,7 @@
   {
    "cell_type": "code",
    "execution_count": 16,
-   "id": "ff056cf9",
+   "id": "6a242f2e",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2887,7 +2887,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "524f259e",
+   "id": "329b50d2",
    "metadata": {
     "editable": true
    },
@@ -2895,24 +2895,24 @@
     "Doing the inversion directly turns out to be a bad idea since the matrix\n",
     "$\\boldsymbol{X}^T\\boldsymbol{X}$ is singular. An alternative approach is to use the **singular\n",
     "value decomposition**. Using the definition of the Moore-Penrose\n",
-    "pseudoinverse we can write the equation for $\\boldsymbol{\\beta}$ as"
+    "pseudoinverse we can write the equation for $\\boldsymbol{\\theta}$ as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "52be4553",
+   "id": "2b544864",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\boldsymbol{\\beta} = \\boldsymbol{X}^{+}\\boldsymbol{y},\n",
+    "\\boldsymbol{\\theta} = \\boldsymbol{X}^{+}\\boldsymbol{y},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c37ff09a",
+   "id": "bf0d128d",
    "metadata": {
     "editable": true
    },
@@ -2922,7 +2922,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ea49a105",
+   "id": "67e527de",
    "metadata": {
     "editable": true
    },
@@ -2934,7 +2934,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bf31067a",
+   "id": "d4ef2ece",
    "metadata": {
     "editable": true
    },
@@ -2947,7 +2947,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5ef2c839",
+   "id": "c4c467f8",
    "metadata": {
     "editable": true
    },
@@ -2957,7 +2957,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "    \\boldsymbol{\\beta} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^{+} \\boldsymbol{U}^T \\boldsymbol{y}.\n",
+    "    \\boldsymbol{\\theta} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^{+} \\boldsymbol{U}^T \\boldsymbol{y}.\n",
     "\\label{_auto6} \\tag{6}\n",
     "\\end{equation}\n",
     "$$"
@@ -2965,7 +2965,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "96bbf134",
+   "id": "ffd1ba3d",
    "metadata": {
     "editable": true
    },
@@ -2981,7 +2981,7 @@
   {
    "cell_type": "code",
    "execution_count": 17,
-   "id": "5e729b1e",
+   "id": "9cdf3698",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2996,19 +2996,19 @@
   {
    "cell_type": "code",
    "execution_count": 18,
-   "id": "80b923d1",
+   "id": "164ccc0c",
    "metadata": {
     "collapsed": false,
     "editable": true
    },
    "outputs": [],
    "source": [
-    "beta = ols_svd(X_train_own,y_train)"
+    "theta = ols_svd(X_train_own,y_train)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b4cf2f1a",
+   "id": "a23f2826",
    "metadata": {
     "editable": true
    },
@@ -3019,19 +3019,19 @@
   {
    "cell_type": "code",
    "execution_count": 19,
-   "id": "93be2c0d",
+   "id": "a164aca2",
    "metadata": {
     "collapsed": false,
     "editable": true
    },
    "outputs": [],
    "source": [
-    "J = beta[1:].reshape(L, L)"
+    "J = theta[1:].reshape(L, L)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ae869089",
+   "id": "3bb35ae1",
    "metadata": {
     "editable": true
    },
@@ -3042,7 +3042,7 @@
   {
    "cell_type": "code",
    "execution_count": 20,
-   "id": "8d6d8152",
+   "id": "93445ba4",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3061,7 +3061,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8092ac84",
+   "id": "f602ed47",
    "metadata": {
     "editable": true
    },
@@ -3084,7 +3084,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d9251cde",
+   "id": "86d25ab0",
    "metadata": {
     "editable": true
    },
@@ -3102,7 +3102,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "abe454c9",
+   "id": "2008a25b",
    "metadata": {
     "editable": true
    },
@@ -3115,7 +3115,7 @@
   {
    "cell_type": "code",
    "execution_count": 21,
-   "id": "89f8fbd0",
+   "id": "01b2b75a",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3147,7 +3147,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e06639d4",
+   "id": "d6b5a067",
    "metadata": {
     "editable": true
    },
@@ -3157,7 +3157,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "43ef0f40",
+   "id": "497d84f1",
    "metadata": {
     "editable": true
    },
@@ -3175,7 +3175,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "523dc03b",
+   "id": "38cc642a",
    "metadata": {
     "editable": true
    },
@@ -3187,7 +3187,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0f389eef",
+   "id": "bec7f36d",
    "metadata": {
     "editable": true
    },
@@ -3205,7 +3205,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b6e00f24",
+   "id": "14ad8f62",
    "metadata": {
     "editable": true
    },
@@ -3217,7 +3217,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5ea1084d",
+   "id": "4ccac4ea",
    "metadata": {
     "editable": true
    },
@@ -3227,7 +3227,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "    \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon}.\n",
+    "    \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\theta} + \\boldsymbol{\\epsilon}.\n",
     "\\label{_auto10} \\tag{10}\n",
     "\\end{equation}\n",
     "$$"
@@ -3235,7 +3235,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5551bb05",
+   "id": "cabb6ced",
    "metadata": {
     "editable": true
    },
@@ -3246,7 +3246,7 @@
   {
    "cell_type": "code",
    "execution_count": 22,
-   "id": "f5dd7795",
+   "id": "ce90ccbe",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3272,7 +3272,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f36c807b",
+   "id": "6f6d672f",
    "metadata": {
     "editable": true
    },
@@ -3283,7 +3283,7 @@
   {
    "cell_type": "code",
    "execution_count": 23,
-   "id": "eb701dd4",
+   "id": "945777f9",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3295,7 +3295,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bd356ada",
+   "id": "ffc8317f",
    "metadata": {
     "editable": true
    },
@@ -3306,7 +3306,7 @@
   {
    "cell_type": "code",
    "execution_count": 24,
-   "id": "ae737db8",
+   "id": "e17c598f",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3318,7 +3318,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d152a32a",
+   "id": "3396f010",
    "metadata": {
     "editable": true
    },
@@ -3329,7 +3329,7 @@
   {
    "cell_type": "code",
    "execution_count": 25,
-   "id": "8d713977",
+   "id": "311d3cea",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3348,7 +3348,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ede30c18",
+   "id": "13705eed",
    "metadata": {
     "editable": true
    },
@@ -3358,13 +3358,13 @@
     "Having explored the ordinary least squares we move on to ridge\n",
     "regression. In ridge regression we include a **regularizer**. This\n",
     "involves a new cost function which leads to a new estimate for the\n",
-    "weights $\\boldsymbol{\\beta}$. This results in a penalized regression problem. The\n",
+    "weights $\\boldsymbol{\\theta}$. This results in a penalized regression problem. The\n",
     "cost function is given by"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "eac772e9",
+   "id": "4cd9530c",
    "metadata": {
     "editable": true
    },
@@ -3374,7 +3374,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "    C(\\boldsymbol{X}, \\boldsymbol{\\beta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y}) + \\lambda \\boldsymbol{\\beta}^T\\boldsymbol{\\beta}.\n",
+    "    C(\\boldsymbol{X}, \\boldsymbol{\\theta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y}) + \\lambda \\boldsymbol{\\theta}^T\\boldsymbol{\\theta}.\n",
     "\\label{_auto11} \\tag{11}\n",
     "\\end{equation}\n",
     "$$"
@@ -3383,7 +3383,7 @@
   {
    "cell_type": "code",
    "execution_count": 26,
-   "id": "b66675f0",
+   "id": "d1c9a462",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3406,7 +3406,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "71aa37b5",
+   "id": "97e6303c",
    "metadata": {
     "editable": true
    },
@@ -3416,7 +3416,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8b70ef66",
+   "id": "8631d307",
    "metadata": {
     "editable": true
    },
@@ -3426,7 +3426,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "    C(\\boldsymbol{X}, \\boldsymbol{\\beta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y}) + \\lambda \\sqrt{\\boldsymbol{\\beta}^T\\boldsymbol{\\beta}}.\n",
+    "    C(\\boldsymbol{X}, \\boldsymbol{\\theta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\theta} - \\boldsymbol{y}) + \\lambda \\sqrt{\\boldsymbol{\\theta}^T\\boldsymbol{\\theta}}.\n",
     "\\label{_auto12} \\tag{12}\n",
     "\\end{equation}\n",
     "$$"
@@ -3434,7 +3434,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "079981ff",
+   "id": "dc2a06fb",
    "metadata": {
     "editable": true
    },
@@ -3445,7 +3445,7 @@
   {
    "cell_type": "code",
    "execution_count": 27,
-   "id": "c8d4a180",
+   "id": "2f3333c6",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3467,7 +3467,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "57839941",
+   "id": "692844ac",
    "metadata": {
     "editable": true
    },
@@ -3482,7 +3482,7 @@
   {
    "cell_type": "code",
    "execution_count": 28,
-   "id": "0e3bbb4e",
+   "id": "d40c4d10",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3529,7 +3529,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "14711abd",
+   "id": "965b51d2",
    "metadata": {
     "editable": true
    },
@@ -3550,7 +3550,7 @@
   {
    "cell_type": "code",
    "execution_count": 29,
-   "id": "9ff29a72",
+   "id": "04fe5f87",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3591,7 +3591,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9c10be56",
+   "id": "845d9279",
    "metadata": {
     "editable": true
    },
@@ -3603,7 +3603,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "efce3b63",
+   "id": "6d8674ba",
    "metadata": {
     "editable": true
    },
@@ -3629,7 +3629,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ef5a4df0",
+   "id": "b26b187d",
    "metadata": {
     "editable": true
    },
@@ -3644,7 +3644,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b085eff4",
+   "id": "76e8040d",
    "metadata": {
     "editable": true
    },
@@ -3674,7 +3674,7 @@
   {
    "cell_type": "code",
    "execution_count": 30,
-   "id": "5056dccb",
+   "id": "7608edfe",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3724,7 +3724,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f5e72aef",
+   "id": "c0d0b45f",
    "metadata": {
     "editable": true
    },
@@ -3741,13 +3741,13 @@
     "value decomposition from e.g., **numpy** ) or use your code from\n",
     "homeworks 1 and 2 and perform a standard least square regression\n",
     "analysis using polynomials in $x$ and $y$ up to fifth order. Find the\n",
-    "[confidence intervals](https://en.wikipedia.org/wiki/Confidence_interval) of the parameters (estimators) $\\beta$ by computing their\n",
+    "[confidence intervals](https://en.wikipedia.org/wiki/Confidence_interval) of the parameters (estimators) $\\theta$ by computing their\n",
     "variances, evaluate the Mean Squared error (MSE)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "0ba33237",
+   "id": "76861250",
    "metadata": {
     "editable": true
    },
@@ -3760,7 +3760,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1f9af2ef",
+   "id": "90074ad8",
    "metadata": {
     "editable": true
    },
@@ -3772,7 +3772,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "77d5b0f2",
+   "id": "31d7a9ad",
    "metadata": {
     "editable": true
    },
@@ -3784,7 +3784,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6c1f9df9",
+   "id": "b6bcc7b7",
    "metadata": {
     "editable": true
    },
@@ -3794,7 +3794,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1d06102c",
+   "id": "45f3893f",
    "metadata": {
     "editable": true
    },
@@ -3806,7 +3806,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8fb40b08",
+   "id": "8fea6a0a",
    "metadata": {
     "editable": true
    },
@@ -3831,7 +3831,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1701de47",
+   "id": "6283b437",
    "metadata": {
     "editable": true
    },
@@ -3863,7 +3863,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "15110cdf",
+   "id": "62f77e00",
    "metadata": {
     "editable": true
    },
@@ -3875,7 +3875,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "33046595",
+   "id": "7380e759",
    "metadata": {
     "editable": true
    },
@@ -3885,28 +3885,28 @@
     "\n",
     "In our derivation of the ordinary least squares method we defined then\n",
     "an approximation to the function $f$ in terms of the parameters\n",
-    "$\\boldsymbol{\\beta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n",
-    "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\beta}$.\n",
+    "$\\boldsymbol{\\theta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n",
+    "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\theta}$.\n",
     "\n",
-    "The parameters $\\boldsymbol{\\beta}$ are in turn found by optimizing the means\n",
+    "The parameters $\\boldsymbol{\\theta}$ are in turn found by optimizing the means\n",
     "squared error via the so-called cost function"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "84527747",
+   "id": "b5c040e0",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n",
+    "C(\\boldsymbol{X},\\boldsymbol{\\theta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "da2f876d",
+   "id": "2cf7f08f",
    "metadata": {
     "editable": true
    },
@@ -3918,7 +3918,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1845a7dc",
+   "id": "8cf351f9",
    "metadata": {
     "editable": true
    },
@@ -3930,7 +3930,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d2f1d7d7",
+   "id": "53fff2ad",
    "metadata": {
     "editable": true
    },
@@ -3950,7 +3950,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e898c902",
+   "id": "89dec1fb",
    "metadata": {
     "editable": true
    },
@@ -3975,7 +3975,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d23dc734",
+   "id": "2d5f7364",
    "metadata": {
     "editable": true
    },
@@ -3995,7 +3995,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "39a35330",
+   "id": "d56e6dc8",
    "metadata": {
     "editable": true
    },
@@ -4012,7 +4012,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "44d9e821",
+   "id": "608453fd",
    "metadata": {
     "editable": true
    },
@@ -4040,7 +4040,7 @@
   {
    "cell_type": "code",
    "execution_count": 31,
-   "id": "0a4e6d7e",
+   "id": "9399d087",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4052,7 +4052,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "16a73292",
+   "id": "ae1d9c8f",
    "metadata": {
     "editable": true
    },
@@ -4064,7 +4064,7 @@
   {
    "cell_type": "code",
    "execution_count": 32,
-   "id": "168356a4",
+   "id": "381e9da0",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4092,7 +4092,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e9ec3e68",
+   "id": "6e1c4af1",
    "metadata": {
     "editable": true
    },
diff --git a/doc/LectureNotes/chapter4.ipynb b/doc/LectureNotes/chapter4.ipynb
index bdfb97b2b..934d454cf 100644
--- a/doc/LectureNotes/chapter4.ipynb
+++ b/doc/LectureNotes/chapter4.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "f63b8c1a",
+   "id": "225f0ca7",
    "metadata": {
     "editable": true
    },
@@ -13,7 +13,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9a284cf5",
+   "id": "a48a2920",
    "metadata": {
     "editable": true
    },
@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d78b73e2",
+   "id": "90b6e822",
    "metadata": {
     "editable": true
    },
@@ -38,7 +38,7 @@
     "analytical expressions for standard ordinary Least Squares or Ridge\n",
     "regression (in terms of matrices to invert) for several quantities,\n",
     "ranging from the variance and thereby the confidence intervals of the\n",
-    "optimal parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n",
+    "optimal parameters $\\hat{\\theta}$ to the mean squared error. If we can invert\n",
     "the product of the design matrices, linear regression gives then a\n",
     "simple recipe for fitting our data.\n",
     "\n",
@@ -61,7 +61,7 @@
     "Logistic regression will also serve as our stepping stone towards\n",
     "neural network algorithms and supervised deep learning. For logistic\n",
     "learning, the minimization of the cost function leads to a non-linear\n",
-    "equation in the parameters $\\hat{\\beta}$. The optimization of the\n",
+    "equation in the parameters $\\hat{\\theta}$. The optimization of the\n",
     "problem calls therefore for minimization algorithms. This forms the\n",
     "bottle neck of all machine learning algorithms, namely how to find\n",
     "reliable minima of a multi-variable function. This leads us to the\n",
@@ -75,7 +75,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d0c829b4",
+   "id": "08ebdf8e",
    "metadata": {
     "editable": true
    },
@@ -100,7 +100,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f1f4d5f2",
+   "id": "f08eca95",
    "metadata": {
     "editable": true
    },
@@ -112,7 +112,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8e2f836b",
+   "id": "d82a2a29",
    "metadata": {
     "editable": true
    },
@@ -128,7 +128,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6b58f1ba",
+   "id": "467f1d23",
    "metadata": {
     "editable": true
    },
@@ -138,7 +138,7 @@
     "\n",
     "$$\n",
     "\\begin{equation}\n",
-    "\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\beta} +  \\boldsymbol{\\epsilon},\n",
+    "\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\theta} +  \\boldsymbol{\\epsilon},\n",
     "\\label{_auto1} \\tag{1}\n",
     "\\end{equation}\n",
     "$$"
@@ -146,13 +146,13 @@
   },
   {
    "cell_type": "markdown",
-   "id": "54e3604b",
+   "id": "0f2560d0",
    "metadata": {
     "editable": true
    },
    "source": [
     "where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n",
-    "$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors.\n",
+    "$n\\times p$ design matrix and $\\boldsymbol{\\theta}$ represents our estimators/predictors.\n",
     "\n",
     "The main problem with our function is that it takes values on the\n",
     "entire real axis. In the case of logistic regression, however, the\n",
@@ -175,7 +175,7 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "cd1dd1c5",
+   "id": "3916f494",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -242,7 +242,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2e06ee82",
+   "id": "df52e736",
    "metadata": {
     "editable": true
    },
@@ -253,7 +253,7 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "a774f001",
+   "id": "07b5c0a7",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -272,7 +272,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e102fa56",
+   "id": "25ce17fc",
    "metadata": {
     "editable": true
    },
@@ -283,19 +283,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "75d995ba",
+   "id": "818e2f55",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "f(y_i\\vert x_i)=\\beta_0+\\beta_1 x_i.\n",
+    "f(y_i\\vert x_i)=\\theta_0+\\theta_1 x_i.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "e2cacca5",
+   "id": "b2616b20",
    "metadata": {
     "editable": true
    },
@@ -314,7 +314,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "af9ac586",
+   "id": "ab85db3c",
    "metadata": {
     "editable": true
    },
@@ -336,7 +336,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "80aa3cbe",
+   "id": "efbd4772",
    "metadata": {
     "editable": true
    },
@@ -348,7 +348,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "28d49915",
+   "id": "1b9231ea",
    "metadata": {
     "editable": true
    },
@@ -358,7 +358,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eeff6cd9",
+   "id": "ad6acb90",
    "metadata": {
     "editable": true
    },
@@ -371,7 +371,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "5ecec05e",
+   "id": "bf66d0dd",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -436,56 +436,56 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5802efe3",
+   "id": "363760d3",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\beta$ in our fitting of the Sigmoid function, that is we define probabilities"
+    "We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\theta$ in our fitting of the Sigmoid function, that is we define probabilities"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f85d14ea",
+   "id": "eefe7b8d",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "p(y_i=1|x_i,\\boldsymbol{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
-    "p(y_i=0|x_i,\\boldsymbol{\\beta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\beta}),\n",
+    "p(y_i=1|x_i,\\boldsymbol{\\theta}) &= \\frac{\\exp{(\\theta_0+\\theta_1x_i)}}{1+\\exp{(\\theta_0+\\theta_1x_i)}},\\nonumber\\\\\n",
+    "p(y_i=0|x_i,\\boldsymbol{\\theta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\theta}),\n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6c4e0334",
+   "id": "102fb347",
    "metadata": {
     "editable": true
    },
    "source": [
-    "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
+    "where $\\boldsymbol{\\theta}$ are the weights we wish to extract from data, in our case $\\theta_0$ and $\\theta_1$. \n",
     "\n",
     "Note that we used"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "df7facc9",
+   "id": "0b230504",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n",
+    "p(y_i=0\\vert x_i, \\boldsymbol{\\theta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\theta}).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "a920e6d8",
+   "id": "669b4c9f",
    "metadata": {
     "editable": true
    },
@@ -500,21 +500,21 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a780bfe1",
+   "id": "5dc36460",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
     "\\begin{align*}\n",
-    "P(\\mathcal{D}|\\boldsymbol{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n",
+    "P(\\mathcal{D}|\\boldsymbol{\\theta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\theta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\theta}))\\right]^{1-y_i}\\nonumber \\\\\n",
     "\\end{align*}\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "9f810a44",
+   "id": "0874a9c6",
    "metadata": {
     "editable": true
    },
@@ -524,19 +524,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3fac4ef4",
+   "id": "212397d3",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n",
+    "\\mathcal{C}(\\boldsymbol{\\theta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\theta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\theta}))\\right]\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "75aa2c14",
+   "id": "f99396bf",
    "metadata": {
     "editable": true
    },
@@ -546,42 +546,42 @@
   },
   {
    "cell_type": "markdown",
-   "id": "780f2038",
+   "id": "fd5e7b5f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n  \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
+    "\\mathcal{C}(\\boldsymbol{\\theta}) = \\sum_{i=1}^n  \\left(y_i(\\theta_0+\\theta_1x_i) -\\log{(1+\\exp{(\\theta_0+\\theta_1x_i)})}\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c8c940aa",
+   "id": "80dbca63",
    "metadata": {
     "editable": true
    },
    "source": [
-    "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n",
+    "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\theta$.\n",
     "Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "9d4a527b",
+   "id": "f5d6c58f",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n  \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
+    "\\mathcal{C}(\\boldsymbol{\\theta})=-\\sum_{i=1}^n  \\left(y_i(\\theta_0+\\theta_1x_i) -\\log{(1+\\exp{(\\theta_0+\\theta_1x_i)})}\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c1706606",
+   "id": "57e33c81",
    "metadata": {
     "editable": true
    },
@@ -589,28 +589,28 @@
     "This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n",
     "in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.\n",
     "\n",
-    "The cross entropy is a convex function of the weights $\\boldsymbol{\\beta}$ and,\n",
+    "The cross entropy is a convex function of the weights $\\boldsymbol{\\theta}$ and,\n",
     "therefore, any local minimizer is a global minimizer. \n",
     "\n",
     "Minimizing this\n",
-    "cost function with respect to the two parameters $\\beta_0$ and $\\beta_1$ we obtain"
+    "cost function with respect to the two parameters $\\theta_0$ and $\\theta_1$ we obtain"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "fe8fb387",
+   "id": "1833cc5d",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n  \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
+    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\theta_0} = -\\sum_{i=1}^n  \\left(y_i -\\frac{\\exp{(\\theta_0+\\theta_1x_i)}}{1+\\exp{(\\theta_0+\\theta_1x_i)}}\\right),\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "bb564e81",
+   "id": "ea34e8e6",
    "metadata": {
     "editable": true
    },
@@ -620,67 +620,67 @@
   },
   {
    "cell_type": "markdown",
-   "id": "700c5443",
+   "id": "f1bc373e",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n  \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
+    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\theta_1} = -\\sum_{i=1}^n  \\left(y_ix_i -x_i\\frac{\\exp{(\\theta_0+\\theta_1x_i)}}{1+\\exp{(\\theta_0+\\theta_1x_i)}}\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f442c956",
+   "id": "617c6d86",
    "metadata": {
     "editable": true
    },
    "source": [
     "Let us now define a vector $\\boldsymbol{y}$ with $n$ elements $y_i$, an\n",
     "$n\\times p$ matrix $\\boldsymbol{X}$ which contains the $x_i$ values and a\n",
-    "vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\beta})$. We can rewrite in a more compact form the first\n",
+    "vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\theta})$. We can rewrite in a more compact form the first\n",
     "derivative of cost function as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "83937f6f",
+   "id": "e79426c6",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
+    "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "71819c1d",
+   "id": "b1dd9451",
    "metadata": {
     "editable": true
    },
    "source": [
     "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
-    "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
+    "$p(y_i\\vert x_i,\\boldsymbol{\\theta})(1-p(y_i\\vert x_i,\\boldsymbol{\\theta})$, we can obtain a compact expression of the second derivative as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "23dfd975",
+   "id": "ac6500cc",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
+    "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}\\partial \\boldsymbol{\\theta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6ca9f74b",
+   "id": "12f4d5d0",
    "metadata": {
     "editable": true
    },
@@ -690,41 +690,41 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ddb3d93a",
+   "id": "b4733616",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
+    "\\log{ \\frac{p(\\boldsymbol{\\theta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\theta}\\boldsymbol{x})}} = \\theta_0+\\theta_1x_1+\\theta_2x_2+\\dots+\\theta_px_p.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "0cfae560",
+   "id": "eec7d85e",
    "metadata": {
     "editable": true
    },
    "source": [
-    "Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
+    "Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\theta}=[\\theta_0, \\theta_1, \\dots, \\theta_p]$ leading to"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7c427fc3",
+   "id": "152e6fa3",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
+    "p(\\boldsymbol{\\theta}\\boldsymbol{x})=\\frac{ \\exp{(\\theta_0+\\theta_1x_1+\\theta_2x_2+\\dots+\\theta_px_p)}}{1+\\exp{(\\theta_0+\\theta_1x_1+\\theta_2x_2+\\dots+\\theta_px_p)}}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6a6814bf",
+   "id": "b8168ae0",
    "metadata": {
     "editable": true
    },
@@ -736,19 +736,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "85282137",
+   "id": "3af519ee",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\beta_{10}+\\beta_{11}x_1,\n",
+    "\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\theta_{10}+\\theta_{11}x_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4864f76d",
+   "id": "fa02c958",
    "metadata": {
     "editable": true
    },
@@ -758,19 +758,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "390e9a55",
+   "id": "c550e015",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n",
+    "\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\theta_{20}+\\theta_{21}x_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "c3f39c44",
+   "id": "8398ec7a",
    "metadata": {
     "editable": true
    },
@@ -780,19 +780,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "92f32a03",
+   "id": "6be24150",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n",
+    "\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\theta_{(K-1)0}+\\theta_{(K-1)1}x_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "677a5c4d",
+   "id": "2153c8e3",
    "metadata": {
     "editable": true
    },
@@ -810,25 +810,25 @@
     "multinomial logistic regression and linear discriminant analysis, the\n",
     "input to the function is the result of $K$ distinct linear functions,\n",
     "and the predicted probability for the $k$-th class given a sample\n",
-    "vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\beta}$ is (with two\n",
+    "vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\theta}$ is (with two\n",
     "predictors):"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "fa70e3bf",
+   "id": "ab540777",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n",
+    "p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\theta_{k0}+\\theta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\theta_{l0}+\\theta_{l1}x_1)}}.\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4c59400c",
+   "id": "b7bcd7e8",
    "metadata": {
     "editable": true
    },
@@ -838,19 +838,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f4073652",
+   "id": "c9605782",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n",
+    "p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\theta_{l0}+\\theta_{l1}x_1)}},\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "810e872a",
+   "id": "856dd896",
    "metadata": {
     "editable": true
    },
@@ -867,7 +867,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f4ebc1da",
+   "id": "73029fad",
    "metadata": {
     "editable": true
    },
@@ -882,7 +882,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "b7a8cbac",
+   "id": "d81a44e3",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -918,7 +918,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "10f74b93",
+   "id": "83a2e903",
    "metadata": {
     "editable": true
    },
@@ -930,7 +930,7 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "d0b8025d",
+   "id": "976fc321",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -975,7 +975,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9d84a2aa",
+   "id": "ea10d953",
    "metadata": {
     "editable": true
    },
@@ -998,7 +998,7 @@
   {
    "cell_type": "code",
    "execution_count": 6,
-   "id": "6144ea0a",
+   "id": "42204f9c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1010,7 +1010,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5627f5ae",
+   "id": "a56de206",
    "metadata": {
     "editable": true
    },
@@ -1021,7 +1021,7 @@
   {
    "cell_type": "code",
    "execution_count": 7,
-   "id": "a7c8662a",
+   "id": "0f2ff030",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1033,7 +1033,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d03ee2a7",
+   "id": "68abe92d",
    "metadata": {
     "editable": true
    },
@@ -1059,7 +1059,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "44ce5095",
+   "id": "3cea1c5a",
    "metadata": {
     "editable": true
    },
@@ -1072,7 +1072,7 @@
   {
    "cell_type": "code",
    "execution_count": 8,
-   "id": "24ff3dd3",
+   "id": "f42e1d87",
    "metadata": {
     "collapsed": false,
     "editable": true