diff --git a/doc/BookChapters/chapter1.dlog b/doc/BookChapters/chapter1.dlog index 6162c8df1..875fd7507 100644 --- a/doc/BookChapters/chapter1.dlog +++ b/doc/BookChapters/chapter1.dlog @@ -34,3 +34,12 @@ Translating doconce text in chapter1.do.txt to ipynb Failed to remove ans_at_end environment Failed to remove sol_at_end environment output in chapter1.ipynb +Translating doconce text in chapter1.do.txt to ipynb +*** replacing \bm{...} by \boldsymbol{...} (\bm is not supported by MathJax) + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. Stick to \[ ... \], equation, equation*, align, or align* environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. Stick to \[ ... \], equation, equation*, align, or align* environments in math environments. +Failed to remove ans_at_end environment +Failed to remove sol_at_end environment +output in chapter1.ipynb diff --git a/doc/BookChapters/chapter1.do.txt b/doc/BookChapters/chapter1.do.txt index bcf98478a..5e738504e 100644 --- a/doc/BookChapters/chapter1.do.txt +++ b/doc/BookChapters/chapter1.do.txt @@ -236,7 +236,7 @@ We start with perhaps our simplest possible example, using _Scikit-Learn_ to per What follows is a simple Python code where we have defined a function $y$ in terms of the variable $x$. Both are defined as vectors with $100$ entries. -The numbers in the vector $\hat{x}$ are given +The numbers in the vector $\bm{x}$ are given by random numbers generated with a uniform distribution with entries $x_i \in [0,1]$ (more about probability distribution functions later). These values are then used to define a function $y(x)$ @@ -264,7 +264,7 @@ where $N(0,1)$ represents random numbers generated by the normal distribution. From _Scikit-Learn_ we import then the _LinearRegression_ functionality and make a prediction $\tilde{y} = \alpha + \beta x$ using the function _fit(x,y)_. We call the set of -data $(\hat{x},\hat{y})$ for our training data. The Python package +data $(\bm{x},\bm{y})$ for our training data. The Python package _scikit-learn_ has also a functionality which extracts the above fitting parameters $\alpha$ and $\beta$ (see below). Later we will distinguish between training data and test data. @@ -359,7 +359,7 @@ the relative error (why would we prefer the MSE instead of the relative error?) !bt \[ -\epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}. +\epsilon_{\mathrm{relative}}= \frac{\vert \bm{y} -\bm{\tilde{y}}\vert}{\vert \bm{y}\vert}. \] !et @@ -435,7 +435,7 @@ plt.show() The function _coef_ gives us the parameter $\beta$ of our fit while _intercept_ yields $\alpha$. Depending on the constant in front of the normal distribution, we get values near or far from $alpha =2$ and $\beta =5$. Try to play around with different parameters in front of the normal distribution. The function _meansquarederror_ gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as !bt -\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\[ MSE(\bm{y},\bm{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, \] !et @@ -448,16 +448,16 @@ The _r2score_ function computes $R^2$, the coefficient of determination. It provides a measure of how well future samples are likely to be predicted by the model. Best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A -constant model that always predicts the expected value of $\hat{y}$, +constant model that always predicts the expected value of $\bm{y}$, disregarding the input features, would get a $R^2$ score of $0.0$. -If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +If $\tilde{\bm{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as !bt \[ -R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +R^2(\bm{y}, \tilde{\bm{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, \] !et -where we have defined the mean value of $\hat{y}$ as +where we have defined the mean value of $\bm{y}$ as !bt \[ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. @@ -468,14 +468,14 @@ Another quantity taht we will meet again in our discussions of regression analys The MAE is defined as follows !bt \[ -\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|. +\text{MAE}(\bm{y}, \bm{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|. \] !et We present the squared logarithmic (quadratic) error !bt \[ -\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2, +\text{MSLE}(\bm{y}, \bm{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2, \] !et @@ -2117,18 +2117,18 @@ o Write your own code (following the examples under the "regression notes":"http o Use thereafter _scikit-learn_ (see again the examples in the regression slides) and compare with your own code. o Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as !bt -\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\[ MSE(\bm{y},\bm{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, \] !et and the $R^2$ score function. -If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +If $\tilde{\bm{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as !bt \[ -R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +R^2(\bm{y}, \tilde{\bm{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, \] !et -where we have defined the mean value of $\hat{y}$ as +where we have defined the mean value of $\bm{y}$ as !bt \[ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. diff --git a/doc/BookChapters/chapter3.do.txt b/doc/BookChapters/chapter3.do.txt index 823e50767..5c1b8e90a 100644 --- a/doc/BookChapters/chapter3.do.txt +++ b/doc/BookChapters/chapter3.do.txt @@ -526,7 +526,7 @@ Before we discuss the link between for example Ridge regression and the singular the definition of the covariance and the correlation function. These are quantities Suppose we have defined two vectors -$\hat{x}$ and $\hat{y}$ with $n$ elements each. The covariance matrix $\bm{C}$ is defined as +$\bm{x}$ and $\bm{y}$ with $n$ elements each. The covariance matrix $\bm{C}$ is defined as !bt \[ \bm{C}[\bm{x},\bm{y}] = \begin{bmatrix} \mathrm{cov}[\bm{x},\bm{x}] & \mathrm{cov}[\bm{x},\bm{y}] \\ diff --git a/doc/BookChapters/chapter4.do.txt b/doc/BookChapters/chapter4.do.txt index bd2cbcebe..c7a2771c6 100644 --- a/doc/BookChapters/chapter4.do.txt +++ b/doc/BookChapters/chapter4.do.txt @@ -10,11 +10,11 @@ In linear regression our main interest was centered on learning the coefficients of a functional fit (say a polynomial) in order to be able to predict the response of a continuous variable on some unseen data. The fit to the continuous variable $y_i$ is based on some -independent variables $\hat{x}_i$. Linear regression resulted in +independent variables $x_i$. Linear regression resulted in analytical expressions for standard ordinary Least Squares or Ridge regression (in terms of matrices to invert) for several quantities, ranging from the variance and thereby the confidence intervals of the -parameters $\hat{\beta}$ to the mean squared error. If we can invert +optimal parameters $\hat{\beta}$ to the mean squared error. If we can invert the product of the design matrices, linear regression gives then a simple recipe for fitting our data. @@ -59,7 +59,7 @@ responses or the outcomes, $y_i$ are discrete and only take values from $k=0,\dots,K-1$ (i.e. $K$ classes). The goal is to predict the -output classes from the design matrix $\hat{X}\in\mathbb{R}^{n\times p}$ +output classes from the design matrix $\bm{X}\in\mathbb{R}^{n\times p}$ made of $n$ samples, each of which carries $p$ features or predictors. The primary goal is to identify the classes to which new unseen samples belong. @@ -88,11 +88,11 @@ We would then have our weighted linear combination, namely !bt \begin{equation} -\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon}, +\bm{y} = \bm{X}^T\bm{\beta} + \bm{\epsilon}, \end{equation} !et -where $\hat{y}$ is a vector representing the possible outcomes, $\hat{X}$ is our -$n\times p$ design matrix and $\hat{\beta}$ represents our estimators/predictors. +where $\bm{y}$ is a vector representing the possible outcomes, $\bm{X}$ is our +$n\times p$ design matrix and $\bm{\beta}$ represents our estimators/predictors. The main problem with our function is that it takes values on the @@ -296,16 +296,16 @@ plt.show() We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\beta$ in our fitting of the Sigmoid function, that is we define probabilities !bt \begin{align*} -p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ -p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}), +p(y_i=1|x_i,\bm{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\bm{\beta}) &= 1 - p(y_i=1|x_i,\bm{\beta}), \end{align*} !et -where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$. +where $\bm{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$. Note that we used !bt \[ -p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}). +p(y_i=0\vert x_i, \bm{\beta}) = 1-p(y_i=1\vert x_i, \bm{\beta}). \] !et @@ -318,13 +318,13 @@ the probability of seeing the observed data. We can then approximate the likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is !bt \begin{align*} -P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\ +P(\mathcal{D}|\bm{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\bm{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\bm{\beta}))\right]^{1-y_i}\nonumber \\ \end{align*} !et from which we obtain the log-likelihood and our _cost/loss_ function !bt \[ -\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right). +\mathcal{C}(\bm{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\bm{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\bm{\beta}))\right]\right). \] !et @@ -332,7 +332,7 @@ from which we obtain the log-likelihood and our _cost/loss_ function Reordering the logarithms, we can rewrite the _cost/loss_ function as !bt \[ -\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\mathcal{C}(\bm{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). \] !et @@ -340,14 +340,14 @@ The maximum likelihood estimator is defined as the set of parameters that maximi Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that !bt \[ -\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\mathcal{C}(\bm{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). \] !et This equation is known in statistics as the _cross entropy_. Finally, we note that just as in linear regression, in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression. -The cross entropy is a convex function of the weights $\hat{\beta}$ and, +The cross entropy is a convex function of the weights $\bm{\beta}$ and, therefore, any local minimizer is a global minimizer. @@ -356,34 +356,34 @@ cost function with respect to the two parameters $\beta_0$ and $\beta_1$ we obta !bt \[ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), +\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), \] !et and !bt \[ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). +\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). \] !et -Let us now define a vector $\hat{y}$ with $n$ elements $y_i$, an -$n\times p$ matrix $\hat{X}$ which contains the $x_i$ values and a -vector $\hat{p}$ of fitted probabilities $p(y_i\vert x_i,\hat{\beta})$. We can rewrite in a more compact form the first +Let us now define a vector $\bm{y}$ with $n$ elements $y_i$, an +$n\times p$ matrix $\bm{X}$ which contains the $x_i$ values and a +vector $\bm{p}$ of fitted probabilities $p(y_i\vert x_i,\bm{\beta})$. We can rewrite in a more compact form the first derivative of cost function as !bt \[ -\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}} = -\bm{X}^T\left(\bm{y}-\bm{p}\right). \] !et -If we in addition define a diagonal matrix $\hat{W}$ with elements -$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as +If we in addition define a diagonal matrix $\bm{W}$ with elements +$p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compact expression of the second derivative as !bt \[ -\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +\frac{\partial^2 \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T} = \bm{X}^T\bm{W}\bm{X}. \] !et @@ -391,13 +391,13 @@ $p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a com Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors !bt \[ -\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. +\log{ \frac{p(\bm{\beta}\bm{x})}{1-p(\bm{\beta}\bm{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. \] !et -Here we defined $\hat{x}=[1,x_1,x_2,\dots,x_p]$ and $\hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to +Here we defined $\bm{x}=[1,x_1,x_2,\dots,x_p]$ and $\bm{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to !bt \[ -p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. +p(\bm{\beta}\bm{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. \] !et @@ -439,7 +439,7 @@ Bayes classifiers, and artificial neural networks. Specifically, in multinomial logistic regression and linear discriminant analysis, the input to the function is the result of $K$ distinct linear functions, and the predicted probability for the $k$-th class given a sample -vector $\hat{x}$ and a weighting vector $\hat{\beta}$ is (with two +vector $\bm{x}$ and a weighting vector $\bm{\beta}$ is (with two predictors): !bt @@ -794,10 +794,10 @@ which we Taylor expand to obtain \end{array}. \] !et -Defining the Jacobian matrix ${\bf \bm{J}}$ we have +Defining the Jacobian matrix $\bm{J}$ we have !bt \[ - {\bf \bm{J}}=\left( \begin{array}{cc} + \bm{J}=\left( \begin{array}{cc} \partial f_1/\partial x_1 & \partial f_1/\partial x_2 \\ \partial f_2/\partial x_1 &\partial f_2/\partial x_2 \end{array} \right), @@ -815,13 +815,13 @@ where we have defined !bt \[ \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)= - -{\bf \bm{J}}^{-1} + -\bm{J}^{-1} \left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right). \] !et We need thus to compute the inverse of the Jacobian matrix and it is to understand that difficulties may -arise in case ${\bf \bm{J}}$ is nearly singular. +arise in case $\bm{J}$ is nearly singular. It is rather straightforward to extend the above scheme to systems of more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function. diff --git a/doc/LectureNotes/_build/.doctrees/chapter1.doctree b/doc/LectureNotes/_build/.doctrees/chapter1.doctree index ddcd320c9..754a7576c 100644 Binary files a/doc/LectureNotes/_build/.doctrees/chapter1.doctree and b/doc/LectureNotes/_build/.doctrees/chapter1.doctree differ diff --git a/doc/LectureNotes/_build/.doctrees/chapter2.doctree b/doc/LectureNotes/_build/.doctrees/chapter2.doctree index cdf8c34d4..545f752d2 100644 Binary files a/doc/LectureNotes/_build/.doctrees/chapter2.doctree and b/doc/LectureNotes/_build/.doctrees/chapter2.doctree differ diff --git a/doc/LectureNotes/_build/.doctrees/chapter4.doctree b/doc/LectureNotes/_build/.doctrees/chapter4.doctree index 701796f1f..fc69eb650 100644 Binary files a/doc/LectureNotes/_build/.doctrees/chapter4.doctree and b/doc/LectureNotes/_build/.doctrees/chapter4.doctree differ diff --git a/doc/LectureNotes/_build/.doctrees/environment.pickle b/doc/LectureNotes/_build/.doctrees/environment.pickle index f0b519cf7..313d56012 100644 Binary files a/doc/LectureNotes/_build/.doctrees/environment.pickle and b/doc/LectureNotes/_build/.doctrees/environment.pickle differ diff --git a/doc/LectureNotes/_build/html/_images/chapter1_11_0.png b/doc/LectureNotes/_build/html/_images/chapter1_11_0.png index 4385faaab..f67778338 100644 Binary files a/doc/LectureNotes/_build/html/_images/chapter1_11_0.png and b/doc/LectureNotes/_build/html/_images/chapter1_11_0.png differ diff --git a/doc/LectureNotes/_build/html/_images/chapter1_13_1.png b/doc/LectureNotes/_build/html/_images/chapter1_13_1.png index 0e285bdbd..a2e04aaa5 100644 Binary files a/doc/LectureNotes/_build/html/_images/chapter1_13_1.png and b/doc/LectureNotes/_build/html/_images/chapter1_13_1.png differ diff --git a/doc/LectureNotes/_build/html/_images/chapter1_27_0.png b/doc/LectureNotes/_build/html/_images/chapter1_27_0.png index 59f455ed6..c1228b486 100644 Binary files a/doc/LectureNotes/_build/html/_images/chapter1_27_0.png and b/doc/LectureNotes/_build/html/_images/chapter1_27_0.png differ diff --git a/doc/LectureNotes/_build/html/_images/chapter1_3_0.png b/doc/LectureNotes/_build/html/_images/chapter1_3_0.png index 8f3073f66..c7183345f 100644 Binary files a/doc/LectureNotes/_build/html/_images/chapter1_3_0.png and b/doc/LectureNotes/_build/html/_images/chapter1_3_0.png differ diff --git a/doc/LectureNotes/_build/html/_images/chapter1_61_10.png b/doc/LectureNotes/_build/html/_images/chapter1_61_10.png new file mode 100644 index 000000000..97a6c4434 Binary files /dev/null and b/doc/LectureNotes/_build/html/_images/chapter1_61_10.png differ diff --git a/doc/LectureNotes/_build/html/_images/chapter2_25_2.png b/doc/LectureNotes/_build/html/_images/chapter2_25_2.png index 40725aa11..3c8dab8ed 100644 Binary files a/doc/LectureNotes/_build/html/_images/chapter2_25_2.png and b/doc/LectureNotes/_build/html/_images/chapter2_25_2.png differ diff --git a/doc/LectureNotes/_build/html/_sources/chapter1.ipynb b/doc/LectureNotes/_build/html/_sources/chapter1.ipynb index 039da6f28..6209e3881 100644 --- a/doc/LectureNotes/_build/html/_sources/chapter1.ipynb +++ b/doc/LectureNotes/_build/html/_sources/chapter1.ipynb @@ -239,7 +239,7 @@ "\n", "What follows is a simple Python code where we have defined a function\n", "$y$ in terms of the variable $x$. Both are defined as vectors with $100$ entries. \n", - "The numbers in the vector $\\hat{x}$ are given\n", + "The numbers in the vector $\\boldsymbol{x}$ are given\n", "by random numbers generated with a uniform distribution with entries\n", "$x_i \\in [0,1]$ (more about probability distribution functions\n", "later). These values are then used to define a function $y(x)$\n", @@ -275,7 +275,7 @@ "distribution. From **Scikit-Learn** we import then the\n", "**LinearRegression** functionality and make a prediction $\\tilde{y} =\n", "\\alpha + \\beta x$ using the function **fit(x,y)**. We call the set of\n", - "data $(\\hat{x},\\hat{y})$ for our training data. The Python package\n", + "data $(\\boldsymbol{x},\\boldsymbol{y})$ for our training data. The Python package\n", "**scikit-learn** has also a functionality which extracts the above\n", "fitting parameters $\\alpha$ and $\\beta$ (see below). Later we will\n", "distinguish between training data and test data.\n", @@ -407,7 +407,7 @@ "metadata": {}, "source": [ "$$\n", - "\\epsilon_{\\mathrm{relative}}= \\frac{\\vert \\hat{y} -\\hat{\\tilde{y}}\\vert}{\\vert \\hat{y}\\vert}.\n", + "\\epsilon_{\\mathrm{relative}}= \\frac{\\vert \\boldsymbol{y} -\\boldsymbol{\\tilde{y}}\\vert}{\\vert \\boldsymbol{y}\\vert}.\n", "$$" ] }, @@ -521,7 +521,7 @@ "metadata": {}, "source": [ "$$\n", - "MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n", + "MSE(\\boldsymbol{y},\\boldsymbol{\\tilde{y}}) = \\frac{1}{n}\n", "\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n", "$$" ] @@ -538,10 +538,10 @@ "determination. It provides a measure of how well future samples are\n", "likely to be predicted by the model. Best possible score is 1.0 and it\n", "can be negative (because the model can be arbitrarily worse). A\n", - "constant model that always predicts the expected value of $\\hat{y}$,\n", + "constant model that always predicts the expected value of $\\boldsymbol{y}$,\n", "disregarding the input features, would get a $R^2$ score of $0.0$.\n", "\n", - "If $\\tilde{\\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as" + "If $\\tilde{\\boldsymbol{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as" ] }, { @@ -549,7 +549,7 @@ "metadata": {}, "source": [ "$$\n", - "R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n", + "R^2(\\boldsymbol{y}, \\tilde{\\boldsymbol{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n", "$$" ] }, @@ -557,7 +557,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where we have defined the mean value of $\\hat{y}$ as" + "where we have defined the mean value of $\\boldsymbol{y}$ as" ] }, { @@ -583,7 +583,7 @@ "metadata": {}, "source": [ "$$\n", - "\\text{MAE}(\\hat{y}, \\hat{\\tilde{y}}) = \\frac{1}{n} \\sum_{i=0}^{n-1} \\left| y_i - \\tilde{y}_i \\right|.\n", + "\\text{MAE}(\\boldsymbol{y}, \\boldsymbol{\\tilde{y}}) = \\frac{1}{n} \\sum_{i=0}^{n-1} \\left| y_i - \\tilde{y}_i \\right|.\n", "$$" ] }, @@ -600,7 +600,7 @@ "metadata": {}, "source": [ "$$\n", - "\\text{MSLE}(\\hat{y}, \\hat{\\tilde{y}}) = \\frac{1}{n} \\sum_{i=0}^{n - 1} (\\log_e (1 + y_i) - \\log_e (1 + \\tilde{y}_i) )^2,\n", + "\\text{MSLE}(\\boldsymbol{y}, \\boldsymbol{\\tilde{y}}) = \\frac{1}{n} \\sum_{i=0}^{n - 1} (\\log_e (1 + y_i) - \\log_e (1 + \\tilde{y}_i) )^2,\n", "$$" ] }, @@ -3314,7 +3314,7 @@ "metadata": {}, "source": [ "$$\n", - "MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n", + "MSE(\\boldsymbol{y},\\boldsymbol{\\tilde{y}}) = \\frac{1}{n}\n", "\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n", "$$" ] @@ -3324,7 +3324,7 @@ "metadata": {}, "source": [ "and the $R^2$ score function.\n", - "If $\\tilde{\\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as" + "If $\\tilde{\\boldsymbol{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as" ] }, { @@ -3332,7 +3332,7 @@ "metadata": {}, "source": [ "$$\n", - "R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n", + "R^2(\\boldsymbol{y}, \\tilde{\\boldsymbol{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n", "$$" ] }, @@ -3340,7 +3340,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where we have defined the mean value of $\\hat{y}$ as" + "where we have defined the mean value of $\\boldsymbol{y}$ as" ] }, { diff --git a/doc/LectureNotes/_build/html/_sources/chapter2.ipynb b/doc/LectureNotes/_build/html/_sources/chapter2.ipynb index 721b43cde..d7f8222d5 100644 --- a/doc/LectureNotes/_build/html/_sources/chapter2.ipynb +++ b/doc/LectureNotes/_build/html/_sources/chapter2.ipynb @@ -6,9 +6,6 @@ "source": [ "# Resampling Methods\n", "\n", - "[Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSept3.mp4?vrtx=view-as-webpage)\n", - "\n", - "\n", "## Introduction\n", "\n", "Resampling methods are an indispensable tool in modern\n", diff --git a/doc/LectureNotes/_build/html/_sources/chapter4.ipynb b/doc/LectureNotes/_build/html/_sources/chapter4.ipynb index 80fb450d4..7cd60a191 100644 --- a/doc/LectureNotes/_build/html/_sources/chapter4.ipynb +++ b/doc/LectureNotes/_build/html/_sources/chapter4.ipynb @@ -16,11 +16,11 @@ "coefficients of a functional fit (say a polynomial) in order to be\n", "able to predict the response of a continuous variable on some unseen\n", "data. The fit to the continuous variable $y_i$ is based on some\n", - "independent variables $\\hat{x}_i$. Linear regression resulted in\n", + "independent variables $x_i$. Linear regression resulted in\n", "analytical expressions for standard ordinary Least Squares or Ridge\n", "regression (in terms of matrices to invert) for several quantities,\n", "ranging from the variance and thereby the confidence intervals of the\n", - "parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n", + "optimal parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n", "the product of the design matrices, linear regression gives then a\n", "simple recipe for fitting our data.\n", "\n", @@ -65,7 +65,7 @@ "from $k=0,\\dots,K-1$ (i.e. $K$ classes).\n", "\n", "The goal is to predict the\n", - "output classes from the design matrix $\\hat{X}\\in\\mathbb{R}^{n\\times p}$\n", + "output classes from the design matrix $\\boldsymbol{X}\\in\\mathbb{R}^{n\\times p}$\n", "made of $n$ samples, each of which carries $p$ features or predictors. The\n", "primary goal is to identify the classes to which new unseen samples\n", "belong.\n", @@ -107,7 +107,7 @@ "\n", "$$\n", "\\begin{equation}\n", - "\\hat{y} = \\hat{X}^T\\hat{\\beta} + \\hat{\\epsilon},\n", + "\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon},\n", "\\label{_auto1} \\tag{1}\n", "\\end{equation}\n", "$$" @@ -117,8 +117,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where $\\hat{y}$ is a vector representing the possible outcomes, $\\hat{X}$ is our\n", - "$n\\times p$ design matrix and $\\hat{\\beta}$ represents our estimators/predictors.\n", + "where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n", + "$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors.\n", "\n", "\n", "The main problem with our function is that it takes values on the\n", @@ -380,8 +380,8 @@ "source": [ "$$\n", "\\begin{align*}\n", - "p(y_i=1|x_i,\\hat{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n", - "p(y_i=0|x_i,\\hat{\\beta}) &= 1 - p(y_i=1|x_i,\\hat{\\beta}),\n", + "p(y_i=1|x_i,\\boldsymbol{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n", + "p(y_i=0|x_i,\\boldsymbol{\\beta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\beta}),\n", "\\end{align*}\n", "$$" ] @@ -390,7 +390,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where $\\hat{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n", + "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n", "\n", "Note that we used" ] @@ -400,7 +400,7 @@ "metadata": {}, "source": [ "$$\n", - "p(y_i=0\\vert x_i, \\hat{\\beta}) = 1-p(y_i=1\\vert x_i, \\hat{\\beta}).\n", + "p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n", "$$" ] }, @@ -422,7 +422,7 @@ "source": [ "$$\n", "\\begin{align*}\n", - "P(\\mathcal{D}|\\hat{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\hat{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n", + "P(\\mathcal{D}|\\boldsymbol{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\boldsymbol{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n", "\\end{align*}\n", "$$" ] @@ -439,7 +439,7 @@ "metadata": {}, "source": [ "$$\n", - "\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\hat{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]\\right).\n", + "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n", "$$" ] }, @@ -455,7 +455,7 @@ "metadata": {}, "source": [ "$$\n", - "\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", + "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", "$$" ] }, @@ -472,7 +472,7 @@ "metadata": {}, "source": [ "$$\n", - "\\mathcal{C}(\\hat{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", + "\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", "$$" ] }, @@ -484,7 +484,7 @@ "in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.\n", "\n", "\n", - "The cross entropy is a convex function of the weights $\\hat{\\beta}$ and,\n", + "The cross entropy is a convex function of the weights $\\boldsymbol{\\beta}$ and,\n", "therefore, any local minimizer is a global minimizer. \n", "\n", "\n", @@ -497,7 +497,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n", "$$" ] }, @@ -513,7 +513,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n", "$$" ] }, @@ -521,9 +521,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Let us now define a vector $\\hat{y}$ with $n$ elements $y_i$, an\n", - "$n\\times p$ matrix $\\hat{X}$ which contains the $x_i$ values and a\n", - "vector $\\hat{p}$ of fitted probabilities $p(y_i\\vert x_i,\\hat{\\beta})$. We can rewrite in a more compact form the first\n", + "Let us now define a vector $\\boldsymbol{y}$ with $n$ elements $y_i$, an\n", + "$n\\times p$ matrix $\\boldsymbol{X}$ which contains the $x_i$ values and a\n", + "vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\beta})$. We can rewrite in a more compact form the first\n", "derivative of cost function as" ] }, @@ -532,7 +532,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}} = -\\hat{X}^T\\left(\\hat{y}-\\hat{p}\\right).\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", "$$" ] }, @@ -540,8 +540,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If we in addition define a diagonal matrix $\\hat{W}$ with elements \n", - "$p(y_i\\vert x_i,\\hat{\\beta})(1-p(y_i\\vert x_i,\\hat{\\beta})$, we can obtain a compact expression of the second derivative as" + "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", + "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" ] }, { @@ -549,7 +549,7 @@ "metadata": {}, "source": [ "$$\n", - "\\frac{\\partial^2 \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}\\partial \\hat{\\beta}^T} = \\hat{X}^T\\hat{W}\\hat{X}.\n", + "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", "$$" ] }, @@ -565,7 +565,7 @@ "metadata": {}, "source": [ "$$\n", - "\\log{ \\frac{p(\\hat{\\beta}\\hat{x})}{1-p(\\hat{\\beta}\\hat{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n", + "\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n", "$$" ] }, @@ -573,7 +573,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Here we defined $\\hat{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\hat{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to" + "Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to" ] }, { @@ -581,7 +581,7 @@ "metadata": {}, "source": [ "$$\n", - "p(\\hat{\\beta}\\hat{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n", + "p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n", "$$" ] }, @@ -654,7 +654,7 @@ "multinomial logistic regression and linear discriminant analysis, the\n", "input to the function is the result of $K$ distinct linear functions,\n", "and the predicted probability for the $k$-th class given a sample\n", - "vector $\\hat{x}$ and a weighting vector $\\hat{\\beta}$ is (with two\n", + "vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\beta}$ is (with two\n", "predictors):" ] }, @@ -1183,7 +1183,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" + "Defining the Jacobian matrix $\\boldsymbol{J}$ we have" ] }, { @@ -1191,7 +1191,7 @@ "metadata": {}, "source": [ "$$\n", - "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", + "\\boldsymbol{J}=\\left( \\begin{array}{cc}\n", " \\partial f_1/\\partial x_1 & \\partial f_1/\\partial x_2 \\\\\n", " \\partial f_2/\\partial x_1 &\\partial f_2/\\partial x_2\n", " \\end{array} \\right),\n", @@ -1229,7 +1229,7 @@ "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", - " -{\\bf \\boldsymbol{J}}^{-1}\n", + " -\\boldsymbol{J}^{-1}\n", " \\left(\\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\\\ f_2(x_1^{n},x_2^{n}) \\end{array} \\right).\n", "$$" ] @@ -1240,7 +1240,7 @@ "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", - "arise in case ${\\bf \\boldsymbol{J}}$ is nearly singular.\n", + "arise in case $\\boldsymbol{J}$ is nearly singular.\n", "\n", "It is rather straightforward to extend the above scheme to systems of\n", "more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function. \n", diff --git a/doc/LectureNotes/_build/html/chapter1.html b/doc/LectureNotes/_build/html/chapter1.html index e993c11f4..6aea9d947 100644 --- a/doc/LectureNotes/_build/html/chapter1.html +++ b/doc/LectureNotes/_build/html/chapter1.html @@ -588,7 +588,7 @@ may first try the simplest class of models, namely linear models, followed obvio

We start with perhaps our simplest possible example, using Scikit-Learn to perform linear regression analysis on a data set produced by us.

What follows is a simple Python code where we have defined a function \(y\) in terms of the variable \(x\). Both are defined as vectors with \(100\) entries. -The numbers in the vector \(\hat{x}\) are given +The numbers in the vector \(\boldsymbol{x}\) are given by random numbers generated with a uniform distribution with entries \(x_i \in [0,1]\) (more about probability distribution functions later). These values are then used to define a function \(y(x)\) @@ -611,7 +611,7 @@ y = 2x+N(0,1), distribution. From Scikit-Learn we import then the LinearRegression functionality and make a prediction \(\tilde{y} = \alpha + \beta x\) using the function fit(x,y). We call the set of -data \((\hat{x},\hat{y})\) for our training data. The Python package +data \((\boldsymbol{x},\boldsymbol{y})\) for our training data. The Python package scikit-learn has also a functionality which extracts the above fitting parameters \(\alpha\) and \(\beta\) (see below). Later we will distinguish between training data and test data.

@@ -703,7 +703,7 @@ the \(\chi^2\) function becom the relative error (why would we prefer the MSE instead of the relative error?) as

\[ -\epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}. +\epsilon_{\mathrm{relative}}= \frac{\vert \boldsymbol{y} -\boldsymbol{\tilde{y}}\vert}{\vert \boldsymbol{y}\vert}. \]

The squared cost function results in an arithmetic mean-unbiased estimator, and the absolute-value cost function results in a @@ -781,13 +781,13 @@ example of the functionality of Scikit-Learn.

The intercept alpha: 
- [1.98452685]
+ [2.15024669]
 Coefficient beta : 
- [[5.00109273]]
-Mean squared error: 0.19
-Variance score: 0.92
+ [[4.89975818]]
+Mean squared error: 0.26
+Variance score: 0.90
 Mean squared log error: 0.01
-Mean absolute error: 0.35
+Mean absolute error: 0.42
_images/chapter1_13_1.png @@ -797,7 +797,7 @@ Mean absolute error: 0.35 \(\alpha\). Depending on the constant in front of the normal distribution, we get values near or far from \(alpha =2\) and \(\beta =5\). Try to play around with different parameters in front of the normal distribution. The function meansquarederror gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as

\[ -MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +MSE(\boldsymbol{y},\boldsymbol{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, \]

The smaller the value, the better the fit. Ideally we would like to @@ -807,14 +807,14 @@ this function as being similar to the \(\hat{y}\), +constant model that always predicts the expected value of \(\boldsymbol{y}\), disregarding the input features, would get a \(R^2\) score of \(0.0\).

-

If \(\tilde{\hat{y}}_i\) is the predicted value of the \(i-th\) sample and \(y_i\) is the corresponding true value, then the score \(R^2\) is defined as

+

If \(\tilde{\boldsymbol{y}}_i\) is the predicted value of the \(i-th\) sample and \(y_i\) is the corresponding true value, then the score \(R^2\) is defined as

\[ -R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, \]
-

where we have defined the mean value of \(\hat{y}\) as

+

where we have defined the mean value of \(\boldsymbol{y}\) as

\[ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. @@ -824,13 +824,13 @@ the mean absolute error (MAE), a risk metric corresponding to the expected value The MAE is defined as follows

\[ -\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|. +\text{MAE}(\boldsymbol{y}, \boldsymbol{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|. \]

We present the squared logarithmic (quadratic) error

\[ -\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2, +\text{MSLE}(\boldsymbol{y}, \boldsymbol{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2, \]

where \(\log_e (x)\) stands for the natural logarithm of \(x\). This error estimate is best to use when targets having exponential growth, such @@ -888,7 +888,7 @@ a linear \(x\)-dependence we s

_images/chapter1_27_0.png -
0.004999999999999996
+
0.0050000000000000044
@@ -1225,7 +1225,7 @@ A 270 3344 160 110 270 Ds 7.253775 7.253775 [267 rows x 6 columns] -0.009883615646716184 +0.009883615646716182
@@ -1283,8 +1283,6 @@ functionality.

warnings.warn( /Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet. warnings.warn( -/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet. - warnings.warn( 
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
@@ -1307,6 +1305,14 @@ functionality.

   warnings.warn(
+
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
+  warnings.warn(
+/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
+  warnings.warn(
+/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
+  warnings.warn(
+
+
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
   warnings.warn(
 /Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
@@ -1331,14 +1337,14 @@ functionality.

   warnings.warn(
 /Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
   warnings.warn(
-/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
-  warnings.warn(
 
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
   warnings.warn(
 /Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
   warnings.warn(
+/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
+  warnings.warn(
 
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
@@ -1349,7 +1355,7 @@ functionality.

   warnings.warn(
-_images/chapter1_61_9.png +_images/chapter1_61_10.png @@ -2324,13 +2330,13 @@ but now splitting the data into a training set and a test set.

Training R2
-0.9999864543345858
+0.9999868619217517
 Training MSE
-6.180092462880674
+5.965885569080809
 Test R2
-0.9999822527140678
+0.9999794306626945
 Test MSE
-7.205466494327873
+8.300162456113691
@@ -2741,31 +2747,31 @@ techniques.

MSE before scaling: 0.00
-R2 score before scaling 0.99
+R2 score before scaling 1.00
 Feature min values before scaling:
- [1.00000000e+00 1.10094646e-03 9.51523276e-04 1.21208310e-06
- 1.04757618e-06 9.05396545e-07 1.33443859e-09 1.15332528e-09
- 9.96793117e-10 8.61505887e-10 1.46914544e-12 1.26974938e-12
- 1.09741585e-12 9.48471852e-13 8.19742904e-13 1.61745046e-15
- 1.39792608e-15 1.20819609e-15 1.04421672e-15 9.02493044e-16
- 7.80004454e-16]
+ [1.00000000e+00 2.54152940e-03 1.38207279e-03 6.45937170e-06
+ 3.51257863e-06 1.91012519e-06 1.64166831e-08 8.92732185e-09
+ 4.85463934e-09 2.63993205e-09 4.17234827e-11 2.26890510e-11
+ 1.23382086e-11 6.70946493e-12 3.64857826e-12 1.06041458e-13
+ 5.76648901e-14 3.13579199e-14 1.70523024e-14 9.27296891e-15
+ 5.04260073e-15]
 Feature max values before scaling:
- [1.         0.99825997 0.99883879 0.99652296 0.99710078 0.99767893
- 0.99478898 0.9953658  0.99594294 0.99652042 0.99305802 0.99363383
- 0.99420997 0.99478645 0.99536326 0.99133007 0.99190487 0.99248001
- 0.99305549 0.99363129 0.99420743]
+ [1.         0.99817842 0.99945628 0.99636015 0.99763569 0.99891285
+ 0.9945452  0.99581841 0.99709325 0.99836973 0.99273355 0.99400444
+ 0.99527696 0.99655111 0.99782689 0.9909252  0.99219378 0.99346398
+ 0.99473581 0.99600927 0.99728435]
 Feature min values after scaling:
- [ 0.         -1.61869821 -1.66880047 -1.06209126 -1.08170444 -1.10131725
- -0.84087101 -0.85396354 -0.86692943 -0.87972591 -0.71351486 -0.7240496
- -0.73453972 -0.74495014 -0.75524378 -0.62783293 -0.63680118 -0.64580686
- -0.65482578 -0.66383151 -0.67279536]
+ [ 0.         -1.63437572 -1.76504618 -1.03871832 -1.08415761 -1.13457922
+ -0.81620806 -0.83935285 -0.86436607 -0.8914984  -0.69347005 -0.70790937
+ -0.72312577 -0.73921714 -0.75629493 -0.61234223 -0.6226921  -0.63339159
+ -0.64447921 -0.65599927 -0.66800261]
 Feature max values after scaling:
- [0.         1.78944806 1.68342382 2.34172919 2.25457052 2.16553696
- 2.7899453  2.71281409 2.63374631 2.55280484 3.17117385 3.10307631
- 3.03303359 2.96104648 2.88712946 3.50394742 3.44395541 3.38216436
- 3.31853484 3.25303483 3.1856411 ]
+ [0.         1.86574276 1.72218808 2.41770932 2.31730641 2.21347282
+ 2.88297395 2.79328828 2.70234019 2.60999846 3.291614   3.20772452
+ 3.12283463 3.03697069 2.95014575 3.65766387 3.57871326 3.49865673
+ 3.41754964 3.33544681 3.25240108]
 MSE after  scaling: 0.00
-R2 score for  scaled data: 0.99
+R2 score for  scaled data: 1.00
@@ -2849,16 +2855,16 @@ The following simple Python instructions define our \[ -MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +MSE(\boldsymbol{y},\boldsymbol{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, \]

and the \(R^2\) score function. -If \(\tilde{\hat{y}}_i\) is the predicted value of the \(i-th\) sample and \(y_i\) is the corresponding true value, then the score \(R^2\) is defined as

+If \(\tilde{\boldsymbol{y}}_i\) is the predicted value of the \(i-th\) sample and \(y_i\) is the corresponding true value, then the score \(R^2\) is defined as

\[ -R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, \]
-

where we have defined the mean value of \(\hat{y}\) as

+

where we have defined the mean value of \(\boldsymbol{y}\) as

\[ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. diff --git a/doc/LectureNotes/_build/html/chapter2.html b/doc/LectureNotes/_build/html/chapter2.html index 5f81da517..21ac40195 100644 --- a/doc/LectureNotes/_build/html/chapter2.html +++ b/doc/LectureNotes/_build/html/chapter2.html @@ -55,7 +55,7 @@ - + @@ -139,7 +139,7 @@