diff --git a/doc/BookChapters/chapter1.dlog b/doc/BookChapters/chapter1.dlog
index 0e51fe222..49f0f53cc 100644
--- a/doc/BookChapters/chapter1.dlog
+++ b/doc/BookChapters/chapter1.dlog
@@ -67,3 +67,13 @@ found info about 5 exercises
 
 *** warning: latex envir \begin{bmatrix} does not work well in Markdown.    Stick to \[ ... \], equation, equation*, align, or align*    environments in math environments.
 output in chapter1.ipynb
+Translating doconce text in chapter1.do.txt to ipynb
+*** replacing \bm{...} by \boldsymbol{...} (\bm is not supported by MathJax)
+found info about 5 exercises
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.    Stick to \[ ... \], equation, equation*, align, or align*    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.    Stick to \[ ... \], equation, equation*, align, or align*    environments in math environments.
+
+*** warning: latex envir \begin{bmatrix} does not work well in Markdown.    Stick to \[ ... \], equation, equation*, align, or align*    environments in math environments.
+output in chapter1.ipynb
diff --git a/doc/BookChapters/chapter1.do.txt b/doc/BookChapters/chapter1.do.txt
index 7995dcd7e..a35734a39 100644
--- a/doc/BookChapters/chapter1.do.txt
+++ b/doc/BookChapters/chapter1.do.txt
@@ -88,7 +88,7 @@ role when we develop a specific machine learning algorithm.
 Machine learning is an extremely rich field, in spite of its young
 age. The increases we have seen during the last three decades in
 computational capabilities have been followed by developments of
-methods and techniques for analyzing and handling large date sets,
+methods and techniques for analyzing and handling large data sets,
 relying heavily on statistics, computer science and mathematics.  The
 field is rather new and developing rapidly. Popular software packages
 written in Python for machine learning like
@@ -113,7 +113,7 @@ two main categories.  In *supervised learning* we know the answer to a
 problem, and let the computer deduce the logic behind it. On the other
 hand, *unsupervised learning* is a method for finding patterns and
 relationship in data sets without any prior knowledge of the system.
-Some authours also operate with a third category, namely
+Some authors also operate with a third category, namely
 *reinforcement learning*. This is a paradigm of learning inspired by
 behavioral psychology, where learning is achieved by trial-and-error,
 solely from rewards and punishment.
@@ -176,14 +176,14 @@ what is the likelihood of finding $B$.
 In science and engineering we often end up in situations where we want to infer (or learn) a
 quantitative model $M$ for a given set of sample points $\bm{X} \in [x_1, x_2,\dots x_N]$.
 
-As we will see repeatedely in these lectures, we could try to fit these data points to a model given by a
+As we will see repeatedly in these lectures, we could try to fit these data points to a model given by a
 straight line, or if we wish to be more sophisticated to a more complex
 function.
 
 The reason for inferring such a model is that it
 serves many useful purposes. On the one hand, the model can reveal information
 encoded in the data or underlying mechanisms from which the data were generated. For instance, we could discover important
-corelations that relate interesting physics interpretations.
+correlations that relate interesting physics interpretations.
 
 In addition, it can simplify the representation of the given data set and help
 us in making predictions about  future data samples.
@@ -328,7 +328,7 @@ y = 10x+0.01 \times N(0,1),
 where $x$ is defined as before.  Does the fit look better? Indeed, by
 reducing the role of the noise given by the normal distribution we see immediately that
 our linear prediction seemingly reproduces better the training
-set. However, this testing 'by the eye' is obviouly not satisfactory in the
+set. However, this testing 'by the eye' is obviously not satisfactory in the
 long run. Here we have only defined the training data and our model, and 
 have not discussed a more rigorous approach to the _cost_ function.
 
diff --git a/doc/BookChapters/chapter2.do.txt b/doc/BookChapters/chapter2.do.txt
index f7bb2b79c..45c884f59 100644
--- a/doc/BookChapters/chapter2.do.txt
+++ b/doc/BookChapters/chapter2.do.txt
@@ -104,7 +104,7 @@ later shrinkage methods like Ridge and Lasso regressions.
 This is given by the _Singular Value Decomposition_ (SVD) algorithm,
 perhaps the most powerful linear algebra algorithm.  The SVD provides
 a numerically stable matrix decomposition that is used in a large
-swath oc applications and the decomposition is always stable
+swath of applications and the decomposition is always stable
 numerically.
 
 In machine learning it plays a central role in dealing with for
@@ -123,7 +123,7 @@ when the matrix $\bm{X}$ (our so-called design matrix) is high-dimensional,
 are problems with near singular or singular matrices. The column vectors of $\bm{X}$ 
 may be linearly dependent, normally referred to as super-collinearity.  
 This means that the matrix may be rank deficient and it is basically impossible to 
-to model the data using linear regression. As an example, consider the matrix
+model the data using linear regression. As an example, consider the matrix
 !bt
 \begin{align*}
 \mathbf{X} & =  \left[
@@ -143,7 +143,7 @@ The columns of $\bm{X}$ are linearly dependent. We see this easily since the
 the first column is the row-wise sum of the other two columns. The rank (more correct,
 the column rank) of a matrix is the dimension of the space spanned by the
 column vectors. Hence, the rank of $\mathbf{X}$ is equal to the number
-of linearly independent columns. In this particular case the matrix has rank 2.
+of linearly independent columns. In this particular case the matrix has rank 1.
 
 Super-collinearity of an $(n \times p)$-dimensional design matrix $\mathbf{X}$ implies
 that the inverse of the matrix $\bm{X}^T\bm{X}$ (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this
@@ -170,7 +170,7 @@ If our design matrix $\bm{X}$ which enters the linear regression problem
 !et
 has linearly dependent column vectors, we will not be able to compute the inverse
 of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\beta_i$. 
-The estimators are only well-defined if $(\bm{X}^{T}\bm{X})^{-1}$ exits. 
+The estimators are only well-defined if $(\bm{X}^{T}\bm{X})$ can be inverted. 
 This is more likely to happen when the matrix $\bm{X}$ is high-dimensional. In this case it is likely to encounter a situation where 
 the regression parameters $\beta_i$ cannot be estimated.
 
@@ -188,7 +188,7 @@ where $\bm{I}$ is the identity matrix.  When we discuss _Ridge_ regression this
 ===== Basic math of the SVD =====
 
 
-From standard linear algebra we know that a square matrix $\bm{X}$ can be diagonalized if and only it is 
+From standard linear algebra we know that a square matrix $\bm{X}$ can be diagonalized if and only if it is 
 a so-called "normal matrix":"https://en.wikipedia.org/wiki/Normal_matrix", that is if $\bm{X}\in {\mathbb{R}}^{n\times n}$
 we have $\bm{X}\bm{X}^T=\bm{X}^T\bm{X}$ or if $\bm{X}\in {\mathbb{C}}^{n\times n}$ we have $\bm{X}\bm{X}^{\dagger}=\bm{X}^{\dagger}\bm{X}$.
 The matrix has then a set of eigenpairs 
@@ -381,7 +381,6 @@ def SVDinv(A):
     return np.matmul(V,np.matmul(invD,UT))
 
 
-#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])
 # Non-singular square matrix
 X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])
 print(X)
@@ -402,7 +401,7 @@ rectangular matrices where the number of rows and columns are not equal.
 It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse.
 It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices.
 
-Using the SVD we can obtain the pseudoinverse of a matrix $\bm{A}$ (labeled here as $\bm{A}_{\mathrm{PI}}$
+Using the SVD we can obtain the pseudoinverse (PI) of a matrix $\bm{A}$ (labeled here as $\bm{A}_{\mathrm{PI}}$
 !bt
 \[
 \bm{A}_{\mathrm{PI}}= \bm{V}\bm{D}_{\mathrm{PI}}\bm{U}^T,
@@ -469,7 +468,7 @@ We can SVD decompose our matrix as
 !et
 where $\bm{U}$ is an orthogonal matrix of dimension $n\times n$, meaning that $\bm{U}\bm{U}^T=\bm{U}^T\bm{U}=\bm{I}_n$. Here $\bm{I}_n$ is the unit matrix of dimension $n \times n$.
 
-Similarly, $\bm{V}$ is an orthogonal matrix of dimension $p\times p$, meaning that $\bm{V}\bm{V}^T=\bm{V}^T\bm{V}=\bm{I}_p$. Here $\bm{I}_n$ is the unit matrix of dimension $p \times p$.
+Similarly, $\bm{V}$ is an orthogonal matrix of dimension $p\times p$, meaning that $\bm{V}\bm{V}^T=\bm{V}^T\bm{V}=\bm{I}_p$. Here $\bm{I}_p$ is the unit matrix of dimension $p \times p$.
 
 Finally $\bm{\Sigma}$ contains the singular values $\sigma_i$. This matrix has dimension $n\times p$ and the singular values $\sigma_i$ are all positive. The non-zero values are ordered in descending order, that is
 
@@ -656,7 +655,7 @@ function, that is we have
 \frac{\partial^2 C(\bm{\beta})}{\partial \bm{\beta}^T\partial \bm{\beta}} =\frac{2}{n}\bm{X}^T\bm{X}.  
 \]
 !et
-This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
+This quantity defines  what is called the Hessian matrix (the second derivative of the cost function we want to optimize).
 
 The Hessian matrix plays an important role and is defined in this course as
 
@@ -777,7 +776,7 @@ with a given vector
 !et
 
 With these definitions, we can now rewrite our $2\times 2$
-correlation/covariance matrix in terms of a moe general design/feature
+correlation/covariance matrix in terms of a more general design/feature
 matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. This leads to a $p\times p$
 covariance matrix for the vectors $\bm{x}_i$ with $i=0,1,\dots,p-1$
 
@@ -883,7 +882,7 @@ The above procedure with _numpy_ can be made more compact if we use _pandas_.
 
 
 
-We whow here how we can set up the correlation matrix using _pandas_, as done in this simple code
+We know here how we can set up the correlation matrix using _pandas_, as done in this simple code
 !bc pycod
 import numpy as np
 import pandas as pd
@@ -1344,7 +1343,7 @@ This equation does not lead to a nice analytical equation as in Ridge regression
 Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the
 diagonal. In this case we have an equal number of rows and columns $n=p$.
 
-Our model approximation is just $\tilde{\bm{y}}=\bm{\beta}$ and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term $1/n$) 
+Our model approximation is just $\tilde{\bm{y}}=\bm{\beta}$ and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term $1/n$) 
 !bt
 \[
 C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2,
@@ -1396,7 +1395,7 @@ which leads to
 Plotting these results ("figure in handwritten notes for week 36":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf") shows clearly that Lasso regression suppresses (sets to zero) values of $\beta_i$ for specific values of $\lambda$. Ridge regression reduces on the other hand the values of $\beta_i$ as function of $\lambda$.
 
 
-As another examples, 
+As another example, 
 let us assume we have a data set with outputs/targets given by the vector
 
 !bt
@@ -1600,13 +1599,13 @@ plt.show()
 
 !ec
 
-We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\bm{\beta}$, we that they are getting smaller and smaller and our error stabilizes since the predicted values of $\tilde{\bm{y}}$ approach zero.
+We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\bm{\beta}$, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of $\tilde{\bm{y}}$ approach zero.
 
 This happens also for Lasso regression, as seen from the next code
 output. The difference is that Lasso shrinks the values of $\beta$ to
 zero at a much earlier stage and the results flatten out. We see that
 Lasso gives also an excellent fit for small values of $\lambda$ and
-shows rthe best performance of the three regression methods.
+shows the best performance of the three regression methods.
 
 !bc pycod
 import os
@@ -1967,7 +1966,7 @@ p(y_i, \bm{X}\vert\bm{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_
 !et
 which reads as finding the likelihood of an event $y_i$ with the input variables $\bm{X}$ given the parameters (to be determined) $\bm{\beta}$.
 
-Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event $\bm{y}$ as the product of the single events, that is we have
+Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event $\bm{y}$ as the product of the single events, that is we have
 
 !bt
 \[
@@ -2072,7 +2071,7 @@ p(X \cup Y)= p(X)+p(Y)-p(X \cap Y).
 The product rule (aka joint probability) is given by
 !bt
 \[
-p(X \cup Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X),
+p(X \cap Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X),
 \]
 !et
 where we read $p(X\vert Y)$ as the likelihood of obtaining $X$ given $Y$.
@@ -2352,7 +2351,7 @@ parameters $\beta_j$ as function of polynomial order and of the added
 noise. Here we recommend to use $\sigma^2=1$ as variance for the
 added noise (which follows a normal distribution with mean value zero).
 Comment your results. If you have a large noise term, do the parameters $\beta_j$ vary more as function
-model complexity? And what about their variance? 
+of model complexity? And what about their variance? 
 
 
 
@@ -2393,7 +2392,7 @@ p(\bm{\beta}\vert\bm{D})\propto p(\bm{D}\vert\bm{\beta})p(\bm{\beta}).
 \]
 !et
 
-We have a model for $p(\bm{D}\vert\bm{\beta})$ but need one for the _prior_ $p(\bm{\beta}$!   
+We have a model for $p(\bm{D}\vert\bm{\beta})$ but need one for the _prior_ $p(\bm{\beta})$!   
 
 
 
@@ -2464,14 +2463,14 @@ constants terms that do not depend on $\beta$, we have
 
 !bt
 \[
-C(\bm{\beta}=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\bm{\beta}\vert\vert_1,
+C(\bm{\beta})=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\bm{\beta}\vert\vert_1,
 \]
 !et
 and replacing $1/\tau$ with $\lambda$ we have
 
 !bt
 \[
-C(\bm{\beta}=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\bm{\beta}\vert\vert_1,
+C(\bm{\beta})=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\bm{\beta}\vert\vert_1,
 \]
 !et
 which is our Lasso cost function!  
diff --git a/doc/BookChapters/chapteroptimization.do.txt b/doc/BookChapters/chapteroptimization.do.txt
index d22d9ead4..692f5f28b 100644
--- a/doc/BookChapters/chapteroptimization.do.txt
+++ b/doc/BookChapters/chapteroptimization.do.txt
@@ -280,9 +280,19 @@ The convex subsets of $\mathbb{R}$ are the intervals of
 $\mathbb{R}$. Examples of convex sets of $\mathbb{R}^2$ are the
 regular polygons (triangles, rectangles, pentagons, etc...).
 
+_Convex function_: Let $X \subset \mathbb{R}^n$ be a convex
+set. Assume that the function $f: X \rightarrow \mathbb{R}$ is
+continuous, then $f$ is said to be convex if
+$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2)$
+for all
+$x_1, x_2 \in X$ and for all $t \in [0,1]$.
 
-
-_Convex function_: Let $X \subset \mathbb{R}^n$ be a convex set. Assume that the function $f: X \rightarrow \mathbb{R}$ is continuous, then $f$ is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all $x_1, x_2 \in X$ and for all $t \in [0,1]$. If $\leq$ is replaced with a strict inequaltiy in the definition, we demand $x_1 \neq x_2$ and $t\in(0,1)$ then $f$ is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the value of the function on the interval $[x_1,x_2]$ is always below the line as illustrated below.
+If $\leq$ is replaced with a strict inequality in the
+definition, we demand $x_1 \neq x_2$ and $t\in(0,1)$ then $f$ is said
+to be strictly convex. For a single variable function, convexity means
+that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the
+value of the function on the interval $[x_1,x_2]$ is always below the
+line as discussed below.
 
 
 In the following we state first and second-order conditions which
@@ -296,7 +306,7 @@ all $x$ in the domain of $f$). Then $f$ is convex if and only if $D_f$
 is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds
 for all $x,y \in D_f$. This condition means that for a convex function
 the first order Taylor expansion (right hand side above) at any point
-a global under estimator of the function. To convince yourself you can
+is a global under estimator of the function. To convince yourself you can
 make a drawing of $f(x) = x^2+1$ and draw the tangent line to $f(x)$ and
 note that it is always below the graph.  
 !eblock
@@ -1756,7 +1766,14 @@ a /=b
 
 
 
-=====  Using Autograd with OLS =====
+!split
+===== Replace or not =====
+
+In the above code, we have use replacement in setting up the
+mini-batches. The discussion
+"here":"https://sebastianraschka.com/faq/docs/sgd-methods.html" may be
+useful.  
+=====  Using Autograd  =====
 
 We conclude the part on optmization by showing how we can make codes
 for linear regression and logistic regression using _autograd_. The
@@ -1816,8 +1833,110 @@ plt.show()
 !ec
 
 
+===== Same code but now with momentum gradient descent =====
+!bc pycod
+# Using Autograd to calculate gradients for OLS
+from random import random, seed
+import numpy as np
+import autograd.numpy as np
+import matplotlib.pyplot as plt
+from autograd import grad
 
-=== Including Stochastic Gradient Descent with Autograd ===
+def CostOLS(beta):
+    return (1.0/n)*np.sum((y-X @ beta)**2)
+
+n = 100
+x = 2*np.random.rand(n,1)
+y = 4+3*x#+np.random.randn(n,1)
+
+X = np.c_[np.ones((n,1)), x]
+XT_X = X.T @ X
+theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
+print("Own inversion")
+print(theta_linreg)
+# Hessian matrix
+H = (2.0/n)* XT_X
+EigValues, EigVectors = np.linalg.eig(H)
+print(f"Eigenvalues of Hessian Matrix:{EigValues}")
+
+theta = np.random.randn(2,1)
+eta = 1.0/np.max(EigValues)
+Niterations = 30
+
+# define the gradient
+training_gradient = grad(CostOLS)
+
+for iter in range(Niterations):
+    gradients = training_gradient(theta)
+    theta -= eta*gradients
+    print(iter,gradients[0],gradients[1])
+print("theta from own gd")
+print(theta)
+
+# Now improve with momentum gradient descent
+change = 0.0
+delta_momentum = 0.3
+for iter in range(Niterations):
+    # calculate gradient
+    gradients = training_gradient(theta)
+    # calculate update
+    new_change = eta*gradients+delta_momentum*change
+    # take a step
+    theta -= new_change
+    # save the change
+    change = new_change
+    print(iter,gradients[0],gradients[1])
+print("theta from own gd wth momentum")
+print(theta)
+
+!ec
+We note indeed a considerable increase in efficiency here, we less iterations needed.
+However, if we can invert the Hessian matrix, this is the preferred approach, as shown in the example here.
+
+!bc pycod
+# Using Newton's method
+from random import random, seed
+import numpy as np
+import autograd.numpy as np
+import matplotlib.pyplot as plt
+from autograd import grad
+
+def CostOLS(beta):
+    return (1.0/n)*np.sum((y-X @ beta)**2)
+
+n = 100
+x = 2*np.random.rand(n,1)
+y = 4+3*x+np.random.randn(n,1)
+
+X = np.c_[np.ones((n,1)), x]
+XT_X = X.T @ X
+beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
+print("Own inversion")
+print(beta_linreg)
+# Hessian matrix
+H = (2.0/n)* XT_X
+# Note that here the Hessian does not depend on the parameters beta
+invH = np.linalg.pinv(H)
+EigValues, EigVectors = np.linalg.eig(H)
+print(f"Eigenvalues of Hessian Matrix:{EigValues}")
+
+beta = np.random.randn(2,1)
+Niterations = 5
+
+# define the gradient
+training_gradient = grad(CostOLS)
+
+for iter in range(Niterations):
+    gradients = training_gradient(beta)
+    beta -= invH @ gradients
+    print(iter,gradients[0],gradients[1])
+print("beta from own Newton code")
+print(beta)
+!ec
+
+
+
+===== Including Stochastic Gradient Descent with Autograd =====
 In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using _autograd_.
 
 !bc pycod
@@ -1898,43 +2017,223 @@ print(theta)
 
 !ec
 
-
-=== And Logistic Regression ===
-
+Here we include momentum in the standard gradient descent approach.
 !bc pycod
+# Using Autograd to calculate gradients using SGD
+# OLS example
+from random import random, seed
+import numpy as np
 import autograd.numpy as np
+import matplotlib.pyplot as plt
 from autograd import grad
 
-def sigmoid(x):
-    return 0.5 * (np.tanh(x / 2.) + 1)
+# Note change from previous example
+def CostOLS(y,X,theta):
+    return np.sum((y-X @ theta)**2)
 
-def logistic_predictions(weights, inputs):
-    # Outputs probability of a label being true according to logistic model.
-    return sigmoid(np.dot(inputs, weights))
+n = 100
+x = 2*np.random.rand(n,1)
+y = 4+3*x+np.random.randn(n,1)
 
-def training_loss(weights):
-    # Training loss is the negative log-likelihood of the training labels.
-    preds = logistic_predictions(weights, inputs)
-    label_probabilities = preds * targets + (1 - preds) * (1 - targets)
-    return -np.sum(np.log(label_probabilities))
+X = np.c_[np.ones((n,1)), x]
+XT_X = X.T @ X
+theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
+print("Own inversion")
+print(theta_linreg)
+# Hessian matrix
+H = (2.0/n)* XT_X
+EigValues, EigVectors = np.linalg.eig(H)
+print(f"Eigenvalues of Hessian Matrix:{EigValues}")
 
-# Build a toy dataset.
-inputs = np.array([[0.52, 1.12,  0.77],
-                   [0.88, -1.08, 0.15],
-                   [0.52, 0.06, -1.30],
-                   [0.74, -2.49, 1.39]])
-targets = np.array([True, True, False, True])
+theta = np.random.randn(2,1)
+eta = 1.0/np.max(EigValues)
+Niterations = 100
 
-# Define a function that returns gradients of training loss using Autograd.
-training_gradient_fun = grad(training_loss)
+# Note that we request the derivative wrt third argument (theta, 2 here)
+training_gradient = grad(CostOLS,2)
 
-# Optimize weights using gradient descent.
-weights = np.array([0.0, 0.0, 0.0])
-print("Initial loss:", training_loss(weights))
-for i in range(100):
-    weights -= training_gradient_fun(weights) * 0.01
+for iter in range(Niterations):
+    gradients = (1.0/n)*training_gradient(y, X, theta)
+    theta -= eta*gradients
+print("theta from own gd")
+print(theta)
 
-print("Trained loss:", training_loss(weights))
+
+n_epochs = 50
+M = 5   #size of each minibatch
+m = int(n/M) #number of minibatches
+t0, t1 = 5, 50
+def learning_schedule(t):
+    return t0/(t+t1)
+
+theta = np.random.randn(2,1)
+
+change = 0.0
+delta_momentum = 0.3
+
+for epoch in range(n_epochs):
+    for i in range(m):
+        random_index = M*np.random.randint(m)
+        xi = X[random_index:random_index+M]
+        yi = y[random_index:random_index+M]
+        gradients = (1.0/M)*training_gradient(yi, xi, theta)
+        eta = learning_schedule(epoch*m+i)
+        # calculate update
+        new_change = eta*gradients+delta_momentum*change
+        # take a step
+        theta -= new_change
+        # save the change
+        change = new_change
+print("theta from own sdg with momentum")
+print(theta)
 !ec
 
 
+
+=== Similar (second order function now) problem but now with AdaGrad ===
+!bc pycod
+# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent
+# OLS example
+from random import random, seed
+import numpy as np
+import autograd.numpy as np
+import matplotlib.pyplot as plt
+from autograd import grad
+
+# Note change from previous example
+def CostOLS(y,X,theta):
+    return np.sum((y-X @ theta)**2)
+
+n = 10000
+x = np.random.rand(n,1)
+y = 2.0+3*x +4*x*x# +np.random.randn(n,1)
+
+X = np.c_[np.ones((n,1)), x, x*x]
+XT_X = X.T @ X
+theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
+print("Own inversion")
+print(theta_linreg)
+
+
+# Note that we request the derivative wrt third argument (theta, 2 here)
+training_gradient = grad(CostOLS,2)
+# Define parameters for Stochastic Gradient Descent
+n_epochs = 50
+M = 5   #size of each minibatch
+m = int(n/M) #number of minibatches
+# Guess for unknown parameters theta
+theta = np.random.randn(3,1)
+
+# Value for learning rate
+eta = 0.01
+# Including AdaGrad parameter to avoid possible division by zero
+delta  = 1e-8
+for epoch in range(n_epochs):
+    # The outer product is calculated from scratch for each epoch
+    Giter = np.zeros(shape=(3,3))
+    for i in range(m):
+        random_index = M*np.random.randint(m)
+        xi = X[random_index:random_index+M]
+        yi = y[random_index:random_index+M]
+        gradients = (1.0/M)*training_gradient(yi, xi, theta)
+	# Calculate the outer product of the gradients
+        Giter +=gradients @ gradients.T
+	# Simpler algorithm with only diagonal elements
+        Ginverse = np.c_[eta/(delta+np.sqrt(np.diagonal(Giter)))]
+        # compute update
+        update = np.multiply(Ginverse,gradients)
+        theta -= update
+print("theta from own AdaGrad")
+print(theta)
+!ec
+
+Running this code we note an almost perfect agreement with the results from matrix inversion.
+
+Similarly, here is our implementation of RMSprop.
+!bc pycod
+# Using Autograd to calculate gradients using RMSprop  and Stochastic Gradient descent
+# OLS example
+from random import random, seed
+import numpy as np
+import autograd.numpy as np
+import matplotlib.pyplot as plt
+from autograd import grad
+
+# Note change from previous example
+def CostOLS(y,X,theta):
+    return np.sum((y-X @ theta)**2)
+
+n = 10000
+x = np.random.rand(n,1)
+y = 2.0+3*x +4*x*x# +np.random.randn(n,1)
+
+X = np.c_[np.ones((n,1)), x, x*x]
+XT_X = X.T @ X
+theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
+print("Own inversion")
+print(theta_linreg)
+
+
+# Note that we request the derivative wrt third argument (theta, 2 here)
+training_gradient = grad(CostOLS,2)
+# Define parameters for Stochastic Gradient Descent
+n_epochs = 50
+M = 5   #size of each minibatch
+m = int(n/M) #number of minibatches
+# Guess for unknown parameters theta
+theta = np.random.randn(3,1)
+
+# Value for learning rate
+eta = 0.01
+# Value for parameter rho
+rho = 0.99
+# Including AdaGrad parameter to avoid possible division by zero
+delta  = 1e-8
+for epoch in range(n_epochs):
+    Giter = np.zeros(shape=(3,3))
+    for i in range(m):
+        random_index = M*np.random.randint(m)
+        xi = X[random_index:random_index+M]
+        yi = y[random_index:random_index+M]
+        gradients = (1.0/M)*training_gradient(yi, xi, theta)
+	# Previous value for the outer product of gradients
+        Previous = Giter
+	# Accumulated gradient
+        Giter +=gradients @ gradients.T
+	# Scaling with rho the new and the previous results
+        Gnew = (rho*Previous+(1-rho)*Giter)
+	# Taking the diagonal only and inverting
+        Ginverse = np.c_[eta/(delta+np.sqrt(np.diagonal(Gnew)))]
+	# Hadamard product
+        update = np.multiply(Ginverse,gradients)
+        theta -= update
+print("theta from own RMSprop")
+print(theta)
+!ec
+
+
+
+
+===== Introducing "JAX":"https://jax.readthedocs.io/en/latest/" =====
+
+Presently, instead of using _autograd_, we recommend using "JAX":"https://jax.readthedocs.io/en/latest/"
+
+_JAX_ is Autograd and "XLA (Accelerated Linear Algebra))":"https://www.tensorflow.org/xla",
+brought together for high-performance numerical computing and machine learning research.
+It provides composable transformations of Python+NumPy programs: differentiate, vectorize, parallelize, Just-In-Time compile to GPU/TPU, and more.
+
+Here's a simple example on how you can use _JAX_ to compute the derivate of the logistic function.
+
+!bc pycod
+import jax.numpy as jnp
+from jax import grad, jit, vmap
+
+def sum_logistic(x):
+  return jnp.sum(1.0 / (1.0 + jnp.exp(-x)))
+
+x_small = jnp.arange(3.)
+derivative_fn = grad(sum_logistic)
+print(derivative_fn(x_small))
+
+!ec
+
diff --git a/doc/LectureNotes/chapter1.ipynb b/doc/LectureNotes/chapter1.ipynb
index 74e22cf88..2e528819b 100644
--- a/doc/LectureNotes/chapter1.ipynb
+++ b/doc/LectureNotes/chapter1.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "d9b4c9c8",
+   "id": "27b96943",
    "metadata": {
     "editable": true
    },
@@ -13,7 +13,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "65e44d92",
+   "id": "ee293e0b",
    "metadata": {
     "editable": true
    },
@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "06b62649",
+   "id": "f6481760",
    "metadata": {
     "editable": true
    },
@@ -65,7 +65,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8de26fd1",
+   "id": "fd2c7eef",
    "metadata": {
     "editable": true
    },
@@ -117,7 +117,7 @@
     "Machine learning is an extremely rich field, in spite of its young\n",
     "age. The increases we have seen during the last three decades in\n",
     "computational capabilities have been followed by developments of\n",
-    "methods and techniques for analyzing and handling large date sets,\n",
+    "methods and techniques for analyzing and handling large data sets,\n",
     "relying heavily on statistics, computer science and mathematics.  The\n",
     "field is rather new and developing rapidly. Popular software packages\n",
     "written in Python for machine learning like\n",
@@ -140,7 +140,7 @@
     "problem, and let the computer deduce the logic behind it. On the other\n",
     "hand, *unsupervised learning* is a method for finding patterns and\n",
     "relationship in data sets without any prior knowledge of the system.\n",
-    "Some authours also operate with a third category, namely\n",
+    "Some authors also operate with a third category, namely\n",
     "*reinforcement learning*. This is a paradigm of learning inspired by\n",
     "behavioral psychology, where learning is achieved by trial-and-error,\n",
     "solely from rewards and punishment.\n",
@@ -167,7 +167,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1ff6afb4",
+   "id": "ff3dccf1",
    "metadata": {
     "editable": true
    },
@@ -202,7 +202,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "98d9014b",
+   "id": "89a1286b",
    "metadata": {
     "editable": true
    },
@@ -212,14 +212,14 @@
     "In science and engineering we often end up in situations where we want to infer (or learn) a\n",
     "quantitative model $M$ for a given set of sample points $\\boldsymbol{X} \\in [x_1, x_2,\\dots x_N]$.\n",
     "\n",
-    "As we will see repeatedely in these lectures, we could try to fit these data points to a model given by a\n",
+    "As we will see repeatedly in these lectures, we could try to fit these data points to a model given by a\n",
     "straight line, or if we wish to be more sophisticated to a more complex\n",
     "function.\n",
     "\n",
     "The reason for inferring such a model is that it\n",
     "serves many useful purposes. On the one hand, the model can reveal information\n",
     "encoded in the data or underlying mechanisms from which the data were generated. For instance, we could discover important\n",
-    "corelations that relate interesting physics interpretations.\n",
+    "correlations that relate interesting physics interpretations.\n",
     "\n",
     "In addition, it can simplify the representation of the given data set and help\n",
     "us in making predictions about  future data samples.\n",
@@ -253,7 +253,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4214f050",
+   "id": "bf0c0745",
    "metadata": {
     "editable": true
    },
@@ -286,7 +286,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "827d0b8e",
+   "id": "aff5ae6b",
    "metadata": {
     "editable": true
    },
@@ -298,7 +298,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "860f3619",
+   "id": "9bf3106e",
    "metadata": {
     "editable": true
    },
@@ -335,7 +335,7 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "b9b8fe1b",
+   "id": "8f9fe452",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -368,7 +368,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "97a78fd5",
+   "id": "a7ce677a",
    "metadata": {
     "editable": true
    },
@@ -385,7 +385,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5036f5a0",
+   "id": "a9463a86",
    "metadata": {
     "editable": true
    },
@@ -397,7 +397,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0e276aea",
+   "id": "87b5de04",
    "metadata": {
     "editable": true
    },
@@ -405,7 +405,7 @@
     "where $x$ is defined as before.  Does the fit look better? Indeed, by\n",
     "reducing the role of the noise given by the normal distribution we see immediately that\n",
     "our linear prediction seemingly reproduces better the training\n",
-    "set. However, this testing 'by the eye' is obviouly not satisfactory in the\n",
+    "set. However, this testing 'by the eye' is obviously not satisfactory in the\n",
     "long run. Here we have only defined the training data and our model, and \n",
     "have not discussed a more rigorous approach to the **cost** function.\n",
     "\n",
@@ -418,7 +418,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "66f6fd13",
+   "id": "e9e757d9",
    "metadata": {
     "editable": true
    },
@@ -431,7 +431,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f29b024f",
+   "id": "40298a87",
    "metadata": {
     "editable": true
    },
@@ -462,7 +462,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dcce4d43",
+   "id": "9c2c9d9d",
    "metadata": {
     "editable": true
    },
@@ -474,7 +474,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "20cd881f",
+   "id": "9baeab03",
    "metadata": {
     "editable": true
    },
@@ -492,7 +492,7 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "aca4204c",
+   "id": "bd76613f",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -520,7 +520,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9925c6cb",
+   "id": "5c78adc9",
    "metadata": {
     "editable": true
    },
@@ -542,7 +542,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "5f927fc2",
+   "id": "d38e9c7c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -580,7 +580,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3c238d46",
+   "id": "c76d7e9f",
    "metadata": {
     "editable": true
    },
@@ -591,7 +591,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "428d6164",
+   "id": "6a528c0f",
    "metadata": {
     "editable": true
    },
@@ -604,7 +604,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a4046ea1",
+   "id": "74377872",
    "metadata": {
     "editable": true
    },
@@ -625,7 +625,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3c665298",
+   "id": "27ad828e",
    "metadata": {
     "editable": true
    },
@@ -637,7 +637,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dff6fbc4",
+   "id": "b7bf4db8",
    "metadata": {
     "editable": true
    },
@@ -647,7 +647,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "81466e4e",
+   "id": "7530e179",
    "metadata": {
     "editable": true
    },
@@ -659,7 +659,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "40f0760e",
+   "id": "4672d1e1",
    "metadata": {
     "editable": true
    },
@@ -671,7 +671,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2489c24a",
+   "id": "00840b14",
    "metadata": {
     "editable": true
    },
@@ -683,7 +683,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3f625a67",
+   "id": "0c5e43af",
    "metadata": {
     "editable": true
    },
@@ -694,7 +694,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6608f259",
+   "id": "a4a68023",
    "metadata": {
     "editable": true
    },
@@ -706,7 +706,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d63ae4a9",
+   "id": "adbc5f5f",
    "metadata": {
     "editable": true
    },
@@ -728,7 +728,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ba2ac7cd",
+   "id": "be191d7c",
    "metadata": {
     "editable": true
    },
@@ -740,7 +740,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "31861c56",
+   "id": "f4549dc2",
    "metadata": {
     "editable": true
    },
@@ -755,7 +755,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "e2e82632",
+   "id": "a6bd8c76",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -796,7 +796,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9f77ad8a",
+   "id": "6c8b9965",
    "metadata": {
     "editable": true
    },
@@ -811,7 +811,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "79ba67a3",
+   "id": "2852a934",
    "metadata": {
     "editable": true
    },
@@ -823,7 +823,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6ed1b749",
+   "id": "74fd99a0",
    "metadata": {
     "editable": true
    },
@@ -833,7 +833,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7f1a81f1",
+   "id": "386c640d",
    "metadata": {
     "editable": true
    },
@@ -845,7 +845,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3a92d75a",
+   "id": "9e7c7132",
    "metadata": {
     "editable": true
    },
@@ -855,7 +855,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2717347f",
+   "id": "3bfd0139",
    "metadata": {
     "editable": true
    },
@@ -867,7 +867,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "96b2d4fb",
+   "id": "0ddf3bb3",
    "metadata": {
     "editable": true
    },
@@ -877,7 +877,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c7705b30",
+   "id": "e9c65dac",
    "metadata": {
     "editable": true
    },
@@ -889,7 +889,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4e679c00",
+   "id": "6addb221",
    "metadata": {
     "editable": true
    },
@@ -905,7 +905,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e5e89d2d",
+   "id": "75fd0e61",
    "metadata": {
     "editable": true
    },
@@ -917,7 +917,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d17826c1",
+   "id": "209a6361",
    "metadata": {
     "editable": true
    },
@@ -928,7 +928,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8cc3306a",
+   "id": "ecf9b9da",
    "metadata": {
     "editable": true
    },
@@ -940,7 +940,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b2ba5419",
+   "id": "a931cfa6",
    "metadata": {
     "editable": true
    },
@@ -954,7 +954,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a94d4658",
+   "id": "f2cbfc44",
    "metadata": {
     "editable": true
    },
@@ -966,7 +966,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0c7a9577",
+   "id": "719d770d",
    "metadata": {
     "editable": true
    },
@@ -991,7 +991,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "acb382a9",
+   "id": "4983af73",
    "metadata": {
     "editable": true
    },
@@ -1008,7 +1008,7 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "828ac44b",
+   "id": "b5c8db7d",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1052,7 +1052,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12105e10",
+   "id": "b54a87d1",
    "metadata": {
     "editable": true
    },
@@ -1063,7 +1063,7 @@
   {
    "cell_type": "code",
    "execution_count": 6,
-   "id": "f3512753",
+   "id": "180faa92",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1085,7 +1085,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8d18c1cd",
+   "id": "6fe942e7",
    "metadata": {
     "editable": true
    },
@@ -1102,7 +1102,7 @@
   {
    "cell_type": "code",
    "execution_count": 7,
-   "id": "9b164e78",
+   "id": "68ed7165",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1123,7 +1123,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f812fd67",
+   "id": "2b775c80",
    "metadata": {
     "editable": true
    },
@@ -1137,7 +1137,7 @@
   {
    "cell_type": "code",
    "execution_count": 8,
-   "id": "56d7b683",
+   "id": "2c94c74f",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1166,7 +1166,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "247978a1",
+   "id": "7be1a100",
    "metadata": {
     "editable": true
    },
@@ -1186,7 +1186,7 @@
   {
    "cell_type": "code",
    "execution_count": 9,
-   "id": "f62c82f4",
+   "id": "b1e553c9",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1203,7 +1203,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "17ef28cf",
+   "id": "b1e4faff",
    "metadata": {
     "editable": true
    },
@@ -1215,7 +1215,7 @@
   {
    "cell_type": "code",
    "execution_count": 10,
-   "id": "261f54be",
+   "id": "c460ff45",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1233,7 +1233,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e22ef3d0",
+   "id": "8095634f",
    "metadata": {
     "editable": true
    },
@@ -1249,7 +1249,7 @@
   {
    "cell_type": "code",
    "execution_count": 11,
-   "id": "952c162e",
+   "id": "61ce83d5",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1262,7 +1262,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d73978fc",
+   "id": "4fd2409e",
    "metadata": {
     "editable": true
    },
@@ -1274,7 +1274,7 @@
   {
    "cell_type": "code",
    "execution_count": 12,
-   "id": "600af054",
+   "id": "dbaacd6c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1304,7 +1304,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a10883fb",
+   "id": "b4982c89",
    "metadata": {
     "editable": true
    },
@@ -1315,7 +1315,7 @@
   {
    "cell_type": "code",
    "execution_count": 13,
-   "id": "0f967044",
+   "id": "931f039d",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1356,7 +1356,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12cfe86f",
+   "id": "e3f80f20",
    "metadata": {
     "editable": true
    },
@@ -1378,7 +1378,7 @@
   {
    "cell_type": "code",
    "execution_count": 14,
-   "id": "75baf4ec",
+   "id": "e56d424f",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1418,7 +1418,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "11e6623d",
+   "id": "27dbd129",
    "metadata": {
     "editable": true
    },
@@ -1488,7 +1488,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1d144bd9",
+   "id": "936b8537",
    "metadata": {
     "editable": true
    },
@@ -1500,7 +1500,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "83596488",
+   "id": "d6fb0e7c",
    "metadata": {
     "editable": true
    },
@@ -1519,7 +1519,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0d0443e0",
+   "id": "c370ec54",
    "metadata": {
     "editable": true
    },
@@ -1531,7 +1531,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "09b0cf81",
+   "id": "df7b365e",
    "metadata": {
     "editable": true
    },
@@ -1543,7 +1543,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a0877d1e",
+   "id": "96f2187d",
    "metadata": {
     "editable": true
    },
@@ -1561,7 +1561,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3a067e9a",
+   "id": "6433b033",
    "metadata": {
     "editable": true
    },
@@ -1571,7 +1571,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4e083f08",
+   "id": "d7d4ff3b",
    "metadata": {
     "editable": true
    },
@@ -1583,7 +1583,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e1fa2d76",
+   "id": "5a52a847",
    "metadata": {
     "editable": true
    },
@@ -1593,7 +1593,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c33752b2",
+   "id": "3911b9d1",
    "metadata": {
     "editable": true
    },
@@ -1605,7 +1605,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8ea0d39b",
+   "id": "069cbce7",
    "metadata": {
     "editable": true
    },
@@ -1615,7 +1615,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2c4dbdc3",
+   "id": "98c47380",
    "metadata": {
     "editable": true
    },
@@ -1627,7 +1627,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "13be3123",
+   "id": "69851a2e",
    "metadata": {
     "editable": true
    },
@@ -1637,7 +1637,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c548ec0c",
+   "id": "27367380",
    "metadata": {
     "editable": true
    },
@@ -1656,7 +1656,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "64ab4abc",
+   "id": "57e81e74",
    "metadata": {
     "editable": true
    },
@@ -1666,7 +1666,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cacc8343",
+   "id": "306e51b5",
    "metadata": {
     "editable": true
    },
@@ -1678,7 +1678,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8270eaa5",
+   "id": "076e49a1",
    "metadata": {
     "editable": true
    },
@@ -1694,7 +1694,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "154017fc",
+   "id": "6ca152bd",
    "metadata": {
     "editable": true
    },
@@ -1714,7 +1714,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0053d6cc",
+   "id": "aa30be64",
    "metadata": {
     "editable": true
    },
@@ -1726,7 +1726,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "65929876",
+   "id": "138531d5",
    "metadata": {
     "editable": true
    },
@@ -1745,7 +1745,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1b28cdfa",
+   "id": "e2760fce",
    "metadata": {
     "editable": true
    },
@@ -1755,7 +1755,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "35773b63",
+   "id": "8df047bc",
    "metadata": {
     "editable": true
    },
@@ -1767,7 +1767,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bf783586",
+   "id": "dee93942",
    "metadata": {
     "editable": true
    },
@@ -1779,7 +1779,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8da3867f",
+   "id": "2dcb17e6",
    "metadata": {
     "editable": true
    },
@@ -1799,7 +1799,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "927facb3",
+   "id": "ecf88b59",
    "metadata": {
     "editable": true
    },
@@ -1816,7 +1816,7 @@
   {
    "cell_type": "code",
    "execution_count": 15,
-   "id": "7aca7c7a",
+   "id": "f56b01eb",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1896,7 +1896,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c1260224",
+   "id": "9987d583",
    "metadata": {
     "editable": true
    },
@@ -1906,7 +1906,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a9549e65",
+   "id": "de65ddd1",
    "metadata": {
     "editable": true
    },
@@ -1918,7 +1918,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "29c67e39",
+   "id": "a11d4702",
    "metadata": {
     "editable": true
    },
@@ -1930,7 +1930,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7fe12004",
+   "id": "2050adb4",
    "metadata": {
     "editable": true
    },
@@ -1942,7 +1942,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a974412a",
+   "id": "233838f8",
    "metadata": {
     "editable": true
    },
@@ -1952,7 +1952,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "03241cbf",
+   "id": "a849e753",
    "metadata": {
     "editable": true
    },
@@ -1964,7 +1964,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bb96d559",
+   "id": "56d0fde7",
    "metadata": {
     "editable": true
    },
@@ -1974,7 +1974,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "108c564b",
+   "id": "4b42a997",
    "metadata": {
     "editable": true
    },
@@ -1986,7 +1986,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "041588ea",
+   "id": "02d1ba2c",
    "metadata": {
     "editable": true
    },
@@ -1999,7 +1999,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2c2ac147",
+   "id": "1ca3f7c3",
    "metadata": {
     "editable": true
    },
@@ -2011,7 +2011,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f4155adc",
+   "id": "2bab5254",
    "metadata": {
     "editable": true
    },
@@ -2023,7 +2023,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f7a73ebb",
+   "id": "0a2dcdbd",
    "metadata": {
     "editable": true
    },
@@ -2035,7 +2035,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "058f09a6",
+   "id": "d00d31ad",
    "metadata": {
     "editable": true
    },
@@ -2046,7 +2046,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dcdfbbc5",
+   "id": "0b9759c4",
    "metadata": {
     "editable": true
    },
@@ -2058,7 +2058,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "206a6652",
+   "id": "f98e5455",
    "metadata": {
     "editable": true
    },
@@ -2077,7 +2077,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "84f4071a",
+   "id": "03e2f2c5",
    "metadata": {
     "editable": true
    },
@@ -2090,7 +2090,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "aec4af35",
+   "id": "e8920d79",
    "metadata": {
     "editable": true
    },
@@ -2100,7 +2100,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "de7dcc20",
+   "id": "98ad07e2",
    "metadata": {
     "editable": true
    },
@@ -2112,7 +2112,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ce6f83f2",
+   "id": "d0783b2b",
    "metadata": {
     "editable": true
    },
@@ -2122,7 +2122,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1780e810",
+   "id": "a33f49bc",
    "metadata": {
     "editable": true
    },
@@ -2134,7 +2134,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6b4d0be0",
+   "id": "bb935afa",
    "metadata": {
     "editable": true
    },
@@ -2144,7 +2144,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "021fe0da",
+   "id": "19eacabb",
    "metadata": {
     "editable": true
    },
@@ -2156,7 +2156,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "90bf5b47",
+   "id": "5975a309",
    "metadata": {
     "editable": true
    },
@@ -2166,7 +2166,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "15e45481",
+   "id": "ed1537ed",
    "metadata": {
     "editable": true
    },
@@ -2178,7 +2178,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d758a151",
+   "id": "c897d003",
    "metadata": {
     "editable": true
    },
@@ -2188,7 +2188,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c5a05d3f",
+   "id": "199d48f4",
    "metadata": {
     "editable": true
    },
@@ -2200,7 +2200,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f57d857f",
+   "id": "931416f6",
    "metadata": {
     "editable": true
    },
@@ -2210,7 +2210,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e7c11cc",
+   "id": "fc031f0e",
    "metadata": {
     "editable": true
    },
@@ -2222,7 +2222,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6d0438bc",
+   "id": "6e04f9f2",
    "metadata": {
     "editable": true
    },
@@ -2246,7 +2246,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "81350349",
+   "id": "c35c6bf5",
    "metadata": {
     "editable": true
    },
@@ -2258,7 +2258,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "470abb5b",
+   "id": "cf43264c",
    "metadata": {
     "editable": true
    },
@@ -2268,7 +2268,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e49b3680",
+   "id": "f5f866f4",
    "metadata": {
     "editable": true
    },
@@ -2280,7 +2280,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "71cc2366",
+   "id": "d9fa677c",
    "metadata": {
     "editable": true
    },
@@ -2290,7 +2290,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "494e1da4",
+   "id": "eaade1fd",
    "metadata": {
     "editable": true
    },
@@ -2302,7 +2302,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "88fa8938",
+   "id": "612c99ac",
    "metadata": {
     "editable": true
    },
@@ -2315,7 +2315,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f2886bf5",
+   "id": "d5b941a8",
    "metadata": {
     "editable": true
    },
@@ -2327,7 +2327,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2b2dc285",
+   "id": "eba447d8",
    "metadata": {
     "editable": true
    },
@@ -2339,7 +2339,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "762f5b0f",
+   "id": "f08187a8",
    "metadata": {
     "editable": true
    },
@@ -2351,7 +2351,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "66c7bf97",
+   "id": "177b3ca3",
    "metadata": {
     "editable": true
    },
@@ -2366,7 +2366,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c57edc78",
+   "id": "20190e91",
    "metadata": {
     "editable": true
    },
@@ -2378,7 +2378,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6d750248",
+   "id": "c4ece1ae",
    "metadata": {
     "editable": true
    },
@@ -2388,7 +2388,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae22e8f6",
+   "id": "bc014cc9",
    "metadata": {
     "editable": true
    },
@@ -2400,7 +2400,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "70c9ad4d",
+   "id": "21b86234",
    "metadata": {
     "editable": true
    },
@@ -2410,7 +2410,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "02fec10f",
+   "id": "99699900",
    "metadata": {
     "editable": true
    },
@@ -2422,7 +2422,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8e2a2e61",
+   "id": "2c98eb03",
    "metadata": {
     "editable": true
    },
@@ -2438,7 +2438,7 @@
   {
    "cell_type": "code",
    "execution_count": 16,
-   "id": "9f6f9239",
+   "id": "8da0a506",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2453,7 +2453,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "604dbc32",
+   "id": "c1c6fbb0",
    "metadata": {
     "editable": true
    },
@@ -2464,7 +2464,7 @@
   {
    "cell_type": "code",
    "execution_count": 17,
-   "id": "00a1ced4",
+   "id": "104e0f22",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2477,7 +2477,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7ceb5efd",
+   "id": "340d198a",
    "metadata": {
     "editable": true
    },
@@ -2488,7 +2488,7 @@
   {
    "cell_type": "code",
    "execution_count": 18,
-   "id": "e462c872",
+   "id": "ade033f9",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2511,7 +2511,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e0ef3a79",
+   "id": "9b722931",
    "metadata": {
     "editable": true
    },
@@ -2523,7 +2523,7 @@
   {
    "cell_type": "code",
    "execution_count": 19,
-   "id": "1f66c40c",
+   "id": "bf4c610b",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2536,7 +2536,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e46854b6",
+   "id": "4d3682f3",
    "metadata": {
     "editable": true
    },
@@ -2547,7 +2547,7 @@
   {
    "cell_type": "code",
    "execution_count": 20,
-   "id": "ec1fee3e",
+   "id": "28c41936",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2559,7 +2559,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5a3558b1",
+   "id": "643d4ba8",
    "metadata": {
     "editable": true
    },
@@ -2570,7 +2570,7 @@
   {
    "cell_type": "code",
    "execution_count": 21,
-   "id": "4aae389c",
+   "id": "28f76048",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2586,7 +2586,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a04587b4",
+   "id": "1a31aa74",
    "metadata": {
     "editable": true
    },
@@ -2597,7 +2597,7 @@
   {
    "cell_type": "code",
    "execution_count": 22,
-   "id": "19a649a5",
+   "id": "cf5ba28a",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2611,7 +2611,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4e526f30",
+   "id": "1fcd9105",
    "metadata": {
     "editable": true
    },
@@ -2633,7 +2633,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f55ada0e",
+   "id": "7c37062a",
    "metadata": {
     "editable": true
    },
@@ -2645,7 +2645,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b58ad660",
+   "id": "7494594e",
    "metadata": {
     "editable": true
    },
@@ -2657,7 +2657,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3e135503",
+   "id": "31f0536d",
    "metadata": {
     "editable": true
    },
@@ -2669,7 +2669,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1a659dc5",
+   "id": "80b342bd",
    "metadata": {
     "editable": true
    },
@@ -2679,7 +2679,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e66bd75c",
+   "id": "6ae0b01c",
    "metadata": {
     "editable": true
    },
@@ -2691,7 +2691,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4d6a21c9",
+   "id": "4738cefc",
    "metadata": {
     "editable": true
    },
@@ -2701,7 +2701,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "31ca7b17",
+   "id": "b60ccc66",
    "metadata": {
     "editable": true
    },
@@ -2713,7 +2713,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "66443630",
+   "id": "14251ed9",
    "metadata": {
     "editable": true
    },
@@ -2725,7 +2725,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "28f84c45",
+   "id": "d7774e34",
    "metadata": {
     "editable": true
    },
@@ -2737,7 +2737,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a292188",
+   "id": "b44744ad",
    "metadata": {
     "editable": true
    },
@@ -2747,7 +2747,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "19acde5f",
+   "id": "2d48b6d2",
    "metadata": {
     "editable": true
    },
@@ -2759,7 +2759,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f4bfc2ba",
+   "id": "68536d4b",
    "metadata": {
     "editable": true
    },
@@ -2769,7 +2769,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ffe9cdc3",
+   "id": "ec458e7f",
    "metadata": {
     "editable": true
    },
@@ -2781,7 +2781,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e6cd175c",
+   "id": "ad31b9e9",
    "metadata": {
     "editable": true
    },
@@ -2791,7 +2791,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4520bd32",
+   "id": "c188e550",
    "metadata": {
     "editable": true
    },
@@ -2803,7 +2803,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "018f1bcc",
+   "id": "80f77d5d",
    "metadata": {
     "editable": true
    },
@@ -2813,7 +2813,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "657cad6a",
+   "id": "a2916ab9",
    "metadata": {
     "editable": true
    },
@@ -2825,7 +2825,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "24fe79df",
+   "id": "1f74a112",
    "metadata": {
     "editable": true
    },
@@ -2835,7 +2835,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6c7db826",
+   "id": "d4dc1c29",
    "metadata": {
     "editable": true
    },
@@ -2847,7 +2847,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1927e34a",
+   "id": "338246d7",
    "metadata": {
     "editable": true
    },
@@ -2857,7 +2857,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "406ca333",
+   "id": "ca1e4df5",
    "metadata": {
     "editable": true
    },
@@ -2869,7 +2869,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6acb7f48",
+   "id": "bea336e4",
    "metadata": {
     "editable": true
    },
@@ -2879,7 +2879,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7cf70bf5",
+   "id": "ce3eb821",
    "metadata": {
     "editable": true
    },
@@ -2891,7 +2891,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b0413ba5",
+   "id": "a117a708",
    "metadata": {
     "editable": true
    },
@@ -2901,7 +2901,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b613c67c",
+   "id": "8dd2c2d0",
    "metadata": {
     "editable": true
    },
@@ -2913,7 +2913,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "85693726",
+   "id": "a131a505",
    "metadata": {
     "editable": true
    },
@@ -2923,7 +2923,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "64582090",
+   "id": "db3ad647",
    "metadata": {
     "editable": true
    },
@@ -2935,7 +2935,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "061bb39a",
+   "id": "1e2b9af1",
    "metadata": {
     "editable": true
    },
@@ -2946,7 +2946,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c750e91e",
+   "id": "8bb14967",
    "metadata": {
     "editable": true
    },
@@ -2958,7 +2958,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "28124d83",
+   "id": "e111117b",
    "metadata": {
     "editable": true
    },
@@ -2970,7 +2970,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d7128979",
+   "id": "dbbc1f3b",
    "metadata": {
     "editable": true
    },
@@ -2982,7 +2982,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ec2f92ad",
+   "id": "13a154e3",
    "metadata": {
     "editable": true
    },
@@ -2994,7 +2994,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b0ac5589",
+   "id": "a2529176",
    "metadata": {
     "editable": true
    },
@@ -3006,7 +3006,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b8de935b",
+   "id": "4c93ae77",
    "metadata": {
     "editable": true
    },
@@ -3016,7 +3016,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ffb07e70",
+   "id": "b4e80da6",
    "metadata": {
     "editable": true
    },
@@ -3028,7 +3028,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f52a2b52",
+   "id": "5b40d231",
    "metadata": {
     "editable": true
    },
@@ -3040,7 +3040,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "21d55a92",
+   "id": "ea093e3a",
    "metadata": {
     "editable": true
    },
@@ -3054,7 +3054,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "35eff2a6",
+   "id": "0e3a2d2b",
    "metadata": {
     "editable": true
    },
@@ -3080,7 +3080,7 @@
   {
    "cell_type": "code",
    "execution_count": 23,
-   "id": "28b5fe82",
+   "id": "b0874c6b",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3162,7 +3162,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2b3abee3",
+   "id": "202c5365",
    "metadata": {
     "editable": true
    },
@@ -3173,7 +3173,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1b449839",
+   "id": "9d0addca",
    "metadata": {
     "editable": true
    },
@@ -3201,7 +3201,7 @@
   {
    "cell_type": "code",
    "execution_count": 24,
-   "id": "d99bbcf6",
+   "id": "270f88ea",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3250,7 +3250,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2e67c926",
+   "id": "6f52faab",
    "metadata": {
     "editable": true
    },
@@ -3261,7 +3261,7 @@
   {
    "cell_type": "code",
    "execution_count": 25,
-   "id": "72bf715e",
+   "id": "9249b1f2",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3286,7 +3286,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7cea6c55",
+   "id": "7ce8cc8a",
    "metadata": {
     "editable": true
    },
@@ -3303,7 +3303,7 @@
   {
    "cell_type": "code",
    "execution_count": 26,
-   "id": "32a2c51d",
+   "id": "9413ed10",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3378,7 +3378,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "988b4c3c",
+   "id": "5ef898c1",
    "metadata": {
     "editable": true
    },
@@ -3422,7 +3422,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b08e5c3f",
+   "id": "6270da59",
    "metadata": {
     "editable": true
    },
@@ -3434,7 +3434,7 @@
   {
    "cell_type": "code",
    "execution_count": 27,
-   "id": "3b5864e7",
+   "id": "a1b6612b",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3450,7 +3450,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f96254bd",
+   "id": "b88aea50",
    "metadata": {
     "editable": true
    },
@@ -3461,7 +3461,7 @@
   {
    "cell_type": "code",
    "execution_count": 28,
-   "id": "784aee0c",
+   "id": "8f40fe64",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3479,7 +3479,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d5616e2a",
+   "id": "b86e3fa0",
    "metadata": {
     "editable": true
    },
@@ -3490,7 +3490,7 @@
   {
    "cell_type": "code",
    "execution_count": 29,
-   "id": "8d91d13b",
+   "id": "1f8b1d5a",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3504,7 +3504,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f81c3406",
+   "id": "222c84bd",
    "metadata": {
     "editable": true
    },
@@ -3515,7 +3515,7 @@
   {
    "cell_type": "code",
    "execution_count": 30,
-   "id": "434d4f3c",
+   "id": "780459dc",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3528,7 +3528,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c1bf7530",
+   "id": "53bd2e9b",
    "metadata": {
     "editable": true
    },
@@ -3539,7 +3539,7 @@
   {
    "cell_type": "code",
    "execution_count": 31,
-   "id": "7fd8c004",
+   "id": "5bac6f50",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3556,7 +3556,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a1b4260a",
+   "id": "7c03d8ad",
    "metadata": {
     "editable": true
    },
@@ -3567,7 +3567,7 @@
   {
    "cell_type": "code",
    "execution_count": 32,
-   "id": "e5303f58",
+   "id": "d32ac0d5",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3583,7 +3583,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2a9b8649",
+   "id": "9bbbf5d2",
    "metadata": {
     "editable": true
    },
@@ -3594,7 +3594,7 @@
   {
    "cell_type": "code",
    "execution_count": 33,
-   "id": "c16b948c",
+   "id": "435eabf3",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3618,7 +3618,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e16e8946",
+   "id": "1d0316d8",
    "metadata": {
     "editable": true
    },
@@ -3629,7 +3629,7 @@
   {
    "cell_type": "code",
    "execution_count": 34,
-   "id": "b9fe4c39",
+   "id": "428ca983",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3642,7 +3642,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "91b51561",
+   "id": "1f48b507",
    "metadata": {
     "editable": true
    },
@@ -3653,7 +3653,7 @@
   {
    "cell_type": "code",
    "execution_count": 35,
-   "id": "e3b9fbd9",
+   "id": "82c7f88e",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3673,7 +3673,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f4c894b3",
+   "id": "af5e67ec",
    "metadata": {
     "editable": true
    },
@@ -3684,7 +3684,7 @@
   {
    "cell_type": "code",
    "execution_count": 36,
-   "id": "b879380d",
+   "id": "2e5a49d5",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3727,7 +3727,7 @@
   {
    "cell_type": "code",
    "execution_count": 37,
-   "id": "5c646824",
+   "id": "193f2fa9",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3742,7 +3742,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fcbd54b7",
+   "id": "7fd81229",
    "metadata": {
     "editable": true
    },
@@ -3819,7 +3819,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dc5eefba",
+   "id": "7791df68",
    "metadata": {
     "editable": true
    },
@@ -3831,7 +3831,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "74092b8e",
+   "id": "7540dd3b",
    "metadata": {
     "editable": true
    },
@@ -3851,7 +3851,7 @@
   {
    "cell_type": "code",
    "execution_count": 38,
-   "id": "29a1b167",
+   "id": "3336893a",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3885,7 +3885,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae1b0689",
+   "id": "68146245",
    "metadata": {
     "editable": true
    },
@@ -3900,7 +3900,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ce25e983",
+   "id": "5f73e12a",
    "metadata": {
     "editable": true
    },
@@ -3912,7 +3912,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "48a5391f",
+   "id": "3a6f0106",
    "metadata": {
     "editable": true
    },
@@ -3922,7 +3922,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cce21123",
+   "id": "489bf6a3",
    "metadata": {
     "editable": true
    },
@@ -3946,7 +3946,7 @@
   {
    "cell_type": "code",
    "execution_count": 39,
-   "id": "6e847f61",
+   "id": "dc6f77ee",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3990,7 +3990,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0020a5f3",
+   "id": "2e94dcd3",
    "metadata": {
     "editable": true
    },
@@ -4000,7 +4000,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a226cdd9",
+   "id": "7298cf1d",
    "metadata": {
     "editable": true
    },
@@ -4069,7 +4069,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9fc89d8c",
+   "id": "10db6d60",
    "metadata": {
     "editable": true
    },
@@ -4083,7 +4083,7 @@
   {
    "cell_type": "code",
    "execution_count": 40,
-   "id": "e417626d",
+   "id": "1db38720",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4096,7 +4096,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cf385569",
+   "id": "3802c10c",
    "metadata": {
     "editable": true
    },
@@ -4110,7 +4110,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9b6db66e",
+   "id": "0e298fa7",
    "metadata": {
     "editable": true
    },
@@ -4123,7 +4123,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "91003866",
+   "id": "6e6b1642",
    "metadata": {
     "editable": true
    },
@@ -4134,7 +4134,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "341e67fe",
+   "id": "4d691079",
    "metadata": {
     "editable": true
    },
@@ -4146,7 +4146,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "486fb461",
+   "id": "aec0693e",
    "metadata": {
     "editable": true
    },
@@ -4156,7 +4156,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b6785408",
+   "id": "082dce1f",
    "metadata": {
     "editable": true
    },
@@ -4168,7 +4168,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "46c12d87",
+   "id": "980a908b",
    "metadata": {
     "editable": true
    },
@@ -4184,7 +4184,7 @@
   {
    "cell_type": "code",
    "execution_count": 41,
-   "id": "ff067689",
+   "id": "dfd1e291",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4235,7 +4235,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b7704c33",
+   "id": "2b9113a4",
    "metadata": {
     "editable": true
    },
@@ -4245,7 +4245,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1c2ac182",
+   "id": "e1b66025",
    "metadata": {
     "editable": true
    },
@@ -4291,7 +4291,7 @@
   {
    "cell_type": "code",
    "execution_count": 42,
-   "id": "dc94e107",
+   "id": "1f61fef1",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4304,7 +4304,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eb016ff4",
+   "id": "2a387540",
    "metadata": {
     "editable": true
    },
@@ -4315,7 +4315,7 @@
   {
    "cell_type": "code",
    "execution_count": 43,
-   "id": "1c3b0985",
+   "id": "d4ca9adf",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4330,7 +4330,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fd966077",
+   "id": "549fd34a",
    "metadata": {
     "editable": true
    },
@@ -4348,7 +4348,7 @@
   {
    "cell_type": "code",
    "execution_count": 44,
-   "id": "dff4770a",
+   "id": "452e7ddf",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4365,7 +4365,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9876ac7f",
+   "id": "d3b3c08c",
    "metadata": {
     "editable": true
    },
@@ -4381,7 +4381,7 @@
   {
    "cell_type": "code",
    "execution_count": 45,
-   "id": "305042b6",
+   "id": "4aad4eb0",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4425,7 +4425,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "87233cf1",
+   "id": "1224d3b5",
    "metadata": {
     "editable": true
    },
@@ -4435,7 +4435,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "41cbe8c3",
+   "id": "0f4f1cc3",
    "metadata": {
     "editable": true
    },
@@ -4446,7 +4446,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d4294280",
+   "id": "aca40014",
    "metadata": {
     "editable": true
    },
@@ -4457,7 +4457,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0f5ec60d",
+   "id": "2afa7dbc",
    "metadata": {
     "editable": true
    },
@@ -4468,7 +4468,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "650a7cf6",
+   "id": "50a555bd",
    "metadata": {
     "editable": true
    },
@@ -4491,7 +4491,7 @@
   {
    "cell_type": "code",
    "execution_count": 46,
-   "id": "57da61e9",
+   "id": "8286a35b",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4504,7 +4504,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4f6c6ed4",
+   "id": "5167b3ba",
    "metadata": {
     "editable": true
    },
@@ -4521,7 +4521,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "83164bcc",
+   "id": "cb3d13c4",
    "metadata": {
     "editable": true
    },
@@ -4534,7 +4534,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "35e2f214",
+   "id": "0001b4b7",
    "metadata": {
     "editable": true
    },
@@ -4545,7 +4545,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e7fcd4bf",
+   "id": "54d64bca",
    "metadata": {
     "editable": true
    },
@@ -4557,7 +4557,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f980f5dd",
+   "id": "a0d12cd4",
    "metadata": {
     "editable": true
    },
@@ -4567,7 +4567,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "404c4590",
+   "id": "e106067a",
    "metadata": {
     "editable": true
    },
@@ -4579,7 +4579,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "de7b647e",
+   "id": "bd211dd6",
    "metadata": {
     "editable": true
    },
@@ -4596,7 +4596,7 @@
   {
    "cell_type": "code",
    "execution_count": 47,
-   "id": "c64fe530",
+   "id": "4394cc31",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4688,7 +4688,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "687ff081",
+   "id": "27144669",
    "metadata": {
     "editable": true
    },
@@ -4698,7 +4698,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2e5938cd",
+   "id": "f8f1b0c1",
    "metadata": {
     "editable": true
    },
@@ -4720,7 +4720,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d9e99254",
+   "id": "3bb29f8e",
    "metadata": {
     "editable": true
    },
@@ -4732,7 +4732,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f6caf857",
+   "id": "1ca6a904",
    "metadata": {
     "editable": true
    },
@@ -4742,7 +4742,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "004eea35",
+   "id": "e096bb31",
    "metadata": {
     "editable": true
    },
@@ -4754,7 +4754,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fd3037ee",
+   "id": "2c4a87f8",
    "metadata": {
     "editable": true
    },
@@ -4764,7 +4764,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "53700a04",
+   "id": "9d6848d3",
    "metadata": {
     "editable": true
    },
@@ -4776,7 +4776,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3e15d191",
+   "id": "9f070f76",
    "metadata": {
     "editable": true
    },
@@ -4791,7 +4791,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "75c00aa1",
+   "id": "14885913",
    "metadata": {
     "editable": true
    },
@@ -4803,7 +4803,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "765be39e",
+   "id": "f5b4f42b",
    "metadata": {
     "editable": true
    },
@@ -4813,7 +4813,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "371bb281",
+   "id": "abca9fed",
    "metadata": {
     "editable": true
    },
@@ -4825,7 +4825,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7531bc5c",
+   "id": "a3931447",
    "metadata": {
     "editable": true
    },
@@ -4835,7 +4835,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "096367a8",
+   "id": "bff5fde9",
    "metadata": {
     "editable": true
    },
@@ -4847,7 +4847,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f2b48135",
+   "id": "cfcbbd41",
    "metadata": {
     "editable": true
    },
@@ -4857,7 +4857,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "64e7a83f",
+   "id": "f21eecf5",
    "metadata": {
     "editable": true
    },
@@ -4869,7 +4869,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7cd415f6",
+   "id": "5d1bccf8",
    "metadata": {
     "editable": true
    },
@@ -4879,7 +4879,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b1d73cf8",
+   "id": "cd0ac1aa",
    "metadata": {
     "editable": true
    },
@@ -4891,7 +4891,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "23c1b258",
+   "id": "e2689f44",
    "metadata": {
     "editable": true
    },
@@ -4901,7 +4901,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f26c3a7c",
+   "id": "3578ef22",
    "metadata": {
     "editable": true
    },
@@ -4913,7 +4913,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "73594acc",
+   "id": "1b9ae1e2",
    "metadata": {
     "editable": true
    },
@@ -4923,7 +4923,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9e2c6bff",
+   "id": "3c67b152",
    "metadata": {
     "editable": true
    },
@@ -4935,7 +4935,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f0ecd236",
+   "id": "7e4a7fe2",
    "metadata": {
     "editable": true
    },
@@ -4945,7 +4945,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9a6ec43b",
+   "id": "0c255fd7",
    "metadata": {
     "editable": true
    },
@@ -4957,7 +4957,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eb98056f",
+   "id": "f0b00e3d",
    "metadata": {
     "editable": true
    },
@@ -4967,7 +4967,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5e5ac90a",
+   "id": "7c9a7b92",
    "metadata": {
     "editable": true
    },
@@ -4979,7 +4979,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c8bdfe01",
+   "id": "a8e3e7e8",
    "metadata": {
     "editable": true
    },
@@ -4989,7 +4989,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4dacfd83",
+   "id": "007406a0",
    "metadata": {
     "editable": true
    },
@@ -5001,7 +5001,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "66ba7f8c",
+   "id": "20f80df6",
    "metadata": {
     "editable": true
    },
diff --git a/doc/LectureNotes/chapter2.ipynb b/doc/LectureNotes/chapter2.ipynb
index 880b4de0f..31a6d7135 100644
--- a/doc/LectureNotes/chapter2.ipynb
+++ b/doc/LectureNotes/chapter2.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "7d4ecb5d",
+   "id": "74d0d498",
    "metadata": {
     "editable": true
    },
@@ -13,7 +13,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1b85d5f1",
+   "id": "acd544cd",
    "metadata": {
     "editable": true
    },
@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a252fe86",
+   "id": "0acaa875",
    "metadata": {
     "editable": true
    },
@@ -37,7 +37,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bc95be71",
+   "id": "0924b488",
    "metadata": {
     "editable": true
    },
@@ -49,7 +49,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c77a2611",
+   "id": "b7307eb7",
    "metadata": {
     "editable": true
    },
@@ -61,7 +61,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d427f76c",
+   "id": "8096b1f0",
    "metadata": {
     "editable": true
    },
@@ -73,7 +73,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f0fe9791",
+   "id": "d64adaf2",
    "metadata": {
     "editable": true
    },
@@ -83,7 +83,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "76d07f64",
+   "id": "78b35483",
    "metadata": {
     "editable": true
    },
@@ -95,7 +95,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12f77a55",
+   "id": "f4a3ff68",
    "metadata": {
     "editable": true
    },
@@ -105,7 +105,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5ce0346b",
+   "id": "5704d260",
    "metadata": {
     "editable": true
    },
@@ -117,7 +117,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2aeccd66",
+   "id": "12fa775a",
    "metadata": {
     "editable": true
    },
@@ -130,7 +130,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e36d0875",
+   "id": "a694774d",
    "metadata": {
     "editable": true
    },
@@ -142,7 +142,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "48cdf012",
+   "id": "e0775433",
    "metadata": {
     "editable": true
    },
@@ -154,7 +154,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "89fb4555",
+   "id": "ec16ae7d",
    "metadata": {
     "editable": true
    },
@@ -166,7 +166,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3cce2fbf",
+   "id": "258fbf6a",
    "metadata": {
     "editable": true
    },
@@ -176,7 +176,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "623e11d4",
+   "id": "05ea3605",
    "metadata": {
     "editable": true
    },
@@ -188,7 +188,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3677f8ed",
+   "id": "db5ed2d4",
    "metadata": {
     "editable": true
    },
@@ -198,7 +198,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "65a16065",
+   "id": "e99ad8e8",
    "metadata": {
     "editable": true
    },
@@ -210,7 +210,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e4dc61d4",
+   "id": "1cc7938a",
    "metadata": {
     "editable": true
    },
@@ -220,7 +220,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6d333c04",
+   "id": "0e3fc9d5",
    "metadata": {
     "editable": true
    },
@@ -246,7 +246,7 @@
     "This is given by the **Singular Value Decomposition** (SVD) algorithm,\n",
     "perhaps the most powerful linear algebra algorithm.  The SVD provides\n",
     "a numerically stable matrix decomposition that is used in a large\n",
-    "swath oc applications and the decomposition is always stable\n",
+    "swath of applications and the decomposition is always stable\n",
     "numerically.\n",
     "\n",
     "In machine learning it plays a central role in dealing with for\n",
@@ -262,12 +262,12 @@
     "are problems with near singular or singular matrices. The column vectors of $\\boldsymbol{X}$ \n",
     "may be linearly dependent, normally referred to as super-collinearity.  \n",
     "This means that the matrix may be rank deficient and it is basically impossible to \n",
-    "to model the data using linear regression. As an example, consider the matrix"
+    "model the data using linear regression. As an example, consider the matrix"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "baba6f65",
+   "id": "6d2bc570",
    "metadata": {
     "editable": true
    },
@@ -290,7 +290,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "97aaa550",
+   "id": "fdda638f",
    "metadata": {
     "editable": true
    },
@@ -299,7 +299,7 @@
     "the first column is the row-wise sum of the other two columns. The rank (more correct,\n",
     "the column rank) of a matrix is the dimension of the space spanned by the\n",
     "column vectors. Hence, the rank of $\\mathbf{X}$ is equal to the number\n",
-    "of linearly independent columns. In this particular case the matrix has rank 2.\n",
+    "of linearly independent columns. In this particular case the matrix has rank 1.\n",
     "\n",
     "Super-collinearity of an $(n \\times p)$-dimensional design matrix $\\mathbf{X}$ implies\n",
     "that the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this"
@@ -307,7 +307,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d196128e",
+   "id": "29a3a620",
    "metadata": {
     "editable": true
    },
@@ -326,7 +326,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d9d92d89",
+   "id": "3278b038",
    "metadata": {
     "editable": true
    },
@@ -339,7 +339,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b3d65641",
+   "id": "7b66f623",
    "metadata": {
     "editable": true
    },
@@ -357,14 +357,14 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ef75ca28",
+   "id": "c97ae6c5",
    "metadata": {
     "editable": true
    },
    "source": [
     "has linearly dependent column vectors, we will not be able to compute the inverse\n",
     "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\beta_i$. \n",
-    "The estimators are only well-defined if $(\\boldsymbol{X}^{T}\\boldsymbol{X})^{-1}$ exits. \n",
+    "The estimators are only well-defined if $(\\boldsymbol{X}^{T}\\boldsymbol{X})$ can be inverted. \n",
     "This is more likely to happen when the matrix $\\boldsymbol{X}$ is high-dimensional. In this case it is likely to encounter a situation where \n",
     "the regression parameters $\\beta_i$ cannot be estimated.\n",
     "\n",
@@ -373,7 +373,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bcba41b4",
+   "id": "af0bfc59",
    "metadata": {
     "editable": true
    },
@@ -385,7 +385,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dda8ceea",
+   "id": "6469bfe8",
    "metadata": {
     "editable": true
    },
@@ -395,14 +395,14 @@
   },
   {
    "cell_type": "markdown",
-   "id": "454a5885",
+   "id": "75d45f30",
    "metadata": {
     "editable": true
    },
    "source": [
     "## Basic math of the SVD\n",
     "\n",
-    "From standard linear algebra we know that a square matrix $\\boldsymbol{X}$ can be diagonalized if and only it is \n",
+    "From standard linear algebra we know that a square matrix $\\boldsymbol{X}$ can be diagonalized if and only if it is \n",
     "a so-called [normal matrix](https://en.wikipedia.org/wiki/Normal_matrix), that is if $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times n}$\n",
     "we have $\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{X}^T\\boldsymbol{X}$ or if $\\boldsymbol{X}\\in {\\mathbb{C}}^{n\\times n}$ we have $\\boldsymbol{X}\\boldsymbol{X}^{\\dagger}=\\boldsymbol{X}^{\\dagger}\\boldsymbol{X}$.\n",
     "The matrix has then a set of eigenpairs"
@@ -410,7 +410,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "be408a56",
+   "id": "28b4fac2",
    "metadata": {
     "editable": true
    },
@@ -422,7 +422,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5f9d9612",
+   "id": "085b2de7",
    "metadata": {
     "editable": true
    },
@@ -432,7 +432,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "208d2b7b",
+   "id": "1fb42fdc",
    "metadata": {
     "editable": true
    },
@@ -444,7 +444,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "abe59381",
+   "id": "0f7b8c0d",
    "metadata": {
     "editable": true
    },
@@ -454,7 +454,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1105a012",
+   "id": "057f3ed2",
    "metadata": {
     "editable": true
    },
@@ -466,7 +466,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "be0d3499",
+   "id": "628031b9",
    "metadata": {
     "editable": true
    },
@@ -478,7 +478,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d2a7f70e",
+   "id": "3177dcfe",
    "metadata": {
     "editable": true
    },
@@ -493,7 +493,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2927bf84",
+   "id": "a392a54d",
    "metadata": {
     "editable": true
    },
@@ -514,7 +514,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f0881611",
+   "id": "3496079e",
    "metadata": {
     "editable": true
    },
@@ -526,7 +526,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2b00326b",
+   "id": "77b61c68",
    "metadata": {
     "editable": true
    },
@@ -536,7 +536,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0b928939",
+   "id": "dbb84089",
    "metadata": {
     "editable": true
    },
@@ -548,7 +548,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dd9b5714",
+   "id": "04fc9eec",
    "metadata": {
     "editable": true
    },
@@ -594,7 +594,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1524a26d",
+   "id": "4f1e3bb4",
    "metadata": {
     "editable": true
    },
@@ -605,7 +605,7 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "0c5ee0f0",
+   "id": "a39cbeb0",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -645,7 +645,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "541be9f2",
+   "id": "dca36481",
    "metadata": {
     "editable": true
    },
@@ -675,7 +675,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8088b20a",
+   "id": "b9b3c7db",
    "metadata": {
     "editable": true
    },
@@ -689,7 +689,7 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "0a1aeb6a",
+   "id": "6aa5981e",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -701,7 +701,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c94ae586",
+   "id": "ae58132b",
    "metadata": {
     "editable": true
    },
@@ -712,7 +712,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "1604fc1a",
+   "id": "761d48f5",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -739,7 +739,6 @@
     "    return np.matmul(V,np.matmul(invD,UT))\n",
     "\n",
     "\n",
-    "#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])\n",
     "# Non-singular square matrix\n",
     "X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])\n",
     "print(X)\n",
@@ -752,7 +751,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ec171bab",
+   "id": "6297c4ec",
    "metadata": {
     "editable": true
    },
@@ -765,12 +764,12 @@
     "It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse.\n",
     "It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices.\n",
     "\n",
-    "Using the SVD we can obtain the pseudoinverse of a matrix $\\boldsymbol{A}$ (labeled here as $\\boldsymbol{A}_{\\mathrm{PI}}$"
+    "Using the SVD we can obtain the pseudoinverse (PI) of a matrix $\\boldsymbol{A}$ (labeled here as $\\boldsymbol{A}_{\\mathrm{PI}}$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "28d8e04d",
+   "id": "2bb1f4b8",
    "metadata": {
     "editable": true
    },
@@ -782,7 +781,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d61db81e",
+   "id": "832a6404",
    "metadata": {
     "editable": true
    },
@@ -793,7 +792,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "42ee64fb",
+   "id": "d393c8c7",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -827,7 +826,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "de54283f",
+   "id": "3d0570c2",
    "metadata": {
     "editable": true
    },
@@ -837,7 +836,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12dcb1c3",
+   "id": "071ff3b6",
    "metadata": {
     "editable": true
    },
@@ -851,7 +850,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e9df537",
+   "id": "113247f9",
    "metadata": {
     "editable": true
    },
@@ -870,7 +869,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4ca18cc5",
+   "id": "05603c38",
    "metadata": {
     "editable": true
    },
@@ -880,7 +879,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fb195f64",
+   "id": "8848e319",
    "metadata": {
     "editable": true
    },
@@ -892,21 +891,21 @@
   },
   {
    "cell_type": "markdown",
-   "id": "76f68d8d",
+   "id": "5b4146e5",
    "metadata": {
     "editable": true
    },
    "source": [
     "where $\\boldsymbol{U}$ is an orthogonal matrix of dimension $n\\times n$, meaning that $\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{U}^T\\boldsymbol{U}=\\boldsymbol{I}_n$. Here $\\boldsymbol{I}_n$ is the unit matrix of dimension $n \\times n$.\n",
     "\n",
-    "Similarly, $\\boldsymbol{V}$ is an orthogonal matrix of dimension $p\\times p$, meaning that $\\boldsymbol{V}\\boldsymbol{V}^T=\\boldsymbol{V}^T\\boldsymbol{V}=\\boldsymbol{I}_p$. Here $\\boldsymbol{I}_n$ is the unit matrix of dimension $p \\times p$.\n",
+    "Similarly, $\\boldsymbol{V}$ is an orthogonal matrix of dimension $p\\times p$, meaning that $\\boldsymbol{V}\\boldsymbol{V}^T=\\boldsymbol{V}^T\\boldsymbol{V}=\\boldsymbol{I}_p$. Here $\\boldsymbol{I}_p$ is the unit matrix of dimension $p \\times p$.\n",
     "\n",
     "Finally $\\boldsymbol{\\Sigma}$ contains the singular values $\\sigma_i$. This matrix has dimension $n\\times p$ and the singular values $\\sigma_i$ are all positive. The non-zero values are ordered in descending order, that is"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "258a1c95",
+   "id": "43701d21",
    "metadata": {
     "editable": true
    },
@@ -918,7 +917,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1e5d87a5",
+   "id": "7cdca00d",
    "metadata": {
     "editable": true
    },
@@ -930,7 +929,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "640e515f",
+   "id": "add83821",
    "metadata": {
     "editable": true
    },
@@ -947,7 +946,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0f289cf7",
+   "id": "3746dd8d",
    "metadata": {
     "editable": true
    },
@@ -957,7 +956,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c9ffceed",
+   "id": "ae591c49",
    "metadata": {
     "editable": true
    },
@@ -973,7 +972,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a99d261a",
+   "id": "63009724",
    "metadata": {
     "editable": true
    },
@@ -983,7 +982,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e3819654",
+   "id": "c674e8da",
    "metadata": {
     "editable": true
    },
@@ -999,7 +998,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c1429159",
+   "id": "9f60659a",
    "metadata": {
     "editable": true
    },
@@ -1009,7 +1008,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2b0a9b45",
+   "id": "27e17584",
    "metadata": {
     "editable": true
    },
@@ -1025,7 +1024,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f48b9161",
+   "id": "3efc62c4",
    "metadata": {
     "editable": true
    },
@@ -1035,7 +1034,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9441e20d",
+   "id": "195d30c4",
    "metadata": {
     "editable": true
    },
@@ -1052,7 +1051,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "76ff02e6",
+   "id": "a00d95e8",
    "metadata": {
     "editable": true
    },
@@ -1066,7 +1065,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3906a87a",
+   "id": "985428ec",
    "metadata": {
     "editable": true
    },
@@ -1078,7 +1077,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d711a76a",
+   "id": "0affd014",
    "metadata": {
     "editable": true
    },
@@ -1088,7 +1087,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2749322c",
+   "id": "146dcd49",
    "metadata": {
     "editable": true
    },
@@ -1100,7 +1099,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b4f29652",
+   "id": "63bc8186",
    "metadata": {
     "editable": true
    },
@@ -1112,7 +1111,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e2c0284a",
+   "id": "4bea8a7d",
    "metadata": {
     "editable": true
    },
@@ -1124,7 +1123,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4f3c3e78",
+   "id": "5321e8fc",
    "metadata": {
     "editable": true
    },
@@ -1134,7 +1133,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "de70ea98",
+   "id": "c16d0109",
    "metadata": {
     "editable": true
    },
@@ -1146,7 +1145,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "87fe93a3",
+   "id": "4ca7bc79",
    "metadata": {
     "editable": true
    },
@@ -1156,7 +1155,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "789c8ec6",
+   "id": "795969aa",
    "metadata": {
     "editable": true
    },
@@ -1168,7 +1167,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "16964421",
+   "id": "36fd11ec",
    "metadata": {
     "editable": true
    },
@@ -1178,7 +1177,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6650d68a",
+   "id": "8b59e361",
    "metadata": {
     "editable": true
    },
@@ -1190,7 +1189,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5cf19771",
+   "id": "4212728f",
    "metadata": {
     "editable": true
    },
@@ -1202,7 +1201,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e384c5b",
+   "id": "8d6940eb",
    "metadata": {
     "editable": true
    },
@@ -1212,7 +1211,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e12b5364",
+   "id": "eabfdb9c",
    "metadata": {
     "editable": true
    },
@@ -1224,7 +1223,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3be11012",
+   "id": "bbc85eab",
    "metadata": {
     "editable": true
    },
@@ -1235,7 +1234,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "80c0acc6",
+   "id": "f5b1bf52",
    "metadata": {
     "editable": true
    },
@@ -1247,7 +1246,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "56d3e48e",
+   "id": "efcc256a",
    "metadata": {
     "editable": true
    },
@@ -1257,7 +1256,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f4622cb0",
+   "id": "0650a070",
    "metadata": {
     "editable": true
    },
@@ -1269,7 +1268,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "951d407b",
+   "id": "c168d425",
    "metadata": {
     "editable": true
    },
@@ -1279,7 +1278,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5afab1bf",
+   "id": "30fb2230",
    "metadata": {
     "editable": true
    },
@@ -1291,7 +1290,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4faa04b1",
+   "id": "d46b1fd5",
    "metadata": {
     "editable": true
    },
@@ -1302,7 +1301,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dddd3a83",
+   "id": "16a93afd",
    "metadata": {
     "editable": true
    },
@@ -1314,7 +1313,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "db2c339c",
+   "id": "80e47710",
    "metadata": {
     "editable": true
    },
@@ -1332,7 +1331,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8ab3d49f",
+   "id": "1dc0f98b",
    "metadata": {
     "editable": true
    },
@@ -1348,7 +1347,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e57f948b",
+   "id": "c5f9ce8b",
    "metadata": {
     "editable": true
    },
@@ -1360,19 +1359,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6312fe8c",
+   "id": "c803fdb7",
    "metadata": {
     "editable": true
    },
    "source": [
-    "This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).\n",
+    "This quantity defines  what is called the Hessian matrix (the second derivative of the cost function we want to optimize).\n",
     "\n",
     "The Hessian matrix plays an important role and is defined in this course as"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "2cfe093b",
+   "id": "eef3c89e",
    "metadata": {
     "editable": true
    },
@@ -1384,7 +1383,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3d95fdcc",
+   "id": "e886f303",
    "metadata": {
     "editable": true
    },
@@ -1403,7 +1402,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e0dfeff9",
+   "id": "efcb2b9e",
    "metadata": {
     "editable": true
    },
@@ -1417,7 +1416,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5f89d8a5",
+   "id": "caf380d7",
    "metadata": {
     "editable": true
    },
@@ -1427,7 +1426,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c38e45e0",
+   "id": "9d676665",
    "metadata": {
     "editable": true
    },
@@ -1439,7 +1438,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "358b23d9",
+   "id": "301b0d53",
    "metadata": {
     "editable": true
    },
@@ -1449,7 +1448,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1b8ed042",
+   "id": "7c134c39",
    "metadata": {
     "editable": true
    },
@@ -1461,7 +1460,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "83057d1c",
+   "id": "12aec852",
    "metadata": {
     "editable": true
    },
@@ -1471,7 +1470,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ba3bed27",
+   "id": "54a5de6b",
    "metadata": {
     "editable": true
    },
@@ -1485,7 +1484,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7efd03ba",
+   "id": "3edc1dd1",
    "metadata": {
     "editable": true
    },
@@ -1508,7 +1507,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "70f28bbc",
+   "id": "9d2d7b4e",
    "metadata": {
     "editable": true
    },
@@ -1520,7 +1519,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "81605545",
+   "id": "dcee4258",
    "metadata": {
     "editable": true
    },
@@ -1533,7 +1532,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "74018ba5",
+   "id": "25100476",
    "metadata": {
     "editable": true
    },
@@ -1547,7 +1546,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5b002e32",
+   "id": "5c46fdef",
    "metadata": {
     "editable": true
    },
@@ -1560,7 +1559,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c1e095b5",
+   "id": "dcfa6d43",
    "metadata": {
     "editable": true
    },
@@ -1579,7 +1578,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ddf9f672",
+   "id": "89dd5e4e",
    "metadata": {
     "editable": true
    },
@@ -1591,7 +1590,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7899034b",
+   "id": "7edacd6c",
    "metadata": {
     "editable": true
    },
@@ -1603,7 +1602,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d675f83c",
+   "id": "94e8929f",
    "metadata": {
     "editable": true
    },
@@ -1613,7 +1612,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0724af92",
+   "id": "3f10861a",
    "metadata": {
     "editable": true
    },
@@ -1625,20 +1624,20 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e85a73eb",
+   "id": "173fbc84",
    "metadata": {
     "editable": true
    },
    "source": [
     "With these definitions, we can now rewrite our $2\\times 2$\n",
-    "correlation/covariance matrix in terms of a moe general design/feature\n",
+    "correlation/covariance matrix in terms of a more general design/feature\n",
     "matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$. This leads to a $p\\times p$\n",
     "covariance matrix for the vectors $\\boldsymbol{x}_i$ with $i=0,1,\\dots,p-1$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "21d58ff5",
+   "id": "0913dc51",
    "metadata": {
     "editable": true
    },
@@ -1657,7 +1656,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "997fe46a",
+   "id": "70346587",
    "metadata": {
     "editable": true
    },
@@ -1667,7 +1666,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "618f4836",
+   "id": "86feec9f",
    "metadata": {
     "editable": true
    },
@@ -1686,7 +1685,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1154e7fa",
+   "id": "25ed76e9",
    "metadata": {
     "editable": true
    },
@@ -1702,7 +1701,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1d40c592",
+   "id": "cea26675",
    "metadata": {
     "editable": true
    },
@@ -1716,7 +1715,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "43b93642",
+   "id": "1fa1423f",
    "metadata": {
     "editable": true
    },
@@ -1731,7 +1730,7 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "e422a526",
+   "id": "f277b044",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1752,7 +1751,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "456786dc",
+   "id": "02205795",
    "metadata": {
     "editable": true
    },
@@ -1767,7 +1766,7 @@
   {
    "cell_type": "code",
    "execution_count": 6,
-   "id": "aee5e3f0",
+   "id": "6c182a7d",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1799,7 +1798,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "333e1d75",
+   "id": "16ff9454",
    "metadata": {
     "editable": true
    },
@@ -1810,13 +1809,13 @@
     "\n",
     "The above procedure with **numpy** can be made more compact if we use **pandas**.\n",
     "\n",
-    "We whow here how we can set up the correlation matrix using **pandas**, as done in this simple code"
+    "We know here how we can set up the correlation matrix using **pandas**, as done in this simple code"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 7,
-   "id": "cbb245b4",
+   "id": "903635fb",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1841,7 +1840,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "598beae1",
+   "id": "034c38ef",
    "metadata": {
     "editable": true
    },
@@ -1852,7 +1851,7 @@
   {
    "cell_type": "code",
    "execution_count": 8,
-   "id": "5bb6727b",
+   "id": "91afb8cb",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1906,7 +1905,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6841e847",
+   "id": "9d8768f8",
    "metadata": {
     "editable": true
    },
@@ -1925,7 +1924,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "494c9f8f",
+   "id": "a7de38a0",
    "metadata": {
     "editable": true
    },
@@ -1937,7 +1936,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "28031e99",
+   "id": "4855248a",
    "metadata": {
     "editable": true
    },
@@ -1947,7 +1946,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f9e03888",
+   "id": "3a150f73",
    "metadata": {
     "editable": true
    },
@@ -1964,7 +1963,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f50a073c",
+   "id": "6ab3f1f7",
    "metadata": {
     "editable": true
    },
@@ -1974,7 +1973,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e8332df1",
+   "id": "ae35f475",
    "metadata": {
     "editable": true
    },
@@ -1989,7 +1988,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "40e85f2e",
+   "id": "9f6b4b6b",
    "metadata": {
     "editable": true
    },
@@ -1999,7 +1998,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b38921ca",
+   "id": "af0c59a3",
    "metadata": {
     "editable": true
    },
@@ -2013,7 +2012,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6cb1f379",
+   "id": "824ae63b",
    "metadata": {
     "editable": true
    },
@@ -2027,7 +2026,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6f71019b",
+   "id": "5b480160",
    "metadata": {
     "editable": true
    },
@@ -2039,7 +2038,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae0b0c45",
+   "id": "506a98da",
    "metadata": {
     "editable": true
    },
@@ -2051,7 +2050,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "89acbca7",
+   "id": "72b1e665",
    "metadata": {
     "editable": true
    },
@@ -2061,7 +2060,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9e52517d",
+   "id": "6ea9123f",
    "metadata": {
     "editable": true
    },
@@ -2073,7 +2072,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5dd4bae9",
+   "id": "ab471475",
    "metadata": {
     "editable": true
    },
@@ -2083,7 +2082,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d0c48d0a",
+   "id": "ff364443",
    "metadata": {
     "editable": true
    },
@@ -2100,7 +2099,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "238fd7dc",
+   "id": "4619ec8f",
    "metadata": {
     "editable": true
    },
@@ -2110,7 +2109,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5b30b7ea",
+   "id": "529d5ed0",
    "metadata": {
     "editable": true
    },
@@ -2122,7 +2121,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "38a1ee92",
+   "id": "762e07ea",
    "metadata": {
     "editable": true
    },
@@ -2132,7 +2131,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fb441672",
+   "id": "f45c2ce1",
    "metadata": {
     "editable": true
    },
@@ -2144,7 +2143,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "20d9a077",
+   "id": "35835939",
    "metadata": {
     "editable": true
    },
@@ -2156,7 +2155,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7b4732c4",
+   "id": "901c3505",
    "metadata": {
     "editable": true
    },
@@ -2168,7 +2167,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d6de6e76",
+   "id": "63486657",
    "metadata": {
     "editable": true
    },
@@ -2190,7 +2189,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b85d60d8",
+   "id": "b8cb7b04",
    "metadata": {
     "editable": true
    },
@@ -2202,7 +2201,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "681abce5",
+   "id": "8962eeb3",
    "metadata": {
     "editable": true
    },
@@ -2219,7 +2218,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a0061112",
+   "id": "ff83bf5a",
    "metadata": {
     "editable": true
    },
@@ -2231,7 +2230,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "67af75b4",
+   "id": "67c405ee",
    "metadata": {
     "editable": true
    },
@@ -2241,7 +2240,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "14e03bab",
+   "id": "d2279b06",
    "metadata": {
     "editable": true
    },
@@ -2253,7 +2252,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ef2fc352",
+   "id": "c08bd913",
    "metadata": {
     "editable": true
    },
@@ -2263,7 +2262,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cb03894b",
+   "id": "585fcec4",
    "metadata": {
     "editable": true
    },
@@ -2275,7 +2274,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "322818b0",
+   "id": "c5ee7c18",
    "metadata": {
     "editable": true
    },
@@ -2285,7 +2284,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "10f1ad94",
+   "id": "250d6a7b",
    "metadata": {
     "editable": true
    },
@@ -2297,7 +2296,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "17865864",
+   "id": "26dca3eb",
    "metadata": {
     "editable": true
    },
@@ -2314,7 +2313,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "30dddf0b",
+   "id": "43ae52cf",
    "metadata": {
     "editable": true
    },
@@ -2327,7 +2326,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8633ef50",
+   "id": "fa5c2da2",
    "metadata": {
     "editable": true
    },
@@ -2339,7 +2338,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a7ec868f",
+   "id": "ef125654",
    "metadata": {
     "editable": true
    },
@@ -2349,7 +2348,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "772f70dc",
+   "id": "3e398c3d",
    "metadata": {
     "editable": true
    },
@@ -2362,7 +2361,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eda842c9",
+   "id": "6e4285d1",
    "metadata": {
     "editable": true
    },
@@ -2372,7 +2371,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4f4cbc79",
+   "id": "c1d8fde6",
    "metadata": {
     "editable": true
    },
@@ -2384,7 +2383,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c960bc44",
+   "id": "5b082cd2",
    "metadata": {
     "editable": true
    },
@@ -2397,7 +2396,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1d5e2196",
+   "id": "306b6062",
    "metadata": {
     "editable": true
    },
@@ -2410,7 +2409,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "000880d6",
+   "id": "f9c22f99",
    "metadata": {
     "editable": true
    },
@@ -2422,7 +2421,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a9d1e159",
+   "id": "8fa998f9",
    "metadata": {
     "editable": true
    },
@@ -2434,7 +2433,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6882c9cf",
+   "id": "954b3b63",
    "metadata": {
     "editable": true
    },
@@ -2444,7 +2443,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a21757d3",
+   "id": "40d226a6",
    "metadata": {
     "editable": true
    },
@@ -2457,7 +2456,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f072b07c",
+   "id": "571654e9",
    "metadata": {
     "editable": true
    },
@@ -2469,7 +2468,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "733a6413",
+   "id": "4b96f90f",
    "metadata": {
     "editable": true
    },
@@ -2481,7 +2480,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c1fe1a2c",
+   "id": "6af80b33",
    "metadata": {
     "editable": true
    },
@@ -2491,7 +2490,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d39258b1",
+   "id": "ed4e7687",
    "metadata": {
     "editable": true
    },
@@ -2503,7 +2502,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "59c47ad9",
+   "id": "2eb319cc",
    "metadata": {
     "editable": true
    },
@@ -2517,7 +2516,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eb60430f",
+   "id": "8528934c",
    "metadata": {
     "editable": true
    },
@@ -2529,7 +2528,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "265d31a8",
+   "id": "22a7e810",
    "metadata": {
     "editable": true
    },
@@ -2539,7 +2538,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8ab4a19c",
+   "id": "01d9c1c9",
    "metadata": {
     "editable": true
    },
@@ -2551,7 +2550,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e1cf6baf",
+   "id": "5d910fd1",
    "metadata": {
     "editable": true
    },
@@ -2563,7 +2562,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0769ccf1",
+   "id": "b8bbfa39",
    "metadata": {
     "editable": true
    },
@@ -2575,7 +2574,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b2efb7e5",
+   "id": "51ac6ef1",
    "metadata": {
     "editable": true
    },
@@ -2594,7 +2593,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2fac59a0",
+   "id": "12ee6646",
    "metadata": {
     "editable": true
    },
@@ -2606,7 +2605,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6783041f",
+   "id": "22852ccf",
    "metadata": {
     "editable": true
    },
@@ -2616,7 +2615,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cb5777d1",
+   "id": "68d87eac",
    "metadata": {
     "editable": true
    },
@@ -2628,7 +2627,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4b124b41",
+   "id": "0219a1a8",
    "metadata": {
     "editable": true
    },
@@ -2640,7 +2639,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "379a857f",
+   "id": "be1f339c",
    "metadata": {
     "editable": true
    },
@@ -2652,7 +2651,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2125cb49",
+   "id": "7e1d5221",
    "metadata": {
     "editable": true
    },
@@ -2670,7 +2669,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f40d87ac",
+   "id": "0b4bc87c",
    "metadata": {
     "editable": true
    },
@@ -2682,7 +2681,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "04c2c26c",
+   "id": "de7e0986",
    "metadata": {
     "editable": true
    },
@@ -2692,7 +2691,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8d6fe816",
+   "id": "0ae7dd26",
    "metadata": {
     "editable": true
    },
@@ -2704,7 +2703,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "77db7bc0",
+   "id": "5caa1086",
    "metadata": {
     "editable": true
    },
@@ -2714,7 +2713,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "77ebce0a",
+   "id": "c737b14f",
    "metadata": {
     "editable": true
    },
@@ -2726,7 +2725,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "30e3d7f9",
+   "id": "60570cf0",
    "metadata": {
     "editable": true
    },
@@ -2742,7 +2741,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b76bf6d6",
+   "id": "f0c2e386",
    "metadata": {
     "editable": true
    },
@@ -2754,7 +2753,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ec68ccb3",
+   "id": "5f3e91d2",
    "metadata": {
     "editable": true
    },
@@ -2764,7 +2763,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7849b038",
+   "id": "66b637e8",
    "metadata": {
     "editable": true
    },
@@ -2776,7 +2775,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e2596a14",
+   "id": "4082c969",
    "metadata": {
     "editable": true
    },
@@ -2786,7 +2785,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5912b05e",
+   "id": "2c0b085b",
    "metadata": {
     "editable": true
    },
@@ -2798,7 +2797,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5a85cd53",
+   "id": "bf355533",
    "metadata": {
     "editable": true
    },
@@ -2808,7 +2807,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "048012bd",
+   "id": "928d6cb3",
    "metadata": {
     "editable": true
    },
@@ -2820,7 +2819,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "efef81d5",
+   "id": "39a40fd9",
    "metadata": {
     "editable": true
    },
@@ -2830,12 +2829,12 @@
     "Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the\n",
     "diagonal. In this case we have an equal number of rows and columns $n=p$.\n",
     "\n",
-    "Our model approximation is just $\\tilde{\\boldsymbol{y}}=\\boldsymbol{\\beta}$ and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term $1/n$)"
+    "Our model approximation is just $\\tilde{\\boldsymbol{y}}=\\boldsymbol{\\beta}$ and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term $1/n$)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "54ca6b87",
+   "id": "5d891e5c",
    "metadata": {
     "editable": true
    },
@@ -2847,7 +2846,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1e2ef9fc",
+   "id": "f9ea4d16",
    "metadata": {
     "editable": true
    },
@@ -2857,7 +2856,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "87f035f4",
+   "id": "59b56446",
    "metadata": {
     "editable": true
    },
@@ -2869,7 +2868,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f9ebafc4",
+   "id": "b178aedb",
    "metadata": {
     "editable": true
    },
@@ -2879,7 +2878,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ff097451",
+   "id": "3c7c60e4",
    "metadata": {
     "editable": true
    },
@@ -2891,7 +2890,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6e5a4735",
+   "id": "2f5d3c17",
    "metadata": {
     "editable": true
    },
@@ -2901,7 +2900,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2b787e5e",
+   "id": "2a612078",
    "metadata": {
     "editable": true
    },
@@ -2913,7 +2912,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3349660d",
+   "id": "dfb11d07",
    "metadata": {
     "editable": true
    },
@@ -2923,7 +2922,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "da17bea5",
+   "id": "9331f4a6",
    "metadata": {
     "editable": true
    },
@@ -2935,7 +2934,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ba3eb166",
+   "id": "b4661ad6",
    "metadata": {
     "editable": true
    },
@@ -2945,7 +2944,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5003889b",
+   "id": "60adb333",
    "metadata": {
     "editable": true
    },
@@ -2957,7 +2956,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8c9da994",
+   "id": "d5fa48d0",
    "metadata": {
     "editable": true
    },
@@ -2967,7 +2966,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "171cb28c",
+   "id": "bee441a6",
    "metadata": {
     "editable": true
    },
@@ -2981,20 +2980,20 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5d749681",
+   "id": "dc6929bb",
    "metadata": {
     "editable": true
    },
    "source": [
     "Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$.\n",
     "\n",
-    "As another examples, \n",
+    "As another example, \n",
     "let us assume we have a data set with outputs/targets given by the vector"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "ec36a482",
+   "id": "954061f9",
    "metadata": {
     "editable": true
    },
@@ -3006,7 +3005,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "153e6fdb",
+   "id": "97efa82b",
    "metadata": {
     "editable": true
    },
@@ -3016,7 +3015,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2dbc8ef3",
+   "id": "f2ed5f9b",
    "metadata": {
     "editable": true
    },
@@ -3028,7 +3027,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c5077dc7",
+   "id": "1c424fc1",
    "metadata": {
     "editable": true
    },
@@ -3040,7 +3039,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cb8e4e64",
+   "id": "95467447",
    "metadata": {
     "editable": true
    },
@@ -3052,7 +3051,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b7761fa1",
+   "id": "5da0739a",
    "metadata": {
     "editable": true
    },
@@ -3062,7 +3061,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12478361",
+   "id": "47e7e72f",
    "metadata": {
     "editable": true
    },
@@ -3074,7 +3073,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "49d1c0e3",
+   "id": "974570c8",
    "metadata": {
     "editable": true
    },
@@ -3086,7 +3085,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bbf2a7e1",
+   "id": "dd9e2341",
    "metadata": {
     "editable": true
    },
@@ -3098,7 +3097,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5465e48e",
+   "id": "70f08735",
    "metadata": {
     "editable": true
    },
@@ -3108,7 +3107,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6ae8cea7",
+   "id": "c0a08b76",
    "metadata": {
     "editable": true
    },
@@ -3120,7 +3119,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "073f7084",
+   "id": "1d8ed72a",
    "metadata": {
     "editable": true
    },
@@ -3135,7 +3134,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9f700c62",
+   "id": "9d4ef6fd",
    "metadata": {
     "editable": true
    },
@@ -3147,7 +3146,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dc606935",
+   "id": "3ba1fd07",
    "metadata": {
     "editable": true
    },
@@ -3159,7 +3158,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9f027657",
+   "id": "2c6a0484",
    "metadata": {
     "editable": true
    },
@@ -3169,7 +3168,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "73ab89da",
+   "id": "7f86e745",
    "metadata": {
     "editable": true
    },
@@ -3181,7 +3180,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d90641d4",
+   "id": "46c8a1af",
    "metadata": {
     "editable": true
    },
@@ -3191,7 +3190,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "783e4acf",
+   "id": "33d6b311",
    "metadata": {
     "editable": true
    },
@@ -3203,7 +3202,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "54e34208",
+   "id": "1e4eae3a",
    "metadata": {
     "editable": true
    },
@@ -3213,7 +3212,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e1d51c0f",
+   "id": "7a68a399",
    "metadata": {
     "editable": true
    },
@@ -3225,7 +3224,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "df26eb7d",
+   "id": "89874fbd",
    "metadata": {
     "editable": true
    },
@@ -3238,7 +3237,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "52773a3b",
+   "id": "a849fabc",
    "metadata": {
     "editable": true
    },
@@ -3250,7 +3249,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7147420f",
+   "id": "445f0b08",
    "metadata": {
     "editable": true
    },
@@ -3262,7 +3261,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0148b8ed",
+   "id": "669385c5",
    "metadata": {
     "editable": true
    },
@@ -3272,7 +3271,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a289ee68",
+   "id": "d5d14c30",
    "metadata": {
     "editable": true
    },
@@ -3284,7 +3283,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a134de53",
+   "id": "1db05ce9",
    "metadata": {
     "editable": true
    },
@@ -3303,7 +3302,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "42525db3",
+   "id": "48598bde",
    "metadata": {
     "editable": true
    },
@@ -3315,7 +3314,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8cb8693f",
+   "id": "458cc863",
    "metadata": {
     "editable": true
    },
@@ -3325,7 +3324,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "941c059c",
+   "id": "ff948eca",
    "metadata": {
     "editable": true
    },
@@ -3337,7 +3336,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c9505d7b",
+   "id": "38dfba54",
    "metadata": {
     "editable": true
    },
@@ -3347,7 +3346,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "706d9352",
+   "id": "bc715180",
    "metadata": {
     "editable": true
    },
@@ -3359,7 +3358,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a36f3223",
+   "id": "62e9a17c",
    "metadata": {
     "editable": true
    },
@@ -3369,7 +3368,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a16a4703",
+   "id": "e564b775",
    "metadata": {
     "editable": true
    },
@@ -3381,7 +3380,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "62e13728",
+   "id": "2bdbcac1",
    "metadata": {
     "editable": true
    },
@@ -3397,7 +3396,7 @@
   {
    "cell_type": "code",
    "execution_count": 9,
-   "id": "4b9e0425",
+   "id": "a1e3dee1",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3459,24 +3458,24 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7ad7e409",
+   "id": "5973148f",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\\boldsymbol{\\beta}$, we that they are getting smaller and smaller and our error stabilizes since the predicted values of $\\tilde{\\boldsymbol{y}}$ approach zero.\n",
+    "We see here that we reach a plateau for the Ridge results. Writing out the coefficients $\\boldsymbol{\\beta}$, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of $\\tilde{\\boldsymbol{y}}$ approach zero.\n",
     "\n",
     "This happens also for Lasso regression, as seen from the next code\n",
     "output. The difference is that Lasso shrinks the values of $\\beta$ to\n",
     "zero at a much earlier stage and the results flatten out. We see that\n",
     "Lasso gives also an excellent fit for small values of $\\lambda$ and\n",
-    "shows rthe best performance of the three regression methods."
+    "shows the best performance of the three regression methods."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 10,
-   "id": "5618fe91",
+   "id": "a961f69c",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3543,7 +3542,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d11661fb",
+   "id": "60cfd641",
    "metadata": {
     "editable": true
    },
@@ -3562,7 +3561,7 @@
   {
    "cell_type": "code",
    "execution_count": 11,
-   "id": "22f6ce96",
+   "id": "171876b3",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3651,7 +3650,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2ab56f06",
+   "id": "947928e7",
    "metadata": {
     "editable": true
    },
@@ -3666,7 +3665,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "86c7b9aa",
+   "id": "9559d0a8",
    "metadata": {
     "editable": true
    },
@@ -3695,7 +3694,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1a8cf62c",
+   "id": "6810eb7d",
    "metadata": {
     "editable": true
    },
@@ -3711,7 +3710,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "03c65270",
+   "id": "bd997167",
    "metadata": {
     "editable": true
    },
@@ -3734,7 +3733,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "423800c4",
+   "id": "4ff740b9",
    "metadata": {
     "editable": true
    },
@@ -3746,7 +3745,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "54abfaf4",
+   "id": "fe0b2250",
    "metadata": {
     "editable": true
    },
@@ -3757,7 +3756,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "630d72b6",
+   "id": "87e6b9b3",
    "metadata": {
     "editable": true
    },
@@ -3769,7 +3768,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5de630a0",
+   "id": "68d0de57",
    "metadata": {
     "editable": true
    },
@@ -3779,7 +3778,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "aa85b4cd",
+   "id": "2f239890",
    "metadata": {
     "editable": true
    },
@@ -3795,7 +3794,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2f903151",
+   "id": "2a724679",
    "metadata": {
     "editable": true
    },
@@ -3806,7 +3805,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a9378db4",
+   "id": "2d710e45",
    "metadata": {
     "editable": true
    },
@@ -3829,7 +3828,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5eeef81e",
+   "id": "488a73d8",
    "metadata": {
     "editable": true
    },
@@ -3842,7 +3841,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e3aa6792",
+   "id": "9b1dca9a",
    "metadata": {
     "editable": true
    },
@@ -3854,7 +3853,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d9f4d5aa",
+   "id": "07089a59",
    "metadata": {
     "editable": true
    },
@@ -3868,7 +3867,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "52725327",
+   "id": "690bd104",
    "metadata": {
     "editable": true
    },
@@ -3899,7 +3898,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ae6de7ff",
+   "id": "6a9132ce",
    "metadata": {
     "editable": true
    },
@@ -3921,7 +3920,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3b7642a3",
+   "id": "68cce775",
    "metadata": {
     "editable": true
    },
@@ -3933,7 +3932,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6519d923",
+   "id": "a9c3f89a",
    "metadata": {
     "editable": true
    },
@@ -3946,7 +3945,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d5c90c4b",
+   "id": "f9e2f9d7",
    "metadata": {
     "editable": true
    },
@@ -3958,7 +3957,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "32d45fbe",
+   "id": "58443fe8",
    "metadata": {
     "editable": true
    },
@@ -3970,7 +3969,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e7e08c2",
+   "id": "cc34c059",
    "metadata": {
     "editable": true
    },
@@ -3982,7 +3981,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1f3b3e08",
+   "id": "6ad9c8e3",
    "metadata": {
     "editable": true
    },
@@ -3994,7 +3993,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8c4539a9",
+   "id": "7c09657d",
    "metadata": {
     "editable": true
    },
@@ -4017,7 +4016,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6e53e7aa",
+   "id": "abe9915b",
    "metadata": {
     "editable": true
    },
@@ -4029,7 +4028,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0a8f5e0e",
+   "id": "326e0c75",
    "metadata": {
     "editable": true
    },
@@ -4040,7 +4039,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "de7e332e",
+   "id": "567fb1b1",
    "metadata": {
     "editable": true
    },
@@ -4052,19 +4051,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e4873c9c",
+   "id": "107abe1c",
    "metadata": {
     "editable": true
    },
    "source": [
     "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\beta}$.\n",
     "\n",
-    "Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event $\\boldsymbol{y}$ as the product of the single events, that is we have"
+    "Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event $\\boldsymbol{y}$ as the product of the single events, that is we have"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7bd0f3cb",
+   "id": "f11ddf78",
    "metadata": {
     "editable": true
    },
@@ -4076,7 +4075,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "181adb8d",
+   "id": "2abd6e3b",
    "metadata": {
     "editable": true
    },
@@ -4087,7 +4086,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "90fc5963",
+   "id": "caddb652",
    "metadata": {
     "editable": true
    },
@@ -4099,7 +4098,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ee0229b0",
+   "id": "291e1dd6",
    "metadata": {
     "editable": true
    },
@@ -4110,7 +4109,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "543c8ca5",
+   "id": "73ac95c1",
    "metadata": {
     "editable": true
    },
@@ -4122,7 +4121,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "38881820",
+   "id": "ce49493b",
    "metadata": {
     "editable": true
    },
@@ -4157,7 +4156,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0f1fea5a",
+   "id": "1ff54861",
    "metadata": {
     "editable": true
    },
@@ -4169,7 +4168,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4c07b706",
+   "id": "e8cdd425",
    "metadata": {
     "editable": true
    },
@@ -4179,7 +4178,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "20a2d9f1",
+   "id": "95d54be7",
    "metadata": {
     "editable": true
    },
@@ -4191,7 +4190,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d65c8233",
+   "id": "731e3e2a",
    "metadata": {
     "editable": true
    },
@@ -4201,7 +4200,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "251536f6",
+   "id": "8c40a24c",
    "metadata": {
     "editable": true
    },
@@ -4213,7 +4212,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c24acd35",
+   "id": "1e298a02",
    "metadata": {
     "editable": true
    },
@@ -4223,7 +4222,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2594074f",
+   "id": "fd5c3e4f",
    "metadata": {
     "editable": true
    },
@@ -4235,7 +4234,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bb7f7b33",
+   "id": "d3aab131",
    "metadata": {
     "editable": true
    },
@@ -4255,7 +4254,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "84b10307",
+   "id": "64646b7c",
    "metadata": {
     "editable": true
    },
@@ -4267,7 +4266,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2ed6eff3",
+   "id": "0ad4cc29",
    "metadata": {
     "editable": true
    },
@@ -4277,19 +4276,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bee02bb6",
+   "id": "8dfd2150",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "p(X \\cup Y)= p(X,Y)= p(X\\vert Y)p(Y)=p(Y\\vert X)p(X),\n",
+    "p(X \\cap Y)= p(X,Y)= p(X\\vert Y)p(Y)=p(Y\\vert X)p(X),\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "f9cb00d6",
+   "id": "9c0313b7",
    "metadata": {
     "editable": true
    },
@@ -4303,7 +4302,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bf8b6016",
+   "id": "af94800f",
    "metadata": {
     "editable": true
    },
@@ -4315,7 +4314,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "192d323f",
+   "id": "3ed2ccef",
    "metadata": {
     "editable": true
    },
@@ -4325,7 +4324,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f593b004",
+   "id": "7a74ee19",
    "metadata": {
     "editable": true
    },
@@ -4337,7 +4336,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "515aeb1a",
+   "id": "5191a71e",
    "metadata": {
     "editable": true
    },
@@ -4347,7 +4346,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4f069d49",
+   "id": "5d5de8f7",
    "metadata": {
     "editable": true
    },
@@ -4359,7 +4358,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bfeebc26",
+   "id": "cca75f59",
    "metadata": {
     "editable": true
    },
@@ -4369,7 +4368,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8a0fadd6",
+   "id": "9113e675",
    "metadata": {
     "editable": true
    },
@@ -4381,7 +4380,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "24bd831c",
+   "id": "a21d13da",
    "metadata": {
     "editable": true
    },
@@ -4417,7 +4416,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ed5051fd",
+   "id": "0c7abec6",
    "metadata": {
     "editable": true
    },
@@ -4429,7 +4428,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3ee252d1",
+   "id": "2eccb1af",
    "metadata": {
     "editable": true
    },
@@ -4442,7 +4441,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5f96bad5",
+   "id": "3c6635e5",
    "metadata": {
     "editable": true
    },
@@ -4454,7 +4453,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c58a4b8e",
+   "id": "ecf0a0b6",
    "metadata": {
     "editable": true
    },
@@ -4468,7 +4467,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5415897f",
+   "id": "166345a1",
    "metadata": {
     "editable": true
    },
@@ -4480,7 +4479,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8a2bffd1",
+   "id": "8a73e80e",
    "metadata": {
     "editable": true
    },
@@ -4491,7 +4490,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2dd3ffc5",
+   "id": "01441388",
    "metadata": {
     "editable": true
    },
@@ -4503,7 +4502,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d7f4ee89",
+   "id": "93fe1e0a",
    "metadata": {
     "editable": true
    },
@@ -4515,7 +4514,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "45729327",
+   "id": "d6860415",
    "metadata": {
     "editable": true
    },
@@ -4533,7 +4532,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c2f81821",
+   "id": "9710dd92",
    "metadata": {
     "editable": true
    },
@@ -4551,7 +4550,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e97b2f6",
+   "id": "23e3912e",
    "metadata": {
     "editable": true
    },
@@ -4561,7 +4560,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "43e854ca",
+   "id": "cf500f71",
    "metadata": {
     "editable": true
    },
@@ -4590,7 +4589,7 @@
   {
    "cell_type": "code",
    "execution_count": 12,
-   "id": "62e17aee",
+   "id": "134d0a22",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4664,7 +4663,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c03f356d",
+   "id": "7dcfe550",
    "metadata": {
     "editable": true
    },
@@ -4694,7 +4693,7 @@
   {
    "cell_type": "code",
    "execution_count": 13,
-   "id": "1ecda570",
+   "id": "5ff5d7c2",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -4743,7 +4742,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bd3186fd",
+   "id": "f1680928",
    "metadata": {
     "editable": true
    },
@@ -4754,12 +4753,12 @@
     "noise. Here we recommend to use $\\sigma^2=1$ as variance for the\n",
     "added noise (which follows a normal distribution with mean value zero).\n",
     "Comment your results. If you have a large noise term, do the parameters $\\beta_j$ vary more as function\n",
-    "model complexity? And what about their variance?"
+    "of model complexity? And what about their variance?"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "10f8e3d1",
+   "id": "5b458403",
    "metadata": {
     "editable": true
    },
@@ -4777,7 +4776,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6a0b1e1c",
+   "id": "6e2d6bf6",
    "metadata": {
     "editable": true
    },
@@ -4789,7 +4788,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "642cfda6",
+   "id": "098b6cbd",
    "metadata": {
     "editable": true
    },
@@ -4799,7 +4798,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dc785785",
+   "id": "8a02d0aa",
    "metadata": {
     "editable": true
    },
@@ -4811,7 +4810,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a2591c04",
+   "id": "97f22408",
    "metadata": {
     "editable": true
    },
@@ -4821,7 +4820,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "37438bbe",
+   "id": "27038459",
    "metadata": {
     "editable": true
    },
@@ -4833,7 +4832,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9ad42344",
+   "id": "f682a8c3",
    "metadata": {
     "editable": true
    },
@@ -4843,7 +4842,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a95978e7",
+   "id": "7fdaa748",
    "metadata": {
     "editable": true
    },
@@ -4855,12 +4854,12 @@
   },
   {
    "cell_type": "markdown",
-   "id": "434a17cd",
+   "id": "1e7fa52c",
    "metadata": {
     "editable": true
    },
    "source": [
-    "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$!   \n",
+    "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta})$!   \n",
     "\n",
     "With the posterior probability defined by a likelihood which we have\n",
     "already modeled and an unknown prior, we are now ready to make\n",
@@ -4873,7 +4872,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c3581bd6",
+   "id": "501d66f4",
    "metadata": {
     "editable": true
    },
@@ -4885,7 +4884,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "19277aa0",
+   "id": "f029c143",
    "metadata": {
     "editable": true
    },
@@ -4895,7 +4894,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3aa17c65",
+   "id": "7f7c3e11",
    "metadata": {
     "editable": true
    },
@@ -4907,7 +4906,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "169fd184",
+   "id": "1f39114c",
    "metadata": {
     "editable": true
    },
@@ -4920,7 +4919,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8340686e",
+   "id": "81cc7b03",
    "metadata": {
     "editable": true
    },
@@ -4932,7 +4931,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "46d497c8",
+   "id": "1e614b9b",
    "metadata": {
     "editable": true
    },
@@ -4942,7 +4941,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "db839f5a",
+   "id": "77252afc",
    "metadata": {
     "editable": true
    },
@@ -4954,7 +4953,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7b412be2",
+   "id": "14953579",
    "metadata": {
     "editable": true
    },
@@ -4966,7 +4965,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "98250878",
+   "id": "36f1f63d",
    "metadata": {
     "editable": true
    },
@@ -4978,7 +4977,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e0374cc2",
+   "id": "50dd90a5",
    "metadata": {
     "editable": true
    },
@@ -4988,7 +4987,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f251157b",
+   "id": "de39cb12",
    "metadata": {
     "editable": true
    },
@@ -5000,7 +4999,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5beb906a",
+   "id": "ad1fc46e",
    "metadata": {
     "editable": true
    },
@@ -5012,19 +5011,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "18264240",
+   "id": "ff8695d4",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
+    "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "b977c7cd",
+   "id": "0de8080e",
    "metadata": {
     "editable": true
    },
@@ -5034,19 +5033,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "80b2f3e2",
+   "id": "3965e5ef",
    "metadata": {
     "editable": true
    },
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
+    "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
     "$$"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "dd62775a",
+   "id": "5c978cdf",
    "metadata": {
     "editable": true
    },
diff --git a/doc/LectureNotes/chapteroptimization.ipynb b/doc/LectureNotes/chapteroptimization.ipynb
index 5d83c5b50..b85c38adf 100644
--- a/doc/LectureNotes/chapteroptimization.ipynb
+++ b/doc/LectureNotes/chapteroptimization.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "4d72e1df",
+   "id": "f84a9d0d",
    "metadata": {
     "editable": true
    },
@@ -13,7 +13,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fb6e8fcd",
+   "id": "d40fde79",
    "metadata": {
     "editable": true
    },
@@ -39,7 +39,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a507b82a",
+   "id": "5e16ce97",
    "metadata": {
     "editable": true
    },
@@ -54,7 +54,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4071429e",
+   "id": "48b8210e",
    "metadata": {
     "editable": true
    },
@@ -70,7 +70,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a3cca0b2",
+   "id": "160bfec6",
    "metadata": {
     "editable": true
    },
@@ -82,7 +82,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12a4ad28",
+   "id": "cf829958",
    "metadata": {
     "editable": true
    },
@@ -93,7 +93,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5d6e7796",
+   "id": "a0ee64a1",
    "metadata": {
     "editable": true
    },
@@ -105,7 +105,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b324ae78",
+   "id": "9d6e5d8f",
    "metadata": {
     "editable": true
    },
@@ -119,7 +119,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7c39cc3f",
+   "id": "4af11aa5",
    "metadata": {
     "editable": true
    },
@@ -131,7 +131,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "10749f5d",
+   "id": "8801711a",
    "metadata": {
     "editable": true
    },
@@ -141,7 +141,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3ec50136",
+   "id": "34c3d5a7",
    "metadata": {
     "editable": true
    },
@@ -153,7 +153,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0145d9eb",
+   "id": "452bbf0a",
    "metadata": {
     "editable": true
    },
@@ -181,7 +181,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "43c17534",
+   "id": "e7d6fca4",
    "metadata": {
     "editable": true
    },
@@ -197,7 +197,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "32ef04f0",
+   "id": "afcf6c29",
    "metadata": {
     "editable": true
    },
@@ -208,7 +208,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6700017c",
+   "id": "e0e9aca3",
    "metadata": {
     "editable": true
    },
@@ -220,7 +220,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8791dec4",
+   "id": "5d70696d",
    "metadata": {
     "editable": true
    },
@@ -230,7 +230,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "69008872",
+   "id": "df759be1",
    "metadata": {
     "editable": true
    },
@@ -242,7 +242,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fce4ef1f",
+   "id": "d9f2bf15",
    "metadata": {
     "editable": true
    },
@@ -252,7 +252,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e64aef7e",
+   "id": "70cf6701",
    "metadata": {
     "editable": true
    },
@@ -264,7 +264,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9acecc44",
+   "id": "e2a70850",
    "metadata": {
     "editable": true
    },
@@ -287,7 +287,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "18bc9fd6",
+   "id": "883881e4",
    "metadata": {
     "editable": true
    },
@@ -300,7 +300,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7dcad370",
+   "id": "8704ba9a",
    "metadata": {
     "editable": true
    },
@@ -310,7 +310,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f1724121",
+   "id": "f5ae25a5",
    "metadata": {
     "editable": true
    },
@@ -328,7 +328,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d04bfa9f",
+   "id": "884ca09b",
    "metadata": {
     "editable": true
    },
@@ -338,7 +338,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a4d3a9e3",
+   "id": "9852d9e6",
    "metadata": {
     "editable": true
    },
@@ -353,7 +353,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d9d6ff69",
+   "id": "475ecae5",
    "metadata": {
     "editable": true
    },
@@ -363,7 +363,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "362a86a3",
+   "id": "d20911ef",
    "metadata": {
     "editable": true
    },
@@ -377,7 +377,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4b8c2937",
+   "id": "8c9e3f44",
    "metadata": {
     "editable": true
    },
@@ -387,7 +387,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7580aa9a",
+   "id": "eec377bf",
    "metadata": {
     "editable": true
    },
@@ -401,7 +401,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a2e3adc3",
+   "id": "93d87180",
    "metadata": {
     "editable": true
    },
@@ -416,7 +416,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "83a585c9",
+   "id": "e1165ae3",
    "metadata": {
     "editable": true
    },
@@ -433,7 +433,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e127ea11",
+   "id": "a84cf785",
    "metadata": {
     "editable": true
    },
@@ -445,7 +445,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "bce435bc",
+   "id": "387af099",
    "metadata": {
     "editable": true
    },
@@ -464,7 +464,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2691da5f",
+   "id": "71b6681c",
    "metadata": {
     "editable": true
    },
@@ -476,7 +476,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e7ae7322",
+   "id": "0b3f7fe6",
    "metadata": {
     "editable": true
    },
@@ -517,7 +517,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a6a44ea7",
+   "id": "c1a307a1",
    "metadata": {
     "editable": true
    },
@@ -536,7 +536,19 @@
     "$\\mathbb{R}$. Examples of convex sets of $\\mathbb{R}^2$ are the\n",
     "regular polygons (triangles, rectangles, pentagons, etc...).\n",
     "\n",
-    "**Convex function**: Let $X \\subset \\mathbb{R}^n$ be a convex set. Assume that the function $f: X \\rightarrow \\mathbb{R}$ is continuous, then $f$ is said to be convex if $$f(tx_1 + (1-t)x_2) \\leq tf(x_1) + (1-t)f(x_2) $$ for all $x_1, x_2 \\in X$ and for all $t \\in [0,1]$. If $\\leq$ is replaced with a strict inequaltiy in the definition, we demand $x_1 \\neq x_2$ and $t\\in(0,1)$ then $f$ is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the value of the function on the interval $[x_1,x_2]$ is always below the line as illustrated below.\n",
+    "**Convex function**: Let $X \\subset \\mathbb{R}^n$ be a convex\n",
+    "set. Assume that the function $f: X \\rightarrow \\mathbb{R}$ is\n",
+    "continuous, then $f$ is said to be convex if\n",
+    "$f(tx_1 + (1-t)x_2) \\leq tf(x_1) + (1-t)f(x_2)$\n",
+    "for all\n",
+    "$x_1, x_2 \\in X$ and for all $t \\in [0,1]$.\n",
+    "\n",
+    "If $\\leq$ is replaced with a strict inequality in the\n",
+    "definition, we demand $x_1 \\neq x_2$ and $t\\in(0,1)$ then $f$ is said\n",
+    "to be strictly convex. For a single variable function, convexity means\n",
+    "that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the\n",
+    "value of the function on the interval $[x_1,x_2]$ is always below the\n",
+    "line as discussed below.\n",
     "\n",
     "In the following we state first and second-order conditions which\n",
     "ensures convexity of a function $f$. We write $D_f$ to denote the\n",
@@ -550,7 +562,7 @@
     "is a convex set and $$f(y) \\geq f(x) + \\nabla f(x)^T (y-x) $$ holds\n",
     "for all $x,y \\in D_f$. This condition means that for a convex function\n",
     "the first order Taylor expansion (right hand side above) at any point\n",
-    "a global under estimator of the function. To convince yourself you can\n",
+    "is a global under estimator of the function. To convince yourself you can\n",
     "make a drawing of $f(x) = x^2+1$ and draw the tangent line to $f(x)$ and\n",
     "note that it is always below the graph.\n",
     "\n",
@@ -586,7 +598,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "809f8f01",
+   "id": "d648ed6e",
    "metadata": {
     "editable": true
    },
@@ -616,7 +628,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f3b91277",
+   "id": "0ca978c0",
    "metadata": {
     "editable": true
    },
@@ -636,7 +648,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ec752109",
+   "id": "82886118",
    "metadata": {
     "editable": true
    },
@@ -648,7 +660,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eb64c8e7",
+   "id": "d228b529",
    "metadata": {
     "editable": true
    },
@@ -658,7 +670,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e99eb7f",
+   "id": "04ffb6a6",
    "metadata": {
     "editable": true
    },
@@ -670,7 +682,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3a3f6414",
+   "id": "d7fcfada",
    "metadata": {
     "editable": true
    },
@@ -684,7 +696,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a88175a4",
+   "id": "825c07b3",
    "metadata": {
     "editable": true
    },
@@ -696,7 +708,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6992cc4f",
+   "id": "42b48221",
    "metadata": {
     "editable": true
    },
@@ -710,7 +722,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "00e22e2b",
+   "id": "69559dc1",
    "metadata": {
     "editable": true
    },
@@ -722,7 +734,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "faab896d",
+   "id": "24613200",
    "metadata": {
     "editable": true
    },
@@ -732,7 +744,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "02cb8061",
+   "id": "5c2d4c87",
    "metadata": {
     "editable": true
    },
@@ -744,7 +756,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fe52c448",
+   "id": "8087b072",
    "metadata": {
     "editable": true
    },
@@ -756,7 +768,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f698b7fe",
+   "id": "c18b46dd",
    "metadata": {
     "editable": true
    },
@@ -768,7 +780,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "022bda7f",
+   "id": "ce8a22ec",
    "metadata": {
     "editable": true
    },
@@ -780,7 +792,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b64077bc",
+   "id": "e439bb4c",
    "metadata": {
     "editable": true
    },
@@ -792,7 +804,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dec06904",
+   "id": "e794533b",
    "metadata": {
     "editable": true
    },
@@ -805,7 +817,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b566de75",
+   "id": "d9b643ae",
    "metadata": {
     "editable": true
    },
@@ -817,7 +829,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2c2d16e7",
+   "id": "5f99f3c9",
    "metadata": {
     "editable": true
    },
@@ -827,7 +839,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6c97f03d",
+   "id": "dcabf00c",
    "metadata": {
     "editable": true
    },
@@ -839,7 +851,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2b40818b",
+   "id": "405bd44d",
    "metadata": {
     "editable": true
    },
@@ -849,7 +861,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6619d064",
+   "id": "b917b358",
    "metadata": {
     "editable": true
    },
@@ -861,7 +873,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "467c71be",
+   "id": "dc88b39b",
    "metadata": {
     "editable": true
    },
@@ -871,7 +883,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d58fd1af",
+   "id": "8bec0c4c",
    "metadata": {
     "editable": true
    },
@@ -883,7 +895,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "38e32957",
+   "id": "b2b3ee64",
    "metadata": {
     "editable": true
    },
@@ -893,7 +905,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "98043fd6",
+   "id": "86bd52fd",
    "metadata": {
     "editable": true
    },
@@ -906,7 +918,7 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "8c7efe84",
+   "id": "4ce511b9",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -939,7 +951,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3bdeb3c1",
+   "id": "0221d5fb",
    "metadata": {
     "editable": true
    },
@@ -950,7 +962,7 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "e6e460db",
+   "id": "107c7b63",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -964,7 +976,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d165add7",
+   "id": "87224aa4",
    "metadata": {
     "editable": true
    },
@@ -975,7 +987,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "236615ae",
+   "id": "683d97c5",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -988,7 +1000,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4b90442c",
+   "id": "b3bdcc8f",
    "metadata": {
     "editable": true
    },
@@ -999,7 +1011,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "0346fd1d",
+   "id": "a633351f",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1017,7 +1029,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ecff8b5b",
+   "id": "73b12221",
    "metadata": {
     "editable": true
    },
@@ -1028,7 +1040,7 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "id": "0d9b7732",
+   "id": "54f7d8ef",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1043,7 +1055,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b0f8920b",
+   "id": "3c8bd3d6",
    "metadata": {
     "editable": true
    },
@@ -1057,7 +1069,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "67b215c6",
+   "id": "beada985",
    "metadata": {
     "editable": true
    },
@@ -1069,7 +1081,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "efc1c192",
+   "id": "ff86fb98",
    "metadata": {
     "editable": true
    },
@@ -1080,7 +1092,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eb4e1832",
+   "id": "5aeb1a79",
    "metadata": {
     "editable": true
    },
@@ -1092,7 +1104,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "775cf999",
+   "id": "a3832e7e",
    "metadata": {
     "editable": true
    },
@@ -1105,7 +1117,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "95893950",
+   "id": "a08e7ad5",
    "metadata": {
     "editable": true
    },
@@ -1117,7 +1129,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "16bfad5c",
+   "id": "9e7fb89c",
    "metadata": {
     "editable": true
    },
@@ -1130,7 +1142,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e2577f1c",
+   "id": "6112eb0a",
    "metadata": {
     "editable": true
    },
@@ -1142,7 +1154,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e4defaaf",
+   "id": "87453181",
    "metadata": {
     "editable": true
    },
@@ -1154,7 +1166,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b4632612",
+   "id": "02ee78ea",
    "metadata": {
     "editable": true
    },
@@ -1166,7 +1178,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a0a06bf2",
+   "id": "51806386",
    "metadata": {
     "editable": true
    },
@@ -1176,7 +1188,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1db7cc6a",
+   "id": "a706ddd7",
    "metadata": {
     "editable": true
    },
@@ -1188,7 +1200,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e7aa5384",
+   "id": "d74cb3eb",
    "metadata": {
     "editable": true
    },
@@ -1198,7 +1210,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "884580b0",
+   "id": "d3232b9d",
    "metadata": {
     "editable": true
    },
@@ -1210,7 +1222,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "934990e4",
+   "id": "eb42d3d2",
    "metadata": {
     "editable": true
    },
@@ -1220,7 +1232,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8be96384",
+   "id": "41db5953",
    "metadata": {
     "editable": true
    },
@@ -1232,7 +1244,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "44f4d8d5",
+   "id": "4489afa1",
    "metadata": {
     "editable": true
    },
@@ -1250,7 +1262,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a8739d7d",
+   "id": "fddfd922",
    "metadata": {
     "editable": true
    },
@@ -1262,7 +1274,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d871e171",
+   "id": "3cd48147",
    "metadata": {
     "editable": true
    },
@@ -1272,7 +1284,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7ed84e84",
+   "id": "ed97f976",
    "metadata": {
     "editable": true
    },
@@ -1284,7 +1296,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b690f75b",
+   "id": "72df369b",
    "metadata": {
     "editable": true
    },
@@ -1296,7 +1308,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "86ce9e8a",
+   "id": "808626d7",
    "metadata": {
     "editable": true
    },
@@ -1308,7 +1320,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7968787f",
+   "id": "ca35d289",
    "metadata": {
     "editable": true
    },
@@ -1320,7 +1332,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0d510b11",
+   "id": "4f2819b2",
    "metadata": {
     "editable": true
    },
@@ -1332,7 +1344,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c7c47a8e",
+   "id": "ecb94eeb",
    "metadata": {
     "editable": true
    },
@@ -1347,7 +1359,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5dc0e4ac",
+   "id": "121043dd",
    "metadata": {
     "editable": true
    },
@@ -1359,7 +1371,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0aecb4b3",
+   "id": "9ab7bf59",
    "metadata": {
     "editable": true
    },
@@ -1375,7 +1387,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f3dfc701",
+   "id": "7cfc8a8e",
    "metadata": {
     "editable": true
    },
@@ -1387,7 +1399,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12e92892",
+   "id": "b3627c04",
    "metadata": {
     "editable": true
    },
@@ -1397,7 +1409,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1514b03f",
+   "id": "e791510f",
    "metadata": {
     "editable": true
    },
@@ -1409,7 +1421,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7b09db3d",
+   "id": "a043cbbf",
    "metadata": {
     "editable": true
    },
@@ -1419,7 +1431,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4a0638f8",
+   "id": "e14a453e",
    "metadata": {
     "editable": true
    },
@@ -1431,7 +1443,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "298f5e46",
+   "id": "124c177b",
    "metadata": {
     "editable": true
    },
@@ -1441,7 +1453,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "acd35abb",
+   "id": "8f8a774d",
    "metadata": {
     "editable": true
    },
@@ -1453,7 +1465,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "79625f01",
+   "id": "1485ffbe",
    "metadata": {
     "editable": true
    },
@@ -1463,7 +1475,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dc5888e2",
+   "id": "72383dcb",
    "metadata": {
     "editable": true
    },
@@ -1475,7 +1487,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d11b96a9",
+   "id": "b94fbe5f",
    "metadata": {
     "editable": true
    },
@@ -1499,7 +1511,7 @@
   {
    "cell_type": "code",
    "execution_count": 6,
-   "id": "cf7c349e",
+   "id": "685e34ab",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1513,7 +1525,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "adf34219",
+   "id": "f6db7782",
    "metadata": {
     "editable": true
    },
@@ -1524,7 +1536,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d28eb4e0",
+   "id": "7c9405b5",
    "metadata": {
     "editable": true
    },
@@ -1536,7 +1548,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "60060fcb",
+   "id": "18b543bb",
    "metadata": {
     "editable": true
    },
@@ -1546,7 +1558,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "82983f16",
+   "id": "f51c5b8a",
    "metadata": {
     "editable": true
    },
@@ -1558,7 +1570,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "60fc4bc0",
+   "id": "ae853b5e",
    "metadata": {
     "editable": true
    },
@@ -1570,7 +1582,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0c5dac72",
+   "id": "2b0dba62",
    "metadata": {
     "editable": true
    },
@@ -1586,7 +1598,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "002d8c9b",
+   "id": "9523ebb8",
    "metadata": {
     "editable": true
    },
@@ -1596,7 +1608,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b91d8b5a",
+   "id": "b2fce916",
    "metadata": {
     "editable": true
    },
@@ -1608,7 +1620,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "d0fc6c20",
+   "id": "fe25134c",
    "metadata": {
     "editable": true
    },
@@ -1620,7 +1632,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3f99dab5",
+   "id": "fa866d39",
    "metadata": {
     "editable": true
    },
@@ -1634,7 +1646,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e1305a22",
+   "id": "1a26f3e1",
    "metadata": {
     "editable": true
    },
@@ -1646,7 +1658,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ef62073f",
+   "id": "13e655d3",
    "metadata": {
     "editable": true
    },
@@ -1661,7 +1673,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dfb78235",
+   "id": "4077ffc7",
    "metadata": {
     "editable": true
    },
@@ -1673,7 +1685,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "56cee86c",
+   "id": "ae32f491",
    "metadata": {
     "editable": true
    },
@@ -1685,7 +1697,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6cc71454",
+   "id": "2755532a",
    "metadata": {
     "editable": true
    },
@@ -1703,7 +1715,7 @@
   {
    "cell_type": "code",
    "execution_count": 7,
-   "id": "90fab6b7",
+   "id": "0babeef2",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1760,7 +1772,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "48ba87fe",
+   "id": "fe2faeda",
    "metadata": {
     "editable": true
    },
@@ -1771,7 +1783,7 @@
   {
    "cell_type": "code",
    "execution_count": 8,
-   "id": "7522775d",
+   "id": "89600ea6",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1798,7 +1810,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "90552e2f",
+   "id": "c277543b",
    "metadata": {
     "editable": true
    },
@@ -1808,7 +1820,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5d7c032c",
+   "id": "f929cf55",
    "metadata": {
     "editable": true
    },
@@ -1820,7 +1832,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ed86f1ba",
+   "id": "dff0af0d",
    "metadata": {
     "editable": true
    },
@@ -1830,7 +1842,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0386e23c",
+   "id": "3722fdb9",
    "metadata": {
     "editable": true
    },
@@ -1844,7 +1856,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b65523fc",
+   "id": "074790fe",
    "metadata": {
     "editable": true
    },
@@ -1854,7 +1866,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c62584ee",
+   "id": "3469f911",
    "metadata": {
     "editable": true
    },
@@ -1867,7 +1879,7 @@
   {
    "cell_type": "code",
    "execution_count": 9,
-   "id": "e8d43667",
+   "id": "61a75ef6",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -1921,7 +1933,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1ca83847",
+   "id": "967a83d3",
    "metadata": {
     "editable": true
    },
@@ -1943,7 +1955,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dcf3e808",
+   "id": "ba71e94a",
    "metadata": {
     "editable": true
    },
@@ -1982,7 +1994,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "473b1af6",
+   "id": "02c72775",
    "metadata": {
     "editable": true
    },
@@ -1995,7 +2007,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3353fe2a",
+   "id": "11665768",
    "metadata": {
     "editable": true
    },
@@ -2006,7 +2018,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6e8e47c3",
+   "id": "0995d8db",
    "metadata": {
     "editable": true
    },
@@ -2019,7 +2031,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "a2eeb6ad",
+   "id": "3cdc1697",
    "metadata": {
     "editable": true
    },
@@ -2046,7 +2058,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5a7a0f8b",
+   "id": "5af510b6",
    "metadata": {
     "editable": true
    },
@@ -2061,7 +2073,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b7b5884f",
+   "id": "490f4197",
    "metadata": {
     "editable": true
    },
@@ -2071,7 +2083,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6492d660",
+   "id": "219a6868",
    "metadata": {
     "editable": true
    },
@@ -2084,7 +2096,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "584164f4",
+   "id": "8d3502b5",
    "metadata": {
     "editable": true
    },
@@ -2099,7 +2111,7 @@
   {
    "cell_type": "code",
    "execution_count": 10,
-   "id": "42d97cf8",
+   "id": "2a2c6fab",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2124,7 +2136,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ca9c6c58",
+   "id": "de5275cd",
    "metadata": {
     "editable": true
    },
@@ -2164,7 +2176,7 @@
   {
    "cell_type": "code",
    "execution_count": 11,
-   "id": "d2921658",
+   "id": "1f118c97",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2199,7 +2211,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "84469eb8",
+   "id": "9250c537",
    "metadata": {
     "editable": true
    },
@@ -2209,7 +2221,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b4b94e7a",
+   "id": "54c13f2b",
    "metadata": {
     "editable": true
    },
@@ -2220,7 +2232,7 @@
   {
    "cell_type": "code",
    "execution_count": 12,
-   "id": "71dcdb35",
+   "id": "250dbe92",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2300,7 +2312,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6231b86f",
+   "id": "e0bdcd22",
    "metadata": {
     "editable": true
    },
@@ -2313,7 +2325,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8d50214c",
+   "id": "86fcc0af",
    "metadata": {
     "editable": true
    },
@@ -2328,7 +2340,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "45d43ca3",
+   "id": "3b656ad4",
    "metadata": {
     "editable": true
    },
@@ -2340,7 +2352,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dcdd91bf",
+   "id": "ec7ef032",
    "metadata": {
     "editable": true
    },
@@ -2358,7 +2370,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "c7519d1e",
+   "id": "842b17dd",
    "metadata": {
     "editable": true
    },
@@ -2377,7 +2389,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f4d4340d",
+   "id": "549a9b7e",
    "metadata": {
     "editable": true
    },
@@ -2389,7 +2401,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "41ad532a",
+   "id": "310fe216",
    "metadata": {
     "editable": true
    },
@@ -2405,7 +2417,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "eef8ae92",
+   "id": "e1b6edcb",
    "metadata": {
     "editable": true
    },
@@ -2417,7 +2429,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "80481a6d",
+   "id": "49e7b650",
    "metadata": {
     "editable": true
    },
@@ -2427,7 +2439,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6aab8db7",
+   "id": "67564cfb",
    "metadata": {
     "editable": true
    },
@@ -2439,7 +2451,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dd6414d2",
+   "id": "ebb0e17a",
    "metadata": {
     "editable": true
    },
@@ -2449,7 +2461,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b76a8372",
+   "id": "6d0cfa1c",
    "metadata": {
     "editable": true
    },
@@ -2461,7 +2473,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "07bbe6db",
+   "id": "e8252d81",
    "metadata": {
     "editable": true
    },
@@ -2475,7 +2487,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a9a2ecf",
+   "id": "ed9da45e",
    "metadata": {
     "editable": true
    },
@@ -2487,7 +2499,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fe5a2bbc",
+   "id": "11c47009",
    "metadata": {
     "editable": true
    },
@@ -2520,7 +2532,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "04540023",
+   "id": "7fd92874",
    "metadata": {
     "editable": true
    },
@@ -2532,7 +2544,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dfbf53a5",
+   "id": "5fa3a569",
    "metadata": {
     "editable": true
    },
@@ -2550,7 +2562,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "32ac3869",
+   "id": "1534657f",
    "metadata": {
     "editable": true
    },
@@ -2581,7 +2593,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6f575b16",
+   "id": "85b9db6c",
    "metadata": {
     "editable": true
    },
@@ -2596,7 +2608,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "274604c4",
+   "id": "0593ecb5",
    "metadata": {
     "editable": true
    },
@@ -2614,7 +2626,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cd679323",
+   "id": "8d8d8609",
    "metadata": {
     "editable": true
    },
@@ -2626,7 +2638,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0a7f8e9d",
+   "id": "e22f9446",
    "metadata": {
     "editable": true
    },
@@ -2638,7 +2650,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "48ba8fff",
+   "id": "1b88fbaa",
    "metadata": {
     "editable": true
    },
@@ -2656,7 +2668,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "83140ab2",
+   "id": "558d9648",
    "metadata": {
     "editable": true
    },
@@ -2679,7 +2691,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "61874dc3",
+   "id": "3711c9bc",
    "metadata": {
     "editable": true
    },
@@ -2697,7 +2709,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "24e19d86",
+   "id": "86d38c7e",
    "metadata": {
     "editable": true
    },
@@ -2709,7 +2721,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "506f78ea",
+   "id": "8fee2361",
    "metadata": {
     "editable": true
    },
@@ -2721,7 +2733,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cb4b8585",
+   "id": "705e9f9b",
    "metadata": {
     "editable": true
    },
@@ -2733,7 +2745,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9b5b11b1",
+   "id": "281da053",
    "metadata": {
     "editable": true
    },
@@ -2745,7 +2757,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "92292a73",
+   "id": "e5ed01f4",
    "metadata": {
     "editable": true
    },
@@ -2757,7 +2769,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1a264832",
+   "id": "7ab5a8ef",
    "metadata": {
     "editable": true
    },
@@ -2774,7 +2786,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6a307202",
+   "id": "f47fe0de",
    "metadata": {
     "editable": true
    },
@@ -2793,7 +2805,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3285f010",
+   "id": "78c5a239",
    "metadata": {
     "editable": true
    },
@@ -2805,7 +2817,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "657349da",
+   "id": "23d5750c",
    "metadata": {
     "editable": true
    },
@@ -2823,7 +2835,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "65044ac7",
+   "id": "78629315",
    "metadata": {
     "editable": true
    },
@@ -2861,7 +2873,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dacf05cf",
+   "id": "39d7472b",
    "metadata": {
     "editable": true
    },
@@ -2873,7 +2885,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "da4ad36e",
+   "id": "79a08e26",
    "metadata": {
     "editable": true
    },
@@ -2883,7 +2895,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f4c3e6c4",
+   "id": "2fca7cf2",
    "metadata": {
     "editable": true
    },
@@ -2895,7 +2907,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1c8bfd4a",
+   "id": "f55b402b",
    "metadata": {
     "editable": true
    },
@@ -2906,7 +2918,7 @@
   {
    "cell_type": "code",
    "execution_count": 13,
-   "id": "e1d91b8b",
+   "id": "83ffc6ab",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2951,7 +2963,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "cd1158a5",
+   "id": "09c698dc",
    "metadata": {
     "editable": true
    },
@@ -2966,7 +2978,7 @@
   {
    "cell_type": "code",
    "execution_count": 14,
-   "id": "e2e9faff",
+   "id": "23ec9da1",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -2994,7 +3006,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e4a83059",
+   "id": "c1d2dcbf",
    "metadata": {
     "editable": true
    },
@@ -3007,7 +3019,7 @@
   {
    "cell_type": "code",
    "execution_count": 15,
-   "id": "f65983d8",
+   "id": "0ff70c42",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3051,7 +3063,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "1d369f97",
+   "id": "8ad0ae71",
    "metadata": {
     "editable": true
    },
@@ -3062,7 +3074,7 @@
   {
    "cell_type": "code",
    "execution_count": 16,
-   "id": "46ea0652",
+   "id": "b9bf2265",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3090,7 +3102,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "9cc6674a",
+   "id": "0fae646d",
    "metadata": {
     "editable": true
    },
@@ -3106,7 +3118,7 @@
   {
    "cell_type": "code",
    "execution_count": 17,
-   "id": "17813055",
+   "id": "44a8fa94",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3135,7 +3147,7 @@
   {
    "cell_type": "code",
    "execution_count": 18,
-   "id": "6da49540",
+   "id": "962c21ba",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3161,7 +3173,7 @@
   {
    "cell_type": "code",
    "execution_count": 19,
-   "id": "a8ff2c17",
+   "id": "83468113",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3197,7 +3209,7 @@
   {
    "cell_type": "code",
    "execution_count": 20,
-   "id": "ec67d4a3",
+   "id": "9d7685e7",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3218,7 +3230,7 @@
   {
    "cell_type": "code",
    "execution_count": 21,
-   "id": "742a2d68",
+   "id": "666db882",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3256,7 +3268,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e7be6348",
+   "id": "5461498d",
    "metadata": {
     "editable": true
    },
@@ -3271,7 +3283,7 @@
   {
    "cell_type": "code",
    "execution_count": 22,
-   "id": "c551058c",
+   "id": "3543f315",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3293,7 +3305,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7a13b21d",
+   "id": "cc829644",
    "metadata": {
     "editable": true
    },
@@ -3304,7 +3316,7 @@
   {
    "cell_type": "code",
    "execution_count": 23,
-   "id": "19c7502b",
+   "id": "9dd71229",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3326,7 +3338,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4450885d",
+   "id": "128e658a",
    "metadata": {
     "editable": true
    },
@@ -3339,7 +3351,7 @@
   {
    "cell_type": "code",
    "execution_count": 24,
-   "id": "013fc7f8",
+   "id": "098a13f3",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3364,7 +3376,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8d360f7d",
+   "id": "2db7526b",
    "metadata": {
     "editable": true
    },
@@ -3375,7 +3387,7 @@
   {
    "cell_type": "code",
    "execution_count": 25,
-   "id": "e29a24eb",
+   "id": "314170c6",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3390,12 +3402,27 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ad8fbbb7",
+   "id": "5aa6151e",
    "metadata": {
     "editable": true
    },
    "source": [
-    "## Using Autograd with OLS\n",
+    "## Replace or not\n",
+    "\n",
+    "In the above code, we have use replacement in setting up the\n",
+    "mini-batches. The discussion\n",
+    "[here](https://sebastianraschka.com/faq/docs/sgd-methods.html) may be\n",
+    "useful."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2d017a74",
+   "metadata": {
+    "editable": true
+   },
+   "source": [
+    "## Using Autograd\n",
     "\n",
     "We conclude the part on optmization by showing how we can make codes\n",
     "for linear regression and logistic regression using **autograd**. The\n",
@@ -3405,7 +3432,7 @@
   {
    "cell_type": "code",
    "execution_count": 26,
-   "id": "904f65dc",
+   "id": "f784b385",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3465,20 +3492,155 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ce338980",
+   "id": "9eb6dbe2",
    "metadata": {
     "editable": true
    },
    "source": [
-    "### Including Stochastic Gradient Descent with Autograd\n",
-    "\n",
-    "In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**."
+    "## Same code but now with momentum gradient descent"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 27,
-   "id": "de261f10",
+   "id": "408e6211",
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "# Using Autograd to calculate gradients for OLS\n",
+    "from random import random, seed\n",
+    "import numpy as np\n",
+    "import autograd.numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from autograd import grad\n",
+    "\n",
+    "def CostOLS(beta):\n",
+    "    return (1.0/n)*np.sum((y-X @ beta)**2)\n",
+    "\n",
+    "n = 100\n",
+    "x = 2*np.random.rand(n,1)\n",
+    "y = 4+3*x#+np.random.randn(n,1)\n",
+    "\n",
+    "X = np.c_[np.ones((n,1)), x]\n",
+    "XT_X = X.T @ X\n",
+    "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
+    "print(\"Own inversion\")\n",
+    "print(theta_linreg)\n",
+    "# Hessian matrix\n",
+    "H = (2.0/n)* XT_X\n",
+    "EigValues, EigVectors = np.linalg.eig(H)\n",
+    "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
+    "\n",
+    "theta = np.random.randn(2,1)\n",
+    "eta = 1.0/np.max(EigValues)\n",
+    "Niterations = 30\n",
+    "\n",
+    "# define the gradient\n",
+    "training_gradient = grad(CostOLS)\n",
+    "\n",
+    "for iter in range(Niterations):\n",
+    "    gradients = training_gradient(theta)\n",
+    "    theta -= eta*gradients\n",
+    "    print(iter,gradients[0],gradients[1])\n",
+    "print(\"theta from own gd\")\n",
+    "print(theta)\n",
+    "\n",
+    "# Now improve with momentum gradient descent\n",
+    "change = 0.0\n",
+    "delta_momentum = 0.3\n",
+    "for iter in range(Niterations):\n",
+    "    # calculate gradient\n",
+    "    gradients = training_gradient(theta)\n",
+    "    # calculate update\n",
+    "    new_change = eta*gradients+delta_momentum*change\n",
+    "    # take a step\n",
+    "    theta -= new_change\n",
+    "    # save the change\n",
+    "    change = new_change\n",
+    "    print(iter,gradients[0],gradients[1])\n",
+    "print(\"theta from own gd wth momentum\")\n",
+    "print(theta)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "07f1dd70",
+   "metadata": {
+    "editable": true
+   },
+   "source": [
+    "We note indeed a considerable increase in efficiency here, we less iterations needed.\n",
+    "However, if we can invert the Hessian matrix, this is the preferred approach, as shown in the example here."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "id": "7eff4d61",
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "# Using Newton's method\n",
+    "from random import random, seed\n",
+    "import numpy as np\n",
+    "import autograd.numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from autograd import grad\n",
+    "\n",
+    "def CostOLS(beta):\n",
+    "    return (1.0/n)*np.sum((y-X @ beta)**2)\n",
+    "\n",
+    "n = 100\n",
+    "x = 2*np.random.rand(n,1)\n",
+    "y = 4+3*x+np.random.randn(n,1)\n",
+    "\n",
+    "X = np.c_[np.ones((n,1)), x]\n",
+    "XT_X = X.T @ X\n",
+    "beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
+    "print(\"Own inversion\")\n",
+    "print(beta_linreg)\n",
+    "# Hessian matrix\n",
+    "H = (2.0/n)* XT_X\n",
+    "# Note that here the Hessian does not depend on the parameters beta\n",
+    "invH = np.linalg.pinv(H)\n",
+    "EigValues, EigVectors = np.linalg.eig(H)\n",
+    "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
+    "\n",
+    "beta = np.random.randn(2,1)\n",
+    "Niterations = 5\n",
+    "\n",
+    "# define the gradient\n",
+    "training_gradient = grad(CostOLS)\n",
+    "\n",
+    "for iter in range(Niterations):\n",
+    "    gradients = training_gradient(beta)\n",
+    "    beta -= invH @ gradients\n",
+    "    print(iter,gradients[0],gradients[1])\n",
+    "print(\"beta from own Newton code\")\n",
+    "print(beta)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "98ae5663",
+   "metadata": {
+    "editable": true
+   },
+   "source": [
+    "## Including Stochastic Gradient Descent with Autograd\n",
+    "In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "id": "cd7caecf",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -3562,57 +3724,287 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ccd8829e",
+   "id": "87e8ab65",
    "metadata": {
     "editable": true
    },
    "source": [
-    "### And Logistic Regression"
+    "Here we include momentum in the standard gradient descent approach."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 28,
-   "id": "cc5811d1",
+   "execution_count": 30,
+   "id": "3183015a",
    "metadata": {
     "collapsed": false,
     "editable": true
    },
    "outputs": [],
    "source": [
+    "# Using Autograd to calculate gradients using SGD\n",
+    "# OLS example\n",
+    "from random import random, seed\n",
+    "import numpy as np\n",
     "import autograd.numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
     "from autograd import grad\n",
     "\n",
-    "def sigmoid(x):\n",
-    "    return 0.5 * (np.tanh(x / 2.) + 1)\n",
+    "# Note change from previous example\n",
+    "def CostOLS(y,X,theta):\n",
+    "    return np.sum((y-X @ theta)**2)\n",
     "\n",
-    "def logistic_predictions(weights, inputs):\n",
-    "    # Outputs probability of a label being true according to logistic model.\n",
-    "    return sigmoid(np.dot(inputs, weights))\n",
+    "n = 100\n",
+    "x = 2*np.random.rand(n,1)\n",
+    "y = 4+3*x+np.random.randn(n,1)\n",
     "\n",
-    "def training_loss(weights):\n",
-    "    # Training loss is the negative log-likelihood of the training labels.\n",
-    "    preds = logistic_predictions(weights, inputs)\n",
-    "    label_probabilities = preds * targets + (1 - preds) * (1 - targets)\n",
-    "    return -np.sum(np.log(label_probabilities))\n",
+    "X = np.c_[np.ones((n,1)), x]\n",
+    "XT_X = X.T @ X\n",
+    "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
+    "print(\"Own inversion\")\n",
+    "print(theta_linreg)\n",
+    "# Hessian matrix\n",
+    "H = (2.0/n)* XT_X\n",
+    "EigValues, EigVectors = np.linalg.eig(H)\n",
+    "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
     "\n",
-    "# Build a toy dataset.\n",
-    "inputs = np.array([[0.52, 1.12,  0.77],\n",
-    "                   [0.88, -1.08, 0.15],\n",
-    "                   [0.52, 0.06, -1.30],\n",
-    "                   [0.74, -2.49, 1.39]])\n",
-    "targets = np.array([True, True, False, True])\n",
+    "theta = np.random.randn(2,1)\n",
+    "eta = 1.0/np.max(EigValues)\n",
+    "Niterations = 100\n",
     "\n",
-    "# Define a function that returns gradients of training loss using Autograd.\n",
-    "training_gradient_fun = grad(training_loss)\n",
+    "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
+    "training_gradient = grad(CostOLS,2)\n",
     "\n",
-    "# Optimize weights using gradient descent.\n",
-    "weights = np.array([0.0, 0.0, 0.0])\n",
-    "print(\"Initial loss:\", training_loss(weights))\n",
-    "for i in range(100):\n",
-    "    weights -= training_gradient_fun(weights) * 0.01\n",
+    "for iter in range(Niterations):\n",
+    "    gradients = (1.0/n)*training_gradient(y, X, theta)\n",
+    "    theta -= eta*gradients\n",
+    "print(\"theta from own gd\")\n",
+    "print(theta)\n",
     "\n",
-    "print(\"Trained loss:\", training_loss(weights))"
+    "\n",
+    "n_epochs = 50\n",
+    "M = 5   #size of each minibatch\n",
+    "m = int(n/M) #number of minibatches\n",
+    "t0, t1 = 5, 50\n",
+    "def learning_schedule(t):\n",
+    "    return t0/(t+t1)\n",
+    "\n",
+    "theta = np.random.randn(2,1)\n",
+    "\n",
+    "change = 0.0\n",
+    "delta_momentum = 0.3\n",
+    "\n",
+    "for epoch in range(n_epochs):\n",
+    "    for i in range(m):\n",
+    "        random_index = M*np.random.randint(m)\n",
+    "        xi = X[random_index:random_index+M]\n",
+    "        yi = y[random_index:random_index+M]\n",
+    "        gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
+    "        eta = learning_schedule(epoch*m+i)\n",
+    "        # calculate update\n",
+    "        new_change = eta*gradients+delta_momentum*change\n",
+    "        # take a step\n",
+    "        theta -= new_change\n",
+    "        # save the change\n",
+    "        change = new_change\n",
+    "print(\"theta from own sdg with momentum\")\n",
+    "print(theta)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9ba705cd",
+   "metadata": {
+    "editable": true
+   },
+   "source": [
+    "### Similar (second order function now) problem but now with AdaGrad"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "id": "be7a85c7",
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent\n",
+    "# OLS example\n",
+    "from random import random, seed\n",
+    "import numpy as np\n",
+    "import autograd.numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from autograd import grad\n",
+    "\n",
+    "# Note change from previous example\n",
+    "def CostOLS(y,X,theta):\n",
+    "    return np.sum((y-X @ theta)**2)\n",
+    "\n",
+    "n = 10000\n",
+    "x = np.random.rand(n,1)\n",
+    "y = 2.0+3*x +4*x*x# +np.random.randn(n,1)\n",
+    "\n",
+    "X = np.c_[np.ones((n,1)), x, x*x]\n",
+    "XT_X = X.T @ X\n",
+    "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
+    "print(\"Own inversion\")\n",
+    "print(theta_linreg)\n",
+    "\n",
+    "\n",
+    "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
+    "training_gradient = grad(CostOLS,2)\n",
+    "# Define parameters for Stochastic Gradient Descent\n",
+    "n_epochs = 50\n",
+    "M = 5   #size of each minibatch\n",
+    "m = int(n/M) #number of minibatches\n",
+    "# Guess for unknown parameters theta\n",
+    "theta = np.random.randn(3,1)\n",
+    "\n",
+    "# Value for learning rate\n",
+    "eta = 0.01\n",
+    "# Including AdaGrad parameter to avoid possible division by zero\n",
+    "delta  = 1e-8\n",
+    "for epoch in range(n_epochs):\n",
+    "    # The outer product is calculated from scratch for each epoch\n",
+    "    Giter = np.zeros(shape=(3,3))\n",
+    "    for i in range(m):\n",
+    "        random_index = M*np.random.randint(m)\n",
+    "        xi = X[random_index:random_index+M]\n",
+    "        yi = y[random_index:random_index+M]\n",
+    "        gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
+    "\t# Calculate the outer product of the gradients\n",
+    "        Giter +=gradients @ gradients.T\n",
+    "\t# Simpler algorithm with only diagonal elements\n",
+    "        Ginverse = np.c_[eta/(delta+np.sqrt(np.diagonal(Giter)))]\n",
+    "        # compute update\n",
+    "        update = np.multiply(Ginverse,gradients)\n",
+    "        theta -= update\n",
+    "print(\"theta from own AdaGrad\")\n",
+    "print(theta)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0e711ae2",
+   "metadata": {
+    "editable": true
+   },
+   "source": [
+    "Running this code we note an almost perfect agreement with the results from matrix inversion.\n",
+    "\n",
+    "Similarly, here is our implementation of RMSprop."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "id": "8b34e5b1",
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "# Using Autograd to calculate gradients using RMSprop  and Stochastic Gradient descent\n",
+    "# OLS example\n",
+    "from random import random, seed\n",
+    "import numpy as np\n",
+    "import autograd.numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from autograd import grad\n",
+    "\n",
+    "# Note change from previous example\n",
+    "def CostOLS(y,X,theta):\n",
+    "    return np.sum((y-X @ theta)**2)\n",
+    "\n",
+    "n = 10000\n",
+    "x = np.random.rand(n,1)\n",
+    "y = 2.0+3*x +4*x*x# +np.random.randn(n,1)\n",
+    "\n",
+    "X = np.c_[np.ones((n,1)), x, x*x]\n",
+    "XT_X = X.T @ X\n",
+    "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
+    "print(\"Own inversion\")\n",
+    "print(theta_linreg)\n",
+    "\n",
+    "\n",
+    "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
+    "training_gradient = grad(CostOLS,2)\n",
+    "# Define parameters for Stochastic Gradient Descent\n",
+    "n_epochs = 50\n",
+    "M = 5   #size of each minibatch\n",
+    "m = int(n/M) #number of minibatches\n",
+    "# Guess for unknown parameters theta\n",
+    "theta = np.random.randn(3,1)\n",
+    "\n",
+    "# Value for learning rate\n",
+    "eta = 0.01\n",
+    "# Value for parameter rho\n",
+    "rho = 0.99\n",
+    "# Including AdaGrad parameter to avoid possible division by zero\n",
+    "delta  = 1e-8\n",
+    "for epoch in range(n_epochs):\n",
+    "    Giter = np.zeros(shape=(3,3))\n",
+    "    for i in range(m):\n",
+    "        random_index = M*np.random.randint(m)\n",
+    "        xi = X[random_index:random_index+M]\n",
+    "        yi = y[random_index:random_index+M]\n",
+    "        gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
+    "\t# Previous value for the outer product of gradients\n",
+    "        Previous = Giter\n",
+    "\t# Accumulated gradient\n",
+    "        Giter +=gradients @ gradients.T\n",
+    "\t# Scaling with rho the new and the previous results\n",
+    "        Gnew = (rho*Previous+(1-rho)*Giter)\n",
+    "\t# Taking the diagonal only and inverting\n",
+    "        Ginverse = np.c_[eta/(delta+np.sqrt(np.diagonal(Gnew)))]\n",
+    "\t# Hadamard product\n",
+    "        update = np.multiply(Ginverse,gradients)\n",
+    "        theta -= update\n",
+    "print(\"theta from own RMSprop\")\n",
+    "print(theta)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7a3b6455",
+   "metadata": {
+    "editable": true
+   },
+   "source": [
+    "## Introducing [JAX](https://jax.readthedocs.io/en/latest/)\n",
+    "\n",
+    "Presently, instead of using **autograd**, we recommend using [JAX](https://jax.readthedocs.io/en/latest/)\n",
+    "\n",
+    "**JAX** is Autograd and [XLA (Accelerated Linear Algebra))](https://www.tensorflow.org/xla),\n",
+    "brought together for high-performance numerical computing and machine learning research.\n",
+    "It provides composable transformations of Python+NumPy programs: differentiate, vectorize, parallelize, Just-In-Time compile to GPU/TPU, and more.\n",
+    "\n",
+    "Here's a simple example on how you can use **JAX** to compute the derivate of the logistic function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 33,
+   "id": "2c30f41b",
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "import jax.numpy as jnp\n",
+    "from jax import grad, jit, vmap\n",
+    "\n",
+    "def sum_logistic(x):\n",
+    "  return jnp.sum(1.0 / (1.0 + jnp.exp(-x)))\n",
+    "\n",
+    "x_small = jnp.arange(3.)\n",
+    "derivative_fn = grad(sum_logistic)\n",
+    "print(derivative_fn(x_small))"
    ]
   }
  ],