has linearly dependent column vectors, we will not be able to compute the inverse of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \theta_i \). -The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits. +The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exists. This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where the regression parameters \( \theta_i \) cannot be estimated.
@@ -388,7 +383,7 @@ $$and reordering we have
$$ -\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}. +\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=\boldsymbol{X}^T\boldsymbol{y}. $$We can redefine \( \lambda \) to absorb the constant \( n/2 \) and we rewrite the last equation as
$$ -\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}. +\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=\boldsymbol{X}^T\boldsymbol{y}. $$This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms.We will discuss how to code the above methods using gradient descent methods.
@@ -397,7 +392,7 @@ $$
- -
We have also discussed Ridge regression where the loss function contains a regularized term given by the \( L_2 \) norm of \( \theta \),
+$$ +C_{\text{ridge}}(\theta) = \frac{1}{n}||X\theta -\mathbf{y}||^2 + \lambda ||\theta||^2, \ \lambda \geq 0. +$$ - -# Importing various packages
-from random import random, seed
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.linear_model import SGDRegressor
+In order to minimize \( C_{\text{ridge}}(\theta) \) using GD we adjust the gradient as follows
+$$
+\nabla_\theta C_{\text{ridge}}(\theta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\theta_0+\theta_1x_i-y_i\right) \\
+\sum_{i=1}^{100}\left( x_i (\theta_0+\theta_1x_i)-y_ix_i\right) \\
+\end{bmatrix} + 2\lambda\begin{bmatrix} \theta_0 \\ \theta_1\end{bmatrix} = 2 (\frac{1}{n}X^T(X\theta - \mathbf{y})+\lambda \theta).
+$$
-n = 100
-x = 2*np.random.rand(n,1)
-y = 4+3*x+np.random.randn(n,1)
-
-X = np.c_[np.ones((n,1)), x]
-theta_linreg = np.linalg.inv(X.T @ X) @ (X.T @ y)
-print(theta_linreg)
-sgdreg = SGDRegressor(max_iter = 50, penalty=None, eta0=0.1)
-sgdreg.fit(x,y.ravel())
-print(sgdreg.intercept_, sgdreg.coef_)
-
-We can easily extend our program to minimize \( C_{\text{ridge}}(\theta) \) using gradient descent and compare with the analytical solution given by
+$$ +\theta_{\text{ridge}} = \left(X^T X + n\lambda I_{2 \times 2} \right)^{-1} X^T \mathbf{y}. +$$
@@ -407,7 +381,7 @@ sgdreg.fit(x,y.
We have also discussed Ridge regression where the loss function contains a regularized term given by the \( L_2 \) norm of \( \theta \), The Hessian matrix of Ridge Regression for our simple example is given by In order to minimize \( C_{\text{ridge}}(\theta) \) using GD we adjust the gradient as follows We can easily extend our program to minimize \( C_{\text{ridge}}(\theta) \) using gradient descent and compare with the analytical solution given by This implies that the Hessian matrix is positive definite, hence the stationary point is a
+minimum.
+Note that the Ridge cost function is convex being a sum of two convex
+functions. Therefore, the stationary point is a global
+minimum of this function.
+
@@ -386,7 +377,7 @@ $$
The Hessian matrix of Ridge Regression for our simple example is given by This implies that the Hessian matrix is positive definite, hence the stationary point is a
-minimum.
-Note that the Ridge cost function is convex being a sum of two convex
-functions. Therefore, the stationary point is a global
-minimum of this function.
-
@@ -382,7 +435,7 @@ minimum of this function.
The material here contains a summary of the lecture on Monday and discussion of SVD, Ridge and Lasso regression with examples
We used the SVD to analyse the matrix to invert in ordinary lineat regression Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is meaning we can write Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get The material here contains a summary of the lecture on Monday and discussion of SVD, Ridge and Lasso regression with examples
@@ -370,7 +395,7 @@ MathJax.Hub.Config({
We used the SVD to analyse the matrix to invert in ordinary lineat regression Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \)
+are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues
+given by the singular values squared, that is
+ meaning we can write Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive
+square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that
+the columns of \( \boldsymbol{V} \) are the eigenvectors of
+\( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of
+\( \boldsymbol{X} \) in a descending order, it means that the column vectors
+\( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they
+encode from the columns of \( \boldsymbol{X} \).
+ Note that these are also the eigenvectors and eigenvalues of the
+Hessian matrix.
+
@@ -400,7 +385,7 @@ $$
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \)
-are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues
-given by the singular values squared, that is
+ Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is
+our optimization problem is
In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive
-square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that
-the columns of \( \boldsymbol{V} \) are the eigenvectors of
-\( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of
-\( \boldsymbol{X} \) in a descending order, it means that the column vectors
-\( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they
-encode from the columns of \( \boldsymbol{X} \).
- or we can state it as where we have used the definition of a norm-2 vector, that is Note that these are also the eigenvectors and eigenvalues of the
-Hessian matrix.
-
@@ -390,7 +382,7 @@ Hessian matrix.
Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is
-our optimization problem is
+ By minimizing the above equation with respect to the parameters
+\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the
+parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by
+defining a new cost function to be optimized, that is
or we can state it as where we have used the definition of a norm-2 vector, that is which leads to the Ridge regression minimization problem where we
+require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is
+a finite number larger than zero. We do not include such a constraints in the discussions here.
+ By defining we have a new optimization equation which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. Here we have defined the norm-1 as By minimizing the above equation with respect to the parameters
-\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the
-parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by
-defining a new cost function to be optimized, that is
+ Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have and
+taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then
+a slightly modified matrix inversion problem which for finite values
+of \( \lambda \) does not suffer from singularity problems. We obtain
+the optimal parameters
which leads to the Ridge regression minimization problem where we
-require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is
-a finite number larger than zero. We do not include such a constraints in the discussions here.
- By defining with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that we have a new optimization equation which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. Here we have defined the norm-1 as with \( t \) a finite positive number.
@@ -404,7 +387,7 @@ $$
Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have Note well that a library like Scikit-Learn does not include the \( 1/n \) factor in the expression for the mean-squared error. If you include it, the optimal parameter \( \theta \) becomes and
-taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then
-a slightly modified matrix inversion problem which for finite values
-of \( \lambda \) does not suffer from singularity problems. We obtain
-the optimal parameters
- with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that with \( t \) a finite positive number. In our codes where we compare our own codes with Scikit-Learn, we do thus not include the \( 1/n \) factor in the cost function.
@@ -392,7 +371,7 @@ $$
Note well that a library like Scikit-Learn does not include the \( 1/n \) factor in the expression for the mean-squared error. If you include it, the optimal parameter \( \theta \) becomes When we compare this with the ordinary least squares result we have In our codes where we compare our own codes with Scikit-Learn, we do thus not include the \( 1/n \) factor in the cost function. which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). We see that Ridge regression is nothing but the standard OLS with a
+modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
+particular for our discussion of the bias-variance tradeoff are rather
+interesting. We will see that for specific values of \( \lambda \), we may
+even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
+
@@ -376,7 +376,7 @@ $$
When we compare this with the ordinary least squares result we have which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). We see that Ridge regression is nothing but the standard OLS with a
-modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
-particular for our discussion of the bias-variance tradeoff are rather
-interesting. We will see that for specific values of \( \lambda \), we may
-even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
+ Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
+We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
For Ridge regression this becomes with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).
@@ -381,7 +378,7 @@ even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). Th
Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
-We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
+ Since \( \lambda \geq 0 \), it means that compared to OLS, we have Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the
+orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by
+\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has
+eigenvalues ordered in a descending way, that is \( \sigma_i \geq
+\sigma_{i+1} \).
For Ridge regression this becomes with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \). For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom.
@@ -383,7 +378,7 @@ $$
Since \( \lambda \geq 0 \), it means that compared to OLS, we have For the sake of simplicity, let us assume that the design matrix is orthonormal, that is Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the
-orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by
-\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has
-eigenvalues ordered in a descending way, that is \( \sigma_i \geq
-\sigma_{i+1} \).
+ In this case the standard OLS results in and that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and
+the Ridge estimator converges to zero when the hyperparameter goes to
+infinity.
For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. We will come back to more interpreations after we have gone through some of the statistical analysis part. For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended.
+Similarly, Mehta et al's article is also recommended.
+
@@ -383,7 +391,7 @@ eigenvalues ordered in a descending way, that is \( \sigma_i \geq
For the sake of simplicity, let us assume that the design matrix is orthonormal, that is Using the matrix-vector expression for Lasso regression, we have the following cost function In this case the standard OLS results in Taking the derivative with respect to \( \boldsymbol{\theta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicity) and we have that the derivative of the cost function is that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and
-the Ridge estimator converges to zero when the hyperparameter goes to
-infinity.
- and reordering we have We will come back to more interpreations after we have gone through some of the statistical analysis part. For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended.
-Similarly, Mehta et al's article is also recommended.
- This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. We have absorbed the factor \( 2/n \) in a redefinition of the parameter \( \lambda \). We will solve this type of problems using libraries like scikit-learn and using our own gradient descent code in project 1.
@@ -396,7 +387,7 @@ Similarly, Mehta et al
Using the matrix-vector expression for Lasso regression, we have the following cost function Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the
+diagonal. In this case we have an equal number of rows and columns \( n=p \).
+ Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\theta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \)) Taking the derivative with respect to \( \boldsymbol{\theta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicity) and minimizing we have that we have that the derivative of the cost function is and reordering we have This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. We have absorbed the factor \( 2/n \) in a redefinition of the parameter \( \lambda \). We will solve this type of problems using libraries like scikit-learn and using our own gradient descent code in project 1.
@@ -392,7 +378,7 @@ $$
Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the
-diagonal. In this case we have an equal number of rows and columns \( n=p \).
- Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\theta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \)) For Ridge regression our cost function is and minimizing we have that For Ridge regression our cost function is For Lasso regression our cost function is and minimizing we have that which leads to Plotting these results shows clearly that Lasso regression suppresses (sets to zero) values of \( \theta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \theta_i \) as function of \( \lambda \).
@@ -378,8 +381,6 @@ $$
Let us assume we have a data set with outputs/targets given by the vector For Lasso regression our cost function is and minimizing we have that and our inputs as a \( 3\times 2 \) design matrix which leads to Plotting these results shows clearly that Lasso regression suppresses (sets to zero) values of \( \theta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \theta_i \) as function of \( \lambda \). meaning that we have two features and two unknown parameters \( \theta_0 \) and \( \theta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression.
@@ -385,7 +374,6 @@ $$
Let us assume we have a data set with outputs/targets given by the vector For ordinary least squares (OLS) we know that the optimal solution is and our inputs as a \( 3\times 2 \) design matrix Inserting the above values we obtain that meaning that we have two features and two unknown parameters \( \theta_0 \) and \( \theta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression. The code which implements this simpler case is presented after the discussion of Ridge and Lasso.
@@ -378,7 +374,6 @@ $$
For ordinary least squares (OLS) we know that the optimal solution is For Ridge regression we have Inserting the above values we obtain that The code which implements this simpler case is presented after the discussion of Ridge and Lasso. There is normally a constraint on the value of \( \vert\vert \boldsymbol{\theta}\vert\vert_2 \) via the parameter \( \lambda \).
+Let us for simplicity assume that \( \theta_0^2+\theta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \theta \) and \( \lambda \).
+ To see this, let us write the cost function for Ridge regression.
@@ -378,7 +377,6 @@ $$
For Ridge regression we have We define the MSE without the \( 1/n \) factor and have then, using that Inserting the above values we obtain that There is normally a constraint on the value of \( \vert\vert \boldsymbol{\theta}\vert\vert_2 \) via the parameter \( \lambda \).
-Let us for simplicity assume that \( \theta_0^2+\theta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \theta \) and \( \lambda \).
- and taking the derivative with respect to \( \theta_0 \) we get To see this, let us write the cost function for Ridge regression. and for \( \theta_1 \) we obtain Using the constraint for \( \theta_0^2+\theta_1^2=1 \) we can constrain \( \lambda \) by solving which gives \( \lambda=4.571 \) and \( \theta_0=0.933 \) and \( \theta_1=0.359 \).
@@ -381,7 +385,6 @@ Let us for simplicity assume that \( \theta_0^2+\theta_1^2=1 \) as constraint. T
We define the MSE without the \( 1/n \) factor and have then, using that For Lasso we need now, keeping a constraint on \( \vert\theta_0\vert+\vert\theta_1\vert=1 \), to take the derivative of the absolute values of \( \theta_0 \)
+and \( \theta_1 \). This gives us the following derivatives of the cost function
+ and taking the derivative with respect to \( \theta_0 \) we get and and for \( \theta_1 \) we obtain Using the constraint for \( \theta_0^2+\theta_1^2=1 \) we can constrain \( \lambda \) by solving which gives \( \lambda=4.571 \) and \( \theta_0=0.933 \) and \( \theta_1=0.359 \). We have now four cases to solve besides the trivial cases \( \theta_0 \) and/or \( \theta_1 \) are zero, namely
For Lasso we need now, keeping a constraint on \( \vert\theta_0\vert+\vert\theta_1\vert=1 \), to take the derivative of the absolute values of \( \theta_0 \)
-and \( \theta_1 \). This gives us the following derivatives of the cost function
- If we consider the first case, we have then and We have now four cases to solve besides the trivial cases \( \theta_0 \) and/or \( \theta_1 \) are zero, namely which yields and Using the constraint on \( \theta_0 \) and \( \theta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you.
If we consider the first case, we have then Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values. and First we study and compare the OLS and Ridge results. The next code compares all three methods. which yields and Using the constraint on \( \theta_0 \) and \( \theta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you. We see here that we reach a plateau. What is actually happening?
@@ -383,7 +432,6 @@ $$
Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values. First we study and compare the OLS and Ridge results. The next code compares all three methods.Gradient descent and Ridge
-
-The Hessian matrix for Ridge Regression
+The Hessian matrix for Ridge Regression
-Program example for gradient descent with Ridge Regression
+
+
+from random import random, seed
+import numpy as np
+import matplotlib.pyplot as plt
+from mpl_toolkits.mplot3d import Axes3D
+from matplotlib import cm
+from matplotlib.ticker import LinearLocator, FormatStrFormatter
+import sys
+
+# the number of datapoints
+n = 100
+x = 2*np.random.rand(n,1)
+y = 4+3*x+np.random.randn(n,1)
+
+X = np.c_[np.ones((n,1)), x]
+XT_X = X.T @ X
+
+#Ridge parameter lambda
+lmbda = 0.001
+Id = n*lmbda* np.eye(XT_X.shape[0])
+
+# Hessian matrix
+H = (2.0/n)* XT_X+2*lmbda* np.eye(XT_X.shape[0])
+# Get the eigenvalues
+EigValues, EigVectors = np.linalg.eig(H)
+print(f"Eigenvalues of Hessian Matrix:{EigValues}")
+
+
+theta_linreg = np.linalg.inv(XT_X+Id) @ X.T @ y
+print(theta_linreg)
+# Start plain gradient descent
+theta = np.random.randn(2,1)
+
+eta = 1.0/np.max(EigValues)
+Niterations = 100
+
+for iter in range(Niterations):
+ gradients = 2.0/n*X.T @ (X @ (theta)-y)+2*lmbda*theta
+ theta -= eta*gradients
+
+print(theta)
+ypredict = X @ theta
+ypredict2 = X @ theta_linreg
+plt.plot(x, ypredict, "r-")
+plt.plot(x, ypredict2, "b-")
+plt.plot(x, y ,'ro')
+plt.axis([0,2.0,0, 15.0])
+plt.xlabel(r'$x$')
+plt.ylabel(r'$y$')
+plt.title(r'Gradient descent example for Ridge')
+plt.show()
+
+Program example for gradient descent with Ridge Regression
-
-
-from random import random, seed
-import numpy as np
-import matplotlib.pyplot as plt
-from mpl_toolkits.mplot3d import Axes3D
-from matplotlib import cm
-from matplotlib.ticker import LinearLocator, FormatStrFormatter
-import sys
-
-# the number of datapoints
-n = 100
-x = 2*np.random.rand(n,1)
-y = 4+3*x+np.random.randn(n,1)
-
-X = np.c_[np.ones((n,1)), x]
-XT_X = X.T @ X
-
-#Ridge parameter lambda
-lmbda = 0.001
-Id = n*lmbda* np.eye(XT_X.shape[0])
-
-# Hessian matrix
-H = (2.0/n)* XT_X+2*lmbda* np.eye(XT_X.shape[0])
-# Get the eigenvalues
-EigValues, EigVectors = np.linalg.eig(H)
-print(f"Eigenvalues of Hessian Matrix:{EigValues}")
-
-
-theta_linreg = np.linalg.inv(XT_X+Id) @ X.T @ y
-print(theta_linreg)
-# Start plain gradient descent
-theta = np.random.randn(2,1)
-
-eta = 1.0/np.max(EigValues)
-Niterations = 100
-
-for iter in range(Niterations):
- gradients = 2.0/n*X.T @ (X @ (theta)-y)+2*lmbda*theta
- theta -= eta*gradients
-
-print(theta)
-ypredict = X @ theta
-ypredict2 = X @ theta_linreg
-plt.plot(x, ypredict, "r-")
-plt.plot(x, ypredict2, "b-")
-plt.plot(x, y ,'ro')
-plt.axis([0,2.0,0, 15.0])
-plt.xlabel(r'$x$')
-plt.ylabel(r'$y$')
-plt.title(r'Gradient descent example for Ridge')
-plt.show()
-
-Using gradient descent methods, limitations
+
+
Using gradient descent methods, limitations
+Material for lab sessions sessions Tuesday and Wednesday
+
+
-
Material for lab sessions sessions Tuesday and Wednesday
+Linear Regression and the SVD
+
+Linear Regression and the SVD
+What does it mean?
-What does it mean?
+Ridge and LASSO Regression
-Ridge and LASSO Regression
+From OLS to Ridge and Lasso
-From OLS to Ridge and Lasso
+Deriving the Ridge Regression Equations
-Deriving the Ridge Regression Equations
+Note on Scikit-Learn
-Note on Scikit-Learn
-
-Comparison with OLS
+Comparison with OLS
-SVD analysis
-SVD analysis
+Interpreting the Ridge results
-Interpreting the Ridge results
+More interpretations
-More interpretations
+Deriving the Lasso Regression Equations
-Deriving the Lasso Regression Equations
+Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
-Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
+Ridge Regression
-Ridge Regression
+Lasso Regression
-Lasso Regression
+Yet another Example
+
+Yet another Example
+The OLS case
-The OLS case
+The Ridge case
-The Ridge case
-
-Writing the Cost Function
+Writing the Cost Function
+Lasso case
-
+
Lasso case
+The first Case
-
-
+The first Case
+Simple code for solving the above problem
-import os
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
-
+Simple code for solving the above problem
-
-With Lasso Regression
@@ -358,6 +349,7 @@ MathJax.Hub.Config({
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
+from sklearn import linear_model
def R2(y_data, y_model):
return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
@@ -386,19 +378,25 @@ ypredictOLS = X = np.eye(2,2)
# Decide which values of lambda to use
nlambdas = 100
-MSEPredict = np.zeros(nlambdas)
+MSERidgePredict = np.zeros(nlambdas)
+MSELassoPredict = np.zeros(nlambdas)
lambdas = np.logspace(-4, 4, nlambdas)
for i in range(nlambdas):
lmb = lambdas[i]
Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
-# print(Ridgebeta)
+ print(Ridgebeta)
# and then make the prediction
ypredictRidge = X @ Ridgebeta
- MSEPredict[i] = MSE(y,ypredictRidge)
-# print(MSEPredict[i])
- # Now plot the results
+ MSERidgePredict[i] = MSE(y,ypredictRidge)
+ RegLasso = linear_model.Lasso(lmb,fit_intercept=False)
+ RegLasso.fit(X,y)
+ ypredictLasso = RegLasso.predict(X)
+ print(RegLasso.coef_)
+ MSELassoPredict[i] = MSE(y,ypredictLasso)
+# Now plot the results
plt.figure()
-plt.plot(np.log10(lambdas), MSEPredict, 'r--', label = 'MSE Ridge Train')
+plt.plot(np.log10(lambdas), MSERidgePredict, 'r--', label = 'MSE Ridge Train')
+plt.plot(np.log10(lambdas), MSELassoPredict, 'r--', label = 'MSE Lasso Train')
plt.xlabel('log10(lambda)')
plt.ylabel('MSE')
plt.legend()
@@ -418,7 +416,6 @@ plt.show()
We see here that we reach a plateau. What is actually happening?
@@ -436,7 +433,6 @@ plt.show()
-
has linearly dependent column vectors, we will not be able to compute the inverse of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \theta_i \). -The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits. +The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exists. This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where the regression parameters \( \theta_i \) cannot be estimated.
@@ -1279,14 +1279,14 @@ $$and reordering we have
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=\boldsymbol{X}^T\boldsymbol{y}.
$$
We can redefine \( \lambda \) to absorb the constant \( n/2 \) and we rewrite the last equation as
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=\boldsymbol{X}^T\boldsymbol{y}.
$$
has linearly dependent column vectors, we will not be able to compute the inverse
of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \theta_i \).
-The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits.
+The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exists.
This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where
the regression parameters \( \theta_i \) cannot be estimated.
and reordering we have We can redefine \( \lambda \) to absorb the constant \( n/2 \) and we rewrite the last equation as This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms.We will discuss how to code the above methods using gradient descent methods. has linearly dependent column vectors, we will not be able to compute the inverse
of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \theta_i \).
-The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits.
+The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exists.
This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where
the regression parameters \( \theta_i \) cannot be estimated.
and reordering we have We can redefine \( \lambda \) to absorb the constant \( n/2 \) and we rewrite the last equation as This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms.We will discuss how to code the above methods using gradient descent methods.
@@ -1857,48 +1857,6 @@ plt.show()
-And a corresponding example using scikit-learn
-
-
-
-# Importing various packages
-from random import random, seed
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.linear_model import SGDRegressor
-
-n = 100
-x = 2*np.random.rand(n,1)
-y = 4+3*x+np.random.randn(n,1)
-
-X = np.c_[np.ones((n,1)), x]
-theta_linreg = np.linalg.inv(X.T @ X) @ (X.T @ y)
-print(theta_linreg)
-sgdreg = SGDRegressor(max_iter = 50, penalty=None, eta0=0.1)
-sgdreg.fit(x,y.ravel())
-print(sgdreg.intercept_, sgdreg.coef_)
-
-Gradient descent and Ridge
diff --git a/doc/pub/week36/html/week36-solarized.html b/doc/pub/week36/html/week36-solarized.html
index 30bfa6fd8..0da5f2ac4 100644
--- a/doc/pub/week36/html/week36-solarized.html
+++ b/doc/pub/week36/html/week36-solarized.html
@@ -173,10 +173,6 @@ div.toc p,a {
('The Hessian matrix', 2, None, 'the-hessian-matrix'),
('Simple program', 2, None, 'simple-program'),
('Gradient Descent Example', 2, None, 'gradient-descent-example'),
- ('And a corresponding example using _scikit-learn_',
- 2,
- None,
- 'and-a-corresponding-example-using-scikit-learn'),
('Gradient descent and Ridge',
2,
None,
@@ -485,7 +481,7 @@ $$
-And a corresponding example using scikit-learn
-
-
-
-# Importing various packages
-from random import random, seed
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.linear_model import SGDRegressor
-
-n = 100
-x = 2*np.random.rand(n,1)
-y = 4+3*x+np.random.randn(n,1)
-
-X = np.c_[np.ones((n,1)), x]
-theta_linreg = np.linalg.inv(X.T @ X) @ (X.T @ y)
-print(theta_linreg)
-sgdreg = SGDRegressor(max_iter = 50, penalty=None, eta0=0.1)
-sgdreg.fit(x,y.ravel())
-print(sgdreg.intercept_, sgdreg.coef_)
-
-Gradient descent and Ridge
diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html
index f848354a3..94f911021 100644
--- a/doc/pub/week36/html/week36.html
+++ b/doc/pub/week36/html/week36.html
@@ -250,10 +250,6 @@ div.toc p,a {
('The Hessian matrix', 2, None, 'the-hessian-matrix'),
('Simple program', 2, None, 'simple-program'),
('Gradient Descent Example', 2, None, 'gradient-descent-example'),
- ('And a corresponding example using _scikit-learn_',
- 2,
- None,
- 'and-a-corresponding-example-using-scikit-learn'),
('Gradient descent and Ridge',
2,
None,
@@ -562,7 +558,7 @@ $$
-And a corresponding example using scikit-learn
-
-
-
-# Importing various packages
-from random import random, seed
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.linear_model import SGDRegressor
-
-n = 100
-x = 2*np.random.rand(n,1)
-y = 4+3*x+np.random.randn(n,1)
-
-X = np.c_[np.ones((n,1)), x]
-theta_linreg = np.linalg.inv(X.T @ X) @ (X.T @ y)
-print(theta_linreg)
-sgdreg = SGDRegressor(max_iter = 50, penalty=None, eta0=0.1)
-sgdreg.fit(x,y.ravel())
-print(sgdreg.intercept_, sgdreg.coef_)
-
-Gradient descent and Ridge
diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
index 76a80034f..54702a8db 100644
Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ
diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb
index d03acd92b..d0d3dee3f 100644
--- a/doc/pub/week36/ipynb/week36.ipynb
+++ b/doc/pub/week36/ipynb/week36.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "b8f45e2d",
+ "id": "3586da6c",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
- "id": "b3f2ead9",
+ "id": "5afa3a45",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
- "id": "04553781",
+ "id": "739da2e4",
"metadata": {
"editable": true
},
@@ -57,7 +57,7 @@
},
{
"cell_type": "markdown",
- "id": "1016bcdb",
+ "id": "66bc8f20",
"metadata": {
"editable": true
},
@@ -67,7 +67,7 @@
},
{
"cell_type": "markdown",
- "id": "839e657a",
+ "id": "d6b147a8",
"metadata": {
"editable": true
},
@@ -81,7 +81,7 @@
},
{
"cell_type": "markdown",
- "id": "12dc9112",
+ "id": "69ee41c8",
"metadata": {
"editable": true
},
@@ -93,7 +93,7 @@
},
{
"cell_type": "markdown",
- "id": "df3e7f6d",
+ "id": "eec38b3f",
"metadata": {
"editable": true
},
@@ -105,7 +105,7 @@
},
{
"cell_type": "markdown",
- "id": "cc1caa8e",
+ "id": "7ca66dca",
"metadata": {
"editable": true
},
@@ -117,7 +117,7 @@
},
{
"cell_type": "markdown",
- "id": "aaf95f40",
+ "id": "4a67a7b4",
"metadata": {
"editable": true
},
@@ -127,7 +127,7 @@
},
{
"cell_type": "markdown",
- "id": "5e615b93",
+ "id": "4b3a2043",
"metadata": {
"editable": true
},
@@ -139,7 +139,7 @@
},
{
"cell_type": "markdown",
- "id": "7f764f4e",
+ "id": "841bde50",
"metadata": {
"editable": true
},
@@ -149,7 +149,7 @@
},
{
"cell_type": "markdown",
- "id": "4e0d9985",
+ "id": "ebca953b",
"metadata": {
"editable": true
},
@@ -161,7 +161,7 @@
},
{
"cell_type": "markdown",
- "id": "a11e3b77",
+ "id": "7be2de84",
"metadata": {
"editable": true
},
@@ -172,7 +172,7 @@
},
{
"cell_type": "markdown",
- "id": "b175d8ad",
+ "id": "9916c24e",
"metadata": {
"editable": true
},
@@ -184,7 +184,7 @@
},
{
"cell_type": "markdown",
- "id": "0901c5d9",
+ "id": "03c35639",
"metadata": {
"editable": true
},
@@ -196,7 +196,7 @@
},
{
"cell_type": "markdown",
- "id": "7263285d",
+ "id": "63aaa0d6",
"metadata": {
"editable": true
},
@@ -206,7 +206,7 @@
},
{
"cell_type": "markdown",
- "id": "c0f54e1a",
+ "id": "563e4883",
"metadata": {
"editable": true
},
@@ -218,7 +218,7 @@
},
{
"cell_type": "markdown",
- "id": "035835e7",
+ "id": "207fb037",
"metadata": {
"editable": true
},
@@ -230,7 +230,7 @@
},
{
"cell_type": "markdown",
- "id": "a0dca7fe",
+ "id": "d0ebfe7a",
"metadata": {
"editable": true
},
@@ -240,7 +240,7 @@
},
{
"cell_type": "markdown",
- "id": "5808efd5",
+ "id": "e5eba895",
"metadata": {
"editable": true
},
@@ -252,7 +252,7 @@
},
{
"cell_type": "markdown",
- "id": "cca67bb8",
+ "id": "20736fd2",
"metadata": {
"editable": true
},
@@ -262,7 +262,7 @@
},
{
"cell_type": "markdown",
- "id": "a189ce2b",
+ "id": "8ebaa24c",
"metadata": {
"editable": true
},
@@ -274,7 +274,7 @@
},
{
"cell_type": "markdown",
- "id": "7835a4fb",
+ "id": "9e0e8b1b",
"metadata": {
"editable": true
},
@@ -284,7 +284,7 @@
},
{
"cell_type": "markdown",
- "id": "1ba547c3",
+ "id": "a9dc6df7",
"metadata": {
"editable": true
},
@@ -324,7 +324,7 @@
},
{
"cell_type": "markdown",
- "id": "55bc2c91",
+ "id": "4a1ebfbc",
"metadata": {
"editable": true
},
@@ -341,7 +341,7 @@
},
{
"cell_type": "markdown",
- "id": "c702f5e0",
+ "id": "632a74c9",
"metadata": {
"editable": true
},
@@ -364,7 +364,7 @@
},
{
"cell_type": "markdown",
- "id": "86cc8cb6",
+ "id": "d8173cef",
"metadata": {
"editable": true
},
@@ -381,7 +381,7 @@
},
{
"cell_type": "markdown",
- "id": "1797b7e5",
+ "id": "8beab768",
"metadata": {
"editable": true
},
@@ -400,7 +400,7 @@
},
{
"cell_type": "markdown",
- "id": "dc73a5a2",
+ "id": "7f8598e9",
"metadata": {
"editable": true
},
@@ -411,7 +411,7 @@
},
{
"cell_type": "markdown",
- "id": "20b19ac5",
+ "id": "55c77d47",
"metadata": {
"editable": true
},
@@ -423,7 +423,7 @@
},
{
"cell_type": "markdown",
- "id": "c269e5e9",
+ "id": "695e94a9",
"metadata": {
"editable": true
},
@@ -441,14 +441,14 @@
},
{
"cell_type": "markdown",
- "id": "64cdb37b",
+ "id": "7cf05588",
"metadata": {
"editable": true
},
"source": [
"has linearly dependent column vectors, we will not be able to compute the inverse\n",
"of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\theta_i$. \n",
- "The estimators are only well-defined if $(\\boldsymbol{X}^{T}\\boldsymbol{X})^{-1}$ exits. \n",
+ "The estimators are only well-defined if $(\\boldsymbol{X}^{T}\\boldsymbol{X})^{-1}$ exists. \n",
"This is more likely to happen when the matrix $\\boldsymbol{X}$ is high-dimensional. In this case it is likely to encounter a situation where \n",
"the regression parameters $\\theta_i$ cannot be estimated.\n",
"\n",
@@ -457,7 +457,7 @@
},
{
"cell_type": "markdown",
- "id": "3e6e5a3e",
+ "id": "bed94018",
"metadata": {
"editable": true
},
@@ -469,7 +469,7 @@
},
{
"cell_type": "markdown",
- "id": "fd83bb4b",
+ "id": "aae1d2f2",
"metadata": {
"editable": true
},
@@ -479,7 +479,7 @@
},
{
"cell_type": "markdown",
- "id": "45b376e8",
+ "id": "d7b1e50f",
"metadata": {
"editable": true
},
@@ -492,7 +492,7 @@
},
{
"cell_type": "markdown",
- "id": "90bb63a4",
+ "id": "5aaa4dd5",
"metadata": {
"editable": true
},
@@ -504,7 +504,7 @@
},
{
"cell_type": "markdown",
- "id": "de110cca",
+ "id": "23b4b363",
"metadata": {
"editable": true
},
@@ -514,7 +514,7 @@
},
{
"cell_type": "markdown",
- "id": "a8f75c00",
+ "id": "0d156e68",
"metadata": {
"editable": true
},
@@ -527,7 +527,7 @@
},
{
"cell_type": "markdown",
- "id": "e3d1ecbd",
+ "id": "386fac67",
"metadata": {
"editable": true
},
@@ -537,7 +537,7 @@
},
{
"cell_type": "markdown",
- "id": "a932ed24",
+ "id": "e0083fa6",
"metadata": {
"editable": true
},
@@ -549,7 +549,7 @@
},
{
"cell_type": "markdown",
- "id": "9ec04c56",
+ "id": "c3bf20b5",
"metadata": {
"editable": true
},
@@ -562,7 +562,7 @@
},
{
"cell_type": "markdown",
- "id": "c191b236",
+ "id": "8a09eb45",
"metadata": {
"editable": true
},
@@ -575,7 +575,7 @@
},
{
"cell_type": "markdown",
- "id": "214affcd",
+ "id": "b9e924fa",
"metadata": {
"editable": true
},
@@ -587,7 +587,7 @@
},
{
"cell_type": "markdown",
- "id": "cebe8ac0",
+ "id": "14e649b1",
"metadata": {
"editable": true
},
@@ -599,7 +599,7 @@
},
{
"cell_type": "markdown",
- "id": "a10a36e8",
+ "id": "2357300f",
"metadata": {
"editable": true
},
@@ -609,7 +609,7 @@
},
{
"cell_type": "markdown",
- "id": "51e17519",
+ "id": "d5b9128f",
"metadata": {
"editable": true
},
@@ -622,7 +622,7 @@
},
{
"cell_type": "markdown",
- "id": "eb706113",
+ "id": "9b8f2719",
"metadata": {
"editable": true
},
@@ -634,7 +634,7 @@
},
{
"cell_type": "markdown",
- "id": "f2252659",
+ "id": "ccaec8bc",
"metadata": {
"editable": true
},
@@ -646,7 +646,7 @@
},
{
"cell_type": "markdown",
- "id": "bcf67e2c",
+ "id": "b9bbc55c",
"metadata": {
"editable": true
},
@@ -658,7 +658,7 @@
},
{
"cell_type": "markdown",
- "id": "4053a663",
+ "id": "4c128865",
"metadata": {
"editable": true
},
@@ -670,7 +670,7 @@
},
{
"cell_type": "markdown",
- "id": "8c8d8ce5",
+ "id": "d9b05a5a",
"metadata": {
"editable": true
},
@@ -684,7 +684,7 @@
},
{
"cell_type": "markdown",
- "id": "fd306ffb",
+ "id": "4c3de377",
"metadata": {
"editable": true
},
@@ -696,7 +696,7 @@
},
{
"cell_type": "markdown",
- "id": "f819a5cd",
+ "id": "64b28254",
"metadata": {
"editable": true
},
@@ -706,7 +706,7 @@
},
{
"cell_type": "markdown",
- "id": "95c2e048",
+ "id": "8123860e",
"metadata": {
"editable": true
},
@@ -718,7 +718,7 @@
},
{
"cell_type": "markdown",
- "id": "071a983e",
+ "id": "0e327aac",
"metadata": {
"editable": true
},
@@ -730,7 +730,7 @@
},
{
"cell_type": "markdown",
- "id": "36d40252",
+ "id": "134d6cb8",
"metadata": {
"editable": true
},
@@ -742,7 +742,7 @@
},
{
"cell_type": "markdown",
- "id": "e70d5af4",
+ "id": "97d5602d",
"metadata": {
"editable": true
},
@@ -754,7 +754,7 @@
},
{
"cell_type": "markdown",
- "id": "d0becda4",
+ "id": "90137256",
"metadata": {
"editable": true
},
@@ -766,7 +766,7 @@
},
{
"cell_type": "markdown",
- "id": "c8ee3aff",
+ "id": "cfd34e4d",
"metadata": {
"editable": true
},
@@ -789,13 +789,10 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "19ea825b",
+ "id": "35da6db9",
"metadata": {
"collapsed": false,
- "editable": true,
- "jupyter": {
- "outputs_hidden": false
- }
+ "editable": true
},
"outputs": [],
"source": [
@@ -870,7 +867,7 @@
},
{
"cell_type": "markdown",
- "id": "85080d04",
+ "id": "2c678f39",
"metadata": {
"editable": true
},
@@ -882,7 +879,7 @@
},
{
"cell_type": "markdown",
- "id": "cefa4cb7",
+ "id": "cf9d2b92",
"metadata": {
"editable": true
},
@@ -897,7 +894,7 @@
},
{
"cell_type": "markdown",
- "id": "10147fa0",
+ "id": "e1a5fd8a",
"metadata": {
"editable": true
},
@@ -909,7 +906,7 @@
},
{
"cell_type": "markdown",
- "id": "2405041d",
+ "id": "7f511458",
"metadata": {
"editable": true
},
@@ -919,7 +916,7 @@
},
{
"cell_type": "markdown",
- "id": "a1a6b707",
+ "id": "c40d1b84",
"metadata": {
"editable": true
},
@@ -931,7 +928,7 @@
},
{
"cell_type": "markdown",
- "id": "ae672df8",
+ "id": "045cf436",
"metadata": {
"editable": true
},
@@ -941,7 +938,7 @@
},
{
"cell_type": "markdown",
- "id": "53ea5d3b",
+ "id": "2c532237",
"metadata": {
"editable": true
},
@@ -953,7 +950,7 @@
},
{
"cell_type": "markdown",
- "id": "19d5965a",
+ "id": "ac5cf334",
"metadata": {
"editable": true
},
@@ -965,7 +962,7 @@
},
{
"cell_type": "markdown",
- "id": "20608e44",
+ "id": "e2bd47a2",
"metadata": {
"editable": true
},
@@ -980,7 +977,7 @@
},
{
"cell_type": "markdown",
- "id": "7a04cc8a",
+ "id": "ce9b602b",
"metadata": {
"editable": true
},
@@ -991,7 +988,7 @@
},
{
"cell_type": "markdown",
- "id": "77a30599",
+ "id": "71339dde",
"metadata": {
"editable": true
},
@@ -1011,7 +1008,7 @@
},
{
"cell_type": "markdown",
- "id": "e97e9982",
+ "id": "93b3d2ca",
"metadata": {
"editable": true
},
@@ -1023,7 +1020,7 @@
},
{
"cell_type": "markdown",
- "id": "6b89ea8c",
+ "id": "b13455c8",
"metadata": {
"editable": true
},
@@ -1033,7 +1030,7 @@
},
{
"cell_type": "markdown",
- "id": "148873d3",
+ "id": "74108a43",
"metadata": {
"editable": true
},
@@ -1045,7 +1042,7 @@
},
{
"cell_type": "markdown",
- "id": "b008fef3",
+ "id": "ee5ebfa5",
"metadata": {
"editable": true
},
@@ -1074,7 +1071,7 @@
},
{
"cell_type": "markdown",
- "id": "82103acd",
+ "id": "6eb960f4",
"metadata": {
"editable": true
},
@@ -1101,7 +1098,7 @@
},
{
"cell_type": "markdown",
- "id": "4fea6570",
+ "id": "cc951a64",
"metadata": {
"editable": true
},
@@ -1112,13 +1109,10 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "ac796dd0",
+ "id": "a08edf11",
"metadata": {
"collapsed": false,
- "editable": true,
- "jupyter": {
- "outputs_hidden": false
- }
+ "editable": true
},
"outputs": [],
"source": [
@@ -1155,7 +1149,7 @@
},
{
"cell_type": "markdown",
- "id": "db53512b",
+ "id": "c5f84b6d",
"metadata": {
"editable": true
},
@@ -1172,7 +1166,7 @@
},
{
"cell_type": "markdown",
- "id": "502d1308",
+ "id": "3e6537bc",
"metadata": {
"editable": true
},
@@ -1195,7 +1189,7 @@
},
{
"cell_type": "markdown",
- "id": "8c2bb770",
+ "id": "c4b0d7d0",
"metadata": {
"editable": true
},
@@ -1209,7 +1203,7 @@
},
{
"cell_type": "markdown",
- "id": "ee7c0203",
+ "id": "2710e74d",
"metadata": {
"editable": true
},
@@ -1228,7 +1222,7 @@
},
{
"cell_type": "markdown",
- "id": "672a491b",
+ "id": "d33e794c",
"metadata": {
"editable": true
},
@@ -1238,7 +1232,7 @@
},
{
"cell_type": "markdown",
- "id": "8e3af41a",
+ "id": "6940c156",
"metadata": {
"editable": true
},
@@ -1250,7 +1244,7 @@
},
{
"cell_type": "markdown",
- "id": "3c0db076",
+ "id": "0a9fa0f9",
"metadata": {
"editable": true
},
@@ -1264,7 +1258,7 @@
},
{
"cell_type": "markdown",
- "id": "a6c05d18",
+ "id": "2ac8bc02",
"metadata": {
"editable": true
},
@@ -1276,7 +1270,7 @@
},
{
"cell_type": "markdown",
- "id": "ff6c524a",
+ "id": "f6197461",
"metadata": {
"editable": true
},
@@ -1286,7 +1280,7 @@
},
{
"cell_type": "markdown",
- "id": "ccca66ed",
+ "id": "7fa22eba",
"metadata": {
"editable": true
},
@@ -1298,7 +1292,7 @@
},
{
"cell_type": "markdown",
- "id": "da2a1c44",
+ "id": "a70285f6",
"metadata": {
"editable": true
},
@@ -1315,7 +1309,7 @@
},
{
"cell_type": "markdown",
- "id": "2f2704ab",
+ "id": "bf5774b8",
"metadata": {
"editable": true
},
@@ -1325,7 +1319,7 @@
},
{
"cell_type": "markdown",
- "id": "e7286128",
+ "id": "8ca229ff",
"metadata": {
"editable": true
},
@@ -1341,7 +1335,7 @@
},
{
"cell_type": "markdown",
- "id": "560e14d4",
+ "id": "b6bf96ea",
"metadata": {
"editable": true
},
@@ -1351,7 +1345,7 @@
},
{
"cell_type": "markdown",
- "id": "5f35139c",
+ "id": "d49ba9c3",
"metadata": {
"editable": true
},
@@ -1367,7 +1361,7 @@
},
{
"cell_type": "markdown",
- "id": "a5732633",
+ "id": "183d0b9c",
"metadata": {
"editable": true
},
@@ -1377,7 +1371,7 @@
},
{
"cell_type": "markdown",
- "id": "cb54add9",
+ "id": "5442ca49",
"metadata": {
"editable": true
},
@@ -1393,7 +1387,7 @@
},
{
"cell_type": "markdown",
- "id": "f03fc8ea",
+ "id": "823779a1",
"metadata": {
"editable": true
},
@@ -1403,7 +1397,7 @@
},
{
"cell_type": "markdown",
- "id": "d686fb8e",
+ "id": "bc9b3b47",
"metadata": {
"editable": true
},
@@ -1420,7 +1414,7 @@
},
{
"cell_type": "markdown",
- "id": "3b44a880",
+ "id": "db89583c",
"metadata": {
"editable": true
},
@@ -1432,7 +1426,7 @@
},
{
"cell_type": "markdown",
- "id": "d2dcc6d7",
+ "id": "cbb0223f",
"metadata": {
"editable": true
},
@@ -1444,7 +1438,7 @@
},
{
"cell_type": "markdown",
- "id": "a2f18583",
+ "id": "3388eb62",
"metadata": {
"editable": true
},
@@ -1456,7 +1450,7 @@
},
{
"cell_type": "markdown",
- "id": "1b1b3ce0",
+ "id": "37c056f5",
"metadata": {
"editable": true
},
@@ -1466,7 +1460,7 @@
},
{
"cell_type": "markdown",
- "id": "d96fcd4a",
+ "id": "bbbad4ae",
"metadata": {
"editable": true
},
@@ -1478,7 +1472,7 @@
},
{
"cell_type": "markdown",
- "id": "98506ca6",
+ "id": "23ebbb1c",
"metadata": {
"editable": true
},
@@ -1490,7 +1484,7 @@
},
{
"cell_type": "markdown",
- "id": "f7df719c",
+ "id": "bdcfe054",
"metadata": {
"editable": true
},
@@ -1502,7 +1496,7 @@
},
{
"cell_type": "markdown",
- "id": "1263babe",
+ "id": "3f26e5a9",
"metadata": {
"editable": true
},
@@ -1512,7 +1506,7 @@
},
{
"cell_type": "markdown",
- "id": "13a792f3",
+ "id": "ab396718",
"metadata": {
"editable": true
},
@@ -1524,7 +1518,7 @@
},
{
"cell_type": "markdown",
- "id": "ff69a1d7",
+ "id": "8ef06c70",
"metadata": {
"editable": true
},
@@ -1534,7 +1528,7 @@
},
{
"cell_type": "markdown",
- "id": "7e4a31aa",
+ "id": "13417de5",
"metadata": {
"editable": true
},
@@ -1546,7 +1540,7 @@
},
{
"cell_type": "markdown",
- "id": "50768853",
+ "id": "0e6345d4",
"metadata": {
"editable": true
},
@@ -1563,7 +1557,7 @@
},
{
"cell_type": "markdown",
- "id": "e70b34dc",
+ "id": "83a8abf0",
"metadata": {
"editable": true
},
@@ -1575,7 +1569,7 @@
},
{
"cell_type": "markdown",
- "id": "e505b123",
+ "id": "6f8e392e",
"metadata": {
"editable": true
},
@@ -1587,7 +1581,7 @@
},
{
"cell_type": "markdown",
- "id": "e143b570",
+ "id": "f8af87f5",
"metadata": {
"editable": true
},
@@ -1597,7 +1591,7 @@
},
{
"cell_type": "markdown",
- "id": "5135ebb4",
+ "id": "a00ee63c",
"metadata": {
"editable": true
},
@@ -1609,7 +1603,7 @@
},
{
"cell_type": "markdown",
- "id": "07ce9ec4",
+ "id": "489ab15e",
"metadata": {
"editable": true
},
@@ -1620,7 +1614,7 @@
},
{
"cell_type": "markdown",
- "id": "80932acf",
+ "id": "6e6bc889",
"metadata": {
"editable": true
},
@@ -1632,7 +1626,7 @@
},
{
"cell_type": "markdown",
- "id": "73eeda35",
+ "id": "06429f0f",
"metadata": {
"editable": true
},
@@ -1642,7 +1636,7 @@
},
{
"cell_type": "markdown",
- "id": "47aff789",
+ "id": "1ddbfde2",
"metadata": {
"editable": true
},
@@ -1654,7 +1648,7 @@
},
{
"cell_type": "markdown",
- "id": "2460c817",
+ "id": "3a910378",
"metadata": {
"editable": true
},
@@ -1664,7 +1658,7 @@
},
{
"cell_type": "markdown",
- "id": "5e3b0d9c",
+ "id": "b03e080e",
"metadata": {
"editable": true
},
@@ -1676,7 +1670,7 @@
},
{
"cell_type": "markdown",
- "id": "fbf2cb86",
+ "id": "925e17ec",
"metadata": {
"editable": true
},
@@ -1687,7 +1681,7 @@
},
{
"cell_type": "markdown",
- "id": "ebce4ea0",
+ "id": "6825e5f1",
"metadata": {
"editable": true
},
@@ -1699,7 +1693,7 @@
},
{
"cell_type": "markdown",
- "id": "2795da2d",
+ "id": "fe851d64",
"metadata": {
"editable": true
},
@@ -1717,7 +1711,7 @@
},
{
"cell_type": "markdown",
- "id": "6a26ccff",
+ "id": "40e14e19",
"metadata": {
"editable": true
},
@@ -1730,7 +1724,7 @@
},
{
"cell_type": "markdown",
- "id": "702137ae",
+ "id": "7a7da817",
"metadata": {
"editable": true
},
@@ -1742,7 +1736,7 @@
},
{
"cell_type": "markdown",
- "id": "6df222d6",
+ "id": "815d4798",
"metadata": {
"editable": true
},
@@ -1752,7 +1746,7 @@
},
{
"cell_type": "markdown",
- "id": "9cc370f4",
+ "id": "9d5e0b49",
"metadata": {
"editable": true
},
@@ -1765,7 +1759,7 @@
},
{
"cell_type": "markdown",
- "id": "9a71ac65",
+ "id": "ef2a20e5",
"metadata": {
"editable": true
},
@@ -1775,7 +1769,7 @@
},
{
"cell_type": "markdown",
- "id": "c4242498",
+ "id": "7ce0f92d",
"metadata": {
"editable": true
},
@@ -1787,7 +1781,7 @@
},
{
"cell_type": "markdown",
- "id": "6deb1e52",
+ "id": "5cca9be8",
"metadata": {
"editable": true
},
@@ -1800,7 +1794,7 @@
},
{
"cell_type": "markdown",
- "id": "4b9320dd",
+ "id": "fc5e1f2c",
"metadata": {
"editable": true
},
@@ -1813,7 +1807,7 @@
},
{
"cell_type": "markdown",
- "id": "43cca84c",
+ "id": "c9cb5c03",
"metadata": {
"editable": true
},
@@ -1825,7 +1819,7 @@
},
{
"cell_type": "markdown",
- "id": "91672a95",
+ "id": "3a4a16f9",
"metadata": {
"editable": true
},
@@ -1837,7 +1831,7 @@
},
{
"cell_type": "markdown",
- "id": "f3e9c0e4",
+ "id": "de9d17d1",
"metadata": {
"editable": true
},
@@ -1847,7 +1841,7 @@
},
{
"cell_type": "markdown",
- "id": "0cc6fab9",
+ "id": "a25cf6f8",
"metadata": {
"editable": true
},
@@ -1860,7 +1854,7 @@
},
{
"cell_type": "markdown",
- "id": "71f360d2",
+ "id": "cca2f8c4",
"metadata": {
"editable": true
},
@@ -1872,7 +1866,7 @@
},
{
"cell_type": "markdown",
- "id": "7a27d39c",
+ "id": "d02c6998",
"metadata": {
"editable": true
},
@@ -1884,7 +1878,7 @@
},
{
"cell_type": "markdown",
- "id": "d554f3b2",
+ "id": "19b2ac7d",
"metadata": {
"editable": true
},
@@ -1901,7 +1895,7 @@
},
{
"cell_type": "markdown",
- "id": "e9597e67",
+ "id": "f2d4c46a",
"metadata": {
"editable": true
},
@@ -1913,7 +1907,7 @@
},
{
"cell_type": "markdown",
- "id": "d0c80c08",
+ "id": "c084dde9",
"metadata": {
"editable": true
},
@@ -1923,7 +1917,7 @@
},
{
"cell_type": "markdown",
- "id": "2b33f0e5",
+ "id": "3bacabfd",
"metadata": {
"editable": true
},
@@ -1935,7 +1929,7 @@
},
{
"cell_type": "markdown",
- "id": "7db2a530",
+ "id": "6ac8a1a8",
"metadata": {
"editable": true
},
@@ -1945,7 +1939,7 @@
},
{
"cell_type": "markdown",
- "id": "177b3058",
+ "id": "43eb347b",
"metadata": {
"editable": true
},
@@ -1957,7 +1951,7 @@
},
{
"cell_type": "markdown",
- "id": "304c0b90",
+ "id": "11e0a485",
"metadata": {
"editable": true
},
@@ -1969,7 +1963,7 @@
},
{
"cell_type": "markdown",
- "id": "14936ac3",
+ "id": "012a0ea1",
"metadata": {
"editable": true
},
@@ -1985,7 +1979,7 @@
},
{
"cell_type": "markdown",
- "id": "797ca0aa",
+ "id": "1f8bf06b",
"metadata": {
"editable": true
},
@@ -1997,7 +1991,7 @@
},
{
"cell_type": "markdown",
- "id": "a469a907",
+ "id": "afbc2061",
"metadata": {
"editable": true
},
@@ -2009,7 +2003,7 @@
},
{
"cell_type": "markdown",
- "id": "e9452a64",
+ "id": "22d903c1",
"metadata": {
"editable": true
},
@@ -2019,7 +2013,7 @@
},
{
"cell_type": "markdown",
- "id": "699b712d",
+ "id": "09655feb",
"metadata": {
"editable": true
},
@@ -2031,7 +2025,7 @@
},
{
"cell_type": "markdown",
- "id": "e4af99e7",
+ "id": "976449ac",
"metadata": {
"editable": true
},
@@ -2041,7 +2035,7 @@
},
{
"cell_type": "markdown",
- "id": "a5a8ee8c",
+ "id": "3aeb28d8",
"metadata": {
"editable": true
},
@@ -2053,7 +2047,7 @@
},
{
"cell_type": "markdown",
- "id": "03c00df5",
+ "id": "59348b08",
"metadata": {
"editable": true
},
@@ -2070,7 +2064,7 @@
},
{
"cell_type": "markdown",
- "id": "023042c7",
+ "id": "a3a38146",
"metadata": {
"editable": true
},
@@ -2082,7 +2076,7 @@
},
{
"cell_type": "markdown",
- "id": "b0cd7107",
+ "id": "4de2d8d6",
"metadata": {
"editable": true
},
@@ -2094,7 +2088,7 @@
},
{
"cell_type": "markdown",
- "id": "4c34ce5f",
+ "id": "5539ce38",
"metadata": {
"editable": true
},
@@ -2104,7 +2098,7 @@
},
{
"cell_type": "markdown",
- "id": "d489ea51",
+ "id": "519481d2",
"metadata": {
"editable": true
},
@@ -2116,7 +2110,7 @@
},
{
"cell_type": "markdown",
- "id": "9921bc50",
+ "id": "433bee8f",
"metadata": {
"editable": true
},
@@ -2126,7 +2120,7 @@
},
{
"cell_type": "markdown",
- "id": "c568250e",
+ "id": "afea6f71",
"metadata": {
"editable": true
},
@@ -2138,7 +2132,7 @@
},
{
"cell_type": "markdown",
- "id": "21d0b485",
+ "id": "32b32b42",
"metadata": {
"editable": true
},
@@ -2148,19 +2142,19 @@
},
{
"cell_type": "markdown",
- "id": "6722483b",
+ "id": "54a0c1f6",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta}+\\frac{n}{2}\\lambda sgn(\\boldsymbol{\\theta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+ "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta}+\\frac{n}{2}\\lambda sgn(\\boldsymbol{\\theta})=\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "9777f5cc",
+ "id": "ec7d68de",
"metadata": {
"editable": true
},
@@ -2170,19 +2164,19 @@
},
{
"cell_type": "markdown",
- "id": "23101fe6",
+ "id": "d26bacac",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta}+\\lambda sgn(\\boldsymbol{\\theta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+ "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta}+\\lambda sgn(\\boldsymbol{\\theta})=\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "4f604170",
+ "id": "ac45c3e5",
"metadata": {
"editable": true
},
@@ -2192,7 +2186,7 @@
},
{
"cell_type": "markdown",
- "id": "5c7b9070",
+ "id": "451fbbf2",
"metadata": {
"editable": true
},
@@ -2211,7 +2205,7 @@
},
{
"cell_type": "markdown",
- "id": "0af8105c",
+ "id": "e0569839",
"metadata": {
"editable": true
},
@@ -2231,7 +2225,7 @@
},
{
"cell_type": "markdown",
- "id": "c93d44b0",
+ "id": "de0e44d6",
"metadata": {
"editable": true
},
@@ -2247,7 +2241,7 @@
},
{
"cell_type": "markdown",
- "id": "b1c868cd",
+ "id": "bae50b3b",
"metadata": {
"editable": true
},
@@ -2263,7 +2257,7 @@
},
{
"cell_type": "markdown",
- "id": "7c257671",
+ "id": "2cc7033f",
"metadata": {
"editable": true
},
@@ -2274,7 +2268,7 @@
},
{
"cell_type": "markdown",
- "id": "ddc77aef",
+ "id": "c4ca8926",
"metadata": {
"editable": true
},
@@ -2286,7 +2280,7 @@
},
{
"cell_type": "markdown",
- "id": "026777a7",
+ "id": "1eaf376f",
"metadata": {
"editable": true
},
@@ -2296,7 +2290,7 @@
},
{
"cell_type": "markdown",
- "id": "3707790b",
+ "id": "9c025697",
"metadata": {
"editable": true
},
@@ -2308,7 +2302,7 @@
},
{
"cell_type": "markdown",
- "id": "13ab6bec",
+ "id": "2d64abe7",
"metadata": {
"editable": true
},
@@ -2318,7 +2312,7 @@
},
{
"cell_type": "markdown",
- "id": "0a6d69ef",
+ "id": "b83c29d5",
"metadata": {
"editable": true
},
@@ -2330,7 +2324,7 @@
},
{
"cell_type": "markdown",
- "id": "336ec1b1",
+ "id": "cfcc809e",
"metadata": {
"editable": true
},
@@ -2352,7 +2346,7 @@
},
{
"cell_type": "markdown",
- "id": "3081ba44",
+ "id": "bdaa56d1",
"metadata": {
"editable": true
},
@@ -2365,7 +2359,7 @@
},
{
"cell_type": "markdown",
- "id": "5c3faff2",
+ "id": "ffa4afe6",
"metadata": {
"editable": true
},
@@ -2378,7 +2372,7 @@
},
{
"cell_type": "markdown",
- "id": "68fd52c0",
+ "id": "5c6b9239",
"metadata": {
"editable": true
},
@@ -2388,7 +2382,7 @@
},
{
"cell_type": "markdown",
- "id": "5e29a567",
+ "id": "5ddb446d",
"metadata": {
"editable": true
},
@@ -2406,7 +2400,7 @@
},
{
"cell_type": "markdown",
- "id": "f787c3a2",
+ "id": "9a0a5b04",
"metadata": {
"editable": true
},
@@ -2416,7 +2410,7 @@
},
{
"cell_type": "markdown",
- "id": "1c5e1214",
+ "id": "6fe881ee",
"metadata": {
"editable": true
},
@@ -2431,7 +2425,7 @@
},
{
"cell_type": "markdown",
- "id": "1cdd5ba1",
+ "id": "83b83194",
"metadata": {
"editable": true
},
@@ -2441,7 +2435,7 @@
},
{
"cell_type": "markdown",
- "id": "276e07a3",
+ "id": "09e77119",
"metadata": {
"editable": true
},
@@ -2455,7 +2449,7 @@
},
{
"cell_type": "markdown",
- "id": "80fe7f13",
+ "id": "afdce492",
"metadata": {
"editable": true
},
@@ -2465,7 +2459,7 @@
},
{
"cell_type": "markdown",
- "id": "e8838b8f",
+ "id": "b6e29c29",
"metadata": {
"editable": true
},
@@ -2479,7 +2473,7 @@
},
{
"cell_type": "markdown",
- "id": "2ed6fbd0",
+ "id": "ad6e09d7",
"metadata": {
"editable": true
},
@@ -2494,7 +2488,7 @@
},
{
"cell_type": "markdown",
- "id": "0796429c",
+ "id": "a514df33",
"metadata": {
"editable": true
},
@@ -2511,7 +2505,7 @@
},
{
"cell_type": "markdown",
- "id": "fc44df4f",
+ "id": "ae758019",
"metadata": {
"editable": true
},
@@ -2523,7 +2517,7 @@
},
{
"cell_type": "markdown",
- "id": "c88146a9",
+ "id": "1bcffe5e",
"metadata": {
"editable": true
},
@@ -2537,7 +2531,7 @@
},
{
"cell_type": "markdown",
- "id": "d1b85445",
+ "id": "67467328",
"metadata": {
"editable": true
},
@@ -2552,7 +2546,7 @@
},
{
"cell_type": "markdown",
- "id": "8edde795",
+ "id": "4bac20b4",
"metadata": {
"editable": true
},
@@ -2564,7 +2558,7 @@
},
{
"cell_type": "markdown",
- "id": "f92e4583",
+ "id": "01e84611",
"metadata": {
"editable": true
},
@@ -2575,7 +2569,7 @@
},
{
"cell_type": "markdown",
- "id": "0bf80264",
+ "id": "8f742c3b",
"metadata": {
"editable": true
},
@@ -2603,7 +2597,7 @@
},
{
"cell_type": "markdown",
- "id": "5b6b24c7",
+ "id": "77acd12d",
"metadata": {
"editable": true
},
@@ -2625,7 +2619,7 @@
},
{
"cell_type": "markdown",
- "id": "e36297bd",
+ "id": "99ab46db",
"metadata": {
"editable": true
},
@@ -2647,7 +2641,7 @@
},
{
"cell_type": "markdown",
- "id": "f8ec331d",
+ "id": "5b4b6243",
"metadata": {
"editable": true
},
@@ -2659,7 +2653,7 @@
},
{
"cell_type": "markdown",
- "id": "0b0d0c67",
+ "id": "2ba66a00",
"metadata": {
"editable": true
},
@@ -2696,7 +2690,7 @@
},
{
"cell_type": "markdown",
- "id": "bdfac28b",
+ "id": "c9a65dbe",
"metadata": {
"editable": true
},
@@ -2724,7 +2718,7 @@
},
{
"cell_type": "markdown",
- "id": "2032fb83",
+ "id": "7efa8d14",
"metadata": {
"editable": true
},
@@ -2754,7 +2748,7 @@
},
{
"cell_type": "markdown",
- "id": "f5a00d18",
+ "id": "93d8901e",
"metadata": {
"editable": true
},
@@ -2778,13 +2772,10 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "68f97538",
+ "id": "6a775d3c",
"metadata": {
"collapsed": false,
- "editable": true,
- "jupyter": {
- "outputs_hidden": false
- }
+ "editable": true
},
"outputs": [],
"source": [
@@ -2794,7 +2785,7 @@
},
{
"cell_type": "markdown",
- "id": "a7fbe32f",
+ "id": "b8360996",
"metadata": {
"editable": true
},
@@ -2805,7 +2796,7 @@
},
{
"cell_type": "markdown",
- "id": "1b64af25",
+ "id": "0b4a5c74",
"metadata": {
"editable": true
},
@@ -2817,7 +2808,7 @@
},
{
"cell_type": "markdown",
- "id": "ef56e132",
+ "id": "f27ef437",
"metadata": {
"editable": true
},
@@ -2827,7 +2818,7 @@
},
{
"cell_type": "markdown",
- "id": "6d7f0c20",
+ "id": "6a7d8179",
"metadata": {
"editable": true
},
@@ -2839,7 +2830,7 @@
},
{
"cell_type": "markdown",
- "id": "3ac315d2",
+ "id": "24b3bcb3",
"metadata": {
"editable": true
},
@@ -2853,7 +2844,7 @@
},
{
"cell_type": "markdown",
- "id": "040d030b",
+ "id": "196152f9",
"metadata": {
"editable": true
},
@@ -2869,7 +2860,7 @@
},
{
"cell_type": "markdown",
- "id": "1cb9c213",
+ "id": "6b935471",
"metadata": {
"editable": true
},
@@ -2879,7 +2870,7 @@
},
{
"cell_type": "markdown",
- "id": "7f6ab192",
+ "id": "d5261a7f",
"metadata": {
"editable": true
},
@@ -2891,7 +2882,7 @@
},
{
"cell_type": "markdown",
- "id": "bc154729",
+ "id": "398173a2",
"metadata": {
"editable": true
},
@@ -2901,7 +2892,7 @@
},
{
"cell_type": "markdown",
- "id": "cda14c18",
+ "id": "0f29a1a0",
"metadata": {
"editable": true
},
@@ -2913,7 +2904,7 @@
},
{
"cell_type": "markdown",
- "id": "ffc6db98",
+ "id": "22f27a92",
"metadata": {
"editable": true
},
@@ -2927,7 +2918,7 @@
},
{
"cell_type": "markdown",
- "id": "f5dd4f8d",
+ "id": "a2b8898e",
"metadata": {
"editable": true
},
@@ -2937,7 +2928,7 @@
},
{
"cell_type": "markdown",
- "id": "54cf8a09",
+ "id": "42b415d8",
"metadata": {
"editable": true
},
@@ -2948,7 +2939,7 @@
},
{
"cell_type": "markdown",
- "id": "c06fcb28",
+ "id": "b522b609",
"metadata": {
"editable": true
},
@@ -2963,7 +2954,7 @@
},
{
"cell_type": "markdown",
- "id": "16b2e122",
+ "id": "2e2e2e39",
"metadata": {
"editable": true
},
@@ -2973,7 +2964,7 @@
},
{
"cell_type": "markdown",
- "id": "9e48ac99",
+ "id": "6f14f45f",
"metadata": {
"editable": true
},
@@ -2985,7 +2976,7 @@
},
{
"cell_type": "markdown",
- "id": "9a536e9c",
+ "id": "d9210eed",
"metadata": {
"editable": true
},
@@ -2997,7 +2988,7 @@
},
{
"cell_type": "markdown",
- "id": "e9afa223",
+ "id": "5050ec60",
"metadata": {
"editable": true
},
@@ -3012,7 +3003,7 @@
},
{
"cell_type": "markdown",
- "id": "f4f76d91",
+ "id": "04ac7d83",
"metadata": {
"editable": true
},
@@ -3024,38 +3015,13 @@
},
{
"cell_type": "code",
- "execution_count": 18,
- "id": "801ec4de",
+ "execution_count": 4,
+ "id": "6b2e19f6",
"metadata": {
"collapsed": false,
- "editable": true,
- "jupyter": {
- "outputs_hidden": false
- }
+ "editable": true
},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Eigenvalues of Hessian Matrix:[0.29828983 4.4698111 ]\n",
- "[[4.]\n",
- " [3.]]\n",
- "[[4.]\n",
- " [3.]]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHFCAYAAADxOP3DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAABEnUlEQVR4nO3dd3hUZeL28XvSA4RIElpIAtmIhCKgqKiINAERshQVsSCwFhQUKTYWXUGFiBVeEQGV4o+mSChiA6UIi7Ig2BVBKSGAFDGhGUjyvH+MM2SSmTSSTCbn+7muudg5c8pzMjmb26fajDFGAAAAFuXn7QIAAAB4E2EIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIcOPbb7/VXXfdpYSEBIWGhio0NFQNGzbU4MGDtWXLlnIrx9ixY2Wz2Vy2NWjQQAMHDizT627cuFFjx47Vn3/+eV7nGThwoBo0aFAqZaoo9u/fr7Fjx+rrr7/2dlEqjLVr18pms2nt2rXeLgpQIgHeLgBQ0UyfPl0PPPCAGjVqpIceekhNmzaVzWbTTz/9pAULFujyyy/Xzp07lZCQ4JXyLVmyRNWrVy/Ta2zcuFHjxo3TwIEDdcEFF5TptXzN/v37NW7cODVo0EAtW7b0dnEAlALCEJDLf//7Xw0ZMkTdu3fXe++9p6CgIOdnHTt21NChQ7Vo0SKFhoYWeJ5Tp06pSpUqZVLGSy65pEzOCwBWRTMZkMuECRPk7++v6dOnuwSh3G6++WZFR0c73w8cOFDVqlXTd999py5duigsLEydOnWSJK1atUo9e/ZUTEyMQkJCdOGFF2rw4ME6cuRIvvN+8MEHatmypYKDgxUfH68XX3zR7fXdNZNlZGTo4YcfVnx8vIKCglSvXj0NHz5cJ0+edNnPZrPpgQce0P/93/+pcePGqlKlilq0aKEVK1Y49xk7dqweeeQRSVJ8fLxsNluRmkBmz56tRo0aKTg4WI0bN9bbb7/tdr8zZ87o2WefVWJiooKDg1WzZk0NGjRIhw8fdtlv9erVat++vSIjIxUaGqq4uDjdeOONOnXqlHOfzMxMPf3002rcuLFCQkIUGRmpDh06aOPGjc59jDGaOnWqWrZsqdDQUNWoUUM33XSTfvvtN5frtW/fXs2aNdPmzZvVtm1bValSRf/4xz/03HPPKScnR5K9Oejyyy+XJA0aNMj5sxk7dmyBP5uDBw9q8ODBiomJUVBQkOLj4zVu3DhlZWU5y3jDDTcoMjJSe/fudR536tQpNW3aVI0bN3Z+lzt37tSgQYPUsGFDValSRfXq1VNSUpK+++47l2s6mq7mz5+vxx57THXr1lW1atWUlJSk33//XcePH9e9996rqKgoRUVFadCgQTpx4oTLORy/L9OnT9dFF12k4OBgNWnSRAsXLizwfh22bNmif/7zn4qIiFBISIguueQSvfvuu0U6FihXBoAxxpisrCwTGhpqrrrqqmIdN2DAABMYGGgaNGhgkpOTzWeffWY++eQTY4wxr7/+uklOTjbLly8369atM3PmzDEtWrQwjRo1MmfOnHGe49NPPzX+/v7mmmuuMSkpKWbRokXm8ssvN3FxcSbvY1q/fn0zYMAA5/uTJ0+ali1bmqioKPPyyy+bTz/91EyePNmEh4ebjh07mpycHOe+kkyDBg3MFVdcYd59913z4Ycfmvbt25uAgADz66+/GmOMSU1NNQ8++KCRZFJSUswXX3xhvvjiC5Oenu7xZzBr1iwjyfTs2dO8//77Zu7cuebCCy80sbGxpn79+s79srOzzfXXX2+qVq1qxo0bZ1atWmXefPNNU69ePdOkSRNz6tQpY4wxu3btMiEhIaZz585m6dKlZu3atWbevHmmf//+5tixY8YYY86ePWs6dOhgAgICzMMPP2w+/PBDs3z5cvPvf//bLFiwwHnNe+65xwQGBppRo0aZjz/+2MyfP98kJiaa2rVrm4MHDzr3a9eunYmMjDQNGzY006ZNM6tWrTJDhgwxksycOXOMMcakp6c77/WJJ55w/mxSU1M9/mwOHDjg/DlMnz7dfPrpp+aZZ54xwcHBZuDAgc79jhw5YmJiYkzr1q2dvxsDBgwwoaGh5ttvv3Xut27dOjNq1Cjz3nvvmXXr1pklS5aYXr16mdDQUPPzzz8791uzZo2RZOrXr28GDhxoPv74YzNt2jRTrVo106FDB9O5c2fz8MMPm5UrV5qJEycaf39/8+CDD7qUXZKJjY01TZo0MQsWLDDLly83119/vZFkFi1alO9aa9ascW5bvXq1CQoKMm3btjXvvPOO+fjjj83AgQONJDNr1iyPPy/AGwhDwN8OHjxoJJl+/frl+ywrK8ucPXvW+codMAYMGGAkmZkzZxZ4/pycHHP27FmzZ88eI8ksW7bM+Vnr1q1NdHS0OX36tHNbRkaGiYiIKDQMJScnGz8/P7N582aX/d577z0jyXz44YfObZJM7dq1TUZGhst9+/n5meTkZOe2F154wUgyu3btKvCejLEHnOjoaHPppZe6/Fx2795tAgMDXcLQggULjCSzePFil3Ns3rzZSDJTp051KfvXX3/t8bpvv/22kWTeeOMNj/t88cUXRpJ56aWXXLanpqaa0NBQ8+ijjzq3tWvXzkgymzZtctm3SZMmpmvXrvnKWtQ/6IMHDzbVqlUze/bscdn+4osvGknmhx9+cG7bsGGDCQgIMMOHDzczZ840ksybb75Z4PmzsrLMmTNnTMOGDc2IESOc2x0BJSkpyWX/4cOHG0lm2LBhLtt79eplIiIiXLZJMqGhoS6hMSsryyQmJpoLL7ww37Vyh6HExERzySWXmLNnz7qcs0ePHqZu3bomOzu7wPsCyhPNZEARtGrVSoGBgc7XSy+9lG+fG2+8Md+2Q4cO6b777lNsbKwCAgIUGBio+vXrS5J++uknSdLJkye1efNm9enTRyEhIc5jw8LClJSUVGjZVqxYoWbNmqlly5bKyspyvrp27eq2eatDhw4KCwtzvq9du7Zq1aqlPXv2FOlnkdf27du1f/9+3XbbbS4j3+rXr6+rr746X1kvuOACJSUluZS1ZcuWqlOnjrOsLVu2VFBQkO69917NmTMnX5OWJH300UcKCQnRv/71L49lW7FihWw2m+644w6X69WpU0ctWrTI97OpU6eOrrjiCpdtzZs3L/HPxlGGDh06KDo62qUM3bp1kyStW7fOuW+bNm00fvx4TZo0Sffff7/uuOMO3XXXXS7ny8rK0oQJE9SkSRMFBQUpICBAQUFB2rFjh/N3KrcePXq4vG/cuLEkqXv37vm2//HHH/mayjp16qTatWs73/v7++uWW27Rzp07tW/fPrf3vHPnTv3888+6/fbbnWV2vG644QYdOHBA27dvL/DnBpQnwhDwt6ioKIWGhrr9wzd//nxt3rxZy5cvd3tslSpV8o3wysnJUZcuXZSSkqJHH31Un332mf73v//pyy+/lCSdPn1aknTs2DHl5OSoTp06+c7rbltev//+u7799luXsBYYGKiwsDAZY/L1T4qMjMx3juDgYGd5iuvo0aMey5p32++//64///xTQUFB+cp78OBBZ1kTEhL06aefqlatWho6dKgSEhKUkJCgyZMnO891+PBhRUdHy8/P8/+N/f777zLGqHbt2vmu9+WXX5b5z8ZRhvfffz/f9Zs2bSpJ+cpw++23KygoSJmZmc6+W7mNHDlSTz75pHr16qX3339fmzZt0ubNm9WiRQu35YyIiHB57+gL52n7X3/95bK9oO/V8d27u2dJevjhh/Pd95AhQ9zeN+BNjCYD/ubv76+OHTtq5cqVOnDggOrWrev8rEmTJpKk3bt3uz0271xAkvT999/rm2++0ezZszVgwADn9p07d7rsV6NGDdlsNh08eDDfOdxty8sR4mbOnOnx87LkCBBFKX9UVJQiIyP18ccfuz1X7hqrtm3bqm3btsrOztaWLVv06quvavjw4apdu7b69eunmjVrasOGDcrJyfEYiKKiomSz2bR+/XoFBwfn+9zdttIWFRWl5s2ba/z48W4/z90ZPzs7W7fffrtq1Kih4OBg3XXXXfrvf//r0pl/7ty5uvPOOzVhwgSX8xw5cqRMpkEo6Ht1Fx6lc79zo0ePVp8+fdzu06hRo1IqIXD+CENALqNHj9ZHH32k++67T++9954CAwNLfC5HQMr7B3f69Oku76tWraorrrhCKSkpeuGFF5xNZcePH9f7779f6HV69OihCRMmKDIyUvHx8SUub26OMhelRqRRo0aqW7euFixYoJEjRzrve8+ePdq4caPLH/sePXpo4cKFys7OVuvWrYtUFn9/f7Vu3VqJiYmaN2+etm7dqn79+qlbt25asGCBZs+e7bGprEePHnruueeUlpamvn37Ful6hSnOz8ZRhg8//FAJCQmqUaNGgfs+9dRTWr9+vVauXKmqVavq2muv1SOPPOJSI2az2fL9Tn3wwQdKS0vThRdeWMy7Kdxnn32m33//3dlUlp2drXfeeUcJCQmKiYlxe0yjRo3UsGFDffPNN/lCG1AREYaAXNq0aaPXXntNDz74oC699FLde++9atq0qfz8/HTgwAEtXrxYkoo06WFiYqISEhL0+OOPyxijiIgIvf/++1q1alW+fZ955hldf/316ty5s0aNGqXs7GxNnDhRVatW1R9//FHgdYYPH67Fixfr2muv1YgRI9S8eXPl5ORo7969WrlypUaNGlXk4OFw8cUXS5ImT56sAQMGKDAwUI0aNXKpuXHw8/PTM888o7vvvlu9e/fWPffcoz///FNjx47N18TSr18/zZs3TzfccIMeeughXXHFFQoMDNS+ffu0Zs0a9ezZU71799a0adO0evVqde/eXXFxcfrrr7+cNV/XXXedJOnWW2/VrFmzdN9992n79u3q0KGDcnJytGnTJjVu3Fj9+vVTmzZtdO+992rQoEHasmWLrr32WlWtWlUHDhzQhg0bdPHFF+v+++8v1s/GMSv5vHnz1LhxY1WrVk3R0dEuoS+3p59+WqtWrdLVV1+tYcOGqVGjRvrrr7+0e/duffjhh5o2bZpiYmK0atUqJScn68knn3ROzZCcnKyHH35Y7du3V+/evSXZw9Xs2bOVmJio5s2b66uvvtILL7zgMZicr6ioKHXs2FFPPvmkqlatqqlTp+rnn38udHj99OnT1a1bN3Xt2lUDBw5UvXr19Mcff+inn37S1q1btWjRojIpL1AiXu7ADVRIX3/9tRk0aJCJj483wcHBJiQkxFx44YXmzjvvNJ999pnLvgMGDDBVq1Z1e54ff/zRdO7c2YSFhZkaNWqYm2++2ezdu9dIMk899ZTLvsuXLzfNmzc3QUFBJi4uzjz33HPmqaeeKnQ0mTHGnDhxwjzxxBOmUaNGJigoyISHh5uLL77YjBgxwmUkkCQzdOjQfOV0d87Ro0eb6Oho4+fnl2+kkDtvvvmmadiwoQkKCjIXXXSRmTlzphkwYIDLaDJj7EPiX3zxRdOiRQsTEhJiqlWrZhITE83gwYPNjh07jDH2UWC9e/c29evXN8HBwSYyMtK0a9fOLF++3OVcp0+fNv/5z3+c142MjDQdO3Y0GzdudNlv5syZpnXr1qZq1aomNDTUJCQkmDvvvNNs2bLFuU+7du1M06ZN892Xu3tYsGCBSUxMNIGBgW6/y7wOHz5shg0bZuLj401gYKCJiIgwrVq1MmPGjDEnTpww+/fvN7Vq1TIdO3Z0GWWVk5NjkpKSzAUXXOAc2Xfs2DFz1113mVq1apkqVaqYa665xqxfv960a9fOtGvXznmsY4RX7iHwxpybBiHv6EPH79rhw4ed2xy/L1OnTjUJCQkmMDDQJCYmmnnz5rkc6240mTHGfPPNN6Zv376mVq1aJjAw0NSpU8d07NjRTJs2rcCfF1DebMYY460gBgCouGw2m4YOHaopU6Z4uyhAmWI0GQAAsDTCEAAAsDQ6UAMA3KIXBayCmiEAAGBphCEAAGBphCEAAGBplugzlJOTo/379yssLMztsgkAAKDiMcbo+PHjha5DeL4sEYb279+v2NhYbxcDAACUQGpqapnNsi5ZJAw5lhBITU0t0jIKAADA+zIyMhQbG+t2KaDSZIkw5Ggaq169OmEIAAAfU9ZdXOhADQAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALI0wBAAALM3rYejzzz9XUlKSoqOjZbPZtHTpUo/7Dh48WDabTZMmTSq38gEAgMrN62Ho5MmTatGihaZMmVLgfkuXLtWmTZsUHR1dTiUDAABWEODtAnTr1k3dunUrcJ+0tDQ98MAD+uSTT9S9e/dyKhkAALACr9cMFSYnJ0f9+/fXI488oqZNm3q7OAAAoJLxes1QYSZOnKiAgAANGzasyMdkZmYqMzPT+T4jI6MsigYAACqBCl0z9NVXX2ny5MmaPXu2bDZbkY9LTk5WeHi48xUbG1uGpQQAAL6sQoeh9evX69ChQ4qLi1NAQIACAgK0Z88ejRo1Sg0aNPB43OjRo5Wenu58paamll+hAQCAT6nQzWT9+/fXdddd57Kta9eu6t+/vwYNGuTxuODgYAUHB5d18QAAQCXg9TB04sQJ7dy50/l+165d+vrrrxUREaG4uDhFRka67B8YGKg6deqoUaNG5V1UAABQCXk9DG3ZskUdOnRwvh85cqQkacCAAZo9e7aXSgUAAKzC62Goffv2MsYUef/du3eXXWEAAIDlVOgO1AAAAGWNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACzN62Ho888/V1JSkqKjo2Wz2bR06VLnZ2fPntVjjz2miy++WFWrVlV0dLTuvPNO7d+/33sFBgAAlYrXw9DJkyfVokULTZkyJd9np06d0tatW/Xkk09q69atSklJ0S+//KJ//vOfXigpAACojGzGGOPtQjjYbDYtWbJEvXr18rjP5s2bdcUVV2jPnj2Ki4sr0nkzMjIUHh6u9PR0Va9evZRKCwAAylJ5/f0OKLMzl5H09HTZbDZdcMEFHvfJzMxUZmam831GRkY5lAwAAPgirzeTFcdff/2lxx9/XLfddluBCTE5OVnh4eHOV2xsbDmWEgAA+BKfCUNnz55Vv379lJOTo6lTpxa47+jRo5Wenu58paamllMpAQCAr/GJZrKzZ8+qb9++2rVrl1avXl1ou2FwcLCCg4PLqXQAAMCXVfgw5AhCO3bs0Jo1axQZGentIgEAgErE62HoxIkT2rlzp/P9rl279PXXXysiIkLR0dG66aabtHXrVq1YsULZ2dk6ePCgJCkiIkJBQUHeKjYAAKgkvD60fu3aterQoUO+7QMGDNDYsWMVHx/v9rg1a9aoffv2RboGQ+sBAPA9lhla3759exWUxyrQNEgAAKAS8pnRZAAAAGWBMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACyNMAQAACzN62Ho888/V1JSkqKjo2Wz2bR06VKXz40xGjt2rKKjoxUaGqr27dvrhx9+8E5hAQBApeP1MHTy5Em1aNFCU6ZMcfv5888/r5dffllTpkzR5s2bVadOHXXu3FnHjx8v55ICAIDKKMDbBejWrZu6devm9jNjjCZNmqQxY8aoT58+kqQ5c+aodu3amj9/vgYPHlyeRQUAAJWQ18NQQXbt2qWDBw+qS5cuzm3BwcFq166dNm7c6DEMZWZmKjMz0/k+IyOjzMsKAAAKkJ0trV1rf0lS+/b2l79/vl1PHDyhTyf/oJRFx8qlaBU6DB08eFCSVLt2bZfttWvX1p49ezwel5ycrHHjxpVp2QAAQCGys6X166Vly6S33pJyd3F59lkpMlKaMUPq00d7/rtPKyb/qvdXV9WaoxfrjFpLKp/KjAodhhxsNpvLe2NMvm25jR49WiNHjnS+z8jIUGxsbJmVDwAA5JGSIj30kLRvn8ddzNGj0o03akTAq5qc9YCkGOdn8QF71SXxe03/vuyLWqHDUJ06dSTZa4jq1q3r3H7o0KF8tUW5BQcHKzg4uMzLBwCApTlqfg4ckOrWldq2tTd7paRIN90kGVPg4TZJRtLIrIl6Tffqquo/qUebY0oaEqvEG/6h4ycu0PTwsr+NCh2G4uPjVadOHa1atUqXXHKJJOnMmTNat26dJk6c6OXSAQBgYe5qfmJipFdekUaMKDQIOdgkxWmfjs79RNVvTyqbshbC62HoxIkT2rlzp/P9rl279PXXXysiIkJxcXEaPny4JkyYoIYNG6phw4aaMGGCqlSpottuu82LpQYAwMI81fykpcncfLM8d2TxrLrfiVIpWkl4PQxt2bJFHTp0cL539PUZMGCAZs+erUcffVSnT5/WkCFDdOzYMbVu3VorV65UWFiYt4oMAEDld+aMNHWq9OuvUkKCNGSIFBRkbxp76CH3NT/GyEglCkPK1R2mvNmMKWI9lg/LyMhQeHi40tPTVb16dW8XBwCAiid3/59ly6RFi6ScnHOf+/tLI0dKN9wg5arEKBUxMdLu3fmG2ZfX32+v1wwBAAAvK8rIr+xs6YUX9MO0DWpW2tefPNntfEPlhTAEAICVFXPkV+Pjm4p+bput4PPmmmfImwhDAABUVo6mr7Q06fBhqWZNqV69c0PgC+r/44ZNkr9ydCa4mgLPnJTN3XE2m73Z66WX7M1quWuboqKkdu2kxo0LnIG6vBGGAACoTHLP+jxvnj0E5RUTY2+aiogosGnMk6CObaWPP85f8+OYEHnSJHttT58+7uchqmAIQwAAVBaLFtlHfR05UuBuZt8+6cYbtbzKLepZkut06SLdfbf7eYYcQUiyB5/27UtyhXLFaDIAACqDRx+VXnihyLvnyKbDilJtuak5Koi/v3Tq1Llh9mVY88NoMgAAUDTvvVesICRJfjKqrcM6G1ZDASf+dN//x52RI+1BSPKZmp/C+Hm7AAAAwI0zZ+xNTg8+aP/3zBn3+2Vn25vGSijwrgH2SRILWABdkj34PPKI9PzzJb5WRUXNEAAAFc3DD9vX+Mo96eHDD9trZXKFkZysHP08+m01cddJuqh69rQ3b7nr/9O5s1S1qusM1JUQYQgAgIqkVy/7SLC8/p70MPN4plZU66cVS7P0wa+J6mRCtKAk13EMgXf08+nZ0ydGfpUFwhAAABXFO++4D0J/M5L8p72mfnpBWbLX0mToPNbqnDTpXOCpJP1/SoIwBABAWSvKqKvsbGnw4AJPY5MUoGw95fes/mjZUT1uC9c1d3eSmsUUb76g2FjXIfAWRxgCAKCsZGdLzzwjvfyydPz4ue2OSQ9zhZHj736ksPT0Ip32iSHHpFfbn9sweXLBS2p07y5dd13+GaghiTAEAEDZSEmRBgyQTpzI/9m+fdJNNynt369p4TeNtWJDuKL//FPzinruhATX93362IfX5+0EXbOmNHWqPSjBIyZdBADgfJ05Yw8dv/5qDyp160r9+hV4SI6kfYpVvHYpR/5qp7Vaqw6FX8vPTzp92v3IrjKeBLG8MekiAAC+4NFH7c1g2dnFOsxPUpxSNSLsTcV1TlT3++KkfxWh78+IEZ6HuFu4E/T5IAwBAFCQ7Gzps8+k//s/e5PXNdfYJ0IMCir2EhjuvDi9unRrO/ubwvr+9OwpvfjieV0P+TEDNQAAnqSkSBdcIHXtKs2dKy1dap/8MDTUPgHiyy+f/zXq1j33vx19f2JiXPcJD5cWLrRfH6WOPkMAAOTm6HezbJl9+LkHRvah7uelZk17/x53w+wrUd+fkqLPEAAA5S0lJf+IrLL02mvuQw59f8oVYQgAAMkehArqr5PHedcKPfKIdPPN53sWlALCEAAA2dn2GqHy6DlSvbr01lvM/VOBEIYAAJaWk5Wj7WPeVuOyaBobNUrq1k1au9b+vn17+8uC/X8qMsIQAKByKEan4xMHT2jVK99rxZKz+uDXRuqQU8KV3/397TVKkye7zjPk728fbfb88/b3nTqV5OwoJ4QhAIBvy86Wxo+3B5I//ji3Pc/6X7s37NOKyb9qxZqqWnP0Yp3Rlc5d01XCkUqOwJOc7DoD9ZAhnidGRIXD0HoAgG/IzrY3N61eLe3dK8XFSYGB0pQp0tGj+XY3NptkpHkXjdXEPbfq+8yGLp//I2CPkprtVo9bw3Tt4MYKanaRlJZWtH5Dfn72JjBHzQ/KRHn9/SYMAQAqrtxz/sycKWVkFOvwHNm0TzGK1y5JUpvq3yvpmmPqcX+sEm/4h2x+ucaEOUaTSa6ByGazv7/2WikiwnUGapQp5hkCAFhP7tqfDRukbduk48dLfDo/GcUpVZ/0nKpLX7pDEQktPO/saeX3mBj75It/N7eh8iEMAQC8K3ftz1tvnVf48eS6W6KkhBqF79inj339L2Z/thTCEACg/DkC0JIl0pw5Unp62V4v9/pfhWH2Z8shDAEAyld5L3kRG2uv3QE8IAwBAMpW7vl/duyQnnqqfK8/aRLNXCgQYQgAUHbKuxYot8hIacYMOj6jUIQhAEDZKObCp6UmMlIaNkwaM4YaIRQJYQgAUPrKc+HTiAj7tRo2ZPQXSoQwBAAoXDHW/ZKkI28tU1RZNI2FhUmdO0tVqthnoO7YkYVPcd4IQwCAgrnr95Nn3a+crBxtW7hd77/5u1ZsrqWGpzJLtvCpJ9WqSY88QtMXygRhCADgmad+P2lpMjfdpC29J2jGD1frgx0X6UBOY0mNJUnVdOj8rtu6tXTRRfZh8dT+oIxV+LXJsrKyNHbsWM2bN08HDx5U3bp1NXDgQD3xxBPy8/Mr0jlYmwwASiA7W2rQwONIsNzrfuXIX1V1Ql3qfq+k68+q24MXqs4/ryj6wqcONWvaV393rBEGS2Ntsr9NnDhR06ZN05w5c9S0aVNt2bJFgwYNUnh4uB566CFvFw8AfItj7a+1a+3v27f3XOuyfn2BQ+Id6369HP+qGt/TVu2GNlNw9SvP7TB5sj3UOBY6dbD9vTjq2LFSQoJ0+LA9BNWrR+dneEWFD0NffPGFevbsqe7du0uSGjRooAULFmjLli1eLhkA+JiUFOnee6WjR89te/bZfPPxZOzL0KrJP+jQ22t1fxFO+9D42tKtrfJ/wMKn8BEVPgxdc801mjZtmn755RdddNFF+uabb7RhwwZNmjTJ20UDgIqtqDM/Hz0qc+ON+rD105q04wat++NindVVaqdM3a9xhV+noHW/WPgUPqDCh6HHHntM6enpSkxMlL+/v7KzszV+/HjdeuutHo/JzMxUZmam831GRkZ5FBUAKo4SzPx88aYZWq1/K0f+ujBwt1o1y9Zfv9VScMZh2dz1+7HZ7LU8ha37xcKnqOCK1gPZi9555x3NnTtX8+fP19atWzVnzhy9+OKLmjNnjsdjkpOTFR4e7nzFxsaWY4kBwMscI8CKEYRskuK0T/OvfFXbP96lHWca6KWtnRQy83XZpHP9fJwH/P2edb9QCVT40WSxsbF6/PHHNXToUOe2Z599VnPnztXPP//s9hh3NUOxsbGMJgPg24oy8WEhI8AKNX++lLfm3V0tU2ws/X5Q5hhN9rdTp07lG0Lv7++vnJwcj8cEBwcrODi4rIsGAOWnCBMfStLZlasVeD4zP7vr/0O/H1RyFT4MJSUlafz48YqLi1PTpk21bds2vfzyy/rXv/7l7aIBQPkoYOJD3XSTMibP0rIfEvT+B/6qtm+vZpb0OgX1/6HfDyqxCt9Mdvz4cT355JNasmSJDh06pOjoaN166636z3/+o6CgoCKdg0kXAfgEd81gUrEmPmyntVqrDiW7/uLFNHuhQimvv98VPgyVBsIQgArLEYCWLZPmzpWOHDn3WUyMdM89nofE53J34BzVuSJOSQNq6Iqne8hWnJmf88wzBFQU9BkCgMqukOHvJi1Neuop2dx+6urNOYHSre3tbyILmPnZGPvMz1lZ9m0FzUANWARhCAC8wVM/oFxsxihHKlIYcun4zMzPQLEUq5ksNTXVJ+fsoZkMQLkqbAh8CYa/G3kIRY6JD3ftcj/MnhFg8GHl9fe7WJMuJiYm6sknn9TJkyfLqjwA4NtSUuxBp0MH6bbb7P82aGDf/rfDbywt9jxAJZr40DEC7NZbaQoDClCsMLRq1SqtXLlSDRs21KxZs8qqTADgmzzM/GzS0mRuvEkLmjytS6r8rGH3nyn+uceNs6/qnltMjL05jGYv4LyUaDTZ22+/rTFjxigqKkqvvPKK2lfwuSdoJgNQajw1PRXS9JV7CPy1Wqc16lS06+VuBpNo9oKlVMhmMoc777xTv/zyi5KSktS9e3f17t1bO3fuLO2yAUDFUlAT2Pr1BTZ9+ckoTqn64Iapeve7pvaAk7fZyxNHMxjNXkCZKPFCrcYYdenSRffee6+WL1+uZs2aadSoUTp+/Hhplg8Ayld2trR2rbRggf3f7Gz7dk9NYPvSZG68UW93ml2k019/R5RqNqttX0ZDKjgQxcbSDAaUg2INrZ82bZo2b96szZs366effpK/v7+aN2+uoUOHqmXLlpo3b56aNGmiJUuW6LLLLiurMgNA2fC0/tcrr0gjRrgdBm+TUY5s6przYdGu4RgC72n4e82a0u2329cCoxkMKBfF6jMUGxurK6+80vm67LLL8i2IOmHCBM2fP1/ff/99qRe2pOgzBCCfvH1/Dh+Wbrklf+Cx2WSMKdJcPzkRkfI79of7uYM8DYFn+Dvgkc8ux/H7778rOjpa2Y6q5QqAMATAhbsaIEcnaDdyVMQ+BcOHn2v+yjvzs0STF1BMFboDdUFq1aql1atXl/ZpAaB0eOj74ykIScX4P8qePe2BhyHwgE9hoVYAlceZM9LUqdKvv0oJCdKQIVJQ0LnPSzDzc5HkbQKj6QsoFSzUCgDF8eij0ssvu9bwPPywNHKk9PzzkqQzH3+moPMNQu4WP5VcZ4F2DIEH4BNKvZkMAMpc3uHvjzwivfBC/qau7GyZF17Qd81v000xX2hwjxIGIZvNPsz93XdpAgMqIZrJAPiGM2ekV1+1B5LvvpNOny7SYUZStvwVqlNqo41aqw7Fu27ezs80gQHlhmYyAHB49FHppZeknJxiH2qTFKBsLf/HSEU9MVjmPzGypaW5H/4u5R9VFhNjbwJz1PzQBAZUOoQhAN5XUG3Lo4/am8DOU7cbbNKgi6XwyfbRZJ76/ixcKEVFUfMDWAhhCIB3eZr1efJkqUcPe6fo0pCQYP/X08zPeWuAAFgGfYYAeI9jzh83sz5L0tlbblfgwrnnfx1/f+nUqfzD7On7A1Ro9BkCULnkDR9XX22vnXH332PGKEfSXwuXKLA0rj1ypGsQkuj7A8CJMASg7LlrCqtZ074emAd+ksJ08vyu6+/vMs8QALhDGAJQtjw0hZnDh4u0+Kmx2WQramv+woX2midPM1ADgBuEIQDF52jySkuz1+7UrGmfjDBvv5vsbI9NYUUJQpJk69tXeuedgneKjJRmzKDzM4ASIQwBKB53TV4OjlFgffoo668s/fjYbDUv6fIXjvW+5s2T4uLczzMUEiKNHi2NGUPnZwAlxmgyAIVz1AQtW2Yffu6B4/9MJtUcr2eO3K+u5mMt0G3Fv17eWZ+lczNQb9ggVasm9e8vdepECAIqsfL6+00YAnCOY82vtWvt79u3l/74w94JuYg1PDmS9ilW8dql7vpAy9Wz8IOioqQjR869j41lzh8ADK0HUM7ee0+66y4pI+PctmefLfZp/CTFKVVfPzRbjcffISXG2PsWufvvLkdT2M6d0saNzPkDwCsIQ4DVuKv9+egje5+cUnRx6ypS1WB7H6KClr+YNMk+4os5fwB4CWEIsJJSqv0pkrp17f+y/AWACo4+Q0Bl5+j8/NJL0ooV5XPN2Fhp1678w+xZ/gJAMdBnCMD5K2gYfFmx2ew1PnmDDstfAKigCEOALyuotsXTIqhliVFgAHwQYQjwJbnDz44d0htv5O+HM3my1LOn50VQS9NDD0kNGniegRoAfABhCPAVRWnySkuTbrpJ6feMUnhZNo1RAwSgEiEMARWRowYoNVXatMk+D88nnxR+nDHKkU1nZ8wqvbJUqybNmmWfGJHOzwAqIcIQUNGcZ6dnPxlF6ej5lyMszD7z9JNPEnwAVGqEIcDbzpyRpk6Vfv1VOnFCmj27dM4bESEdO1a0fkNhYdLMmdT+ALAkwhBQnvLO/vzLL/YJCfOuxl4aHnpIGjs2/8zPuVH7AwC+EYbS0tL02GOP6aOPPtLp06d10UUX6a233lKrVq28XTSg6FJSpHvvlY6WQhNWQRzrfY0ZIzVrlr/JrWZN6fbb7SPOqP0BgIofho4dO6Y2bdqoQ4cO+uijj1SrVi39+uuvuuCCC7xdNKDoUlKkG28s++vkXu/L398+2qtnT2Z+BoACVPgwNHHiRMXGxmrWrHOjYxo0aOC9AgHFlZ0tDRtWPtdyt94XMz8DQIEqfBhavny5unbtqptvvlnr1q1TvXr1NGTIEN1zzz0ej8nMzFRmZqbzfUbuRSmB0lLUtbbWr7fP/1MWYmKke+6RGjak1gcASsjP2wUozG+//abXX39dDRs21CeffKL77rtPw4YN09tvv+3xmOTkZIWHhztfsbGx5VhiWEJKin3m5Q4dpNtus//boIF9+99MjtGPy3fq/eGflf71e/SQ1qyRdu+W/vMf6dZb7bU/BCEAKLYKv2p9UFCQLrvsMm3cuNG5bdiwYdq8ebO++OILt8e4qxmKjY1l1XqUDk9rftlsMpK+vW2iZn5/hd7/IV67suLUTmu1Vh1K59r+/vbRX88/XzrnA4AKjFXr/1a3bl01adLEZVvjxo21ePFij8cEBwcrODi4rIsGK8rO9rzmlzEysqnGvFc1RbuUI38FKVNVI0N0/GSUqv11RLbiXOuWW6Tu3e0zUBtjbwobMkQKCiqtuwEAyAfCUJs2bbR9+3aXbb/88ovq16/vpRLB0tavL3BmaD8ZxSlVz8W8poZ3tNZ1DzVVtTpXSinTiz6aLG/tT//+pVBwAIAnFT4MjRgxQldffbUmTJigvn376n//+59mzJihGTNmeLtosJDTf5zWmle/18E3V+tfRdj/kedrSre2PrehTx9p8WL38wxFRkr//KdUtaqUkEDtDwCUswrfZ0iSVqxYodGjR2vHjh2Kj4/XyJEjCxxNlld5tTmictm/9aA+mPSLVqwK1qcHm+mUqha9/8+aNe6Hs+edgbp9ezo+A4AH5fX32yfC0PkiDFlcEYfA52TlaOv8n7Vi5iGt2FxLX51y7asW479fSRdt10tp/RRy/LBs7h4dx+zPu3YRcADgPNGBGigN7laAj4mRJk+W+vTRyUMn9emk77UiJVMf7LhIB3KaSLKHIJtydEXVH9XjyiPqcU9dtbj5Itn8oqWU1+2jyfKu+ZV39mcAgE+gZgiVg7van2XL3A6BN7JJMhpffaKezRimTIU4P6um4+oS/YN6dD2rG0Ymqnazmu6v5y5kxcbmn/0ZAFBiNJOVIsJQJZM3+Bw+bB99lbf25/Rpj4ui5simfYpRvHYpLmC/kpr8ph79qqnd0GYKrl7EaRmKOgM1AKBEaCYD3HFXI+OG2bevwDl9HEPgdz6zUA3+fZtsfiWYpZw1vwCgUiAMoWLLXfuyY4f01FNFOqyokxvGJ/hJfsWaChEAUMkQhlBxFbEW6LzUrVt25wYA+ATCELynoDl3PK3/VVocQ+Dbti2b8wMAfAZhCN6RkpJ/NuZnn7XPxvz66/YO0WUZhCSGwAMAJBGG4A0pKZ7X6Tp6VKZv3+ItaOqJzSZFREghIVJa2rntMTEMgQcAOBGGUDYcHZ/T0uxD32vWlOrVk66+2t4PqKw5an9mzJB69mQIPADAI8IQSl9BHZ+joqQjRwo8vFRqhfLW/jAEHgDgAWEIpauQjs/myJHSCTu5xcZKL79sD1rU/gAAiokwhOIpaARYdra9RqiAjs/FDkKe1v8aO1Zq2JDgAwA4b4QhFM7R/2fZMmnmTCkj49xnjhFgM2bYOyuX1pxAMTHSK69II0bkX2aDzs8AgFJEGEJ+eWd9fuONAkOOOXpUuvFGfVbjJl1XWmX4e1V59e5N52cAQJkiDOGc7Gxp/Hh7EPnjjyIfZl8DXmp+bF3xrle9umstk3SulslR88P6XwCAMkYYgp27SRCLwSaplg4rs8oFCjr1Z8F9gxyzP+/caa/1cdf/CACAckIYQqkufRF870B7zVJh55o0SQoKkjp1sr8AAPASP28XAF5WhBFgxdKzp/Tee/aaH3diY+2f0wEaAFBBUDNUWeXuBF1Ax+Ps1WvlX5ojwBzXccz6nHcGajpAAwAqGMJQZZA3+Bw54n5I+t8jtP7ck65PXv5BK97PUciu3/RGaZVj8uRzQYeOzwAAH0EY8nUFLX2Ri9mXJt14o8ZWeUETTj2kLF0tSWqnrPMvQ94RYAAA+BDCkC8rRsdnm4xyZNNdp/6fntUIJQb9qqQWqepxe3WZF2Jk259W9H5DNWtK7dpJiYmMAAMA+DzCkK8qQcdnPxnFKVV7X1ykeqP6SUqwfxA72R6q8i594RAZKQ0bxvIXAIBKiTBUkRS07lde69eXeOmLetF5Ak+fPvYRXnmb2xwhaMwYwg8AoNIiDFUU7iY9zL3u19/9cTIzMvX51O+VNnW1Bpb0WnXr5t/Wp8+5EWAsfQEAsBCbMaU1wUzFlZGRofDwcKWnp6t69ereLk5+KSnSjTd6/NhIWtPhGb22/Tqt3N9UJxSmdlqrtepQvOs4Zn7etYuQAwCo8Mrr7zeTLnqbo+9PIS5cM11L91+uEwpTHb/f1bChdDqsloytwIUvznHsN2kSQQgAgFwIQ95WhL4/Nklx2qe3WryqzXN+VFpmTb3xS3uFzn7dvgZYUQJRTAwzPwMA4AZ9hrxo/9aD+umJ1SrqylwDH6st3drk3AZPHZ9jY6WXXrIPgaf/DwAABSIMlYYiLn2Rk5WjrfN/1oqZh/T+/2pr6+nGaqeO6qRninYdOj4DAFDqCEMl5QhAy5ZJ8+bZ199yyLX0xclDJ/XppO/1/uIz+mDnRTqY00SSvXbHphydqVJDGdlRCss8ogIbuxzrfrnD0hcAAJQYYagkClkCw7H0xbNhEzX++DBlqrXzs2o6ri7RP6hH17O6YWSiajdrIaVML3A0mSTXdb8AAECpYWh9Qdw1fy1bVqQlMHJk0z7FKF67FBewX0lNflPSrdV07ZBmCq4enP8Ad/MMSaz7BQCwrPIaWk8Y8sRd7U+9etJff+UPLAXY/exc1R99m2x+RRjxVZwZqAEAqOTKKwxZt5nMUeuTmipt2mSv6WnYUBoyRFqxwm3tj0lLK7hfjxsN/uEnFSUISfbQ06mT/QUAAMqFtcKQo+Zl2TJp7lzpyJH8+4waJVWp4rYZrLhBSJL7EWAAAKDCsFYYSkiQjh0rcBeTkyPbiRPnfy3H0heeRoABAIAKwedmoE5OTpbNZtPw4cOLf3AhQUgqYe1PvpOw9AUAAL7Cp8LQ5s2bNWPGDDVv3tzbRSkYS18AAOAzfCYMnThxQrfffrveeOMN1ahRw3sFsdnsw91jYly316wpDR8urVljXxWeIAQAgE/wmT5DQ4cOVffu3XXdddfp2WefLXDfzMxMZWZmOt9nZGSUTiEczV8zZrAEBgAAlYRPhKGFCxdq69at2rx5c5H2T05O1rhx487von5+9pCTlnZuW0yMvR+Qo9aHJTAAAPB5Fb6ZLDU1VQ899JDmzp2rkJCQIh0zevRopaenO1+pqanFv/CoUdKePfZmr/nzaf4CAKCSqvAzUC9dulS9e/eWf64mqOzsbNlsNvn5+SkzM9PlM3ecM1hKKnT+Sn9/aeRI6fnnz7vsAACg5JiB+m+dOnXSd99957Jt0KBBSkxM1GOPPVZoECpQ9erSgAFSTo7rDNRBQedZagAA4CsqfBgKCwtTs2bNXLZVrVpVkZGR+bYXWUSEfd2xMWPo9AwAgMVV+DBUqt580z4LNSO/AADA3yp8n6HSUF5tjgAAoPSU19/vCj+aDAAAoCwRhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKURhgAAgKVV+DCUnJysyy+/XGFhYapVq5Z69eql7du3e7tYAACgkqjwYWjdunUaOnSovvzyS61atUpZWVnq0qWLTp486e2iAQCASsBmjDHeLkRxHD58WLVq1dK6det07bXXFumYjIwMhYeHKz09XdWrVy/jEgIAgNJQXn+/A8rszGUkPT1dkhQREeFxn8zMTGVmZjrfZ2RklHm5AACAb6rwzWS5GWM0cuRIXXPNNWrWrJnH/ZKTkxUeHu58xcbGlmMpAQCAL/GpZrKhQ4fqgw8+0IYNGxQTE+NxP3c1Q7GxsTSTAQDgQ2gmy+PBBx/U8uXL9fnnnxcYhCQpODhYwcHB5VQyAADgyyp8GDLG6MEHH9SSJUu0du1axcfHe7tIAACgEqnwYWjo0KGaP3++li1bprCwMB08eFCSFB4ertDQUC+XDgAA+LoK32fIZrO53T5r1iwNHDiwSOdgaD0AAL6HPkN/q+BZDQAA+DifGloPAABQ2ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0ghDAADA0nwmDE2dOlXx8fEKCQlRq1attH79em8XCQAAVAI+EYbeeecdDR8+XGPGjNG2bdvUtm1bdevWTXv37vV20QAAgI+zGWOMtwtRmNatW+vSSy/V66+/7tzWuHFj9erVS8nJyYUen5GRofDwcKWnp6t69eplWVQAAFBKyuvvd4WvGTpz5oy++uordenSxWV7ly5dtHHjRi+VCgAAVBYB3i5AYY4cOaLs7GzVrl3bZXvt2rV18OBBt8dkZmYqMzPT+T49PV2SPWECAADf4Pi7XdaNWBU+DDnYbDaX98aYfNsckpOTNW7cuHzbY2Njy6RsAACg7Bw9elTh4eFldv4KH4aioqLk7++frxbo0KFD+WqLHEaPHq2RI0c63//555+qX7++9u7dW6Y/zIomIyNDsbGxSk1NtVRfKe6b+7YC7pv7toL09HTFxcUpIiKiTK9T4cNQUFCQWrVqpVWrVql3797O7atWrVLPnj3dHhMcHKzg4OB828PDwy31S+RQvXp17ttCuG9r4b6txar37edXtl2cK3wYkqSRI0eqf//+uuyyy3TVVVdpxowZ2rt3r+677z5vFw0AAPg4nwhDt9xyi44ePaqnn35aBw4cULNmzfThhx+qfv363i4aAADwcT4RhiRpyJAhGjJkSImODQ4O1lNPPeW26awy4765byvgvrlvK+C+y/a+fWLSRQAAgLJS4SddBAAAKEuEIQAAYGmEIQAAYGmEIQAAYGk+GYamTp2q+Ph4hYSEqFWrVlq/fn2B+69bt06tWrVSSEiI/vGPf2jatGn59lm8eLGaNGmi4OBgNWnSREuWLCmr4pdYce47JSVFnTt3Vs2aNVW9enVdddVV+uSTT1z2mT17tmw2W77XX3/9Vda3UizFue+1a9e6vaeff/7ZZb/K9n0PHDjQ7X03bdrUuY8vfN+ff/65kpKSFB0dLZvNpqVLlxZ6TGV4vot735Xl+S7ufVeW57u4911Znu/k5GRdfvnlCgsLU61atdSrVy9t37690OPK4xn3uTD0zjvvaPjw4RozZoy2bdumtm3bqlu3btq7d6/b/Xft2qUbbrhBbdu21bZt2/Tvf/9bw4YN0+LFi537fPHFF7rlllvUv39/ffPNN+rfv7/69u2rTZs2lddtFaq49/3555+rc+fO+vDDD/XVV1+pQ4cOSkpK0rZt21z2q169ug4cOODyCgkJKY9bKpLi3rfD9u3bXe6pYcOGzs8q4/c9efJkl/tNTU1VRESEbr75Zpf9Kvr3ffLkSbVo0UJTpkwp0v6V5fku7n1Xlue7uPft4OvPd3Hvu7I83+vWrdPQoUP15ZdfatWqVcrKylKXLl108uRJj8eU2zNufMwVV1xh7rvvPpdtiYmJ5vHHH3e7/6OPPmoSExNdtg0ePNhceeWVzvd9+/Y1119/vcs+Xbt2Nf369SulUp+/4t63O02aNDHjxo1zvp81a5YJDw8vrSKWieLe95o1a4wkc+zYMY/ntML3vWTJEmOz2czu3bud23zh+85NklmyZEmB+1SW5zu3oty3O774fOdWlPuuLM93biX5vivD822MMYcOHTKSzLp16zzuU17PuE/VDJ05c0ZfffWVunTp4rK9S5cu2rhxo9tjvvjii3z7d+3aVVu2bNHZs2cL3MfTOctbSe47r5ycHB0/fjzfYncnTpxQ/fr1FRMTox49euT7L0tvOp/7vuSSS1S3bl116tRJa9ascfnMCt/3W2+9peuuuy7fLO0V+fsuicrwfJcGX3y+z4cvP9+lobI83+np6ZJU4CKs5fWM+1QYOnLkiLKzs/OtVl+7du18q9o7HDx40O3+WVlZOnLkSIH7eDpneSvJfef10ksv6eTJk+rbt69zW2JiombPnq3ly5drwYIFCgkJUZs2bbRjx45SLX9JleS+69atqxkzZmjx4sVKSUlRo0aN1KlTJ33++efOfSr7933gwAF99NFHuvvuu122V/TvuyQqw/NdGnzx+S6JyvB8n6/K8nwbYzRy5Ehdc801atasmcf9yusZ95nlOHKz2Wwu740x+bYVtn/e7cU9pzeUtIwLFizQ2LFjtWzZMtWqVcu5/corr9SVV17pfN+mTRtdeumlevXVV/X//t//K72Cn6fi3HejRo3UqFEj5/urrrpKqampevHFF3XttdeW6JzeUtIyzp49WxdccIF69erlst1Xvu/iqizPd0n5+vNdHJXp+S6pyvJ8P/DAA/r222+1YcOGQvctj2fcp2qGoqKi5O/vny/tHTp0KF8qdKhTp47b/QMCAhQZGVngPp7OWd5Kct8O77zzju666y69++67uu666wrc18/PT5dffnmF+S+J87nv3K688kqXe6rM37cxRjNnzlT//v0VFBRU4L4V7fsuicrwfJ8PX36+S4uvPd/no7I83w8++KCWL1+uNWvWKCYmpsB9y+sZ96kwFBQUpFatWmnVqlUu21etWqWrr77a7TFXXXVVvv1Xrlypyy67TIGBgQXu4+mc5a0k9y3Z/4tx4MCBmj9/vrp3717odYwx+vrrr1W3bt3zLnNpKOl957Vt2zaXe6qs37dkH62xc+dO3XXXXYVep6J93yVRGZ7vkvL157u0+NrzfT58/fk2xuiBBx5QSkqKVq9erfj4+EKPKbdnvMhdrSuIhQsXmsDAQPPWW2+ZH3/80QwfPtxUrVrV2av+8ccfN/3793fu/9tvv5kqVaqYESNGmB9//NG89dZbJjAw0Lz33nvOff773/8af39/89xzz5mffvrJPPfccyYgIMB8+eWX5X5/nhT3vufPn28CAgLMa6+9Zg4cOOB8/fnnn859xo4daz7++GPz66+/mm3btplBgwaZgIAAs2nTpnK/P0+Ke9+vvPKKWbJkifnll1/M999/bx5//HEjySxevNi5T2X8vh3uuOMO07p1a7fn9IXv+/jx42bbtm1m27ZtRpJ5+eWXzbZt28yePXuMMZX3+S7ufVeW57u4911Znu/i3reDrz/f999/vwkPDzdr1651+b09deqUcx9vPeM+F4aMMea1114z9evXN0FBQebSSy91GZY3YMAA065dO5f9165day655BITFBRkGjRoYF5//fV851y0aJFp1KiRCQwMNImJiS4PV0VRnPtu166dkZTvNWDAAOc+w4cPN3FxcSYoKMjUrFnTdOnSxWzcuLEc76hoinPfEydONAkJCSYkJMTUqFHDXHPNNeaDDz7Id87K9n0bY8yff/5pQkNDzYwZM9yezxe+b8fQaU+/t5X1+S7ufVeW57u4911Znu+S/J5Xhufb3T1LMrNmzXLu461n3PZ3AQEAACzJp/oMAQAAlDbCEAAAsDTCEAAAsDTCEAAAsDTCEAAAsDTCEAAAsDTCEAAAsDTCEAAAsDTCEAAAsDTCEAAAsDTCEACftGDBAoWEhCgtLc257e6771bz5s2Vnp7uxZIB8DWsTQbAJxlj1LJlS7Vt21ZTpkzRuHHj9Oabb+rLL79UvXr1vF08AD4kwNsFAICSsNlsGj9+vG666SZFR0dr8uTJWr9+PUEIQLFRMwTAp1166aX64YcftHLlSrVr187bxQHgg+gzBMBnffLJJ/r555+VnZ2t2rVre7s4AHwUNUMAfNLWrVvVvn17vfbaa1q4cKGqVKmiRYsWebtYAHwQfYYA+Jzdu3ere/fuevzxx9W/f381adJEl19+ub766iu1atXK28UD4GOoGQLgU/744w+1adNG1157raZPn+7c3rNnT2VmZurjjz/2YukA+CLCEAAAsDQ6UAMAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEv7/5vxx3z/T4NjAAAAAElFTkSuQmCC",
- "text/plain": [
- "