diff --git a/doc/pub/week39/html/week39-bs.html b/doc/pub/week39/html/week39-bs.html index 76a8950d4..899adc049 100644 --- a/doc/pub/week39/html/week39-bs.html +++ b/doc/pub/week39/html/week39-bs.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 39: Optimization and Gradient Methods - + - - -
@@ -250,7 +243,6 @@ MathJax.Hub.Config({ Week 39: Optimization and Gradient Methods
-
-
-

 

 

 

- - - -
-

Week 39: Optimization and Gradient Methods

+
+

Week 39: Optimization and Gradient Methods

+
-

-

Morten Hjorth-Jensen [1, 2]
- -

+

+[1] Department of Physics, University of Oslo +
+
+[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
+
+
+

Nov 2, 2021

+
+
-
[1] Department of Physics, University of Oslo
-
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-
-

-

Oct 1, 2021

-
-

Read »

@@ -392,25 +380,18 @@ MathJax.Hub.Config({
  • »
    • -
    - - -
    © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html index 953c05fd4..bd6faa596 100644 --- a/doc/pub/week39/html/week39-reveal.html +++ b/doc/pub/week39/html/week39-reveal.html @@ -1,18 +1,17 @@ + - + + - Week 39: Optimization and Gradient Methods - - - - - - @@ -55,36 +54,81 @@ document.getElementsByTagName( 'head' )[0].appendChild( link ); - - - +
    +

    Week 39: Optimization and Gradient Methods

    +
    - - -

    Week 39: Optimization and Gradient Methods

    - -

    -

    Morten Hjorth-Jensen [1, 2]
    - -

    +

    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 2, 2021

    +
    +
    -
    [1] Department of Physics, University of Oslo
    -
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    -
    -

    -

    Oct 1, 2021

    -
    -











    -

    Plan for week 39

    - -See lecture notes for week 39. +

    See lecture notes for week 39. For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. +

    -

    For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11. -These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. +

    These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning.

    -











    -

    Thursday September 30

    -

    Overview Video, why do we care about gradient methods? -











    -

    Searching for Optimal Regularization Parameters \( \lambda \)

    -

    -In project 1, when using Ridge and Lasso regression, we end up +

    In project 1, when using Ridge and Lasso regression, we end up searching for the optimal parameter \( \lambda \) which minimizes our selected scores (MSE or \( R2 \) values for example). The brute force approach, as discussed in the code here for Ridge regression, consists in evaluating the MSE as function of different \( \lambda \) values. Based on these calculations, one tries then to determine the value of the hyperparameter \( \lambda \) which results in optimal scores (for example the smallest MSE or an \( R2=1 \)). -

    +

    -
    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
 from sklearn.model_selection import train_test_split
@@ -370,26 +366,42 @@ plt.xlabel('log10(lambda)')
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function RidgeCV instead of just calling the Ridge function. For RidgeCV we need to pass the array of \( \lambda \) values.
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function RidgeCV instead of just calling the Ridge function. For RidgeCV we need to pass the array of \( \lambda \) values. By inspecting the figure we can in turn determine which is the optimal regularization parameter. -This becomes however less functional in the long run. +This becomes however less functional in the long run. +

    -











    -

    -

    -An alternative is to use the so-called grid search functionality +

    An alternative is to use the so-called grid search functionality included with the library Scikit-Learn, as demonstrated for the same example here. +

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 from sklearn.model_selection import train_test_split
 from sklearn.linear_model import Ridge
 from sklearn.model_selection import GridSearchCV
@@ -431,23 +443,33 @@ ypredictRidge = gridsearch.predict(X_test)
 print(f"Best estimated lambda-value: {gridsearch.best_estimator_.alpha}")
 print(f"MSE score: {MSE(y_test,ypredictRidge)}")
 print(f"R2 score: {R2(y_test,ypredictRidge)}")
-
    
-
    
-By default the grid search function includes cross validation with
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    By default the grid search function includes cross validation with five folds. The Scikit-Learn documentation contains more information on how to set the different parameters. +

    -

    -If we take out the random noise, running the above codes results in \( \lambda=0 \) yielding the best fit. +

    If we take out the random noise, running the above codes results in \( \lambda=0 \) yielding the best fit.

    -











    -

    -

    -An alternative to the above manual grid set up, is to use a random +

    An alternative to the above manual grid set up, is to use a random search where the parameters are tuned from a random distribution (uniform below) for a fixed number of iterations. A model is constructed and evaluated for each combination of chosen parameters. @@ -455,11 +477,16 @@ We repeat the previous example but now with a random search. Note that values of \( \lambda \) are now limited to be within \( x\in [0,1] \). This domain may not be the most relevant one for the specific case under study. +

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 from sklearn.model_selection import train_test_split
 from sklearn.linear_model import Ridge
 from sklearn.model_selection import GridSearchCV
@@ -502,14 +529,26 @@ ypredictRidge = gridsearch.predict(X_test)
 print(f"Best estimated lambda-value: {gridsearch.best_estimator_.alpha}")
 print(f"MSE score: {MSE(y_test,ypredictRidge)}")
 print(f"R2 score: {R2(y_test,ypredictRidge)}")
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Optimization, the central part of any Machine Learning algortithm

    -

    -Almost every problem in machine learning and data science starts with +

    Almost every problem in machine learning and data science starts with a dataset \( X \), a model \( g(\beta) \), which is a function of the parameters \( \beta \) and a cost function \( C(X, g(\beta)) \) that allows us to judge how well the model \( g(\beta) \) explains the observations @@ -517,19 +556,18 @@ us to judge how well the model \( g(\beta) \) explains the observations the cost function. Ideally we would be able to solve for \( \beta \) analytically, however this is not possible in general and we must use some approximative/numerical method to compute the minimum. +

    -











    -

    Revisiting our Logistic Regression case

    -

    -In our discussion on Logistic Regression we studied the +

    In our discussion on Logistic Regression we studied the case of two classes, with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assumed also that we have only two parameters \( \beta \) in our fitting, that is we defined probabilities +

    $$ \begin{align*} @@ -538,120 +576,105 @@ p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}), \end{align*} $$ -where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \). +

    where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).

    -











    -

    The equations to solve

    -

    -Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \) +

    Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \) elements \( y_i \), an \( n\times p \) matrix \( \boldsymbol{X} \) which contains the \( x_i \) values and a vector \( \boldsymbol{p} \) of fitted probabilities \( p(y_i\vert x_i,\boldsymbol{\beta}) \). We rewrote in a more compact form the first derivative of the cost function as +

    $$ \frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right). $$ -

    -If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements +

    If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements \( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as +

    $$ \frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}. $$ -This defines what is called the Hessian matrix. +

    This defines what is called the Hessian matrix.

    -











    -

    Solving using Newton-Raphson's method

    -

    -If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives. +

    If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives.

    -

    -Our iterative scheme is then given by +

    Our iterative scheme is then given by

    $$ \boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T}\right)^{-1}_{\boldsymbol{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}\right)_{\boldsymbol{\beta}^{\mathrm{old}}}, $$ -or in matrix form as +

    or in matrix form as

    $$ \boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X} \right)^{-1}\times \left(-\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{p}) \right)_{\boldsymbol{\beta}^{\mathrm{old}}}. $$ -The right-hand side is computed with the old values of \( \beta \). +

    The right-hand side is computed with the old values of \( \beta \).

    -

    -If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement. +

    If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement.

    -











    -

    Brief reminder on Newton-Raphson's method

    -

    -Let us quickly remind ourselves how we derive the above method. +

    Let us quickly remind ourselves how we derive the above method.

    -

    -Perhaps the most celebrated of all one-dimensional root-finding +

    Perhaps the most celebrated of all one-dimensional root-finding routines is Newton's method, also called the Newton-Raphson method. This method requires the evaluation of both the function \( f \) and its derivative \( f' \) at arbitrary points. If you can only calculate the derivative numerically and/or your function is not of the smooth type, we normally discourage the use of this method. +

    -











    -

    The equations

    -

    -The Newton-Raphson formula consists geometrically of extending the +

    The Newton-Raphson formula consists geometrically of extending the tangent line at a current point until it crosses zero, then setting the next guess to the abscissa of that zero-crossing. The mathematics behind this method is rather simple. Employing a Taylor expansion for \( x \) sufficiently close to the solution \( s \), we have +

    $$ f(s)=0=f(x)+(s-x)f'(x)+\frac{(s-x)^2}{2}f''(x) +\dots. \label{eq:taylornr} $$ -

    -For small enough values of the function and for well-behaved +

    For small enough values of the function and for well-behaved functions, the terms beyond linear are unimportant, hence we obtain +

    $$ f(x)+(s-x)f'(x)\approx 0, $$ -yielding +

    yielding

    $$ s\approx x-\frac{f(x)}{f'(x)}. $$ -

    -Having in mind an iterative procedure, it is natural to start iterating with +

    Having in mind an iterative procedure, it is natural to start iterating with

    $$ x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)}. $$ -

    -









    +









    Simple geometric interpretation

    -

    -The above is Newton-Raphson's method. It has a simple geometric +

    The above is Newton-Raphson's method. It has a simple geometric interpretation, namely \( x_{n+1} \) is the point where the tangent from \( (x_n,f(x_n)) \) crosses the \( x \)-axis. Close to the solution, Newton-Raphson converges fast to the desired result. However, if we @@ -662,21 +685,20 @@ from the true root as to let the search interval include a local maximum or minimum of the function. If an iteration places a trial guess near such a local extremum, so that the first derivative nearly vanishes, then Newton-Raphson may fail totally +

    -











    -

    Extending to more than one variable

    -

    -Newton's method can be generalized to systems of several non-linear equations +

    Newton's method can be generalized to systems of several non-linear equations and variables. Consider the case with two equations +

    $$ \begin{array}{cc} f_1(x_1,x_2) &=0\\ f_2(x_1,x_2) &=0,\end{array} $$ -which we Taylor expand to obtain +

    which we Taylor expand to obtain

    $$ \begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1 @@ -688,7 +710,7 @@ $$ \end{array}. $$ -Defining the Jacobian matrix \( {\bf \boldsymbol{J}} \) we have +

    Defining the Jacobian matrix \( {\bf \boldsymbol{J}} \) we have

    $$ {\bf \boldsymbol{J}}=\left( \begin{array}{cc} \partial f_1/\partial x_1 & \partial f_1/\partial x_2 \\ @@ -696,78 +718,71 @@ $$ \end{array} \right), $$ -we can rephrase Newton's method as +

    we can rephrase Newton's method as

    $$ \left(\begin{array}{c} x_1^{n+1} \\ x_2^{n+1} \end{array} \right)= \left(\begin{array}{c} x_1^{n} \\ x_2^{n} \end{array} \right)+ \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right), $$ -where we have defined +

    where we have defined

    $$ \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)= -{\bf \boldsymbol{J}}^{-1} \left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right). $$ -We need thus to compute the inverse of the Jacobian matrix and it +

    We need thus to compute the inverse of the Jacobian matrix and it is to understand that difficulties may arise in case \( {\bf \boldsymbol{J}} \) is nearly singular. +

    -

    -It is rather straightforward to extend the above scheme to systems of -more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function. +

    It is rather straightforward to extend the above scheme to systems of +more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function. +

    -











    -

    Steepest descent

    -

    -The basic idea of gradient descent is +

    The basic idea of gradient descent is that a function \( F(\mathbf{x}) \), \( \mathbf{x} \equiv (x_1,\cdots,x_n) \), decreases fastest if one goes from \( \bf {x} \) in the direction of the negative gradient \( -\nabla F(\mathbf{x}) \). +

    -

    -It can be shown that if +

    It can be shown that if

    $$ \mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), $$ -with \( \gamma_k > 0 \). +

    with \( \gamma_k > 0 \).

    -

    -For \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq +

    For \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq F(\mathbf{x}_k) \). This means that for a sufficiently small \( \gamma_k \) we are always moving towards smaller function values, i.e a minimum. +

    -

    -

    More on Steepest descent

    -

    -The previous observation is the basis of the method of steepest +

    The previous observation is the basis of the method of steepest descent, which is also referred to as just gradient descent (GD). One starts with an initial guess \( \mathbf{x}_0 \) for a minimum of \( F \) and computes new approximations according to +

    $$ \mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ k \geq 0. $$ -

    -The parameter \( \gamma_k \) is often referred to as the step length or +

    The parameter \( \gamma_k \) is often referred to as the step length or the learning rate within the context of Machine Learning. +

    -

    -

    The ideal

    -

    -Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global +

    Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global minimum of the function \( F \). In general we do not know if we are in a global or local minimum. In the special case when \( F \) is a convex function, all local minima are also global minima, so in this case @@ -775,188 +790,165 @@ gradient descent can converge to the global solution. The advantage of this scheme is that it is conceptually simple and straightforward to implement. However the method in this form has some severe limitations: +

    -

    -In machine learing we are often faced with non-convex high dimensional +

    In machine learing we are often faced with non-convex high dimensional cost functions with many local minima. Since GD is deterministic we will get stuck in a local minimum, if the method converges, unless we have a very good intial guess. This also implies that the scheme is sensitive to the chosen initial condition. +

    -

    -Note that the gradient is a function of \( \mathbf{x} = +

    Note that the gradient is a function of \( \mathbf{x} = (x_1,\cdots,x_n) \) which makes it expensive to compute numerically. +

    -

    -

    The sensitiveness of the gradient descent

    -

    -The gradient descent method +

    The gradient descent method is sensitive to the choice of learning rate \( \gamma_k \). This is due to the fact that we are only guaranteed that \( F(\mathbf{x}_{k+1}) \leq F(\mathbf{x}_k) \) for sufficiently small \( \gamma_k \). The problem is to determine an optimal learning rate. If the learning rate is chosen too small the method will take a long time to converge and if it is too large we can experience erratic behavior. +

    -

    -Many of these shortcomings can be alleviated by introducing +

    Many of these shortcomings can be alleviated by introducing randomness. One such method is that of Stochastic Gradient Descent (SGD), see below. +

    -

    -

    Convex functions

    -

    -Ideally we want our cost/loss function to be convex(concave). +

    Ideally we want our cost/loss function to be convex(concave).

    -

    -First we give the definition of a convex set: A set \( C \) in +

    First we give the definition of a convex set: A set \( C \) in \( \mathbb{R}^n \) is said to be convex if, for all \( x \) and \( y \) in \( C \) and all \( t \in (0,1) \) , the point \( (1 − t)x + ty \) also belongs to C. Geometrically this means that every point on the line segment connecting \( x \) and \( y \) is in \( C \) as discussed below. +

    -

    -The convex subsets of \( \mathbb{R} \) are the intervals of +

    The convex subsets of \( \mathbb{R} \) are the intervals of \( \mathbb{R} \). Examples of convex sets of \( \mathbb{R}^2 \) are the regular polygons (triangles, rectangles, pentagons, etc...). +

    -











    -

    Convex function

    -

    -Convex function: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below. +

    Convex function: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below.

    -











    -

    Conditions on convex functions

    -

    -In the following we state first and second-order conditions which +

    In the following we state first and second-order conditions which ensures convexity of a function \( f \). We write \( D_f \) to denote the domain of \( f \), i.e the subset of \( R^n \) where \( f \) is defined. For more details and proofs we refer to: S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press. +

    -

    First order condition

    -Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for +

    Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \) is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds for all \( x,y \in D_f \). This condition means that for a convex function the first order Taylor expansion (right hand side above) at any point a global under estimator of the function. To convince yourself you can make a drawing of \( f(x) = x^2+1 \) and draw the tangent line to \( f(x) \) and -note that it is always below the graph. +note that it is always below the graph. +

    -

    Second order condition

    -Assume that \( f \) is twice +

    Assume that \( f \) is twice differentiable, i.e the Hessian matrix exists at each point in \( D_f \). Then \( f \) is convex if and only if \( D_f \) is a convex set and its Hessian is positive semi-definite for all \( x\in D_f \). For a single-variable function this reduces to \( f''(x) \geq 0 \). Geometrically this means that \( f \) has nonnegative curvature everywhere. +

    -

    -This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition. +

    This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.

    -











    -

    More on convex functions

    -

    -The next result is of great importance to us and the reason why we are +

    The next result is of great importance to us and the reason why we are going on about convex functions. In machine learning we frequently have to minimize a loss/cost function in order to find the best -parameters for the model we are considering. +parameters for the model we are considering. +

    -

    -Ideally we want the +

    Ideally we want the global minimum (for high-dimensional models it is hard to know if we have local or global minimum). However, if the cost/loss function is convex the following result provides invaluable information: +

    -

    Any minimum is global for convex functions

    -Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \) +

    Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \) is minimal, where \( f \) is convex and differentiable. Then, any point \( x^* \) that satisfies \( \nabla f(x^*) = 0 \) is a global minimum. +

    -

    -This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum. +

    This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.

    -











    -

    Some simple problems

    1. Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1] \) $\lambda f(x)+(1-\lambda)f(y)-f(\lambda x + (1-\lambda) y ) \geq 0$.
    2. Using the second order condition show that the following functions are convex on the specified domain.
      3. - -
    3. Let \( f(x) = x^2 \) and \( g(x) = e^x \). Show that \( f(g(x)) \) and \( g(f(x)) \) is convex for \( x \in \mathbb{R} \). Also show that if \( f(x) \) is any convex function than \( h(x) = e^{f(x)} \) is convex.
    4. A norm is any function that satisfy the following properties
      5. - -
    +

    Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).

    -Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this). - -











    -

    Standard steepest descent

    -

    -Before we proceed, we would like to discuss the approach called the +

    Before we proceed, we would like to discuss the approach called the standard Steepest descent (different from the above steepest descent discussion), which again leads to us having to be able to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG). +

    -

    The success of the CG method -for finding solutions of non-linear problems is based on the theory +

    for finding solutions of non-linear problems is based on the theory of conjugate gradients for linear systems of equations. It belongs to the class of iterative methods for solving problems from linear algebra of the type +

    $$ \begin{equation*} \boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}. \end{equation*} $$ -

    -In the iterative process we end up with a problem like +

    In the iterative process we end up with a problem like

    $$ \begin{equation*} @@ -964,116 +956,106 @@ $$ \end{equation*} $$ -where \( \boldsymbol{r} \) is the so-called residual or error in the iterative process. +

    where \( \boldsymbol{r} \) is the so-called residual or error in the iterative process.

    -

    -When we have found the exact solution, \( \boldsymbol{r}=0 \). +

    When we have found the exact solution, \( \boldsymbol{r}=0 \).

    -











    -

    Gradient method

    -

    -The residual is zero when we reach the minimum of the quadratic equation +

    The residual is zero when we reach the minimum of the quadratic equation

    $$ \begin{equation*} P(\boldsymbol{x})=\frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T\boldsymbol{b}, \end{equation*} $$ -

    -with the constraint that the matrix \( \boldsymbol{A} \) is positive definite and -symmetric. This defines also the Hessian and we want it to be positive definite. +

    with the constraint that the matrix \( \boldsymbol{A} \) is positive definite and +symmetric. This defines also the Hessian and we want it to be positive definite. +

    -











    -

    Steepest descent method

    -

    -We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). +

    We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). We can assume without loss of generality that +

    $$ \begin{equation*} \boldsymbol{x}_0=0, \end{equation*} $$ -or consider the system +

    or consider the system

    $$ \begin{equation*} \boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0, \end{equation*} $$ -instead. +

    instead.

    -











    -

    Steepest descent method

    -One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form +

    One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form

    $$ \begin{equation*} f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n. \end{equation*} $$ -This suggests taking the first basis vector \( \boldsymbol{r}_1 \) (see below for definition) +

    This suggests taking the first basis vector \( \boldsymbol{r}_1 \) (see below for definition) to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \), which equals +

    $$ \begin{equation*} \boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b}, \end{equation*} $$ -and +

    and \( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \). - - +

    -











    -

    Final expressions

    -We can compute the residual iteratively as +

    We can compute the residual iteratively as

    $$ \begin{equation*} \boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1}, \end{equation*} $$ -which equals +

    which equals

    $$ \begin{equation*} \boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{r}_k), \end{equation*} $$ -or +

    or

    $$ \begin{equation*} (\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{r}_k, \end{equation*} $$ -which gives +

    which gives

    $$ \alpha_k = \frac{\boldsymbol{r}_k^T\boldsymbol{r}_k}{\boldsymbol{r}_k^T\boldsymbol{A}\boldsymbol{r}_k} $$ -leading to the iterative scheme +

    leading to the iterative scheme

    $$ \begin{equation*} \boldsymbol{x}_{k+1}=\boldsymbol{x}_k-\alpha_k\boldsymbol{r}_{k}, @@ -1082,15 +1064,17 @@ $$
    -











    -

    Steepest descent example

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 import numpy.linalg as la
 
 import scipy.optimize as sopt
@@ -1110,116 +1094,196 @@ ax = fig.gca(projection="3d")
 xmesh, ymesh = np.mgrid[-3:3:50j,-3:3:50j]
 fmesh = f(np.array([xmesh, ymesh]))
 ax.plot_surface(xmesh, ymesh, fmesh)
-
    
-
    
-And then as countor plot
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    And then as countor plot

    -
    pt.axis("equal")
+
    
+  
    
+    
    
+      
    
+        
    
+  
    pt.axis("equal")
 pt.contour(xmesh, ymesh, fmesh)
 guesses = [np.array([2, 2./5])]
-
    
-
    
-Find guesses
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Find guesses

    -
    x = guesses[-1]
+
    
+  
    
+    
    
+      
    
+        
    
+  
    x = guesses[-1]
 s = -df(x)
-
    
-
    
-Run it!
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Run it!

    -
    def f1d(alpha):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def f1d(alpha):
     return f(x + alpha*s)
 
 alpha_opt = sopt.golden(f1d)
 next_guess = x + alpha_opt * s
 guesses.append(next_guess)
 print(next_guess)
-
    
-
    
-What happened?
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    What happened?

    -
    pt.axis("equal")
+
    
+  
    
+    
    
+      
    
+        
    
+  
    pt.axis("equal")
 pt.contour(xmesh, ymesh, fmesh, 50)
 it_array = np.array(guesses)
 pt.plot(it_array.T[0], it_array.T[1], "x-")
-
    
-
    
-Note that we did only one iteration here. We can easily add more using our previous guesses.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Note that we did only one iteration here. We can easily add more using our previous guesses.

    -











    -

    Conjugate gradient method

    -In the CG method we define so-called conjugate directions and two vectors +

    In the CG method we define so-called conjugate directions and two vectors \( \boldsymbol{s} \) and \( \boldsymbol{t} \) are said to be conjugate if +

    $$ \begin{equation*} \boldsymbol{s}^T\boldsymbol{A}\boldsymbol{t}= 0. \end{equation*} $$ -The philosophy of the CG method is to perform searches in various conjugate directions +

    The philosophy of the CG method is to perform searches in various conjugate directions of our vectors \( \boldsymbol{x}_i \) obeying the above criterion, namely +

    $$ \begin{equation*} \boldsymbol{x}_i^T\boldsymbol{A}\boldsymbol{x}_j= 0. \end{equation*} $$ -Two vectors are conjugate if they are orthogonal with respect to +

    Two vectors are conjugate if they are orthogonal with respect to this inner product. Being conjugate is a symmetric relation: if \( \boldsymbol{s} \) is conjugate to \( \boldsymbol{t} \), then \( \boldsymbol{t} \) is conjugate to \( \boldsymbol{s} \). +

    -











    -

    Conjugate gradient method

    -An example is given by the eigenvectors of the matrix +

    An example is given by the eigenvectors of the matrix

    $$ \begin{equation*} \boldsymbol{v}_i^T\boldsymbol{A}\boldsymbol{v}_j= \lambda\boldsymbol{v}_i^T\boldsymbol{v}_j, \end{equation*} $$ -which is zero unless \( i=j \). +

    which is zero unless \( i=j \).

    -











    -

    Conjugate gradient method

    -Assume now that we have a symmetric positive-definite matrix \( \boldsymbol{A} \) of size +

    Assume now that we have a symmetric positive-definite matrix \( \boldsymbol{A} \) of size \( n\times n \). At each iteration \( i+1 \) we obtain the conjugate direction of a vector +

    $$ \begin{equation*} \boldsymbol{x}_{i+1}=\boldsymbol{x}_{i}+\alpha_i\boldsymbol{p}_{i}. \end{equation*} $$ -We assume that \( \boldsymbol{p}_{i} \) is a sequence of \( n \) mutually conjugate directions. +

    We assume that \( \boldsymbol{p}_{i} \) is a sequence of \( n \) mutually conjugate directions. Then the \( \boldsymbol{p}_{i} \) form a basis of \( R^n \) and we can expand the solution $ \boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}$ in this basis, namely +

    $$ \begin{equation*} @@ -1229,21 +1293,19 @@ $$
    -











    -

    Conjugate gradient method

    -The coefficients are given by +

    The coefficients are given by

    $$ \begin{equation*} \mathbf{A}\mathbf{x} = \sum^{n}_{i=1} \alpha_i \mathbf{A} \mathbf{p}_i = \mathbf{b}. \end{equation*} $$ -Multiplying with \( \boldsymbol{p}_k^T \) from the left gives +

    Multiplying with \( \boldsymbol{p}_k^T \) from the left gives

    $$ \begin{equation*} @@ -1251,7 +1313,7 @@ $$ \end{equation*} $$ -and we can define the coefficients \( \alpha_k \) as +

    and we can define the coefficients \( \alpha_k \) as

    $$ \begin{equation*} @@ -1261,93 +1323,90 @@ $$
    -











    -

    Conjugate gradient method and iterations

    -

    -If we choose the conjugate vectors \( \boldsymbol{p}_k \) carefully, +

    If we choose the conjugate vectors \( \boldsymbol{p}_k \) carefully, then we may not need all of them to obtain a good approximation to the solution \( \boldsymbol{x} \). We want to regard the conjugate gradient method as an iterative method. This will us to solve systems where \( n \) is so large that the direct method would take too much time. +

    -

    -We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). +

    We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). We can assume without loss of generality that +

    $$ \begin{equation*} \boldsymbol{x}_0=0, \end{equation*} $$ -or consider the system +

    or consider the system

    $$ \begin{equation*} \boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0, \end{equation*} $$ -instead. +

    instead.

    -











    -

    Conjugate gradient method

    -One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form +

    One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form

    $$ \begin{equation*} f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n. \end{equation*} $$ -This suggests taking the first basis vector \( \boldsymbol{p}_1 \) +

    This suggests taking the first basis vector \( \boldsymbol{p}_1 \) to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \), which equals +

    $$ \begin{equation*} \boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b}, \end{equation*} $$ -and +

    and \( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \). The other vectors in the basis will be conjugate to the gradient, hence the name conjugate gradient method. +

    -











    -

    Conjugate gradient method

    -Let \( \boldsymbol{r}_k \) be the residual at the \( k \)-th step: +

    Let \( \boldsymbol{r}_k \) be the residual at the \( k \)-th step:

    $$ \begin{equation*} \boldsymbol{r}_k=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k. \end{equation*} $$ -Note that \( \boldsymbol{r}_k \) is the negative gradient of \( f \) at +

    Note that \( \boldsymbol{r}_k \) is the negative gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_k \), so the gradient descent method would be to move in the direction \( \boldsymbol{r}_k \). Here, we insist that the directions \( \boldsymbol{p}_k \) are conjugate to each other, so we take the direction closest to the gradient \( \boldsymbol{r}_k \) under the conjugacy constraint. This gives the following expression +

    $$ \begin{equation*} \boldsymbol{p}_{k+1}=\boldsymbol{r}_k-\frac{\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{r}_k}{\boldsymbol{p}_k^T\boldsymbol{A}\boldsymbol{p}_k} \boldsymbol{p}_k. @@ -1356,35 +1415,33 @@ $$
    -











    -

    Conjugate gradient method

    -We can also compute the residual iteratively as +

    We can also compute the residual iteratively as

    $$ \begin{equation*} \boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1}, \end{equation*} $$ -which equals +

    which equals

    $$ \begin{equation*} \boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{p}_k), \end{equation*} $$ -or +

    or

    $$ \begin{equation*} (\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{p}_k, \end{equation*} $$ -which gives +

    which gives

    $$ \begin{equation*} @@ -1394,53 +1451,65 @@ $$
    -

    -

    Revisiting our first homework

    -

    -We will use linear regression as a case study for the gradient descent +

    We will use linear regression as a case study for the gradient descent methods. Linear regression is a great test case for the gradient descent methods discussed in the lectures since it has several desirable properties such as: +

    1. An analytical solution (recall homework set 1).
    2. The gradient can be computed analytically.
    3. The cost function is convex which guarantees that gradient descent converges for small enough learning rates
    +

    We revisit an example similar to what we had in the first homework set. We had a function of the type

    -We revisit an example similar to what we had in the first homework set. We had a function of the type - -

    -

    x = 2*np.random.rand(m,1)
+
    
+  
    
+    
    
+      
    
+        
    
+  
    x = 2*np.random.rand(m,1)
 y = 4+3*x+np.random.randn(m,1)
-
    
-
    
-with \( x_i \in [0,1]  \) is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). 
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    with \( x_i \in [0,1] \) is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). The linear regression model is given by +

    $$ h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x, $$ -such that +

    such that

    $$ \boldsymbol{y}_i = \beta_0 + \beta_1 x_i. $$ -

    - +

    Gradient descent example

    -

    -Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \) +

    Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)

    -

    -It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by (we keep the intercept here) +

    It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by (we keep the intercept here)

    $$ X \equiv \begin{bmatrix} 1 & x_1 \\ @@ -1449,33 +1518,28 @@ X \equiv \begin{bmatrix} \end{bmatrix}. $$ -The cost/loss/risk function is given by ( +

    The cost/loss/risk function is given by (

    $$ C(\beta) = \frac{1}{n}||X\beta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2\right] $$ -and we want to find \( \beta \) such that \( C(\beta) \) is minimized. +

    and we want to find \( \beta \) such that \( C(\beta) \) is minimized.

    -











    -

    The derivative of the cost/loss function

    -

    -Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as +

    Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as

    $$ \nabla_{\beta} C(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ \sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ \end{bmatrix} = \frac{2}{n}X^T(X\beta - \mathbf{y}), $$ -where \( X \) is the design matrix defined above. +

    where \( X \) is the design matrix defined above.

    -











    -

    The Hessian matrix

    -The Hessian matrix of \( C(\beta) \) is given by +

    The Hessian matrix of \( C(\beta) \) is given by

    $$ \boldsymbol{H} \equiv \begin{bmatrix} \frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ @@ -1483,39 +1547,37 @@ $$ \end{bmatrix} = \frac{2}{n}X^T X. $$ -This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite. +

    This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite.

    -











    -

    Simple program

    -

    -We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to +

    We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to

    $$ \beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots $$ -

    -We can use the expression we computed for the gradient and let use a +

    We can use the expression we computed for the gradient and let use a \( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). Note that the code below does not include the latter stop criterion. +

    -

    -And finally we can compare our solution for \( \beta \) with the analytic result given by +

    And finally we can compare our solution for \( \beta \) with the analytic result given by \( \beta= (X^TX)^{-1} X^T \mathbf{y} \). +

    -











    -

    Gradient Descent Example

    -

    -Here our simple example -

    +

    Here our simple example

    -
    # Importing various packages
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Importing various packages
 from random import random, seed
 import numpy as np
 import matplotlib.pyplot as plt
@@ -1560,16 +1622,33 @@ plt.xlabel(r'$x$')
 plt.ylabel(r'$y$')
 plt.title(r'Gradient descent example')
 plt.show()
-
    
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    And a corresponding example using scikit-learn

    -

    And a corresponding example using scikit-learn

    - -

    -

    # Importing various packages
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Importing various packages
 from random import random, seed
 import numpy as np
 import matplotlib.pyplot as plt
@@ -1585,40 +1664,53 @@ beta_linreg = np.linalg.inv(X.T @ X) @ (X.T @ y)
 sgdreg = SGDRegressor(max_iter = 50, penalty=None, eta0=0.1)
 sgdreg.fit(x,y.ravel())
 print(sgdreg.intercept_, sgdreg.coef_)
-
    
-
    
-
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Gradient descent and Ridge

    -

    -We have also discussed Ridge regression where the loss function contains a regularized term given by the \( L_2 \) norm of \( \beta \), +

    We have also discussed Ridge regression where the loss function contains a regularized term given by the \( L_2 \) norm of \( \beta \),

    $$ C_{\text{ridge}}(\beta) = \frac{1}{n}||X\beta -\mathbf{y}||^2 + \lambda ||\beta||^2, \ \lambda \geq 0. $$ -

    -In order to minimize \( C_{\text{ridge}}(\beta) \) using GD we only have adjust the gradient as follows +

    In order to minimize \( C_{\text{ridge}}(\beta) \) using GD we adjust the gradient as follows

    $$ \nabla_\beta C_{\text{ridge}}(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ \sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ -\end{bmatrix} + 2\lambda\begin{bmatrix} \beta_0 \\ \beta_1\end{bmatrix} = 2 (X^T(X\beta - \mathbf{y})+\lambda \beta). +\end{bmatrix} + 2\lambda\begin{bmatrix} \beta_0 \\ \beta_1\end{bmatrix} = 2 (\frac{1}{n}X^T(X\beta - \mathbf{y})+\lambda \beta). $$ -

    -We can easily extend our program to minimize \( C_{\text{ridge}}(\beta) \) using gradient descent and compare with the analytical solution given by +

    We can easily extend our program to minimize \( C_{\text{ridge}}(\beta) \) using gradient descent and compare with the analytical solution given by

    $$ -\beta_{\text{ridge}} = \left(X^T X + \lambda I_{2 \times 2} \right)^{-1} X^T \mathbf{y}. +\beta_{\text{ridge}} = \left(X^T X + n\lambda I_{2 \times 2} \right)^{-1} X^T \mathbf{y}. $$ -

    +









    -

    Program example for gradient descent with Ridge Regression

    -

    -

    from random import random, seed
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from random import random, seed
 import numpy as np
 import matplotlib.pyplot as plt
 from mpl_toolkits.mplot3d import Axes3D
@@ -1661,10 +1753,23 @@ plt.xlabel(r'$x$')
 plt.ylabel(r'$y$')
 plt.title(r'Gradient descent example for Ridge')
 plt.show()
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Using gradient descent methods, limitations

    -









    -

    Challenge yourself

    -

    -Write a code which implements gradient descent for a logistic regression example. +

    Write a code which implements gradient descent for a logistic regression example.

    -











    -

    Friday October 1

    -











    -

    Stochastic Gradient Descent

    -

    -Stochastic gradient descent (SGD) and variants thereof address some of +

    Stochastic gradient descent (SGD) and variants thereof address some of the shortcomings of the Gradient descent method discussed above. +

    -

    -The underlying idea of SGD comes from the observation that the cost +

    The underlying idea of SGD comes from the observation that the cost function, which we want to minimize, can almost always be written as a sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \), +

    $$ C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, \mathbf{\beta}). $$ -

    -









    +









    Computation of gradients

    -

    -This in turn means that the gradient can be +

    This in turn means that the gradient can be computed as a sum over \( i \)-gradients +

    $$ \nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, \mathbf{\beta}). $$ -

    -Stochasticity/randomness is introduced by only taking the +

    Stochasticity/randomness is introduced by only taking the gradient on a subset of the data called minibatches. If there are \( n \) data points and the size of each minibatch is \( M \), there will be \( n/M \) minibatches. We denote these minibatches by \( B_k \) where \( k=1,\cdots,n/M \). +

    -











    -

    SGD example

    -As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \) +

    As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \) and we choose to have \( M=5 \) minibathces, then each minibatch contains two data points. In particular we have \( B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 = @@ -1738,11 +1833,12 @@ then each minibatch contains two data points. In particular we have have only a single batch with all data points and on the other extreme, you may choose \( M=n \) resulting in a minibatch for each datapoint, i.e \( B_k = \mathbf{x}_k \). +

    -

    -The idea is now to approximate the gradient by replacing the sum over +

    The idea is now to approximate the gradient by replacing the sum over all data points with a sum over the data points in one the minibatches picked at random in each gradient descent step +

    $$ \nabla_{\beta} C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, @@ -1750,34 +1846,34 @@ C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, c_i(\mathbf{x}_i, \mathbf{\beta}). $$ -

    -









    +









    The gradient step

    -

    -Thus a gradient descent step now looks like +

    Thus a gradient descent step now looks like

    $$ \beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i, \mathbf{\beta}) $$ -

    -where \( k \) is picked at random with equal +

    where \( k \) is picked at random with equal probability from \( [1,n/M] \). An iteration over the number of minibathces (n/M) is commonly referred to as an epoch. Thus it is typical to choose a number of epochs and for each epoch iterate over the number of minibatches, as exemplified in the code below. +

    -











    -

    Simple example code

    -

    -

    import numpy as np 
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np 
 
 n = 100 #100 datapoints 
 M = 5   #size of each minibatch
@@ -1791,23 +1887,34 @@ j = 0
         #Compute the gradient using the data in minibatch Bk
         #Compute new suggestion for 
         j += 1
-
    
-
    
-Taking the gradient only on a subset of the data has two important
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Taking the gradient only on a subset of the data has two important benefits. First, it introduces randomness which decreases the chance that our opmization scheme gets stuck in a local minima. Second, if the size of the minibatches are small relative to the number of datapoints (\( M < n \)), the computation of the gradient is much cheaper since we sum over the datapoints in the \( k-th \) minibatch and not all \( n \) datapoints. +

    -











    -

    When do we stop?

    -

    -A natural question is when do we stop the search for a new minimum? +

    A natural question is when do we stop the search for a new minimum? One possibility is to compute the full gradient after a given number of epochs and check if the norm of the gradient is smaller than some threshold and stop if true. However, the condition that the gradient @@ -1817,31 +1924,33 @@ evaluate the cost function at this point, store the result and continue the search. If the test kicks in at a later stage we can compare the values of the cost function and keep the \( \beta \) that gave the lowest value. +

    -











    -

    Slightly different approach

    -

    -Another approach is to let the step length \( \gamma_j \) depend on the +

    Another approach is to let the step length \( \gamma_j \) depend on the number of epochs in such a way that it becomes very small after a reasonable time such that we do not move at all. +

    -

    -As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \). +

    As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).

    -

    -In this way we can fix the number of epochs, compute \( \beta \) and +

    In this way we can fix the number of epochs, compute \( \beta \) and evaluate the cost function at the end. Repeating the computation will give a different result since the scheme is random by design. Then we pick the final \( \beta \) that gives the lowest value of the cost function. +

    -

    -

    import numpy as np 
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np 
 
 def step_length(t,t0,t1):
     return t0/(t+t1)
@@ -1865,16 +1974,33 @@ j = 0
         j += 1
 
 print("gamma_j after %d epochs: %g" % (n_epochs,gamma_j))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Program for stochastic gradient

    -

    -

    # Importing various packages
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Importing various packages
 from math import exp, sqrt
 from random import random, seed
 import numpy as np
@@ -1938,20 +2064,31 @@ plt.xlabel(r'$x$')
 plt.ylabel(r'$y$')
 plt.title(r'Random numbers ')
 plt.show()
-
    
-
    
-Challenge: try to write a similar code for a Logistic Regression case.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Challenge: try to write a similar code for a Logistic Regression case.

    -











    -

    Momentum based GD

    -

    -The stochastic gradient descent (SGD) is almost always used with a +

    The stochastic gradient descent (SGD) is almost always used with a momentum or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically implemented as follows +

    $$ \begin{align} @@ -1961,8 +2098,7 @@ $$ \end{align} $$ -

    -where we have introduced a momentum parameter \( \gamma \), with +

    where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum @@ -1972,67 +2108,62 @@ running average of recently encountered gradients and used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is +

    $$ \Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t), $$ -where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\boldsymbol{\theta}_{t-1} \). +

    where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\boldsymbol{\theta}_{t-1} \).

    -











    -

    More on momentum based approaches

    -

    -Let us try to get more intuition from these equations. It is helpful +

    Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential \( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by +

    $$ m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}). $$ -

    -We can discretize this equation in the usual way to get +

    We can discretize this equation in the usual way to get

    $$ m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}). $$ -

    -Rearranging this equation, we can rewrite this as +

    Rearranging this equation, we can rewrite this as

    $$ \Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t. $$ -

    -









    +









    Momentum parameter

    -

    -Notice that this equation is identical to previous one if we identify +

    Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as: +

    $$ \gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}. $$ -

    -Thus, as the name suggests, the momentum parameter is proportional to +

    Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \). +

    -

    -Why is momentum useful? SGD momentum helps the gradient descent +

    Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in @@ -2041,18 +2172,19 @@ and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models. +

    -

    -These beneficial properties of momentum can sometimes become even more +

    These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG). +

    -

    -In the NAG algorithm, rather than calculating the gradient at the +

    In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule +

    $$ \begin{align} @@ -2062,16 +2194,12 @@ $$ \end{align} $$ -

    -One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \). +

    One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).

    -











    -

    Second moment of the gradient

    -

    -In stochastic gradient descent, with and without momentum, we still +

    In stochastic gradient descent, with and without momentum, we still have to specify a schedule for tuning the learning rates \( \eta_t \) as a function of time. As discussed in the context of Newton's method, this presents a number of dilemmas. The learning rate is @@ -2086,23 +2214,22 @@ extremely large models. Ideally, we would like to be able to adaptively change the step size to match the landscape without paying the steep computational price of calculating or approximating Hessians. +

    -

    -Recently, a number of methods have been introduced that accomplish +

    Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM. +

    -











    -

    RMS prop

    -

    -In RMS prop, in addition to keeping a running average of the first +

    In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by +

    $$ \begin{align} @@ -2113,8 +2240,7 @@ $$ \end{align} $$ -

    -where \( \beta \) controls the averaging time of the second moment and is +

    where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication @@ -2123,14 +2249,12 @@ is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions. +

    -











    -

    ADAM optimizer

    -

    -A related algorithm is the ADAM optimizer. In ADAM, we keep a running +

    A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and @@ -2142,6 +2266,7 @@ are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below) +

    $$ \begin{align} @@ -2156,26 +2281,25 @@ $$ \end{align} $$ -

    -where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and +

    where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop. +

    -

    -Like in RMSprop, the effective step size of a parameter depends on the +

    Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \boldsymbol{\mathbf{s}}_t - (\boldsymbol{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by +

    $$ \Delta \theta_{t+1}= -\eta_t { \boldsymbol{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. $$ -

    -









    +









    Practical tips

    +

    Geron's text, see chapter 11, has several interesting discussions.

    -Geron's text, see chapter 11, has several interesting discussions. - -











    -

    Automatic differentiation

    -

    -Automatic differentiation (AD), +

    Automatic differentiation (AD), also called algorithmic differentiation or computational differentiation,is a set of techniques to numerically evaluate the derivative of a function @@ -2206,38 +2326,42 @@ operations, derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program. +

    -

    -Automatic differentiation is neither: +

    Automatic differentiation is neither:

    - -Symbolic differentiation can lead to inefficient code and faces the +

    Symbolic differentiation can lead to inefficient code and faces the difficulty of converting a computer program into a single expression, while numerical differentiation can introduce round-off errors in the discretization process and cancellation +

    -

    -Python has tools for so-called automatic differentiation. +

    Python has tools for so-called automatic differentiation. Consider the following example +

    $$ f(x) = \sin\left(2\pi x + x^2\right) $$ -which has the following derivative +

    which has the following derivative

    $$ f'(x) = \cos\left(2\pi x + x^2\right)\left(2\pi + 2x\right) $$ -Using autograd we have +

    Using autograd we have

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 
 # To do elementwise differentiation:
 from autograd import elementwise_grad as egrad 
@@ -2271,23 +2395,40 @@ plt.legend()
 plt.show()
 
 print("The max absolute difference is: %g"%(np.max(np.abs(computed - analytic))))
-
    
-
    
-
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Using autograd

    -

    -Here we +

    Here we experiment with what kind of functions Autograd is capable of finding the gradient of. The following Python functions are just meant to illustrate what Autograd can do, but please feel free to experiment with other, possibly more complicated, functions as well. +

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 
 def f1(x):
@@ -2304,21 +2445,38 @@ a = 1.0
 # Compare with the analytical derivative, that is f1'(x) = 3*x**2 
 grad_analytical = 3*a**2
 print("The gradient of f1 evaluated at a = %g by finding the analytic expression is: %g"%(a,grad_analytical))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Autograd with more complicated functions

    -

    -To differentiate with respect to two (or more) arguments of a Python +

    To differentiate with respect to two (or more) arguments of a Python function, Autograd need to know at which variable the function if being differentiated with respect to. +

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f2(x1,x2):
     return 3*x1**3 + x2*(x1 - 5) + 1
@@ -2351,19 +2509,34 @@ f2_grad_x2_analytical = x1 - 5
 
 print("The derivative of f2 w.r.t x2: %g"%( f2_grad_x2(x1,x2) ))
 print("The analytical derivative of f2 w.r.t x2: %g"%( f2_grad_x2(x1,x2) ))
-
    
-
    
-Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.

    -











    -

    More complicated functions using the elements of their arguments directly

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f3(x): # Assumes x is an array of length 5 or higher
     return 2*x[0] + 3*x[1] + 5*x[2] + 7*x[3] + 11*x[4]**2
@@ -2380,24 +2553,40 @@ f3_grad_analytical = np.array([2, # Print the analytical gradient:
 print("The analytical gradient of f3 is: ", f3_grad_analytical)
-
    
-
    
-Note that in this case, when sending an array as input argument, the
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Note that in this case, when sending an array as input argument, the output from Autograd is another array. This is the true gradient of the function, as opposed to the function in the previous example. By using arrays to represent the variables, the output from Autograd might be easier to work with, as the output is closer to what one could expect form a gradient-evaluting function. +

    -

    -

    Functions using mathematical functions from Numpy

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f4(x):
     return np.sqrt(1+x**2) + np.exp(x) + np.sin(2*np.pi*x)
@@ -2414,16 +2603,33 @@ f4_grad_analytical = x/np.sqrt(1 + x**# Print the analytical gradient:
 print("The analytical gradient of f4 at x = %g is: %g"%(x,f4_grad_analytical))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    More autograd

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f5(x):
     if x >= 0:
@@ -2437,16 +2643,33 @@ x = 2.7
 
 # Print the computed derivative:
 print("The computed derivative of f5 at x = %g is: %g"%(x,f5_grad(x)))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    And with loops

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f6_for(x):
     val = 0
@@ -2470,11 +2693,29 @@ x = 0.5
 # Print the computed derivaties of f6_for and f6_while
 print("The computed derivative of f6_for at x = %g is: %g"%(x,f6_for_grad(x)))
 print("The computed derivative of f6_while at x = %g is: %g"%(x,f6_while_grad(x)))
-
    
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + -
    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 # Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9
 # The analytical derivative is: sum(i*x**(i-1)) 
@@ -2483,15 +2724,32 @@ f6_grad_analytical = 0
     f6_grad_analytical += i*x**(i-1)
 
 print("The analytical derivative of f6 at x = %g is: %g"%(x,f6_grad_analytical))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Using recursion

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 
 def f7(n): # Assume that n is an integer
@@ -2518,22 +2776,36 @@ f7_grad_analytical = 0
     f7_grad_analytical += tmp
 
 print("The analytical derivative of f7 at n = %d is: %g"%(n,f7_grad_analytical))
-
    
-
    
-Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.

    -











    -

    Unsupported functions

    -Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd. +

    Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.

    -

    -Assigning a value to the variable being differentiated with respect to -

    +

    Assigning a value to the variable being differentiated with respect to

    -
    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f8(x): # Assume x is an array
     x[2] = 3
@@ -2544,18 +2816,33 @@ f8_grad = grad(f8)
 x = 8.4
 
 print("The derivative of f8 is:",f8_grad(x))
-
    
-
    
-Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.

    -











    -

    The syntax a.dot(b) when finding the dot product

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f9(a): # Assume a is an array with 2 elements
     b = np.array([1.0,2.0])
@@ -2566,16 +2853,34 @@ f9_grad = grad(f9)
 x = np.array([1.0,0.0])
 
 print("The derivative of f9 is:",f9_grad(x))
-
    
-
    
-Here we are told that the 'dot' function does not belong to Autograd's
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Here we are told that the 'dot' function does not belong to Autograd's version of a Numpy array. To overcome this, an alternative syntax which also computed the dot product can be used: +

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f9_alternative(x): # Assume a is an array with 2 elements
     b = np.array([1.0,2.0])
@@ -2589,31 +2894,56 @@ x = np.array([3.0,# The analytical gradient of the dot product of vectors x and b with two elements (x_1,x_2) and (b_1, b_2) respectively
 # w.r.t x is (b_1, b_2).
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    -The documentation recommends to avoid inplace operations such as -

    +

    The documentation recommends to avoid inplace operations such as

    -
    a += b
+
    
+  
    
+    
    
+      
    
+        
    
+  
    a += b
 a -= b
 a*= b
 a /=b
-
    
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + - -
    © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html index dcb043762..5c0ddf7a9 100644 --- a/doc/pub/week39/html/week39.html +++ b/doc/pub/week39/html/week39.html @@ -1,6 +1,7 @@ @@ -8,29 +9,97 @@ Automatically generated HTML file from DocOnce source - Week 39: Optimization and Gradient Methods - - - - +
    +

    Week 39: Optimization and Gradient Methods

    +
    - - -

    Week 39: Optimization and Gradient Methods

    - -

    -

    Morten Hjorth-Jensen [1, 2]
    - -

    +

    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 2, 2021

    +
    +
    -
    [1] Department of Physics, University of Oslo
    -
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    -
    -

    -

    Oct 1, 2021

    -
    -











    -

    Plan for week 39

    - -See lecture notes for week 39. +

    See lecture notes for week 39. For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. +

    -

    For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11. -These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. +

    These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning.

    -











    -

    Thursday September 30

    -

    Overview Video, why do we care about gradient methods? -











    -

    Searching for Optimal Regularization Parameters \( \lambda \)

    -

    -In project 1, when using Ridge and Lasso regression, we end up +

    In project 1, when using Ridge and Lasso regression, we end up searching for the optimal parameter \( \lambda \) which minimizes our selected scores (MSE or \( R2 \) values for example). The brute force approach, as discussed in the code here for Ridge regression, consists in evaluating the MSE as function of different \( \lambda \) values. Based on these calculations, one tries then to determine the value of the hyperparameter \( \lambda \) which results in optimal scores (for example the smallest MSE or an \( R2=1 \)). -

    +

    -
    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
 from sklearn.model_selection import train_test_split
@@ -375,26 +443,42 @@ plt.xlabel('
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function RidgeCV instead of just calling the Ridge function. For RidgeCV we need to pass the array of \( \lambda \) values.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Here we have performed a rather data greedy calculation as function of the regularization parameter \( \lambda \). There is no resampling here. The latter can easily be added by employing the function RidgeCV instead of just calling the Ridge function. For RidgeCV we need to pass the array of \( \lambda \) values. By inspecting the figure we can in turn determine which is the optimal regularization parameter. -This becomes however less functional in the long run. +This becomes however less functional in the long run. +

    -











    -

    -

    -An alternative is to use the so-called grid search functionality +

    An alternative is to use the so-called grid search functionality included with the library Scikit-Learn, as demonstrated for the same example here. +

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 from sklearn.model_selection import train_test_split
 from sklearn.linear_model import Ridge
 from sklearn.model_selection import GridSearchCV
@@ -436,23 +520,33 @@ ypredictRidge = gridsearchprint(f"Best estimated lambda-value: {gridsearch.best_estimator_.alpha}")
 print(f"MSE score: {MSE(y_test,ypredictRidge)}")
 print(f"R2 score: {R2(y_test,ypredictRidge)}")
-
    
-
    
-By default the grid search function includes cross validation with
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    By default the grid search function includes cross validation with five folds. The Scikit-Learn documentation contains more information on how to set the different parameters. +

    -

    -If we take out the random noise, running the above codes results in \( \lambda=0 \) yielding the best fit. +

    If we take out the random noise, running the above codes results in \( \lambda=0 \) yielding the best fit.

    -











    -

    -

    -An alternative to the above manual grid set up, is to use a random +

    An alternative to the above manual grid set up, is to use a random search where the parameters are tuned from a random distribution (uniform below) for a fixed number of iterations. A model is constructed and evaluated for each combination of chosen parameters. @@ -460,11 +554,16 @@ We repeat the previous example but now with a random search. Note that values of \( \lambda \) are now limited to be within \( x\in [0,1] \). This domain may not be the most relevant one for the specific case under study. +

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 from sklearn.model_selection import train_test_split
 from sklearn.linear_model import Ridge
 from sklearn.model_selection import GridSearchCV
@@ -507,14 +606,26 @@ ypredictRidge = gridsearchprint(f"Best estimated lambda-value: {gridsearch.best_estimator_.alpha}")
 print(f"MSE score: {MSE(y_test,ypredictRidge)}")
 print(f"R2 score: {R2(y_test,ypredictRidge)}")
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Optimization, the central part of any Machine Learning algortithm

    -

    -Almost every problem in machine learning and data science starts with +

    Almost every problem in machine learning and data science starts with a dataset \( X \), a model \( g(\beta) \), which is a function of the parameters \( \beta \) and a cost function \( C(X, g(\beta)) \) that allows us to judge how well the model \( g(\beta) \) explains the observations @@ -522,19 +633,18 @@ us to judge how well the model \( g(\beta) \) explains the observations the cost function. Ideally we would be able to solve for \( \beta \) analytically, however this is not possible in general and we must use some approximative/numerical method to compute the minimum. +

    -











    -

    Revisiting our Logistic Regression case

    -

    -In our discussion on Logistic Regression we studied the +

    In our discussion on Logistic Regression we studied the case of two classes, with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assumed also that we have only two parameters \( \beta \) in our fitting, that is we defined probabilities +

    $$ \begin{align*} @@ -543,120 +653,105 @@ p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}), \end{align*} $$ -where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \). +

    where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).

    -











    -

    The equations to solve

    -

    -Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \) +

    Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \) elements \( y_i \), an \( n\times p \) matrix \( \boldsymbol{X} \) which contains the \( x_i \) values and a vector \( \boldsymbol{p} \) of fitted probabilities \( p(y_i\vert x_i,\boldsymbol{\beta}) \). We rewrote in a more compact form the first derivative of the cost function as +

    $$ \frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right). $$ -

    -If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements +

    If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements \( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as +

    $$ \frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}. $$ -This defines what is called the Hessian matrix. +

    This defines what is called the Hessian matrix.

    -











    -

    Solving using Newton-Raphson's method

    -

    -If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives. +

    If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives.

    -

    -Our iterative scheme is then given by +

    Our iterative scheme is then given by

    $$ \boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T}\right)^{-1}_{\boldsymbol{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}\right)_{\boldsymbol{\beta}^{\mathrm{old}}}, $$ -or in matrix form as +

    or in matrix form as

    $$ \boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X} \right)^{-1}\times \left(-\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{p}) \right)_{\boldsymbol{\beta}^{\mathrm{old}}}. $$ -The right-hand side is computed with the old values of \( \beta \). +

    The right-hand side is computed with the old values of \( \beta \).

    -

    -If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement. +

    If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement.

    -











    -

    Brief reminder on Newton-Raphson's method

    -

    -Let us quickly remind ourselves how we derive the above method. +

    Let us quickly remind ourselves how we derive the above method.

    -

    -Perhaps the most celebrated of all one-dimensional root-finding +

    Perhaps the most celebrated of all one-dimensional root-finding routines is Newton's method, also called the Newton-Raphson method. This method requires the evaluation of both the function \( f \) and its derivative \( f' \) at arbitrary points. If you can only calculate the derivative numerically and/or your function is not of the smooth type, we normally discourage the use of this method. +

    -











    -

    The equations

    -

    -The Newton-Raphson formula consists geometrically of extending the +

    The Newton-Raphson formula consists geometrically of extending the tangent line at a current point until it crosses zero, then setting the next guess to the abscissa of that zero-crossing. The mathematics behind this method is rather simple. Employing a Taylor expansion for \( x \) sufficiently close to the solution \( s \), we have +

    $$ f(s)=0=f(x)+(s-x)f'(x)+\frac{(s-x)^2}{2}f''(x) +\dots. \label{eq:taylornr} $$ -

    -For small enough values of the function and for well-behaved +

    For small enough values of the function and for well-behaved functions, the terms beyond linear are unimportant, hence we obtain +

    $$ f(x)+(s-x)f'(x)\approx 0, $$ -yielding +

    yielding

    $$ s\approx x-\frac{f(x)}{f'(x)}. $$ -

    -Having in mind an iterative procedure, it is natural to start iterating with +

    Having in mind an iterative procedure, it is natural to start iterating with

    $$ x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)}. $$ -

    -









    +









    Simple geometric interpretation

    -

    -The above is Newton-Raphson's method. It has a simple geometric +

    The above is Newton-Raphson's method. It has a simple geometric interpretation, namely \( x_{n+1} \) is the point where the tangent from \( (x_n,f(x_n)) \) crosses the \( x \)-axis. Close to the solution, Newton-Raphson converges fast to the desired result. However, if we @@ -667,21 +762,20 @@ from the true root as to let the search interval include a local maximum or minimum of the function. If an iteration places a trial guess near such a local extremum, so that the first derivative nearly vanishes, then Newton-Raphson may fail totally +

    -











    -

    Extending to more than one variable

    -

    -Newton's method can be generalized to systems of several non-linear equations +

    Newton's method can be generalized to systems of several non-linear equations and variables. Consider the case with two equations +

    $$ \begin{array}{cc} f_1(x_1,x_2) &=0\\ f_2(x_1,x_2) &=0,\end{array} $$ -which we Taylor expand to obtain +

    which we Taylor expand to obtain

    $$ \begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1 @@ -693,7 +787,7 @@ $$ \end{array}. $$ -Defining the Jacobian matrix \( {\bf \boldsymbol{J}} \) we have +

    Defining the Jacobian matrix \( {\bf \boldsymbol{J}} \) we have

    $$ {\bf \boldsymbol{J}}=\left( \begin{array}{cc} \partial f_1/\partial x_1 & \partial f_1/\partial x_2 \\ @@ -701,78 +795,71 @@ $$ \end{array} \right), $$ -we can rephrase Newton's method as +

    we can rephrase Newton's method as

    $$ \left(\begin{array}{c} x_1^{n+1} \\ x_2^{n+1} \end{array} \right)= \left(\begin{array}{c} x_1^{n} \\ x_2^{n} \end{array} \right)+ \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right), $$ -where we have defined +

    where we have defined

    $$ \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)= -{\bf \boldsymbol{J}}^{-1} \left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right). $$ -We need thus to compute the inverse of the Jacobian matrix and it +

    We need thus to compute the inverse of the Jacobian matrix and it is to understand that difficulties may arise in case \( {\bf \boldsymbol{J}} \) is nearly singular. +

    -

    -It is rather straightforward to extend the above scheme to systems of -more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function. +

    It is rather straightforward to extend the above scheme to systems of +more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function. +

    -











    -

    Steepest descent

    -

    -The basic idea of gradient descent is +

    The basic idea of gradient descent is that a function \( F(\mathbf{x}) \), \( \mathbf{x} \equiv (x_1,\cdots,x_n) \), decreases fastest if one goes from \( \bf {x} \) in the direction of the negative gradient \( -\nabla F(\mathbf{x}) \). +

    -

    -It can be shown that if +

    It can be shown that if

    $$ \mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), $$ -with \( \gamma_k > 0 \). +

    with \( \gamma_k > 0 \).

    -

    -For \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq +

    For \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq F(\mathbf{x}_k) \). This means that for a sufficiently small \( \gamma_k \) we are always moving towards smaller function values, i.e a minimum. +

    -

    -

    More on Steepest descent

    -

    -The previous observation is the basis of the method of steepest +

    The previous observation is the basis of the method of steepest descent, which is also referred to as just gradient descent (GD). One starts with an initial guess \( \mathbf{x}_0 \) for a minimum of \( F \) and computes new approximations according to +

    $$ \mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ k \geq 0. $$ -

    -The parameter \( \gamma_k \) is often referred to as the step length or +

    The parameter \( \gamma_k \) is often referred to as the step length or the learning rate within the context of Machine Learning. +

    -

    -

    The ideal

    -

    -Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global +

    Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global minimum of the function \( F \). In general we do not know if we are in a global or local minimum. In the special case when \( F \) is a convex function, all local minima are also global minima, so in this case @@ -780,188 +867,165 @@ gradient descent can converge to the global solution. The advantage of this scheme is that it is conceptually simple and straightforward to implement. However the method in this form has some severe limitations: +

    -

    -In machine learing we are often faced with non-convex high dimensional +

    In machine learing we are often faced with non-convex high dimensional cost functions with many local minima. Since GD is deterministic we will get stuck in a local minimum, if the method converges, unless we have a very good intial guess. This also implies that the scheme is sensitive to the chosen initial condition. +

    -

    -Note that the gradient is a function of \( \mathbf{x} = +

    Note that the gradient is a function of \( \mathbf{x} = (x_1,\cdots,x_n) \) which makes it expensive to compute numerically. +

    -

    -

    The sensitiveness of the gradient descent

    -

    -The gradient descent method +

    The gradient descent method is sensitive to the choice of learning rate \( \gamma_k \). This is due to the fact that we are only guaranteed that \( F(\mathbf{x}_{k+1}) \leq F(\mathbf{x}_k) \) for sufficiently small \( \gamma_k \). The problem is to determine an optimal learning rate. If the learning rate is chosen too small the method will take a long time to converge and if it is too large we can experience erratic behavior. +

    -

    -Many of these shortcomings can be alleviated by introducing +

    Many of these shortcomings can be alleviated by introducing randomness. One such method is that of Stochastic Gradient Descent (SGD), see below. +

    -

    -

    Convex functions

    -

    -Ideally we want our cost/loss function to be convex(concave). +

    Ideally we want our cost/loss function to be convex(concave).

    -

    -First we give the definition of a convex set: A set \( C \) in +

    First we give the definition of a convex set: A set \( C \) in \( \mathbb{R}^n \) is said to be convex if, for all \( x \) and \( y \) in \( C \) and all \( t \in (0,1) \) , the point \( (1 − t)x + ty \) also belongs to C. Geometrically this means that every point on the line segment connecting \( x \) and \( y \) is in \( C \) as discussed below. +

    -

    -The convex subsets of \( \mathbb{R} \) are the intervals of +

    The convex subsets of \( \mathbb{R} \) are the intervals of \( \mathbb{R} \). Examples of convex sets of \( \mathbb{R}^2 \) are the regular polygons (triangles, rectangles, pentagons, etc...). +

    -











    -

    Convex function

    -

    -Convex function: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below. +

    Convex function: Let \( X \subset \mathbb{R}^n \) be a convex set. Assume that the function \( f: X \rightarrow \mathbb{R} \) is continuous, then \( f \) is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all \( x_1, x_2 \in X \) and for all \( t \in [0,1] \). If \( \leq \) is replaced with a strict inequaltiy in the definition, we demand \( x_1 \neq x_2 \) and \( t\in(0,1) \) then \( f \) is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \( f(x_1) \) and \( f(x_2) \), the value of the function on the interval \( [x_1,x_2] \) is always below the line as illustrated below.

    -











    -

    Conditions on convex functions

    -

    -In the following we state first and second-order conditions which +

    In the following we state first and second-order conditions which ensures convexity of a function \( f \). We write \( D_f \) to denote the domain of \( f \), i.e the subset of \( R^n \) where \( f \) is defined. For more details and proofs we refer to: S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press. +

    -

    First order condition

    -Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for +

    Suppose \( f \) is differentiable (i.e \( \nabla f(x) \) is well defined for all \( x \) in the domain of \( f \)). Then \( f \) is convex if and only if \( D_f \) is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds for all \( x,y \in D_f \). This condition means that for a convex function the first order Taylor expansion (right hand side above) at any point a global under estimator of the function. To convince yourself you can make a drawing of \( f(x) = x^2+1 \) and draw the tangent line to \( f(x) \) and -note that it is always below the graph. +note that it is always below the graph. +

    -

    Second order condition

    -Assume that \( f \) is twice +

    Assume that \( f \) is twice differentiable, i.e the Hessian matrix exists at each point in \( D_f \). Then \( f \) is convex if and only if \( D_f \) is a convex set and its Hessian is positive semi-definite for all \( x\in D_f \). For a single-variable function this reduces to \( f''(x) \geq 0 \). Geometrically this means that \( f \) has nonnegative curvature everywhere. +

    -

    -This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition. +

    This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.

    -











    -

    More on convex functions

    -

    -The next result is of great importance to us and the reason why we are +

    The next result is of great importance to us and the reason why we are going on about convex functions. In machine learning we frequently have to minimize a loss/cost function in order to find the best -parameters for the model we are considering. +parameters for the model we are considering. +

    -

    -Ideally we want the +

    Ideally we want the global minimum (for high-dimensional models it is hard to know if we have local or global minimum). However, if the cost/loss function is convex the following result provides invaluable information: +

    -

    Any minimum is global for convex functions

    -Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \) +

    Consider the problem of finding \( x \in \mathbb{R}^n \) such that \( f(x) \) is minimal, where \( f \) is convex and differentiable. Then, any point \( x^* \) that satisfies \( \nabla f(x^*) = 0 \) is a global minimum. +

    -

    -This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum. +

    This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.

    -











    -

    Some simple problems

    1. Show that \( f(x)=x^2 \) is convex for \( x \in \mathbb{R} \) using the definition of convexity. Hint: If you re-write the definition, \( f \) is convex if the following holds for all \( x,y \in D_f \) and any \( \lambda \in [0,1] \) $\lambda f(x)+(1-\lambda)f(y)-f(\lambda x + (1-\lambda) y ) \geq 0$.
    2. Using the second order condition show that the following functions are convex on the specified domain.
      3. - -
    3. Let \( f(x) = x^2 \) and \( g(x) = e^x \). Show that \( f(g(x)) \) and \( g(f(x)) \) is convex for \( x \in \mathbb{R} \). Also show that if \( f(x) \) is any convex function than \( h(x) = e^{f(x)} \) is convex.
    4. A norm is any function that satisfy the following properties
      5. - -
    +

    Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).

    -Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this). - -











    -

    Standard steepest descent

    -

    -Before we proceed, we would like to discuss the approach called the +

    Before we proceed, we would like to discuss the approach called the standard Steepest descent (different from the above steepest descent discussion), which again leads to us having to be able to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG). +

    -

    The success of the CG method -for finding solutions of non-linear problems is based on the theory +

    for finding solutions of non-linear problems is based on the theory of conjugate gradients for linear systems of equations. It belongs to the class of iterative methods for solving problems from linear algebra of the type +

    $$ \begin{equation*} \boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}. \end{equation*} $$ -

    -In the iterative process we end up with a problem like +

    In the iterative process we end up with a problem like

    $$ \begin{equation*} @@ -969,116 +1033,106 @@ $$ \end{equation*} $$ -where \( \boldsymbol{r} \) is the so-called residual or error in the iterative process. +

    where \( \boldsymbol{r} \) is the so-called residual or error in the iterative process.

    -

    -When we have found the exact solution, \( \boldsymbol{r}=0 \). +

    When we have found the exact solution, \( \boldsymbol{r}=0 \).

    -











    -

    Gradient method

    -

    -The residual is zero when we reach the minimum of the quadratic equation +

    The residual is zero when we reach the minimum of the quadratic equation

    $$ \begin{equation*} P(\boldsymbol{x})=\frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T\boldsymbol{b}, \end{equation*} $$ -

    -with the constraint that the matrix \( \boldsymbol{A} \) is positive definite and -symmetric. This defines also the Hessian and we want it to be positive definite. +

    with the constraint that the matrix \( \boldsymbol{A} \) is positive definite and +symmetric. This defines also the Hessian and we want it to be positive definite. +

    -











    -

    Steepest descent method

    -

    -We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). +

    We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). We can assume without loss of generality that +

    $$ \begin{equation*} \boldsymbol{x}_0=0, \end{equation*} $$ -or consider the system +

    or consider the system

    $$ \begin{equation*} \boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0, \end{equation*} $$ -instead. +

    instead.

    -











    -

    Steepest descent method

    -One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form +

    One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form

    $$ \begin{equation*} f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n. \end{equation*} $$ -This suggests taking the first basis vector \( \boldsymbol{r}_1 \) (see below for definition) +

    This suggests taking the first basis vector \( \boldsymbol{r}_1 \) (see below for definition) to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \), which equals +

    $$ \begin{equation*} \boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b}, \end{equation*} $$ -and +

    and \( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \). - - +

    -











    -

    Final expressions

    -We can compute the residual iteratively as +

    We can compute the residual iteratively as

    $$ \begin{equation*} \boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1}, \end{equation*} $$ -which equals +

    which equals

    $$ \begin{equation*} \boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{r}_k), \end{equation*} $$ -or +

    or

    $$ \begin{equation*} (\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{r}_k, \end{equation*} $$ -which gives +

    which gives

    $$ \alpha_k = \frac{\boldsymbol{r}_k^T\boldsymbol{r}_k}{\boldsymbol{r}_k^T\boldsymbol{A}\boldsymbol{r}_k} $$ -leading to the iterative scheme +

    leading to the iterative scheme

    $$ \begin{equation*} \boldsymbol{x}_{k+1}=\boldsymbol{x}_k-\alpha_k\boldsymbol{r}_{k}, @@ -1087,15 +1141,17 @@ $$
    -











    -

    Steepest descent example

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 import numpy.linalg as la
 
 import scipy.optimize as sopt
@@ -1115,116 +1171,196 @@ ax = fig.= np.mgrid[-3:3:50j,-3:3:50j]
 fmesh = f(np.array([xmesh, ymesh]))
 ax.plot_surface(xmesh, ymesh, fmesh)
-
    
-
    
-And then as countor plot
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    And then as countor plot

    -
    pt.axis("equal")
+
    
+  
    
+    
    
+      
    
+        
    
+  
    pt.axis("equal")
 pt.contour(xmesh, ymesh, fmesh)
 guesses = [np.array([2, 2./5])]
-
    
-
    
-Find guesses
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Find guesses

    -
    x = guesses[-1]
+
    
+  
    
+    
    
+      
    
+        
    
+  
    x = guesses[-1]
 s = -df(x)
-
    
-
    
-Run it!
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Run it!

    -
    def f1d(alpha):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def f1d(alpha):
     return f(x + alpha*s)
 
 alpha_opt = sopt.golden(f1d)
 next_guess = x + alpha_opt * s
 guesses.append(next_guess)
 print(next_guess)
-
    
-
    
-What happened?
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    What happened?

    -
    pt.axis("equal")
+
    
+  
    
+    
    
+      
    
+        
    
+  
    pt.axis("equal")
 pt.contour(xmesh, ymesh, fmesh, 50)
 it_array = np.array(guesses)
 pt.plot(it_array.T[0], it_array.T[1], "x-")
-
    
-
    
-Note that we did only one iteration here. We can easily add more using our previous guesses.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Note that we did only one iteration here. We can easily add more using our previous guesses.

    -











    -

    Conjugate gradient method

    -In the CG method we define so-called conjugate directions and two vectors +

    In the CG method we define so-called conjugate directions and two vectors \( \boldsymbol{s} \) and \( \boldsymbol{t} \) are said to be conjugate if +

    $$ \begin{equation*} \boldsymbol{s}^T\boldsymbol{A}\boldsymbol{t}= 0. \end{equation*} $$ -The philosophy of the CG method is to perform searches in various conjugate directions +

    The philosophy of the CG method is to perform searches in various conjugate directions of our vectors \( \boldsymbol{x}_i \) obeying the above criterion, namely +

    $$ \begin{equation*} \boldsymbol{x}_i^T\boldsymbol{A}\boldsymbol{x}_j= 0. \end{equation*} $$ -Two vectors are conjugate if they are orthogonal with respect to +

    Two vectors are conjugate if they are orthogonal with respect to this inner product. Being conjugate is a symmetric relation: if \( \boldsymbol{s} \) is conjugate to \( \boldsymbol{t} \), then \( \boldsymbol{t} \) is conjugate to \( \boldsymbol{s} \). +

    -











    -

    Conjugate gradient method

    -An example is given by the eigenvectors of the matrix +

    An example is given by the eigenvectors of the matrix

    $$ \begin{equation*} \boldsymbol{v}_i^T\boldsymbol{A}\boldsymbol{v}_j= \lambda\boldsymbol{v}_i^T\boldsymbol{v}_j, \end{equation*} $$ -which is zero unless \( i=j \). +

    which is zero unless \( i=j \).

    -











    -

    Conjugate gradient method

    -Assume now that we have a symmetric positive-definite matrix \( \boldsymbol{A} \) of size +

    Assume now that we have a symmetric positive-definite matrix \( \boldsymbol{A} \) of size \( n\times n \). At each iteration \( i+1 \) we obtain the conjugate direction of a vector +

    $$ \begin{equation*} \boldsymbol{x}_{i+1}=\boldsymbol{x}_{i}+\alpha_i\boldsymbol{p}_{i}. \end{equation*} $$ -We assume that \( \boldsymbol{p}_{i} \) is a sequence of \( n \) mutually conjugate directions. +

    We assume that \( \boldsymbol{p}_{i} \) is a sequence of \( n \) mutually conjugate directions. Then the \( \boldsymbol{p}_{i} \) form a basis of \( R^n \) and we can expand the solution $ \boldsymbol{A}\boldsymbol{x} = \boldsymbol{b}$ in this basis, namely +

    $$ \begin{equation*} @@ -1234,21 +1370,19 @@ $$
    -











    -

    Conjugate gradient method

    -The coefficients are given by +

    The coefficients are given by

    $$ \begin{equation*} \mathbf{A}\mathbf{x} = \sum^{n}_{i=1} \alpha_i \mathbf{A} \mathbf{p}_i = \mathbf{b}. \end{equation*} $$ -Multiplying with \( \boldsymbol{p}_k^T \) from the left gives +

    Multiplying with \( \boldsymbol{p}_k^T \) from the left gives

    $$ \begin{equation*} @@ -1256,7 +1390,7 @@ $$ \end{equation*} $$ -and we can define the coefficients \( \alpha_k \) as +

    and we can define the coefficients \( \alpha_k \) as

    $$ \begin{equation*} @@ -1266,93 +1400,90 @@ $$
    -











    -

    Conjugate gradient method and iterations

    -

    -If we choose the conjugate vectors \( \boldsymbol{p}_k \) carefully, +

    If we choose the conjugate vectors \( \boldsymbol{p}_k \) carefully, then we may not need all of them to obtain a good approximation to the solution \( \boldsymbol{x} \). We want to regard the conjugate gradient method as an iterative method. This will us to solve systems where \( n \) is so large that the direct method would take too much time. +

    -

    -We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). +

    We denote the initial guess for \( \boldsymbol{x} \) as \( \boldsymbol{x}_0 \). We can assume without loss of generality that +

    $$ \begin{equation*} \boldsymbol{x}_0=0, \end{equation*} $$ -or consider the system +

    or consider the system

    $$ \begin{equation*} \boldsymbol{A}\boldsymbol{z} = \boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_0, \end{equation*} $$ -instead. +

    instead.

    -











    -

    Conjugate gradient method

    -One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form +

    One can show that the solution \( \boldsymbol{x} \) is also the unique minimizer of the quadratic form

    $$ \begin{equation*} f(\boldsymbol{x}) = \frac{1}{2}\boldsymbol{x}^T\boldsymbol{A}\boldsymbol{x} - \boldsymbol{x}^T \boldsymbol{x} , \quad \boldsymbol{x}\in\mathbf{R}^n. \end{equation*} $$ -This suggests taking the first basis vector \( \boldsymbol{p}_1 \) +

    This suggests taking the first basis vector \( \boldsymbol{p}_1 \) to be the gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_0 \), which equals +

    $$ \begin{equation*} \boldsymbol{A}\boldsymbol{x}_0-\boldsymbol{b}, \end{equation*} $$ -and +

    and \( \boldsymbol{x}_0=0 \) it is equal \( -\boldsymbol{b} \). The other vectors in the basis will be conjugate to the gradient, hence the name conjugate gradient method. +

    -











    -

    Conjugate gradient method

    -Let \( \boldsymbol{r}_k \) be the residual at the \( k \)-th step: +

    Let \( \boldsymbol{r}_k \) be the residual at the \( k \)-th step:

    $$ \begin{equation*} \boldsymbol{r}_k=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k. \end{equation*} $$ -Note that \( \boldsymbol{r}_k \) is the negative gradient of \( f \) at +

    Note that \( \boldsymbol{r}_k \) is the negative gradient of \( f \) at \( \boldsymbol{x}=\boldsymbol{x}_k \), so the gradient descent method would be to move in the direction \( \boldsymbol{r}_k \). Here, we insist that the directions \( \boldsymbol{p}_k \) are conjugate to each other, so we take the direction closest to the gradient \( \boldsymbol{r}_k \) under the conjugacy constraint. This gives the following expression +

    $$ \begin{equation*} \boldsymbol{p}_{k+1}=\boldsymbol{r}_k-\frac{\boldsymbol{p}_k^T \boldsymbol{A}\boldsymbol{r}_k}{\boldsymbol{p}_k^T\boldsymbol{A}\boldsymbol{p}_k} \boldsymbol{p}_k. @@ -1361,35 +1492,33 @@ $$
    -











    -

    Conjugate gradient method

    -We can also compute the residual iteratively as +

    We can also compute the residual iteratively as

    $$ \begin{equation*} \boldsymbol{r}_{k+1}=\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_{k+1}, \end{equation*} $$ -which equals +

    which equals

    $$ \begin{equation*} \boldsymbol{b}-\boldsymbol{A}(\boldsymbol{x}_k+\alpha_k\boldsymbol{p}_k), \end{equation*} $$ -or +

    or

    $$ \begin{equation*} (\boldsymbol{b}-\boldsymbol{A}\boldsymbol{x}_k)-\alpha_k\boldsymbol{A}\boldsymbol{p}_k, \end{equation*} $$ -which gives +

    which gives

    $$ \begin{equation*} @@ -1399,53 +1528,65 @@ $$
    -

    -

    Revisiting our first homework

    -

    -We will use linear regression as a case study for the gradient descent +

    We will use linear regression as a case study for the gradient descent methods. Linear regression is a great test case for the gradient descent methods discussed in the lectures since it has several desirable properties such as: +

    1. An analytical solution (recall homework set 1).
    2. The gradient can be computed analytically.
    3. The cost function is convex which guarantees that gradient descent converges for small enough learning rates
    +

    We revisit an example similar to what we had in the first homework set. We had a function of the type

    -We revisit an example similar to what we had in the first homework set. We had a function of the type - -

    -

    x = 2*np.random.rand(m,1)
+
    
+  
    
+    
    
+      
    
+        
    
+  
    x = 2*np.random.rand(m,1)
 y = 4+3*x+np.random.randn(m,1)
-
    
-
    
-with \( x_i \in [0,1]  \) is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). 
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    with \( x_i \in [0,1] \) is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). The linear regression model is given by +

    $$ h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x, $$ -such that +

    such that

    $$ \boldsymbol{y}_i = \beta_0 + \beta_1 x_i. $$ -

    - +

    Gradient descent example

    -

    -Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \) +

    Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)

    -

    -It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by (we keep the intercept here) +

    It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by (we keep the intercept here)

    $$ X \equiv \begin{bmatrix} 1 & x_1 \\ @@ -1454,33 +1595,28 @@ X \equiv \begin{bmatrix} \end{bmatrix}. $$ -The cost/loss/risk function is given by ( +

    The cost/loss/risk function is given by (

    $$ C(\beta) = \frac{1}{n}||X\beta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2\right] $$ -and we want to find \( \beta \) such that \( C(\beta) \) is minimized. +

    and we want to find \( \beta \) such that \( C(\beta) \) is minimized.

    -











    -

    The derivative of the cost/loss function

    -

    -Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as +

    Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as

    $$ \nabla_{\beta} C(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ \sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ \end{bmatrix} = \frac{2}{n}X^T(X\beta - \mathbf{y}), $$ -where \( X \) is the design matrix defined above. +

    where \( X \) is the design matrix defined above.

    -











    -

    The Hessian matrix

    -The Hessian matrix of \( C(\beta) \) is given by +

    The Hessian matrix of \( C(\beta) \) is given by

    $$ \boldsymbol{H} \equiv \begin{bmatrix} \frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ @@ -1488,39 +1624,37 @@ $$ \end{bmatrix} = \frac{2}{n}X^T X. $$ -This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite. +

    This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite.

    -











    -

    Simple program

    -

    -We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to +

    We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to

    $$ \beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots $$ -

    -We can use the expression we computed for the gradient and let use a +

    We can use the expression we computed for the gradient and let use a \( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). Note that the code below does not include the latter stop criterion. +

    -

    -And finally we can compare our solution for \( \beta \) with the analytic result given by +

    And finally we can compare our solution for \( \beta \) with the analytic result given by \( \beta= (X^TX)^{-1} X^T \mathbf{y} \). +

    -











    -

    Gradient Descent Example

    -

    -Here our simple example -

    +

    Here our simple example

    -
    # Importing various packages
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Importing various packages
 from random import random, seed
 import numpy as np
 import matplotlib.pyplot as plt
@@ -1565,16 +1699,33 @@ plt.xlabel(r
 plt.ylabel(r'$y$')
 plt.title(r'Gradient descent example')
 plt.show()
-
    
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    And a corresponding example using scikit-learn

    -

    And a corresponding example using scikit-learn

    - -

    -

    # Importing various packages
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Importing various packages
 from random import random, seed
 import numpy as np
 import matplotlib.pyplot as plt
@@ -1590,40 +1741,53 @@ beta_linreg = np= SGDRegressor(max_iter = 50, penalty=None, eta0=0.1)
 sgdreg.fit(x,y.ravel())
 print(sgdreg.intercept_, sgdreg.coef_)
-
    
-
    
-
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Gradient descent and Ridge

    -

    -We have also discussed Ridge regression where the loss function contains a regularized term given by the \( L_2 \) norm of \( \beta \), +

    We have also discussed Ridge regression where the loss function contains a regularized term given by the \( L_2 \) norm of \( \beta \),

    $$ C_{\text{ridge}}(\beta) = \frac{1}{n}||X\beta -\mathbf{y}||^2 + \lambda ||\beta||^2, \ \lambda \geq 0. $$ -

    -In order to minimize \( C_{\text{ridge}}(\beta) \) using GD we only have adjust the gradient as follows +

    In order to minimize \( C_{\text{ridge}}(\beta) \) using GD we adjust the gradient as follows

    $$ \nabla_\beta C_{\text{ridge}}(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ \sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ -\end{bmatrix} + 2\lambda\begin{bmatrix} \beta_0 \\ \beta_1\end{bmatrix} = 2 (X^T(X\beta - \mathbf{y})+\lambda \beta). +\end{bmatrix} + 2\lambda\begin{bmatrix} \beta_0 \\ \beta_1\end{bmatrix} = 2 (\frac{1}{n}X^T(X\beta - \mathbf{y})+\lambda \beta). $$ -

    -We can easily extend our program to minimize \( C_{\text{ridge}}(\beta) \) using gradient descent and compare with the analytical solution given by +

    We can easily extend our program to minimize \( C_{\text{ridge}}(\beta) \) using gradient descent and compare with the analytical solution given by

    $$ -\beta_{\text{ridge}} = \left(X^T X + \lambda I_{2 \times 2} \right)^{-1} X^T \mathbf{y}. +\beta_{\text{ridge}} = \left(X^T X + n\lambda I_{2 \times 2} \right)^{-1} X^T \mathbf{y}. $$ -

    +









    -

    Program example for gradient descent with Ridge Regression

    -

    -

    from random import random, seed
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from random import random, seed
 import numpy as np
 import matplotlib.pyplot as plt
 from mpl_toolkits.mplot3d import Axes3D
@@ -1666,10 +1830,23 @@ plt.xlabel(r
 plt.ylabel(r'$y$')
 plt.title(r'Gradient descent example for Ridge')
 plt.show()
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Using gradient descent methods, limitations

    -









    -

    Challenge yourself

    -

    -Write a code which implements gradient descent for a logistic regression example. +

    Write a code which implements gradient descent for a logistic regression example.

    -











    -

    Friday October 1

    -











    -

    Stochastic Gradient Descent

    -

    -Stochastic gradient descent (SGD) and variants thereof address some of +

    Stochastic gradient descent (SGD) and variants thereof address some of the shortcomings of the Gradient descent method discussed above. +

    -

    -The underlying idea of SGD comes from the observation that the cost +

    The underlying idea of SGD comes from the observation that the cost function, which we want to minimize, can almost always be written as a sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \), +

    $$ C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, \mathbf{\beta}). $$ -

    -









    +









    Computation of gradients

    -

    -This in turn means that the gradient can be +

    This in turn means that the gradient can be computed as a sum over \( i \)-gradients +

    $$ \nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, \mathbf{\beta}). $$ -

    -Stochasticity/randomness is introduced by only taking the +

    Stochasticity/randomness is introduced by only taking the gradient on a subset of the data called minibatches. If there are \( n \) data points and the size of each minibatch is \( M \), there will be \( n/M \) minibatches. We denote these minibatches by \( B_k \) where \( k=1,\cdots,n/M \). +

    -











    -

    SGD example

    -As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \) +

    As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \) and we choose to have \( M=5 \) minibathces, then each minibatch contains two data points. In particular we have \( B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 = @@ -1743,11 +1910,12 @@ then each minibatch contains two data points. In particular we have have only a single batch with all data points and on the other extreme, you may choose \( M=n \) resulting in a minibatch for each datapoint, i.e \( B_k = \mathbf{x}_k \). +

    -

    -The idea is now to approximate the gradient by replacing the sum over +

    The idea is now to approximate the gradient by replacing the sum over all data points with a sum over the data points in one the minibatches picked at random in each gradient descent step +

    $$ \nabla_{\beta} C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, @@ -1755,34 +1923,34 @@ C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, c_i(\mathbf{x}_i, \mathbf{\beta}). $$ -

    -









    +









    The gradient step

    -

    -Thus a gradient descent step now looks like +

    Thus a gradient descent step now looks like

    $$ \beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i, \mathbf{\beta}) $$ -

    -where \( k \) is picked at random with equal +

    where \( k \) is picked at random with equal probability from \( [1,n/M] \). An iteration over the number of minibathces (n/M) is commonly referred to as an epoch. Thus it is typical to choose a number of epochs and for each epoch iterate over the number of minibatches, as exemplified in the code below. +

    -











    -

    Simple example code

    -

    -

    import numpy as np 
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np 
 
 n = 100 #100 datapoints 
 M = 5   #size of each minibatch
@@ -1796,23 +1964,34 @@ j = 0
         #Compute the gradient using the data in minibatch Bk
         #Compute new suggestion for 
         j += 1
-
    
-
    
-Taking the gradient only on a subset of the data has two important
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Taking the gradient only on a subset of the data has two important benefits. First, it introduces randomness which decreases the chance that our opmization scheme gets stuck in a local minima. Second, if the size of the minibatches are small relative to the number of datapoints (\( M < n \)), the computation of the gradient is much cheaper since we sum over the datapoints in the \( k-th \) minibatch and not all \( n \) datapoints. +

    -











    -

    When do we stop?

    -

    -A natural question is when do we stop the search for a new minimum? +

    A natural question is when do we stop the search for a new minimum? One possibility is to compute the full gradient after a given number of epochs and check if the norm of the gradient is smaller than some threshold and stop if true. However, the condition that the gradient @@ -1822,31 +2001,33 @@ evaluate the cost function at this point, store the result and continue the search. If the test kicks in at a later stage we can compare the values of the cost function and keep the \( \beta \) that gave the lowest value. +

    -











    -

    Slightly different approach

    -

    -Another approach is to let the step length \( \gamma_j \) depend on the +

    Another approach is to let the step length \( \gamma_j \) depend on the number of epochs in such a way that it becomes very small after a reasonable time such that we do not move at all. +

    -

    -As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \). +

    As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \).

    -

    -In this way we can fix the number of epochs, compute \( \beta \) and +

    In this way we can fix the number of epochs, compute \( \beta \) and evaluate the cost function at the end. Repeating the computation will give a different result since the scheme is random by design. Then we pick the final \( \beta \) that gives the lowest value of the cost function. +

    -

    -

    import numpy as np 
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np 
 
 def step_length(t,t0,t1):
     return t0/(t+t1)
@@ -1870,16 +2051,33 @@ j = 0
         j += 1
 
 print("gamma_j after %d epochs: %g" % (n_epochs,gamma_j))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Program for stochastic gradient

    -

    -

    # Importing various packages
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Importing various packages
 from math import exp, sqrt
 from random import random, seed
 import numpy as np
@@ -1943,20 +2141,31 @@ plt.xlabel(r
 plt.ylabel(r'$y$')
 plt.title(r'Random numbers ')
 plt.show()
-
    
-
    
-Challenge: try to write a similar code for a Logistic Regression case.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Challenge: try to write a similar code for a Logistic Regression case.

    -











    -

    Momentum based GD

    -

    -The stochastic gradient descent (SGD) is almost always used with a +

    The stochastic gradient descent (SGD) is almost always used with a momentum or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically implemented as follows +

    $$ \begin{align} @@ -1966,8 +2175,7 @@ $$ \end{align} $$ -

    -where we have introduced a momentum parameter \( \gamma \), with +

    where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum @@ -1977,67 +2185,62 @@ running average of recently encountered gradients and used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is +

    $$ \Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t), $$ -where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\boldsymbol{\theta}_{t-1} \). +

    where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\boldsymbol{\theta}_{t-1} \).

    -











    -

    More on momentum based approaches

    -

    -Let us try to get more intuition from these equations. It is helpful +

    Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential \( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by +

    $$ m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}). $$ -

    -We can discretize this equation in the usual way to get +

    We can discretize this equation in the usual way to get

    $$ m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}). $$ -

    -Rearranging this equation, we can rewrite this as +

    Rearranging this equation, we can rewrite this as

    $$ \Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t. $$ -

    -









    +









    Momentum parameter

    -

    -Notice that this equation is identical to previous one if we identify +

    Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as: +

    $$ \gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}. $$ -

    -Thus, as the name suggests, the momentum parameter is proportional to +

    Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \). +

    -

    -Why is momentum useful? SGD momentum helps the gradient descent +

    Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in @@ -2046,18 +2249,19 @@ and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models. +

    -

    -These beneficial properties of momentum can sometimes become even more +

    These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG). +

    -

    -In the NAG algorithm, rather than calculating the gradient at the +

    In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule +

    $$ \begin{align} @@ -2067,16 +2271,12 @@ $$ \end{align} $$ -

    -One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \). +

    One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).

    -











    -

    Second moment of the gradient

    -

    -In stochastic gradient descent, with and without momentum, we still +

    In stochastic gradient descent, with and without momentum, we still have to specify a schedule for tuning the learning rates \( \eta_t \) as a function of time. As discussed in the context of Newton's method, this presents a number of dilemmas. The learning rate is @@ -2091,23 +2291,22 @@ extremely large models. Ideally, we would like to be able to adaptively change the step size to match the landscape without paying the steep computational price of calculating or approximating Hessians. +

    -

    -Recently, a number of methods have been introduced that accomplish +

    Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM. +

    -











    -

    RMS prop

    -

    -In RMS prop, in addition to keeping a running average of the first +

    In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by +

    $$ \begin{align} @@ -2118,8 +2317,7 @@ $$ \end{align} $$ -

    -where \( \beta \) controls the averaging time of the second moment and is +

    where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication @@ -2128,14 +2326,12 @@ is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions. +

    -











    -

    ADAM optimizer

    -

    -A related algorithm is the ADAM optimizer. In ADAM, we keep a running +

    A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and @@ -2147,6 +2343,7 @@ are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below) +

    $$ \begin{align} @@ -2161,26 +2358,25 @@ $$ \end{align} $$ -

    -where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and +

    where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop. +

    -

    -Like in RMSprop, the effective step size of a parameter depends on the +

    Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \boldsymbol{\mathbf{s}}_t - (\boldsymbol{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by +

    $$ \Delta \theta_{t+1}= -\eta_t { \boldsymbol{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. $$ -

    -









    +









    Practical tips

    +

    Geron's text, see chapter 11, has several interesting discussions.

    -Geron's text, see chapter 11, has several interesting discussions. - -











    -

    Automatic differentiation

    -

    -Automatic differentiation (AD), +

    Automatic differentiation (AD), also called algorithmic differentiation or computational differentiation,is a set of techniques to numerically evaluate the derivative of a function @@ -2211,38 +2403,42 @@ operations, derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program. +

    -

    -Automatic differentiation is neither: +

    Automatic differentiation is neither:

    - -Symbolic differentiation can lead to inefficient code and faces the +

    Symbolic differentiation can lead to inefficient code and faces the difficulty of converting a computer program into a single expression, while numerical differentiation can introduce round-off errors in the discretization process and cancellation +

    -

    -Python has tools for so-called automatic differentiation. +

    Python has tools for so-called automatic differentiation. Consider the following example +

    $$ f(x) = \sin\left(2\pi x + x^2\right) $$ -which has the following derivative +

    which has the following derivative

    $$ f'(x) = \cos\left(2\pi x + x^2\right)\left(2\pi + 2x\right) $$ -Using autograd we have +

    Using autograd we have

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 
 # To do elementwise differentiation:
 from autograd import elementwise_grad as egrad 
@@ -2276,23 +2472,40 @@ plt.legend()
 plt.show()
 
 print("The max absolute difference is: %g"%(np.max(np.abs(computed - analytic))))
-
    
-
    
-
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Using autograd

    -

    -Here we +

    Here we experiment with what kind of functions Autograd is capable of finding the gradient of. The following Python functions are just meant to illustrate what Autograd can do, but please feel free to experiment with other, possibly more complicated, functions as well. +

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 
 def f1(x):
@@ -2309,21 +2522,38 @@ a = 1.0
 # Compare with the analytical derivative, that is f1'(x) = 3*x**2 
 grad_analytical = 3*a**2
 print("The gradient of f1 evaluated at a = %g by finding the analytic expression is: %g"%(a,grad_analytical))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Autograd with more complicated functions

    -

    -To differentiate with respect to two (or more) arguments of a Python +

    To differentiate with respect to two (or more) arguments of a Python function, Autograd need to know at which variable the function if being differentiated with respect to. +

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f2(x1,x2):
     return 3*x1**3 + x2*(x1 - 5) + 1
@@ -2356,19 +2586,34 @@ f2_grad_x2_analytical = x1 print("The derivative of f2 w.r.t x2: %g"%( f2_grad_x2(x1,x2) ))
 print("The analytical derivative of f2 w.r.t x2: %g"%( f2_grad_x2(x1,x2) ))
-
    
-
    
-Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.

    -











    -

    More complicated functions using the elements of their arguments directly

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f3(x): # Assumes x is an array of length 5 or higher
     return 2*x[0] + 3*x[1] + 5*x[2] + 7*x[3] + 11*x[4]**2
@@ -2385,24 +2630,40 @@ f3_grad_analytical = np# Print the analytical gradient:
 print("The analytical gradient of f3 is: ", f3_grad_analytical)
-
    
-
    
-Note that in this case, when sending an array as input argument, the
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Note that in this case, when sending an array as input argument, the output from Autograd is another array. This is the true gradient of the function, as opposed to the function in the previous example. By using arrays to represent the variables, the output from Autograd might be easier to work with, as the output is closer to what one could expect form a gradient-evaluting function. +

    -

    -

    Functions using mathematical functions from Numpy

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f4(x):
     return np.sqrt(1+x**2) + np.exp(x) + np.sin(2*np.pi*x)
@@ -2419,16 +2680,33 @@ f4_grad_analytical = x# Print the analytical gradient:
 print("The analytical gradient of f4 at x = %g is: %g"%(x,f4_grad_analytical))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    More autograd

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f5(x):
     if x >= 0:
@@ -2442,16 +2720,33 @@ x = 2.7
 
 # Print the computed derivative:
 print("The computed derivative of f5 at x = %g is: %g"%(x,f5_grad(x)))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    And with loops

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f6_for(x):
     val = 0
@@ -2475,11 +2770,29 @@ x = 0.5
 # Print the computed derivaties of f6_for and f6_while
 print("The computed derivative of f6_for at x = %g is: %g"%(x,f6_for_grad(x)))
 print("The computed derivative of f6_while at x = %g is: %g"%(x,f6_while_grad(x)))
-
    
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + -
    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 # Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9
 # The analytical derivative is: sum(i*x**(i-1)) 
@@ -2488,15 +2801,32 @@ f6_grad_analytical = += i*x**(i-1)
 
 print("The analytical derivative of f6 at x = %g is: %g"%(x,f6_grad_analytical))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Using recursion

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 
 def f7(n): # Assume that n is an integer
@@ -2523,22 +2853,36 @@ f7_grad_analytical = += tmp
 
 print("The analytical derivative of f7 at n = %d is: %g"%(n,f7_grad_analytical))
-
    
-
    
-Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.

    -











    -

    Unsupported functions

    -Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd. +

    Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.

    -

    -Assigning a value to the variable being differentiated with respect to -

    +

    Assigning a value to the variable being differentiated with respect to

    -
    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f8(x): # Assume x is an array
     x[2] = 3
@@ -2549,18 +2893,33 @@ f8_grad = grad(f8)
 x = 8.4
 
 print("The derivative of f8 is:",f8_grad(x))
-
    
-
    
-Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.

    -











    -

    The syntax a.dot(b) when finding the dot product

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f9(a): # Assume a is an array with 2 elements
     b = np.array([1.0,2.0])
@@ -2571,16 +2930,34 @@ f9_grad = grad(f9)
 x = np.array([1.0,0.0])
 
 print("The derivative of f9 is:",f9_grad(x))
-
    
-
    
-Here we are told that the 'dot' function does not belong to Autograd's
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Here we are told that the 'dot' function does not belong to Autograd's version of a Numpy array. To overcome this, an alternative syntax which also computed the dot product can be used: +

    -

    -

    import autograd.numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import autograd.numpy as np
 from autograd import grad
 def f9_alternative(x): # Assume a is an array with 2 elements
     b = np.array([1.0,2.0])
@@ -2594,31 +2971,56 @@ x = np.a
 
 # The analytical gradient of the dot product of vectors x and b with two elements (x_1,x_2) and (b_1, b_2) respectively
 # w.r.t x is (b_1, b_2).
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    -The documentation recommends to avoid inplace operations such as -

    +

    The documentation recommends to avoid inplace operations such as

    -
    a += b
+
    
+  
    
+    
    
+      
    
+        
    
+  
    a += b
 a -= b
 a*= b
 a /=b
-
    
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + - -
    © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz index b66734120..0b3705d4b 100644 Binary files a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz and b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz differ diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index 6a3d62427..d28a86b34 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -2,20 +2,38 @@ "cells": [ { "cell_type": "markdown", - "metadata": {}, + "id": "6263ac2c", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "814f46e8", + "metadata": { + "editable": true + }, "source": [ - "\n", "# Week 39: Optimization and Gradient Methods\n", - "\n", - " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Oct 1, 2021**\n", - "\n", - "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", - "\n", - "\n", + "Date: **Nov 2, 2021**\n", "\n", + "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license" + ] + }, + { + "cell_type": "markdown", + "id": "cd34f9be", + "metadata": { + "editable": true + }, + "source": [ "## Plan for week 39\n", "\n", "* Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Discussion of project 1 and examples on how to implement Logistic Regression\n", @@ -32,13 +50,28 @@ "For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.\n", "\n", "**For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.55 and 5.7-5.11**.\n", - "These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. \n", - "\n", + "These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning." + ] + }, + { + "cell_type": "markdown", + "id": "68f08321", + "metadata": { + "editable": true + }, + "source": [ "## Thursday September 30\n", "\n", - "[Overview Video, why do we care about gradient methods?](https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverarchingAimsWeek39.mp4?vrtx=view-as-webpage)\n", - "\n", - "\n", + "[Overview Video, why do we care about gradient methods?](https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h20/forelesningsvideoer/OverarchingAimsWeek39.mp4?vrtx=view-as-webpage)" + ] + }, + { + "cell_type": "markdown", + "id": "618b34d4", + "metadata": { + "editable": true + }, + "source": [ "## Searching for Optimal Regularization Parameters $\\lambda$\n", "\n", "In project 1, when using Ridge and Lasso regression, we end up\n", @@ -53,26 +86,12 @@ { "cell_type": "code", "execution_count": 1, - "metadata": {}, - "outputs": [ - { - "ename": "ModuleNotFoundError", - "evalue": "No module named 'matplotlib'", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", - "\u001b[0;32m/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_7618/3906882771.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_line_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'matplotlib'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'inline'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpandas\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpyplot\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/opt/homebrew/lib/python3.9/site-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_line_magic\u001b[0;34m(self, magic_name, line, _stack_depth)\u001b[0m\n\u001b[1;32m 2349\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'local_ns'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_local_scope\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstack_depth\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2350\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2351\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2352\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2353\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/opt/homebrew/lib/python3.9/site-packages/decorator.py\u001b[0m in \u001b[0;36mfun\u001b[0;34m(*args, **kw)\u001b[0m\n\u001b[1;32m 230\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mkwsyntax\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 231\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkw\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 232\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mcaller\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mextras\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 233\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 234\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__doc__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/opt/homebrew/lib/python3.9/site-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m 185\u001b[0m \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 186\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 187\u001b[0;31m \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 188\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 189\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/opt/homebrew/lib/python3.9/site-packages/IPython/core/magics/pylab.py\u001b[0m in \u001b[0;36mmatplotlib\u001b[0;34m(self, line)\u001b[0m\n\u001b[1;32m 97\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Available matplotlib backends: %s\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mbackends_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 98\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 99\u001b[0;31m \u001b[0mgui\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbackend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshell\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menable_matplotlib\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgui\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgui\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgui\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 100\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_show_matplotlib_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgui\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbackend\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/opt/homebrew/lib/python3.9/site-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36menable_matplotlib\u001b[0;34m(self, gui)\u001b[0m\n\u001b[1;32m 3518\u001b[0m \"\"\"\n\u001b[1;32m 3519\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mIPython\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcore\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpylabtools\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mpt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3520\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mmatplotlib_inline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackend_inline\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mconfigure_inline_support\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3521\u001b[0m \u001b[0mgui\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbackend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind_gui_and_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgui\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpylab_gui_select\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3522\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/opt/homebrew/lib/python3.9/site-packages/matplotlib_inline/backend_inline.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;31m# Distributed under the terms of the BSD 3-Clause License.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 7\u001b[0m from matplotlib.backends.backend_agg import ( # noqa\n\u001b[1;32m 8\u001b[0m \u001b[0mnew_figure_manager\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'matplotlib'" - ] - } - ], + "id": "ff12897b", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "%matplotlib inline\n", "\n", @@ -124,16 +143,25 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "75fdf948", + "metadata": { + "editable": true + }, "source": [ "Here we have performed a rather data greedy calculation as function of the regularization parameter $\\lambda$. There is no resampling here. The latter can easily be added by employing the function **RidgeCV** instead of just calling the **Ridge** function. For **RidgeCV** we need to pass the array of $\\lambda$ values.\n", "By inspecting the figure we can in turn determine which is the optimal regularization parameter.\n", - "This becomes however less functional in the long run. \n", - "\n", - "\n", + "This becomes however less functional in the long run." + ] + }, + { + "cell_type": "markdown", + "id": "21b4ddd2", + "metadata": { + "editable": true + }, + "source": [ "## Grid Search\n", "\n", - "\n", "An alternative is to use the so-called grid search functionality\n", "included with the library **Scikit-Learn**, as demonstrated for the same\n", "example here." @@ -141,8 +169,12 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 2, + "id": "0be364ae", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -191,16 +223,26 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b1705a00", + "metadata": { + "editable": true + }, "source": [ "By default the grid search function includes cross validation with\n", "five folds. The [Scikit-Learn\n", "documentation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV)\n", "contains more information on how to set the different parameters.\n", "\n", - "If we take out the random noise, running the above codes results in $\\lambda=0$ yielding the best fit. \n", - "\n", - "\n", + "If we take out the random noise, running the above codes results in $\\lambda=0$ yielding the best fit." + ] + }, + { + "cell_type": "markdown", + "id": "031b93e6", + "metadata": { + "editable": true + }, + "source": [ "## Randomized Grid Search\n", "\n", "An alternative to the above manual grid set up, is to use a random\n", @@ -215,8 +257,12 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 3, + "id": "ffe995f0", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -266,7 +312,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "fc03fd95", + "metadata": { + "editable": true + }, "source": [ "## Optimization, the central part of any Machine Learning algortithm\n", "\n", @@ -277,9 +326,16 @@ "$X$. The model is fit by finding the values of $\\beta$ that minimize\n", "the cost function. Ideally we would be able to solve for $\\beta$\n", "analytically, however this is not possible in general and we must use\n", - "some approximative/numerical method to compute the minimum.\n", - "\n", - "\n", + "some approximative/numerical method to compute the minimum." + ] + }, + { + "cell_type": "markdown", + "id": "2718f706", + "metadata": { + "editable": true + }, + "source": [ "## Revisiting our Logistic Regression case\n", "\n", "In our discussion on Logistic Regression we studied the \n", @@ -292,7 +348,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3ad8bf69", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -304,10 +363,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b86907b5", + "metadata": { + "editable": true + }, + "source": [ + "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$." + ] + }, + { + "cell_type": "markdown", + "id": "c03cb57f", + "metadata": { + "editable": true + }, "source": [ - "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n", - "\n", "## The equations to solve\n", "\n", "Our compact equations used a definition of a vector $\\boldsymbol{y}$ with $n$\n", @@ -319,7 +389,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "27c1a970", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -328,7 +401,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "954ac945", + "metadata": { + "editable": true + }, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -336,7 +412,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0463bd79", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -345,14 +424,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7905b925", + "metadata": { + "editable": true + }, + "source": [ + "This defines what is called the Hessian matrix." + ] + }, + { + "cell_type": "markdown", + "id": "25d628e3", + "metadata": { + "editable": true + }, "source": [ - "This defines what is called the Hessian matrix.\n", - "\n", - "\n", - "\n", - "\n", - "\n", "## Solving using Newton-Raphson's method\n", "\n", "If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives. \n", @@ -362,7 +448,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "196e94f7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", @@ -371,14 +460,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "31e06821", + "metadata": { + "editable": true + }, "source": [ "or in matrix form as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "49cf7f77", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", @@ -387,13 +482,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "091103af", + "metadata": { + "editable": true + }, "source": [ "The right-hand side is computed with the old values of $\\beta$. \n", "\n", - "If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement. \n", - "\n", - "\n", + "If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement." + ] + }, + { + "cell_type": "markdown", + "id": "1e13788e", + "metadata": { + "editable": true + }, + "source": [ "## Brief reminder on Newton-Raphson's method\n", "\n", "Let us quickly remind ourselves how we derive the above method.\n", @@ -404,8 +509,16 @@ "function $f$ and its derivative $f'$ at arbitrary points. \n", "If you can only calculate the derivative\n", "numerically and/or your function is not of the smooth type, we\n", - "normally discourage the use of this method.\n", - "\n", + "normally discourage the use of this method." + ] + }, + { + "cell_type": "markdown", + "id": "81b72086", + "metadata": { + "editable": true + }, + "source": [ "## The equations\n", "\n", "The Newton-Raphson formula consists geometrically of extending the\n", @@ -417,7 +530,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "154ff7ec", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -430,7 +546,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2dc77c1c", + "metadata": { + "editable": true + }, "source": [ "For small enough values of the function and for well-behaved\n", "functions, the terms beyond linear are unimportant, hence we obtain" @@ -438,7 +557,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "436d5e10", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(x)+(s-x)f'(x)\\approx 0,\n", @@ -447,14 +569,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "eded5059", + "metadata": { + "editable": true + }, "source": [ "yielding" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "0c8f72c4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "s\\approx x-\\frac{f(x)}{f'(x)}.\n", @@ -463,14 +591,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b43097ee", + "metadata": { + "editable": true + }, "source": [ "Having in mind an iterative procedure, it is natural to start iterating with" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "78cacabf", + "metadata": { + "editable": true + }, "source": [ "$$\n", "x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n", @@ -479,7 +613,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "4f490616", + "metadata": { + "editable": true + }, "source": [ "## Simple geometric interpretation\n", "\n", @@ -493,9 +630,16 @@ "from the true root as to let the search interval include a local\n", "maximum or minimum of the function. If an iteration places a trial\n", "guess near such a local extremum, so that the first derivative nearly\n", - "vanishes, then Newton-Raphson may fail totally\n", - "\n", - "\n", + "vanishes, then Newton-Raphson may fail totally" + ] + }, + { + "cell_type": "markdown", + "id": "1e808c2b", + "metadata": { + "editable": true + }, + "source": [ "## Extending to more than one variable\n", "\n", "Newton's method can be generalized to systems of several non-linear equations\n", @@ -504,7 +648,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1e13adc6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n", @@ -514,14 +661,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "df5ed837", + "metadata": { + "editable": true + }, "source": [ "which we Taylor expand to obtain" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "b1e83c72", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n", @@ -536,14 +689,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d913e2b4", + "metadata": { + "editable": true + }, "source": [ "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "c6a3a141", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", @@ -555,14 +714,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f80cc614", + "metadata": { + "editable": true + }, "source": [ "we can rephrase Newton's method as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "65dc9a58", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n", @@ -573,14 +738,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "339350ce", + "metadata": { + "editable": true + }, "source": [ "where we have defined" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "b05423ff", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", @@ -591,17 +762,26 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "bc4379f3", + "metadata": { + "editable": true + }, "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", "arise in case ${\\bf \\boldsymbol{J}}$ is nearly singular.\n", "\n", "It is rather straightforward to extend the above scheme to systems of\n", - "more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function. \n", - "\n", - "\n", - "\n", + "more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function." + ] + }, + { + "cell_type": "markdown", + "id": "7ed3a930", + "metadata": { + "editable": true + }, + "source": [ "## Steepest descent\n", "\n", "The basic idea of gradient descent is\n", @@ -614,7 +794,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a817ecc9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n", @@ -623,15 +806,25 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "73e4664e", + "metadata": { + "editable": true + }, "source": [ "with $\\gamma_k > 0$.\n", "\n", "For $\\gamma_k$ small enough, then $F(\\mathbf{x}_{k+1}) \\leq\n", "F(\\mathbf{x}_k)$. This means that for a sufficiently small $\\gamma_k$\n", - "we are always moving towards smaller function values, i.e a minimum.\n", - "\n", - "\n", + "we are always moving towards smaller function values, i.e a minimum." + ] + }, + { + "cell_type": "markdown", + "id": "42b23010", + "metadata": { + "editable": true + }, + "source": [ "## More on Steepest descent\n", "\n", "The previous observation is the basis of the method of steepest\n", @@ -642,7 +835,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b2ce0e1e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n", @@ -651,12 +847,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8d9c258d", + "metadata": { + "editable": true + }, "source": [ "The parameter $\\gamma_k$ is often referred to as the step length or\n", - "the learning rate within the context of Machine Learning.\n", - "\n", - "\n", + "the learning rate within the context of Machine Learning." + ] + }, + { + "cell_type": "markdown", + "id": "75b014b2", + "metadata": { + "editable": true + }, + "source": [ "## The ideal\n", "\n", "Ideally the sequence $\\{\\mathbf{x}_k \\}_{k=0}$ converges to a global\n", @@ -675,10 +881,16 @@ "sensitive to the chosen initial condition.\n", "\n", "Note that the gradient is a function of $\\mathbf{x} =\n", - "(x_1,\\cdots,x_n)$ which makes it expensive to compute numerically.\n", - "\n", - "\n", - "\n", + "(x_1,\\cdots,x_n)$ which makes it expensive to compute numerically." + ] + }, + { + "cell_type": "markdown", + "id": "2c3e43f6", + "metadata": { + "editable": true + }, + "source": [ "## The sensitiveness of the gradient descent\n", "\n", "The gradient descent method \n", @@ -691,10 +903,16 @@ "\n", "Many of these shortcomings can be alleviated by introducing\n", "randomness. One such method is that of Stochastic Gradient Descent\n", - "(SGD), see below.\n", - "\n", - "\n", - "\n", + "(SGD), see below." + ] + }, + { + "cell_type": "markdown", + "id": "fc987448", + "metadata": { + "editable": true + }, + "source": [ "## Convex functions\n", "\n", "Ideally we want our cost/loss function to be convex(concave).\n", @@ -707,12 +925,28 @@ "\n", "The convex subsets of $\\mathbb{R}$ are the intervals of\n", "$\\mathbb{R}$. Examples of convex sets of $\\mathbb{R}^2$ are the\n", - "regular polygons (triangles, rectangles, pentagons, etc...).\n", - "\n", + "regular polygons (triangles, rectangles, pentagons, etc...)." + ] + }, + { + "cell_type": "markdown", + "id": "2ec7230c", + "metadata": { + "editable": true + }, + "source": [ "## Convex function\n", "\n", - "**Convex function**: Let $X \\subset \\mathbb{R}^n$ be a convex set. Assume that the function $f: X \\rightarrow \\mathbb{R}$ is continuous, then $f$ is said to be convex if $$f(tx_1 + (1-t)x_2) \\leq tf(x_1) + (1-t)f(x_2) $$ for all $x_1, x_2 \\in X$ and for all $t \\in [0,1]$. If $\\leq$ is replaced with a strict inequaltiy in the definition, we demand $x_1 \\neq x_2$ and $t\\in(0,1)$ then $f$ is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the value of the function on the interval $[x_1,x_2]$ is always below the line as illustrated below.\n", - "\n", + "**Convex function**: Let $X \\subset \\mathbb{R}^n$ be a convex set. Assume that the function $f: X \\rightarrow \\mathbb{R}$ is continuous, then $f$ is said to be convex if $$f(tx_1 + (1-t)x_2) \\leq tf(x_1) + (1-t)f(x_2) $$ for all $x_1, x_2 \\in X$ and for all $t \\in [0,1]$. If $\\leq$ is replaced with a strict inequaltiy in the definition, we demand $x_1 \\neq x_2$ and $t\\in(0,1)$ then $f$ is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the value of the function on the interval $[x_1,x_2]$ is always below the line as illustrated below." + ] + }, + { + "cell_type": "markdown", + "id": "2f329d26", + "metadata": { + "editable": true + }, + "source": [ "## Conditions on convex functions\n", "\n", "In the following we state first and second-order conditions which\n", @@ -731,8 +965,6 @@ "make a drawing of $f(x) = x^2+1$ and draw the tangent line to $f(x)$ and\n", "note that it is always below the graph.\n", "\n", - "\n", - "\n", "**Second order condition.**\n", "\n", "Assume that $f$ is twice\n", @@ -742,10 +974,16 @@ "single-variable function this reduces to $f''(x) \\geq 0$. Geometrically this means that $f$ has nonnegative curvature\n", "everywhere.\n", "\n", - "\n", - "\n", - "This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.\n", - "\n", + "This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition." + ] + }, + { + "cell_type": "markdown", + "id": "b5d52917", + "metadata": { + "editable": true + }, + "source": [ "## More on convex functions\n", "\n", "The next result is of great importance to us and the reason why we are\n", @@ -764,10 +1002,16 @@ "is minimal, where $f$ is convex and differentiable. Then, any point\n", "$x^*$ that satisfies $\\nabla f(x^*) = 0$ is a global minimum.\n", "\n", - "\n", - "\n", - "This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.\n", - "\n", + "This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum." + ] + }, + { + "cell_type": "markdown", + "id": "e504719e", + "metadata": { + "editable": true + }, + "source": [ "## Some simple problems\n", "\n", "1. Show that $f(x)=x^2$ is convex for $x \\in \\mathbb{R}$ using the definition of convexity. Hint: If you re-write the definition, $f$ is convex if the following holds for all $x,y \\in D_f$ and any $\\lambda \\in [0,1]$ $\\lambda f(x)+(1-\\lambda)f(y)-f(\\lambda x + (1-\\lambda) y ) \\geq 0$.\n", @@ -778,7 +1022,6 @@ "\n", " * $g(x) = -\\ln(x)$ is convex for $x \\in (0,\\infty)$.\n", "\n", - "\n", "3. Let $f(x) = x^2$ and $g(x) = e^x$. Show that $f(g(x))$ and $g(f(x))$ is convex for $x \\in \\mathbb{R}$. Also show that if $f(x)$ is any convex function than $h(x) = e^{f(x)}$ is convex.\n", "\n", "4. A norm is any function that satisfy the following properties\n", @@ -789,15 +1032,18 @@ "\n", " * $f(x) \\leq 0$ for all $x \\in \\mathbb{R}^n$ with equality if and only if $x = 0$\n", "\n", - "\n", - "Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).\n", - "\n", - "\n", - "\n", - "\n", + "Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this)." + ] + }, + { + "cell_type": "markdown", + "id": "a1649131", + "metadata": { + "editable": true + }, + "source": [ "## Standard steepest descent\n", "\n", - "\n", "Before we proceed, we would like to discuss the approach called the\n", "**standard Steepest descent** (different from the above steepest descent discussion), which again leads to us having to be able\n", "to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG).\n", @@ -811,7 +1057,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "4ef42fd6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}.\n", @@ -820,14 +1069,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "159fe760", + "metadata": { + "editable": true + }, "source": [ "In the iterative process we end up with a problem like" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "29ebbb6f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}= \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x},\n", @@ -836,12 +1091,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5f4a35d4", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{r}$ is the so-called residual or error in the iterative process.\n", "\n", - "When we have found the exact solution, $\\boldsymbol{r}=0$.\n", - "\n", + "When we have found the exact solution, $\\boldsymbol{r}=0$." + ] + }, + { + "cell_type": "markdown", + "id": "90f63cc8", + "metadata": { + "editable": true + }, + "source": [ "## Gradient method\n", "\n", "The residual is zero when we reach the minimum of the quadratic equation" @@ -849,7 +1115,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b7bd384e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "P(\\boldsymbol{x})=\\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T\\boldsymbol{b},\n", @@ -858,12 +1127,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8a95c1a9", + "metadata": { + "editable": true + }, "source": [ "with the constraint that the matrix $\\boldsymbol{A}$ is positive definite and\n", - "symmetric. This defines also the Hessian and we want it to be positive definite. \n", - "\n", - "\n", + "symmetric. This defines also the Hessian and we want it to be positive definite." + ] + }, + { + "cell_type": "markdown", + "id": "30180639", + "metadata": { + "editable": true + }, + "source": [ "## Steepest descent method\n", "\n", "We denote the initial guess for $\\boldsymbol{x}$ as $\\boldsymbol{x}_0$. \n", @@ -872,7 +1151,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "09b1c9f1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -881,14 +1163,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6fcbfec6", + "metadata": { + "editable": true + }, "source": [ "or consider the system" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "d00fcff7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -897,18 +1185,31 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7ff48637", + "metadata": { + "editable": true + }, + "source": [ + "instead." + ] + }, + { + "cell_type": "markdown", + "id": "023bb623", + "metadata": { + "editable": true + }, "source": [ - "instead.\n", - "\n", - "\n", "## Steepest descent method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "d4cd6b69", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -917,7 +1218,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "438c0b5a", + "metadata": { + "editable": true + }, "source": [ "This suggests taking the first basis vector $\\boldsymbol{r}_1$ (see below for definition) \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -926,7 +1230,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "764cde40", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -935,20 +1242,32 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "4dd2012f", + "metadata": { + "editable": true + }, "source": [ "and \n", - "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n", - "\n", - "\n", - "\n", + "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$." + ] + }, + { + "cell_type": "markdown", + "id": "e5a711d2", + "metadata": { + "editable": true + }, + "source": [ "## Final expressions\n", "We can compute the residual iteratively as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "e7eec8f6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -957,14 +1276,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ac4bc943", + "metadata": { + "editable": true + }, "source": [ "which equals" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "fcf6d10d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_k),\n", @@ -973,14 +1298,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "07d74d47", + "metadata": { + "editable": true + }, "source": [ "or" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "10181a0a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{r}_k,\n", @@ -989,14 +1320,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "30ddb647", + "metadata": { + "editable": true + }, "source": [ "which gives" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "c40c0ed3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{r}_k^T\\boldsymbol{r}_k}{\\boldsymbol{r}_k^T\\boldsymbol{A}\\boldsymbol{r}_k}\n", @@ -1005,14 +1342,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "60898dba", + "metadata": { + "editable": true + }, "source": [ "leading to the iterative scheme" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "eee23b5f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_{k+1}=\\boldsymbol{x}_k-\\alpha_k\\boldsymbol{r}_{k},\n", @@ -1021,15 +1364,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "50fdc629", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent example" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 4, + "id": "defa6ef0", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -1056,15 +1406,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "669108ca", + "metadata": { + "editable": true + }, "source": [ "And then as countor plot" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 5, + "id": "206f3608", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1074,15 +1431,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "137b9e45", + "metadata": { + "editable": true + }, "source": [ "Find guesses" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 6, + "id": "d2410c8a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = guesses[-1]\n", @@ -1091,15 +1455,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "de7aa713", + "metadata": { + "editable": true + }, "source": [ "Run it!" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 7, + "id": "16efee06", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def f1d(alpha):\n", @@ -1113,15 +1484,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c8f6e2bd", + "metadata": { + "editable": true + }, "source": [ "What happened?" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 8, + "id": "8540f54d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1132,10 +1510,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6f483a63", + "metadata": { + "editable": true + }, + "source": [ + "Note that we did only one iteration here. We can easily add more using our previous guesses." + ] + }, + { + "cell_type": "markdown", + "id": "2367ba74", + "metadata": { + "editable": true + }, "source": [ - "Note that we did only one iteration here. We can easily add more using our previous guesses.\n", - "\n", "## Conjugate gradient method\n", "In the CG method we define so-called conjugate directions and two vectors \n", "$\\boldsymbol{s}$ and $\\boldsymbol{t}$\n", @@ -1145,7 +1534,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "20788d85", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{s}^T\\boldsymbol{A}\\boldsymbol{t}= 0.\n", @@ -1154,7 +1546,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ea535aea", + "metadata": { + "editable": true + }, "source": [ "The philosophy of the CG method is to perform searches in various conjugate directions\n", "of our vectors $\\boldsymbol{x}_i$ obeying the above criterion, namely" @@ -1162,7 +1557,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "165adcbc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_i^T\\boldsymbol{A}\\boldsymbol{x}_j= 0.\n", @@ -1171,20 +1569,32 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f7aaa007", + "metadata": { + "editable": true + }, "source": [ "Two vectors are conjugate if they are orthogonal with respect to \n", - "this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$.\n", - "\n", - "\n", - "\n", + "this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$." + ] + }, + { + "cell_type": "markdown", + "id": "23efba9c", + "metadata": { + "editable": true + }, + "source": [ "## Conjugate gradient method\n", "An example is given by the eigenvectors of the matrix" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "d5439727", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{v}_i^T\\boldsymbol{A}\\boldsymbol{v}_j= \\lambda\\boldsymbol{v}_i^T\\boldsymbol{v}_j,\n", @@ -1193,13 +1603,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "631c623f", + "metadata": { + "editable": true + }, + "source": [ + "which is zero unless $i=j$." + ] + }, + { + "cell_type": "markdown", + "id": "4580be44", + "metadata": { + "editable": true + }, "source": [ - "which is zero unless $i=j$.\n", - "\n", - "\n", - "\n", - "\n", "## Conjugate gradient method\n", "Assume now that we have a symmetric positive-definite matrix $\\boldsymbol{A}$ of size\n", "$n\\times n$. At each iteration $i+1$ we obtain the conjugate direction of a vector" @@ -1207,7 +1625,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3fc0d9f5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_{i+1}=\\boldsymbol{x}_{i}+\\alpha_i\\boldsymbol{p}_{i}.\n", @@ -1216,7 +1637,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "4b0f7db6", + "metadata": { + "editable": true + }, "source": [ "We assume that $\\boldsymbol{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n", "Then the $\\boldsymbol{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n", @@ -1225,7 +1649,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6062951a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i \\boldsymbol{p}_i.\n", @@ -1234,7 +1661,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "fd0a2f6e", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "The coefficients are given by" @@ -1242,7 +1672,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b83ad4fb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n", @@ -1251,14 +1684,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0cd2222f", + "metadata": { + "editable": true + }, "source": [ "Multiplying with $\\boldsymbol{p}_k^T$ from the left gives" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "de54cca2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{p}_i= \\boldsymbol{p}_k^T \\boldsymbol{b},\n", @@ -1267,14 +1706,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a090dc8e", + "metadata": { + "editable": true + }, "source": [ "and we can define the coefficients $\\alpha_k$ as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "4c5bb87a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{p}_k^T \\boldsymbol{b}}{\\boldsymbol{p}_k^T \\boldsymbol{A} \\boldsymbol{p}_k}\n", @@ -1283,7 +1728,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e1dc0a5d", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method and iterations\n", "\n", @@ -1300,7 +1748,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "94780e34", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -1309,14 +1760,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7a0f6afb", + "metadata": { + "editable": true + }, "source": [ "or consider the system" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "fd14657e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -1325,20 +1782,31 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ce919c46", + "metadata": { + "editable": true + }, + "source": [ + "instead." + ] + }, + { + "cell_type": "markdown", + "id": "2af9b699", + "metadata": { + "editable": true + }, "source": [ - "instead.\n", - "\n", - "\n", - "\n", - "\n", "## Conjugate gradient method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "312f6dc3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -1347,7 +1815,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e75a06a8", + "metadata": { + "editable": true + }, "source": [ "This suggests taking the first basis vector $\\boldsymbol{p}_1$ \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -1356,7 +1827,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "12232187", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -1365,23 +1839,34 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e44907f2", + "metadata": { + "editable": true + }, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n", "The other vectors in the basis will be conjugate to the gradient, \n", - "hence the name conjugate gradient method.\n", - "\n", - "\n", - "\n", - "\n", + "hence the name conjugate gradient method." + ] + }, + { + "cell_type": "markdown", + "id": "3297c1f5", + "metadata": { + "editable": true + }, + "source": [ "## Conjugate gradient method\n", "Let $\\boldsymbol{r}_k$ be the residual at the $k$-th step:" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "abae27bc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_k=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k.\n", @@ -1390,7 +1875,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "835b302b", + "metadata": { + "editable": true + }, "source": [ "Note that $\\boldsymbol{r}_k$ is the negative gradient of $f$ at \n", "$\\boldsymbol{x}=\\boldsymbol{x}_k$, \n", @@ -1403,7 +1891,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f7a21361", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{p}_{k+1}=\\boldsymbol{r}_k-\\frac{\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{r}_k}{\\boldsymbol{p}_k^T\\boldsymbol{A}\\boldsymbol{p}_k} \\boldsymbol{p}_k.\n", @@ -1412,7 +1903,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5277ae72", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "We can also compute the residual iteratively as" @@ -1420,7 +1914,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "cd385e88", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1429,14 +1926,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "20361729", + "metadata": { + "editable": true + }, "source": [ "which equals" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "dd6b22b4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{p}_k),\n", @@ -1445,14 +1948,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "150beb1c", + "metadata": { + "editable": true + }, "source": [ "or" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "77530681", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{p}_k,\n", @@ -1461,14 +1970,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "04ba2383", + "metadata": { + "editable": true + }, "source": [ "which gives" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "ff1a8f53", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{r}_k-\\boldsymbol{A}\\boldsymbol{p}_{k},\n", @@ -1477,9 +1992,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f8596b4e", + "metadata": { + "editable": true + }, "source": [ - "\n", "## Revisiting our first homework\n", "\n", "We will use linear regression as a case study for the gradient descent\n", @@ -1498,8 +2015,12 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 9, + "id": "260fdca9", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -1508,7 +2029,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8884fffe", + "metadata": { + "editable": true + }, "source": [ "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" @@ -1516,7 +2040,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7269998b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", @@ -1525,14 +2052,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f8d3cadb", + "metadata": { + "editable": true + }, "source": [ "such that" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "3e9fb7fa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", @@ -1541,9 +2074,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d25601f3", + "metadata": { + "editable": true + }, "source": [ - "\n", "## Gradient descent example\n", "\n", "Let $\\mathbf{y} = (y_1,\\cdots,y_n)^T$, $\\mathbf{\\boldsymbol{y}} = (\\boldsymbol{y}_1,\\cdots,\\boldsymbol{y}_n)^T$ and $\\beta = (\\beta_0, \\beta_1)^T$\n", @@ -1553,7 +2088,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3a341c25", + "metadata": { + "editable": true + }, "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", @@ -1566,14 +2104,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "aa05f072", + "metadata": { + "editable": true + }, "source": [ "The cost/loss/risk function is given by (" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "d9d8e0de", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", @@ -1582,10 +2126,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1d693be2", + "metadata": { + "editable": true + }, + "source": [ + "and we want to find $\\beta$ such that $C(\\beta)$ is minimized." + ] + }, + { + "cell_type": "markdown", + "id": "2ea777b4", + "metadata": { + "editable": true + }, "source": [ - "and we want to find $\\beta$ such that $C(\\beta)$ is minimized.\n", - "\n", "## The derivative of the cost/loss function\n", "\n", "Computing $\\partial C(\\beta) / \\partial \\beta_0$ and $\\partial C(\\beta) / \\partial \\beta_1$ we can show that the gradient can be written as" @@ -1593,7 +2148,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "24834799", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -1604,17 +2162,31 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ca03d184", + "metadata": { + "editable": true + }, + "source": [ + "where $X$ is the design matrix defined above." + ] + }, + { + "cell_type": "markdown", + "id": "76b10741", + "metadata": { + "editable": true + }, "source": [ - "where $X$ is the design matrix defined above.\n", - "\n", "## The Hessian matrix\n", "The Hessian matrix of $C(\\beta)$ is given by" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "03d8886f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -1626,13 +2198,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "97b0ddb8", + "metadata": { + "editable": true + }, + "source": [ + "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite." + ] + }, + { + "cell_type": "markdown", + "id": "928eefaf", + "metadata": { + "editable": true + }, "source": [ - "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite.\n", - "\n", - "\n", - "\n", - "\n", "## Simple program\n", "\n", "We can now write a program that minimizes $C(\\beta)$ using the gradient descent method with a constant learning rate $\\gamma$ according to" @@ -1640,7 +2220,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "20ca3352", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n", @@ -1649,15 +2232,26 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c9e83371", + "metadata": { + "editable": true + }, "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", "when $||\\nabla_\\beta C(\\beta_k) || \\leq \\epsilon = 10^{-8}$. **Note that the code below does not include the latter stop criterion**.\n", "\n", "And finally we can compare our solution for $\\beta$ with the analytic result given by \n", - "$\\beta= (X^TX)^{-1} X^T \\mathbf{y}$.\n", - "\n", + "$\\beta= (X^TX)^{-1} X^T \\mathbf{y}$." + ] + }, + { + "cell_type": "markdown", + "id": "37af4676", + "metadata": { + "editable": true + }, + "source": [ "## Gradient Descent Example\n", "\n", "Here our simple example" @@ -1665,8 +2259,12 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 10, + "id": "8a3afc9b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\n", @@ -1719,15 +2317,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "fef51920", + "metadata": { + "editable": true + }, "source": [ "## And a corresponding example using **scikit-learn**" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 11, + "id": "3421b06e", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -1750,9 +2355,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9be51010", + "metadata": { + "editable": true + }, "source": [ - "\n", "## Gradient descent and Ridge\n", "\n", "We have also discussed Ridge regression where the loss function contains a regularized term given by the $L_2$ norm of $\\beta$," @@ -1760,7 +2367,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f749c999", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n", @@ -1769,49 +2379,68 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a3e198f9", + "metadata": { + "editable": true + }, "source": [ - "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we only have adjust the gradient as follows" + "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "1d6eb713", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", "\\sum_{i=1}^{100}\\left( x_i (\\beta_0+\\beta_1x_i)-y_ix_i\\right) \\\\\n", - "\\end{bmatrix} + 2\\lambda\\begin{bmatrix} \\beta_0 \\\\ \\beta_1\\end{bmatrix} = 2 (X^T(X\\beta - \\mathbf{y})+\\lambda \\beta).\n", + "\\end{bmatrix} + 2\\lambda\\begin{bmatrix} \\beta_0 \\\\ \\beta_1\\end{bmatrix} = 2 (\\frac{1}{n}X^T(X\\beta - \\mathbf{y})+\\lambda \\beta).\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "91fab79a", + "metadata": { + "editable": true + }, "source": [ "We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "9107f00b", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "\\beta_{\\text{ridge}} = \\left(X^T X + \\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", + "\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "961a44f6", + "metadata": { + "editable": true + }, "source": [ "## Program example for gradient descent with Ridge Regression" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 12, + "id": "b1f705b3", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from random import random, seed\n", @@ -1861,7 +2490,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b2c5981d", + "metadata": { + "editable": true + }, "source": [ "## Using gradient descent methods, limitations\n", "\n", @@ -1875,18 +2507,38 @@ "\n", "* **GD treats all directions in parameter space uniformly.** Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive. \n", "\n", - "* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.\n", - "\n", + "* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points." + ] + }, + { + "cell_type": "markdown", + "id": "4c5330ea", + "metadata": { + "editable": true + }, + "source": [ "## Challenge yourself\n", "\n", - "Write a code which implements gradient descent for a logistic regression example.\n", - "\n", - "\n", - "\n", - "\n", - "## Friday October 1\n", - "\n", - "\n", + "Write a code which implements gradient descent for a logistic regression example." + ] + }, + { + "cell_type": "markdown", + "id": "c15cdad1", + "metadata": { + "editable": true + }, + "source": [ + "## Friday October 1" + ] + }, + { + "cell_type": "markdown", + "id": "fe4260f1", + "metadata": { + "editable": true + }, + "source": [ "## Stochastic Gradient Descent\n", "\n", "Stochastic gradient descent (SGD) and variants thereof address some of\n", @@ -1899,7 +2551,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6880a7be", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", @@ -1909,7 +2564,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "da55002d", + "metadata": { + "editable": true + }, "source": [ "## Computation of gradients\n", "\n", @@ -1919,7 +2577,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "291d4f67", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -1929,14 +2590,25 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b5587460", + "metadata": { + "editable": true + }, "source": [ "Stochasticity/randomness is introduced by only taking the\n", "gradient on a subset of the data called minibatches. If there are $n$\n", "data points and the size of each minibatch is $M$, there will be $n/M$\n", "minibatches. We denote these minibatches by $B_k$ where\n", - "$k=1,\\cdots,n/M$.\n", - "\n", + "$k=1,\\cdots,n/M$." + ] + }, + { + "cell_type": "markdown", + "id": "347deeb7", + "metadata": { + "editable": true + }, + "source": [ "## SGD example\n", "As an example, suppose we have $10$ data points $(\\mathbf{x}_1,\\cdots, \\mathbf{x}_{10})$ \n", "and we choose to have $M=5$ minibathces,\n", @@ -1954,7 +2626,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "4ff76de7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_{\\beta}\n", @@ -1966,7 +2641,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0dabde37", + "metadata": { + "editable": true + }, "source": [ "## The gradient step\n", "\n", @@ -1975,7 +2653,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "85053899", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -1985,21 +2666,36 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c5d2743e", + "metadata": { + "editable": true + }, "source": [ "where $k$ is picked at random with equal\n", "probability from $[1,n/M]$. An iteration over the number of\n", "minibathces (n/M) is commonly referred to as an epoch. Thus it is\n", "typical to choose a number of epochs and for each epoch iterate over\n", - "the number of minibatches, as exemplified in the code below.\n", - "\n", + "the number of minibatches, as exemplified in the code below." + ] + }, + { + "cell_type": "markdown", + "id": "a3aa78b6", + "metadata": { + "editable": true + }, + "source": [ "## Simple example code" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 13, + "id": "3e876868", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np \n", @@ -2020,7 +2716,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "69f96a04", + "metadata": { + "editable": true + }, "source": [ "Taking the gradient only on a subset of the data has two important\n", "benefits. First, it introduces randomness which decreases the chance\n", @@ -2028,8 +2727,16 @@ "the size of the minibatches are small relative to the number of\n", "datapoints ($M < n$), the computation of the gradient is much\n", "cheaper since we sum over the datapoints in the $k-th$ minibatch and not\n", - "all $n$ datapoints.\n", - "\n", + "all $n$ datapoints." + ] + }, + { + "cell_type": "markdown", + "id": "ab8f6541", + "metadata": { + "editable": true + }, + "source": [ "## When do we stop?\n", "\n", "A natural question is when do we stop the search for a new minimum?\n", @@ -2041,8 +2748,16 @@ "evaluate the cost function at this point, store the result and\n", "continue the search. If the test kicks in at a later stage we can\n", "compare the values of the cost function and keep the $\\beta$ that\n", - "gave the lowest value.\n", - "\n", + "gave the lowest value." + ] + }, + { + "cell_type": "markdown", + "id": "d52641ac", + "metadata": { + "editable": true + }, + "source": [ "## Slightly different approach\n", "\n", "Another approach is to let the step length $\\gamma_j$ depend on the\n", @@ -2060,8 +2775,12 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 14, + "id": "8168bc67", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np \n", @@ -2092,15 +2811,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2c2826ce", + "metadata": { + "editable": true + }, "source": [ "## Program for stochastic gradient" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 15, + "id": "445663c0", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -2171,14 +2897,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "aca2efe7", + "metadata": { + "editable": true + }, + "source": [ + "**Challenge**: try to write a similar code for a Logistic Regression case." + ] + }, + { + "cell_type": "markdown", + "id": "c3f489f9", + "metadata": { + "editable": true + }, "source": [ - "**Challenge**: try to write a similar code for a Logistic Regression case.\n", - "\n", - "\n", - "\n", - "\n", - "\n", "## Momentum based GD\n", "\n", "The stochastic gradient descent (SGD) is almost always used with a\n", @@ -2189,7 +2922,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "36735fa8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n", @@ -2198,7 +2934,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8e0fd47e", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -2213,7 +2952,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "165c45c1", + "metadata": { + "editable": true + }, "source": [ "where we have introduced a momentum parameter $\\gamma$, with\n", "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n", @@ -2229,7 +2971,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "34e9d01d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n", @@ -2238,10 +2983,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "4fcd3fe2", + "metadata": { + "editable": true + }, + "source": [ + "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$." + ] + }, + { + "cell_type": "markdown", + "id": "eef12186", + "metadata": { + "editable": true + }, "source": [ - "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$.\n", - "\n", "## More on momentum based approaches\n", "\n", "Let us try to get more intuition from these equations. It is helpful\n", @@ -2253,7 +3009,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f1b327ae", + "metadata": { + "editable": true + }, "source": [ "$$\n", "m {d^2 \\mathbf{w} \\over dt^2} + \\mu {d \\mathbf{w} \\over dt }= -\\nabla_w E(\\mathbf{w}).\n", @@ -2262,14 +3021,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "843e5f0e", + "metadata": { + "editable": true + }, "source": [ "We can discretize this equation in the usual way to get" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "0b256db8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "m { \\mathbf{w}_{t+\\Delta t}-2 \\mathbf{w}_{t} +\\mathbf{w}_{t-\\Delta t} \\over (\\Delta t)^2}+\\mu {\\mathbf{w}_{t+\\Delta t}- \\mathbf{w}_{t} \\over \\Delta t} = -\\nabla_w E(\\mathbf{w}).\n", @@ -2278,14 +3043,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a486e7aa", + "metadata": { + "editable": true + }, "source": [ "Rearranging this equation, we can rewrite this as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "bd91d3e0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\Delta \\mathbf{w}_{t +\\Delta t}= - { (\\Delta t)^2 \\over m +\\mu \\Delta t} \\nabla_w E(\\mathbf{w})+ {m \\over m +\\mu \\Delta t} \\Delta \\mathbf{w}_t.\n", @@ -2294,7 +3065,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "fc17ad1e", + "metadata": { + "editable": true + }, "source": [ "## Momentum parameter\n", "\n", @@ -2307,7 +3081,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f4f69cbf", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\gamma= {m \\over m +\\mu \\Delta t }, \\qquad \\eta = {(\\Delta t)^2 \\over m +\\mu \\Delta t}.\n", @@ -2316,7 +3093,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "bedea29a", + "metadata": { + "editable": true + }, "source": [ "Thus, as the name suggests, the momentum parameter is proportional to\n", "the mass of the particle and effectively provides inertia.\n", @@ -2346,7 +3126,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ff6d5b2c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1}) \\nonumber\n", @@ -2355,7 +3138,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7a21b635", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -2370,14 +3156,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "092e7434", + "metadata": { + "editable": true + }, + "source": [ + "One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\\gamma$." + ] + }, + { + "cell_type": "markdown", + "id": "ba10fb4a", + "metadata": { + "editable": true + }, "source": [ - "One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\\gamma$.\n", - "\n", - "\n", "## Second moment of the gradient\n", "\n", - "\n", "In stochastic gradient descent, with and without momentum, we still\n", "have to specify a schedule for tuning the learning rates $\\eta_t$\n", "as a function of time. As discussed in the context of Newton's\n", @@ -2397,8 +3192,16 @@ "Recently, a number of methods have been introduced that accomplish\n", "this by tracking not only the gradient, but also the second moment of\n", "the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and\n", - "ADAM.\n", - "\n", + "ADAM." + ] + }, + { + "cell_type": "markdown", + "id": "bd7a13b2", + "metadata": { + "editable": true + }, + "source": [ "## RMS prop\n", "\n", "In RMS prop, in addition to keeping a running average of the first\n", @@ -2409,7 +3212,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6ace73dd", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -2424,7 +3230,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "33888962", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{s}_t =\\beta \\mathbf{s}_{t-1} +(1-\\beta)\\mathbf{g}_t^2 \\nonumber\n", @@ -2433,7 +3242,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7665c788", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\mathbf{g}_t \\over \\sqrt{\\mathbf{s}_t +\\epsilon}}, \\nonumber\n", @@ -2442,7 +3254,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "02b499b9", + "metadata": { + "editable": true + }, "source": [ "where $\\beta$ controls the averaging time of the second moment and is\n", "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n", @@ -2452,9 +3267,16 @@ "is clear from this formula that the learning rate is reduced in\n", "directions where the norm of the gradient is consistently large. This\n", "greatly speeds up the convergence by allowing us to use a larger\n", - "learning rate for flat directions.\n", - "\n", - "\n", + "learning rate for flat directions." + ] + }, + { + "cell_type": "markdown", + "id": "9ddf4fca", + "metadata": { + "editable": true + }, + "source": [ "## ADAM optimizer\n", "\n", "A related algorithm is the ADAM optimizer. In ADAM, we keep a running\n", @@ -2473,7 +3295,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e5867cac", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -2488,7 +3313,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a89d53a0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{m}_t = \\beta_1 \\mathbf{m}_{t-1} + (1-\\beta_1) \\mathbf{g}_t \\nonumber\n", @@ -2497,7 +3325,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0bdc8b50", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{s}_t =\\beta_2 \\mathbf{s}_{t-1} +(1-\\beta_2)\\mathbf{g}_t^2 \\nonumber\n", @@ -2506,7 +3337,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f9edf07c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", @@ -2515,7 +3349,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1abbdd9c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", @@ -2524,7 +3361,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7b00f144", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n", @@ -2533,7 +3373,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "90a0afbf", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -2547,7 +3390,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c4f96471", + "metadata": { + "editable": true + }, "source": [ "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n", "second moment and are typically taken to be $0.9$ and $0.99$\n", @@ -2563,7 +3409,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "26b924de", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n", @@ -2572,7 +3421,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "4d98827e", + "metadata": { + "editable": true + }, "source": [ "## Practical tips\n", "\n", @@ -2584,10 +3436,16 @@ "\n", "* **Adaptive optimization methods don't always have good generalization.** Recent studies have shown that adaptive methods such as ADAM, RMSPorp, and AdaGrad tend to have poor generalization compared to SGD or SGD with momentum, particularly in the high-dimensional limit (i.e. the number of parameters exceeds the number of data points). Although it is not clear at this stage why these methods perform so well in training deep neural networks, simpler procedures like properly-tuned SGD may work as well or better in these applications.\n", "\n", - "Geron's text, see chapter 11, has several interesting discussions.\n", - "\n", - "\n", - "\n", + "Geron's text, see chapter 11, has several interesting discussions." + ] + }, + { + "cell_type": "markdown", + "id": "7aff0f96", + "metadata": { + "editable": true + }, + "source": [ "## Automatic differentiation\n", "\n", "[Automatic differentiation (AD)](https://en.wikipedia.org/wiki/Automatic_differentiation), \n", @@ -2615,15 +3473,16 @@ "while numerical differentiation can introduce round-off errors in the\n", "discretization process and cancellation\n", "\n", - "\n", - "\n", "Python has tools for so-called **automatic differentiation**.\n", "Consider the following example" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "8b61c524", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(x) = \\sin\\left(2\\pi x + x^2\\right)\n", @@ -2632,14 +3491,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a605388f", + "metadata": { + "editable": true + }, "source": [ "which has the following derivative" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "5116fd5f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f'(x) = \\cos\\left(2\\pi x + x^2\\right)\\left(2\\pi + 2x\\right)\n", @@ -2648,15 +3513,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b0b06320", + "metadata": { + "editable": true + }, "source": [ "Using **autograd** we have" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 16, + "id": "972ed554", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2697,9 +3569,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ff530d43", + "metadata": { + "editable": true + }, "source": [ - "\n", "## Using autograd\n", "\n", "Here we\n", @@ -2711,8 +3585,12 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 17, + "id": "f9c4b9f9", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2736,7 +3614,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5bb16647", + "metadata": { + "editable": true + }, "source": [ "## Autograd with more complicated functions\n", "\n", @@ -2747,8 +3628,12 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 18, + "id": "f5c25a9a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2788,18 +3673,32 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8691e0f5", + "metadata": { + "editable": true + }, + "source": [ + "Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable." + ] + }, + { + "cell_type": "markdown", + "id": "f5c1a799", + "metadata": { + "editable": true + }, "source": [ - "Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.\n", - "\n", - "\n", "## More complicated functions using the elements of their arguments directly" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 19, + "id": "7a630992", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2823,23 +3722,37 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5b1de1ef", + "metadata": { + "editable": true + }, "source": [ "Note that in this case, when sending an array as input argument, the\n", "output from Autograd is another array. This is the true gradient of\n", "the function, as opposed to the function in the previous example. By\n", "using arrays to represent the variables, the output from Autograd\n", "might be easier to work with, as the output is closer to what one\n", - "could expect form a gradient-evaluting function.\n", - "\n", - "\n", + "could expect form a gradient-evaluting function." + ] + }, + { + "cell_type": "markdown", + "id": "1d6f8533", + "metadata": { + "editable": true + }, + "source": [ "## Functions using mathematical functions from Numpy" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 20, + "id": "6732b4e4", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2863,15 +3776,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "903e1d95", + "metadata": { + "editable": true + }, "source": [ "## More autograd" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 21, + "id": "14f83c2d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2892,46 +3812,58 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8a55e3b3", + "metadata": { + "editable": true + }, "source": [ "## And with loops" ] }, { - "cell_type": "markdown", - "metadata": {}, + "cell_type": "code", + "execution_count": 22, + "id": "c894ef86", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ - "2\n", - "1\n", - " \n", - "<\n", - "<\n", - "<\n", - "!\n", - "!\n", - "C\n", - "O\n", - "D\n", - "E\n", - "_\n", - "B\n", - "L\n", - "O\n", - "C\n", - "K\n", - " \n", - " \n", - "p\n", - "y\n", - "c\n", - "o\n", - "d" + "import autograd.numpy as np\n", + "from autograd import grad\n", + "def f6_for(x):\n", + " val = 0\n", + " for i in range(10):\n", + " val = val + x**i\n", + " return val\n", + "\n", + "def f6_while(x):\n", + " val = 0\n", + " i = 0\n", + " while i < 10:\n", + " val = val + x**i\n", + " i = i + 1\n", + " return val\n", + "\n", + "f6_for_grad = grad(f6_for)\n", + "f6_while_grad = grad(f6_while)\n", + "\n", + "x = 0.5\n", + "\n", + "# Print the computed derivaties of f6_for and f6_while\n", + "print(\"The computed derivative of f6_for at x = %g is: %g\"%(x,f6_for_grad(x)))\n", + "print(\"The computed derivative of f6_while at x = %g is: %g\"%(x,f6_while_grad(x)))" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 23, + "id": "83b8201a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2947,15 +3879,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0f7fe07d", + "metadata": { + "editable": true + }, "source": [ "## Using recursion" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 24, + "id": "b1f755f4", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2989,10 +3928,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0a1df639", + "metadata": { + "editable": true + }, + "source": [ + "Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input." + ] + }, + { + "cell_type": "markdown", + "id": "22ea217a", + "metadata": { + "editable": true + }, "source": [ - "Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.\n", - "\n", "## Unsupported functions\n", "Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.\n", "\n", @@ -3001,8 +3951,12 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 25, + "id": "0dd3c4cb", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3020,17 +3974,32 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "030639cd", + "metadata": { + "editable": true + }, + "source": [ + "Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible." + ] + }, + { + "cell_type": "markdown", + "id": "90cc352d", + "metadata": { + "editable": true + }, "source": [ - "Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.\n", - "\n", "## The syntax a.dot(b) when finding the dot product" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 26, + "id": "40ed6032", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3048,7 +4017,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "67e6173f", + "metadata": { + "editable": true + }, "source": [ "Here we are told that the 'dot' function does not belong to Autograd's\n", "version of a Numpy array. To overcome this, an alternative syntax\n", @@ -3057,8 +4029,12 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 27, + "id": "cbfc002f", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3079,7 +4055,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "dd550ee3", + "metadata": { + "editable": true + }, "source": [ "## Recommended to avoid\n", "The documentation recommends to avoid inplace operations such as" @@ -3087,8 +4066,12 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 28, + "id": "0ab38cff", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "a += b\n", @@ -3098,25 +4081,7 @@ ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.7" - } - }, + "metadata": {}, "nbformat": 4, - "nbformat_minor": 4 + "nbformat_minor": 5 } diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt index 68d3bfd83..bdf1ceb4b 100644 --- a/doc/src/week39/week39.do.txt +++ b/doc/src/week39/week39.do.txt @@ -1148,19 +1148,19 @@ C_{\text{ridge}}(\beta) = \frac{1}{n}||X\beta -\mathbf{y}||^2 + \lambda ||\beta| \] !et -In order to minimize $C_{\text{ridge}}(\beta)$ using GD we only have adjust the gradient as follows +In order to minimize $C_{\text{ridge}}(\beta)$ using GD we adjust the gradient as follows !bt \[ \nabla_\beta C_{\text{ridge}}(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ \sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ -\end{bmatrix} + 2\lambda\begin{bmatrix} \beta_0 \\ \beta_1\end{bmatrix} = 2 (X^T(X\beta - \mathbf{y})+\lambda \beta). +\end{bmatrix} + 2\lambda\begin{bmatrix} \beta_0 \\ \beta_1\end{bmatrix} = 2 (\frac{1}{n}X^T(X\beta - \mathbf{y})+\lambda \beta). \] !et We can easily extend our program to minimize $C_{\text{ridge}}(\beta)$ using gradient descent and compare with the analytical solution given by !bt \[ -\beta_{\text{ridge}} = \left(X^T X + \lambda I_{2 \times 2} \right)^{-1} X^T \mathbf{y}. +\beta_{\text{ridge}} = \left(X^T X + n\lambda I_{2 \times 2} \right)^{-1} X^T \mathbf{y}. \] !et diff --git a/doc/src/week43/week43-bs.html b/doc/src/week43/week43-bs.html new file mode 100644 index 000000000..20e7af09c --- /dev/null +++ b/doc/src/week43/week43-bs.html @@ -0,0 +1,3444 @@ + + + + + + + +Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis + + + + + + + + + + + + + + + + + + +
    +
    + + Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis +
    +
    + +
    +
    + +
    +

     

     

     

    + +
    +
    +

    Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis

    +
    + + +
    +Morten Hjorth-Jensen [1, 2] +
    + +
    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 2, 2021

    +
    +
    + + +
    + + +

    Plans for week 43

    + +
      +
    • Thursday: Summary of Convolutional Neural Networks from week 42 and Recurrent Neural Networks
      • + +
    • Friday: Recurrent Neural Networks and other Deep Learning methods such as Generalized Adversarial Neural Networks. Start discussing Principal component analysis
      • + +
    +
    +
    + + +
    +
    + + +
    +
    + + +
    +
    + + + +

    Reading Recommendations

    + +
      +
    • Goodfellow et al, chapter 10 on Recurrent NNs, chapters 11 and 12 on various practicalities around deep learning are also recommended.
      • +
    • Aurelien Geron, chapter 14 on RNNs.
      • +
    + +

    Summary on Deep Learning Methods

    + +

    We have studied fully connected neural networks (also called artifical nueral networks) and convolutional neural networks (CNNs).

    + +

    The first type of deep learning networks work very well on homogeneous and structured input data while CCNs are normally tailored to recognizing images.

    + + +

    CNNs in brief

    + +

    In summary:

    + +
      +
    • A CNN architecture is in the simplest case a list of Layers that transform the image volume into an output volume (e.g. holding the class scores)
      • +
    • There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular)
      • +
    • Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function
      • +
    • Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don’t)
      • +
    • Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn’t)
      • +
    +

    For more material on convolutional networks, we strongly recommend +the course +IN5400 – Machine Learning for Image Analysis +and the slides of CS231 which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs. +

    + +

    However, both standard feed forwards networks and CNNs perform well on data with unknown length.

    + +

    This is where recurrent nueral networks (RNNs) come to our rescue.

    + + +

    Recurrent neural networks: Overarching view

    + +

    Till now our focus has been, including convolutional neural networks +as well, on feedforward neural networks. The output or the activations +flow only in one direction, from the input layer to the output layer. +

    + +

    A recurrent neural network (RNN) looks very much like a feedforward +neural network, except that it also has connections pointing +backward. +

    + +

    RNNs are used to analyze time series data such as stock prices, and +tell you when to buy or sell. In autonomous driving systems, they can +anticipate car trajectories and help avoid accidents. More generally, +they can work on sequences of arbitrary lengths, rather than on +fixed-sized inputs like all the nets we have discussed so far. For +example, they can take sentences, documents, or audio samples as +input, making them extremely useful for natural language processing +systems such as automatic translation and speech-to-text. +

    + + +

    Set up of an RNN

    + +

    More to text to be added

    + + +

    A simple example

    + + + +
    +
    +
    +
    +
    + 
    # Start importing packages
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import tensorflow as tf
+from tensorflow.keras import datasets, layers, models
+from tensorflow.keras.layers import Input
+from tensorflow.keras.models import Model, Sequential 
+from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU
+from tensorflow.keras import optimizers     
+from tensorflow.keras import regularizers           
+from tensorflow.keras.utils import to_categorical 
+
+
+
+# convert into dataset matrix
+def convertToMatrix(data, step):
+ X, Y =[], []
+ for i in range(len(data)-step):
+  d=i+step  
+  X.append(data[i:d,])
+  Y.append(data[d,])
+ return np.array(X), np.array(Y)
+
+step = 4
+N = 1000    
+Tp = 800    
+
+t=np.arange(0,N)
+x=np.sin(0.02*t)+2*np.random.rand(N)
+df = pd.DataFrame(x)
+df.head()
+
+plt.plot(df)
+plt.show()
+
+values=df.values
+train,test = values[0:Tp,:], values[Tp:N,:]
+
+# add step elements into train and test
+test = np.append(test,np.repeat(test[-1,],step))
+train = np.append(train,np.repeat(train[-1,],step))
+ 
+trainX,trainY =convertToMatrix(train,step)
+testX,testY =convertToMatrix(test,step)
+trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
+testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))
+
+model = Sequential()
+model.add(SimpleRNN(units=32, input_shape=(1,step), activation="relu"))
+model.add(Dense(8, activation="relu")) 
+model.add(Dense(1))
+model.compile(loss='mean_squared_error', optimizer='rmsprop')
+model.summary()
+
+model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)
+trainPredict = model.predict(trainX)
+testPredict= model.predict(testX)
+predicted=np.concatenate((trainPredict,testPredict),axis=0)
+
+trainScore = model.evaluate(trainX, trainY, verbose=0)
+print(trainScore)
+
+index = df.index.values
+plt.plot(index,df)
+plt.plot(index,predicted)
+plt.axvline(df.index[Tp], c="r")
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    An extrapolation example

    + +

    The following code provides an example of how recurrent neural +networks can be used to extrapolate to unknown values of physics data +sets. Specifically, the data sets used in this program come from +a quantum mechanical many-body calculation of energies as functions of the number of particles. +

    + + + +
    +
    +
    +
    +
    + 
    # For matrices and calculations
+import numpy as np
+# For machine learning (backend for keras)
+import tensorflow as tf
+# User-friendly machine learning library
+# Front end for TensorFlow
+import tensorflow.keras
+# Different methods from Keras needed to create an RNN
+# This is not necessary but it shortened function calls 
+# that need to be used in the code.
+from tensorflow.keras import datasets, layers, models
+from tensorflow.keras.layers import Input
+from tensorflow.keras import regularizers
+from tensorflow.keras.models import Model, Sequential
+from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU
+# For timing the code
+from timeit import default_timer as timer
+# For plotting
+import matplotlib.pyplot as plt
+
+
+# The data set
+datatype='VaryDimension'
+X_tot = np.arange(2, 42, 2)
+y_tot = np.array([-0.03077640549, -0.08336233266, -0.1446729567, -0.2116753732, -0.2830637392, -0.3581341341, -0.436462435, -0.5177783846,
+	-0.6019067271, -0.6887363571, -0.7782028952, -0.8702784034, -0.9649652536, -1.062292565, -1.16231451, 
+	-1.265109911, -1.370782966, -1.479465113, -1.591317992, -1.70653767])
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Formatting the Data

    + +

    The way the recurrent neural networks are trained in this program +differs from how machine learning algorithms are usually trained. +Typically a machine learning algorithm is trained by learning the +relationship between the x data and the y data. In this program, the +recurrent neural network will be trained to recognize the relationship +in a sequence of y values. This is type of data formatting is +typically used time series forcasting, but it can also be used in any +extrapolation (time series forecasting is just a specific type of +extrapolation along the time axis). This method of data formatting +does not use the x data and assumes that the y data are evenly spaced. +

    + +

    For a standard machine learning algorithm, the training data has the +form of (x,y) so the machine learning algorithm learns to assiciate a +y value with a given x value. This is useful when the test data has x +values within the same range as the training data. However, for this +application, the x values of the test data are outside of the x values +of the training data and the traditional method of training a machine +learning algorithm does not work as well. For this reason, the +recurrent neural network is trained on sequences of y values of the +form ((y1, y2), y3), so that the network is concerned with learning +the pattern of the y data and not the relation between the x and y +data. As long as the pattern of y data outside of the training region +stays relatively stable compared to what was inside the training +region, this method of training can produce accurate extrapolations to +y values far removed from the training data set. +

    + + + + + + + + + + +
    +
    +
    +
    +
    + 
    # FORMAT_DATA
+def format_data(data, length_of_sequence = 2):  
+    """
+        Inputs:
+            data(a numpy array): the data that will be the inputs to the recurrent neural
+                network
+            length_of_sequence (an int): the number of elements in one iteration of the
+                sequence patter.  For a function approximator use length_of_sequence = 2.
+        Returns:
+            rnn_input (a 3D numpy array): the input data for the recurrent neural network.  Its
+                dimensions are length of data - length of sequence, length of sequence, 
+                dimnsion of data
+            rnn_output (a numpy array): the training data for the neural network
+        Formats data to be used in a recurrent neural network.
+    """
+
+    X, Y = [], []
+    for i in range(len(data)-length_of_sequence):
+        # Get the next length_of_sequence elements
+        a = data[i:i+length_of_sequence]
+        # Get the element that immediately follows that
+        b = data[i+length_of_sequence]
+        # Reshape so that each data point is contained in its own array
+        a = np.reshape (a, (len(a), 1))
+        X.append(a)
+        Y.append(b)
+    rnn_input = np.array(X)
+    rnn_output = np.array(Y)
+
+    return rnn_input, rnn_output
+
+
+# ## Defining the Recurrent Neural Network Using Keras
+# 
+# The following method defines a simple recurrent neural network in keras consisting of one input layer, one hidden layer, and one output layer.
+
+def rnn(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with one hidden layer and returns the model.
+    """
+    # Number of neurons in the input and output layers
+    in_out_neurons = 1
+    # Number of neurons in the hidden layer
+    hidden_neurons = 200
+    # Define the input layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons))  
+    # Define the hidden layer as a simple RNN layer with a set number of neurons and add it to 
+    # the network immediately after the input layer
+    rnn = SimpleRNN(hidden_neurons, 
+                    return_sequences=False,
+                    stateful = stateful,
+                    name="RNN")(inp)
+    # Define the output layer as a dense neural network layer (standard neural network layer)
+    #and add it to the network immediately after the hidden layer.
+    dens = Dense(in_out_neurons,name="dense")(rnn)
+    # Create the machine learning model starting with the input layer and ending with the 
+    # output layer
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the machine learning model using the mean squared error function as the loss 
+    # function and an Adams optimizer.
+    model.compile(loss="mean_squared_error", optimizer="adam")  
+    return model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Predicting New Points With A Trained Recurrent Neural Network

    + + + +
    +
    +
    +
    +
    + 
    def test_rnn (x1, y_test, plot_min, plot_max):
+    """
+        Inputs:
+            x1 (a list or numpy array): The complete x component of the data set
+            y_test (a list or numpy array): The complete y component of the data set
+            plot_min (an int or float): the smallest x value used in the training data
+            plot_max (an int or float): the largest x valye used in the training data
+        Returns:
+            None.
+        Uses a trained recurrent neural network model to predict future points in the 
+        series.  Computes the MSE of the predicted data set from the true data set, saves
+        the predicted data set to a csv file, and plots the predicted and true data sets w
+        while also displaying the data range used for training.
+    """
+    # Add the training data as the first dim points in the predicted data array as these
+    # are known values.
+    y_pred = y_test[:dim].tolist()
+    # Generate the first input to the trained recurrent neural network using the last two 
+    # points of the training data.  Based on how the network was trained this means that it
+    # will predict the first point in the data set after the training data.  All of the 
+    # brackets are necessary for Tensorflow.
+    next_input = np.array([[[y_test[dim-2]], [y_test[dim-1]]]])
+    # Save the very last point in the training data set.  This will be used later.
+    last = [y_test[dim-1]]
+
+    # Iterate until the complete data set is created.
+    for i in range (dim, len(y_test)):
+        # Predict the next point in the data set using the previous two points.
+        next = model.predict(next_input)
+        # Append just the number of the predicted data set
+        y_pred.append(next[0][0])
+        # Create the input that will be used to predict the next data point in the data set.
+        next_input = np.array([[last, next[0]]], dtype=np.float64)
+        last = next
+
+    # Print the mean squared error between the known data set and the predicted data set.
+    print('MSE: ', np.square(np.subtract(y_test, y_pred)).mean())
+    # Save the predicted data set as a csv file for later use
+    name = datatype + 'Predicted'+str(dim)+'.csv'
+    np.savetxt(name, y_pred, delimiter=',')
+    # Plot the known data set and the predicted data set.  The red box represents the region that was used
+    # for the training data.
+    fig, ax = plt.subplots()
+    ax.plot(x1, y_test, label="true", linewidth=3)
+    ax.plot(x1, y_pred, 'g-.',label="predicted", linewidth=4)
+    ax.legend()
+    # Created a red region to represent the points used in the training data.
+    ax.axvspan(plot_min, plot_max, alpha=0.25, color='red')
+    plt.show()
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+
+# Generate the training data for the RNN, using a sequence of 2
+rnn_input, rnn_training = format_data(y_train, 2)
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+model = rnn(length_of_sequences = rnn_input.shape[1])
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict more points of the data set
+test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Other Things to Try

    + +

    Changing the size of the recurrent neural network and its parameters +can drastically change the results you get from the model. The below +code takes the simple recurrent neural network from above and adds a +second hidden layer, changes the number of neurons in the hidden +layer, and explicitly declares the activation function of the hidden +layers to be a sigmoid function. The loss function and optimizer can +also be changed but are kept the same as the above network. These +parameters can be tuned to provide the optimal result from the +network. For some ideas on how to improve the performance of a +recurrent neural network. +

    + + + +
    +
    +
    +
    +
    + 
    def rnn_2layers(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with two hidden layers and returns the model.
+    """
+    # Number of neurons in the input and output layers
+    in_out_neurons = 1
+    # Number of neurons in the hidden layer, increased from the first network
+    hidden_neurons = 500
+    # Define the input layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons))  
+    # Create two hidden layers instead of one hidden layer.  Explicitly set the activation
+    # function to be the sigmoid function (the default value is hyperbolic tangent)
+    rnn1 = SimpleRNN(hidden_neurons, 
+                    return_sequences=True,  # This needs to be True if another hidden layer is to follow
+                    stateful = stateful, activation = 'sigmoid',
+                    name="RNN1")(inp)
+    rnn2 = SimpleRNN(hidden_neurons, 
+                    return_sequences=False, activation = 'sigmoid',
+                    stateful = stateful,
+                    name="RNN2")(rnn1)
+    # Define the output layer as a dense neural network layer (standard neural network layer)
+    #and add it to the network immediately after the hidden layer.
+    dens = Dense(in_out_neurons,name="dense")(rnn2)
+    # Create the machine learning model starting with the input layer and ending with the 
+    # output layer
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the machine learning model using the mean squared error function as the loss 
+    # function and an Adams optimizer.
+    model.compile(loss="mean_squared_error", optimizer="adam")  
+    return model
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+
+# Generate the training data for the RNN, using a sequence of 2
+rnn_input, rnn_training = format_data(y_train, 2)
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+model = rnn_2layers(length_of_sequences = 2)
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+
+# This section plots the training loss and the validation loss as a function of training iteration.
+# This is not required for analyzing the couple cluster data but can help determine if the network is
+# being overtrained.
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict more points of the data set
+test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Other Types of Recurrent Neural Networks

    + +

    Besides a simple recurrent neural network layer, there are two other +commonly used types of recurrent neural network layers: Long Short +Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short +introduction to these layers see https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b +and https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b. +

    + +

    The first network created below is similar to the previous network, +but it replaces the SimpleRNN layers with LSTM layers. The second +network below has two hidden layers made up of GRUs, which are +preceeded by two dense (feeddorward) neural network layers. These +dense layers "preprocess" the data before it reaches the recurrent +layers. This architecture has been shown to improve the performance +of recurrent neural networks (see the link above and also +https://arxiv.org/pdf/1807.02857.pdf. +

    + + + +
    +
    +
    +
    +
    + 
    def lstm_2layers(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.
+    """
+    # Number of neurons on the input/output layer and the number of neurons in the hidden layer
+    in_out_neurons = 1
+    hidden_neurons = 250
+    # Input Layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons)) 
+    # Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)
+    rnn= LSTM(hidden_neurons, 
+                    return_sequences=True,
+                    stateful = stateful,
+                    name="RNN", use_bias=True, activation='tanh')(inp)
+    rnn1 = LSTM(hidden_neurons, 
+                    return_sequences=False,
+                    stateful = stateful,
+                    name="RNN1", use_bias=True, activation='tanh')(rnn)
+    # Output layer
+    dens = Dense(in_out_neurons,name="dense")(rnn1)
+    # Define the midel
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the model
+    model.compile(loss='mean_squared_error', optimizer='adam')  
+    # Return the model
+    return model
+
+def dnn2_gru2(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with four hidden layers (two dense followed by
+        two GRU layers) and returns the model.
+    """    
+    # Number of neurons on the input/output layers and hidden layers
+    in_out_neurons = 1
+    hidden_neurons = 250
+    # Input layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons)) 
+    # Hidden Dense (feedforward) layers
+    dnn = Dense(hidden_neurons/2, activation='relu', name='dnn')(inp)
+    dnn1 = Dense(hidden_neurons/2, activation='relu', name='dnn1')(dnn)
+    # Hidden GRU layers
+    rnn1 = GRU(hidden_neurons, 
+                    return_sequences=True,
+                    stateful = stateful,
+                    name="RNN1", use_bias=True)(dnn1)
+    rnn = GRU(hidden_neurons, 
+                    return_sequences=False,
+                    stateful = stateful,
+                    name="RNN", use_bias=True)(rnn1)
+    # Output layer
+    dens = Dense(in_out_neurons,name="dense")(rnn)
+    # Define the model
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the mdoel
+    model.compile(loss='mean_squared_error', optimizer='adam')  
+    # Return the model
+    return model
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+
+# Generate the training data for the RNN, using a sequence of 2
+rnn_input, rnn_training = format_data(y_train, 2)
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+# Change the method name to reflect which network you want to use
+model = dnn2_gru2(length_of_sequences = 2)
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+
+# This section plots the training loss and the validation loss as a function of training iteration.
+# This is not required for analyzing the couple cluster data but can help determine if the network is
+# being overtrained.
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict more points of the data set
+test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
+
+# ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)
+# 
+# Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+# Reshape the data for Keras specifications
+X_train = X_train.reshape((dim, 1))
+y_train = y_train.reshape((dim, 1))
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+# Set the sequence length to 1 for regular data formatting 
+model = rnn(length_of_sequences = 1)
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(X_train, y_train, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+
+# This section plots the training loss and the validation loss as a function of training iteration.
+# This is not required for analyzing the couple cluster data but can help determine if the network is
+# being overtrained.
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict the remaining data points
+X_pred = X_tot[dim:]
+X_pred = X_pred.reshape((len(X_pred), 1))
+y_model = model.predict(X_pred)
+y_pred = np.concatenate((y_tot[:dim], y_model.flatten()))
+
+# Plot the known data set and the predicted data set.  The red box represents the region that was used
+# for the training data.
+fig, ax = plt.subplots()
+ax.plot(X_tot, y_tot, label="true", linewidth=3)
+ax.plot(X_tot, y_pred, 'g-.',label="predicted", linewidth=4)
+ax.legend()
+# Created a red region to represent the points used in the training data.
+ax.axvspan(X_tot[0], X_tot[dim], alpha=0.25, color='red')
+plt.show()
+
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Generative Models

    + +

    Generative models describe a class of statistical models that are a contrast +to discriminative models. Informally we say that generative models can +generate new data instances while discriminative models discriminate between +different kinds of data instances. A generative model could generate new photos +of animals that look like 'real' animals while a discriminative model could tell +a dog from a cat. More formally, given a data set \( x \) and a set of labels / +targets \( y \). Generative models capture the joint probability \( p(x, y) \), or +just \( p(x) \) if there are no labels, while discriminative models capture the +conditional probability \( p(y | x) \). Discriminative models generally try to draw +boundaries in the data space (often high dimensional), while generative models +try to model how data is placed throughout the space. +

    + +

    Note: this material is thanks to Linus Ekstrøm.

    + + +

    Generative Adversarial Networks

    + +

    Generative Adversarial Networks are a type of unsupervised machine learning +algorithm proposed by Goodfellow et. al +in 2014 (short and good article). +

    + +

    The simplest formulation of +the model is based on a game theoretic approach, zero sum game, where we pit +two neural networks against one another. We define two rival networks, one +generator \( g \), and one discriminator \( d \). The generator directly produces +samples +

    +$$ +\begin{equation} + x = g(z; \theta^{(g)}) +\label{_auto1} +\end{equation} +$$ + + + +

    Discriminator

    +

    The discriminator attempts to distinguish between samples drawn from the +training data and samples drawn from the generator. In other words, it tries to +tell the difference between the fake data produced by \( g \) and the actual data +samples we want to do prediction on. The discriminator outputs a probability +value given by +

    + +$$ +\begin{equation} + d(x; \theta^{(d)}) +\label{_auto2} +\end{equation} +$$ + +

    indicating the probability that \( x \) is a real training example rather than a +fake sample the generator has generated. The simplest way to formulate the +learning process in a generative adversarial network is a zero-sum game, in +which a function +

    + +$$ +\begin{equation} + v(\theta^{(g)}, \theta^{(d)}) +\label{_auto3} +\end{equation} +$$ + +

    determines the reward for the discriminator, while the generator gets the +conjugate reward +

    + +$$ +\begin{equation} + -v(\theta^{(g)}, \theta^{(d)}) +\label{_auto4} +\end{equation} +$$ + + + +

    Learning Process

    + +

    During learning both of the networks maximize their own reward function, so that +the generator gets better and better at tricking the discriminator, while the +discriminator gets better and better at telling the difference between the fake +and real data. The generator and discriminator alternate on which one trains at +one time (i.e. for one epoch). In other words, we keep the generator constant +and train the discriminator, then we keep the discriminator constant to train +the generator and repeat. It is this back and forth dynamic which lets GANs +tackle otherwise intractable generative problems. As the generator improves with + training, the discriminator's performance gets worse because it cannot easily + tell the difference between real and fake. If the generator ends up succeeding + perfectly, the the discriminator will do no better than random guessing i.e. + 50\%. This progression in the training poses a problem for the convergence + criteria for GANs. The discriminator feedback gets less meaningful over time, + if we continue training after this point then the generator is effectively + training on junk data which can undo the learning up to that point. Therefore, + we stop training when the discriminator starts outputting \( 1/2 \) everywhere. +

    + + +

    More about the Learning Process

    + +

    At convergence we have

    + +$$ +\begin{equation} + g^* = \underset{g}{\mathrm{argmin}}\hspace{2pt} + \underset{d}{\mathrm{max}}v(\theta^{(g)}, \theta^{(d)}) +\label{_auto5} +\end{equation} +$$ + +

    The default choice for \( v \) is

    +$$ +\begin{equation} + v(\theta^{(g)}, \theta^{(d)}) = \mathbb{E}_{x\sim p_\mathrm{data}}\log d(x) + + \mathbb{E}_{x\sim p_\mathrm{model}} + \log (1 - d(x)) +\label{_auto6} +\end{equation} +$$ + +

    The main motivation for the design of GANs is that the learning process requires +neither approximate inference (variational autoencoders for example) nor +approximation of a partition function. In the case where +

    +$$ +\begin{equation} + \underset{d}{\mathrm{max}}v(\theta^{(g)}, \theta^{(d)}) +\label{_auto7} +\end{equation} +$$ + +

    is convex in $\theta^{(g)} then the procedure is guaranteed to converge and is +asymptotically consistent +( Seth Lloyd on QuGANs ). +

    + + +

    Additional References

    +

    This is in +general not the case and it is possible to get situations where the training +process never converges because the generator and discriminator chase one +another around in the parameter space indefinitely. A much deeper discussion on +the currently open research problem of GAN convergence is available +here. To +anyone interested in learning more about GANs it is a highly recommended read. +Direct quote: "In this best-performing formulation, the generator aims to +increase the log probability that the discriminator makes a mistake, rather than +aiming to decrease the log probability that the discriminator makes the correct +prediction." Another interesting read +

    + + +

    Writing Our First Generative Adversarial Network

    +

    Let us now move on to actually implementing a GAN in tensorflow. We will study +the performance of our GAN on the MNIST dataset. This code is based on and +adapted from the +google tutorial +

    + +

    First we import our libraries

    + + + +
    +
    +
    +
    +
    + 
    import os
+import time
+import numpy as np
+import tensorflow as tf
+import matplotlib.pyplot as plt
+from tensorflow.keras import layers
+from tensorflow.keras.utils import plot_model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we define our hyperparameters and import our data the usual way

    + + + +
    +
    +
    +
    +
    + 
    BUFFER_SIZE = 60000
+BATCH_SIZE = 256
+EPOCHS = 30
+
+data = tf.keras.datasets.mnist.load_data()
+(train_images, train_labels), (test_images, test_labels) = data
+train_images = np.reshape(train_images, (train_images.shape[0],
+                                         28,
+                                         28,
+                                         1)).astype('float32')
+
+# we normalize between -1 and 1
+train_images = (train_images - 127.5) / 127.5
+training_dataset = tf.data.Dataset.from_tensor_slices(
+                      train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    MNIST and GANs

    + +

    Let's have a quick look

    + + + +
    +
    +
    +
    +
    + 
    plt.imshow(train_images[0], cmap='Greys')
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we define our two models. This is where the 'magic' happens. There are a +huge amount of possible formulations for both models. A lot of engineering and +trial and error can be done here to try to produce better performing models. For +more advanced GANs this is by far the step where you can 'make or break' a +model. +

    + +

    We start with the generator. As stated in the introductory text the generator +\( g \) upsamples from a random sample to the shape of what we want to predict. In +our case we are trying to predict MNIST images (\( 28\times 28 \) pixels). +

    + + + +
    +
    +
    +
    +
    + 
    def generator_model():
+    """
+    The generator uses upsampling layers tf.keras.layers.Conv2DTranspose() to
+    produce an image from a random seed. We start with a Dense layer taking this
+    random sample as an input and subsequently upsample through multiple
+    convolutional layers.
+    """
+
+    # we define our model
+    model = tf.keras.Sequential()
+
+
+    # adding our input layer. Dense means that every neuron is connected and
+    # the input shape is the shape of our random noise. The units need to match
+    # in some sense the upsampling strides to reach our desired output shape.
+    # we are using 100 random numbers as our seed
+    model.add(layers.Dense(units=7*7*BATCH_SIZE,
+                           use_bias=False,
+                           input_shape=(100, )))
+    # we normalize the output form the Dense layer
+    model.add(layers.BatchNormalization())
+    # and add an activation function to our 'layer'. LeakyReLU avoids vanishing
+    # gradient problem
+    model.add(layers.LeakyReLU())
+    model.add(layers.Reshape((7, 7, BATCH_SIZE)))
+    assert model.output_shape == (None, 7, 7, BATCH_SIZE)
+    # even though we just added four keras layers we think of everything above
+    # as 'one' layer
+
+    # next we add our upscaling convolutional layers
+    model.add(layers.Conv2DTranspose(filters=128,
+                                     kernel_size=(5, 5),
+                                     strides=(1, 1),
+                                     padding='same',
+                                     use_bias=False))
+    model.add(layers.BatchNormalization())
+    model.add(layers.LeakyReLU())
+    assert model.output_shape == (None, 7, 7, 128)
+
+    model.add(layers.Conv2DTranspose(filters=64,
+                                     kernel_size=(5, 5),
+                                     strides=(2, 2),
+                                     padding='same',
+                                     use_bias=False))
+    model.add(layers.BatchNormalization())
+    model.add(layers.LeakyReLU())
+    assert model.output_shape == (None, 14, 14, 64)
+
+    model.add(layers.Conv2DTranspose(filters=1,
+                                     kernel_size=(5, 5),
+                                     strides=(2, 2),
+                                     padding='same',
+                                     use_bias=False,
+                                     activation='tanh'))
+    assert model.output_shape == (None, 28, 28, 1)
+
+    return model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    And there we have our 'simple' generator model. Now we move on to defining our +discriminator model \( d \), which is a convolutional neural network based image +classifier. +

    + + + +
    +
    +
    +
    +
    + 
    def discriminator_model():
+    """
+    The discriminator is a convolutional neural network based image classifier
+    """
+
+    # we define our model
+    model = tf.keras.Sequential()
+    model.add(layers.Conv2D(filters=64,
+                            kernel_size=(5, 5),
+                            strides=(2, 2),
+                            padding='same',
+                            input_shape=[28, 28, 1]))
+    model.add(layers.LeakyReLU())
+    # adding a dropout layer as you do in conv-nets
+    model.add(layers.Dropout(0.3))
+
+
+    model.add(layers.Conv2D(filters=128,
+                            kernel_size=(5, 5),
+                            strides=(2, 2),
+                            padding='same'))
+    model.add(layers.LeakyReLU())
+    # adding a dropout layer as you do in conv-nets
+    model.add(layers.Dropout(0.3))
+
+    model.add(layers.Flatten())
+    model.add(layers.Dense(1))
+
+    return model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Other Models

    +

    Let us take a look at our models. Note: double click images for bigger view.

    + + + +
    +
    +
    +
    +
    + 
    generator = generator_model()
+plot_model(generator, show_shapes=True, rankdir='LR')
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    discriminator = discriminator_model()
+plot_model(discriminator, show_shapes=True, rankdir='LR')
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we need a few helper objects we will use in training

    + + + +
    +
    +
    +
    +
    + 
    cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)
+generator_optimizer = tf.keras.optimizers.Adam(1e-4)
+discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    The first object, cross_entropy is our loss function and the two others are +our optimizers. Notice we use the same learning rate for both \( g \) and \( d \). This +is because they need to improve their accuracy at approximately equal speeds to +get convergence (not necessarily exactly equal). Now we define our loss +functions +

    + + + +
    +
    +
    +
    +
    + 
    def generator_loss(fake_output):
+    loss = cross_entropy(tf.ones_like(fake_output), fake_output)
+
+    return loss
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    def discriminator_loss(real_output, fake_output):
+    real_loss = cross_entropy(tf.ones_like(real_output), real_output)
+    fake_loss = cross_entropy(tf.zeros_liks(fake_output), fake_output)
+    total_loss = real_loss + fake_loss
+
+    return total_loss
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we define a kind of seed to help us compare the learning process over +multiple training epochs. +

    + + + +
    +
    +
    +
    +
    + 
    noise_dimension = 100
+n_examples_to_generate = 16
+seed_images = tf.random.normal([n_examples_to_generate, noise_dimension])
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Training Step

    + +

    Now we have everything we need to define our training step, which we will apply +for every step in our training loop. Notice the @tf.function flag signifying +that the function is tensorflow 'compiled'. Removing this flag doubles the +computation time. +

    + + + +
    +
    +
    +
    +
    + 
    @tf.function
+def train_step(images):
+    noise = tf.random.normal([BATCH_SIZE, noise_dimension])
+
+    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:
+        generated_images = generator(noise, training=True)
+
+        real_output = discriminator(images, training=True)
+        fake_output = discriminator(generated_images, training=True)
+
+        gen_loss = generator_loss(fake_output)
+        disc_loss = discriminator_loss(real_output, fake_output)
+
+    gradients_of_generator = gen_tape.gradient(gen_loss,
+                                            generator.trainable_variables)
+    gradients_of_discriminator = disc_tape.gradient(disc_loss,
+                                            discriminator.trainable_variables)
+    generator_optimizer.apply_gradients(zip(gradients_of_generator,
+                                            generator.trainable_variables))
+    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator,
+                                            discriminator.trainable_variables))
+
+    return gen_loss, disc_loss
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we define a helper function to produce an output over our training epochs +to see the predictive progression of our generator model. Note: I am including +this code here, but comment it out in the training loop. +

    + + +
    +
    +
    +
    +
    + 
    def generate_and_save_images(model, epoch, test_input):
+    # we're making inferences here
+    predictions = model(test_input, training=False)
+
+    fig = plt.figure(figsize=(4, 4))
+
+    for i in range(predictions.shape[0]):
+        plt.subplot(4, 4, i+1)
+        plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')
+        plt.axis('off')
+
+    plt.savefig(f'./images_from_seed_images/image_at_epoch_{str(epoch).zfill(3)}.png')
+    plt.close()
+    #plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Checkpoints

    +

    Setting up checkpoints to periodically save our model during training so that +everything is not lost even if the program were to somehow terminate while +training. +

    + + + +
    +
    +
    +
    +
    + 
    # Setting up checkpoints to save model during training
+checkpoint_dir = './training_checkpoints'
+checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt')
+checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,
+                            discriminator_optimizer=discriminator_optimizer,
+                            generator=generator,
+                            discriminator=discriminator)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we define our training loop

    + + + +
    +
    +
    +
    +
    + 
    def train(dataset, epochs):
+    generator_loss_list = []
+    discriminator_loss_list = []
+
+    for epoch in range(epochs):
+        start = time.time()
+
+        for image_batch in dataset:
+            gen_loss, disc_loss = train_step(image_batch)
+            generator_loss_list.append(gen_loss.numpy())
+            discriminator_loss_list.append(disc_loss.numpy())
+
+        #generate_and_save_images(generator, epoch + 1, seed_images)
+
+        if (epoch + 1) % 15 == 0:
+            checkpoint.save(file_prefix=checkpoint_prefix)
+
+        print(f'Time for epoch {epoch} is {time.time() - start}')
+
+    #generate_and_save_images(generator, epochs, seed_images)
+
+    loss_file = './data/lossfile.txt'
+    with open(loss_file, 'w') as outfile:
+        outfile.write(str(generator_loss_list))
+        outfile.write('\n')
+        outfile.write('\n')
+        outfile.write(str(discriminator_loss_list))
+        outfile.write('\n')
+        outfile.write('\n')
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    To train simply call this function. Warning: this might take a long time so +there is a folder of a pretrained network already included in the repository. +

    + + + +
    +
    +
    +
    +
    + 
    train(train_dataset, EPOCHS)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    And here is the result of training our model for 100 epochs

    + + +

    + +

    Now to avoid having to train and everything, which will take a while depending +on your computer setup we now load in the model which produced the above gif. +

    + + + +
    +
    +
    +
    +
    + 
    checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))
+restored_generator = checkpoint.generator
+restored_discriminator = checkpoint.discriminator
+
+print(restored_generator)
+print(restored_discriminator)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Exploring the Latent Space

    + +

    We have successfully loaded in our latest model. Let us now play around a bit +and see what kind of things we can learn about this model. Our generator takes +an array of 100 numbers. One idea can be to try to systematically change our +input. Let us try and see what we get +

    + + + +
    +
    +
    +
    +
    + 
    def generate_latent_points(number=100, scale_means=1, scale_stds=1):
+    latent_dim = 100
+    means = scale_means * tf.linspace(-1, 1, num=latent_dim)
+    stds = scale_stds * tf.linspace(-1, 1, num=latent_dim)
+    latent_space_value_range = tf.random.normal([number, latent_dim],
+                                                means,
+                                                stds,
+                                                dtype=tf.float64)
+
+    return latent_space_value_range
+
+def generate_images(latent_points):
+    # notice we set training to false because we are making inferences
+    generated_images = restored_generator.predict(latent_points)
+
+    return generated_images
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    def plot_result(generated_images, number=100):
+    # obviously this assumes sqrt number is an int
+    fig, axs = plt.subplots(int(np.sqrt(number)), int(np.sqrt(number)),
+                            figsize=(10, 10))
+
+    for i in range(int(np.sqrt(number))):
+        for j in range(int(np.sqrt(number))):
+            axs[i, j].imshow(generated_images[i*j], cmap='Greys')
+            axs[i, j].axis('off')
+
+    plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    generated_images = generate_images(generate_latent_points())
+plot_result(generated_images)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Getting Results

    +

    We see that the generator generates images that look like MNIST +numbers: \( 1, 4, 7, 9 \). Let's try to tweak it a bit more to see if we are able +to generate a similar plot where we generate every MNIST number. Let us now try +to 'move' a bit around in the latent space. Note: decrease the plot number if +these following cells take too long to run on your computer. +

    + + + +
    +
    +
    +
    +
    + 
    plot_number = 225
+
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=5,
+                                                          scale_stds=1))
+plot_result(generated_images, number=plot_number)
+
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=-5,
+                                                          scale_stds=1))
+plot_result(generated_images, number=plot_number)
+
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=1,
+                                                          scale_stds=5))
+plot_result(generated_images, number=plot_number)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Again, we have found something interesting. Moving around using our means +takes us from digit to digit, while moving around using our standard +deviations seem to increase the number of different digits! In the last image +above, we can barely make out every MNIST digit. Let us make on last plot using +this information by upping the standard deviation of our Gaussian noises. +

    + + + +
    +
    +
    +
    +
    + 
    plot_number = 400
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=1,
+                                                          scale_stds=10))
+plot_result(generated_images, number=plot_number)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    A pretty cool result! We see that our generator indeed has learned a +distribution which qualitatively looks a whole lot like the MNIST dataset. +

    + + +

    Interpolating Between MNIST Digits

    +

    Another interesting way to explore the latent space of our generator model is by +interpolating between the MNIST digits. This section is largely based on +this excellent blogpost +by Jason Brownlee. +

    + +

    So let us start by defining a function to interpolate between two points in the +latent space. +

    + + + +
    +
    +
    +
    +
    + 
    def interpolation(point_1, point_2, n_steps=10):
+    ratios = np.linspace(0, 1, num=n_steps)
+    vectors = []
+    for i, ratio in enumerate(ratios):
+        vectors.append(((1.0 - ratio) * point_1 + ratio * point_2))
+
+    return tf.stack(vectors)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we have all we need to do our interpolation analysis.

    + + + +
    +
    +
    +
    +
    + 
    plot_number = 100
+latent_points = generate_latent_points(number=plot_number)
+results = None
+for i in range(0, 2*np.sqrt(plot_number), 2):
+    interpolated = interpolation(latent_points[i], latent_points[i+1])
+    generated_images = generate_images(interpolated)
+
+    if results is None:
+        results = generated_images
+    else:
+        results = tf.stack((results, generated_images))
+
+plot_results(results, plot_number)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Basic ideas of the Principal Component Analysis (PCA)

    + +

    The principal component analysis deals with the problem of fitting a +low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than +the total dimension \( D \) of the problem at hand (our data +set). Mathematically it can be formulated as a statistical problem or +a geometric problem. In our discussion of the theorem for the +classical PCA, we will stay with a statistical approach. +Historically, the PCA was first formulated in a statistical setting in order to estimate the principal component of a multivariate random variable. +

    + +

    We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)

    +
      +
    • Each data point is determined by \( p \) extrinsic (measurement) variables
      • +
    • We may want to ask the following question: Are there fewer intrinsic variables (say \( d < < p \)) that still approximately describe the data?
      • +
    • If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.
      • +
    +

    A good read is for example Vidal, Ma and Sastry.

    + + +

    Introducing the Covariance and Correlation functions

    + +

    Before we discuss the PCA theorem, we need to remind ourselves about +the definition of the covariance and the correlation function. These are quantities +

    + +

    Suppose we have defined two vectors +\( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as +

    +$$ +\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\ + \end{bmatrix}, +$$ + +

    where for example

    +$$ +\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). +$$ + +

    With this definition and recalling that the variance is defined as

    +$$ +\mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2, +$$ + +

    we can rewrite the covariance matrix as

    +$$ +\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\ + \end{bmatrix}. +$$ + + + +

    More on the covariance

    +

    The covariance takes values between zero and infinity and may thus +lead to problems with loss of numerical precision for particularly +large values. It is common to scale the covariance matrix by +introducing instead the correlation matrix defined via the so-called +correlation function +

    + +$$ +\mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}. +$$ + +

    The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] +\in [-1,1] \). This avoids eventual problems with too large values. We +can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) +and \( \boldsymbol{y} \) as +

    + +$$ +\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{corr}[\boldsymbol{y},\boldsymbol{x}] & 1 \\ + \end{bmatrix}, +$$ + +

    In the above example this is the function we constructed using pandas.

    + + +

    Reminding ourselves about Linear Regression

    +

    In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression +we defined the design/feature matrix \( \boldsymbol{X} \) as +

    + +$$ +\boldsymbol{X}=\begin{bmatrix} +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\ +\dots & \dots & \dots & \dots \dots & \dots \\ +x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ +\end{bmatrix}, +$$ + +

    with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the +entries \( n \) being the row elements. +We can rewrite the design/feature matrix in terms of its column vectors as +

    +$$ +\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix}, +$$ + +

    with a given vector

    +$$ +\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. +$$ + + + +

    Simple Example

    +

    With these definitions, we can now rewrite our \( 2\times 2 \) +correlation/covariance matrix in terms of a moe general design/feature +matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) +covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) +

    + +$$ +\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} +\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +\mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_1] & \mathrm{var}[\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & \mathrm{var}[\boldsymbol{x}_{p-1}]\\ +\end{bmatrix}, +$$ + + + +

    The Correlation Matrix

    + +

    and the correlation matrix

    +$$ +\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} +1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +\mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +\mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\ +\end{bmatrix}, +$$ + + + +

    Numpy Functionality

    + +

    The Numpy function np.cov calculates the covariance elements using +the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have +the exact mean values. The following simple function uses the +np.vstack function which takes each vector of dimension \( 1\times n \) +and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +

    + +$$ +\boldsymbol{W}^T = \begin{bmatrix} x_0 & y_0 \\ + x_1 & y_1 \\ + x_2 & y_2\\ + \dots & \dots \\ + x_{n-2} & y_{n-2}\\ + x_{n-1} & y_{n-1} & + \end{bmatrix}, +$$ + +

    which in turn is converted into into the \( 2\times 2 \) covariance matrix +\( \boldsymbol{C} \) via the Numpy function np.cov(). We note that we can also calculate +the mean value of each set of samples \( \boldsymbol{x} \) etc using the Numpy +function np.mean(x). We can also extract the eigenvalues of the +covariance matrix through the np.linalg.eig() function. +

    + + + +
    +
    +
    +
    +
    + 
    # Importing various packages
+import numpy as np
+n = 100
+x = np.random.normal(size=n)
+print(np.mean(x))
+y = 4+3*x+np.random.normal(size=n)
+print(np.mean(y))
+W = np.vstack((x, y))
+C = np.cov(W)
+print(C)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Correlation Matrix again

    + +

    The previous example can be converted into the correlation matrix by +simply scaling the matrix elements with the variances. We should also +subtract the mean values for each column. This leads to the following +code which sets up the correlations matrix for the previous example in +a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors). +

    + + + +
    +
    +
    +
    +
    + 
    import numpy as np
+n = 100
+# define two vectors                                                                                           
+x = np.random.random(size=n)
+y = 4+3*x+np.random.normal(size=n)
+#scaling the x and y vectors                                                                                   
+x = x - np.mean(x)
+y = y - np.mean(y)
+variance_x = np.sum(x@x)/n
+variance_y = np.sum(y@y)/n
+print(variance_x)
+print(variance_y)
+cov_xy = np.sum(x@y)/n
+cov_xx = np.sum(x@x)/n
+cov_yy = np.sum(y@y)/n
+C = np.zeros((2,2))
+C[0,0]= cov_xx/variance_x
+C[1,1]= cov_yy/variance_y
+C[0,1]= cov_xy/np.sqrt(variance_y*variance_x)
+C[1,0]= C[0,1]
+print(C)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We see that the matrix elements along the diagonal are one as they +should be and that the matrix is symmetric. Furthermore, diagonalizing +this matrix we easily see that it is a positive definite matrix. +

    + +

    The above procedure with numpy can be made more compact if we use pandas.

    + + +

    Using Pandas

    + +

    We whow here how we can set up the correlation matrix using pandas, as done in this simple code

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import pandas as pd
+n = 10
+x = np.random.normal(size=n)
+x = x - np.mean(x)
+y = 4+3*x+np.random.normal(size=n)
+y = y - np.mean(y)
+X = (np.vstack((x, y))).T
+print(X)
+Xpd = pd.DataFrame(X)
+print(Xpd)
+correlation_matrix = Xpd.corr()
+print(correlation_matrix)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    And then the Franke Function

    + +

    We expand this model to the Franke function discussed above.

    + + + +
    +
    +
    +
    +
    + 
    # Common imports
+import numpy as np
+import pandas as pd
+
+
+def FrankeFunction(x,y):
+	term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
+	term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
+	term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
+	term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
+	return term1 + term2 + term3 + term4
+
+
+def create_X(x, y, n ):
+	if len(x.shape) > 1:
+		x = np.ravel(x)
+		y = np.ravel(y)
+
+	N = len(x)
+	l = int((n+1)*(n+2)/2)		# Number of elements in beta
+	X = np.ones((N,l))
+
+	for i in range(1,n+1):
+		q = int((i)*(i+1)/2)
+		for k in range(i+1):
+			X[:,q+k] = (x**(i-k))*(y**k)
+
+	return X
+
+
+# Making meshgrid of datapoints and compute Franke's function
+n = 4
+N = 100
+x = np.sort(np.random.uniform(0, 1, N))
+y = np.sort(np.random.uniform(0, 1, N))
+z = FrankeFunction(x, y)
+X = create_X(x, y, n=n)    
+
+Xpd = pd.DataFrame(X)
+# subtract the mean values and set up the covariance matrix
+Xpd = Xpd - Xpd.mean()
+covariance_matrix = Xpd.cov()
+print(covariance_matrix)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We note here that the covariance is zero for the first rows and +columns since all matrix elements in the design matrix were set to one +(we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept +and wee can simply +drop these elements and construct a correlation +matrix without them by centering our matrix elements by subtracting the mean of each column. +

    + + +

    Lnks with the Design Matrix

    + +

    We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as

    +$$ +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. +$$ + +

    To see this let us simply look at a design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{2\times 2} \)

    +$$ +\boldsymbol{X}=\begin{bmatrix} +x_{00} & x_{01}\\ +x_{10} & x_{11}\\ +\end{bmatrix}=\begin{bmatrix} +\boldsymbol{x}_{0} & \boldsymbol{x}_{1}\\ +\end{bmatrix}. +$$ + + + +

    Computing the Expectation Values

    + +

    If we then compute the expectation value

    +$$ +\mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\begin{bmatrix} +x_{00}^2+x_{01}^2 & x_{00}x_{10}+x_{01}x_{11}\\ +x_{10}x_{00}+x_{11}x_{01} & x_{10}^2+x_{11}^2\\ +\end{bmatrix}, +$$ + +

    which is just

    +$$ +\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]=\begin{bmatrix} \mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] \\ + \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] \\ + \end{bmatrix}, +$$ + +

    where we wrote $$\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]$$ to indicate that this the covariance of the vectors \( \boldsymbol{x} \) of the design/feature matrix \( \boldsymbol{X} \).

    + +

    It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \).

    + + +

    Towards the PCA theorem

    + +

    We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as

    +$$ +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. +$$ + +

    Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices \( \boldsymbol{S} \). +These matrices are defined as \( \boldsymbol{S}\in {\mathbb{R}}^{p\times p} \) and obey the orthogonality requirements \( \boldsymbol{S}\boldsymbol{S}^T=\boldsymbol{S}^T\boldsymbol{S}=\boldsymbol{I} \). The matrix can be written out in terms of the column vectors \( \boldsymbol{s}_i \) as \( \boldsymbol{S}=[\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \) and \( \boldsymbol{s}_i \in {\mathbb{R}}^{p} \). +

    + +

    Assume also that there is a transformation \( \boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}=\boldsymbol{C}[\boldsymbol{y}] \) such that the new matrix \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal with elements \( [\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}] \).

    + +

    That is we have

    +$$ +\boldsymbol{C}[\boldsymbol{y}] = \mathbb{E}[\boldsymbol{S}^T\boldsymbol{X}^T\boldsymbol{X}T\boldsymbol{S}]=\boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}, +$$ + +

    since the matrix \( \boldsymbol{S} \) is not a data dependent matrix. Multiplying with \( \boldsymbol{S} \) from the left we have

    +$$ +\boldsymbol{S}\boldsymbol{C}[\boldsymbol{y}] = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}, +$$ + +

    and since \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal we have for a given eigenvalue \( i \) of the covariance matrix that

    + +$$ +\boldsymbol{S}_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}_i. +$$ + + + +

    More on the PCA Theorem

    + +

    In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is +\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \). +

    + +

    The eigenvalues tell us then how much we need to stretch the +corresponding eigenvectors. Dimensions with large eigenvalues have +thus large variations (large variance) and define therefore useful +dimensions. The data points are more spread out in the direction of +these eigenvectors. Smaller eigenvalues mean on the other hand that +the corresponding eigenvectors are shrunk accordingly and the data +points are tightly bunched together and there is not much variation in +these specific directions. Hopefully then we could leave it out +dimensions where the eigenvalues are very small. If \( p \) is very large, +we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) +features/predictors. +

    + + +

    The Algorithm before theorem

    + +

    Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.

    +
      +
    • Set up the datapoints for the design/feature matrix \( \boldsymbol{X} \) with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) referring to the column numbers and the entries \( n \) being the row elements.
      • +
    +$$ +\boldsymbol{X}=\begin{bmatrix} +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\ +\dots & \dots & \dots & \dots \dots & \dots \\ +x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ +\end{bmatrix}, +$$ + +
      +
    • Center the data by subtracting the mean value for each column. This leads to a new matrix \( \boldsymbol{X}\rightarrow \overline{\boldsymbol{X}} \).
      • +
    • Compute then the covariance/correlation matrix \( \mathbb{E}[\overline{\boldsymbol{X}}^T\overline{\boldsymbol{X}}] \).
      • +
    • Find the eigenpairs of \( \boldsymbol{C} \) with eigenvalues \( [\lambda_0,\lambda_1,\dots,\lambda_{p-1}] \) and eigenvectors \( [\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \).
      • +
    • Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.
      • +
    • Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.
      • +
    + +

    Writing our own PCA code

    + +

    We will use a simple example first with two-dimensional data +drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below): +

    +$$ +\mu = (-1,2) \qquad \Sigma = \begin{bmatrix} 4 & 2 \\ +2 & 2 +\end{bmatrix} +$$ + +

    Note that the mean refers to each column of data. +We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from +this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values. +

    + + +

    Implementing it

    +

    The following Python code aids in setting up the data and writing out the design matrix. +Note that the function multivariate returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \). +

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from IPython.display import display
+n = 10000
+mean = (-1, 2)
+cov = [[4, 2], [2, 2]]
+X = np.random.multivariate_normal(mean, cov, n)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we are going to implement the PCA algorithm. We will break it down into various substeps.

    + + +

    First Step

    + +

    The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is

    +$$ +\mu_n = \frac{1}{n} \sum_{i=1}^n x_i +$$ + +

    and the mean-centered data \( \bar{X} = \{ \bar{x}_1, \ldots, \bar{x}_n \} \) takes the form

    +$$ +\bar{x}_i = x_i - \mu_n. +$$ + +

    When you are done with these steps, print out \( \mu_n \) to verify it is +close to \( \mu \) and plot your mean centered data to verify it is +centered at the origin! +The following code elements perform these operations using pandas or using our own functionality for doing so. The latter, using numpy is rather simple through the mean() function. +

    + + +
    +
    +
    +
    +
    + 
    df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Scaling

    +

    Alternatively, we could use the functions we discussed +earlier for scaling the data set. That is, we could have used the +StandardScaler function in Scikit-Learn, a function which ensures +that for each feature/predictor we study the mean value is zero and +the variance is one (every column in the design/feature matrix). You +would then not get the same results, since we divide by the +variance. The diagonal covariance matrix elements will then be one, +while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our +specific case. +

    + + +

    Centered Data

    + +

    Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation

    +$$ +\begin{equation*} +\Sigma_n = \frac{1}{n-1} \sum_{i=1}^n \bar{x}_i^T \bar{x}_i = \frac{1}{n-1} \sum_{i=1}^n (x_i - \mu_n)^T (x_i - \mu_n) +\end{equation*} +$$ + +

    where the data points \( x_i \in \mathbb{R}^p \) (here in this example \( p = 2 \)) are column vectors and \( x^T \) is the transpose of \( x \). +We can write our own code or simply use either the functionaly of numpy or that of pandas, as follows +

    + + +
    +
    +
    +
    +
    + 
    print(df.cov())
+print(np.cov(X_centered.T))
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Note that the way we define the covariance matrix here has a factor \( n-1 \) instead of \( n \). This is included in the cov() function by numpy and pandas. +Our own code here is not very elegant and asks for obvious improvements. It is tailored to this specific \( 2\times 2 \) covariance matrix. +

    + + +
    +
    +
    +
    +
    + 
    # extract the relevant columns from the centered design matrix of dim n x 2
+x = X_centered[:,0]
+y = X_centered[:,1]
+Cov = np.zeros((2,2))
+Cov[0,1] = np.sum(x.T@y)/(n-1.0)
+Cov[0,0] = np.sum(x.T@x)/(n-1.0)
+Cov[1,1] = np.sum(y.T@y)/(n-1.0)
+Cov[1,0]= Cov[0,1]
+print("Centered covariance using own code")
+print(Cov)
+plt.plot(x, y, 'x')
+plt.axis('equal')
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Exploring

    + +

    Depending on the number of points \( n \), we will get results that are close to the covariance values defined above. +The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed. +

    + + +

    Diagonalize the sample covariance matrix to obtain the principal components

    + +

    Now we are ready to solve for the principal components! To do so we +diagonalize the sample covariance matrix \( \Sigma \). We can use the +function np.linalg.eig to do so. It will return the eigenvalues and +eigenvectors of \( \Sigma \). Once we have these we can perform the +following tasks: +

    + +
      +
    • We compute the percentage of the total variance captured by the first principal component
      • +
    • We plot the mean centered data and lines along the first and second principal components
      • +
    • Then we project the mean centered data onto the first and second principal components, and plot the projected data.
      • +
    • Finally, we approximate the data as
      • +
    +$$ +\begin{equation*} +x_i \approx \tilde{x}_i = \mu_n + \langle x_i, v_0 \rangle v_0 +\end{equation*} +$$ + +

    where \( v_0 \) is the first principal component.

    + + +

    Collecting all Steps

    + +

    Collecting all these steps we can write our own PCA function and +compare this with the functionality included in Scikit-Learn. +

    + +

    The code here outlines some of the elements we could include in the +analysis. Feel free to extend upon this in order to address the above +questions. +

    + + + +
    +
    +
    +
    +
    + 
    # diagonalize and obtain eigenvalues, not necessarily sorted
+EigValues, EigVectors = np.linalg.eig(Cov)
+# sort eigenvectors and eigenvalues
+#permute = EigValues.argsort()
+#EigValues = EigValues[permute]
+#EigVectors = EigVectors[:,permute]
+print("Eigenvalues of Covariance matrix")
+for i in range(2):
+    print(EigValues[i])
+FirstEigvector = EigVectors[:,0]
+SecondEigvector = EigVectors[:,1]
+print("First eigenvector")
+print(FirstEigvector)
+print("Second eigenvector")
+print(SecondEigvector)
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2Dsl = pca.fit_transform(X)
+print("Eigenvector of largest eigenvalue")
+print(pca.components_.T[:, 0])
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    This code does not contain all the above elements, but it shows how we can use Scikit-Learn to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then?

    + + +

    Classical PCA Theorem

    + +

    We assume now that we have a design matrix \( \boldsymbol{X} \) which has been +centered as discussed above. For the sake of simplicity we skip the +overline symbol. The matrix is defined in terms of the various column +vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension +\( \boldsymbol{x}\in {\mathbb{R}}^{n} \). +

    + +

    The PCA theorem states that minimizing the above reconstruction error +corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which +diagonalizes the empirical covariance(correlation) matrix. The optimal +low-dimensional encoding of the data is then given by a set of vectors +\( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the +orthogonal projection of the data onto the columns spanned by the +eigenvectors of the covariance(correlations matrix). +

    + + +

    The PCA Theorem

    + +

    To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as

    + +

    We are almost there, we have obtained a relation between minimizing +the reconstruction error and the variance and the covariance +matrix. Minimizing the error is equivalent to maximizing the variance +of the projected data. +

    + +

    We could trivially maximize the variance of the projection (and +thereby minimize the error in the reconstruction function) by letting +the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we +want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by +\( \vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a +Lagrange multiplier we can then in turn maximize +

    + +$$ +J(\boldsymbol{w}_0)= \boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0+\lambda_0(1-\boldsymbol{w}_0^T\boldsymbol{w}_0). +$$ + +

    Taking the derivative with respect to \( \boldsymbol{w}_0 \) we obtain

    + +$$ +\frac{\partial J(\boldsymbol{w}_0)}{\partial \boldsymbol{w}_0}= 2\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0-2\lambda_0\boldsymbol{w}_0=0, +$$ + +

    meaning that

    +$$ +\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0\boldsymbol{w}_0. +$$ + +

    The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is

    +$$ +\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0. +$$ + +

    If we want to maximize the variance (minimize the construction error) +we simply pick the eigenvector of the covariance matrix with the +largest eigenvalue. This establishes the link between the minimization +of the reconstruction function \( J \) in terms of an orthogonal matrix +and the maximization of the variance and thereby the covariance of our +observations encoded in the design/feature matrix \( \boldsymbol{X} \). +

    + +

    The proof +for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) can be +established by applying the above arguments and using the fact that +our basis of eigenvectors is orthogonal, see Murphy chapter +12.2. The +discussion in chapter 12.2 of Murphy's text has also a nice link with +the Singular Value Decomposition theorem. For categorical data, see +chapter 12.4 and discussion therein. +

    + +

    For more details, see for example Vidal, Ma and Sastry, chapter 2.

    + + + + +

    For a detailed demonstration of the geometric interpretation, see Vidal, Ma and Sastry, section 2.1.2.

    + +

    Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm. +First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it. +

    + +

    The following Python code uses NumPy’s svd() function to obtain all the principal components of the +training set, then extracts the first two principal components. First we center the data using either pandas or our own code +

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 vanilla matrix 
+rows = 10
+cols = 5
+X = np.random.randn(rows,cols)
+df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+display(df)
+
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+# Then check the difference between pandas and our own set up
+print(X_centered-df)
+#Now we do an SVD
+U, s, V = np.linalg.svd(X_centered)
+c1 = V.T[:, 0]
+c2 = V.T[:, 1]
+W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+print(X2D)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering +the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t +forget to center the data first. +

    + +

    Once you have identified all the principal components, you can reduce the dimensionality of the dataset +down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components. +Selecting this hyperplane ensures that the projection will preserve as much variance as possible. +

    + + +
    +
    +
    +
    +
    + 
    W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    PCA and scikit-learn

    + +

    Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The +following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note +that it automatically takes care of centering the data): +

    + + +
    +
    +
    +
    +
    + 
    #thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2D = pca.fit_transform(X)
+print(X2D)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    After fitting the PCA transformer to the dataset, you can access the principal components using the +components variable (note that it contains the PCs as horizontal vectors, so, for example, the first +principal component is equal to +

    + + +
    +
    +
    +
    +
    + 
    pca.components_.T[:, 0]
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Another very useful piece of information is the explained variance ratio of each principal component, +available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s +variance that lies along the axis of each principal component. +

    + + +

    Back to the Cancer Data

    +

    We can now repeat the above but applied to real data, in this case our breast cancer data. +Here we compute performance scores on the training data using logistic regression. +

    + + +
    +
    +
    +
    +
    + 
    import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import  train_test_split 
+from sklearn.datasets import load_breast_cancer
+from sklearn.linear_model import LogisticRegression
+cancer = load_breast_cancer()
+
+X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
+
+logreg = LogisticRegression()
+logreg.fit(X_train, y_train)
+print("Train set accuracy from Logistic Regression: {:.2f}".format(logreg.score(X_train,y_train)))
+# We scale the data
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+# Then perform again a log reg fit
+logreg.fit(X_train_scaled, y_train)
+print("Train set accuracy scaled data: {:.2f}".format(logreg.score(X_train_scaled,y_train)))
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2D_train = pca.fit_transform(X_train_scaled)
+# and finally compute the log reg fit and the score on the training data	
+logreg.fit(X2D_train,y_train)
+print("Train set accuracy scaled and PCA data: {:.2f}".format(logreg.score(X2D_train,y_train)))
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.

    + +

    Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to +choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%). +Unless, of course, you are reducing dimensionality for data visualization — in that case you will +generally want to reduce the dimensionality down to 2 or 3. +The following code computes PCA without reducing dimensionality, then computes the minimum number +of dimensions required to preserve 95% of the training set’s variance: +

    + + +
    +
    +
    +
    +
    + 
    pca = PCA()
+pca.fit(X)
+cumsum = np.cumsum(pca.explained_variance_ratio_)
+d = np.argmax(cumsum >= 0.95) + 1
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead +of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be +a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve: +

    + + +
    +
    +
    +
    +
    + 
    pca = PCA(n_components=0.95)
+X_reduced = pca.fit_transform(X)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Incremental PCA

    + +

    One problem with the preceding implementation of PCA is that it requires the whole training set to fit in +memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have +been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch +at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new +instances arrive). +

    +

    Randomized PCA

    + +

    Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic +algorithm that quickly finds an approximation of the first d principal components. Its computational +complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the +previous algorithms when \( d \) is much smaller than \( n \). +

    +

    Kernel PCA

    + +

    The kernel trick is a mathematical technique that implicitly maps instances into a +very high-dimensional space (called the feature space), enabling nonlinear classification and regression +with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature +space corresponds to a complex nonlinear decision boundary in the original space. +It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear +projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at +preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a +twisted manifold. +For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an +

    + + +
    +
    +
    +
    +
    + 
    from sklearn.decomposition import KernelPCA
+rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04)
+X_reduced = rbf_pca.fit_transform(X)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + + +

    Other techniques

    + +

    There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.

    + +

    Here are some of the most popular:

    +
      +
    • Multidimensional Scaling (MDS) reduces dimensionality while trying to preserve the distances between the instances.
      • +
    • Isomap creates a graph by connecting each instance to its nearest neighbors, then reduces dimensionality while trying to preserve the geodesic distances between the instances.
      • +
    • t-Distributed Stochastic Neighbor Embedding (t-SNE) reduces dimensionality while trying to keep similar instances close and dissimilar instances apart. It is mostly used for visualization, in particular to visualize clusters of instances in high-dimensional space (e.g., to visualize the MNIST images in 2D).
      • +
    • Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it learns the most discriminative axes between the classes, and these axes can then be used to define a hyperplane onto which to project the data. The benefit is that the projection will keep classes as far apart as possible, so LDA is a good technique to reduce dimensionality before running another classification algorithm such as a Support Vector Machine (SVM) classifier discussed in the SVM lectures.
      • +
    + +
    + + + + +
    + © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license +
    + + + diff --git a/doc/src/week43/week43-reveal.html b/doc/src/week43/week43-reveal.html new file mode 100644 index 000000000..b980ae5d4 --- /dev/null +++ b/doc/src/week43/week43-reveal.html @@ -0,0 +1,3604 @@ + + + + + + + + +Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    +
    + + + + + + + + + + + +
    + +
    +

    Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis

    +
    + + +
    +Morten Hjorth-Jensen [1, 2] +
    + +
    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 2, 2021

    +
    +
    + + +
    + © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license +
    +
    + +
    +

    Plans for week 43

    + +
      +

    • Thursday: Summary of Convolutional Neural Networks from week 42 and Recurrent Neural Networks
      • + +

      +

    • Friday: Recurrent Neural Networks and other Deep Learning methods such as Generalized Adversarial Neural Networks. Start discussing Principal component analysis
      • + +

      +

    +

    +

    +Excellent lectures on CNNs and RNNs +

    +

    +
    + + +
    +More resources +

    +

    +
    +
    + +
    +

    Reading Recommendations

    + +
      +

    • Goodfellow et al, chapter 10 on Recurrent NNs, chapters 11 and 12 on various practicalities around deep learning are also recommended.
      • +

    • Aurelien Geron, chapter 14 on RNNs.
      • +
    +
    + +
    +

    Summary on Deep Learning Methods

    + +

    We have studied fully connected neural networks (also called artifical nueral networks) and convolutional neural networks (CNNs).

    + +

    The first type of deep learning networks work very well on homogeneous and structured input data while CCNs are normally tailored to recognizing images.

    +
    + +
    +

    CNNs in brief

    + +

    In summary:

    + +
      +

    • A CNN architecture is in the simplest case a list of Layers that transform the image volume into an output volume (e.g. holding the class scores)
      • +

    • There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular)
      • +

    • Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function
      • +

    • Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don’t)
      • +

    • Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn’t)
      • +
    +

    +

    For more material on convolutional networks, we strongly recommend +the course +IN5400 – Machine Learning for Image Analysis +and the slides of CS231 which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs. +

    + +

    However, both standard feed forwards networks and CNNs perform well on data with unknown length.

    + +

    This is where recurrent nueral networks (RNNs) come to our rescue.

    +
    + +
    +

    Recurrent neural networks: Overarching view

    + +

    Till now our focus has been, including convolutional neural networks +as well, on feedforward neural networks. The output or the activations +flow only in one direction, from the input layer to the output layer. +

    + +

    A recurrent neural network (RNN) looks very much like a feedforward +neural network, except that it also has connections pointing +backward. +

    + +

    RNNs are used to analyze time series data such as stock prices, and +tell you when to buy or sell. In autonomous driving systems, they can +anticipate car trajectories and help avoid accidents. More generally, +they can work on sequences of arbitrary lengths, rather than on +fixed-sized inputs like all the nets we have discussed so far. For +example, they can take sentences, documents, or audio samples as +input, making them extremely useful for natural language processing +systems such as automatic translation and speech-to-text. +

    +
    + +
    +

    Set up of an RNN

    + +

    More to text to be added

    +
    + +
    +

    A simple example

    + + + +
    +
    +
    +
    +
    + 
    # Start importing packages
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import tensorflow as tf
+from tensorflow.keras import datasets, layers, models
+from tensorflow.keras.layers import Input
+from tensorflow.keras.models import Model, Sequential 
+from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU
+from tensorflow.keras import optimizers     
+from tensorflow.keras import regularizers           
+from tensorflow.keras.utils import to_categorical 
+
+
+
+# convert into dataset matrix
+def convertToMatrix(data, step):
+ X, Y =[], []
+ for i in range(len(data)-step):
+  d=i+step  
+  X.append(data[i:d,])
+  Y.append(data[d,])
+ return np.array(X), np.array(Y)
+
+step = 4
+N = 1000    
+Tp = 800    
+
+t=np.arange(0,N)
+x=np.sin(0.02*t)+2*np.random.rand(N)
+df = pd.DataFrame(x)
+df.head()
+
+plt.plot(df)
+plt.show()
+
+values=df.values
+train,test = values[0:Tp,:], values[Tp:N,:]
+
+# add step elements into train and test
+test = np.append(test,np.repeat(test[-1,],step))
+train = np.append(train,np.repeat(train[-1,],step))
+ 
+trainX,trainY =convertToMatrix(train,step)
+testX,testY =convertToMatrix(test,step)
+trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
+testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))
+
+model = Sequential()
+model.add(SimpleRNN(units=32, input_shape=(1,step), activation="relu"))
+model.add(Dense(8, activation="relu")) 
+model.add(Dense(1))
+model.compile(loss='mean_squared_error', optimizer='rmsprop')
+model.summary()
+
+model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)
+trainPredict = model.predict(trainX)
+testPredict= model.predict(testX)
+predicted=np.concatenate((trainPredict,testPredict),axis=0)
+
+trainScore = model.evaluate(trainX, trainY, verbose=0)
+print(trainScore)
+
+index = df.index.values
+plt.plot(index,df)
+plt.plot(index,predicted)
+plt.axvline(df.index[Tp], c="r")
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    An extrapolation example

    + +

    The following code provides an example of how recurrent neural +networks can be used to extrapolate to unknown values of physics data +sets. Specifically, the data sets used in this program come from +a quantum mechanical many-body calculation of energies as functions of the number of particles. +

    + + + +
    +
    +
    +
    +
    + 
    # For matrices and calculations
+import numpy as np
+# For machine learning (backend for keras)
+import tensorflow as tf
+# User-friendly machine learning library
+# Front end for TensorFlow
+import tensorflow.keras
+# Different methods from Keras needed to create an RNN
+# This is not necessary but it shortened function calls 
+# that need to be used in the code.
+from tensorflow.keras import datasets, layers, models
+from tensorflow.keras.layers import Input
+from tensorflow.keras import regularizers
+from tensorflow.keras.models import Model, Sequential
+from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU
+# For timing the code
+from timeit import default_timer as timer
+# For plotting
+import matplotlib.pyplot as plt
+
+
+# The data set
+datatype='VaryDimension'
+X_tot = np.arange(2, 42, 2)
+y_tot = np.array([-0.03077640549, -0.08336233266, -0.1446729567, -0.2116753732, -0.2830637392, -0.3581341341, -0.436462435, -0.5177783846,
+	-0.6019067271, -0.6887363571, -0.7782028952, -0.8702784034, -0.9649652536, -1.062292565, -1.16231451, 
+	-1.265109911, -1.370782966, -1.479465113, -1.591317992, -1.70653767])
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Formatting the Data

    + +

    The way the recurrent neural networks are trained in this program +differs from how machine learning algorithms are usually trained. +Typically a machine learning algorithm is trained by learning the +relationship between the x data and the y data. In this program, the +recurrent neural network will be trained to recognize the relationship +in a sequence of y values. This is type of data formatting is +typically used time series forcasting, but it can also be used in any +extrapolation (time series forecasting is just a specific type of +extrapolation along the time axis). This method of data formatting +does not use the x data and assumes that the y data are evenly spaced. +

    + +

    For a standard machine learning algorithm, the training data has the +form of (x,y) so the machine learning algorithm learns to assiciate a +y value with a given x value. This is useful when the test data has x +values within the same range as the training data. However, for this +application, the x values of the test data are outside of the x values +of the training data and the traditional method of training a machine +learning algorithm does not work as well. For this reason, the +recurrent neural network is trained on sequences of y values of the +form ((y1, y2), y3), so that the network is concerned with learning +the pattern of the y data and not the relation between the x and y +data. As long as the pattern of y data outside of the training region +stays relatively stable compared to what was inside the training +region, this method of training can produce accurate extrapolations to +y values far removed from the training data set. +

    + + + + + + + + + + +
    +
    +
    +
    +
    + 
    # FORMAT_DATA
+def format_data(data, length_of_sequence = 2):  
+    """
+        Inputs:
+            data(a numpy array): the data that will be the inputs to the recurrent neural
+                network
+            length_of_sequence (an int): the number of elements in one iteration of the
+                sequence patter.  For a function approximator use length_of_sequence = 2.
+        Returns:
+            rnn_input (a 3D numpy array): the input data for the recurrent neural network.  Its
+                dimensions are length of data - length of sequence, length of sequence, 
+                dimnsion of data
+            rnn_output (a numpy array): the training data for the neural network
+        Formats data to be used in a recurrent neural network.
+    """
+
+    X, Y = [], []
+    for i in range(len(data)-length_of_sequence):
+        # Get the next length_of_sequence elements
+        a = data[i:i+length_of_sequence]
+        # Get the element that immediately follows that
+        b = data[i+length_of_sequence]
+        # Reshape so that each data point is contained in its own array
+        a = np.reshape (a, (len(a), 1))
+        X.append(a)
+        Y.append(b)
+    rnn_input = np.array(X)
+    rnn_output = np.array(Y)
+
+    return rnn_input, rnn_output
+
+
+# ## Defining the Recurrent Neural Network Using Keras
+# 
+# The following method defines a simple recurrent neural network in keras consisting of one input layer, one hidden layer, and one output layer.
+
+def rnn(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with one hidden layer and returns the model.
+    """
+    # Number of neurons in the input and output layers
+    in_out_neurons = 1
+    # Number of neurons in the hidden layer
+    hidden_neurons = 200
+    # Define the input layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons))  
+    # Define the hidden layer as a simple RNN layer with a set number of neurons and add it to 
+    # the network immediately after the input layer
+    rnn = SimpleRNN(hidden_neurons, 
+                    return_sequences=False,
+                    stateful = stateful,
+                    name="RNN")(inp)
+    # Define the output layer as a dense neural network layer (standard neural network layer)
+    #and add it to the network immediately after the hidden layer.
+    dens = Dense(in_out_neurons,name="dense")(rnn)
+    # Create the machine learning model starting with the input layer and ending with the 
+    # output layer
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the machine learning model using the mean squared error function as the loss 
+    # function and an Adams optimizer.
+    model.compile(loss="mean_squared_error", optimizer="adam")  
+    return model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Predicting New Points With A Trained Recurrent Neural Network

    + + + +
    +
    +
    +
    +
    + 
    def test_rnn (x1, y_test, plot_min, plot_max):
+    """
+        Inputs:
+            x1 (a list or numpy array): The complete x component of the data set
+            y_test (a list or numpy array): The complete y component of the data set
+            plot_min (an int or float): the smallest x value used in the training data
+            plot_max (an int or float): the largest x valye used in the training data
+        Returns:
+            None.
+        Uses a trained recurrent neural network model to predict future points in the 
+        series.  Computes the MSE of the predicted data set from the true data set, saves
+        the predicted data set to a csv file, and plots the predicted and true data sets w
+        while also displaying the data range used for training.
+    """
+    # Add the training data as the first dim points in the predicted data array as these
+    # are known values.
+    y_pred = y_test[:dim].tolist()
+    # Generate the first input to the trained recurrent neural network using the last two 
+    # points of the training data.  Based on how the network was trained this means that it
+    # will predict the first point in the data set after the training data.  All of the 
+    # brackets are necessary for Tensorflow.
+    next_input = np.array([[[y_test[dim-2]], [y_test[dim-1]]]])
+    # Save the very last point in the training data set.  This will be used later.
+    last = [y_test[dim-1]]
+
+    # Iterate until the complete data set is created.
+    for i in range (dim, len(y_test)):
+        # Predict the next point in the data set using the previous two points.
+        next = model.predict(next_input)
+        # Append just the number of the predicted data set
+        y_pred.append(next[0][0])
+        # Create the input that will be used to predict the next data point in the data set.
+        next_input = np.array([[last, next[0]]], dtype=np.float64)
+        last = next
+
+    # Print the mean squared error between the known data set and the predicted data set.
+    print('MSE: ', np.square(np.subtract(y_test, y_pred)).mean())
+    # Save the predicted data set as a csv file for later use
+    name = datatype + 'Predicted'+str(dim)+'.csv'
+    np.savetxt(name, y_pred, delimiter=',')
+    # Plot the known data set and the predicted data set.  The red box represents the region that was used
+    # for the training data.
+    fig, ax = plt.subplots()
+    ax.plot(x1, y_test, label="true", linewidth=3)
+    ax.plot(x1, y_pred, 'g-.',label="predicted", linewidth=4)
+    ax.legend()
+    # Created a red region to represent the points used in the training data.
+    ax.axvspan(plot_min, plot_max, alpha=0.25, color='red')
+    plt.show()
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+
+# Generate the training data for the RNN, using a sequence of 2
+rnn_input, rnn_training = format_data(y_train, 2)
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+model = rnn(length_of_sequences = rnn_input.shape[1])
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict more points of the data set
+test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Other Things to Try

    + +

    Changing the size of the recurrent neural network and its parameters +can drastically change the results you get from the model. The below +code takes the simple recurrent neural network from above and adds a +second hidden layer, changes the number of neurons in the hidden +layer, and explicitly declares the activation function of the hidden +layers to be a sigmoid function. The loss function and optimizer can +also be changed but are kept the same as the above network. These +parameters can be tuned to provide the optimal result from the +network. For some ideas on how to improve the performance of a +recurrent neural network. +

    + + + +
    +
    +
    +
    +
    + 
    def rnn_2layers(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with two hidden layers and returns the model.
+    """
+    # Number of neurons in the input and output layers
+    in_out_neurons = 1
+    # Number of neurons in the hidden layer, increased from the first network
+    hidden_neurons = 500
+    # Define the input layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons))  
+    # Create two hidden layers instead of one hidden layer.  Explicitly set the activation
+    # function to be the sigmoid function (the default value is hyperbolic tangent)
+    rnn1 = SimpleRNN(hidden_neurons, 
+                    return_sequences=True,  # This needs to be True if another hidden layer is to follow
+                    stateful = stateful, activation = 'sigmoid',
+                    name="RNN1")(inp)
+    rnn2 = SimpleRNN(hidden_neurons, 
+                    return_sequences=False, activation = 'sigmoid',
+                    stateful = stateful,
+                    name="RNN2")(rnn1)
+    # Define the output layer as a dense neural network layer (standard neural network layer)
+    #and add it to the network immediately after the hidden layer.
+    dens = Dense(in_out_neurons,name="dense")(rnn2)
+    # Create the machine learning model starting with the input layer and ending with the 
+    # output layer
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the machine learning model using the mean squared error function as the loss 
+    # function and an Adams optimizer.
+    model.compile(loss="mean_squared_error", optimizer="adam")  
+    return model
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+
+# Generate the training data for the RNN, using a sequence of 2
+rnn_input, rnn_training = format_data(y_train, 2)
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+model = rnn_2layers(length_of_sequences = 2)
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+
+# This section plots the training loss and the validation loss as a function of training iteration.
+# This is not required for analyzing the couple cluster data but can help determine if the network is
+# being overtrained.
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict more points of the data set
+test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Other Types of Recurrent Neural Networks

    + +

    Besides a simple recurrent neural network layer, there are two other +commonly used types of recurrent neural network layers: Long Short +Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short +introduction to these layers see https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b +and https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b. +

    + +

    The first network created below is similar to the previous network, +but it replaces the SimpleRNN layers with LSTM layers. The second +network below has two hidden layers made up of GRUs, which are +preceeded by two dense (feeddorward) neural network layers. These +dense layers "preprocess" the data before it reaches the recurrent +layers. This architecture has been shown to improve the performance +of recurrent neural networks (see the link above and also +https://arxiv.org/pdf/1807.02857.pdf. +

    + + + +
    +
    +
    +
    +
    + 
    def lstm_2layers(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.
+    """
+    # Number of neurons on the input/output layer and the number of neurons in the hidden layer
+    in_out_neurons = 1
+    hidden_neurons = 250
+    # Input Layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons)) 
+    # Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)
+    rnn= LSTM(hidden_neurons, 
+                    return_sequences=True,
+                    stateful = stateful,
+                    name="RNN", use_bias=True, activation='tanh')(inp)
+    rnn1 = LSTM(hidden_neurons, 
+                    return_sequences=False,
+                    stateful = stateful,
+                    name="RNN1", use_bias=True, activation='tanh')(rnn)
+    # Output layer
+    dens = Dense(in_out_neurons,name="dense")(rnn1)
+    # Define the midel
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the model
+    model.compile(loss='mean_squared_error', optimizer='adam')  
+    # Return the model
+    return model
+
+def dnn2_gru2(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with four hidden layers (two dense followed by
+        two GRU layers) and returns the model.
+    """    
+    # Number of neurons on the input/output layers and hidden layers
+    in_out_neurons = 1
+    hidden_neurons = 250
+    # Input layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons)) 
+    # Hidden Dense (feedforward) layers
+    dnn = Dense(hidden_neurons/2, activation='relu', name='dnn')(inp)
+    dnn1 = Dense(hidden_neurons/2, activation='relu', name='dnn1')(dnn)
+    # Hidden GRU layers
+    rnn1 = GRU(hidden_neurons, 
+                    return_sequences=True,
+                    stateful = stateful,
+                    name="RNN1", use_bias=True)(dnn1)
+    rnn = GRU(hidden_neurons, 
+                    return_sequences=False,
+                    stateful = stateful,
+                    name="RNN", use_bias=True)(rnn1)
+    # Output layer
+    dens = Dense(in_out_neurons,name="dense")(rnn)
+    # Define the model
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the mdoel
+    model.compile(loss='mean_squared_error', optimizer='adam')  
+    # Return the model
+    return model
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+
+# Generate the training data for the RNN, using a sequence of 2
+rnn_input, rnn_training = format_data(y_train, 2)
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+# Change the method name to reflect which network you want to use
+model = dnn2_gru2(length_of_sequences = 2)
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+
+# This section plots the training loss and the validation loss as a function of training iteration.
+# This is not required for analyzing the couple cluster data but can help determine if the network is
+# being overtrained.
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict more points of the data set
+test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
+
+# ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)
+# 
+# Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+# Reshape the data for Keras specifications
+X_train = X_train.reshape((dim, 1))
+y_train = y_train.reshape((dim, 1))
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+# Set the sequence length to 1 for regular data formatting 
+model = rnn(length_of_sequences = 1)
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(X_train, y_train, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+
+# This section plots the training loss and the validation loss as a function of training iteration.
+# This is not required for analyzing the couple cluster data but can help determine if the network is
+# being overtrained.
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict the remaining data points
+X_pred = X_tot[dim:]
+X_pred = X_pred.reshape((len(X_pred), 1))
+y_model = model.predict(X_pred)
+y_pred = np.concatenate((y_tot[:dim], y_model.flatten()))
+
+# Plot the known data set and the predicted data set.  The red box represents the region that was used
+# for the training data.
+fig, ax = plt.subplots()
+ax.plot(X_tot, y_tot, label="true", linewidth=3)
+ax.plot(X_tot, y_pred, 'g-.',label="predicted", linewidth=4)
+ax.legend()
+# Created a red region to represent the points used in the training data.
+ax.axvspan(X_tot[0], X_tot[dim], alpha=0.25, color='red')
+plt.show()
+
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Generative Models

    + +

    Generative models describe a class of statistical models that are a contrast +to discriminative models. Informally we say that generative models can +generate new data instances while discriminative models discriminate between +different kinds of data instances. A generative model could generate new photos +of animals that look like 'real' animals while a discriminative model could tell +a dog from a cat. More formally, given a data set \( x \) and a set of labels / +targets \( y \). Generative models capture the joint probability \( p(x, y) \), or +just \( p(x) \) if there are no labels, while discriminative models capture the +conditional probability \( p(y | x) \). Discriminative models generally try to draw +boundaries in the data space (often high dimensional), while generative models +try to model how data is placed throughout the space. +

    + +

    Note: this material is thanks to Linus Ekstrøm.

    +
    + +
    +

    Generative Adversarial Networks

    + +

    Generative Adversarial Networks are a type of unsupervised machine learning +algorithm proposed by Goodfellow et. al +in 2014 (short and good article). +

    + +

    The simplest formulation of +the model is based on a game theoretic approach, zero sum game, where we pit +two neural networks against one another. We define two rival networks, one +generator \( g \), and one discriminator \( d \). The generator directly produces +samples +

    +

     
    +$$ +\begin{equation} + x = g(z; \theta^{(g)}) +\tag{1} +\end{equation} +$$ +

     
    +

    + +
    +

    Discriminator

    +

    The discriminator attempts to distinguish between samples drawn from the +training data and samples drawn from the generator. In other words, it tries to +tell the difference between the fake data produced by \( g \) and the actual data +samples we want to do prediction on. The discriminator outputs a probability +value given by +

    + +

     
    +$$ +\begin{equation} + d(x; \theta^{(d)}) +\tag{2} +\end{equation} +$$ +

     
    + +

    indicating the probability that \( x \) is a real training example rather than a +fake sample the generator has generated. The simplest way to formulate the +learning process in a generative adversarial network is a zero-sum game, in +which a function +

    + +

     
    +$$ +\begin{equation} + v(\theta^{(g)}, \theta^{(d)}) +\tag{3} +\end{equation} +$$ +

     
    + +

    determines the reward for the discriminator, while the generator gets the +conjugate reward +

    + +

     
    +$$ +\begin{equation} + -v(\theta^{(g)}, \theta^{(d)}) +\tag{4} +\end{equation} +$$ +

     
    +

    + +
    +

    Learning Process

    + +

    During learning both of the networks maximize their own reward function, so that +the generator gets better and better at tricking the discriminator, while the +discriminator gets better and better at telling the difference between the fake +and real data. The generator and discriminator alternate on which one trains at +one time (i.e. for one epoch). In other words, we keep the generator constant +and train the discriminator, then we keep the discriminator constant to train +the generator and repeat. It is this back and forth dynamic which lets GANs +tackle otherwise intractable generative problems. As the generator improves with + training, the discriminator's performance gets worse because it cannot easily + tell the difference between real and fake. If the generator ends up succeeding + perfectly, the the discriminator will do no better than random guessing i.e. + 50\%. This progression in the training poses a problem for the convergence + criteria for GANs. The discriminator feedback gets less meaningful over time, + if we continue training after this point then the generator is effectively + training on junk data which can undo the learning up to that point. Therefore, + we stop training when the discriminator starts outputting \( 1/2 \) everywhere. +

    +
    + +
    +

    More about the Learning Process

    + +

    At convergence we have

    + +

     
    +$$ +\begin{equation} + g^* = \underset{g}{\mathrm{argmin}}\hspace{2pt} + \underset{d}{\mathrm{max}}v(\theta^{(g)}, \theta^{(d)}) +\tag{5} +\end{equation} +$$ +

     
    + +

    The default choice for \( v \) is

    +

     
    +$$ +\begin{equation} + v(\theta^{(g)}, \theta^{(d)}) = \mathbb{E}_{x\sim p_\mathrm{data}}\log d(x) + + \mathbb{E}_{x\sim p_\mathrm{model}} + \log (1 - d(x)) +\tag{6} +\end{equation} +$$ +

     
    + +

    The main motivation for the design of GANs is that the learning process requires +neither approximate inference (variational autoencoders for example) nor +approximation of a partition function. In the case where +

    +

     
    +$$ +\begin{equation} + \underset{d}{\mathrm{max}}v(\theta^{(g)}, \theta^{(d)}) +\tag{7} +\end{equation} +$$ +

     
    + +

    is convex in $\theta^{(g)} then the procedure is guaranteed to converge and is +asymptotically consistent +( Seth Lloyd on QuGANs ). +

    +
    + +
    +

    Additional References

    +

    This is in +general not the case and it is possible to get situations where the training +process never converges because the generator and discriminator chase one +another around in the parameter space indefinitely. A much deeper discussion on +the currently open research problem of GAN convergence is available +here. To +anyone interested in learning more about GANs it is a highly recommended read. +Direct quote: "In this best-performing formulation, the generator aims to +increase the log probability that the discriminator makes a mistake, rather than +aiming to decrease the log probability that the discriminator makes the correct +prediction." Another interesting read +

    +
    + +
    +

    Writing Our First Generative Adversarial Network

    +

    Let us now move on to actually implementing a GAN in tensorflow. We will study +the performance of our GAN on the MNIST dataset. This code is based on and +adapted from the +google tutorial +

    + +

    First we import our libraries

    + + + +
    +
    +
    +
    +
    + 
    import os
+import time
+import numpy as np
+import tensorflow as tf
+import matplotlib.pyplot as plt
+from tensorflow.keras import layers
+from tensorflow.keras.utils import plot_model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we define our hyperparameters and import our data the usual way

    + + + +
    +
    +
    +
    +
    + 
    BUFFER_SIZE = 60000
+BATCH_SIZE = 256
+EPOCHS = 30
+
+data = tf.keras.datasets.mnist.load_data()
+(train_images, train_labels), (test_images, test_labels) = data
+train_images = np.reshape(train_images, (train_images.shape[0],
+                                         28,
+                                         28,
+                                         1)).astype('float32')
+
+# we normalize between -1 and 1
+train_images = (train_images - 127.5) / 127.5
+training_dataset = tf.data.Dataset.from_tensor_slices(
+                      train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    MNIST and GANs

    + +

    Let's have a quick look

    + + + +
    +
    +
    +
    +
    + 
    plt.imshow(train_images[0], cmap='Greys')
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we define our two models. This is where the 'magic' happens. There are a +huge amount of possible formulations for both models. A lot of engineering and +trial and error can be done here to try to produce better performing models. For +more advanced GANs this is by far the step where you can 'make or break' a +model. +

    + +

    We start with the generator. As stated in the introductory text the generator +\( g \) upsamples from a random sample to the shape of what we want to predict. In +our case we are trying to predict MNIST images (\( 28\times 28 \) pixels). +

    + + + +
    +
    +
    +
    +
    + 
    def generator_model():
+    """
+    The generator uses upsampling layers tf.keras.layers.Conv2DTranspose() to
+    produce an image from a random seed. We start with a Dense layer taking this
+    random sample as an input and subsequently upsample through multiple
+    convolutional layers.
+    """
+
+    # we define our model
+    model = tf.keras.Sequential()
+
+
+    # adding our input layer. Dense means that every neuron is connected and
+    # the input shape is the shape of our random noise. The units need to match
+    # in some sense the upsampling strides to reach our desired output shape.
+    # we are using 100 random numbers as our seed
+    model.add(layers.Dense(units=7*7*BATCH_SIZE,
+                           use_bias=False,
+                           input_shape=(100, )))
+    # we normalize the output form the Dense layer
+    model.add(layers.BatchNormalization())
+    # and add an activation function to our 'layer'. LeakyReLU avoids vanishing
+    # gradient problem
+    model.add(layers.LeakyReLU())
+    model.add(layers.Reshape((7, 7, BATCH_SIZE)))
+    assert model.output_shape == (None, 7, 7, BATCH_SIZE)
+    # even though we just added four keras layers we think of everything above
+    # as 'one' layer
+
+    # next we add our upscaling convolutional layers
+    model.add(layers.Conv2DTranspose(filters=128,
+                                     kernel_size=(5, 5),
+                                     strides=(1, 1),
+                                     padding='same',
+                                     use_bias=False))
+    model.add(layers.BatchNormalization())
+    model.add(layers.LeakyReLU())
+    assert model.output_shape == (None, 7, 7, 128)
+
+    model.add(layers.Conv2DTranspose(filters=64,
+                                     kernel_size=(5, 5),
+                                     strides=(2, 2),
+                                     padding='same',
+                                     use_bias=False))
+    model.add(layers.BatchNormalization())
+    model.add(layers.LeakyReLU())
+    assert model.output_shape == (None, 14, 14, 64)
+
+    model.add(layers.Conv2DTranspose(filters=1,
+                                     kernel_size=(5, 5),
+                                     strides=(2, 2),
+                                     padding='same',
+                                     use_bias=False,
+                                     activation='tanh'))
+    assert model.output_shape == (None, 28, 28, 1)
+
+    return model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    And there we have our 'simple' generator model. Now we move on to defining our +discriminator model \( d \), which is a convolutional neural network based image +classifier. +

    + + + +
    +
    +
    +
    +
    + 
    def discriminator_model():
+    """
+    The discriminator is a convolutional neural network based image classifier
+    """
+
+    # we define our model
+    model = tf.keras.Sequential()
+    model.add(layers.Conv2D(filters=64,
+                            kernel_size=(5, 5),
+                            strides=(2, 2),
+                            padding='same',
+                            input_shape=[28, 28, 1]))
+    model.add(layers.LeakyReLU())
+    # adding a dropout layer as you do in conv-nets
+    model.add(layers.Dropout(0.3))
+
+
+    model.add(layers.Conv2D(filters=128,
+                            kernel_size=(5, 5),
+                            strides=(2, 2),
+                            padding='same'))
+    model.add(layers.LeakyReLU())
+    # adding a dropout layer as you do in conv-nets
+    model.add(layers.Dropout(0.3))
+
+    model.add(layers.Flatten())
+    model.add(layers.Dense(1))
+
+    return model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Other Models

    +

    Let us take a look at our models. Note: double click images for bigger view.

    + + + +
    +
    +
    +
    +
    + 
    generator = generator_model()
+plot_model(generator, show_shapes=True, rankdir='LR')
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    discriminator = discriminator_model()
+plot_model(discriminator, show_shapes=True, rankdir='LR')
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we need a few helper objects we will use in training

    + + + +
    +
    +
    +
    +
    + 
    cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)
+generator_optimizer = tf.keras.optimizers.Adam(1e-4)
+discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    The first object, cross_entropy is our loss function and the two others are +our optimizers. Notice we use the same learning rate for both \( g \) and \( d \). This +is because they need to improve their accuracy at approximately equal speeds to +get convergence (not necessarily exactly equal). Now we define our loss +functions +

    + + + +
    +
    +
    +
    +
    + 
    def generator_loss(fake_output):
+    loss = cross_entropy(tf.ones_like(fake_output), fake_output)
+
+    return loss
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    def discriminator_loss(real_output, fake_output):
+    real_loss = cross_entropy(tf.ones_like(real_output), real_output)
+    fake_loss = cross_entropy(tf.zeros_liks(fake_output), fake_output)
+    total_loss = real_loss + fake_loss
+
+    return total_loss
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we define a kind of seed to help us compare the learning process over +multiple training epochs. +

    + + + +
    +
    +
    +
    +
    + 
    noise_dimension = 100
+n_examples_to_generate = 16
+seed_images = tf.random.normal([n_examples_to_generate, noise_dimension])
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Training Step

    + +

    Now we have everything we need to define our training step, which we will apply +for every step in our training loop. Notice the @tf.function flag signifying +that the function is tensorflow 'compiled'. Removing this flag doubles the +computation time. +

    + + + +
    +
    +
    +
    +
    + 
    @tf.function
+def train_step(images):
+    noise = tf.random.normal([BATCH_SIZE, noise_dimension])
+
+    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:
+        generated_images = generator(noise, training=True)
+
+        real_output = discriminator(images, training=True)
+        fake_output = discriminator(generated_images, training=True)
+
+        gen_loss = generator_loss(fake_output)
+        disc_loss = discriminator_loss(real_output, fake_output)
+
+    gradients_of_generator = gen_tape.gradient(gen_loss,
+                                            generator.trainable_variables)
+    gradients_of_discriminator = disc_tape.gradient(disc_loss,
+                                            discriminator.trainable_variables)
+    generator_optimizer.apply_gradients(zip(gradients_of_generator,
+                                            generator.trainable_variables))
+    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator,
+                                            discriminator.trainable_variables))
+
+    return gen_loss, disc_loss
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we define a helper function to produce an output over our training epochs +to see the predictive progression of our generator model. Note: I am including +this code here, but comment it out in the training loop. +

    + + +
    +
    +
    +
    +
    + 
    def generate_and_save_images(model, epoch, test_input):
+    # we're making inferences here
+    predictions = model(test_input, training=False)
+
+    fig = plt.figure(figsize=(4, 4))
+
+    for i in range(predictions.shape[0]):
+        plt.subplot(4, 4, i+1)
+        plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')
+        plt.axis('off')
+
+    plt.savefig(f'./images_from_seed_images/image_at_epoch_{str(epoch).zfill(3)}.png')
+    plt.close()
+    #plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Checkpoints

    +

    Setting up checkpoints to periodically save our model during training so that +everything is not lost even if the program were to somehow terminate while +training. +

    + + + +
    +
    +
    +
    +
    + 
    # Setting up checkpoints to save model during training
+checkpoint_dir = './training_checkpoints'
+checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt')
+checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,
+                            discriminator_optimizer=discriminator_optimizer,
+                            generator=generator,
+                            discriminator=discriminator)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we define our training loop

    + + + +
    +
    +
    +
    +
    + 
    def train(dataset, epochs):
+    generator_loss_list = []
+    discriminator_loss_list = []
+
+    for epoch in range(epochs):
+        start = time.time()
+
+        for image_batch in dataset:
+            gen_loss, disc_loss = train_step(image_batch)
+            generator_loss_list.append(gen_loss.numpy())
+            discriminator_loss_list.append(disc_loss.numpy())
+
+        #generate_and_save_images(generator, epoch + 1, seed_images)
+
+        if (epoch + 1) % 15 == 0:
+            checkpoint.save(file_prefix=checkpoint_prefix)
+
+        print(f'Time for epoch {epoch} is {time.time() - start}')
+
+    #generate_and_save_images(generator, epochs, seed_images)
+
+    loss_file = './data/lossfile.txt'
+    with open(loss_file, 'w') as outfile:
+        outfile.write(str(generator_loss_list))
+        outfile.write('\n')
+        outfile.write('\n')
+        outfile.write(str(discriminator_loss_list))
+        outfile.write('\n')
+        outfile.write('\n')
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    To train simply call this function. Warning: this might take a long time so +there is a folder of a pretrained network already included in the repository. +

    + + + +
    +
    +
    +
    +
    + 
    train(train_dataset, EPOCHS)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    And here is the result of training our model for 100 epochs

    + + +

    + +

    Now to avoid having to train and everything, which will take a while depending +on your computer setup we now load in the model which produced the above gif. +

    + + + +
    +
    +
    +
    +
    + 
    checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))
+restored_generator = checkpoint.generator
+restored_discriminator = checkpoint.discriminator
+
+print(restored_generator)
+print(restored_discriminator)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Exploring the Latent Space

    + +

    We have successfully loaded in our latest model. Let us now play around a bit +and see what kind of things we can learn about this model. Our generator takes +an array of 100 numbers. One idea can be to try to systematically change our +input. Let us try and see what we get +

    + + + +
    +
    +
    +
    +
    + 
    def generate_latent_points(number=100, scale_means=1, scale_stds=1):
+    latent_dim = 100
+    means = scale_means * tf.linspace(-1, 1, num=latent_dim)
+    stds = scale_stds * tf.linspace(-1, 1, num=latent_dim)
+    latent_space_value_range = tf.random.normal([number, latent_dim],
+                                                means,
+                                                stds,
+                                                dtype=tf.float64)
+
+    return latent_space_value_range
+
+def generate_images(latent_points):
+    # notice we set training to false because we are making inferences
+    generated_images = restored_generator.predict(latent_points)
+
+    return generated_images
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    def plot_result(generated_images, number=100):
+    # obviously this assumes sqrt number is an int
+    fig, axs = plt.subplots(int(np.sqrt(number)), int(np.sqrt(number)),
+                            figsize=(10, 10))
+
+    for i in range(int(np.sqrt(number))):
+        for j in range(int(np.sqrt(number))):
+            axs[i, j].imshow(generated_images[i*j], cmap='Greys')
+            axs[i, j].axis('off')
+
+    plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    generated_images = generate_images(generate_latent_points())
+plot_result(generated_images)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Getting Results

    +

    We see that the generator generates images that look like MNIST +numbers: \( 1, 4, 7, 9 \). Let's try to tweak it a bit more to see if we are able +to generate a similar plot where we generate every MNIST number. Let us now try +to 'move' a bit around in the latent space. Note: decrease the plot number if +these following cells take too long to run on your computer. +

    + + + +
    +
    +
    +
    +
    + 
    plot_number = 225
+
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=5,
+                                                          scale_stds=1))
+plot_result(generated_images, number=plot_number)
+
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=-5,
+                                                          scale_stds=1))
+plot_result(generated_images, number=plot_number)
+
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=1,
+                                                          scale_stds=5))
+plot_result(generated_images, number=plot_number)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Again, we have found something interesting. Moving around using our means +takes us from digit to digit, while moving around using our standard +deviations seem to increase the number of different digits! In the last image +above, we can barely make out every MNIST digit. Let us make on last plot using +this information by upping the standard deviation of our Gaussian noises. +

    + + + +
    +
    +
    +
    +
    + 
    plot_number = 400
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=1,
+                                                          scale_stds=10))
+plot_result(generated_images, number=plot_number)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    A pretty cool result! We see that our generator indeed has learned a +distribution which qualitatively looks a whole lot like the MNIST dataset. +

    +
    + +
    +

    Interpolating Between MNIST Digits

    +

    Another interesting way to explore the latent space of our generator model is by +interpolating between the MNIST digits. This section is largely based on +this excellent blogpost +by Jason Brownlee. +

    + +

    So let us start by defining a function to interpolate between two points in the +latent space. +

    + + + +
    +
    +
    +
    +
    + 
    def interpolation(point_1, point_2, n_steps=10):
+    ratios = np.linspace(0, 1, num=n_steps)
+    vectors = []
+    for i, ratio in enumerate(ratios):
+        vectors.append(((1.0 - ratio) * point_1 + ratio * point_2))
+
+    return tf.stack(vectors)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we have all we need to do our interpolation analysis.

    + + + +
    +
    +
    +
    +
    + 
    plot_number = 100
+latent_points = generate_latent_points(number=plot_number)
+results = None
+for i in range(0, 2*np.sqrt(plot_number), 2):
+    interpolated = interpolation(latent_points[i], latent_points[i+1])
+    generated_images = generate_images(interpolated)
+
+    if results is None:
+        results = generated_images
+    else:
+        results = tf.stack((results, generated_images))
+
+plot_results(results, plot_number)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Basic ideas of the Principal Component Analysis (PCA)

    + +

    The principal component analysis deals with the problem of fitting a +low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than +the total dimension \( D \) of the problem at hand (our data +set). Mathematically it can be formulated as a statistical problem or +a geometric problem. In our discussion of the theorem for the +classical PCA, we will stay with a statistical approach. +Historically, the PCA was first formulated in a statistical setting in order to estimate the principal component of a multivariate random variable. +

    + +

    We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)

    +
      +

    • Each data point is determined by \( p \) extrinsic (measurement) variables
      • +

    • We may want to ask the following question: Are there fewer intrinsic variables (say \( d < < p \)) that still approximately describe the data?
      • +

    • If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.
      • +
    +

    +

    A good read is for example Vidal, Ma and Sastry.

    +
    + +
    +

    Introducing the Covariance and Correlation functions

    + +

    Before we discuss the PCA theorem, we need to remind ourselves about +the definition of the covariance and the correlation function. These are quantities +

    + +

    Suppose we have defined two vectors +\( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as +

    +

     
    +$$ +\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\ + \end{bmatrix}, +$$ +

     
    + +

    where for example

    +

     
    +$$ +\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). +$$ +

     
    + +

    With this definition and recalling that the variance is defined as

    +

     
    +$$ +\mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2, +$$ +

     
    + +

    we can rewrite the covariance matrix as

    +

     
    +$$ +\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\ + \end{bmatrix}. +$$ +

     
    +

    + +
    +

    More on the covariance

    +

    The covariance takes values between zero and infinity and may thus +lead to problems with loss of numerical precision for particularly +large values. It is common to scale the covariance matrix by +introducing instead the correlation matrix defined via the so-called +correlation function +

    + +

     
    +$$ +\mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}. +$$ +

     
    + +

    The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] +\in [-1,1] \). This avoids eventual problems with too large values. We +can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) +and \( \boldsymbol{y} \) as +

    + +

     
    +$$ +\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{corr}[\boldsymbol{y},\boldsymbol{x}] & 1 \\ + \end{bmatrix}, +$$ +

     
    + +

    In the above example this is the function we constructed using pandas.

    +
    + +
    +

    Reminding ourselves about Linear Regression

    +

    In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression +we defined the design/feature matrix \( \boldsymbol{X} \) as +

    + +

     
    +$$ +\boldsymbol{X}=\begin{bmatrix} +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\ +\dots & \dots & \dots & \dots \dots & \dots \\ +x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ +\end{bmatrix}, +$$ +

     
    + +

    with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the +entries \( n \) being the row elements. +We can rewrite the design/feature matrix in terms of its column vectors as +

    +

     
    +$$ +\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix}, +$$ +

     
    + +

    with a given vector

    +

     
    +$$ +\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. +$$ +

     
    +

    + +
    +

    Simple Example

    +

    With these definitions, we can now rewrite our \( 2\times 2 \) +correlation/covariance matrix in terms of a moe general design/feature +matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) +covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) +

    + +

     
    +$$ +\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} +\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +\mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_1] & \mathrm{var}[\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & \mathrm{var}[\boldsymbol{x}_{p-1}]\\ +\end{bmatrix}, +$$ +

     
    +

    + +
    +

    The Correlation Matrix

    + +

    and the correlation matrix

    +

     
    +$$ +\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} +1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +\mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +\mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\ +\end{bmatrix}, +$$ +

     
    +

    + +
    +

    Numpy Functionality

    + +

    The Numpy function np.cov calculates the covariance elements using +the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have +the exact mean values. The following simple function uses the +np.vstack function which takes each vector of dimension \( 1\times n \) +and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +

    + +

     
    +$$ +\boldsymbol{W}^T = \begin{bmatrix} x_0 & y_0 \\ + x_1 & y_1 \\ + x_2 & y_2\\ + \dots & \dots \\ + x_{n-2} & y_{n-2}\\ + x_{n-1} & y_{n-1} & + \end{bmatrix}, +$$ +

     
    + +

    which in turn is converted into into the \( 2\times 2 \) covariance matrix +\( \boldsymbol{C} \) via the Numpy function np.cov(). We note that we can also calculate +the mean value of each set of samples \( \boldsymbol{x} \) etc using the Numpy +function np.mean(x). We can also extract the eigenvalues of the +covariance matrix through the np.linalg.eig() function. +

    + + + +
    +
    +
    +
    +
    + 
    # Importing various packages
+import numpy as np
+n = 100
+x = np.random.normal(size=n)
+print(np.mean(x))
+y = 4+3*x+np.random.normal(size=n)
+print(np.mean(y))
+W = np.vstack((x, y))
+C = np.cov(W)
+print(C)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Correlation Matrix again

    + +

    The previous example can be converted into the correlation matrix by +simply scaling the matrix elements with the variances. We should also +subtract the mean values for each column. This leads to the following +code which sets up the correlations matrix for the previous example in +a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors). +

    + + + +
    +
    +
    +
    +
    + 
    import numpy as np
+n = 100
+# define two vectors                                                                                           
+x = np.random.random(size=n)
+y = 4+3*x+np.random.normal(size=n)
+#scaling the x and y vectors                                                                                   
+x = x - np.mean(x)
+y = y - np.mean(y)
+variance_x = np.sum(x@x)/n
+variance_y = np.sum(y@y)/n
+print(variance_x)
+print(variance_y)
+cov_xy = np.sum(x@y)/n
+cov_xx = np.sum(x@x)/n
+cov_yy = np.sum(y@y)/n
+C = np.zeros((2,2))
+C[0,0]= cov_xx/variance_x
+C[1,1]= cov_yy/variance_y
+C[0,1]= cov_xy/np.sqrt(variance_y*variance_x)
+C[1,0]= C[0,1]
+print(C)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We see that the matrix elements along the diagonal are one as they +should be and that the matrix is symmetric. Furthermore, diagonalizing +this matrix we easily see that it is a positive definite matrix. +

    + +

    The above procedure with numpy can be made more compact if we use pandas.

    +
    + +
    +

    Using Pandas

    + +

    We whow here how we can set up the correlation matrix using pandas, as done in this simple code

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import pandas as pd
+n = 10
+x = np.random.normal(size=n)
+x = x - np.mean(x)
+y = 4+3*x+np.random.normal(size=n)
+y = y - np.mean(y)
+X = (np.vstack((x, y))).T
+print(X)
+Xpd = pd.DataFrame(X)
+print(Xpd)
+correlation_matrix = Xpd.corr()
+print(correlation_matrix)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    And then the Franke Function

    + +

    We expand this model to the Franke function discussed above.

    + + + +
    +
    +
    +
    +
    + 
    # Common imports
+import numpy as np
+import pandas as pd
+
+
+def FrankeFunction(x,y):
+	term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
+	term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
+	term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
+	term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
+	return term1 + term2 + term3 + term4
+
+
+def create_X(x, y, n ):
+	if len(x.shape) > 1:
+		x = np.ravel(x)
+		y = np.ravel(y)
+
+	N = len(x)
+	l = int((n+1)*(n+2)/2)		# Number of elements in beta
+	X = np.ones((N,l))
+
+	for i in range(1,n+1):
+		q = int((i)*(i+1)/2)
+		for k in range(i+1):
+			X[:,q+k] = (x**(i-k))*(y**k)
+
+	return X
+
+
+# Making meshgrid of datapoints and compute Franke's function
+n = 4
+N = 100
+x = np.sort(np.random.uniform(0, 1, N))
+y = np.sort(np.random.uniform(0, 1, N))
+z = FrankeFunction(x, y)
+X = create_X(x, y, n=n)    
+
+Xpd = pd.DataFrame(X)
+# subtract the mean values and set up the covariance matrix
+Xpd = Xpd - Xpd.mean()
+covariance_matrix = Xpd.cov()
+print(covariance_matrix)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We note here that the covariance is zero for the first rows and +columns since all matrix elements in the design matrix were set to one +(we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept +and wee can simply +drop these elements and construct a correlation +matrix without them by centering our matrix elements by subtracting the mean of each column. +

    +
    + +
    +

    Lnks with the Design Matrix

    + +

    We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as

    +

     
    +$$ +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. +$$ +

     
    + +

    To see this let us simply look at a design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{2\times 2} \)

    +

     
    +$$ +\boldsymbol{X}=\begin{bmatrix} +x_{00} & x_{01}\\ +x_{10} & x_{11}\\ +\end{bmatrix}=\begin{bmatrix} +\boldsymbol{x}_{0} & \boldsymbol{x}_{1}\\ +\end{bmatrix}. +$$ +

     
    +

    + +
    +

    Computing the Expectation Values

    + +

    If we then compute the expectation value

    +

     
    +$$ +\mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\begin{bmatrix} +x_{00}^2+x_{01}^2 & x_{00}x_{10}+x_{01}x_{11}\\ +x_{10}x_{00}+x_{11}x_{01} & x_{10}^2+x_{11}^2\\ +\end{bmatrix}, +$$ +

     
    + +

    which is just

    +

     
    +$$ +\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]=\begin{bmatrix} \mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] \\ + \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] \\ + \end{bmatrix}, +$$ +

     
    + +

    where we wrote

     
    +$$\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]$$ +

     
    to indicate that this the covariance of the vectors \( \boldsymbol{x} \) of the design/feature matrix \( \boldsymbol{X} \).

    + +

    It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \).

    +
    + +
    +

    Towards the PCA theorem

    + +

    We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as

    +

     
    +$$ +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. +$$ +

     
    + +

    Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices \( \boldsymbol{S} \). +These matrices are defined as \( \boldsymbol{S}\in {\mathbb{R}}^{p\times p} \) and obey the orthogonality requirements \( \boldsymbol{S}\boldsymbol{S}^T=\boldsymbol{S}^T\boldsymbol{S}=\boldsymbol{I} \). The matrix can be written out in terms of the column vectors \( \boldsymbol{s}_i \) as \( \boldsymbol{S}=[\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \) and \( \boldsymbol{s}_i \in {\mathbb{R}}^{p} \). +

    + +

    Assume also that there is a transformation \( \boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}=\boldsymbol{C}[\boldsymbol{y}] \) such that the new matrix \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal with elements \( [\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}] \).

    + +

    That is we have

    +

     
    +$$ +\boldsymbol{C}[\boldsymbol{y}] = \mathbb{E}[\boldsymbol{S}^T\boldsymbol{X}^T\boldsymbol{X}T\boldsymbol{S}]=\boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}, +$$ +

     
    + +

    since the matrix \( \boldsymbol{S} \) is not a data dependent matrix. Multiplying with \( \boldsymbol{S} \) from the left we have

    +

     
    +$$ +\boldsymbol{S}\boldsymbol{C}[\boldsymbol{y}] = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}, +$$ +

     
    + +

    and since \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal we have for a given eigenvalue \( i \) of the covariance matrix that

    + +

     
    +$$ +\boldsymbol{S}_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}_i. +$$ +

     
    +

    + +
    +

    More on the PCA Theorem

    + +

    In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is +\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \). +

    + +

    The eigenvalues tell us then how much we need to stretch the +corresponding eigenvectors. Dimensions with large eigenvalues have +thus large variations (large variance) and define therefore useful +dimensions. The data points are more spread out in the direction of +these eigenvectors. Smaller eigenvalues mean on the other hand that +the corresponding eigenvectors are shrunk accordingly and the data +points are tightly bunched together and there is not much variation in +these specific directions. Hopefully then we could leave it out +dimensions where the eigenvalues are very small. If \( p \) is very large, +we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) +features/predictors. +

    +
    + +
    +

    The Algorithm before theorem

    + +

    Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.

    +
      +

    • Set up the datapoints for the design/feature matrix \( \boldsymbol{X} \) with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) referring to the column numbers and the entries \( n \) being the row elements.
      • +
    +

    +

     
    +$$ +\boldsymbol{X}=\begin{bmatrix} +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\ +\dots & \dots & \dots & \dots \dots & \dots \\ +x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ +\end{bmatrix}, +$$ +

     
    + +

      +

    • Center the data by subtracting the mean value for each column. This leads to a new matrix \( \boldsymbol{X}\rightarrow \overline{\boldsymbol{X}} \).
      • +

    • Compute then the covariance/correlation matrix \( \mathbb{E}[\overline{\boldsymbol{X}}^T\overline{\boldsymbol{X}}] \).
      • +

    • Find the eigenpairs of \( \boldsymbol{C} \) with eigenvalues \( [\lambda_0,\lambda_1,\dots,\lambda_{p-1}] \) and eigenvectors \( [\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \).
      • +

    • Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.
      • +

    • Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.
      • +
    +
    + +
    +

    Writing our own PCA code

    + +

    We will use a simple example first with two-dimensional data +drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below): +

    +

     
    +$$ +\mu = (-1,2) \qquad \Sigma = \begin{bmatrix} 4 & 2 \\ +2 & 2 +\end{bmatrix} +$$ +

     
    + +

    Note that the mean refers to each column of data. +We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from +this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values. +

    +
    + +
    +

    Implementing it

    +

    The following Python code aids in setting up the data and writing out the design matrix. +Note that the function multivariate returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \). +

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from IPython.display import display
+n = 10000
+mean = (-1, 2)
+cov = [[4, 2], [2, 2]]
+X = np.random.multivariate_normal(mean, cov, n)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we are going to implement the PCA algorithm. We will break it down into various substeps.

    +
    + +
    +

    First Step

    + +

    The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is

    +

     
    +$$ +\mu_n = \frac{1}{n} \sum_{i=1}^n x_i +$$ +

     
    + +

    and the mean-centered data \( \bar{X} = \{ \bar{x}_1, \ldots, \bar{x}_n \} \) takes the form

    +

     
    +$$ +\bar{x}_i = x_i - \mu_n. +$$ +

     
    + +

    When you are done with these steps, print out \( \mu_n \) to verify it is +close to \( \mu \) and plot your mean centered data to verify it is +centered at the origin! +The following code elements perform these operations using pandas or using our own functionality for doing so. The latter, using numpy is rather simple through the mean() function. +

    + + +
    +
    +
    +
    +
    + 
    df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Scaling

    +

    Alternatively, we could use the functions we discussed +earlier for scaling the data set. That is, we could have used the +StandardScaler function in Scikit-Learn, a function which ensures +that for each feature/predictor we study the mean value is zero and +the variance is one (every column in the design/feature matrix). You +would then not get the same results, since we divide by the +variance. The diagonal covariance matrix elements will then be one, +while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our +specific case. +

    +
    + +
    +

    Centered Data

    + +

    Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation

    +

     
    +$$ +\begin{equation*} +\Sigma_n = \frac{1}{n-1} \sum_{i=1}^n \bar{x}_i^T \bar{x}_i = \frac{1}{n-1} \sum_{i=1}^n (x_i - \mu_n)^T (x_i - \mu_n) +\end{equation*} +$$ +

     
    + +

    where the data points \( x_i \in \mathbb{R}^p \) (here in this example \( p = 2 \)) are column vectors and \( x^T \) is the transpose of \( x \). +We can write our own code or simply use either the functionaly of numpy or that of pandas, as follows +

    + + +
    +
    +
    +
    +
    + 
    print(df.cov())
+print(np.cov(X_centered.T))
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Note that the way we define the covariance matrix here has a factor \( n-1 \) instead of \( n \). This is included in the cov() function by numpy and pandas. +Our own code here is not very elegant and asks for obvious improvements. It is tailored to this specific \( 2\times 2 \) covariance matrix. +

    + + +
    +
    +
    +
    +
    + 
    # extract the relevant columns from the centered design matrix of dim n x 2
+x = X_centered[:,0]
+y = X_centered[:,1]
+Cov = np.zeros((2,2))
+Cov[0,1] = np.sum(x.T@y)/(n-1.0)
+Cov[0,0] = np.sum(x.T@x)/(n-1.0)
+Cov[1,1] = np.sum(y.T@y)/(n-1.0)
+Cov[1,0]= Cov[0,1]
+print("Centered covariance using own code")
+print(Cov)
+plt.plot(x, y, 'x')
+plt.axis('equal')
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Exploring

    + +

    Depending on the number of points \( n \), we will get results that are close to the covariance values defined above. +The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed. +

    +
    + +
    +

    Diagonalize the sample covariance matrix to obtain the principal components

    + +

    Now we are ready to solve for the principal components! To do so we +diagonalize the sample covariance matrix \( \Sigma \). We can use the +function np.linalg.eig to do so. It will return the eigenvalues and +eigenvectors of \( \Sigma \). Once we have these we can perform the +following tasks: +

    + +
      +

    • We compute the percentage of the total variance captured by the first principal component
      • +

    • We plot the mean centered data and lines along the first and second principal components
      • +

    • Then we project the mean centered data onto the first and second principal components, and plot the projected data.
      • +

    • Finally, we approximate the data as
      • +
    +

    +

     
    +$$ +\begin{equation*} +x_i \approx \tilde{x}_i = \mu_n + \langle x_i, v_0 \rangle v_0 +\end{equation*} +$$ +

     
    + +

    where \( v_0 \) is the first principal component.

    +
    + +
    +

    Collecting all Steps

    + +

    Collecting all these steps we can write our own PCA function and +compare this with the functionality included in Scikit-Learn. +

    + +

    The code here outlines some of the elements we could include in the +analysis. Feel free to extend upon this in order to address the above +questions. +

    + + + +
    +
    +
    +
    +
    + 
    # diagonalize and obtain eigenvalues, not necessarily sorted
+EigValues, EigVectors = np.linalg.eig(Cov)
+# sort eigenvectors and eigenvalues
+#permute = EigValues.argsort()
+#EigValues = EigValues[permute]
+#EigVectors = EigVectors[:,permute]
+print("Eigenvalues of Covariance matrix")
+for i in range(2):
+    print(EigValues[i])
+FirstEigvector = EigVectors[:,0]
+SecondEigvector = EigVectors[:,1]
+print("First eigenvector")
+print(FirstEigvector)
+print("Second eigenvector")
+print(SecondEigvector)
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2Dsl = pca.fit_transform(X)
+print("Eigenvector of largest eigenvalue")
+print(pca.components_.T[:, 0])
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    This code does not contain all the above elements, but it shows how we can use Scikit-Learn to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then?

    +
    + +
    +

    Classical PCA Theorem

    + +

    We assume now that we have a design matrix \( \boldsymbol{X} \) which has been +centered as discussed above. For the sake of simplicity we skip the +overline symbol. The matrix is defined in terms of the various column +vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension +\( \boldsymbol{x}\in {\mathbb{R}}^{n} \). +

    + +

    The PCA theorem states that minimizing the above reconstruction error +corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which +diagonalizes the empirical covariance(correlation) matrix. The optimal +low-dimensional encoding of the data is then given by a set of vectors +\( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the +orthogonal projection of the data onto the columns spanned by the +eigenvectors of the covariance(correlations matrix). +

    +
    + +
    +

    The PCA Theorem

    + +

    To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as

    + +

    We are almost there, we have obtained a relation between minimizing +the reconstruction error and the variance and the covariance +matrix. Minimizing the error is equivalent to maximizing the variance +of the projected data. +

    + +

    We could trivially maximize the variance of the projection (and +thereby minimize the error in the reconstruction function) by letting +the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we +want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by +\( \vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a +Lagrange multiplier we can then in turn maximize +

    + +

     
    +$$ +J(\boldsymbol{w}_0)= \boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0+\lambda_0(1-\boldsymbol{w}_0^T\boldsymbol{w}_0). +$$ +

     
    + +

    Taking the derivative with respect to \( \boldsymbol{w}_0 \) we obtain

    + +

     
    +$$ +\frac{\partial J(\boldsymbol{w}_0)}{\partial \boldsymbol{w}_0}= 2\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0-2\lambda_0\boldsymbol{w}_0=0, +$$ +

     
    + +

    meaning that

    +

     
    +$$ +\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0\boldsymbol{w}_0. +$$ +

     
    + +

    The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is

    +

     
    +$$ +\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0. +$$ +

     
    + +

    If we want to maximize the variance (minimize the construction error) +we simply pick the eigenvector of the covariance matrix with the +largest eigenvalue. This establishes the link between the minimization +of the reconstruction function \( J \) in terms of an orthogonal matrix +and the maximization of the variance and thereby the covariance of our +observations encoded in the design/feature matrix \( \boldsymbol{X} \). +

    + +

    The proof +for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) can be +established by applying the above arguments and using the fact that +our basis of eigenvectors is orthogonal, see Murphy chapter +12.2. The +discussion in chapter 12.2 of Murphy's text has also a nice link with +the Singular Value Decomposition theorem. For categorical data, see +chapter 12.4 and discussion therein. +

    + +

    For more details, see for example Vidal, Ma and Sastry, chapter 2.

    +
    + +
    + + +

    For a detailed demonstration of the geometric interpretation, see Vidal, Ma and Sastry, section 2.1.2.

    + +

    Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm. +First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it. +

    + +

    The following Python code uses NumPy’s svd() function to obtain all the principal components of the +training set, then extracts the first two principal components. First we center the data using either pandas or our own code +

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 vanilla matrix 
+rows = 10
+cols = 5
+X = np.random.randn(rows,cols)
+df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+display(df)
+
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+# Then check the difference between pandas and our own set up
+print(X_centered-df)
+#Now we do an SVD
+U, s, V = np.linalg.svd(X_centered)
+c1 = V.T[:, 0]
+c2 = V.T[:, 1]
+W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+print(X2D)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering +the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t +forget to center the data first. +

    + +

    Once you have identified all the principal components, you can reduce the dimensionality of the dataset +down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components. +Selecting this hyperplane ensures that the projection will preserve as much variance as possible. +

    + + +
    +
    +
    +
    +
    + 
    W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    PCA and scikit-learn

    + +

    Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The +following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note +that it automatically takes care of centering the data): +

    + + +
    +
    +
    +
    +
    + 
    #thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2D = pca.fit_transform(X)
+print(X2D)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    After fitting the PCA transformer to the dataset, you can access the principal components using the +components variable (note that it contains the PCs as horizontal vectors, so, for example, the first +principal component is equal to +

    + + +
    +
    +
    +
    +
    + 
    pca.components_.T[:, 0]
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Another very useful piece of information is the explained variance ratio of each principal component, +available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s +variance that lies along the axis of each principal component. +

    +
    + +
    +

    Back to the Cancer Data

    +

    We can now repeat the above but applied to real data, in this case our breast cancer data. +Here we compute performance scores on the training data using logistic regression. +

    + + +
    +
    +
    +
    +
    + 
    import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import  train_test_split 
+from sklearn.datasets import load_breast_cancer
+from sklearn.linear_model import LogisticRegression
+cancer = load_breast_cancer()
+
+X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
+
+logreg = LogisticRegression()
+logreg.fit(X_train, y_train)
+print("Train set accuracy from Logistic Regression: {:.2f}".format(logreg.score(X_train,y_train)))
+# We scale the data
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+# Then perform again a log reg fit
+logreg.fit(X_train_scaled, y_train)
+print("Train set accuracy scaled data: {:.2f}".format(logreg.score(X_train_scaled,y_train)))
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2D_train = pca.fit_transform(X_train_scaled)
+# and finally compute the log reg fit and the score on the training data	
+logreg.fit(X2D_train,y_train)
+print("Train set accuracy scaled and PCA data: {:.2f}".format(logreg.score(X2D_train,y_train)))
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.

    + +

    Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to +choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%). +Unless, of course, you are reducing dimensionality for data visualization — in that case you will +generally want to reduce the dimensionality down to 2 or 3. +The following code computes PCA without reducing dimensionality, then computes the minimum number +of dimensions required to preserve 95% of the training set’s variance: +

    + + +
    +
    +
    +
    +
    + 
    pca = PCA()
+pca.fit(X)
+cumsum = np.cumsum(pca.explained_variance_ratio_)
+d = np.argmax(cumsum >= 0.95) + 1
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead +of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be +a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve: +

    + + +
    +
    +
    +
    +
    + 
    pca = PCA(n_components=0.95)
+X_reduced = pca.fit_transform(X)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Incremental PCA

    + +

    One problem with the preceding implementation of PCA is that it requires the whole training set to fit in +memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have +been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch +at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new +instances arrive). +

    +

    Randomized PCA

    + +

    Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic +algorithm that quickly finds an approximation of the first d principal components. Its computational +complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the +previous algorithms when \( d \) is much smaller than \( n \). +

    +

    Kernel PCA

    + +

    The kernel trick is a mathematical technique that implicitly maps instances into a +very high-dimensional space (called the feature space), enabling nonlinear classification and regression +with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature +space corresponds to a complex nonlinear decision boundary in the original space. +It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear +projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at +preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a +twisted manifold. +For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an +

    + + +
    +
    +
    +
    +
    + 
    from sklearn.decomposition import KernelPCA
+rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04)
+X_reduced = rbf_pca.fit_transform(X)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +

    Other techniques

    + +

    There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.

    + +

    Here are some of the most popular:

    +
      +

    • Multidimensional Scaling (MDS) reduces dimensionality while trying to preserve the distances between the instances.
      • +

    • Isomap creates a graph by connecting each instance to its nearest neighbors, then reduces dimensionality while trying to preserve the geodesic distances between the instances.
      • +

    • t-Distributed Stochastic Neighbor Embedding (t-SNE) reduces dimensionality while trying to keep similar instances close and dissimilar instances apart. It is mostly used for visualization, in particular to visualize clusters of instances in high-dimensional space (e.g., to visualize the MNIST images in 2D).
      • +

    • Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it learns the most discriminative axes between the classes, and these axes can then be used to define a hyperplane onto which to project the data. The benefit is that the projection will keep classes as far apart as possible, so LDA is a good technique to reduce dimensionality before running another classification algorithm such as a Support Vector Machine (SVM) classifier discussed in the SVM lectures.
      • +
    +
    + + + +
    +
    + + + + + + + + + + + + + diff --git a/doc/src/week43/week43-solarized.html b/doc/src/week43/week43-solarized.html new file mode 100644 index 000000000..730ceb964 --- /dev/null +++ b/doc/src/week43/week43-solarized.html @@ -0,0 +1,3374 @@ + + + + + + + +Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis + + + + + + + + + + + + + + + + + + +
    +

    Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis

    +
    + + +
    +Morten Hjorth-Jensen [1, 2] +
    + +
    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 2, 2021

    +
    +
    + +









    +

    Plans for week 43

    + +
      +
    • Thursday: Summary of Convolutional Neural Networks from week 42 and Recurrent Neural Networks
      • + +
    • Friday: Recurrent Neural Networks and other Deep Learning methods such as Generalized Adversarial Neural Networks. Start discussing Principal component analysis
      • + +
    +
    +Excellent lectures on CNNs and RNNs +

    +

    +
    + + +
    +More resources +

    +

    +
    + + +









    +

    Reading Recommendations

    + +
      +
    • Goodfellow et al, chapter 10 on Recurrent NNs, chapters 11 and 12 on various practicalities around deep learning are also recommended.
      • +
    • Aurelien Geron, chapter 14 on RNNs.
      • +
    +









    +

    Summary on Deep Learning Methods

    + +

    We have studied fully connected neural networks (also called artifical nueral networks) and convolutional neural networks (CNNs).

    + +

    The first type of deep learning networks work very well on homogeneous and structured input data while CCNs are normally tailored to recognizing images.

    + +









    +

    CNNs in brief

    + +

    In summary:

    + +
      +
    • A CNN architecture is in the simplest case a list of Layers that transform the image volume into an output volume (e.g. holding the class scores)
      • +
    • There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular)
      • +
    • Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function
      • +
    • Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don’t)
      • +
    • Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn’t)
      • +
    +

    For more material on convolutional networks, we strongly recommend +the course +IN5400 – Machine Learning for Image Analysis +and the slides of CS231 which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs. +

    + +

    However, both standard feed forwards networks and CNNs perform well on data with unknown length.

    + +

    This is where recurrent nueral networks (RNNs) come to our rescue.

    + +









    +

    Recurrent neural networks: Overarching view

    + +

    Till now our focus has been, including convolutional neural networks +as well, on feedforward neural networks. The output or the activations +flow only in one direction, from the input layer to the output layer. +

    + +

    A recurrent neural network (RNN) looks very much like a feedforward +neural network, except that it also has connections pointing +backward. +

    + +

    RNNs are used to analyze time series data such as stock prices, and +tell you when to buy or sell. In autonomous driving systems, they can +anticipate car trajectories and help avoid accidents. More generally, +they can work on sequences of arbitrary lengths, rather than on +fixed-sized inputs like all the nets we have discussed so far. For +example, they can take sentences, documents, or audio samples as +input, making them extremely useful for natural language processing +systems such as automatic translation and speech-to-text. +

    + +









    +

    Set up of an RNN

    + +

    More to text to be added

    + +









    +

    A simple example

    + + + +
    +
    +
    +
    +
    + 
    # Start importing packages
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import tensorflow as tf
+from tensorflow.keras import datasets, layers, models
+from tensorflow.keras.layers import Input
+from tensorflow.keras.models import Model, Sequential 
+from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU
+from tensorflow.keras import optimizers     
+from tensorflow.keras import regularizers           
+from tensorflow.keras.utils import to_categorical 
+
+
+
+# convert into dataset matrix
+def convertToMatrix(data, step):
+ X, Y =[], []
+ for i in range(len(data)-step):
+  d=i+step  
+  X.append(data[i:d,])
+  Y.append(data[d,])
+ return np.array(X), np.array(Y)
+
+step = 4
+N = 1000    
+Tp = 800    
+
+t=np.arange(0,N)
+x=np.sin(0.02*t)+2*np.random.rand(N)
+df = pd.DataFrame(x)
+df.head()
+
+plt.plot(df)
+plt.show()
+
+values=df.values
+train,test = values[0:Tp,:], values[Tp:N,:]
+
+# add step elements into train and test
+test = np.append(test,np.repeat(test[-1,],step))
+train = np.append(train,np.repeat(train[-1,],step))
+ 
+trainX,trainY =convertToMatrix(train,step)
+testX,testY =convertToMatrix(test,step)
+trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
+testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))
+
+model = Sequential()
+model.add(SimpleRNN(units=32, input_shape=(1,step), activation="relu"))
+model.add(Dense(8, activation="relu")) 
+model.add(Dense(1))
+model.compile(loss='mean_squared_error', optimizer='rmsprop')
+model.summary()
+
+model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)
+trainPredict = model.predict(trainX)
+testPredict= model.predict(testX)
+predicted=np.concatenate((trainPredict,testPredict),axis=0)
+
+trainScore = model.evaluate(trainX, trainY, verbose=0)
+print(trainScore)
+
+index = df.index.values
+plt.plot(index,df)
+plt.plot(index,predicted)
+plt.axvline(df.index[Tp], c="r")
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    An extrapolation example

    + +

    The following code provides an example of how recurrent neural +networks can be used to extrapolate to unknown values of physics data +sets. Specifically, the data sets used in this program come from +a quantum mechanical many-body calculation of energies as functions of the number of particles. +

    + + + +
    +
    +
    +
    +
    + 
    # For matrices and calculations
+import numpy as np
+# For machine learning (backend for keras)
+import tensorflow as tf
+# User-friendly machine learning library
+# Front end for TensorFlow
+import tensorflow.keras
+# Different methods from Keras needed to create an RNN
+# This is not necessary but it shortened function calls 
+# that need to be used in the code.
+from tensorflow.keras import datasets, layers, models
+from tensorflow.keras.layers import Input
+from tensorflow.keras import regularizers
+from tensorflow.keras.models import Model, Sequential
+from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU
+# For timing the code
+from timeit import default_timer as timer
+# For plotting
+import matplotlib.pyplot as plt
+
+
+# The data set
+datatype='VaryDimension'
+X_tot = np.arange(2, 42, 2)
+y_tot = np.array([-0.03077640549, -0.08336233266, -0.1446729567, -0.2116753732, -0.2830637392, -0.3581341341, -0.436462435, -0.5177783846,
+	-0.6019067271, -0.6887363571, -0.7782028952, -0.8702784034, -0.9649652536, -1.062292565, -1.16231451, 
+	-1.265109911, -1.370782966, -1.479465113, -1.591317992, -1.70653767])
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Formatting the Data

    + +

    The way the recurrent neural networks are trained in this program +differs from how machine learning algorithms are usually trained. +Typically a machine learning algorithm is trained by learning the +relationship between the x data and the y data. In this program, the +recurrent neural network will be trained to recognize the relationship +in a sequence of y values. This is type of data formatting is +typically used time series forcasting, but it can also be used in any +extrapolation (time series forecasting is just a specific type of +extrapolation along the time axis). This method of data formatting +does not use the x data and assumes that the y data are evenly spaced. +

    + +

    For a standard machine learning algorithm, the training data has the +form of (x,y) so the machine learning algorithm learns to assiciate a +y value with a given x value. This is useful when the test data has x +values within the same range as the training data. However, for this +application, the x values of the test data are outside of the x values +of the training data and the traditional method of training a machine +learning algorithm does not work as well. For this reason, the +recurrent neural network is trained on sequences of y values of the +form ((y1, y2), y3), so that the network is concerned with learning +the pattern of the y data and not the relation between the x and y +data. As long as the pattern of y data outside of the training region +stays relatively stable compared to what was inside the training +region, this method of training can produce accurate extrapolations to +y values far removed from the training data set. +

    + + + + + + + + + + +
    +
    +
    +
    +
    + 
    # FORMAT_DATA
+def format_data(data, length_of_sequence = 2):  
+    """
+        Inputs:
+            data(a numpy array): the data that will be the inputs to the recurrent neural
+                network
+            length_of_sequence (an int): the number of elements in one iteration of the
+                sequence patter.  For a function approximator use length_of_sequence = 2.
+        Returns:
+            rnn_input (a 3D numpy array): the input data for the recurrent neural network.  Its
+                dimensions are length of data - length of sequence, length of sequence, 
+                dimnsion of data
+            rnn_output (a numpy array): the training data for the neural network
+        Formats data to be used in a recurrent neural network.
+    """
+
+    X, Y = [], []
+    for i in range(len(data)-length_of_sequence):
+        # Get the next length_of_sequence elements
+        a = data[i:i+length_of_sequence]
+        # Get the element that immediately follows that
+        b = data[i+length_of_sequence]
+        # Reshape so that each data point is contained in its own array
+        a = np.reshape (a, (len(a), 1))
+        X.append(a)
+        Y.append(b)
+    rnn_input = np.array(X)
+    rnn_output = np.array(Y)
+
+    return rnn_input, rnn_output
+
+
+# ## Defining the Recurrent Neural Network Using Keras
+# 
+# The following method defines a simple recurrent neural network in keras consisting of one input layer, one hidden layer, and one output layer.
+
+def rnn(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with one hidden layer and returns the model.
+    """
+    # Number of neurons in the input and output layers
+    in_out_neurons = 1
+    # Number of neurons in the hidden layer
+    hidden_neurons = 200
+    # Define the input layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons))  
+    # Define the hidden layer as a simple RNN layer with a set number of neurons and add it to 
+    # the network immediately after the input layer
+    rnn = SimpleRNN(hidden_neurons, 
+                    return_sequences=False,
+                    stateful = stateful,
+                    name="RNN")(inp)
+    # Define the output layer as a dense neural network layer (standard neural network layer)
+    #and add it to the network immediately after the hidden layer.
+    dens = Dense(in_out_neurons,name="dense")(rnn)
+    # Create the machine learning model starting with the input layer and ending with the 
+    # output layer
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the machine learning model using the mean squared error function as the loss 
+    # function and an Adams optimizer.
+    model.compile(loss="mean_squared_error", optimizer="adam")  
+    return model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Predicting New Points With A Trained Recurrent Neural Network

    + + + +
    +
    +
    +
    +
    + 
    def test_rnn (x1, y_test, plot_min, plot_max):
+    """
+        Inputs:
+            x1 (a list or numpy array): The complete x component of the data set
+            y_test (a list or numpy array): The complete y component of the data set
+            plot_min (an int or float): the smallest x value used in the training data
+            plot_max (an int or float): the largest x valye used in the training data
+        Returns:
+            None.
+        Uses a trained recurrent neural network model to predict future points in the 
+        series.  Computes the MSE of the predicted data set from the true data set, saves
+        the predicted data set to a csv file, and plots the predicted and true data sets w
+        while also displaying the data range used for training.
+    """
+    # Add the training data as the first dim points in the predicted data array as these
+    # are known values.
+    y_pred = y_test[:dim].tolist()
+    # Generate the first input to the trained recurrent neural network using the last two 
+    # points of the training data.  Based on how the network was trained this means that it
+    # will predict the first point in the data set after the training data.  All of the 
+    # brackets are necessary for Tensorflow.
+    next_input = np.array([[[y_test[dim-2]], [y_test[dim-1]]]])
+    # Save the very last point in the training data set.  This will be used later.
+    last = [y_test[dim-1]]
+
+    # Iterate until the complete data set is created.
+    for i in range (dim, len(y_test)):
+        # Predict the next point in the data set using the previous two points.
+        next = model.predict(next_input)
+        # Append just the number of the predicted data set
+        y_pred.append(next[0][0])
+        # Create the input that will be used to predict the next data point in the data set.
+        next_input = np.array([[last, next[0]]], dtype=np.float64)
+        last = next
+
+    # Print the mean squared error between the known data set and the predicted data set.
+    print('MSE: ', np.square(np.subtract(y_test, y_pred)).mean())
+    # Save the predicted data set as a csv file for later use
+    name = datatype + 'Predicted'+str(dim)+'.csv'
+    np.savetxt(name, y_pred, delimiter=',')
+    # Plot the known data set and the predicted data set.  The red box represents the region that was used
+    # for the training data.
+    fig, ax = plt.subplots()
+    ax.plot(x1, y_test, label="true", linewidth=3)
+    ax.plot(x1, y_pred, 'g-.',label="predicted", linewidth=4)
+    ax.legend()
+    # Created a red region to represent the points used in the training data.
+    ax.axvspan(plot_min, plot_max, alpha=0.25, color='red')
+    plt.show()
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+
+# Generate the training data for the RNN, using a sequence of 2
+rnn_input, rnn_training = format_data(y_train, 2)
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+model = rnn(length_of_sequences = rnn_input.shape[1])
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict more points of the data set
+test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Other Things to Try

    + +

    Changing the size of the recurrent neural network and its parameters +can drastically change the results you get from the model. The below +code takes the simple recurrent neural network from above and adds a +second hidden layer, changes the number of neurons in the hidden +layer, and explicitly declares the activation function of the hidden +layers to be a sigmoid function. The loss function and optimizer can +also be changed but are kept the same as the above network. These +parameters can be tuned to provide the optimal result from the +network. For some ideas on how to improve the performance of a +recurrent neural network. +

    + + + +
    +
    +
    +
    +
    + 
    def rnn_2layers(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with two hidden layers and returns the model.
+    """
+    # Number of neurons in the input and output layers
+    in_out_neurons = 1
+    # Number of neurons in the hidden layer, increased from the first network
+    hidden_neurons = 500
+    # Define the input layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons))  
+    # Create two hidden layers instead of one hidden layer.  Explicitly set the activation
+    # function to be the sigmoid function (the default value is hyperbolic tangent)
+    rnn1 = SimpleRNN(hidden_neurons, 
+                    return_sequences=True,  # This needs to be True if another hidden layer is to follow
+                    stateful = stateful, activation = 'sigmoid',
+                    name="RNN1")(inp)
+    rnn2 = SimpleRNN(hidden_neurons, 
+                    return_sequences=False, activation = 'sigmoid',
+                    stateful = stateful,
+                    name="RNN2")(rnn1)
+    # Define the output layer as a dense neural network layer (standard neural network layer)
+    #and add it to the network immediately after the hidden layer.
+    dens = Dense(in_out_neurons,name="dense")(rnn2)
+    # Create the machine learning model starting with the input layer and ending with the 
+    # output layer
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the machine learning model using the mean squared error function as the loss 
+    # function and an Adams optimizer.
+    model.compile(loss="mean_squared_error", optimizer="adam")  
+    return model
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+
+# Generate the training data for the RNN, using a sequence of 2
+rnn_input, rnn_training = format_data(y_train, 2)
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+model = rnn_2layers(length_of_sequences = 2)
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+
+# This section plots the training loss and the validation loss as a function of training iteration.
+# This is not required for analyzing the couple cluster data but can help determine if the network is
+# being overtrained.
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict more points of the data set
+test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Other Types of Recurrent Neural Networks

    + +

    Besides a simple recurrent neural network layer, there are two other +commonly used types of recurrent neural network layers: Long Short +Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short +introduction to these layers see https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b +and https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b. +

    + +

    The first network created below is similar to the previous network, +but it replaces the SimpleRNN layers with LSTM layers. The second +network below has two hidden layers made up of GRUs, which are +preceeded by two dense (feeddorward) neural network layers. These +dense layers "preprocess" the data before it reaches the recurrent +layers. This architecture has been shown to improve the performance +of recurrent neural networks (see the link above and also +https://arxiv.org/pdf/1807.02857.pdf. +

    + + + +
    +
    +
    +
    +
    + 
    def lstm_2layers(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.
+    """
+    # Number of neurons on the input/output layer and the number of neurons in the hidden layer
+    in_out_neurons = 1
+    hidden_neurons = 250
+    # Input Layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons)) 
+    # Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)
+    rnn= LSTM(hidden_neurons, 
+                    return_sequences=True,
+                    stateful = stateful,
+                    name="RNN", use_bias=True, activation='tanh')(inp)
+    rnn1 = LSTM(hidden_neurons, 
+                    return_sequences=False,
+                    stateful = stateful,
+                    name="RNN1", use_bias=True, activation='tanh')(rnn)
+    # Output layer
+    dens = Dense(in_out_neurons,name="dense")(rnn1)
+    # Define the midel
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the model
+    model.compile(loss='mean_squared_error', optimizer='adam')  
+    # Return the model
+    return model
+
+def dnn2_gru2(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with four hidden layers (two dense followed by
+        two GRU layers) and returns the model.
+    """    
+    # Number of neurons on the input/output layers and hidden layers
+    in_out_neurons = 1
+    hidden_neurons = 250
+    # Input layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons)) 
+    # Hidden Dense (feedforward) layers
+    dnn = Dense(hidden_neurons/2, activation='relu', name='dnn')(inp)
+    dnn1 = Dense(hidden_neurons/2, activation='relu', name='dnn1')(dnn)
+    # Hidden GRU layers
+    rnn1 = GRU(hidden_neurons, 
+                    return_sequences=True,
+                    stateful = stateful,
+                    name="RNN1", use_bias=True)(dnn1)
+    rnn = GRU(hidden_neurons, 
+                    return_sequences=False,
+                    stateful = stateful,
+                    name="RNN", use_bias=True)(rnn1)
+    # Output layer
+    dens = Dense(in_out_neurons,name="dense")(rnn)
+    # Define the model
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the mdoel
+    model.compile(loss='mean_squared_error', optimizer='adam')  
+    # Return the model
+    return model
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+
+# Generate the training data for the RNN, using a sequence of 2
+rnn_input, rnn_training = format_data(y_train, 2)
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+# Change the method name to reflect which network you want to use
+model = dnn2_gru2(length_of_sequences = 2)
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+
+# This section plots the training loss and the validation loss as a function of training iteration.
+# This is not required for analyzing the couple cluster data but can help determine if the network is
+# being overtrained.
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict more points of the data set
+test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
+
+# ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)
+# 
+# Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+# Reshape the data for Keras specifications
+X_train = X_train.reshape((dim, 1))
+y_train = y_train.reshape((dim, 1))
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+# Set the sequence length to 1 for regular data formatting 
+model = rnn(length_of_sequences = 1)
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(X_train, y_train, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+
+# This section plots the training loss and the validation loss as a function of training iteration.
+# This is not required for analyzing the couple cluster data but can help determine if the network is
+# being overtrained.
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict the remaining data points
+X_pred = X_tot[dim:]
+X_pred = X_pred.reshape((len(X_pred), 1))
+y_model = model.predict(X_pred)
+y_pred = np.concatenate((y_tot[:dim], y_model.flatten()))
+
+# Plot the known data set and the predicted data set.  The red box represents the region that was used
+# for the training data.
+fig, ax = plt.subplots()
+ax.plot(X_tot, y_tot, label="true", linewidth=3)
+ax.plot(X_tot, y_pred, 'g-.',label="predicted", linewidth=4)
+ax.legend()
+# Created a red region to represent the points used in the training data.
+ax.axvspan(X_tot[0], X_tot[dim], alpha=0.25, color='red')
+plt.show()
+
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Generative Models

    + +

    Generative models describe a class of statistical models that are a contrast +to discriminative models. Informally we say that generative models can +generate new data instances while discriminative models discriminate between +different kinds of data instances. A generative model could generate new photos +of animals that look like 'real' animals while a discriminative model could tell +a dog from a cat. More formally, given a data set \( x \) and a set of labels / +targets \( y \). Generative models capture the joint probability \( p(x, y) \), or +just \( p(x) \) if there are no labels, while discriminative models capture the +conditional probability \( p(y | x) \). Discriminative models generally try to draw +boundaries in the data space (often high dimensional), while generative models +try to model how data is placed throughout the space. +

    + +

    Note: this material is thanks to Linus Ekstrøm.

    + +









    +

    Generative Adversarial Networks

    + +

    Generative Adversarial Networks are a type of unsupervised machine learning +algorithm proposed by Goodfellow et. al +in 2014 (short and good article). +

    + +

    The simplest formulation of +the model is based on a game theoretic approach, zero sum game, where we pit +two neural networks against one another. We define two rival networks, one +generator \( g \), and one discriminator \( d \). The generator directly produces +samples +

    +$$ +\begin{equation} + x = g(z; \theta^{(g)}) +\label{_auto1} +\end{equation} +$$ + + +









    +

    Discriminator

    +

    The discriminator attempts to distinguish between samples drawn from the +training data and samples drawn from the generator. In other words, it tries to +tell the difference between the fake data produced by \( g \) and the actual data +samples we want to do prediction on. The discriminator outputs a probability +value given by +

    + +$$ +\begin{equation} + d(x; \theta^{(d)}) +\label{_auto2} +\end{equation} +$$ + +

    indicating the probability that \( x \) is a real training example rather than a +fake sample the generator has generated. The simplest way to formulate the +learning process in a generative adversarial network is a zero-sum game, in +which a function +

    + +$$ +\begin{equation} + v(\theta^{(g)}, \theta^{(d)}) +\label{_auto3} +\end{equation} +$$ + +

    determines the reward for the discriminator, while the generator gets the +conjugate reward +

    + +$$ +\begin{equation} + -v(\theta^{(g)}, \theta^{(d)}) +\label{_auto4} +\end{equation} +$$ + + +









    +

    Learning Process

    + +

    During learning both of the networks maximize their own reward function, so that +the generator gets better and better at tricking the discriminator, while the +discriminator gets better and better at telling the difference between the fake +and real data. The generator and discriminator alternate on which one trains at +one time (i.e. for one epoch). In other words, we keep the generator constant +and train the discriminator, then we keep the discriminator constant to train +the generator and repeat. It is this back and forth dynamic which lets GANs +tackle otherwise intractable generative problems. As the generator improves with + training, the discriminator's performance gets worse because it cannot easily + tell the difference between real and fake. If the generator ends up succeeding + perfectly, the the discriminator will do no better than random guessing i.e. + 50\%. This progression in the training poses a problem for the convergence + criteria for GANs. The discriminator feedback gets less meaningful over time, + if we continue training after this point then the generator is effectively + training on junk data which can undo the learning up to that point. Therefore, + we stop training when the discriminator starts outputting \( 1/2 \) everywhere. +

    + +









    +

    More about the Learning Process

    + +

    At convergence we have

    + +$$ +\begin{equation} + g^* = \underset{g}{\mathrm{argmin}}\hspace{2pt} + \underset{d}{\mathrm{max}}v(\theta^{(g)}, \theta^{(d)}) +\label{_auto5} +\end{equation} +$$ + +

    The default choice for \( v \) is

    +$$ +\begin{equation} + v(\theta^{(g)}, \theta^{(d)}) = \mathbb{E}_{x\sim p_\mathrm{data}}\log d(x) + + \mathbb{E}_{x\sim p_\mathrm{model}} + \log (1 - d(x)) +\label{_auto6} +\end{equation} +$$ + +

    The main motivation for the design of GANs is that the learning process requires +neither approximate inference (variational autoencoders for example) nor +approximation of a partition function. In the case where +

    +$$ +\begin{equation} + \underset{d}{\mathrm{max}}v(\theta^{(g)}, \theta^{(d)}) +\label{_auto7} +\end{equation} +$$ + +

    is convex in $\theta^{(g)} then the procedure is guaranteed to converge and is +asymptotically consistent +( Seth Lloyd on QuGANs ). +

    + +









    +

    Additional References

    +

    This is in +general not the case and it is possible to get situations where the training +process never converges because the generator and discriminator chase one +another around in the parameter space indefinitely. A much deeper discussion on +the currently open research problem of GAN convergence is available +here. To +anyone interested in learning more about GANs it is a highly recommended read. +Direct quote: "In this best-performing formulation, the generator aims to +increase the log probability that the discriminator makes a mistake, rather than +aiming to decrease the log probability that the discriminator makes the correct +prediction." Another interesting read +

    + +









    +

    Writing Our First Generative Adversarial Network

    +

    Let us now move on to actually implementing a GAN in tensorflow. We will study +the performance of our GAN on the MNIST dataset. This code is based on and +adapted from the +google tutorial +

    + +

    First we import our libraries

    + + + +
    +
    +
    +
    +
    + 
    import os
+import time
+import numpy as np
+import tensorflow as tf
+import matplotlib.pyplot as plt
+from tensorflow.keras import layers
+from tensorflow.keras.utils import plot_model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we define our hyperparameters and import our data the usual way

    + + + +
    +
    +
    +
    +
    + 
    BUFFER_SIZE = 60000
+BATCH_SIZE = 256
+EPOCHS = 30
+
+data = tf.keras.datasets.mnist.load_data()
+(train_images, train_labels), (test_images, test_labels) = data
+train_images = np.reshape(train_images, (train_images.shape[0],
+                                         28,
+                                         28,
+                                         1)).astype('float32')
+
+# we normalize between -1 and 1
+train_images = (train_images - 127.5) / 127.5
+training_dataset = tf.data.Dataset.from_tensor_slices(
+                      train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    MNIST and GANs

    + +

    Let's have a quick look

    + + + +
    +
    +
    +
    +
    + 
    plt.imshow(train_images[0], cmap='Greys')
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we define our two models. This is where the 'magic' happens. There are a +huge amount of possible formulations for both models. A lot of engineering and +trial and error can be done here to try to produce better performing models. For +more advanced GANs this is by far the step where you can 'make or break' a +model. +

    + +

    We start with the generator. As stated in the introductory text the generator +\( g \) upsamples from a random sample to the shape of what we want to predict. In +our case we are trying to predict MNIST images (\( 28\times 28 \) pixels). +

    + + + +
    +
    +
    +
    +
    + 
    def generator_model():
+    """
+    The generator uses upsampling layers tf.keras.layers.Conv2DTranspose() to
+    produce an image from a random seed. We start with a Dense layer taking this
+    random sample as an input and subsequently upsample through multiple
+    convolutional layers.
+    """
+
+    # we define our model
+    model = tf.keras.Sequential()
+
+
+    # adding our input layer. Dense means that every neuron is connected and
+    # the input shape is the shape of our random noise. The units need to match
+    # in some sense the upsampling strides to reach our desired output shape.
+    # we are using 100 random numbers as our seed
+    model.add(layers.Dense(units=7*7*BATCH_SIZE,
+                           use_bias=False,
+                           input_shape=(100, )))
+    # we normalize the output form the Dense layer
+    model.add(layers.BatchNormalization())
+    # and add an activation function to our 'layer'. LeakyReLU avoids vanishing
+    # gradient problem
+    model.add(layers.LeakyReLU())
+    model.add(layers.Reshape((7, 7, BATCH_SIZE)))
+    assert model.output_shape == (None, 7, 7, BATCH_SIZE)
+    # even though we just added four keras layers we think of everything above
+    # as 'one' layer
+
+    # next we add our upscaling convolutional layers
+    model.add(layers.Conv2DTranspose(filters=128,
+                                     kernel_size=(5, 5),
+                                     strides=(1, 1),
+                                     padding='same',
+                                     use_bias=False))
+    model.add(layers.BatchNormalization())
+    model.add(layers.LeakyReLU())
+    assert model.output_shape == (None, 7, 7, 128)
+
+    model.add(layers.Conv2DTranspose(filters=64,
+                                     kernel_size=(5, 5),
+                                     strides=(2, 2),
+                                     padding='same',
+                                     use_bias=False))
+    model.add(layers.BatchNormalization())
+    model.add(layers.LeakyReLU())
+    assert model.output_shape == (None, 14, 14, 64)
+
+    model.add(layers.Conv2DTranspose(filters=1,
+                                     kernel_size=(5, 5),
+                                     strides=(2, 2),
+                                     padding='same',
+                                     use_bias=False,
+                                     activation='tanh'))
+    assert model.output_shape == (None, 28, 28, 1)
+
+    return model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    And there we have our 'simple' generator model. Now we move on to defining our +discriminator model \( d \), which is a convolutional neural network based image +classifier. +

    + + + +
    +
    +
    +
    +
    + 
    def discriminator_model():
+    """
+    The discriminator is a convolutional neural network based image classifier
+    """
+
+    # we define our model
+    model = tf.keras.Sequential()
+    model.add(layers.Conv2D(filters=64,
+                            kernel_size=(5, 5),
+                            strides=(2, 2),
+                            padding='same',
+                            input_shape=[28, 28, 1]))
+    model.add(layers.LeakyReLU())
+    # adding a dropout layer as you do in conv-nets
+    model.add(layers.Dropout(0.3))
+
+
+    model.add(layers.Conv2D(filters=128,
+                            kernel_size=(5, 5),
+                            strides=(2, 2),
+                            padding='same'))
+    model.add(layers.LeakyReLU())
+    # adding a dropout layer as you do in conv-nets
+    model.add(layers.Dropout(0.3))
+
+    model.add(layers.Flatten())
+    model.add(layers.Dense(1))
+
+    return model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Other Models

    +

    Let us take a look at our models. Note: double click images for bigger view.

    + + + +
    +
    +
    +
    +
    + 
    generator = generator_model()
+plot_model(generator, show_shapes=True, rankdir='LR')
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    discriminator = discriminator_model()
+plot_model(discriminator, show_shapes=True, rankdir='LR')
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we need a few helper objects we will use in training

    + + + +
    +
    +
    +
    +
    + 
    cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)
+generator_optimizer = tf.keras.optimizers.Adam(1e-4)
+discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    The first object, cross_entropy is our loss function and the two others are +our optimizers. Notice we use the same learning rate for both \( g \) and \( d \). This +is because they need to improve their accuracy at approximately equal speeds to +get convergence (not necessarily exactly equal). Now we define our loss +functions +

    + + + +
    +
    +
    +
    +
    + 
    def generator_loss(fake_output):
+    loss = cross_entropy(tf.ones_like(fake_output), fake_output)
+
+    return loss
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    def discriminator_loss(real_output, fake_output):
+    real_loss = cross_entropy(tf.ones_like(real_output), real_output)
+    fake_loss = cross_entropy(tf.zeros_liks(fake_output), fake_output)
+    total_loss = real_loss + fake_loss
+
+    return total_loss
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we define a kind of seed to help us compare the learning process over +multiple training epochs. +

    + + + +
    +
    +
    +
    +
    + 
    noise_dimension = 100
+n_examples_to_generate = 16
+seed_images = tf.random.normal([n_examples_to_generate, noise_dimension])
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Training Step

    + +

    Now we have everything we need to define our training step, which we will apply +for every step in our training loop. Notice the @tf.function flag signifying +that the function is tensorflow 'compiled'. Removing this flag doubles the +computation time. +

    + + + +
    +
    +
    +
    +
    + 
    @tf.function
+def train_step(images):
+    noise = tf.random.normal([BATCH_SIZE, noise_dimension])
+
+    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:
+        generated_images = generator(noise, training=True)
+
+        real_output = discriminator(images, training=True)
+        fake_output = discriminator(generated_images, training=True)
+
+        gen_loss = generator_loss(fake_output)
+        disc_loss = discriminator_loss(real_output, fake_output)
+
+    gradients_of_generator = gen_tape.gradient(gen_loss,
+                                            generator.trainable_variables)
+    gradients_of_discriminator = disc_tape.gradient(disc_loss,
+                                            discriminator.trainable_variables)
+    generator_optimizer.apply_gradients(zip(gradients_of_generator,
+                                            generator.trainable_variables))
+    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator,
+                                            discriminator.trainable_variables))
+
+    return gen_loss, disc_loss
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we define a helper function to produce an output over our training epochs +to see the predictive progression of our generator model. Note: I am including +this code here, but comment it out in the training loop. +

    + + +
    +
    +
    +
    +
    + 
    def generate_and_save_images(model, epoch, test_input):
+    # we're making inferences here
+    predictions = model(test_input, training=False)
+
+    fig = plt.figure(figsize=(4, 4))
+
+    for i in range(predictions.shape[0]):
+        plt.subplot(4, 4, i+1)
+        plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')
+        plt.axis('off')
+
+    plt.savefig(f'./images_from_seed_images/image_at_epoch_{str(epoch).zfill(3)}.png')
+    plt.close()
+    #plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Checkpoints

    +

    Setting up checkpoints to periodically save our model during training so that +everything is not lost even if the program were to somehow terminate while +training. +

    + + + +
    +
    +
    +
    +
    + 
    # Setting up checkpoints to save model during training
+checkpoint_dir = './training_checkpoints'
+checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt')
+checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,
+                            discriminator_optimizer=discriminator_optimizer,
+                            generator=generator,
+                            discriminator=discriminator)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we define our training loop

    + + + +
    +
    +
    +
    +
    + 
    def train(dataset, epochs):
+    generator_loss_list = []
+    discriminator_loss_list = []
+
+    for epoch in range(epochs):
+        start = time.time()
+
+        for image_batch in dataset:
+            gen_loss, disc_loss = train_step(image_batch)
+            generator_loss_list.append(gen_loss.numpy())
+            discriminator_loss_list.append(disc_loss.numpy())
+
+        #generate_and_save_images(generator, epoch + 1, seed_images)
+
+        if (epoch + 1) % 15 == 0:
+            checkpoint.save(file_prefix=checkpoint_prefix)
+
+        print(f'Time for epoch {epoch} is {time.time() - start}')
+
+    #generate_and_save_images(generator, epochs, seed_images)
+
+    loss_file = './data/lossfile.txt'
+    with open(loss_file, 'w') as outfile:
+        outfile.write(str(generator_loss_list))
+        outfile.write('\n')
+        outfile.write('\n')
+        outfile.write(str(discriminator_loss_list))
+        outfile.write('\n')
+        outfile.write('\n')
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    To train simply call this function. Warning: this might take a long time so +there is a folder of a pretrained network already included in the repository. +

    + + + +
    +
    +
    +
    +
    + 
    train(train_dataset, EPOCHS)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    And here is the result of training our model for 100 epochs

    + + +

    + +

    Now to avoid having to train and everything, which will take a while depending +on your computer setup we now load in the model which produced the above gif. +

    + + + +
    +
    +
    +
    +
    + 
    checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))
+restored_generator = checkpoint.generator
+restored_discriminator = checkpoint.discriminator
+
+print(restored_generator)
+print(restored_discriminator)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Exploring the Latent Space

    + +

    We have successfully loaded in our latest model. Let us now play around a bit +and see what kind of things we can learn about this model. Our generator takes +an array of 100 numbers. One idea can be to try to systematically change our +input. Let us try and see what we get +

    + + + +
    +
    +
    +
    +
    + 
    def generate_latent_points(number=100, scale_means=1, scale_stds=1):
+    latent_dim = 100
+    means = scale_means * tf.linspace(-1, 1, num=latent_dim)
+    stds = scale_stds * tf.linspace(-1, 1, num=latent_dim)
+    latent_space_value_range = tf.random.normal([number, latent_dim],
+                                                means,
+                                                stds,
+                                                dtype=tf.float64)
+
+    return latent_space_value_range
+
+def generate_images(latent_points):
+    # notice we set training to false because we are making inferences
+    generated_images = restored_generator.predict(latent_points)
+
+    return generated_images
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    def plot_result(generated_images, number=100):
+    # obviously this assumes sqrt number is an int
+    fig, axs = plt.subplots(int(np.sqrt(number)), int(np.sqrt(number)),
+                            figsize=(10, 10))
+
+    for i in range(int(np.sqrt(number))):
+        for j in range(int(np.sqrt(number))):
+            axs[i, j].imshow(generated_images[i*j], cmap='Greys')
+            axs[i, j].axis('off')
+
+    plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    generated_images = generate_images(generate_latent_points())
+plot_result(generated_images)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Getting Results

    +

    We see that the generator generates images that look like MNIST +numbers: \( 1, 4, 7, 9 \). Let's try to tweak it a bit more to see if we are able +to generate a similar plot where we generate every MNIST number. Let us now try +to 'move' a bit around in the latent space. Note: decrease the plot number if +these following cells take too long to run on your computer. +

    + + + +
    +
    +
    +
    +
    + 
    plot_number = 225
+
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=5,
+                                                          scale_stds=1))
+plot_result(generated_images, number=plot_number)
+
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=-5,
+                                                          scale_stds=1))
+plot_result(generated_images, number=plot_number)
+
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=1,
+                                                          scale_stds=5))
+plot_result(generated_images, number=plot_number)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Again, we have found something interesting. Moving around using our means +takes us from digit to digit, while moving around using our standard +deviations seem to increase the number of different digits! In the last image +above, we can barely make out every MNIST digit. Let us make on last plot using +this information by upping the standard deviation of our Gaussian noises. +

    + + + +
    +
    +
    +
    +
    + 
    plot_number = 400
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=1,
+                                                          scale_stds=10))
+plot_result(generated_images, number=plot_number)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    A pretty cool result! We see that our generator indeed has learned a +distribution which qualitatively looks a whole lot like the MNIST dataset. +

    + +









    +

    Interpolating Between MNIST Digits

    +

    Another interesting way to explore the latent space of our generator model is by +interpolating between the MNIST digits. This section is largely based on +this excellent blogpost +by Jason Brownlee. +

    + +

    So let us start by defining a function to interpolate between two points in the +latent space. +

    + + + +
    +
    +
    +
    +
    + 
    def interpolation(point_1, point_2, n_steps=10):
+    ratios = np.linspace(0, 1, num=n_steps)
+    vectors = []
+    for i, ratio in enumerate(ratios):
+        vectors.append(((1.0 - ratio) * point_1 + ratio * point_2))
+
+    return tf.stack(vectors)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we have all we need to do our interpolation analysis.

    + + + +
    +
    +
    +
    +
    + 
    plot_number = 100
+latent_points = generate_latent_points(number=plot_number)
+results = None
+for i in range(0, 2*np.sqrt(plot_number), 2):
+    interpolated = interpolation(latent_points[i], latent_points[i+1])
+    generated_images = generate_images(interpolated)
+
+    if results is None:
+        results = generated_images
+    else:
+        results = tf.stack((results, generated_images))
+
+plot_results(results, plot_number)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Basic ideas of the Principal Component Analysis (PCA)

    + +

    The principal component analysis deals with the problem of fitting a +low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than +the total dimension \( D \) of the problem at hand (our data +set). Mathematically it can be formulated as a statistical problem or +a geometric problem. In our discussion of the theorem for the +classical PCA, we will stay with a statistical approach. +Historically, the PCA was first formulated in a statistical setting in order to estimate the principal component of a multivariate random variable. +

    + +

    We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)

    +
      +
    • Each data point is determined by \( p \) extrinsic (measurement) variables
      • +
    • We may want to ask the following question: Are there fewer intrinsic variables (say \( d < < p \)) that still approximately describe the data?
      • +
    • If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.
      • +
    +

    A good read is for example Vidal, Ma and Sastry.

    + +









    +

    Introducing the Covariance and Correlation functions

    + +

    Before we discuss the PCA theorem, we need to remind ourselves about +the definition of the covariance and the correlation function. These are quantities +

    + +

    Suppose we have defined two vectors +\( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as +

    +$$ +\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\ + \end{bmatrix}, +$$ + +

    where for example

    +$$ +\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). +$$ + +

    With this definition and recalling that the variance is defined as

    +$$ +\mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2, +$$ + +

    we can rewrite the covariance matrix as

    +$$ +\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\ + \end{bmatrix}. +$$ + + +









    +

    More on the covariance

    +

    The covariance takes values between zero and infinity and may thus +lead to problems with loss of numerical precision for particularly +large values. It is common to scale the covariance matrix by +introducing instead the correlation matrix defined via the so-called +correlation function +

    + +$$ +\mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}. +$$ + +

    The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] +\in [-1,1] \). This avoids eventual problems with too large values. We +can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) +and \( \boldsymbol{y} \) as +

    + +$$ +\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{corr}[\boldsymbol{y},\boldsymbol{x}] & 1 \\ + \end{bmatrix}, +$$ + +

    In the above example this is the function we constructed using pandas.

    + +









    +

    Reminding ourselves about Linear Regression

    +

    In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression +we defined the design/feature matrix \( \boldsymbol{X} \) as +

    + +$$ +\boldsymbol{X}=\begin{bmatrix} +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\ +\dots & \dots & \dots & \dots \dots & \dots \\ +x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ +\end{bmatrix}, +$$ + +

    with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the +entries \( n \) being the row elements. +We can rewrite the design/feature matrix in terms of its column vectors as +

    +$$ +\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix}, +$$ + +

    with a given vector

    +$$ +\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. +$$ + + +









    +

    Simple Example

    +

    With these definitions, we can now rewrite our \( 2\times 2 \) +correlation/covariance matrix in terms of a moe general design/feature +matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) +covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) +

    + +$$ +\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} +\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +\mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_1] & \mathrm{var}[\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & \mathrm{var}[\boldsymbol{x}_{p-1}]\\ +\end{bmatrix}, +$$ + + +









    +

    The Correlation Matrix

    + +

    and the correlation matrix

    +$$ +\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} +1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +\mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +\mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\ +\end{bmatrix}, +$$ + + +









    +

    Numpy Functionality

    + +

    The Numpy function np.cov calculates the covariance elements using +the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have +the exact mean values. The following simple function uses the +np.vstack function which takes each vector of dimension \( 1\times n \) +and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +

    + +$$ +\boldsymbol{W}^T = \begin{bmatrix} x_0 & y_0 \\ + x_1 & y_1 \\ + x_2 & y_2\\ + \dots & \dots \\ + x_{n-2} & y_{n-2}\\ + x_{n-1} & y_{n-1} & + \end{bmatrix}, +$$ + +

    which in turn is converted into into the \( 2\times 2 \) covariance matrix +\( \boldsymbol{C} \) via the Numpy function np.cov(). We note that we can also calculate +the mean value of each set of samples \( \boldsymbol{x} \) etc using the Numpy +function np.mean(x). We can also extract the eigenvalues of the +covariance matrix through the np.linalg.eig() function. +

    + + + +
    +
    +
    +
    +
    + 
    # Importing various packages
+import numpy as np
+n = 100
+x = np.random.normal(size=n)
+print(np.mean(x))
+y = 4+3*x+np.random.normal(size=n)
+print(np.mean(y))
+W = np.vstack((x, y))
+C = np.cov(W)
+print(C)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Correlation Matrix again

    + +

    The previous example can be converted into the correlation matrix by +simply scaling the matrix elements with the variances. We should also +subtract the mean values for each column. This leads to the following +code which sets up the correlations matrix for the previous example in +a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors). +

    + + + +
    +
    +
    +
    +
    + 
    import numpy as np
+n = 100
+# define two vectors                                                                                           
+x = np.random.random(size=n)
+y = 4+3*x+np.random.normal(size=n)
+#scaling the x and y vectors                                                                                   
+x = x - np.mean(x)
+y = y - np.mean(y)
+variance_x = np.sum(x@x)/n
+variance_y = np.sum(y@y)/n
+print(variance_x)
+print(variance_y)
+cov_xy = np.sum(x@y)/n
+cov_xx = np.sum(x@x)/n
+cov_yy = np.sum(y@y)/n
+C = np.zeros((2,2))
+C[0,0]= cov_xx/variance_x
+C[1,1]= cov_yy/variance_y
+C[0,1]= cov_xy/np.sqrt(variance_y*variance_x)
+C[1,0]= C[0,1]
+print(C)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We see that the matrix elements along the diagonal are one as they +should be and that the matrix is symmetric. Furthermore, diagonalizing +this matrix we easily see that it is a positive definite matrix. +

    + +

    The above procedure with numpy can be made more compact if we use pandas.

    + +









    +

    Using Pandas

    + +

    We whow here how we can set up the correlation matrix using pandas, as done in this simple code

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import pandas as pd
+n = 10
+x = np.random.normal(size=n)
+x = x - np.mean(x)
+y = 4+3*x+np.random.normal(size=n)
+y = y - np.mean(y)
+X = (np.vstack((x, y))).T
+print(X)
+Xpd = pd.DataFrame(X)
+print(Xpd)
+correlation_matrix = Xpd.corr()
+print(correlation_matrix)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    And then the Franke Function

    + +

    We expand this model to the Franke function discussed above.

    + + + +
    +
    +
    +
    +
    + 
    # Common imports
+import numpy as np
+import pandas as pd
+
+
+def FrankeFunction(x,y):
+	term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
+	term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
+	term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
+	term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
+	return term1 + term2 + term3 + term4
+
+
+def create_X(x, y, n ):
+	if len(x.shape) > 1:
+		x = np.ravel(x)
+		y = np.ravel(y)
+
+	N = len(x)
+	l = int((n+1)*(n+2)/2)		# Number of elements in beta
+	X = np.ones((N,l))
+
+	for i in range(1,n+1):
+		q = int((i)*(i+1)/2)
+		for k in range(i+1):
+			X[:,q+k] = (x**(i-k))*(y**k)
+
+	return X
+
+
+# Making meshgrid of datapoints and compute Franke's function
+n = 4
+N = 100
+x = np.sort(np.random.uniform(0, 1, N))
+y = np.sort(np.random.uniform(0, 1, N))
+z = FrankeFunction(x, y)
+X = create_X(x, y, n=n)    
+
+Xpd = pd.DataFrame(X)
+# subtract the mean values and set up the covariance matrix
+Xpd = Xpd - Xpd.mean()
+covariance_matrix = Xpd.cov()
+print(covariance_matrix)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We note here that the covariance is zero for the first rows and +columns since all matrix elements in the design matrix were set to one +(we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept +and wee can simply +drop these elements and construct a correlation +matrix without them by centering our matrix elements by subtracting the mean of each column. +

    + +









    +

    Lnks with the Design Matrix

    + +

    We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as

    +$$ +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. +$$ + +

    To see this let us simply look at a design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{2\times 2} \)

    +$$ +\boldsymbol{X}=\begin{bmatrix} +x_{00} & x_{01}\\ +x_{10} & x_{11}\\ +\end{bmatrix}=\begin{bmatrix} +\boldsymbol{x}_{0} & \boldsymbol{x}_{1}\\ +\end{bmatrix}. +$$ + + +









    +

    Computing the Expectation Values

    + +

    If we then compute the expectation value

    +$$ +\mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\begin{bmatrix} +x_{00}^2+x_{01}^2 & x_{00}x_{10}+x_{01}x_{11}\\ +x_{10}x_{00}+x_{11}x_{01} & x_{10}^2+x_{11}^2\\ +\end{bmatrix}, +$$ + +

    which is just

    +$$ +\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]=\begin{bmatrix} \mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] \\ + \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] \\ + \end{bmatrix}, +$$ + +

    where we wrote $$\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]$$ to indicate that this the covariance of the vectors \( \boldsymbol{x} \) of the design/feature matrix \( \boldsymbol{X} \).

    + +

    It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \).

    + +









    +

    Towards the PCA theorem

    + +

    We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as

    +$$ +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. +$$ + +

    Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices \( \boldsymbol{S} \). +These matrices are defined as \( \boldsymbol{S}\in {\mathbb{R}}^{p\times p} \) and obey the orthogonality requirements \( \boldsymbol{S}\boldsymbol{S}^T=\boldsymbol{S}^T\boldsymbol{S}=\boldsymbol{I} \). The matrix can be written out in terms of the column vectors \( \boldsymbol{s}_i \) as \( \boldsymbol{S}=[\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \) and \( \boldsymbol{s}_i \in {\mathbb{R}}^{p} \). +

    + +

    Assume also that there is a transformation \( \boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}=\boldsymbol{C}[\boldsymbol{y}] \) such that the new matrix \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal with elements \( [\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}] \).

    + +

    That is we have

    +$$ +\boldsymbol{C}[\boldsymbol{y}] = \mathbb{E}[\boldsymbol{S}^T\boldsymbol{X}^T\boldsymbol{X}T\boldsymbol{S}]=\boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}, +$$ + +

    since the matrix \( \boldsymbol{S} \) is not a data dependent matrix. Multiplying with \( \boldsymbol{S} \) from the left we have

    +$$ +\boldsymbol{S}\boldsymbol{C}[\boldsymbol{y}] = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}, +$$ + +

    and since \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal we have for a given eigenvalue \( i \) of the covariance matrix that

    + +$$ +\boldsymbol{S}_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}_i. +$$ + + +









    +

    More on the PCA Theorem

    + +

    In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is +\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \). +

    + +

    The eigenvalues tell us then how much we need to stretch the +corresponding eigenvectors. Dimensions with large eigenvalues have +thus large variations (large variance) and define therefore useful +dimensions. The data points are more spread out in the direction of +these eigenvectors. Smaller eigenvalues mean on the other hand that +the corresponding eigenvectors are shrunk accordingly and the data +points are tightly bunched together and there is not much variation in +these specific directions. Hopefully then we could leave it out +dimensions where the eigenvalues are very small. If \( p \) is very large, +we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) +features/predictors. +

    + +









    +

    The Algorithm before theorem

    + +

    Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.

    +
      +
    • Set up the datapoints for the design/feature matrix \( \boldsymbol{X} \) with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) referring to the column numbers and the entries \( n \) being the row elements.
      • +
    +$$ +\boldsymbol{X}=\begin{bmatrix} +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\ +\dots & \dots & \dots & \dots \dots & \dots \\ +x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ +\end{bmatrix}, +$$ + +
      +
    • Center the data by subtracting the mean value for each column. This leads to a new matrix \( \boldsymbol{X}\rightarrow \overline{\boldsymbol{X}} \).
      • +
    • Compute then the covariance/correlation matrix \( \mathbb{E}[\overline{\boldsymbol{X}}^T\overline{\boldsymbol{X}}] \).
      • +
    • Find the eigenpairs of \( \boldsymbol{C} \) with eigenvalues \( [\lambda_0,\lambda_1,\dots,\lambda_{p-1}] \) and eigenvectors \( [\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \).
      • +
    • Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.
      • +
    • Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.
      • +
    +









    +

    Writing our own PCA code

    + +

    We will use a simple example first with two-dimensional data +drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below): +

    +$$ +\mu = (-1,2) \qquad \Sigma = \begin{bmatrix} 4 & 2 \\ +2 & 2 +\end{bmatrix} +$$ + +

    Note that the mean refers to each column of data. +We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from +this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values. +

    + +









    +

    Implementing it

    +

    The following Python code aids in setting up the data and writing out the design matrix. +Note that the function multivariate returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \). +

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from IPython.display import display
+n = 10000
+mean = (-1, 2)
+cov = [[4, 2], [2, 2]]
+X = np.random.multivariate_normal(mean, cov, n)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we are going to implement the PCA algorithm. We will break it down into various substeps.

    + +









    +

    First Step

    + +

    The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is

    +$$ +\mu_n = \frac{1}{n} \sum_{i=1}^n x_i +$$ + +

    and the mean-centered data \( \bar{X} = \{ \bar{x}_1, \ldots, \bar{x}_n \} \) takes the form

    +$$ +\bar{x}_i = x_i - \mu_n. +$$ + +

    When you are done with these steps, print out \( \mu_n \) to verify it is +close to \( \mu \) and plot your mean centered data to verify it is +centered at the origin! +The following code elements perform these operations using pandas or using our own functionality for doing so. The latter, using numpy is rather simple through the mean() function. +

    + + +
    +
    +
    +
    +
    + 
    df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Scaling

    +

    Alternatively, we could use the functions we discussed +earlier for scaling the data set. That is, we could have used the +StandardScaler function in Scikit-Learn, a function which ensures +that for each feature/predictor we study the mean value is zero and +the variance is one (every column in the design/feature matrix). You +would then not get the same results, since we divide by the +variance. The diagonal covariance matrix elements will then be one, +while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our +specific case. +

    + +









    +

    Centered Data

    + +

    Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation

    +$$ +\begin{equation*} +\Sigma_n = \frac{1}{n-1} \sum_{i=1}^n \bar{x}_i^T \bar{x}_i = \frac{1}{n-1} \sum_{i=1}^n (x_i - \mu_n)^T (x_i - \mu_n) +\end{equation*} +$$ + +

    where the data points \( x_i \in \mathbb{R}^p \) (here in this example \( p = 2 \)) are column vectors and \( x^T \) is the transpose of \( x \). +We can write our own code or simply use either the functionaly of numpy or that of pandas, as follows +

    + + +
    +
    +
    +
    +
    + 
    print(df.cov())
+print(np.cov(X_centered.T))
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Note that the way we define the covariance matrix here has a factor \( n-1 \) instead of \( n \). This is included in the cov() function by numpy and pandas. +Our own code here is not very elegant and asks for obvious improvements. It is tailored to this specific \( 2\times 2 \) covariance matrix. +

    + + +
    +
    +
    +
    +
    + 
    # extract the relevant columns from the centered design matrix of dim n x 2
+x = X_centered[:,0]
+y = X_centered[:,1]
+Cov = np.zeros((2,2))
+Cov[0,1] = np.sum(x.T@y)/(n-1.0)
+Cov[0,0] = np.sum(x.T@x)/(n-1.0)
+Cov[1,1] = np.sum(y.T@y)/(n-1.0)
+Cov[1,0]= Cov[0,1]
+print("Centered covariance using own code")
+print(Cov)
+plt.plot(x, y, 'x')
+plt.axis('equal')
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Exploring

    + +

    Depending on the number of points \( n \), we will get results that are close to the covariance values defined above. +The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed. +

    + +









    +

    Diagonalize the sample covariance matrix to obtain the principal components

    + +

    Now we are ready to solve for the principal components! To do so we +diagonalize the sample covariance matrix \( \Sigma \). We can use the +function np.linalg.eig to do so. It will return the eigenvalues and +eigenvectors of \( \Sigma \). Once we have these we can perform the +following tasks: +

    + +
      +
    • We compute the percentage of the total variance captured by the first principal component
      • +
    • We plot the mean centered data and lines along the first and second principal components
      • +
    • Then we project the mean centered data onto the first and second principal components, and plot the projected data.
      • +
    • Finally, we approximate the data as
      • +
    +$$ +\begin{equation*} +x_i \approx \tilde{x}_i = \mu_n + \langle x_i, v_0 \rangle v_0 +\end{equation*} +$$ + +

    where \( v_0 \) is the first principal component.

    + +









    +

    Collecting all Steps

    + +

    Collecting all these steps we can write our own PCA function and +compare this with the functionality included in Scikit-Learn. +

    + +

    The code here outlines some of the elements we could include in the +analysis. Feel free to extend upon this in order to address the above +questions. +

    + + + +
    +
    +
    +
    +
    + 
    # diagonalize and obtain eigenvalues, not necessarily sorted
+EigValues, EigVectors = np.linalg.eig(Cov)
+# sort eigenvectors and eigenvalues
+#permute = EigValues.argsort()
+#EigValues = EigValues[permute]
+#EigVectors = EigVectors[:,permute]
+print("Eigenvalues of Covariance matrix")
+for i in range(2):
+    print(EigValues[i])
+FirstEigvector = EigVectors[:,0]
+SecondEigvector = EigVectors[:,1]
+print("First eigenvector")
+print(FirstEigvector)
+print("Second eigenvector")
+print(SecondEigvector)
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2Dsl = pca.fit_transform(X)
+print("Eigenvector of largest eigenvalue")
+print(pca.components_.T[:, 0])
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    This code does not contain all the above elements, but it shows how we can use Scikit-Learn to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then?

    + +









    +

    Classical PCA Theorem

    + +

    We assume now that we have a design matrix \( \boldsymbol{X} \) which has been +centered as discussed above. For the sake of simplicity we skip the +overline symbol. The matrix is defined in terms of the various column +vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension +\( \boldsymbol{x}\in {\mathbb{R}}^{n} \). +

    + +

    The PCA theorem states that minimizing the above reconstruction error +corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which +diagonalizes the empirical covariance(correlation) matrix. The optimal +low-dimensional encoding of the data is then given by a set of vectors +\( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the +orthogonal projection of the data onto the columns spanned by the +eigenvectors of the covariance(correlations matrix). +

    + +









    +

    The PCA Theorem

    + +

    To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as

    + +

    We are almost there, we have obtained a relation between minimizing +the reconstruction error and the variance and the covariance +matrix. Minimizing the error is equivalent to maximizing the variance +of the projected data. +

    + +

    We could trivially maximize the variance of the projection (and +thereby minimize the error in the reconstruction function) by letting +the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we +want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by +\( \vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a +Lagrange multiplier we can then in turn maximize +

    + +$$ +J(\boldsymbol{w}_0)= \boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0+\lambda_0(1-\boldsymbol{w}_0^T\boldsymbol{w}_0). +$$ + +

    Taking the derivative with respect to \( \boldsymbol{w}_0 \) we obtain

    + +$$ +\frac{\partial J(\boldsymbol{w}_0)}{\partial \boldsymbol{w}_0}= 2\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0-2\lambda_0\boldsymbol{w}_0=0, +$$ + +

    meaning that

    +$$ +\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0\boldsymbol{w}_0. +$$ + +

    The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is

    +$$ +\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0. +$$ + +

    If we want to maximize the variance (minimize the construction error) +we simply pick the eigenvector of the covariance matrix with the +largest eigenvalue. This establishes the link between the minimization +of the reconstruction function \( J \) in terms of an orthogonal matrix +and the maximization of the variance and thereby the covariance of our +observations encoded in the design/feature matrix \( \boldsymbol{X} \). +

    + +

    The proof +for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) can be +established by applying the above arguments and using the fact that +our basis of eigenvectors is orthogonal, see Murphy chapter +12.2. The +discussion in chapter 12.2 of Murphy's text has also a nice link with +the Singular Value Decomposition theorem. For categorical data, see +chapter 12.4 and discussion therein. +

    + +

    For more details, see for example Vidal, Ma and Sastry, chapter 2.

    + +









    + + +

    For a detailed demonstration of the geometric interpretation, see Vidal, Ma and Sastry, section 2.1.2.

    + +

    Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm. +First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it. +

    + +

    The following Python code uses NumPy’s svd() function to obtain all the principal components of the +training set, then extracts the first two principal components. First we center the data using either pandas or our own code +

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 vanilla matrix 
+rows = 10
+cols = 5
+X = np.random.randn(rows,cols)
+df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+display(df)
+
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+# Then check the difference between pandas and our own set up
+print(X_centered-df)
+#Now we do an SVD
+U, s, V = np.linalg.svd(X_centered)
+c1 = V.T[:, 0]
+c2 = V.T[:, 1]
+W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+print(X2D)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering +the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t +forget to center the data first. +

    + +

    Once you have identified all the principal components, you can reduce the dimensionality of the dataset +down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components. +Selecting this hyperplane ensures that the projection will preserve as much variance as possible. +

    + + +
    +
    +
    +
    +
    + 
    W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    PCA and scikit-learn

    + +

    Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The +following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note +that it automatically takes care of centering the data): +

    + + +
    +
    +
    +
    +
    + 
    #thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2D = pca.fit_transform(X)
+print(X2D)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    After fitting the PCA transformer to the dataset, you can access the principal components using the +components variable (note that it contains the PCs as horizontal vectors, so, for example, the first +principal component is equal to +

    + + +
    +
    +
    +
    +
    + 
    pca.components_.T[:, 0]
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Another very useful piece of information is the explained variance ratio of each principal component, +available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s +variance that lies along the axis of each principal component. +

    + +









    +

    Back to the Cancer Data

    +

    We can now repeat the above but applied to real data, in this case our breast cancer data. +Here we compute performance scores on the training data using logistic regression. +

    + + +
    +
    +
    +
    +
    + 
    import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import  train_test_split 
+from sklearn.datasets import load_breast_cancer
+from sklearn.linear_model import LogisticRegression
+cancer = load_breast_cancer()
+
+X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
+
+logreg = LogisticRegression()
+logreg.fit(X_train, y_train)
+print("Train set accuracy from Logistic Regression: {:.2f}".format(logreg.score(X_train,y_train)))
+# We scale the data
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+# Then perform again a log reg fit
+logreg.fit(X_train_scaled, y_train)
+print("Train set accuracy scaled data: {:.2f}".format(logreg.score(X_train_scaled,y_train)))
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2D_train = pca.fit_transform(X_train_scaled)
+# and finally compute the log reg fit and the score on the training data	
+logreg.fit(X2D_train,y_train)
+print("Train set accuracy scaled and PCA data: {:.2f}".format(logreg.score(X2D_train,y_train)))
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.

    + +

    Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to +choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%). +Unless, of course, you are reducing dimensionality for data visualization — in that case you will +generally want to reduce the dimensionality down to 2 or 3. +The following code computes PCA without reducing dimensionality, then computes the minimum number +of dimensions required to preserve 95% of the training set’s variance: +

    + + +
    +
    +
    +
    +
    + 
    pca = PCA()
+pca.fit(X)
+cumsum = np.cumsum(pca.explained_variance_ratio_)
+d = np.argmax(cumsum >= 0.95) + 1
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead +of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be +a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve: +

    + + +
    +
    +
    +
    +
    + 
    pca = PCA(n_components=0.95)
+X_reduced = pca.fit_transform(X)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Incremental PCA

    + +

    One problem with the preceding implementation of PCA is that it requires the whole training set to fit in +memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have +been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch +at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new +instances arrive). +

    +

    Randomized PCA

    + +

    Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic +algorithm that quickly finds an approximation of the first d principal components. Its computational +complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the +previous algorithms when \( d \) is much smaller than \( n \). +

    +

    Kernel PCA

    + +

    The kernel trick is a mathematical technique that implicitly maps instances into a +very high-dimensional space (called the feature space), enabling nonlinear classification and regression +with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature +space corresponds to a complex nonlinear decision boundary in the original space. +It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear +projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at +preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a +twisted manifold. +For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an +

    + + +
    +
    +
    +
    +
    + 
    from sklearn.decomposition import KernelPCA
+rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04)
+X_reduced = rbf_pca.fit_transform(X)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Other techniques

    + +

    There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.

    + +

    Here are some of the most popular:

    +
      +
    • Multidimensional Scaling (MDS) reduces dimensionality while trying to preserve the distances between the instances.
      • +
    • Isomap creates a graph by connecting each instance to its nearest neighbors, then reduces dimensionality while trying to preserve the geodesic distances between the instances.
      • +
    • t-Distributed Stochastic Neighbor Embedding (t-SNE) reduces dimensionality while trying to keep similar instances close and dissimilar instances apart. It is mostly used for visualization, in particular to visualize clusters of instances in high-dimensional space (e.g., to visualize the MNIST images in 2D).
      • +
    • Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it learns the most discriminative axes between the classes, and these axes can then be used to define a hyperplane onto which to project the data. The benefit is that the projection will keep classes as far apart as possible, so LDA is a good technique to reduce dimensionality before running another classification algorithm such as a Support Vector Machine (SVM) classifier discussed in the SVM lectures.
      • +
    + +
    + © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license +
    + + + diff --git a/doc/src/week43/week43.html b/doc/src/week43/week43.html new file mode 100644 index 000000000..6b1437d42 --- /dev/null +++ b/doc/src/week43/week43.html @@ -0,0 +1,3451 @@ + + + + + + + +Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis + + + + + + + + + + + + + + +
    +

    Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis

    +
    + + +
    +Morten Hjorth-Jensen [1, 2] +
    + +
    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 2, 2021

    +
    +
    + +









    +

    Plans for week 43

    + +
      +
    • Thursday: Summary of Convolutional Neural Networks from week 42 and Recurrent Neural Networks
      • + +
    • Friday: Recurrent Neural Networks and other Deep Learning methods such as Generalized Adversarial Neural Networks. Start discussing Principal component analysis
      • + +
    +
    +Excellent lectures on CNNs and RNNs +

    +

    +
    + + +
    +More resources +

    +

    +
    + + +









    +

    Reading Recommendations

    + +
      +
    • Goodfellow et al, chapter 10 on Recurrent NNs, chapters 11 and 12 on various practicalities around deep learning are also recommended.
      • +
    • Aurelien Geron, chapter 14 on RNNs.
      • +
    +









    +

    Summary on Deep Learning Methods

    + +

    We have studied fully connected neural networks (also called artifical nueral networks) and convolutional neural networks (CNNs).

    + +

    The first type of deep learning networks work very well on homogeneous and structured input data while CCNs are normally tailored to recognizing images.

    + +









    +

    CNNs in brief

    + +

    In summary:

    + +
      +
    • A CNN architecture is in the simplest case a list of Layers that transform the image volume into an output volume (e.g. holding the class scores)
      • +
    • There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular)
      • +
    • Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function
      • +
    • Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don’t)
      • +
    • Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn’t)
      • +
    +

    For more material on convolutional networks, we strongly recommend +the course +IN5400 – Machine Learning for Image Analysis +and the slides of CS231 which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs. +

    + +

    However, both standard feed forwards networks and CNNs perform well on data with unknown length.

    + +

    This is where recurrent nueral networks (RNNs) come to our rescue.

    + +









    +

    Recurrent neural networks: Overarching view

    + +

    Till now our focus has been, including convolutional neural networks +as well, on feedforward neural networks. The output or the activations +flow only in one direction, from the input layer to the output layer. +

    + +

    A recurrent neural network (RNN) looks very much like a feedforward +neural network, except that it also has connections pointing +backward. +

    + +

    RNNs are used to analyze time series data such as stock prices, and +tell you when to buy or sell. In autonomous driving systems, they can +anticipate car trajectories and help avoid accidents. More generally, +they can work on sequences of arbitrary lengths, rather than on +fixed-sized inputs like all the nets we have discussed so far. For +example, they can take sentences, documents, or audio samples as +input, making them extremely useful for natural language processing +systems such as automatic translation and speech-to-text. +

    + +









    +

    Set up of an RNN

    + +

    More to text to be added

    + +









    +

    A simple example

    + + + +
    +
    +
    +
    +
    + 
    # Start importing packages
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import tensorflow as tf
+from tensorflow.keras import datasets, layers, models
+from tensorflow.keras.layers import Input
+from tensorflow.keras.models import Model, Sequential 
+from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU
+from tensorflow.keras import optimizers     
+from tensorflow.keras import regularizers           
+from tensorflow.keras.utils import to_categorical 
+
+
+
+# convert into dataset matrix
+def convertToMatrix(data, step):
+ X, Y =[], []
+ for i in range(len(data)-step):
+  d=i+step  
+  X.append(data[i:d,])
+  Y.append(data[d,])
+ return np.array(X), np.array(Y)
+
+step = 4
+N = 1000    
+Tp = 800    
+
+t=np.arange(0,N)
+x=np.sin(0.02*t)+2*np.random.rand(N)
+df = pd.DataFrame(x)
+df.head()
+
+plt.plot(df)
+plt.show()
+
+values=df.values
+train,test = values[0:Tp,:], values[Tp:N,:]
+
+# add step elements into train and test
+test = np.append(test,np.repeat(test[-1,],step))
+train = np.append(train,np.repeat(train[-1,],step))
+ 
+trainX,trainY =convertToMatrix(train,step)
+testX,testY =convertToMatrix(test,step)
+trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
+testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))
+
+model = Sequential()
+model.add(SimpleRNN(units=32, input_shape=(1,step), activation="relu"))
+model.add(Dense(8, activation="relu")) 
+model.add(Dense(1))
+model.compile(loss='mean_squared_error', optimizer='rmsprop')
+model.summary()
+
+model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)
+trainPredict = model.predict(trainX)
+testPredict= model.predict(testX)
+predicted=np.concatenate((trainPredict,testPredict),axis=0)
+
+trainScore = model.evaluate(trainX, trainY, verbose=0)
+print(trainScore)
+
+index = df.index.values
+plt.plot(index,df)
+plt.plot(index,predicted)
+plt.axvline(df.index[Tp], c="r")
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    An extrapolation example

    + +

    The following code provides an example of how recurrent neural +networks can be used to extrapolate to unknown values of physics data +sets. Specifically, the data sets used in this program come from +a quantum mechanical many-body calculation of energies as functions of the number of particles. +

    + + + +
    +
    +
    +
    +
    + 
    # For matrices and calculations
+import numpy as np
+# For machine learning (backend for keras)
+import tensorflow as tf
+# User-friendly machine learning library
+# Front end for TensorFlow
+import tensorflow.keras
+# Different methods from Keras needed to create an RNN
+# This is not necessary but it shortened function calls 
+# that need to be used in the code.
+from tensorflow.keras import datasets, layers, models
+from tensorflow.keras.layers import Input
+from tensorflow.keras import regularizers
+from tensorflow.keras.models import Model, Sequential
+from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU
+# For timing the code
+from timeit import default_timer as timer
+# For plotting
+import matplotlib.pyplot as plt
+
+
+# The data set
+datatype='VaryDimension'
+X_tot = np.arange(2, 42, 2)
+y_tot = np.array([-0.03077640549, -0.08336233266, -0.1446729567, -0.2116753732, -0.2830637392, -0.3581341341, -0.436462435, -0.5177783846,
+	-0.6019067271, -0.6887363571, -0.7782028952, -0.8702784034, -0.9649652536, -1.062292565, -1.16231451, 
+	-1.265109911, -1.370782966, -1.479465113, -1.591317992, -1.70653767])
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Formatting the Data

    + +

    The way the recurrent neural networks are trained in this program +differs from how machine learning algorithms are usually trained. +Typically a machine learning algorithm is trained by learning the +relationship between the x data and the y data. In this program, the +recurrent neural network will be trained to recognize the relationship +in a sequence of y values. This is type of data formatting is +typically used time series forcasting, but it can also be used in any +extrapolation (time series forecasting is just a specific type of +extrapolation along the time axis). This method of data formatting +does not use the x data and assumes that the y data are evenly spaced. +

    + +

    For a standard machine learning algorithm, the training data has the +form of (x,y) so the machine learning algorithm learns to assiciate a +y value with a given x value. This is useful when the test data has x +values within the same range as the training data. However, for this +application, the x values of the test data are outside of the x values +of the training data and the traditional method of training a machine +learning algorithm does not work as well. For this reason, the +recurrent neural network is trained on sequences of y values of the +form ((y1, y2), y3), so that the network is concerned with learning +the pattern of the y data and not the relation between the x and y +data. As long as the pattern of y data outside of the training region +stays relatively stable compared to what was inside the training +region, this method of training can produce accurate extrapolations to +y values far removed from the training data set. +

    + + + + + + + + + + +
    +
    +
    +
    +
    + 
    # FORMAT_DATA
+def format_data(data, length_of_sequence = 2):  
+    """
+        Inputs:
+            data(a numpy array): the data that will be the inputs to the recurrent neural
+                network
+            length_of_sequence (an int): the number of elements in one iteration of the
+                sequence patter.  For a function approximator use length_of_sequence = 2.
+        Returns:
+            rnn_input (a 3D numpy array): the input data for the recurrent neural network.  Its
+                dimensions are length of data - length of sequence, length of sequence, 
+                dimnsion of data
+            rnn_output (a numpy array): the training data for the neural network
+        Formats data to be used in a recurrent neural network.
+    """
+
+    X, Y = [], []
+    for i in range(len(data)-length_of_sequence):
+        # Get the next length_of_sequence elements
+        a = data[i:i+length_of_sequence]
+        # Get the element that immediately follows that
+        b = data[i+length_of_sequence]
+        # Reshape so that each data point is contained in its own array
+        a = np.reshape (a, (len(a), 1))
+        X.append(a)
+        Y.append(b)
+    rnn_input = np.array(X)
+    rnn_output = np.array(Y)
+
+    return rnn_input, rnn_output
+
+
+# ## Defining the Recurrent Neural Network Using Keras
+# 
+# The following method defines a simple recurrent neural network in keras consisting of one input layer, one hidden layer, and one output layer.
+
+def rnn(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with one hidden layer and returns the model.
+    """
+    # Number of neurons in the input and output layers
+    in_out_neurons = 1
+    # Number of neurons in the hidden layer
+    hidden_neurons = 200
+    # Define the input layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons))  
+    # Define the hidden layer as a simple RNN layer with a set number of neurons and add it to 
+    # the network immediately after the input layer
+    rnn = SimpleRNN(hidden_neurons, 
+                    return_sequences=False,
+                    stateful = stateful,
+                    name="RNN")(inp)
+    # Define the output layer as a dense neural network layer (standard neural network layer)
+    #and add it to the network immediately after the hidden layer.
+    dens = Dense(in_out_neurons,name="dense")(rnn)
+    # Create the machine learning model starting with the input layer and ending with the 
+    # output layer
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the machine learning model using the mean squared error function as the loss 
+    # function and an Adams optimizer.
+    model.compile(loss="mean_squared_error", optimizer="adam")  
+    return model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Predicting New Points With A Trained Recurrent Neural Network

    + + + +
    +
    +
    +
    +
    + 
    def test_rnn (x1, y_test, plot_min, plot_max):
+    """
+        Inputs:
+            x1 (a list or numpy array): The complete x component of the data set
+            y_test (a list or numpy array): The complete y component of the data set
+            plot_min (an int or float): the smallest x value used in the training data
+            plot_max (an int or float): the largest x valye used in the training data
+        Returns:
+            None.
+        Uses a trained recurrent neural network model to predict future points in the 
+        series.  Computes the MSE of the predicted data set from the true data set, saves
+        the predicted data set to a csv file, and plots the predicted and true data sets w
+        while also displaying the data range used for training.
+    """
+    # Add the training data as the first dim points in the predicted data array as these
+    # are known values.
+    y_pred = y_test[:dim].tolist()
+    # Generate the first input to the trained recurrent neural network using the last two 
+    # points of the training data.  Based on how the network was trained this means that it
+    # will predict the first point in the data set after the training data.  All of the 
+    # brackets are necessary for Tensorflow.
+    next_input = np.array([[[y_test[dim-2]], [y_test[dim-1]]]])
+    # Save the very last point in the training data set.  This will be used later.
+    last = [y_test[dim-1]]
+
+    # Iterate until the complete data set is created.
+    for i in range (dim, len(y_test)):
+        # Predict the next point in the data set using the previous two points.
+        next = model.predict(next_input)
+        # Append just the number of the predicted data set
+        y_pred.append(next[0][0])
+        # Create the input that will be used to predict the next data point in the data set.
+        next_input = np.array([[last, next[0]]], dtype=np.float64)
+        last = next
+
+    # Print the mean squared error between the known data set and the predicted data set.
+    print('MSE: ', np.square(np.subtract(y_test, y_pred)).mean())
+    # Save the predicted data set as a csv file for later use
+    name = datatype + 'Predicted'+str(dim)+'.csv'
+    np.savetxt(name, y_pred, delimiter=',')
+    # Plot the known data set and the predicted data set.  The red box represents the region that was used
+    # for the training data.
+    fig, ax = plt.subplots()
+    ax.plot(x1, y_test, label="true", linewidth=3)
+    ax.plot(x1, y_pred, 'g-.',label="predicted", linewidth=4)
+    ax.legend()
+    # Created a red region to represent the points used in the training data.
+    ax.axvspan(plot_min, plot_max, alpha=0.25, color='red')
+    plt.show()
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+
+# Generate the training data for the RNN, using a sequence of 2
+rnn_input, rnn_training = format_data(y_train, 2)
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+model = rnn(length_of_sequences = rnn_input.shape[1])
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict more points of the data set
+test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Other Things to Try

    + +

    Changing the size of the recurrent neural network and its parameters +can drastically change the results you get from the model. The below +code takes the simple recurrent neural network from above and adds a +second hidden layer, changes the number of neurons in the hidden +layer, and explicitly declares the activation function of the hidden +layers to be a sigmoid function. The loss function and optimizer can +also be changed but are kept the same as the above network. These +parameters can be tuned to provide the optimal result from the +network. For some ideas on how to improve the performance of a +recurrent neural network. +

    + + + +
    +
    +
    +
    +
    + 
    def rnn_2layers(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with two hidden layers and returns the model.
+    """
+    # Number of neurons in the input and output layers
+    in_out_neurons = 1
+    # Number of neurons in the hidden layer, increased from the first network
+    hidden_neurons = 500
+    # Define the input layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons))  
+    # Create two hidden layers instead of one hidden layer.  Explicitly set the activation
+    # function to be the sigmoid function (the default value is hyperbolic tangent)
+    rnn1 = SimpleRNN(hidden_neurons, 
+                    return_sequences=True,  # This needs to be True if another hidden layer is to follow
+                    stateful = stateful, activation = 'sigmoid',
+                    name="RNN1")(inp)
+    rnn2 = SimpleRNN(hidden_neurons, 
+                    return_sequences=False, activation = 'sigmoid',
+                    stateful = stateful,
+                    name="RNN2")(rnn1)
+    # Define the output layer as a dense neural network layer (standard neural network layer)
+    #and add it to the network immediately after the hidden layer.
+    dens = Dense(in_out_neurons,name="dense")(rnn2)
+    # Create the machine learning model starting with the input layer and ending with the 
+    # output layer
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the machine learning model using the mean squared error function as the loss 
+    # function and an Adams optimizer.
+    model.compile(loss="mean_squared_error", optimizer="adam")  
+    return model
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+
+# Generate the training data for the RNN, using a sequence of 2
+rnn_input, rnn_training = format_data(y_train, 2)
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+model = rnn_2layers(length_of_sequences = 2)
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+
+# This section plots the training loss and the validation loss as a function of training iteration.
+# This is not required for analyzing the couple cluster data but can help determine if the network is
+# being overtrained.
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict more points of the data set
+test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Other Types of Recurrent Neural Networks

    + +

    Besides a simple recurrent neural network layer, there are two other +commonly used types of recurrent neural network layers: Long Short +Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short +introduction to these layers see https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b +and https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b. +

    + +

    The first network created below is similar to the previous network, +but it replaces the SimpleRNN layers with LSTM layers. The second +network below has two hidden layers made up of GRUs, which are +preceeded by two dense (feeddorward) neural network layers. These +dense layers "preprocess" the data before it reaches the recurrent +layers. This architecture has been shown to improve the performance +of recurrent neural networks (see the link above and also +https://arxiv.org/pdf/1807.02857.pdf. +

    + + + +
    +
    +
    +
    +
    + 
    def lstm_2layers(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.
+    """
+    # Number of neurons on the input/output layer and the number of neurons in the hidden layer
+    in_out_neurons = 1
+    hidden_neurons = 250
+    # Input Layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons)) 
+    # Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)
+    rnn= LSTM(hidden_neurons, 
+                    return_sequences=True,
+                    stateful = stateful,
+                    name="RNN", use_bias=True, activation='tanh')(inp)
+    rnn1 = LSTM(hidden_neurons, 
+                    return_sequences=False,
+                    stateful = stateful,
+                    name="RNN1", use_bias=True, activation='tanh')(rnn)
+    # Output layer
+    dens = Dense(in_out_neurons,name="dense")(rnn1)
+    # Define the midel
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the model
+    model.compile(loss='mean_squared_error', optimizer='adam')  
+    # Return the model
+    return model
+
+def dnn2_gru2(length_of_sequences, batch_size = None, stateful = False):
+    """
+        Inputs:
+            length_of_sequences (an int): the number of y values in "x data".  This is determined
+                when the data is formatted
+            batch_size (an int): Default value is None.  See Keras documentation of SimpleRNN.
+            stateful (a boolean): Default value is False.  See Keras documentation of SimpleRNN.
+        Returns:
+            model (a Keras model): The recurrent neural network that is built and compiled by this
+                method
+        Builds and compiles a recurrent neural network with four hidden layers (two dense followed by
+        two GRU layers) and returns the model.
+    """    
+    # Number of neurons on the input/output layers and hidden layers
+    in_out_neurons = 1
+    hidden_neurons = 250
+    # Input layer
+    inp = Input(batch_shape=(batch_size, 
+                length_of_sequences, 
+                in_out_neurons)) 
+    # Hidden Dense (feedforward) layers
+    dnn = Dense(hidden_neurons/2, activation='relu', name='dnn')(inp)
+    dnn1 = Dense(hidden_neurons/2, activation='relu', name='dnn1')(dnn)
+    # Hidden GRU layers
+    rnn1 = GRU(hidden_neurons, 
+                    return_sequences=True,
+                    stateful = stateful,
+                    name="RNN1", use_bias=True)(dnn1)
+    rnn = GRU(hidden_neurons, 
+                    return_sequences=False,
+                    stateful = stateful,
+                    name="RNN", use_bias=True)(rnn1)
+    # Output layer
+    dens = Dense(in_out_neurons,name="dense")(rnn)
+    # Define the model
+    model = Model(inputs=[inp],outputs=[dens])
+    # Compile the mdoel
+    model.compile(loss='mean_squared_error', optimizer='adam')  
+    # Return the model
+    return model
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+
+# Generate the training data for the RNN, using a sequence of 2
+rnn_input, rnn_training = format_data(y_train, 2)
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+# Change the method name to reflect which network you want to use
+model = dnn2_gru2(length_of_sequences = 2)
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+
+# This section plots the training loss and the validation loss as a function of training iteration.
+# This is not required for analyzing the couple cluster data but can help determine if the network is
+# being overtrained.
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict more points of the data set
+test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
+
+# ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)
+# 
+# Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.
+
+# Check to make sure the data set is complete
+assert len(X_tot) == len(y_tot)
+
+# This is the number of points that will be used in as the training data
+dim=12
+
+# Separate the training data from the whole data set
+X_train = X_tot[:dim]
+y_train = y_tot[:dim]
+
+# Reshape the data for Keras specifications
+X_train = X_train.reshape((dim, 1))
+y_train = y_train.reshape((dim, 1))
+
+
+# Create a recurrent neural network in Keras and produce a summary of the 
+# machine learning model
+# Set the sequence length to 1 for regular data formatting 
+model = rnn(length_of_sequences = 1)
+model.summary()
+
+# Start the timer.  Want to time training+testing
+start = timer()
+# Fit the model using the training data genenerated above using 150 training iterations and a 5%
+# validation split.  Setting verbose to True prints information about each training iteration.
+hist = model.fit(X_train, y_train, batch_size=None, epochs=150, 
+                 verbose=True,validation_split=0.05)
+
+
+# This section plots the training loss and the validation loss as a function of training iteration.
+# This is not required for analyzing the couple cluster data but can help determine if the network is
+# being overtrained.
+for label in ["loss","val_loss"]:
+    plt.plot(hist.history[label],label=label)
+
+plt.ylabel("loss")
+plt.xlabel("epoch")
+plt.title("The final validation loss: {}".format(hist.history["val_loss"][-1]))
+plt.legend()
+plt.show()
+
+# Use the trained neural network to predict the remaining data points
+X_pred = X_tot[dim:]
+X_pred = X_pred.reshape((len(X_pred), 1))
+y_model = model.predict(X_pred)
+y_pred = np.concatenate((y_tot[:dim], y_model.flatten()))
+
+# Plot the known data set and the predicted data set.  The red box represents the region that was used
+# for the training data.
+fig, ax = plt.subplots()
+ax.plot(X_tot, y_tot, label="true", linewidth=3)
+ax.plot(X_tot, y_pred, 'g-.',label="predicted", linewidth=4)
+ax.legend()
+# Created a red region to represent the points used in the training data.
+ax.axvspan(X_tot[0], X_tot[dim], alpha=0.25, color='red')
+plt.show()
+
+# Stop the timer and calculate the total time needed.
+end = timer()
+print('Time: ', end-start)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Generative Models

    + +

    Generative models describe a class of statistical models that are a contrast +to discriminative models. Informally we say that generative models can +generate new data instances while discriminative models discriminate between +different kinds of data instances. A generative model could generate new photos +of animals that look like 'real' animals while a discriminative model could tell +a dog from a cat. More formally, given a data set \( x \) and a set of labels / +targets \( y \). Generative models capture the joint probability \( p(x, y) \), or +just \( p(x) \) if there are no labels, while discriminative models capture the +conditional probability \( p(y | x) \). Discriminative models generally try to draw +boundaries in the data space (often high dimensional), while generative models +try to model how data is placed throughout the space. +

    + +

    Note: this material is thanks to Linus Ekstrøm.

    + +









    +

    Generative Adversarial Networks

    + +

    Generative Adversarial Networks are a type of unsupervised machine learning +algorithm proposed by Goodfellow et. al +in 2014 (short and good article). +

    + +

    The simplest formulation of +the model is based on a game theoretic approach, zero sum game, where we pit +two neural networks against one another. We define two rival networks, one +generator \( g \), and one discriminator \( d \). The generator directly produces +samples +

    +$$ +\begin{equation} + x = g(z; \theta^{(g)}) +\label{_auto1} +\end{equation} +$$ + + +









    +

    Discriminator

    +

    The discriminator attempts to distinguish between samples drawn from the +training data and samples drawn from the generator. In other words, it tries to +tell the difference between the fake data produced by \( g \) and the actual data +samples we want to do prediction on. The discriminator outputs a probability +value given by +

    + +$$ +\begin{equation} + d(x; \theta^{(d)}) +\label{_auto2} +\end{equation} +$$ + +

    indicating the probability that \( x \) is a real training example rather than a +fake sample the generator has generated. The simplest way to formulate the +learning process in a generative adversarial network is a zero-sum game, in +which a function +

    + +$$ +\begin{equation} + v(\theta^{(g)}, \theta^{(d)}) +\label{_auto3} +\end{equation} +$$ + +

    determines the reward for the discriminator, while the generator gets the +conjugate reward +

    + +$$ +\begin{equation} + -v(\theta^{(g)}, \theta^{(d)}) +\label{_auto4} +\end{equation} +$$ + + +









    +

    Learning Process

    + +

    During learning both of the networks maximize their own reward function, so that +the generator gets better and better at tricking the discriminator, while the +discriminator gets better and better at telling the difference between the fake +and real data. The generator and discriminator alternate on which one trains at +one time (i.e. for one epoch). In other words, we keep the generator constant +and train the discriminator, then we keep the discriminator constant to train +the generator and repeat. It is this back and forth dynamic which lets GANs +tackle otherwise intractable generative problems. As the generator improves with + training, the discriminator's performance gets worse because it cannot easily + tell the difference between real and fake. If the generator ends up succeeding + perfectly, the the discriminator will do no better than random guessing i.e. + 50\%. This progression in the training poses a problem for the convergence + criteria for GANs. The discriminator feedback gets less meaningful over time, + if we continue training after this point then the generator is effectively + training on junk data which can undo the learning up to that point. Therefore, + we stop training when the discriminator starts outputting \( 1/2 \) everywhere. +

    + +









    +

    More about the Learning Process

    + +

    At convergence we have

    + +$$ +\begin{equation} + g^* = \underset{g}{\mathrm{argmin}}\hspace{2pt} + \underset{d}{\mathrm{max}}v(\theta^{(g)}, \theta^{(d)}) +\label{_auto5} +\end{equation} +$$ + +

    The default choice for \( v \) is

    +$$ +\begin{equation} + v(\theta^{(g)}, \theta^{(d)}) = \mathbb{E}_{x\sim p_\mathrm{data}}\log d(x) + + \mathbb{E}_{x\sim p_\mathrm{model}} + \log (1 - d(x)) +\label{_auto6} +\end{equation} +$$ + +

    The main motivation for the design of GANs is that the learning process requires +neither approximate inference (variational autoencoders for example) nor +approximation of a partition function. In the case where +

    +$$ +\begin{equation} + \underset{d}{\mathrm{max}}v(\theta^{(g)}, \theta^{(d)}) +\label{_auto7} +\end{equation} +$$ + +

    is convex in $\theta^{(g)} then the procedure is guaranteed to converge and is +asymptotically consistent +( Seth Lloyd on QuGANs ). +

    + +









    +

    Additional References

    +

    This is in +general not the case and it is possible to get situations where the training +process never converges because the generator and discriminator chase one +another around in the parameter space indefinitely. A much deeper discussion on +the currently open research problem of GAN convergence is available +here. To +anyone interested in learning more about GANs it is a highly recommended read. +Direct quote: "In this best-performing formulation, the generator aims to +increase the log probability that the discriminator makes a mistake, rather than +aiming to decrease the log probability that the discriminator makes the correct +prediction." Another interesting read +

    + +









    +

    Writing Our First Generative Adversarial Network

    +

    Let us now move on to actually implementing a GAN in tensorflow. We will study +the performance of our GAN on the MNIST dataset. This code is based on and +adapted from the +google tutorial +

    + +

    First we import our libraries

    + + + +
    +
    +
    +
    +
    + 
    import os
+import time
+import numpy as np
+import tensorflow as tf
+import matplotlib.pyplot as plt
+from tensorflow.keras import layers
+from tensorflow.keras.utils import plot_model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we define our hyperparameters and import our data the usual way

    + + + +
    +
    +
    +
    +
    + 
    BUFFER_SIZE = 60000
+BATCH_SIZE = 256
+EPOCHS = 30
+
+data = tf.keras.datasets.mnist.load_data()
+(train_images, train_labels), (test_images, test_labels) = data
+train_images = np.reshape(train_images, (train_images.shape[0],
+                                         28,
+                                         28,
+                                         1)).astype('float32')
+
+# we normalize between -1 and 1
+train_images = (train_images - 127.5) / 127.5
+training_dataset = tf.data.Dataset.from_tensor_slices(
+                      train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    MNIST and GANs

    + +

    Let's have a quick look

    + + + +
    +
    +
    +
    +
    + 
    plt.imshow(train_images[0], cmap='Greys')
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we define our two models. This is where the 'magic' happens. There are a +huge amount of possible formulations for both models. A lot of engineering and +trial and error can be done here to try to produce better performing models. For +more advanced GANs this is by far the step where you can 'make or break' a +model. +

    + +

    We start with the generator. As stated in the introductory text the generator +\( g \) upsamples from a random sample to the shape of what we want to predict. In +our case we are trying to predict MNIST images (\( 28\times 28 \) pixels). +

    + + + +
    +
    +
    +
    +
    + 
    def generator_model():
+    """
+    The generator uses upsampling layers tf.keras.layers.Conv2DTranspose() to
+    produce an image from a random seed. We start with a Dense layer taking this
+    random sample as an input and subsequently upsample through multiple
+    convolutional layers.
+    """
+
+    # we define our model
+    model = tf.keras.Sequential()
+
+
+    # adding our input layer. Dense means that every neuron is connected and
+    # the input shape is the shape of our random noise. The units need to match
+    # in some sense the upsampling strides to reach our desired output shape.
+    # we are using 100 random numbers as our seed
+    model.add(layers.Dense(units=7*7*BATCH_SIZE,
+                           use_bias=False,
+                           input_shape=(100, )))
+    # we normalize the output form the Dense layer
+    model.add(layers.BatchNormalization())
+    # and add an activation function to our 'layer'. LeakyReLU avoids vanishing
+    # gradient problem
+    model.add(layers.LeakyReLU())
+    model.add(layers.Reshape((7, 7, BATCH_SIZE)))
+    assert model.output_shape == (None, 7, 7, BATCH_SIZE)
+    # even though we just added four keras layers we think of everything above
+    # as 'one' layer
+
+    # next we add our upscaling convolutional layers
+    model.add(layers.Conv2DTranspose(filters=128,
+                                     kernel_size=(5, 5),
+                                     strides=(1, 1),
+                                     padding='same',
+                                     use_bias=False))
+    model.add(layers.BatchNormalization())
+    model.add(layers.LeakyReLU())
+    assert model.output_shape == (None, 7, 7, 128)
+
+    model.add(layers.Conv2DTranspose(filters=64,
+                                     kernel_size=(5, 5),
+                                     strides=(2, 2),
+                                     padding='same',
+                                     use_bias=False))
+    model.add(layers.BatchNormalization())
+    model.add(layers.LeakyReLU())
+    assert model.output_shape == (None, 14, 14, 64)
+
+    model.add(layers.Conv2DTranspose(filters=1,
+                                     kernel_size=(5, 5),
+                                     strides=(2, 2),
+                                     padding='same',
+                                     use_bias=False,
+                                     activation='tanh'))
+    assert model.output_shape == (None, 28, 28, 1)
+
+    return model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    And there we have our 'simple' generator model. Now we move on to defining our +discriminator model \( d \), which is a convolutional neural network based image +classifier. +

    + + + +
    +
    +
    +
    +
    + 
    def discriminator_model():
+    """
+    The discriminator is a convolutional neural network based image classifier
+    """
+
+    # we define our model
+    model = tf.keras.Sequential()
+    model.add(layers.Conv2D(filters=64,
+                            kernel_size=(5, 5),
+                            strides=(2, 2),
+                            padding='same',
+                            input_shape=[28, 28, 1]))
+    model.add(layers.LeakyReLU())
+    # adding a dropout layer as you do in conv-nets
+    model.add(layers.Dropout(0.3))
+
+
+    model.add(layers.Conv2D(filters=128,
+                            kernel_size=(5, 5),
+                            strides=(2, 2),
+                            padding='same'))
+    model.add(layers.LeakyReLU())
+    # adding a dropout layer as you do in conv-nets
+    model.add(layers.Dropout(0.3))
+
+    model.add(layers.Flatten())
+    model.add(layers.Dense(1))
+
+    return model
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Other Models

    +

    Let us take a look at our models. Note: double click images for bigger view.

    + + + +
    +
    +
    +
    +
    + 
    generator = generator_model()
+plot_model(generator, show_shapes=True, rankdir='LR')
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    discriminator = discriminator_model()
+plot_model(discriminator, show_shapes=True, rankdir='LR')
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we need a few helper objects we will use in training

    + + + +
    +
    +
    +
    +
    + 
    cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)
+generator_optimizer = tf.keras.optimizers.Adam(1e-4)
+discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    The first object, cross_entropy is our loss function and the two others are +our optimizers. Notice we use the same learning rate for both \( g \) and \( d \). This +is because they need to improve their accuracy at approximately equal speeds to +get convergence (not necessarily exactly equal). Now we define our loss +functions +

    + + + +
    +
    +
    +
    +
    + 
    def generator_loss(fake_output):
+    loss = cross_entropy(tf.ones_like(fake_output), fake_output)
+
+    return loss
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    def discriminator_loss(real_output, fake_output):
+    real_loss = cross_entropy(tf.ones_like(real_output), real_output)
+    fake_loss = cross_entropy(tf.zeros_liks(fake_output), fake_output)
+    total_loss = real_loss + fake_loss
+
+    return total_loss
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we define a kind of seed to help us compare the learning process over +multiple training epochs. +

    + + + +
    +
    +
    +
    +
    + 
    noise_dimension = 100
+n_examples_to_generate = 16
+seed_images = tf.random.normal([n_examples_to_generate, noise_dimension])
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Training Step

    + +

    Now we have everything we need to define our training step, which we will apply +for every step in our training loop. Notice the @tf.function flag signifying +that the function is tensorflow 'compiled'. Removing this flag doubles the +computation time. +

    + + + +
    +
    +
    +
    +
    + 
    @tf.function
+def train_step(images):
+    noise = tf.random.normal([BATCH_SIZE, noise_dimension])
+
+    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:
+        generated_images = generator(noise, training=True)
+
+        real_output = discriminator(images, training=True)
+        fake_output = discriminator(generated_images, training=True)
+
+        gen_loss = generator_loss(fake_output)
+        disc_loss = discriminator_loss(real_output, fake_output)
+
+    gradients_of_generator = gen_tape.gradient(gen_loss,
+                                            generator.trainable_variables)
+    gradients_of_discriminator = disc_tape.gradient(disc_loss,
+                                            discriminator.trainable_variables)
+    generator_optimizer.apply_gradients(zip(gradients_of_generator,
+                                            generator.trainable_variables))
+    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator,
+                                            discriminator.trainable_variables))
+
+    return gen_loss, disc_loss
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Next we define a helper function to produce an output over our training epochs +to see the predictive progression of our generator model. Note: I am including +this code here, but comment it out in the training loop. +

    + + +
    +
    +
    +
    +
    + 
    def generate_and_save_images(model, epoch, test_input):
+    # we're making inferences here
+    predictions = model(test_input, training=False)
+
+    fig = plt.figure(figsize=(4, 4))
+
+    for i in range(predictions.shape[0]):
+        plt.subplot(4, 4, i+1)
+        plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')
+        plt.axis('off')
+
+    plt.savefig(f'./images_from_seed_images/image_at_epoch_{str(epoch).zfill(3)}.png')
+    plt.close()
+    #plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Checkpoints

    +

    Setting up checkpoints to periodically save our model during training so that +everything is not lost even if the program were to somehow terminate while +training. +

    + + + +
    +
    +
    +
    +
    + 
    # Setting up checkpoints to save model during training
+checkpoint_dir = './training_checkpoints'
+checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt')
+checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,
+                            discriminator_optimizer=discriminator_optimizer,
+                            generator=generator,
+                            discriminator=discriminator)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we define our training loop

    + + + +
    +
    +
    +
    +
    + 
    def train(dataset, epochs):
+    generator_loss_list = []
+    discriminator_loss_list = []
+
+    for epoch in range(epochs):
+        start = time.time()
+
+        for image_batch in dataset:
+            gen_loss, disc_loss = train_step(image_batch)
+            generator_loss_list.append(gen_loss.numpy())
+            discriminator_loss_list.append(disc_loss.numpy())
+
+        #generate_and_save_images(generator, epoch + 1, seed_images)
+
+        if (epoch + 1) % 15 == 0:
+            checkpoint.save(file_prefix=checkpoint_prefix)
+
+        print(f'Time for epoch {epoch} is {time.time() - start}')
+
+    #generate_and_save_images(generator, epochs, seed_images)
+
+    loss_file = './data/lossfile.txt'
+    with open(loss_file, 'w') as outfile:
+        outfile.write(str(generator_loss_list))
+        outfile.write('\n')
+        outfile.write('\n')
+        outfile.write(str(discriminator_loss_list))
+        outfile.write('\n')
+        outfile.write('\n')
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    To train simply call this function. Warning: this might take a long time so +there is a folder of a pretrained network already included in the repository. +

    + + + +
    +
    +
    +
    +
    + 
    train(train_dataset, EPOCHS)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    And here is the result of training our model for 100 epochs

    + + +

    + +

    Now to avoid having to train and everything, which will take a while depending +on your computer setup we now load in the model which produced the above gif. +

    + + + +
    +
    +
    +
    +
    + 
    checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))
+restored_generator = checkpoint.generator
+restored_discriminator = checkpoint.discriminator
+
+print(restored_generator)
+print(restored_discriminator)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Exploring the Latent Space

    + +

    We have successfully loaded in our latest model. Let us now play around a bit +and see what kind of things we can learn about this model. Our generator takes +an array of 100 numbers. One idea can be to try to systematically change our +input. Let us try and see what we get +

    + + + +
    +
    +
    +
    +
    + 
    def generate_latent_points(number=100, scale_means=1, scale_stds=1):
+    latent_dim = 100
+    means = scale_means * tf.linspace(-1, 1, num=latent_dim)
+    stds = scale_stds * tf.linspace(-1, 1, num=latent_dim)
+    latent_space_value_range = tf.random.normal([number, latent_dim],
+                                                means,
+                                                stds,
+                                                dtype=tf.float64)
+
+    return latent_space_value_range
+
+def generate_images(latent_points):
+    # notice we set training to false because we are making inferences
+    generated_images = restored_generator.predict(latent_points)
+
+    return generated_images
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    def plot_result(generated_images, number=100):
+    # obviously this assumes sqrt number is an int
+    fig, axs = plt.subplots(int(np.sqrt(number)), int(np.sqrt(number)),
+                            figsize=(10, 10))
+
+    for i in range(int(np.sqrt(number))):
+        for j in range(int(np.sqrt(number))):
+            axs[i, j].imshow(generated_images[i*j], cmap='Greys')
+            axs[i, j].axis('off')
+
+    plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +
    +
    +
    +
    +
    + 
    generated_images = generate_images(generate_latent_points())
+plot_result(generated_images)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Getting Results

    +

    We see that the generator generates images that look like MNIST +numbers: \( 1, 4, 7, 9 \). Let's try to tweak it a bit more to see if we are able +to generate a similar plot where we generate every MNIST number. Let us now try +to 'move' a bit around in the latent space. Note: decrease the plot number if +these following cells take too long to run on your computer. +

    + + + +
    +
    +
    +
    +
    + 
    plot_number = 225
+
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=5,
+                                                          scale_stds=1))
+plot_result(generated_images, number=plot_number)
+
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=-5,
+                                                          scale_stds=1))
+plot_result(generated_images, number=plot_number)
+
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=1,
+                                                          scale_stds=5))
+plot_result(generated_images, number=plot_number)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Again, we have found something interesting. Moving around using our means +takes us from digit to digit, while moving around using our standard +deviations seem to increase the number of different digits! In the last image +above, we can barely make out every MNIST digit. Let us make on last plot using +this information by upping the standard deviation of our Gaussian noises. +

    + + + +
    +
    +
    +
    +
    + 
    plot_number = 400
+generated_images = generate_images(generate_latent_points(number=plot_number,
+                                                          scale_means=1,
+                                                          scale_stds=10))
+plot_result(generated_images, number=plot_number)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    A pretty cool result! We see that our generator indeed has learned a +distribution which qualitatively looks a whole lot like the MNIST dataset. +

    + +









    +

    Interpolating Between MNIST Digits

    +

    Another interesting way to explore the latent space of our generator model is by +interpolating between the MNIST digits. This section is largely based on +this excellent blogpost +by Jason Brownlee. +

    + +

    So let us start by defining a function to interpolate between two points in the +latent space. +

    + + + +
    +
    +
    +
    +
    + 
    def interpolation(point_1, point_2, n_steps=10):
+    ratios = np.linspace(0, 1, num=n_steps)
+    vectors = []
+    for i, ratio in enumerate(ratios):
+        vectors.append(((1.0 - ratio) * point_1 + ratio * point_2))
+
+    return tf.stack(vectors)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we have all we need to do our interpolation analysis.

    + + + +
    +
    +
    +
    +
    + 
    plot_number = 100
+latent_points = generate_latent_points(number=plot_number)
+results = None
+for i in range(0, 2*np.sqrt(plot_number), 2):
+    interpolated = interpolation(latent_points[i], latent_points[i+1])
+    generated_images = generate_images(interpolated)
+
+    if results is None:
+        results = generated_images
+    else:
+        results = tf.stack((results, generated_images))
+
+plot_results(results, plot_number)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Basic ideas of the Principal Component Analysis (PCA)

    + +

    The principal component analysis deals with the problem of fitting a +low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than +the total dimension \( D \) of the problem at hand (our data +set). Mathematically it can be formulated as a statistical problem or +a geometric problem. In our discussion of the theorem for the +classical PCA, we will stay with a statistical approach. +Historically, the PCA was first formulated in a statistical setting in order to estimate the principal component of a multivariate random variable. +

    + +

    We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)

    +
      +
    • Each data point is determined by \( p \) extrinsic (measurement) variables
      • +
    • We may want to ask the following question: Are there fewer intrinsic variables (say \( d < < p \)) that still approximately describe the data?
      • +
    • If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.
      • +
    +

    A good read is for example Vidal, Ma and Sastry.

    + +









    +

    Introducing the Covariance and Correlation functions

    + +

    Before we discuss the PCA theorem, we need to remind ourselves about +the definition of the covariance and the correlation function. These are quantities +

    + +

    Suppose we have defined two vectors +\( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as +

    +$$ +\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\ + \end{bmatrix}, +$$ + +

    where for example

    +$$ +\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). +$$ + +

    With this definition and recalling that the variance is defined as

    +$$ +\mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2, +$$ + +

    we can rewrite the covariance matrix as

    +$$ +\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\ + \end{bmatrix}. +$$ + + +









    +

    More on the covariance

    +

    The covariance takes values between zero and infinity and may thus +lead to problems with loss of numerical precision for particularly +large values. It is common to scale the covariance matrix by +introducing instead the correlation matrix defined via the so-called +correlation function +

    + +$$ +\mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}. +$$ + +

    The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] +\in [-1,1] \). This avoids eventual problems with too large values. We +can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) +and \( \boldsymbol{y} \) as +

    + +$$ +\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\ + \mathrm{corr}[\boldsymbol{y},\boldsymbol{x}] & 1 \\ + \end{bmatrix}, +$$ + +

    In the above example this is the function we constructed using pandas.

    + +









    +

    Reminding ourselves about Linear Regression

    +

    In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression +we defined the design/feature matrix \( \boldsymbol{X} \) as +

    + +$$ +\boldsymbol{X}=\begin{bmatrix} +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\ +\dots & \dots & \dots & \dots \dots & \dots \\ +x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ +\end{bmatrix}, +$$ + +

    with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the +entries \( n \) being the row elements. +We can rewrite the design/feature matrix in terms of its column vectors as +

    +$$ +\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix}, +$$ + +

    with a given vector

    +$$ +\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. +$$ + + +









    +

    Simple Example

    +

    With these definitions, we can now rewrite our \( 2\times 2 \) +correlation/covariance matrix in terms of a moe general design/feature +matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) +covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) +

    + +$$ +\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} +\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +\mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_1] & \mathrm{var}[\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & \mathrm{var}[\boldsymbol{x}_{p-1}]\\ +\end{bmatrix}, +$$ + + +









    +

    The Correlation Matrix

    + +

    and the correlation matrix

    +$$ +\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} +1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +\mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +\mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\ +\end{bmatrix}, +$$ + + +









    +

    Numpy Functionality

    + +

    The Numpy function np.cov calculates the covariance elements using +the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have +the exact mean values. The following simple function uses the +np.vstack function which takes each vector of dimension \( 1\times n \) +and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +

    + +$$ +\boldsymbol{W}^T = \begin{bmatrix} x_0 & y_0 \\ + x_1 & y_1 \\ + x_2 & y_2\\ + \dots & \dots \\ + x_{n-2} & y_{n-2}\\ + x_{n-1} & y_{n-1} & + \end{bmatrix}, +$$ + +

    which in turn is converted into into the \( 2\times 2 \) covariance matrix +\( \boldsymbol{C} \) via the Numpy function np.cov(). We note that we can also calculate +the mean value of each set of samples \( \boldsymbol{x} \) etc using the Numpy +function np.mean(x). We can also extract the eigenvalues of the +covariance matrix through the np.linalg.eig() function. +

    + + + +
    +
    +
    +
    +
    + 
    # Importing various packages
+import numpy as np
+n = 100
+x = np.random.normal(size=n)
+print(np.mean(x))
+y = 4+3*x+np.random.normal(size=n)
+print(np.mean(y))
+W = np.vstack((x, y))
+C = np.cov(W)
+print(C)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Correlation Matrix again

    + +

    The previous example can be converted into the correlation matrix by +simply scaling the matrix elements with the variances. We should also +subtract the mean values for each column. This leads to the following +code which sets up the correlations matrix for the previous example in +a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors). +

    + + + +
    +
    +
    +
    +
    + 
    import numpy as np
+n = 100
+# define two vectors                                                                                           
+x = np.random.random(size=n)
+y = 4+3*x+np.random.normal(size=n)
+#scaling the x and y vectors                                                                                   
+x = x - np.mean(x)
+y = y - np.mean(y)
+variance_x = np.sum(x@x)/n
+variance_y = np.sum(y@y)/n
+print(variance_x)
+print(variance_y)
+cov_xy = np.sum(x@y)/n
+cov_xx = np.sum(x@x)/n
+cov_yy = np.sum(y@y)/n
+C = np.zeros((2,2))
+C[0,0]= cov_xx/variance_x
+C[1,1]= cov_yy/variance_y
+C[0,1]= cov_xy/np.sqrt(variance_y*variance_x)
+C[1,0]= C[0,1]
+print(C)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We see that the matrix elements along the diagonal are one as they +should be and that the matrix is symmetric. Furthermore, diagonalizing +this matrix we easily see that it is a positive definite matrix. +

    + +

    The above procedure with numpy can be made more compact if we use pandas.

    + +









    +

    Using Pandas

    + +

    We whow here how we can set up the correlation matrix using pandas, as done in this simple code

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import pandas as pd
+n = 10
+x = np.random.normal(size=n)
+x = x - np.mean(x)
+y = 4+3*x+np.random.normal(size=n)
+y = y - np.mean(y)
+X = (np.vstack((x, y))).T
+print(X)
+Xpd = pd.DataFrame(X)
+print(Xpd)
+correlation_matrix = Xpd.corr()
+print(correlation_matrix)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    And then the Franke Function

    + +

    We expand this model to the Franke function discussed above.

    + + + +
    +
    +
    +
    +
    + 
    # Common imports
+import numpy as np
+import pandas as pd
+
+
+def FrankeFunction(x,y):
+	term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
+	term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
+	term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
+	term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
+	return term1 + term2 + term3 + term4
+
+
+def create_X(x, y, n ):
+	if len(x.shape) > 1:
+		x = np.ravel(x)
+		y = np.ravel(y)
+
+	N = len(x)
+	l = int((n+1)*(n+2)/2)		# Number of elements in beta
+	X = np.ones((N,l))
+
+	for i in range(1,n+1):
+		q = int((i)*(i+1)/2)
+		for k in range(i+1):
+			X[:,q+k] = (x**(i-k))*(y**k)
+
+	return X
+
+
+# Making meshgrid of datapoints and compute Franke's function
+n = 4
+N = 100
+x = np.sort(np.random.uniform(0, 1, N))
+y = np.sort(np.random.uniform(0, 1, N))
+z = FrankeFunction(x, y)
+X = create_X(x, y, n=n)    
+
+Xpd = pd.DataFrame(X)
+# subtract the mean values and set up the covariance matrix
+Xpd = Xpd - Xpd.mean()
+covariance_matrix = Xpd.cov()
+print(covariance_matrix)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We note here that the covariance is zero for the first rows and +columns since all matrix elements in the design matrix were set to one +(we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept +and wee can simply +drop these elements and construct a correlation +matrix without them by centering our matrix elements by subtracting the mean of each column. +

    + +









    +

    Lnks with the Design Matrix

    + +

    We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as

    +$$ +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. +$$ + +

    To see this let us simply look at a design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{2\times 2} \)

    +$$ +\boldsymbol{X}=\begin{bmatrix} +x_{00} & x_{01}\\ +x_{10} & x_{11}\\ +\end{bmatrix}=\begin{bmatrix} +\boldsymbol{x}_{0} & \boldsymbol{x}_{1}\\ +\end{bmatrix}. +$$ + + +









    +

    Computing the Expectation Values

    + +

    If we then compute the expectation value

    +$$ +\mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\begin{bmatrix} +x_{00}^2+x_{01}^2 & x_{00}x_{10}+x_{01}x_{11}\\ +x_{10}x_{00}+x_{11}x_{01} & x_{10}^2+x_{11}^2\\ +\end{bmatrix}, +$$ + +

    which is just

    +$$ +\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]=\begin{bmatrix} \mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] \\ + \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] \\ + \end{bmatrix}, +$$ + +

    where we wrote $$\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]$$ to indicate that this the covariance of the vectors \( \boldsymbol{x} \) of the design/feature matrix \( \boldsymbol{X} \).

    + +

    It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \).

    + +









    +

    Towards the PCA theorem

    + +

    We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as

    +$$ +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. +$$ + +

    Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices \( \boldsymbol{S} \). +These matrices are defined as \( \boldsymbol{S}\in {\mathbb{R}}^{p\times p} \) and obey the orthogonality requirements \( \boldsymbol{S}\boldsymbol{S}^T=\boldsymbol{S}^T\boldsymbol{S}=\boldsymbol{I} \). The matrix can be written out in terms of the column vectors \( \boldsymbol{s}_i \) as \( \boldsymbol{S}=[\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \) and \( \boldsymbol{s}_i \in {\mathbb{R}}^{p} \). +

    + +

    Assume also that there is a transformation \( \boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}=\boldsymbol{C}[\boldsymbol{y}] \) such that the new matrix \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal with elements \( [\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}] \).

    + +

    That is we have

    +$$ +\boldsymbol{C}[\boldsymbol{y}] = \mathbb{E}[\boldsymbol{S}^T\boldsymbol{X}^T\boldsymbol{X}T\boldsymbol{S}]=\boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}, +$$ + +

    since the matrix \( \boldsymbol{S} \) is not a data dependent matrix. Multiplying with \( \boldsymbol{S} \) from the left we have

    +$$ +\boldsymbol{S}\boldsymbol{C}[\boldsymbol{y}] = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}, +$$ + +

    and since \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal we have for a given eigenvalue \( i \) of the covariance matrix that

    + +$$ +\boldsymbol{S}_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}_i. +$$ + + +









    +

    More on the PCA Theorem

    + +

    In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is +\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \). +

    + +

    The eigenvalues tell us then how much we need to stretch the +corresponding eigenvectors. Dimensions with large eigenvalues have +thus large variations (large variance) and define therefore useful +dimensions. The data points are more spread out in the direction of +these eigenvectors. Smaller eigenvalues mean on the other hand that +the corresponding eigenvectors are shrunk accordingly and the data +points are tightly bunched together and there is not much variation in +these specific directions. Hopefully then we could leave it out +dimensions where the eigenvalues are very small. If \( p \) is very large, +we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) +features/predictors. +

    + +









    +

    The Algorithm before theorem

    + +

    Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.

    +
      +
    • Set up the datapoints for the design/feature matrix \( \boldsymbol{X} \) with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) referring to the column numbers and the entries \( n \) being the row elements.
      • +
    +$$ +\boldsymbol{X}=\begin{bmatrix} +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\ +\dots & \dots & \dots & \dots \dots & \dots \\ +x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ +\end{bmatrix}, +$$ + +
      +
    • Center the data by subtracting the mean value for each column. This leads to a new matrix \( \boldsymbol{X}\rightarrow \overline{\boldsymbol{X}} \).
      • +
    • Compute then the covariance/correlation matrix \( \mathbb{E}[\overline{\boldsymbol{X}}^T\overline{\boldsymbol{X}}] \).
      • +
    • Find the eigenpairs of \( \boldsymbol{C} \) with eigenvalues \( [\lambda_0,\lambda_1,\dots,\lambda_{p-1}] \) and eigenvectors \( [\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \).
      • +
    • Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.
      • +
    • Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.
      • +
    +









    +

    Writing our own PCA code

    + +

    We will use a simple example first with two-dimensional data +drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below): +

    +$$ +\mu = (-1,2) \qquad \Sigma = \begin{bmatrix} 4 & 2 \\ +2 & 2 +\end{bmatrix} +$$ + +

    Note that the mean refers to each column of data. +We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from +this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values. +

    + +









    +

    Implementing it

    +

    The following Python code aids in setting up the data and writing out the design matrix. +Note that the function multivariate returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \). +

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from IPython.display import display
+n = 10000
+mean = (-1, 2)
+cov = [[4, 2], [2, 2]]
+X = np.random.multivariate_normal(mean, cov, n)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Now we are going to implement the PCA algorithm. We will break it down into various substeps.

    + +









    +

    First Step

    + +

    The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is

    +$$ +\mu_n = \frac{1}{n} \sum_{i=1}^n x_i +$$ + +

    and the mean-centered data \( \bar{X} = \{ \bar{x}_1, \ldots, \bar{x}_n \} \) takes the form

    +$$ +\bar{x}_i = x_i - \mu_n. +$$ + +

    When you are done with these steps, print out \( \mu_n \) to verify it is +close to \( \mu \) and plot your mean centered data to verify it is +centered at the origin! +The following code elements perform these operations using pandas or using our own functionality for doing so. The latter, using numpy is rather simple through the mean() function. +

    + + +
    +
    +
    +
    +
    + 
    df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Scaling

    +

    Alternatively, we could use the functions we discussed +earlier for scaling the data set. That is, we could have used the +StandardScaler function in Scikit-Learn, a function which ensures +that for each feature/predictor we study the mean value is zero and +the variance is one (every column in the design/feature matrix). You +would then not get the same results, since we divide by the +variance. The diagonal covariance matrix elements will then be one, +while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our +specific case. +

    + +









    +

    Centered Data

    + +

    Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation

    +$$ +\begin{equation*} +\Sigma_n = \frac{1}{n-1} \sum_{i=1}^n \bar{x}_i^T \bar{x}_i = \frac{1}{n-1} \sum_{i=1}^n (x_i - \mu_n)^T (x_i - \mu_n) +\end{equation*} +$$ + +

    where the data points \( x_i \in \mathbb{R}^p \) (here in this example \( p = 2 \)) are column vectors and \( x^T \) is the transpose of \( x \). +We can write our own code or simply use either the functionaly of numpy or that of pandas, as follows +

    + + +
    +
    +
    +
    +
    + 
    print(df.cov())
+print(np.cov(X_centered.T))
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Note that the way we define the covariance matrix here has a factor \( n-1 \) instead of \( n \). This is included in the cov() function by numpy and pandas. +Our own code here is not very elegant and asks for obvious improvements. It is tailored to this specific \( 2\times 2 \) covariance matrix. +

    + + +
    +
    +
    +
    +
    + 
    # extract the relevant columns from the centered design matrix of dim n x 2
+x = X_centered[:,0]
+y = X_centered[:,1]
+Cov = np.zeros((2,2))
+Cov[0,1] = np.sum(x.T@y)/(n-1.0)
+Cov[0,0] = np.sum(x.T@x)/(n-1.0)
+Cov[1,1] = np.sum(y.T@y)/(n-1.0)
+Cov[1,0]= Cov[0,1]
+print("Centered covariance using own code")
+print(Cov)
+plt.plot(x, y, 'x')
+plt.axis('equal')
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Exploring

    + +

    Depending on the number of points \( n \), we will get results that are close to the covariance values defined above. +The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed. +

    + +









    +

    Diagonalize the sample covariance matrix to obtain the principal components

    + +

    Now we are ready to solve for the principal components! To do so we +diagonalize the sample covariance matrix \( \Sigma \). We can use the +function np.linalg.eig to do so. It will return the eigenvalues and +eigenvectors of \( \Sigma \). Once we have these we can perform the +following tasks: +

    + +
      +
    • We compute the percentage of the total variance captured by the first principal component
      • +
    • We plot the mean centered data and lines along the first and second principal components
      • +
    • Then we project the mean centered data onto the first and second principal components, and plot the projected data.
      • +
    • Finally, we approximate the data as
      • +
    +$$ +\begin{equation*} +x_i \approx \tilde{x}_i = \mu_n + \langle x_i, v_0 \rangle v_0 +\end{equation*} +$$ + +

    where \( v_0 \) is the first principal component.

    + +









    +

    Collecting all Steps

    + +

    Collecting all these steps we can write our own PCA function and +compare this with the functionality included in Scikit-Learn. +

    + +

    The code here outlines some of the elements we could include in the +analysis. Feel free to extend upon this in order to address the above +questions. +

    + + + +
    +
    +
    +
    +
    + 
    # diagonalize and obtain eigenvalues, not necessarily sorted
+EigValues, EigVectors = np.linalg.eig(Cov)
+# sort eigenvectors and eigenvalues
+#permute = EigValues.argsort()
+#EigValues = EigValues[permute]
+#EigVectors = EigVectors[:,permute]
+print("Eigenvalues of Covariance matrix")
+for i in range(2):
+    print(EigValues[i])
+FirstEigvector = EigVectors[:,0]
+SecondEigvector = EigVectors[:,1]
+print("First eigenvector")
+print(FirstEigvector)
+print("Second eigenvector")
+print(SecondEigvector)
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2Dsl = pca.fit_transform(X)
+print("Eigenvector of largest eigenvalue")
+print(pca.components_.T[:, 0])
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    This code does not contain all the above elements, but it shows how we can use Scikit-Learn to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then?

    + +









    +

    Classical PCA Theorem

    + +

    We assume now that we have a design matrix \( \boldsymbol{X} \) which has been +centered as discussed above. For the sake of simplicity we skip the +overline symbol. The matrix is defined in terms of the various column +vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension +\( \boldsymbol{x}\in {\mathbb{R}}^{n} \). +

    + +

    The PCA theorem states that minimizing the above reconstruction error +corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which +diagonalizes the empirical covariance(correlation) matrix. The optimal +low-dimensional encoding of the data is then given by a set of vectors +\( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the +orthogonal projection of the data onto the columns spanned by the +eigenvectors of the covariance(correlations matrix). +

    + +









    +

    The PCA Theorem

    + +

    To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as

    + +

    We are almost there, we have obtained a relation between minimizing +the reconstruction error and the variance and the covariance +matrix. Minimizing the error is equivalent to maximizing the variance +of the projected data. +

    + +

    We could trivially maximize the variance of the projection (and +thereby minimize the error in the reconstruction function) by letting +the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we +want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by +\( \vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a +Lagrange multiplier we can then in turn maximize +

    + +$$ +J(\boldsymbol{w}_0)= \boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0+\lambda_0(1-\boldsymbol{w}_0^T\boldsymbol{w}_0). +$$ + +

    Taking the derivative with respect to \( \boldsymbol{w}_0 \) we obtain

    + +$$ +\frac{\partial J(\boldsymbol{w}_0)}{\partial \boldsymbol{w}_0}= 2\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0-2\lambda_0\boldsymbol{w}_0=0, +$$ + +

    meaning that

    +$$ +\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0\boldsymbol{w}_0. +$$ + +

    The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is

    +$$ +\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0. +$$ + +

    If we want to maximize the variance (minimize the construction error) +we simply pick the eigenvector of the covariance matrix with the +largest eigenvalue. This establishes the link between the minimization +of the reconstruction function \( J \) in terms of an orthogonal matrix +and the maximization of the variance and thereby the covariance of our +observations encoded in the design/feature matrix \( \boldsymbol{X} \). +

    + +

    The proof +for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) can be +established by applying the above arguments and using the fact that +our basis of eigenvectors is orthogonal, see Murphy chapter +12.2. The +discussion in chapter 12.2 of Murphy's text has also a nice link with +the Singular Value Decomposition theorem. For categorical data, see +chapter 12.4 and discussion therein. +

    + +

    For more details, see for example Vidal, Ma and Sastry, chapter 2.

    + +









    + + +

    For a detailed demonstration of the geometric interpretation, see Vidal, Ma and Sastry, section 2.1.2.

    + +

    Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm. +First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it. +

    + +

    The following Python code uses NumPy’s svd() function to obtain all the principal components of the +training set, then extracts the first two principal components. First we center the data using either pandas or our own code +

    + + +
    +
    +
    +
    +
    + 
    import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 vanilla matrix 
+rows = 10
+cols = 5
+X = np.random.randn(rows,cols)
+df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+display(df)
+
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+# Then check the difference between pandas and our own set up
+print(X_centered-df)
+#Now we do an SVD
+U, s, V = np.linalg.svd(X_centered)
+c1 = V.T[:, 0]
+c2 = V.T[:, 1]
+W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+print(X2D)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering +the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t +forget to center the data first. +

    + +

    Once you have identified all the principal components, you can reduce the dimensionality of the dataset +down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components. +Selecting this hyperplane ensures that the projection will preserve as much variance as possible. +

    + + +
    +
    +
    +
    +
    + 
    W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    PCA and scikit-learn

    + +

    Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The +following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note +that it automatically takes care of centering the data): +

    + + +
    +
    +
    +
    +
    + 
    #thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2D = pca.fit_transform(X)
+print(X2D)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    After fitting the PCA transformer to the dataset, you can access the principal components using the +components variable (note that it contains the PCs as horizontal vectors, so, for example, the first +principal component is equal to +

    + + +
    +
    +
    +
    +
    + 
    pca.components_.T[:, 0]
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Another very useful piece of information is the explained variance ratio of each principal component, +available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s +variance that lies along the axis of each principal component. +

    + +









    +

    Back to the Cancer Data

    +

    We can now repeat the above but applied to real data, in this case our breast cancer data. +Here we compute performance scores on the training data using logistic regression. +

    + + +
    +
    +
    +
    +
    + 
    import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.model_selection import  train_test_split 
+from sklearn.datasets import load_breast_cancer
+from sklearn.linear_model import LogisticRegression
+cancer = load_breast_cancer()
+
+X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
+
+logreg = LogisticRegression()
+logreg.fit(X_train, y_train)
+print("Train set accuracy from Logistic Regression: {:.2f}".format(logreg.score(X_train,y_train)))
+# We scale the data
+from sklearn.preprocessing import StandardScaler
+scaler = StandardScaler()
+scaler.fit(X_train)
+X_train_scaled = scaler.transform(X_train)
+X_test_scaled = scaler.transform(X_test)
+# Then perform again a log reg fit
+logreg.fit(X_train_scaled, y_train)
+print("Train set accuracy scaled data: {:.2f}".format(logreg.score(X_train_scaled,y_train)))
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2D_train = pca.fit_transform(X_train_scaled)
+# and finally compute the log reg fit and the score on the training data	
+logreg.fit(X2D_train,y_train)
+print("Train set accuracy scaled and PCA data: {:.2f}".format(logreg.score(X2D_train,y_train)))
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.

    + +

    Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to +choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%). +Unless, of course, you are reducing dimensionality for data visualization — in that case you will +generally want to reduce the dimensionality down to 2 or 3. +The following code computes PCA without reducing dimensionality, then computes the minimum number +of dimensions required to preserve 95% of the training set’s variance: +

    + + +
    +
    +
    +
    +
    + 
    pca = PCA()
+pca.fit(X)
+cumsum = np.cumsum(pca.explained_variance_ratio_)
+d = np.argmax(cumsum >= 0.95) + 1
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead +of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be +a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve: +

    + + +
    +
    +
    +
    +
    + 
    pca = PCA(n_components=0.95)
+X_reduced = pca.fit_transform(X)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Incremental PCA

    + +

    One problem with the preceding implementation of PCA is that it requires the whole training set to fit in +memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have +been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch +at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new +instances arrive). +

    +

    Randomized PCA

    + +

    Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic +algorithm that quickly finds an approximation of the first d principal components. Its computational +complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the +previous algorithms when \( d \) is much smaller than \( n \). +

    +

    Kernel PCA

    + +

    The kernel trick is a mathematical technique that implicitly maps instances into a +very high-dimensional space (called the feature space), enabling nonlinear classification and regression +with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature +space corresponds to a complex nonlinear decision boundary in the original space. +It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear +projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at +preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a +twisted manifold. +For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an +

    + + +
    +
    +
    +
    +
    + 
    from sklearn.decomposition import KernelPCA
+rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04)
+X_reduced = rbf_pca.fit_transform(X)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + + +









    +

    Other techniques

    + +

    There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.

    + +

    Here are some of the most popular:

    +
      +
    • Multidimensional Scaling (MDS) reduces dimensionality while trying to preserve the distances between the instances.
      • +
    • Isomap creates a graph by connecting each instance to its nearest neighbors, then reduces dimensionality while trying to preserve the geodesic distances between the instances.
      • +
    • t-Distributed Stochastic Neighbor Embedding (t-SNE) reduces dimensionality while trying to keep similar instances close and dissimilar instances apart. It is mostly used for visualization, in particular to visualize clusters of instances in high-dimensional space (e.g., to visualize the MNIST images in 2D).
      • +
    • Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it learns the most discriminative axes between the classes, and these axes can then be used to define a hyperplane onto which to project the data. The benefit is that the projection will keep classes as far apart as possible, so LDA is a good technique to reduce dimensionality before running another classification algorithm such as a Support Vector Machine (SVM) classifier discussed in the SVM lectures.
      • +
    + +
    + © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license +
    + + +