From 5c0e2988afae31b36d476172b76717b32e303ef6 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Fri, 29 Oct 2021 06:22:31 +0200 Subject: [PATCH] small update --- doc/LectureNotes/schedule.md | 1 + doc/pub/week43/html/week43-bs.html | 155 +++++- doc/pub/week43/html/week43-reveal.html | 104 +++- doc/pub/week43/html/week43-solarized.html | 128 ++++- doc/pub/week43/html/week43.html | 128 ++++- doc/pub/week43/ipynb/ipynb-week43-src.tar.gz | Bin 29 -> 29 bytes doc/pub/week43/ipynb/week43.ipynb | 507 +++++++++++-------- doc/src/week43/week43.do.txt | 70 ++- 8 files changed, 818 insertions(+), 275 deletions(-) diff --git a/doc/LectureNotes/schedule.md b/doc/LectureNotes/schedule.md index e65587b6a..5f08e2068 100644 --- a/doc/LectureNotes/schedule.md +++ b/doc/LectureNotes/schedule.md @@ -134,6 +134,7 @@ For the reading assignments we use the following abbreviations: ### Week 43 October 25-29 - Lab Wednesday: Work on project 2 - Lecture Thursday: Recurrent Neural Networks + - Video of Lecture at https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureOctober28.mp4?vrtx=view-as-webpage - Lecture Friday: Recurrent Neural Networks and principal component analysis (PCA) - Reading recommendations: - See lecture notes for week 43 at https://compphysics.github.io/MachineLearning/doc/web/course.html. diff --git a/doc/pub/week43/html/week43-bs.html b/doc/pub/week43/html/week43-bs.html index c9b21698a..910521043 100644 --- a/doc/pub/week43/html/week43-bs.html +++ b/doc/pub/week43/html/week43-bs.html @@ -97,19 +97,48 @@ doconce format html week43.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'introducing-the-covariance-and-correlation-functions'), - ('Correlation Matrix', 2, None, 'correlation-matrix'), + ('More on the covariance', 2, None, 'more-on-the-covariance'), + ('Reminding ourselves about Linear Regression', + 2, + None, + 'reminding-ourselves-about-linear-regression'), + ('Simple Example', 2, None, 'simple-example'), + ('The Correlation Matrix', 2, None, 'the-correlation-matrix'), + ('Numpy Functionality', 2, None, 'numpy-functionality'), + ('Correlation Matrix again', 2, None, 'correlation-matrix-again'), + ('Using Pandas', 2, None, 'using-pandas'), + ('And then the Franke Function', + 2, + None, + 'and-then-the-franke-function'), + ('Lnks with the Design Matrix', + 2, + None, + 'lnks-with-the-design-matrix'), + ('Computing the Expectation Values', + 2, + None, + 'computing-the-expectation-values'), ('Towards the PCA theorem', 2, None, 'towards-the-pca-theorem'), + ('More on the PCA Theorem', 2, None, 'more-on-the-pca-theorem'), ('The Algorithm before the Theorem', - 3, + 2, None, 'the-algorithm-before-the-theorem'), - ('Writing our own PCA code', 3, None, 'writing-our-own-pca-code'), + ('Writing our own PCA code', 2, None, 'writing-our-own-pca-code'), + ('Implementing it', 2, None, 'implementing-it'), + ('First Step', 2, None, 'first-step'), + ('Scaling', 2, None, 'scaling'), + ('Centered Data', 2, None, 'centered-data'), + ('Exploring', 2, None, 'exploring'), ('Diagonalize the sample covariance matrix to obtain the ' 'principal components', - 3, + 2, None, 'diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components'), + ('Collecting all Steps', 2, None, 'collecting-all-steps'), ('Classical PCA Theorem', 2, None, 'classical-pca-theorem'), + ('The PCA Theorem', 2, None, 'the-pca-theorem'), ('Geometric Interpretation and link with Singular Value ' 'Decomposition', 2, @@ -117,7 +146,7 @@ doconce format html week43.do.txt --html_style=bootstrap --pygments_html_style=d 'geometric-interpretation-and-link-with-singular-value-decomposition'), ('PCA and scikit-learn', 2, None, 'pca-and-scikit-learn'), ('Back to the Cancer Data', 2, None, 'back-to-the-cancer-data'), - ('Incremental PCA', 3, None, 'incremental-pca'), + ('Incremental PCA', 2, None, 'incremental-pca'), ('Randomized PCA', 3, None, 'randomized-pca'), ('Kernel PCA', 3, None, 'kernel-pca'), ('Other techniques', 2, None, 'other-techniques')]} @@ -183,16 +212,33 @@ MathJax.Hub.Config({
  • Interpolating Between MNIST Digits
  • Basic ideas of the Principal Component Analysis (PCA)
  • Introducing the Covariance and Correlation functions
    • -
  • Correlation Matrix
    • +
  • More on the covariance
    • +
  • Reminding ourselves about Linear Regression
    • +
  • Simple Example
    • +
  • The Correlation Matrix
    • +
  • Numpy Functionality
    • +
  • Correlation Matrix again
    • +
  • Using Pandas
    • +
  • And then the Franke Function
    • +
  • Lnks with the Design Matrix
    • +
  • Computing the Expectation Values
  • Towards the PCA theorem
    • -
  •    The Algorithm before the Theorem
    • -
  •    Writing our own PCA code
    • -
  •    Diagonalize the sample covariance matrix to obtain the principal components
    • +
  • More on the PCA Theorem
    • +
  • The Algorithm before the Theorem
    • +
  • Writing our own PCA code
    • +
  • Implementing it
    • +
  • First Step
    • +
  • Scaling
    • +
  • Centered Data
    • +
  • Exploring
    • +
  • Diagonalize the sample covariance matrix to obtain the principal components
    • +
  • Collecting all Steps
  • Classical PCA Theorem
    • +
  • The PCA Theorem
  • Geometric Interpretation and link with Singular Value Decomposition
  • PCA and scikit-learn
  • Back to the Cancer Data
    • -
  •    Incremental PCA
    • +
  • Incremental PCA
  •    Randomized PCA
  •    Kernel PCA
  • Other techniques
    • @@ -224,7 +270,7 @@ MathJax.Hub.Config({
    -

    Oct 28, 2021

    +

    Oct 29, 2021


    @@ -2229,6 +2275,9 @@ $$ \end{bmatrix}. $$ + + +

    More on the covariance

    The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by @@ -2254,6 +2303,8 @@ $$

    In the above example this is the function we constructed using pandas.

    + +

    Reminding ourselves about Linear Regression

    In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as

    @@ -2282,8 +2333,11 @@ $$ \boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. $$ + + +

    Simple Example

    With these definitions, we can now rewrite our \( 2\times 2 \) -correaltion/covariance matrix in terms of a moe general design/feature +correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)

    @@ -2299,6 +2353,10 @@ $$ \end{bmatrix}, $$ + + +

    The Correlation Matrix

    +

    and the correlation matrix

    $$ \boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} @@ -2311,6 +2369,10 @@ $$ \end{bmatrix}, $$ + + +

    Numpy Functionality

    +

    The Numpy function np.cov calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values. The following simple function uses the @@ -2369,7 +2431,7 @@ C = np.c -

    Correlation Matrix

    +

    Correlation Matrix again

    The previous example can be converted into the correlation matrix by simply scaling the matrix elements with the variances. We should also @@ -2428,6 +2490,9 @@ this matrix we easily see that it is a positive definite matrix.

    The above procedure with numpy can be made more compact if we use pandas.

    + +

    Using Pandas

    +

    We whow here how we can set up the correlation matrix using pandas, as done in this simple code

    @@ -2464,6 +2529,10 @@ correlation_matrix = XpdAnd then the Franke Function +

    We expand this model to the Franke function discussed above.

    @@ -2536,9 +2605,12 @@ columns since all matrix elements in the design matrix were set to one (we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept and wee can simply drop these elements and construct a correlation -matrix without them. +matrix without them by centering our matrix elements by subtracting the mean of each column.

    + +

    Lnks with the Design Matrix

    +

    We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as

    $$ \boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. @@ -2554,6 +2626,10 @@ x_{10} & x_{11}\\ \end{bmatrix}. $$ + + +

    Computing the Expectation Values

    +

    If we then compute the expectation value

    $$ \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\begin{bmatrix} @@ -2603,6 +2679,10 @@ $$ \boldsymbol{S}_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}_i. $$ + + +

    More on the PCA Theorem

    +

    In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is \( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).

    @@ -2619,7 +2699,9 @@ dimensions where the eigenvalues are very small. If \( p \) is very large, we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) features/predictors.

    -

    The Algorithm before theorem

    + + +

    The Algorithm before theorem

    Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.

    -

    Writing our own PCA code

    + +

    Writing our own PCA code

    We will use a simple example first with two-dimensional data drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below): @@ -2659,6 +2742,8 @@ We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values.

    + +

    Implementing it

    The following Python code aids in setting up the data and writing out the design matrix. Note that the function multivariate returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).

    @@ -2694,6 +2779,9 @@ X = np.r

    Now we are going to implement the PCA algorithm. We will break it down into various substeps.

    + +

    First Step

    +

    The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is

    $$ \mu_n = \frac{1}{n} \sum_{i=1}^n x_i @@ -2736,6 +2824,9 @@ X_centered = X + + +

    Scaling

    Alternatively, we could use the functions we discussed earlier for scaling the data set. That is, we could have used the StandardScaler function in Scikit-Learn, a function which ensures @@ -2747,6 +2838,9 @@ while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our specific case.

    + +

    Centered Data

    +

    Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation

    $$ \begin{equation*} @@ -2819,10 +2913,16 @@ plt.show() + + +

    Exploring

    +

    Depending on the number of points \( n \), we will get results that are close to the covariance values defined above. The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed.

    -

    Diagonalize the sample covariance matrix to obtain the principal components

    + + +

    Diagonalize the sample covariance matrix to obtain the principal components

    Now we are ready to solve for the principal components! To do so we diagonalize the sample covariance matrix \( \Sigma \). We can use the @@ -2845,6 +2945,9 @@ $$

    where \( v_0 \) is the first principal component.

    + +

    Collecting all Steps

    +

    Collecting all these steps we can write our own PCA function and compare this with the functionality included in Scikit-Learn.

    @@ -2898,6 +3001,8 @@ X2Dsl = pca.

    This code does not contain all the above elements, but it shows how we can use Scikit-Learn to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then?

    + +

    Classical PCA Theorem

    We assume now that we have a design matrix \( \boldsymbol{X} \) which has been @@ -2916,6 +3021,9 @@ orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).

    + +

    The PCA Theorem

    +

    To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as

    We are almost there, we have obtained a relation between minimizing @@ -2971,6 +3079,8 @@ chapter 12.4 and discussion therein.

    For more details, see for example Vidal, Ma and Sastry, chapter 2.

    + +

    For a detailed demonstration of the geometric interpretation, see Vidal, Ma and Sastry, section 2.1.2.

    @@ -3060,6 +3170,9 @@ X2D = X_centered + + +

    PCA and scikit-learn

    Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The @@ -3242,7 +3355,10 @@ X_reduced = pca -

    Incremental PCA

    + + + +

    Incremental PCA

    One problem with the preceding implementation of PCA is that it requires the whole training set to fit in memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have @@ -3293,6 +3409,9 @@ X_reduced = rbf_pcaOther techniques

    There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.

    diff --git a/doc/pub/week43/html/week43-reveal.html b/doc/pub/week43/html/week43-reveal.html index 56f8e8ae3..68fd4bbf5 100644 --- a/doc/pub/week43/html/week43-reveal.html +++ b/doc/pub/week43/html/week43-reveal.html @@ -184,7 +184,7 @@ MathJax.Hub.Config({
    -

    Oct 28, 2021

    +

    Oct 29, 2021


    @@ -2228,7 +2228,10 @@ $$ \end{bmatrix}. $$

     
    + +

    +

    More on the covariance

    The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by @@ -2257,7 +2260,10 @@ $$

     

    In the above example this is the function we constructed using pandas.

    +
    +
    +

    Reminding ourselves about Linear Regression

    In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as

    @@ -2291,9 +2297,12 @@ $$ \boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. $$

     
    +

    +
    +

    Simple Example

    With these definitions, we can now rewrite our \( 2\times 2 \) -correaltion/covariance matrix in terms of a moe general design/feature +correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)

    @@ -2310,6 +2319,10 @@ $$ \end{bmatrix}, $$

     
    +

    + +
    +

    The Correlation Matrix

    and the correlation matrix

     
    @@ -2324,6 +2337,10 @@ $$ \end{bmatrix}, $$

     
    +

    + +
    +

    Numpy Functionality

    The Numpy function np.cov calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have @@ -2385,7 +2402,7 @@ C = np.cov(W)

    -

    Correlation Matrix

    +

    Correlation Matrix again

    The previous example can be converted into the correlation matrix by simply scaling the matrix elements with the variances. We should also @@ -2443,6 +2460,10 @@ this matrix we easily see that it is a positive definite matrix.

    The above procedure with numpy can be made more compact if we use pandas.

    +
    + +
    +

    Using Pandas

    We whow here how we can set up the correlation matrix using pandas, as done in this simple code

    @@ -2479,6 +2500,10 @@ correlation_matrix = Xpd.corr() +
    + +
    +

    And then the Franke Function

    We expand this model to the Franke function discussed above.

    @@ -2552,8 +2577,12 @@ columns since all matrix elements in the design matrix were set to one (we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept and wee can simply drop these elements and construct a correlation -matrix without them. +matrix without them by centering our matrix elements by subtracting the mean of each column.

    +
    + +
    +

    Lnks with the Design Matrix

    We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as

     
    @@ -2573,6 +2602,10 @@ x_{10} & x_{11}\\ \end{bmatrix}. $$

     
    +

    + +
    +

    Computing the Expectation Values

    If we then compute the expectation value

     
    @@ -2637,6 +2670,10 @@ $$ \boldsymbol{S}_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}_i. $$

     
    +

    + +
    +

    More on the PCA Theorem

    In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is \( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \). @@ -2654,7 +2691,10 @@ dimensions where the eigenvalues are very small. If \( p \) is very large, we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) features/predictors.

    -

    The Algorithm before theorem

    +
    + +
    +

    The Algorithm before theorem

    Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.

      @@ -2681,8 +2721,10 @@ $$

    • Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.

    • Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.
    -

    -

    Writing our own PCA code

    +
    + +
    +

    Writing our own PCA code

    We will use a simple example first with two-dimensional data drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below): @@ -2699,7 +2741,10 @@ $$ We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values.

    +
    +
    +

    Implementing it

    The following Python code aids in setting up the data and writing out the design matrix. Note that the function multivariate returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).

    @@ -2734,6 +2779,10 @@ X = np.random.multivariate_normal(mean, cov, n)

    Now we are going to implement the PCA algorithm. We will break it down into various substeps.

    +
    + +
    +

    First Step

    The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is

     
    @@ -2780,7 +2829,10 @@ X_centered = X - X.mean(axis=0) +

    +
    +

    Scaling

    Alternatively, we could use the functions we discussed earlier for scaling the data set. That is, we could have used the StandardScaler function in Scikit-Learn, a function which ensures @@ -2791,6 +2843,10 @@ variance. The diagonal covariance matrix elements will then be one, while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our specific case.

    +
    + +
    +

    Centered Data

    Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation

     
    @@ -2865,11 +2921,18 @@ plt.show() +

    + +
    +

    Exploring

    Depending on the number of points \( n \), we will get results that are close to the covariance values defined above. The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed.

    -

    Diagonalize the sample covariance matrix to obtain the principal components

    +
    + +
    +

    Diagonalize the sample covariance matrix to obtain the principal components

    Now we are ready to solve for the principal components! To do so we diagonalize the sample covariance matrix \( \Sigma \). We can use the @@ -2894,6 +2957,10 @@ $$

     

    where \( v_0 \) is the first principal component.

    +
    + +
    +

    Collecting all Steps

    Collecting all these steps we can write our own PCA function and compare this with the functionality included in Scikit-Learn. @@ -2948,6 +3015,9 @@ X2Dsl = pca.fit_transform(X)

    This code does not contain all the above elements, but it shows how we can use Scikit-Learn to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then?

    +
    + +

    Classical PCA Theorem

    We assume now that we have a design matrix \( \boldsymbol{X} \) which has been @@ -2965,6 +3035,10 @@ low-dimensional encoding of the data is then given by a set of vectors orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).

    +
    + +
    +

    The PCA Theorem

    To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as

    @@ -3029,6 +3103,9 @@ chapter 12.4 and discussion therein.

    For more details, see for example Vidal, Ma and Sastry, chapter 2.

    +
    + +

    For a detailed demonstration of the geometric interpretation, see Vidal, Ma and Sastry, section 2.1.2.

    @@ -3118,6 +3195,9 @@ X2D = X_centered.dot(W2) +
    + +

    PCA and scikit-learn

    Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The @@ -3301,7 +3381,10 @@ X_reduced = pca.fit_transform(X) -

    Incremental PCA

    +
    + +
    +

    Incremental PCA

    One problem with the preceding implementation of PCA is that it requires the whole training set to fit in memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have @@ -3352,6 +3435,9 @@ X_reduced = rbf_pca.fit_transform(X) +

    + +

    Other techniques

    There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.

    diff --git a/doc/pub/week43/html/week43-solarized.html b/doc/pub/week43/html/week43-solarized.html index 6b37d9a1f..76e5c239e 100644 --- a/doc/pub/week43/html/week43-solarized.html +++ b/doc/pub/week43/html/week43-solarized.html @@ -124,19 +124,48 @@ div.toc p,a { 2, None, 'introducing-the-covariance-and-correlation-functions'), - ('Correlation Matrix', 2, None, 'correlation-matrix'), + ('More on the covariance', 2, None, 'more-on-the-covariance'), + ('Reminding ourselves about Linear Regression', + 2, + None, + 'reminding-ourselves-about-linear-regression'), + ('Simple Example', 2, None, 'simple-example'), + ('The Correlation Matrix', 2, None, 'the-correlation-matrix'), + ('Numpy Functionality', 2, None, 'numpy-functionality'), + ('Correlation Matrix again', 2, None, 'correlation-matrix-again'), + ('Using Pandas', 2, None, 'using-pandas'), + ('And then the Franke Function', + 2, + None, + 'and-then-the-franke-function'), + ('Lnks with the Design Matrix', + 2, + None, + 'lnks-with-the-design-matrix'), + ('Computing the Expectation Values', + 2, + None, + 'computing-the-expectation-values'), ('Towards the PCA theorem', 2, None, 'towards-the-pca-theorem'), + ('More on the PCA Theorem', 2, None, 'more-on-the-pca-theorem'), ('The Algorithm before the Theorem', - 3, + 2, None, 'the-algorithm-before-the-theorem'), - ('Writing our own PCA code', 3, None, 'writing-our-own-pca-code'), + ('Writing our own PCA code', 2, None, 'writing-our-own-pca-code'), + ('Implementing it', 2, None, 'implementing-it'), + ('First Step', 2, None, 'first-step'), + ('Scaling', 2, None, 'scaling'), + ('Centered Data', 2, None, 'centered-data'), + ('Exploring', 2, None, 'exploring'), ('Diagonalize the sample covariance matrix to obtain the ' 'principal components', - 3, + 2, None, 'diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components'), + ('Collecting all Steps', 2, None, 'collecting-all-steps'), ('Classical PCA Theorem', 2, None, 'classical-pca-theorem'), + ('The PCA Theorem', 2, None, 'the-pca-theorem'), ('Geometric Interpretation and link with Singular Value ' 'Decomposition', 2, @@ -144,7 +173,7 @@ div.toc p,a { 'geometric-interpretation-and-link-with-singular-value-decomposition'), ('PCA and scikit-learn', 2, None, 'pca-and-scikit-learn'), ('Back to the Cancer Data', 2, None, 'back-to-the-cancer-data'), - ('Incremental PCA', 3, None, 'incremental-pca'), + ('Incremental PCA', 2, None, 'incremental-pca'), ('Randomized PCA', 3, None, 'randomized-pca'), ('Kernel PCA', 3, None, 'kernel-pca'), ('Other techniques', 2, None, 'other-techniques')]} @@ -185,7 +214,7 @@ MathJax.Hub.Config({
    -

    Oct 28, 2021

    +

    Oct 29, 2021


    @@ -2185,6 +2214,9 @@ $$ \end{bmatrix}. $$ + +









    +

    More on the covariance

    The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by @@ -2210,6 +2242,8 @@ $$

    In the above example this is the function we constructed using pandas.

    +









    +

    Reminding ourselves about Linear Regression

    In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as

    @@ -2238,8 +2272,11 @@ $$ \boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. $$ + +









    +

    Simple Example

    With these definitions, we can now rewrite our \( 2\times 2 \) -correaltion/covariance matrix in terms of a moe general design/feature +correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)

    @@ -2255,6 +2292,10 @@ $$ \end{bmatrix}, $$ + +









    +

    The Correlation Matrix

    +

    and the correlation matrix

    $$ \boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} @@ -2267,6 +2308,10 @@ $$ \end{bmatrix}, $$ + +









    +

    Numpy Functionality

    +

    The Numpy function np.cov calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values. The following simple function uses the @@ -2325,7 +2370,7 @@ C = np.cov(W)









    -

    Correlation Matrix

    +

    Correlation Matrix again

    The previous example can be converted into the correlation matrix by simply scaling the matrix elements with the variances. We should also @@ -2384,6 +2429,9 @@ this matrix we easily see that it is a positive definite matrix.

    The above procedure with numpy can be made more compact if we use pandas.

    +









    +

    Using Pandas

    +

    We whow here how we can set up the correlation matrix using pandas, as done in this simple code

    @@ -2420,6 +2468,10 @@ correlation_matrix = Xpd.corr() + +









    +

    And then the Franke Function

    +

    We expand this model to the Franke function discussed above.

    @@ -2492,9 +2544,12 @@ columns since all matrix elements in the design matrix were set to one (we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept and wee can simply drop these elements and construct a correlation -matrix without them. +matrix without them by centering our matrix elements by subtracting the mean of each column.

    +









    +

    Lnks with the Design Matrix

    +

    We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as

    $$ \boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. @@ -2510,6 +2565,10 @@ x_{10} & x_{11}\\ \end{bmatrix}. $$ + +









    +

    Computing the Expectation Values

    +

    If we then compute the expectation value

    $$ \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\begin{bmatrix} @@ -2559,6 +2618,10 @@ $$ \boldsymbol{S}_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}_i. $$ + +









    +

    More on the PCA Theorem

    +

    In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is \( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).

    @@ -2575,7 +2638,9 @@ dimensions where the eigenvalues are very small. If \( p \) is very large, we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) features/predictors.

    -

    The Algorithm before theorem

    + +









    +

    The Algorithm before theorem

    Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.

      @@ -2599,7 +2664,8 @@ $$
    • Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.
    • Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.
    -

    Writing our own PCA code

    +









    +

    Writing our own PCA code

    We will use a simple example first with two-dimensional data drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below): @@ -2615,6 +2681,8 @@ We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values.

    +









    +

    Implementing it

    The following Python code aids in setting up the data and writing out the design matrix. Note that the function multivariate returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).

    @@ -2650,6 +2718,9 @@ X = np.random.multivariate_normal(mean, cov, n)

    Now we are going to implement the PCA algorithm. We will break it down into various substeps.

    +









    +

    First Step

    +

    The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is

    $$ \mu_n = \frac{1}{n} \sum_{i=1}^n x_i @@ -2692,6 +2763,9 @@ X_centered = X - X.mean(axis=0) + +









    +

    Scaling

    Alternatively, we could use the functions we discussed earlier for scaling the data set. That is, we could have used the StandardScaler function in Scikit-Learn, a function which ensures @@ -2703,6 +2777,9 @@ while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our specific case.

    +









    +

    Centered Data

    +

    Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation

    $$ \begin{equation*} @@ -2775,10 +2852,16 @@ plt.show() + +









    +

    Exploring

    +

    Depending on the number of points \( n \), we will get results that are close to the covariance values defined above. The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed.

    -

    Diagonalize the sample covariance matrix to obtain the principal components

    + +









    +

    Diagonalize the sample covariance matrix to obtain the principal components

    Now we are ready to solve for the principal components! To do so we diagonalize the sample covariance matrix \( \Sigma \). We can use the @@ -2801,6 +2884,9 @@ $$

    where \( v_0 \) is the first principal component.

    +









    +

    Collecting all Steps

    +

    Collecting all these steps we can write our own PCA function and compare this with the functionality included in Scikit-Learn.

    @@ -2854,6 +2940,8 @@ X2Dsl = pca.fit_transform(X)

    This code does not contain all the above elements, but it shows how we can use Scikit-Learn to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then?

    + +









    Classical PCA Theorem

    We assume now that we have a design matrix \( \boldsymbol{X} \) which has been @@ -2872,6 +2960,9 @@ orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).

    +









    +

    The PCA Theorem

    +

    To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as

    We are almost there, we have obtained a relation between minimizing @@ -2927,6 +3018,8 @@ chapter 12.4 and discussion therein.

    For more details, see for example Vidal, Ma and Sastry, chapter 2.

    + +









    For a detailed demonstration of the geometric interpretation, see Vidal, Ma and Sastry, section 2.1.2.

    @@ -3016,6 +3109,9 @@ X2D = X_centered.dot(W2) + + +









    PCA and scikit-learn

    Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The @@ -3198,7 +3294,10 @@ X_reduced = pca.fit_transform(X) -

    Incremental PCA

    + + +









    +

    Incremental PCA

    One problem with the preceding implementation of PCA is that it requires the whole training set to fit in memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have @@ -3249,6 +3348,9 @@ X_reduced = rbf_pca.fit_transform(X) + + +









    Other techniques

    There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.

    diff --git a/doc/pub/week43/html/week43.html b/doc/pub/week43/html/week43.html index 9d5cddd98..054bfe344 100644 --- a/doc/pub/week43/html/week43.html +++ b/doc/pub/week43/html/week43.html @@ -201,19 +201,48 @@ div.toc p,a { 2, None, 'introducing-the-covariance-and-correlation-functions'), - ('Correlation Matrix', 2, None, 'correlation-matrix'), + ('More on the covariance', 2, None, 'more-on-the-covariance'), + ('Reminding ourselves about Linear Regression', + 2, + None, + 'reminding-ourselves-about-linear-regression'), + ('Simple Example', 2, None, 'simple-example'), + ('The Correlation Matrix', 2, None, 'the-correlation-matrix'), + ('Numpy Functionality', 2, None, 'numpy-functionality'), + ('Correlation Matrix again', 2, None, 'correlation-matrix-again'), + ('Using Pandas', 2, None, 'using-pandas'), + ('And then the Franke Function', + 2, + None, + 'and-then-the-franke-function'), + ('Lnks with the Design Matrix', + 2, + None, + 'lnks-with-the-design-matrix'), + ('Computing the Expectation Values', + 2, + None, + 'computing-the-expectation-values'), ('Towards the PCA theorem', 2, None, 'towards-the-pca-theorem'), + ('More on the PCA Theorem', 2, None, 'more-on-the-pca-theorem'), ('The Algorithm before the Theorem', - 3, + 2, None, 'the-algorithm-before-the-theorem'), - ('Writing our own PCA code', 3, None, 'writing-our-own-pca-code'), + ('Writing our own PCA code', 2, None, 'writing-our-own-pca-code'), + ('Implementing it', 2, None, 'implementing-it'), + ('First Step', 2, None, 'first-step'), + ('Scaling', 2, None, 'scaling'), + ('Centered Data', 2, None, 'centered-data'), + ('Exploring', 2, None, 'exploring'), ('Diagonalize the sample covariance matrix to obtain the ' 'principal components', - 3, + 2, None, 'diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components'), + ('Collecting all Steps', 2, None, 'collecting-all-steps'), ('Classical PCA Theorem', 2, None, 'classical-pca-theorem'), + ('The PCA Theorem', 2, None, 'the-pca-theorem'), ('Geometric Interpretation and link with Singular Value ' 'Decomposition', 2, @@ -221,7 +250,7 @@ div.toc p,a { 'geometric-interpretation-and-link-with-singular-value-decomposition'), ('PCA and scikit-learn', 2, None, 'pca-and-scikit-learn'), ('Back to the Cancer Data', 2, None, 'back-to-the-cancer-data'), - ('Incremental PCA', 3, None, 'incremental-pca'), + ('Incremental PCA', 2, None, 'incremental-pca'), ('Randomized PCA', 3, None, 'randomized-pca'), ('Kernel PCA', 3, None, 'kernel-pca'), ('Other techniques', 2, None, 'other-techniques')]} @@ -262,7 +291,7 @@ MathJax.Hub.Config({
    -

    Oct 28, 2021

    +

    Oct 29, 2021


    @@ -2262,6 +2291,9 @@ $$ \end{bmatrix}. $$ + +









    +

    More on the covariance

    The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by @@ -2287,6 +2319,8 @@ $$

    In the above example this is the function we constructed using pandas.

    +









    +

    Reminding ourselves about Linear Regression

    In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as

    @@ -2315,8 +2349,11 @@ $$ \boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. $$ + +









    +

    Simple Example

    With these definitions, we can now rewrite our \( 2\times 2 \) -correaltion/covariance matrix in terms of a moe general design/feature +correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)

    @@ -2332,6 +2369,10 @@ $$ \end{bmatrix}, $$ + +









    +

    The Correlation Matrix

    +

    and the correlation matrix

    $$ \boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} @@ -2344,6 +2385,10 @@ $$ \end{bmatrix}, $$ + +









    +

    Numpy Functionality

    +

    The Numpy function np.cov calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values. The following simple function uses the @@ -2402,7 +2447,7 @@ C = np.c









    -

    Correlation Matrix

    +

    Correlation Matrix again

    The previous example can be converted into the correlation matrix by simply scaling the matrix elements with the variances. We should also @@ -2461,6 +2506,9 @@ this matrix we easily see that it is a positive definite matrix.

    The above procedure with numpy can be made more compact if we use pandas.

    +









    +

    Using Pandas

    +

    We whow here how we can set up the correlation matrix using pandas, as done in this simple code

    @@ -2497,6 +2545,10 @@ correlation_matrix = XpdAnd then the Franke Function +

    We expand this model to the Franke function discussed above.

    @@ -2569,9 +2621,12 @@ columns since all matrix elements in the design matrix were set to one (we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept and wee can simply drop these elements and construct a correlation -matrix without them. +matrix without them by centering our matrix elements by subtracting the mean of each column.

    +









    +

    Lnks with the Design Matrix

    +

    We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as

    $$ \boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. @@ -2587,6 +2642,10 @@ x_{10} & x_{11}\\ \end{bmatrix}. $$ + +









    +

    Computing the Expectation Values

    +

    If we then compute the expectation value

    $$ \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\begin{bmatrix} @@ -2636,6 +2695,10 @@ $$ \boldsymbol{S}_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}_i. $$ + +









    +

    More on the PCA Theorem

    +

    In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is \( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).

    @@ -2652,7 +2715,9 @@ dimensions where the eigenvalues are very small. If \( p \) is very large, we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) features/predictors.

    -

    The Algorithm before theorem

    + +









    +

    The Algorithm before theorem

    Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.

      @@ -2676,7 +2741,8 @@ $$
    • Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.
    • Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.
    -

    Writing our own PCA code

    +









    +

    Writing our own PCA code

    We will use a simple example first with two-dimensional data drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below): @@ -2692,6 +2758,8 @@ We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values.

    +









    +

    Implementing it

    The following Python code aids in setting up the data and writing out the design matrix. Note that the function multivariate returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).

    @@ -2727,6 +2795,9 @@ X = np.r

    Now we are going to implement the PCA algorithm. We will break it down into various substeps.

    +









    +

    First Step

    +

    The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is

    $$ \mu_n = \frac{1}{n} \sum_{i=1}^n x_i @@ -2769,6 +2840,9 @@ X_centered = X + +









    +

    Scaling

    Alternatively, we could use the functions we discussed earlier for scaling the data set. That is, we could have used the StandardScaler function in Scikit-Learn, a function which ensures @@ -2780,6 +2854,9 @@ while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our specific case.

    +









    +

    Centered Data

    +

    Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation

    $$ \begin{equation*} @@ -2852,10 +2929,16 @@ plt.show() + +









    +

    Exploring

    +

    Depending on the number of points \( n \), we will get results that are close to the covariance values defined above. The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed.

    -

    Diagonalize the sample covariance matrix to obtain the principal components

    + +









    +

    Diagonalize the sample covariance matrix to obtain the principal components

    Now we are ready to solve for the principal components! To do so we diagonalize the sample covariance matrix \( \Sigma \). We can use the @@ -2878,6 +2961,9 @@ $$

    where \( v_0 \) is the first principal component.

    +









    +

    Collecting all Steps

    +

    Collecting all these steps we can write our own PCA function and compare this with the functionality included in Scikit-Learn.

    @@ -2931,6 +3017,8 @@ X2Dsl = pca.

    This code does not contain all the above elements, but it shows how we can use Scikit-Learn to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then?

    + +









    Classical PCA Theorem

    We assume now that we have a design matrix \( \boldsymbol{X} \) which has been @@ -2949,6 +3037,9 @@ orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).

    +









    +

    The PCA Theorem

    +

    To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as

    We are almost there, we have obtained a relation between minimizing @@ -3004,6 +3095,8 @@ chapter 12.4 and discussion therein.

    For more details, see for example Vidal, Ma and Sastry, chapter 2.

    + +









    For a detailed demonstration of the geometric interpretation, see Vidal, Ma and Sastry, section 2.1.2.

    @@ -3093,6 +3186,9 @@ X2D = X_centered + + +









    PCA and scikit-learn

    Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The @@ -3275,7 +3371,10 @@ X_reduced = pca -

    Incremental PCA

    + + +









    +

    Incremental PCA

    One problem with the preceding implementation of PCA is that it requires the whole training set to fit in memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have @@ -3326,6 +3425,9 @@ X_reduced = rbf_pcaOther techniques

    There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.

    diff --git a/doc/pub/week43/ipynb/ipynb-week43-src.tar.gz b/doc/pub/week43/ipynb/ipynb-week43-src.tar.gz index d2f5d2e2526c69c90b7116fa1e5b0f4210bd8988..2355ac853a8b66dc42d9eec99d97c9310a968a17 100644 GIT binary patch literal 29 kcmb2|=3w|~kC;$Ke diff --git a/doc/pub/week43/ipynb/week43.ipynb b/doc/pub/week43/ipynb/week43.ipynb index be1d02adf..2a4445417 100644 --- a/doc/pub/week43/ipynb/week43.ipynb +++ b/doc/pub/week43/ipynb/week43.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "b1a2512a", + "id": "8f5a249a", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "884a4eec", + "id": "7eee948e", "metadata": { "editable": true }, @@ -22,14 +22,14 @@ "# Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Oct 28, 2021**\n", + "Date: **Oct 29, 2021**\n", "\n", "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license" ] }, { "cell_type": "markdown", - "id": "c6f9bd15", + "id": "09c0a2e9", "metadata": { "editable": true }, @@ -59,7 +59,7 @@ }, { "cell_type": "markdown", - "id": "fc7cc0f8", + "id": "d52cc17d", "metadata": { "editable": true }, @@ -73,7 +73,7 @@ }, { "cell_type": "markdown", - "id": "2627b40f", + "id": "07a83a03", "metadata": { "editable": true }, @@ -87,7 +87,7 @@ }, { "cell_type": "markdown", - "id": "9006e671", + "id": "9726592c", "metadata": { "editable": true }, @@ -118,7 +118,7 @@ }, { "cell_type": "markdown", - "id": "0ddb3db4", + "id": "567f9c73", "metadata": { "editable": true }, @@ -145,7 +145,7 @@ }, { "cell_type": "markdown", - "id": "b6798e9d", + "id": "f5b8a632", "metadata": { "editable": true }, @@ -157,7 +157,7 @@ }, { "cell_type": "markdown", - "id": "802fb3ad", + "id": "c4c8b231", "metadata": { "editable": true }, @@ -168,7 +168,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "2b6e7a93", + "id": "40996f8c", "metadata": { "collapsed": false, "editable": true @@ -249,7 +249,7 @@ }, { "cell_type": "markdown", - "id": "94be91fe", + "id": "a982c648", "metadata": { "editable": true }, @@ -265,7 +265,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "2bd5c1ff", + "id": "7d898aa2", "metadata": { "collapsed": false, "editable": true @@ -304,7 +304,7 @@ }, { "cell_type": "markdown", - "id": "ba39a657", + "id": "8da5014e", "metadata": { "editable": true }, @@ -348,7 +348,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "730c86ce", + "id": "ad49d998", "metadata": { "collapsed": false, "editable": true @@ -431,7 +431,7 @@ }, { "cell_type": "markdown", - "id": "90b2e0a9", + "id": "931e40a5", "metadata": { "editable": true }, @@ -442,7 +442,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "a7e6d1c8", + "id": "aeddbf41", "metadata": { "collapsed": false, "editable": true @@ -544,7 +544,7 @@ }, { "cell_type": "markdown", - "id": "12e77a06", + "id": "baf60d58", "metadata": { "editable": true }, @@ -566,7 +566,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "7ac9b8ef", + "id": "f7005025", "metadata": { "collapsed": false, "editable": true @@ -663,7 +663,7 @@ }, { "cell_type": "markdown", - "id": "5515b214", + "id": "41ffedf9", "metadata": { "editable": true }, @@ -689,7 +689,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "9a4bb356", + "id": "654eaf6e", "metadata": { "collapsed": false, "editable": true @@ -890,7 +890,7 @@ }, { "cell_type": "markdown", - "id": "2ad0be29", + "id": "efc47d03", "metadata": { "editable": true }, @@ -914,7 +914,7 @@ }, { "cell_type": "markdown", - "id": "bba61009", + "id": "33b09eb0", "metadata": { "editable": true }, @@ -934,7 +934,7 @@ }, { "cell_type": "markdown", - "id": "15ffa0cb", + "id": "2c45dde8", "metadata": { "editable": true }, @@ -952,7 +952,7 @@ }, { "cell_type": "markdown", - "id": "83761910", + "id": "0a24e43b", "metadata": { "editable": true }, @@ -967,7 +967,7 @@ }, { "cell_type": "markdown", - "id": "807c499e", + "id": "e56a0574", "metadata": { "editable": true }, @@ -985,7 +985,7 @@ }, { "cell_type": "markdown", - "id": "cd6c5a40", + "id": "4dddc7fe", "metadata": { "editable": true }, @@ -998,7 +998,7 @@ }, { "cell_type": "markdown", - "id": "0547a7c5", + "id": "d01b03de", "metadata": { "editable": true }, @@ -1016,7 +1016,7 @@ }, { "cell_type": "markdown", - "id": "4688234e", + "id": "1cb23473", "metadata": { "editable": true }, @@ -1027,7 +1027,7 @@ }, { "cell_type": "markdown", - "id": "5f92bf31", + "id": "429bc00c", "metadata": { "editable": true }, @@ -1045,7 +1045,7 @@ }, { "cell_type": "markdown", - "id": "063d323f", + "id": "f828e2d9", "metadata": { "editable": true }, @@ -1072,7 +1072,7 @@ }, { "cell_type": "markdown", - "id": "2bfa66eb", + "id": "77878d51", "metadata": { "editable": true }, @@ -1084,7 +1084,7 @@ }, { "cell_type": "markdown", - "id": "da8a3c34", + "id": "ce5362f0", "metadata": { "editable": true }, @@ -1103,7 +1103,7 @@ }, { "cell_type": "markdown", - "id": "57d222af", + "id": "ae0ea76b", "metadata": { "editable": true }, @@ -1113,7 +1113,7 @@ }, { "cell_type": "markdown", - "id": "6c96024d", + "id": "3ab4f47b", "metadata": { "editable": true }, @@ -1133,7 +1133,7 @@ }, { "cell_type": "markdown", - "id": "df853ef1", + "id": "118b710d", "metadata": { "editable": true }, @@ -1145,7 +1145,7 @@ }, { "cell_type": "markdown", - "id": "30243273", + "id": "db719155", "metadata": { "editable": true }, @@ -1163,7 +1163,7 @@ }, { "cell_type": "markdown", - "id": "e77924b7", + "id": "d159205d", "metadata": { "editable": true }, @@ -1175,7 +1175,7 @@ }, { "cell_type": "markdown", - "id": "7a1c6563", + "id": "702470d8", "metadata": { "editable": true }, @@ -1196,7 +1196,7 @@ }, { "cell_type": "markdown", - "id": "e5a01aaf", + "id": "bc0845f3", "metadata": { "editable": true }, @@ -1213,7 +1213,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "42e71a7a", + "id": "e59d95dd", "metadata": { "collapsed": false, "editable": true @@ -1231,7 +1231,7 @@ }, { "cell_type": "markdown", - "id": "4a1970f1", + "id": "7d1a896c", "metadata": { "editable": true }, @@ -1242,7 +1242,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "a0bf7025", + "id": "1421746a", "metadata": { "collapsed": false, "editable": true @@ -1268,7 +1268,7 @@ }, { "cell_type": "markdown", - "id": "e4575f59", + "id": "d5f04732", "metadata": { "editable": true }, @@ -1281,7 +1281,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "73e8d296", + "id": "4fcf0e0c", "metadata": { "collapsed": false, "editable": true @@ -1294,7 +1294,7 @@ }, { "cell_type": "markdown", - "id": "f61a0a7d", + "id": "fecd3e34", "metadata": { "editable": true }, @@ -1313,7 +1313,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "bd5bac76", + "id": "ae3d6584", "metadata": { "collapsed": false, "editable": true @@ -1381,7 +1381,7 @@ }, { "cell_type": "markdown", - "id": "5ed98b7b", + "id": "b89c707d", "metadata": { "editable": true }, @@ -1394,7 +1394,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "e39a1ffa", + "id": "7d8dbd26", "metadata": { "collapsed": false, "editable": true @@ -1434,7 +1434,7 @@ }, { "cell_type": "markdown", - "id": "73d6abf0", + "id": "f71666af", "metadata": { "editable": true }, @@ -1446,7 +1446,7 @@ { "cell_type": "code", "execution_count": 12, - "id": "1dd50a9d", + "id": "df251164", "metadata": { "collapsed": false, "editable": true @@ -1460,7 +1460,7 @@ { "cell_type": "code", "execution_count": 13, - "id": "530ec249", + "id": "afaa12be", "metadata": { "collapsed": false, "editable": true @@ -1473,7 +1473,7 @@ }, { "cell_type": "markdown", - "id": "997a4526", + "id": "75456ae6", "metadata": { "editable": true }, @@ -1484,7 +1484,7 @@ { "cell_type": "code", "execution_count": 14, - "id": "1f8b6253", + "id": "59e30b86", "metadata": { "collapsed": false, "editable": true @@ -1498,7 +1498,7 @@ }, { "cell_type": "markdown", - "id": "44753d04", + "id": "58c0f040", "metadata": { "editable": true }, @@ -1513,7 +1513,7 @@ { "cell_type": "code", "execution_count": 15, - "id": "0c5dfb9e", + "id": "83505ac1", "metadata": { "collapsed": false, "editable": true @@ -1529,7 +1529,7 @@ { "cell_type": "code", "execution_count": 16, - "id": "095a049a", + "id": "329f1dc0", "metadata": { "collapsed": false, "editable": true @@ -1546,7 +1546,7 @@ }, { "cell_type": "markdown", - "id": "a53b31fc", + "id": "14a81976", "metadata": { "editable": true }, @@ -1558,7 +1558,7 @@ { "cell_type": "code", "execution_count": 17, - "id": "9150ff53", + "id": "17457797", "metadata": { "collapsed": false, "editable": true @@ -1572,7 +1572,7 @@ }, { "cell_type": "markdown", - "id": "fbea864e", + "id": "e5789961", "metadata": { "editable": true }, @@ -1588,7 +1588,7 @@ { "cell_type": "code", "execution_count": 18, - "id": "29dfca3c", + "id": "bdc875bd", "metadata": { "collapsed": false, "editable": true @@ -1622,7 +1622,7 @@ }, { "cell_type": "markdown", - "id": "eecb0ffd", + "id": "e6cb3384", "metadata": { "editable": true }, @@ -1635,7 +1635,7 @@ { "cell_type": "code", "execution_count": 19, - "id": "8c6bc3a1", + "id": "08316feb", "metadata": { "collapsed": false, "editable": true @@ -1660,7 +1660,7 @@ }, { "cell_type": "markdown", - "id": "7b93ba3c", + "id": "5edac321", "metadata": { "editable": true }, @@ -1674,7 +1674,7 @@ { "cell_type": "code", "execution_count": 20, - "id": "676c04b4", + "id": "7c1bb30a", "metadata": { "collapsed": false, "editable": true @@ -1692,7 +1692,7 @@ }, { "cell_type": "markdown", - "id": "316d681d", + "id": "6336c57c", "metadata": { "editable": true }, @@ -1703,7 +1703,7 @@ { "cell_type": "code", "execution_count": 21, - "id": "e8057157", + "id": "ca4d8d85", "metadata": { "collapsed": false, "editable": true @@ -1743,7 +1743,7 @@ }, { "cell_type": "markdown", - "id": "13d66d82", + "id": "749dd31a", "metadata": { "editable": true }, @@ -1755,7 +1755,7 @@ { "cell_type": "code", "execution_count": 22, - "id": "3790b732", + "id": "872d50d8", "metadata": { "collapsed": false, "editable": true @@ -1767,7 +1767,7 @@ }, { "cell_type": "markdown", - "id": "32999c8d", + "id": "4226960a", "metadata": { "editable": true }, @@ -1781,7 +1781,7 @@ { "cell_type": "code", "execution_count": 23, - "id": "d27ada3f", + "id": "9fd37e6d", "metadata": { "collapsed": false, "editable": true @@ -1798,7 +1798,7 @@ }, { "cell_type": "markdown", - "id": "cd192dd6", + "id": "4e452b42", "metadata": { "editable": true }, @@ -1812,7 +1812,7 @@ { "cell_type": "code", "execution_count": 24, - "id": "6856d82f", + "id": "f0359775", "metadata": { "collapsed": false, "editable": true @@ -1829,7 +1829,7 @@ }, { "cell_type": "markdown", - "id": "6f24333f", + "id": "faf2f2ce", "metadata": { "editable": true }, @@ -1845,7 +1845,7 @@ { "cell_type": "code", "execution_count": 25, - "id": "e26c535c", + "id": "4b5f60fd", "metadata": { "collapsed": false, "editable": true @@ -1873,7 +1873,7 @@ { "cell_type": "code", "execution_count": 26, - "id": "b3a5ab15", + "id": "8bf339e2", "metadata": { "collapsed": false, "editable": true @@ -1896,7 +1896,7 @@ { "cell_type": "code", "execution_count": 27, - "id": "4066a99d", + "id": "e377cea3", "metadata": { "collapsed": false, "editable": true @@ -1909,7 +1909,7 @@ }, { "cell_type": "markdown", - "id": "22b0a84d", + "id": "ffcac90d", "metadata": { "editable": true }, @@ -1925,7 +1925,7 @@ { "cell_type": "code", "execution_count": 28, - "id": "79b1826d", + "id": "2cb19a2d", "metadata": { "collapsed": false, "editable": true @@ -1952,7 +1952,7 @@ }, { "cell_type": "markdown", - "id": "a0a93e0c", + "id": "97e26693", "metadata": { "editable": true }, @@ -1967,7 +1967,7 @@ { "cell_type": "code", "execution_count": 29, - "id": "c6ed69b5", + "id": "7fc9e31d", "metadata": { "collapsed": false, "editable": true @@ -1983,7 +1983,7 @@ }, { "cell_type": "markdown", - "id": "704f44b3", + "id": "4941cc57", "metadata": { "editable": true }, @@ -1994,7 +1994,7 @@ }, { "cell_type": "markdown", - "id": "913ee2c3", + "id": "a4781239", "metadata": { "editable": true }, @@ -2012,7 +2012,7 @@ { "cell_type": "code", "execution_count": 30, - "id": "4c338adb", + "id": "9a03bbec", "metadata": { "collapsed": false, "editable": true @@ -2030,7 +2030,7 @@ }, { "cell_type": "markdown", - "id": "78eec6c2", + "id": "b75233fe", "metadata": { "editable": true }, @@ -2041,7 +2041,7 @@ { "cell_type": "code", "execution_count": 31, - "id": "feda074f", + "id": "0dd048c6", "metadata": { "collapsed": false, "editable": true @@ -2065,7 +2065,7 @@ }, { "cell_type": "markdown", - "id": "f9bc3f71", + "id": "0cfed2fe", "metadata": { "editable": true }, @@ -2092,7 +2092,7 @@ }, { "cell_type": "markdown", - "id": "949e7840", + "id": "404c4a68", "metadata": { "editable": true }, @@ -2108,7 +2108,7 @@ }, { "cell_type": "markdown", - "id": "bf565e75", + "id": "0fbde7e9", "metadata": { "editable": true }, @@ -2122,7 +2122,7 @@ }, { "cell_type": "markdown", - "id": "dd6062bd", + "id": "1bcb51c6", "metadata": { "editable": true }, @@ -2132,7 +2132,7 @@ }, { "cell_type": "markdown", - "id": "4cfc7ce4", + "id": "a5d2c073", "metadata": { "editable": true }, @@ -2144,7 +2144,7 @@ }, { "cell_type": "markdown", - "id": "6e082802", + "id": "c091fe65", "metadata": { "editable": true }, @@ -2154,7 +2154,7 @@ }, { "cell_type": "markdown", - "id": "e9d6dc5d", + "id": "0c552109", "metadata": { "editable": true }, @@ -2166,7 +2166,7 @@ }, { "cell_type": "markdown", - "id": "dbf919fe", + "id": "f21f246d", "metadata": { "editable": true }, @@ -2176,7 +2176,7 @@ }, { "cell_type": "markdown", - "id": "3f3146ae", + "id": "b87e5542", "metadata": { "editable": true }, @@ -2190,11 +2190,12 @@ }, { "cell_type": "markdown", - "id": "4cb8f154", + "id": "dbe0889e", "metadata": { "editable": true }, "source": [ + "## More on the covariance\n", "The covariance takes values between zero and infinity and may thus\n", "lead to problems with loss of numerical precision for particularly\n", "large values. It is common to scale the covariance matrix by\n", @@ -2204,7 +2205,7 @@ }, { "cell_type": "markdown", - "id": "9c13ad1a", + "id": "8df5a74b", "metadata": { "editable": true }, @@ -2216,7 +2217,7 @@ }, { "cell_type": "markdown", - "id": "cdad15b5", + "id": "be726a47", "metadata": { "editable": true }, @@ -2229,7 +2230,7 @@ }, { "cell_type": "markdown", - "id": "78bf11ff", + "id": "00f2081e", "metadata": { "editable": true }, @@ -2243,20 +2244,29 @@ }, { "cell_type": "markdown", - "id": "1f90b000", + "id": "94d16ece", "metadata": { "editable": true }, "source": [ - "In the above example this is the function we constructed using **pandas**.\n", - "\n", + "In the above example this is the function we constructed using **pandas**." + ] + }, + { + "cell_type": "markdown", + "id": "9e6a8394", + "metadata": { + "editable": true + }, + "source": [ + "## Reminding ourselves about Linear Regression\n", "In our derivation of the various regression algorithms like **Ordinary Least Squares** or **Ridge regression**\n", "we defined the design/feature matrix $\\boldsymbol{X}$ as" ] }, { "cell_type": "markdown", - "id": "9ebf20db", + "id": "9e329f2d", "metadata": { "editable": true }, @@ -2275,7 +2285,7 @@ }, { "cell_type": "markdown", - "id": "cbdfc57a", + "id": "0b35e8ec", "metadata": { "editable": true }, @@ -2287,7 +2297,7 @@ }, { "cell_type": "markdown", - "id": "f99874ea", + "id": "7ed94bc7", "metadata": { "editable": true }, @@ -2299,7 +2309,7 @@ }, { "cell_type": "markdown", - "id": "503a367c", + "id": "077121e6", "metadata": { "editable": true }, @@ -2309,7 +2319,7 @@ }, { "cell_type": "markdown", - "id": "66c5839d", + "id": "30d6d885", "metadata": { "editable": true }, @@ -2321,20 +2331,21 @@ }, { "cell_type": "markdown", - "id": "a7fe488c", + "id": "2db52e35", "metadata": { "editable": true }, "source": [ + "## Simple Example\n", "With these definitions, we can now rewrite our $2\\times 2$\n", - "correaltion/covariance matrix in terms of a moe general design/feature\n", + "correlation/covariance matrix in terms of a moe general design/feature\n", "matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$. This leads to a $p\\times p$\n", "covariance matrix for the vectors $\\boldsymbol{x}_i$ with $i=0,1,\\dots,p-1$" ] }, { "cell_type": "markdown", - "id": "6c38e3bc", + "id": "bf0b66f6", "metadata": { "editable": true }, @@ -2353,17 +2364,19 @@ }, { "cell_type": "markdown", - "id": "1424261b", + "id": "9e9119d3", "metadata": { "editable": true }, "source": [ + "## The Correlation Matrix\n", + "\n", "and the correlation matrix" ] }, { "cell_type": "markdown", - "id": "81c3f7ab", + "id": "a8038cdb", "metadata": { "editable": true }, @@ -2382,11 +2395,13 @@ }, { "cell_type": "markdown", - "id": "1ad546e7", + "id": "05ef28c6", "metadata": { "editable": true }, "source": [ + "## Numpy Functionality\n", + "\n", "The Numpy function **np.cov** calculates the covariance elements using\n", "the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have\n", "the exact mean values. The following simple function uses the\n", @@ -2396,7 +2411,7 @@ }, { "cell_type": "markdown", - "id": "7c67df72", + "id": "466ca667", "metadata": { "editable": true }, @@ -2414,7 +2429,7 @@ }, { "cell_type": "markdown", - "id": "035060f0", + "id": "03922592", "metadata": { "editable": true }, @@ -2429,7 +2444,7 @@ { "cell_type": "code", "execution_count": 32, - "id": "b78e08ee", + "id": "1f95502c", "metadata": { "collapsed": false, "editable": true @@ -2450,12 +2465,12 @@ }, { "cell_type": "markdown", - "id": "1d2f8ec8", + "id": "6a98fbd2", "metadata": { "editable": true }, "source": [ - "## Correlation Matrix\n", + "## Correlation Matrix again\n", "\n", "The previous example can be converted into the correlation matrix by\n", "simply scaling the matrix elements with the variances. We should also\n", @@ -2467,7 +2482,7 @@ { "cell_type": "code", "execution_count": 33, - "id": "7f370bd0", + "id": "efc7ef2a", "metadata": { "collapsed": false, "editable": true @@ -2499,7 +2514,7 @@ }, { "cell_type": "markdown", - "id": "d6867d46", + "id": "4915dd8b", "metadata": { "editable": true }, @@ -2508,7 +2523,17 @@ "should be and that the matrix is symmetric. Furthermore, diagonalizing\n", "this matrix we easily see that it is a positive definite matrix.\n", "\n", - "The above procedure with **numpy** can be made more compact if we use **pandas**.\n", + "The above procedure with **numpy** can be made more compact if we use **pandas**." + ] + }, + { + "cell_type": "markdown", + "id": "df593730", + "metadata": { + "editable": true + }, + "source": [ + "## Using Pandas\n", "\n", "We whow here how we can set up the correlation matrix using **pandas**, as done in this simple code" ] @@ -2516,7 +2541,7 @@ { "cell_type": "code", "execution_count": 34, - "id": "f51f5c99", + "id": "41e1b2df", "metadata": { "collapsed": false, "editable": true @@ -2540,18 +2565,20 @@ }, { "cell_type": "markdown", - "id": "9c819f6e", + "id": "46fa834c", "metadata": { "editable": true }, "source": [ + "## And then the Franke Function\n", + "\n", "We expand this model to the Franke function discussed above." ] }, { "cell_type": "code", "execution_count": 35, - "id": "9188663e", + "id": "07f0267e", "metadata": { "collapsed": false, "editable": true @@ -2605,7 +2632,7 @@ }, { "cell_type": "markdown", - "id": "a0235aa2", + "id": "4bb2bf9a", "metadata": { "editable": true }, @@ -2615,14 +2642,24 @@ "(we are fitting the function in terms of a polynomial of degree $n$). We would however not include the intercept\n", "and wee can simply\n", "drop these elements and construct a correlation\n", - "matrix without them. \n", + "matrix without them by centering our matrix elements by subtracting the mean of each column." + ] + }, + { + "cell_type": "markdown", + "id": "ce40bdd8", + "metadata": { + "editable": true + }, + "source": [ + "## Lnks with the Design Matrix\n", "\n", "We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix $\\boldsymbol{X}$ as" ] }, { "cell_type": "markdown", - "id": "7014c58b", + "id": "c09fc3e3", "metadata": { "editable": true }, @@ -2634,7 +2671,7 @@ }, { "cell_type": "markdown", - "id": "a1d89e07", + "id": "e06bf04c", "metadata": { "editable": true }, @@ -2644,7 +2681,7 @@ }, { "cell_type": "markdown", - "id": "6140d6e6", + "id": "a5735719", "metadata": { "editable": true }, @@ -2661,17 +2698,19 @@ }, { "cell_type": "markdown", - "id": "bc884877", + "id": "b95dcb40", "metadata": { "editable": true }, "source": [ + "## Computing the Expectation Values\n", + "\n", "If we then compute the expectation value" ] }, { "cell_type": "markdown", - "id": "40bb4918", + "id": "33935c81", "metadata": { "editable": true }, @@ -2686,7 +2725,7 @@ }, { "cell_type": "markdown", - "id": "54ebcb5f", + "id": "92417d00", "metadata": { "editable": true }, @@ -2696,7 +2735,7 @@ }, { "cell_type": "markdown", - "id": "119893a2", + "id": "0f58f9ec", "metadata": { "editable": true }, @@ -2710,7 +2749,7 @@ }, { "cell_type": "markdown", - "id": "fe6881a4", + "id": "182fedef", "metadata": { "editable": true }, @@ -2722,7 +2761,7 @@ }, { "cell_type": "markdown", - "id": "fd8d0fa8", + "id": "67ee7af0", "metadata": { "editable": true }, @@ -2734,7 +2773,7 @@ }, { "cell_type": "markdown", - "id": "868c5922", + "id": "c32b672a", "metadata": { "editable": true }, @@ -2746,7 +2785,7 @@ }, { "cell_type": "markdown", - "id": "25c0e9b8", + "id": "c417777b", "metadata": { "editable": true }, @@ -2761,7 +2800,7 @@ }, { "cell_type": "markdown", - "id": "440a2c8c", + "id": "bda501e3", "metadata": { "editable": true }, @@ -2773,7 +2812,7 @@ }, { "cell_type": "markdown", - "id": "4cd3dd52", + "id": "7d478963", "metadata": { "editable": true }, @@ -2783,7 +2822,7 @@ }, { "cell_type": "markdown", - "id": "1674339d", + "id": "99722089", "metadata": { "editable": true }, @@ -2795,7 +2834,7 @@ }, { "cell_type": "markdown", - "id": "5d2acb01", + "id": "51089375", "metadata": { "editable": true }, @@ -2805,7 +2844,7 @@ }, { "cell_type": "markdown", - "id": "4a39e1c9", + "id": "a2424367", "metadata": { "editable": true }, @@ -2817,11 +2856,13 @@ }, { "cell_type": "markdown", - "id": "a61ae494", + "id": "36fd4d35", "metadata": { "editable": true }, "source": [ + "## More on the PCA Theorem\n", + "\n", "In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is\n", "$\\lambda_0 > \\lambda_1 > \\dots > \\lambda_{p-1}$. \n", "\n", @@ -2840,12 +2881,12 @@ }, { "cell_type": "markdown", - "id": "791f5746", + "id": "b0adccc5", "metadata": { "editable": true }, "source": [ - "### The Algorithm before theorem\n", + "## The Algorithm before theorem\n", "\n", "Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here. \n", "* Set up the datapoints for the design/feature matrix $\\boldsymbol{X}$ with $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$, with the predictors/features $p$ referring to the column numbers and the entries $n$ being the row elements." @@ -2853,7 +2894,7 @@ }, { "cell_type": "markdown", - "id": "126813f3", + "id": "e1bbb25f", "metadata": { "editable": true }, @@ -2872,7 +2913,7 @@ }, { "cell_type": "markdown", - "id": "93397469", + "id": "5049383e", "metadata": { "editable": true }, @@ -2890,12 +2931,12 @@ }, { "cell_type": "markdown", - "id": "fc1b0b6b", + "id": "e9a6ebff", "metadata": { "editable": true }, "source": [ - "### Writing our own PCA code\n", + "## Writing our own PCA code\n", "\n", "We will use a simple example first with two-dimensional data\n", "drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below):" @@ -2903,7 +2944,7 @@ }, { "cell_type": "markdown", - "id": "01a3f128", + "id": "9f9d65a0", "metadata": { "editable": true }, @@ -2917,15 +2958,24 @@ }, { "cell_type": "markdown", - "id": "315a2025", + "id": "b8709735", "metadata": { "editable": true }, "source": [ "Note that the mean refers to each column of data. \n", "We will generate $n = 10000$ points $X = \\{ x_1, \\ldots, x_N \\}$ from\n", - "this distribution, and store them in the $1000 \\times 2$ matrix $\\boldsymbol{X}$. This is our design matrix where we have forced the covariance and mean values to take specific values.\n", - "\n", + "this distribution, and store them in the $1000 \\times 2$ matrix $\\boldsymbol{X}$. This is our design matrix where we have forced the covariance and mean values to take specific values." + ] + }, + { + "cell_type": "markdown", + "id": "122a25fa", + "metadata": { + "editable": true + }, + "source": [ + "## Implementing it\n", "The following Python code aids in setting up the data and writing out the design matrix.\n", "Note that the function **multivariate** returns also the covariance discussed above and that it is defined by dividing by $n-1$ instead of $n$." ] @@ -2933,7 +2983,7 @@ { "cell_type": "code", "execution_count": 36, - "id": "83e2b908", + "id": "4a910ae3", "metadata": { "collapsed": false, "editable": true @@ -2952,19 +3002,29 @@ }, { "cell_type": "markdown", - "id": "e5f508bc", + "id": "f61df9bb", "metadata": { "editable": true }, "source": [ - "Now we are going to implement the PCA algorithm. We will break it down into various substeps.\n", + "Now we are going to implement the PCA algorithm. We will break it down into various substeps." + ] + }, + { + "cell_type": "markdown", + "id": "05dc655c", + "metadata": { + "editable": true + }, + "source": [ + "## First Step\n", "\n", "The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is" ] }, { "cell_type": "markdown", - "id": "5f21d61e", + "id": "3de65f74", "metadata": { "editable": true }, @@ -2976,7 +3036,7 @@ }, { "cell_type": "markdown", - "id": "7808934b", + "id": "c82225cb", "metadata": { "editable": true }, @@ -2986,7 +3046,7 @@ }, { "cell_type": "markdown", - "id": "e5a41d70", + "id": "70cee57f", "metadata": { "editable": true }, @@ -2998,7 +3058,7 @@ }, { "cell_type": "markdown", - "id": "282d43cc", + "id": "5c819d68", "metadata": { "editable": true }, @@ -3012,7 +3072,7 @@ { "cell_type": "code", "execution_count": 37, - "id": "09b06064", + "id": "47487240", "metadata": { "collapsed": false, "editable": true @@ -3028,11 +3088,12 @@ }, { "cell_type": "markdown", - "id": "a031cecf", + "id": "1b262d86", "metadata": { "editable": true }, "source": [ + "## Scaling\n", "Alternatively, we could use the functions we discussed\n", "earlier for scaling the data set. That is, we could have used the\n", "**StandardScaler** function in **Scikit-Learn**, a function which ensures\n", @@ -3041,14 +3102,24 @@ "would then not get the same results, since we divide by the\n", "variance. The diagonal covariance matrix elements will then be one,\n", "while the non-diagonal ones need to be divided by $2\\sqrt{2}$ for our\n", - "specific case.\n", + "specific case." + ] + }, + { + "cell_type": "markdown", + "id": "05c59b90", + "metadata": { + "editable": true + }, + "source": [ + "## Centered Data\n", "\n", "Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation" ] }, { "cell_type": "markdown", - "id": "994711d8", + "id": "16727040", "metadata": { "editable": true }, @@ -3060,7 +3131,7 @@ }, { "cell_type": "markdown", - "id": "1e2675fe", + "id": "8ca98122", "metadata": { "editable": true }, @@ -3072,7 +3143,7 @@ { "cell_type": "code", "execution_count": 38, - "id": "10ccad26", + "id": "fe07d5ca", "metadata": { "collapsed": false, "editable": true @@ -3085,7 +3156,7 @@ }, { "cell_type": "markdown", - "id": "1690b383", + "id": "ea51ad69", "metadata": { "editable": true }, @@ -3097,7 +3168,7 @@ { "cell_type": "code", "execution_count": 39, - "id": "84659a84", + "id": "f937d79b", "metadata": { "collapsed": false, "editable": true @@ -3121,23 +3192,25 @@ }, { "cell_type": "markdown", - "id": "54554a75", + "id": "ab3606bc", "metadata": { "editable": true }, "source": [ + "## Exploring\n", + "\n", "Depending on the number of points $n$, we will get results that are close to the covariance values defined above.\n", "The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed." ] }, { "cell_type": "markdown", - "id": "ff07b3a4", + "id": "013d5ff2", "metadata": { "editable": true }, "source": [ - "### Diagonalize the sample covariance matrix to obtain the principal components\n", + "## Diagonalize the sample covariance matrix to obtain the principal components\n", "\n", "Now we are ready to solve for the principal components! To do so we\n", "diagonalize the sample covariance matrix $\\Sigma$. We can use the\n", @@ -3156,7 +3229,7 @@ }, { "cell_type": "markdown", - "id": "2587ff23", + "id": "07e80a9f", "metadata": { "editable": true }, @@ -3168,12 +3241,22 @@ }, { "cell_type": "markdown", - "id": "ee673932", + "id": "4001c555", "metadata": { "editable": true }, "source": [ - "where $v_0$ is the first principal component. \n", + "where $v_0$ is the first principal component." + ] + }, + { + "cell_type": "markdown", + "id": "b31eca70", + "metadata": { + "editable": true + }, + "source": [ + "## Collecting all Steps\n", "\n", "Collecting all these steps we can write our own PCA function and\n", "compare this with the functionality included in **Scikit-Learn**. \n", @@ -3186,7 +3269,7 @@ { "cell_type": "code", "execution_count": 40, - "id": "283b0750", + "id": "796f892f", "metadata": { "collapsed": false, "editable": true @@ -3218,7 +3301,7 @@ }, { "cell_type": "markdown", - "id": "54749e42", + "id": "de283672", "metadata": { "editable": true }, @@ -3228,7 +3311,7 @@ }, { "cell_type": "markdown", - "id": "e631472f", + "id": "320ead2e", "metadata": { "editable": true }, @@ -3247,7 +3330,17 @@ "low-dimensional encoding of the data is then given by a set of vectors\n", "$\\boldsymbol{z}_i$ with at most $l$ vectors, with $l << p$, defined by the\n", "orthogonal projection of the data onto the columns spanned by the\n", - "eigenvectors of the covariance(correlations matrix).\n", + "eigenvectors of the covariance(correlations matrix)." + ] + }, + { + "cell_type": "markdown", + "id": "94a58b84", + "metadata": { + "editable": true + }, + "source": [ + "## The PCA Theorem\n", "\n", "To show the PCA theorem let us start with the assumption that there is one vector $\\boldsymbol{s}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\\boldsymbol{w}_0$ and $\\boldsymbol{z}_0$ as\n", "\n", @@ -3266,7 +3359,7 @@ }, { "cell_type": "markdown", - "id": "710d2d80", + "id": "7b2c7a0d", "metadata": { "editable": true }, @@ -3278,7 +3371,7 @@ }, { "cell_type": "markdown", - "id": "43cbc633", + "id": "8eae4523", "metadata": { "editable": true }, @@ -3288,7 +3381,7 @@ }, { "cell_type": "markdown", - "id": "6045b2bd", + "id": "073d2212", "metadata": { "editable": true }, @@ -3300,7 +3393,7 @@ }, { "cell_type": "markdown", - "id": "9fb10f8c", + "id": "0e9d2aa5", "metadata": { "editable": true }, @@ -3310,7 +3403,7 @@ }, { "cell_type": "markdown", - "id": "5d4daefc", + "id": "37db61a5", "metadata": { "editable": true }, @@ -3322,7 +3415,7 @@ }, { "cell_type": "markdown", - "id": "556f302e", + "id": "e36f8507", "metadata": { "editable": true }, @@ -3332,7 +3425,7 @@ }, { "cell_type": "markdown", - "id": "f22b1a92", + "id": "99a1e313", "metadata": { "editable": true }, @@ -3344,7 +3437,7 @@ }, { "cell_type": "markdown", - "id": "da2606b3", + "id": "d3dc7838", "metadata": { "editable": true }, @@ -3370,7 +3463,7 @@ }, { "cell_type": "markdown", - "id": "0385c8c3", + "id": "a9a3243b", "metadata": { "editable": true }, @@ -3389,7 +3482,7 @@ { "cell_type": "code", "execution_count": 41, - "id": "b73cf27c", + "id": "7142dc2f", "metadata": { "collapsed": false, "editable": true @@ -3424,7 +3517,7 @@ }, { "cell_type": "markdown", - "id": "9593461d", + "id": "8037bbd0", "metadata": { "editable": true }, @@ -3441,7 +3534,7 @@ { "cell_type": "code", "execution_count": 42, - "id": "2d49e0e4", + "id": "9e7c2c5b", "metadata": { "collapsed": false, "editable": true @@ -3454,7 +3547,7 @@ }, { "cell_type": "markdown", - "id": "d3b313c2", + "id": "594c9376", "metadata": { "editable": true }, @@ -3469,7 +3562,7 @@ { "cell_type": "code", "execution_count": 43, - "id": "5765d20d", + "id": "19fe32e2", "metadata": { "collapsed": false, "editable": true @@ -3485,7 +3578,7 @@ }, { "cell_type": "markdown", - "id": "2c33011a", + "id": "8242afb2", "metadata": { "editable": true }, @@ -3498,7 +3591,7 @@ { "cell_type": "code", "execution_count": 44, - "id": "3b53a202", + "id": "fadae4bd", "metadata": { "collapsed": false, "editable": true @@ -3510,7 +3603,7 @@ }, { "cell_type": "markdown", - "id": "5a256bd6", + "id": "11973c56", "metadata": { "editable": true }, @@ -3522,7 +3615,7 @@ }, { "cell_type": "markdown", - "id": "c3ea4d8e", + "id": "4a7c95c0", "metadata": { "editable": true }, @@ -3535,7 +3628,7 @@ { "cell_type": "code", "execution_count": 45, - "id": "4ccb71cd", + "id": "fc346a80", "metadata": { "collapsed": false, "editable": true @@ -3574,7 +3667,7 @@ }, { "cell_type": "markdown", - "id": "985c80ee", + "id": "2fbd7f3e", "metadata": { "editable": true }, @@ -3592,7 +3685,7 @@ { "cell_type": "code", "execution_count": 46, - "id": "213e8674", + "id": "584d9340", "metadata": { "collapsed": false, "editable": true @@ -3607,7 +3700,7 @@ }, { "cell_type": "markdown", - "id": "6c8a6faf", + "id": "24464b99", "metadata": { "editable": true }, @@ -3620,7 +3713,7 @@ { "cell_type": "code", "execution_count": 47, - "id": "5152b41f", + "id": "bd125784", "metadata": { "collapsed": false, "editable": true @@ -3633,12 +3726,12 @@ }, { "cell_type": "markdown", - "id": "2c779e67", + "id": "936b4230", "metadata": { "editable": true }, "source": [ - "### Incremental PCA\n", + "## Incremental PCA\n", "\n", "One problem with the preceding implementation of PCA is that it requires the whole training set to fit in\n", "memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have\n", @@ -3649,7 +3742,7 @@ }, { "cell_type": "markdown", - "id": "cb2e571a", + "id": "fb9fe920", "metadata": { "editable": true }, @@ -3664,7 +3757,7 @@ }, { "cell_type": "markdown", - "id": "b0f40f3a", + "id": "c9182ac2", "metadata": { "editable": true }, @@ -3685,7 +3778,7 @@ { "cell_type": "code", "execution_count": 48, - "id": "a92cd7b1", + "id": "11a6ca19", "metadata": { "collapsed": false, "editable": true @@ -3699,7 +3792,7 @@ }, { "cell_type": "markdown", - "id": "73eb6324", + "id": "ab1e22e4", "metadata": { "editable": true }, diff --git a/doc/src/week43/week43.do.txt b/doc/src/week43/week43.do.txt index 8fb853379..4a4c48e8f 100644 --- a/doc/src/week43/week43.do.txt +++ b/doc/src/week43/week43.do.txt @@ -1390,6 +1390,8 @@ we can rewrite the covariance matrix as \] !et +!split +===== More on the covariance ===== The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by @@ -1417,7 +1419,8 @@ and $\bm{y}$ as In the above example this is the function we constructed using _pandas_. - +!split +===== Reminding ourselves about Linear Regression ===== In our derivation of the various regression algorithms like _Ordinary Least Squares_ or _Ridge regression_ we defined the design/feature matrix $\bm{X}$ as @@ -1448,8 +1451,10 @@ with a given vector \] !et +!split +===== Simple Example ===== With these definitions, we can now rewrite our $2\times 2$ -correaltion/covariance matrix in terms of a moe general design/feature +correlation/covariance matrix in terms of a moe general design/feature matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. This leads to a $p\times p$ covariance matrix for the vectors $\bm{x}_i$ with $i=0,1,\dots,p-1$ @@ -1465,6 +1470,10 @@ covariance matrix for the vectors $\bm{x}_i$ with $i=0,1,\dots,p-1$ \end{bmatrix}, \] !et + +!split +===== The Correlation Matrix ===== + and the correlation matrix !bt \[ @@ -1480,7 +1489,8 @@ and the correlation matrix !et - +!split +===== Numpy Functionality ===== The Numpy function _np.cov_ calculates the covariance elements using the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have @@ -1522,7 +1532,7 @@ print(C) !split -===== Correlation Matrix ===== +===== Correlation Matrix again ===== The previous example can be converted into the correlation matrix by simply scaling the matrix elements with the variances. We should also @@ -1560,6 +1570,8 @@ this matrix we easily see that it is a positive definite matrix. The above procedure with _numpy_ can be made more compact if we use _pandas_. +!split +===== Using Pandas ===== We whow here how we can set up the correlation matrix using _pandas_, as done in this simple code !bc pycod @@ -1578,6 +1590,8 @@ correlation_matrix = Xpd.corr() print(correlation_matrix) !ec +!split +===== And then the Franke Function ===== We expand this model to the Franke function discussed above. @@ -1633,9 +1647,10 @@ columns since all matrix elements in the design matrix were set to one (we are fitting the function in terms of a polynomial of degree $n$). We would however not include the intercept and wee can simply drop these elements and construct a correlation -matrix without them. - +matrix without them by centering our matrix elements by subtracting the mean of each column. +!split +===== Lnks with the Design Matrix ===== We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix $\bm{X}$ as !bt @@ -1656,6 +1671,9 @@ x_{10} & x_{11}\\ \] !et +!split +===== Computing the Expectation Values ===== + If we then compute the expectation value !bt \[ @@ -1712,6 +1730,9 @@ and since $\bm{C}[\bm{y}]$ is diagonal we have for a given eigenvalue $i$ of the \] !et +!split +===== More on the PCA Theorem ===== + In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is $\lambda_0 > \lambda_1 > \dots > \lambda_{p-1}$. @@ -1728,7 +1749,8 @@ dimensions where the eigenvalues are very small. If $p$ is very large, we could then aim at reducing $p$ to $l << p$ and handle only $l$ features/predictors. -=== The Algorithm before the Theorem === +!split +===== The Algorithm before the Theorem ===== Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here. * Set up the datapoints for the design/feature matrix $\bm{X}$ with $\bm{X}\in {\mathbb{R}}^{n\times p}$, with the predictors/features $p$ referring to the column numbers and the entries $n$ being the row elements. @@ -1750,8 +1772,8 @@ x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ * Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues. * Keep only those $l$ eigenvalues larger than a selected threshold value, discarding thus $p-l$ features since we expect small variations in the data here. - -=== Writing our own PCA code === +!split +===== Writing our own PCA code ===== We will use a simple example first with two-dimensional data drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below): @@ -1766,6 +1788,8 @@ Note that the mean refers to each column of data. We will generate $n = 10000$ points $X = \{ x_1, \ldots, x_N \}$ from this distribution, and store them in the $1000 \times 2$ matrix $\bm{X}$. This is our design matrix where we have forced the covariance and mean values to take specific values. +!split +===== Implementing it ===== The following Python code aids in setting up the data and writing out the design matrix. Note that the function _multivariate_ returns also the covariance discussed above and that it is defined by dividing by $n-1$ instead of $n$. !bc pycod @@ -1781,6 +1805,8 @@ X = np.random.multivariate_normal(mean, cov, n) Now we are going to implement the PCA algorithm. We will break it down into various substeps. +!split +===== First Step ===== The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is !bt @@ -1806,6 +1832,8 @@ df = df -df.mean() X_centered = X - X.mean(axis=0) !ec +!split +===== Scaling ===== Alternatively, we could use the functions we discussed earlier for scaling the data set. That is, we could have used the _StandardScaler_ function in _Scikit-Learn_, a function which ensures @@ -1816,6 +1844,8 @@ variance. The diagonal covariance matrix elements will then be one, while the non-diagonal ones need to be divided by $2\sqrt{2}$ for our specific case. +!split +===== Centered Data ===== Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation !bt @@ -1847,10 +1877,14 @@ plt.axis('equal') plt.show() !ec +!split +===== Exploring ===== + Depending on the number of points $n$, we will get results that are close to the covariance values defined above. The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed. -=== Diagonalize the sample covariance matrix to obtain the principal components === +!split +===== Diagonalize the sample covariance matrix to obtain the principal components ===== Now we are ready to solve for the principal components! To do so we diagonalize the sample covariance matrix $\Sigma$. We can use the @@ -1870,6 +1904,9 @@ x_i \approx \tilde{x}_i = \mu_n + \langle x_i, v_0 \rangle v_0 !et where $v_0$ is the first principal component. +!split +===== Collecting all Steps ===== + Collecting all these steps we can write our own PCA function and compare this with the functionality included in _Scikit-Learn_. @@ -1903,7 +1940,7 @@ print(pca.components_.T[:, 0]) !ec This code does not contain all the above elements, but it shows how we can use _Scikit-Learn_ to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then? - +!split ===== Classical PCA Theorem ===== We assume now that we have a design matrix $\bm{X}$ which has been @@ -1924,6 +1961,8 @@ eigenvectors of the covariance(correlations matrix). +!split +===== The PCA Theorem ===== To show the PCA theorem let us start with the assumption that there is one vector $\bm{s}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\bm{w}_0$ and $\bm{z}_0$ as @@ -1985,7 +2024,7 @@ chapter 12.4 and discussion therein. For more details, see for example "Vidal, Ma and Sastry, chapter 2":"https://www.springer.com/gp/book/9780387878102". - +!split ===== Geometric Interpretation and link with Singular Value Decomposition ===== For a detailed demonstration of the geometric interpretation, see "Vidal, Ma and Sastry, section 2.1.2":"https://www.springer.com/gp/book/9780387878102". @@ -2035,6 +2074,7 @@ W2 = V.T[:, :2] X2D = X_centered.dot(W2) !ec +!split ===== PCA and scikit-learn ===== Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The @@ -2116,8 +2156,8 @@ pca = PCA(n_components=0.95) X_reduced = pca.fit_transform(X) !ec - -=== Incremental PCA === +!split +===== Incremental PCA ===== One problem with the preceding implementation of PCA is that it requires the whole training set to fit in memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have @@ -2152,7 +2192,7 @@ rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04) X_reduced = rbf_pca.fit_transform(X) !ec - +!split ===== Other techniques =====