more pca

doc/pub/DimRed/html/._DimRed-bs000.html

@@ -224,7 +224,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Jan 1, 2020</h4></center> <!-- date -->
+<center><h4>Jan 2, 2020</h4></center> <!-- date -->
 <br>
 <p>
 

doc/pub/DimRed/html/._DimRed-bs018.html

@@ -233,12 +233,7 @@ n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
 mean <span style="color: #666666">=</span> (<span style="color: #666666">-1</span>, <span style="color: #666666">2</span>)
 cov <span style="color: #666666">=</span> [[<span style="color: #666666">4</span>, <span style="color: #666666">2</span>], [<span style="color: #666666">2</span>, <span style="color: #666666">2</span>]]
 X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>multivariate_normal(mean, cov, n)
-<span style="color: #408080; font-style: italic"># Print the X-matrix</span>
-<span style="color: #008000; font-weight: bold">print</span>(X)
 </pre></div>
-<p>
-Try to add to this code your own calculation of the covariance matrix.
-
 <p>
 Now we are going to implement the PCA algorithm. We will break it down into various substeps.
 
@@ -258,31 +253,31 @@ $$
 When you are done with these steps, print out \( \mu_n \) to verify it is
 close to \( \mu \) and plot your mean centered data to verify it is
 centered at the origin! Compare your code with the functionality from <b>Scikit-Learn</b> discussed above.
-The following code elements perform these operations using <b>pandas</b> or using our own functionality for doing so. The latter, using <b>numpy</b> is rather simply through the <b>mean()</b> function. 
+The following code elements perform these operations using <b>pandas</b> or using our own functionality for doing so. The latter, using <b>numpy</b> is rather simple through the <b>mean()</b> function. 
 <p>
 
 <!-- code=python (!bc pycod) typeset with pygments style "default" -->
 <div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>df <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(X)
 <span style="color: #408080; font-style: italic"># Pandas does the centering for us</span>
 df <span style="color: #666666">=</span> df <span style="color: #666666">-</span>df<span style="color: #666666">.</span>mean()
-display(df)
-
 <span style="color: #408080; font-style: italic"># we center it ourselves</span>
 X_centered <span style="color: #666666">=</span> X <span style="color: #666666">-</span> X<span style="color: #666666">.</span>mean(axis<span style="color: #666666">=0</span>)
-<span style="color: #408080; font-style: italic"># test that we get the same as Pandas</span>
-<span style="color: #008000; font-weight: bold">print</span>(X_centered<span style="color: #666666">-</span>df)
 </pre></div>
 <p>
-Alternatively, you could also have used the functions we discussed earlier for scaling the data set. 
-That is, we could have used the <b>StandardScaler</b> function in <b>Scikit-Learn</b>, a function which 
-ensures that for each feature/predictor we study the mean value is
-zero and the variance is one (every column in the design/feature
-matrix).  You would then not get the same results, since we divide by the variance. For diagonal covariance matrix elements will then be one, while the non-diagonal ones will be divided by \( 2\sqrt{2} \) for our specific case.
+Alternatively, we could use the functions we discussed
+earlier for scaling the data set.  That is, we could have used the
+<b>StandardScaler</b> function in <b>Scikit-Learn</b>, a function which ensures
+that for each feature/predictor we study the mean value is zero and
+the variance is one (every column in the design/feature matrix).  You
+would then not get the same results, since we divide by the
+variance. The diagonal covariance matrix elements will then be one,
+while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our
+specific case.
 
 <h3 id="___sec19" class="anchor">Compute the sample covariance </h3>
 
 <p>
-Now we are going to use the mean centered data to compute the sample covariance of the data. Recall it is given by:
+Now we are going to use the mean centered data to compute the sample covariance of the data. 
 $$
 \begin{equation*}
 \Sigma_n = \frac{1}{n-1} \sum_{i=1}^n \bar{x}_i^T \bar{x}_i = \frac{1}{n-1} \sum_{i=1}^n (x_i - \mu_n)^T (x_i - \mu_n)
@@ -294,11 +289,29 @@ We can write our own code or simply use either the functionaly of <b>numpy</b> o
 <p>
 
 <!-- code=python (!bc pycod) typeset with pygments style "default" -->
-<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">print</span>(np<span style="color: #666666">.</span>cov(X<span style="color: #666666">.</span>T))
-<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>cov())
+<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>cov())
 <span style="color: #008000; font-weight: bold">print</span>(np<span style="color: #666666">.</span>cov(X_centered<span style="color: #666666">.</span>T))
 </pre></div>
 <p>
+Note that the way we define the covariance matrix here has a factor \( n-1 \) instead of \( n \).
+Our own code here is not very elegant and asks for improvements. 
+<p>
+
+<!-- code=python (!bc pycod) typeset with pygments style "default" -->
+<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># extract the relevant columns from the centered design matrix of dim n x 2                                                      x = X_centered[:,[0]]</span>
+y <span style="color: #666666">=</span> X_centered[:,[<span style="color: #666666">1</span>]]
+Cov <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #666666">2</span>,<span style="color: #666666">2</span>))
+Cov[<span style="color: #666666">0</span>,<span style="color: #666666">1</span>] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(x<span style="color: #666666">.</span>T<span style="color: #AA22FF">@y</span>)<span style="color: #666666">/</span>(n<span style="color: #666666">-1.0</span>)
+Cov[<span style="color: #666666">0</span>,<span style="color: #666666">0</span>] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(x<span style="color: #666666">.</span>T<span style="color: #AA22FF">@x</span>)<span style="color: #666666">/</span>(n<span style="color: #666666">-1.0</span>)
+Cov[<span style="color: #666666">1</span>,<span style="color: #666666">1</span>] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(y<span style="color: #666666">.</span>T<span style="color: #AA22FF">@y</span>)<span style="color: #666666">/</span>(n<span style="color: #666666">-1.0</span>)
+Cov[<span style="color: #666666">1</span>,<span style="color: #666666">0</span>]<span style="color: #666666">=</span> Cov[<span style="color: #666666">0</span>,<span style="color: #666666">1</span>]
+<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Centered covariance using own code&quot;</span>)
+<span style="color: #008000; font-weight: bold">print</span>(Cov)
+plt<span style="color: #666666">.</span>plot(x, y, <span style="color: #BA2121">&#39;x&#39;</span>)
+plt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&#39;equal&#39;</span>)
+plt<span style="color: #666666">.</span>show()
+</pre></div>
+<p>
 Depending on the number of points \( n \), we will get results that are close to the covariance values defined above.
 
 <h3 id="___sec20" class="anchor">Diagonalize the sample covariance matrix to obtain the principal components </h3>

doc/pub/DimRed/html/DimRed-bs.html

@@ -224,7 +224,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Jan 1, 2020</h4></center> <!-- date -->
+<center><h4>Jan 2, 2020</h4></center> <!-- date -->
 <br>
 <p>
 

doc/pub/DimRed/html/DimRed-reveal.html

@@ -148,7 +148,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>&nbsp;<br>
-<center><h4>Jan 1, 2020</h4></center> <!-- date -->
+<center><h4>Jan 2, 2020</h4></center> <!-- date -->
 <br>
 <p>
 
@@ -1039,12 +1039,7 @@ n = <span style="color: #B452CD">100</span>
 mean = (-<span style="color: #B452CD">1</span>, <span style="color: #B452CD">2</span>)
 cov = [[<span style="color: #B452CD">4</span>, <span style="color: #B452CD">2</span>], [<span style="color: #B452CD">2</span>, <span style="color: #B452CD">2</span>]]
 X = np.random.multivariate_normal(mean, cov, n)
-<span style="color: #228B22"># Print the X-matrix</span>
-<span style="color: #8B008B; font-weight: bold">print</span>(X)
 </pre></div>
-<p>
-Try to add to this code your own calculation of the covariance matrix.
-
 <p>
 Now we are going to implement the PCA algorithm. We will break it down into various substeps.
 
@@ -1068,31 +1063,31 @@ $$
 When you are done with these steps, print out \( \mu_n \) to verify it is
 close to \( \mu \) and plot your mean centered data to verify it is
 centered at the origin! Compare your code with the functionality from <b>Scikit-Learn</b> discussed above.
-The following code elements perform these operations using <b>pandas</b> or using our own functionality for doing so. The latter, using <b>numpy</b> is rather simply through the <b>mean()</b> function. 
+The following code elements perform these operations using <b>pandas</b> or using our own functionality for doing so. The latter, using <b>numpy</b> is rather simple through the <b>mean()</b> function. 
 <p>
 
 <!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
 <div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>df = pd.DataFrame(X)
 <span style="color: #228B22"># Pandas does the centering for us</span>
 df = df -df.mean()
-display(df)
-
 <span style="color: #228B22"># we center it ourselves</span>
 X_centered = X - X.mean(axis=<span style="color: #B452CD">0</span>)
-<span style="color: #228B22"># test that we get the same as Pandas</span>
-<span style="color: #8B008B; font-weight: bold">print</span>(X_centered-df)
 </pre></div>
 <p>
-Alternatively, you could also have used the functions we discussed earlier for scaling the data set. 
-That is, we could have used the <b>StandardScaler</b> function in <b>Scikit-Learn</b>, a function which 
-ensures that for each feature/predictor we study the mean value is
-zero and the variance is one (every column in the design/feature
-matrix).  You would then not get the same results, since we divide by the variance. For diagonal covariance matrix elements will then be one, while the non-diagonal ones will be divided by \( 2\sqrt{2} \) for our specific case.
+Alternatively, we could use the functions we discussed
+earlier for scaling the data set.  That is, we could have used the
+<b>StandardScaler</b> function in <b>Scikit-Learn</b>, a function which ensures
+that for each feature/predictor we study the mean value is zero and
+the variance is one (every column in the design/feature matrix).  You
+would then not get the same results, since we divide by the
+variance. The diagonal covariance matrix elements will then be one,
+while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our
+specific case.
 
 <h3 id="___sec19">Compute the sample covariance </h3>
 
 <p>
-Now we are going to use the mean centered data to compute the sample covariance of the data. Recall it is given by:
+Now we are going to use the mean centered data to compute the sample covariance of the data. 
 <p>&nbsp;<br>
 $$
 \begin{equation*}
@@ -1106,11 +1101,29 @@ We can write our own code or simply use either the functionaly of <b>numpy</b> o
 <p>
 
 <!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
-<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">print</span>(np.cov(X.T))
-<span style="color: #8B008B; font-weight: bold">print</span>(df.cov())
+<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">print</span>(df.cov())
 <span style="color: #8B008B; font-weight: bold">print</span>(np.cov(X_centered.T))
 </pre></div>
 <p>
+Note that the way we define the covariance matrix here has a factor \( n-1 \) instead of \( n \).
+Our own code here is not very elegant and asks for improvements. 
+<p>
+
+<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
+<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># extract the relevant columns from the centered design matrix of dim n x 2                                                      x = X_centered[:,[0]]</span>
+y = X_centered[:,[<span style="color: #B452CD">1</span>]]
+Cov = np.zeros((<span style="color: #B452CD">2</span>,<span style="color: #B452CD">2</span>))
+Cov[<span style="color: #B452CD">0</span>,<span style="color: #B452CD">1</span>] = np.sum(x.T<span style="color: #707a7c">@y</span>)/(n-<span style="color: #B452CD">1.0</span>)
+Cov[<span style="color: #B452CD">0</span>,<span style="color: #B452CD">0</span>] = np.sum(x.T<span style="color: #707a7c">@x</span>)/(n-<span style="color: #B452CD">1.0</span>)
+Cov[<span style="color: #B452CD">1</span>,<span style="color: #B452CD">1</span>] = np.sum(y.T<span style="color: #707a7c">@y</span>)/(n-<span style="color: #B452CD">1.0</span>)
+Cov[<span style="color: #B452CD">1</span>,<span style="color: #B452CD">0</span>]= Cov[<span style="color: #B452CD">0</span>,<span style="color: #B452CD">1</span>]
+<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Centered covariance using own code&quot;</span>)
+<span style="color: #8B008B; font-weight: bold">print</span>(Cov)
+plt.plot(x, y, <span style="color: #CD5555">&#39;x&#39;</span>)
+plt.axis(<span style="color: #CD5555">&#39;equal&#39;</span>)
+plt.show()
+</pre></div>
+<p>
 Depending on the number of points \( n \), we will get results that are close to the covariance values defined above.
 
 <h3 id="___sec20">Diagonalize the sample covariance matrix to obtain the principal components </h3>

doc/pub/DimRed/html/DimRed-solarized.html

@@ -179,7 +179,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Jan 1, 2020</h4></center> <!-- date -->
+<center><h4>Jan 2, 2020</h4></center> <!-- date -->
 <br>
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1019,12 +1019,7 @@ n = <span style="color: #B452CD">100</span>
 mean = (-<span style="color: #B452CD">1</span>, <span style="color: #B452CD">2</span>)
 cov = [[<span style="color: #B452CD">4</span>, <span style="color: #B452CD">2</span>], [<span style="color: #B452CD">2</span>, <span style="color: #B452CD">2</span>]]
 X = np.random.multivariate_normal(mean, cov, n)
-<span style="color: #228B22"># Print the X-matrix</span>
-<span style="color: #8B008B; font-weight: bold">print</span>(X)
 </pre></div>
-<p>
-Try to add to this code your own calculation of the covariance matrix.
-
 <p>
 Now we are going to implement the PCA algorithm. We will break it down into various substeps.
 
@@ -1044,31 +1039,31 @@ $$
 When you are done with these steps, print out \( \mu_n \) to verify it is
 close to \( \mu \) and plot your mean centered data to verify it is
 centered at the origin! Compare your code with the functionality from <b>Scikit-Learn</b> discussed above.
-The following code elements perform these operations using <b>pandas</b> or using our own functionality for doing so. The latter, using <b>numpy</b> is rather simply through the <b>mean()</b> function. 
+The following code elements perform these operations using <b>pandas</b> or using our own functionality for doing so. The latter, using <b>numpy</b> is rather simple through the <b>mean()</b> function. 
 <p>
 
 <!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
 <div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>df = pd.DataFrame(X)
 <span style="color: #228B22"># Pandas does the centering for us</span>
 df = df -df.mean()
-display(df)
-
 <span style="color: #228B22"># we center it ourselves</span>
 X_centered = X - X.mean(axis=<span style="color: #B452CD">0</span>)
-<span style="color: #228B22"># test that we get the same as Pandas</span>
-<span style="color: #8B008B; font-weight: bold">print</span>(X_centered-df)
 </pre></div>
 <p>
-Alternatively, you could also have used the functions we discussed earlier for scaling the data set. 
-That is, we could have used the <b>StandardScaler</b> function in <b>Scikit-Learn</b>, a function which 
-ensures that for each feature/predictor we study the mean value is
-zero and the variance is one (every column in the design/feature
-matrix).  You would then not get the same results, since we divide by the variance. For diagonal covariance matrix elements will then be one, while the non-diagonal ones will be divided by \( 2\sqrt{2} \) for our specific case.
+Alternatively, we could use the functions we discussed
+earlier for scaling the data set.  That is, we could have used the
+<b>StandardScaler</b> function in <b>Scikit-Learn</b>, a function which ensures
+that for each feature/predictor we study the mean value is zero and
+the variance is one (every column in the design/feature matrix).  You
+would then not get the same results, since we divide by the
+variance. The diagonal covariance matrix elements will then be one,
+while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our
+specific case.
 
 <h3 id="___sec19">Compute the sample covariance </h3>
 
 <p>
-Now we are going to use the mean centered data to compute the sample covariance of the data. Recall it is given by:
+Now we are going to use the mean centered data to compute the sample covariance of the data. 
 $$
 \begin{equation*}
 \Sigma_n = \frac{1}{n-1} \sum_{i=1}^n \bar{x}_i^T \bar{x}_i = \frac{1}{n-1} \sum_{i=1}^n (x_i - \mu_n)^T (x_i - \mu_n)
@@ -1080,11 +1075,29 @@ We can write our own code or simply use either the functionaly of <b>numpy</b> o
 <p>
 
 <!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
-<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">print</span>(np.cov(X.T))
-<span style="color: #8B008B; font-weight: bold">print</span>(df.cov())
+<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">print</span>(df.cov())
 <span style="color: #8B008B; font-weight: bold">print</span>(np.cov(X_centered.T))
 </pre></div>
 <p>
+Note that the way we define the covariance matrix here has a factor \( n-1 \) instead of \( n \).
+Our own code here is not very elegant and asks for improvements. 
+<p>
+
+<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
+<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># extract the relevant columns from the centered design matrix of dim n x 2                                                      x = X_centered[:,[0]]</span>
+y = X_centered[:,[<span style="color: #B452CD">1</span>]]
+Cov = np.zeros((<span style="color: #B452CD">2</span>,<span style="color: #B452CD">2</span>))
+Cov[<span style="color: #B452CD">0</span>,<span style="color: #B452CD">1</span>] = np.sum(x.T<span style="color: #707a7c">@y</span>)/(n-<span style="color: #B452CD">1.0</span>)
+Cov[<span style="color: #B452CD">0</span>,<span style="color: #B452CD">0</span>] = np.sum(x.T<span style="color: #707a7c">@x</span>)/(n-<span style="color: #B452CD">1.0</span>)
+Cov[<span style="color: #B452CD">1</span>,<span style="color: #B452CD">1</span>] = np.sum(y.T<span style="color: #707a7c">@y</span>)/(n-<span style="color: #B452CD">1.0</span>)
+Cov[<span style="color: #B452CD">1</span>,<span style="color: #B452CD">0</span>]= Cov[<span style="color: #B452CD">0</span>,<span style="color: #B452CD">1</span>]
+<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Centered covariance using own code&quot;</span>)
+<span style="color: #8B008B; font-weight: bold">print</span>(Cov)
+plt.plot(x, y, <span style="color: #CD5555">&#39;x&#39;</span>)
+plt.axis(<span style="color: #CD5555">&#39;equal&#39;</span>)
+plt.show()
+</pre></div>
+<p>
 Depending on the number of points \( n \), we will get results that are close to the covariance values defined above.
 
 <h3 id="___sec20">Diagonalize the sample covariance matrix to obtain the principal components </h3>

doc/pub/DimRed/html/DimRed.html

@@ -184,7 +184,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Jan 1, 2020</h4></center> <!-- date -->
+<center><h4>Jan 2, 2020</h4></center> <!-- date -->
 <br>
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1024,12 +1024,7 @@ n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
 mean <span style="color: #666666">=</span> (<span style="color: #666666">-1</span>, <span style="color: #666666">2</span>)
 cov <span style="color: #666666">=</span> [[<span style="color: #666666">4</span>, <span style="color: #666666">2</span>], [<span style="color: #666666">2</span>, <span style="color: #666666">2</span>]]
 X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>multivariate_normal(mean, cov, n)
-<span style="color: #408080; font-style: italic"># Print the X-matrix</span>
-<span style="color: #008000; font-weight: bold">print</span>(X)
 </pre></div>
-<p>
-Try to add to this code your own calculation of the covariance matrix.
-
 <p>
 Now we are going to implement the PCA algorithm. We will break it down into various substeps.
 
@@ -1049,31 +1044,31 @@ $$
 When you are done with these steps, print out \( \mu_n \) to verify it is
 close to \( \mu \) and plot your mean centered data to verify it is
 centered at the origin! Compare your code with the functionality from <b>Scikit-Learn</b> discussed above.
-The following code elements perform these operations using <b>pandas</b> or using our own functionality for doing so. The latter, using <b>numpy</b> is rather simply through the <b>mean()</b> function. 
+The following code elements perform these operations using <b>pandas</b> or using our own functionality for doing so. The latter, using <b>numpy</b> is rather simple through the <b>mean()</b> function. 
 <p>
 
 <!-- code=python (!bc pycod) typeset with pygments style "default" -->
 <div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>df <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(X)
 <span style="color: #408080; font-style: italic"># Pandas does the centering for us</span>
 df <span style="color: #666666">=</span> df <span style="color: #666666">-</span>df<span style="color: #666666">.</span>mean()
-display(df)
-
 <span style="color: #408080; font-style: italic"># we center it ourselves</span>
 X_centered <span style="color: #666666">=</span> X <span style="color: #666666">-</span> X<span style="color: #666666">.</span>mean(axis<span style="color: #666666">=0</span>)
-<span style="color: #408080; font-style: italic"># test that we get the same as Pandas</span>
-<span style="color: #008000; font-weight: bold">print</span>(X_centered<span style="color: #666666">-</span>df)
 </pre></div>
 <p>
-Alternatively, you could also have used the functions we discussed earlier for scaling the data set. 
-That is, we could have used the <b>StandardScaler</b> function in <b>Scikit-Learn</b>, a function which 
-ensures that for each feature/predictor we study the mean value is
-zero and the variance is one (every column in the design/feature
-matrix).  You would then not get the same results, since we divide by the variance. For diagonal covariance matrix elements will then be one, while the non-diagonal ones will be divided by \( 2\sqrt{2} \) for our specific case.
+Alternatively, we could use the functions we discussed
+earlier for scaling the data set.  That is, we could have used the
+<b>StandardScaler</b> function in <b>Scikit-Learn</b>, a function which ensures
+that for each feature/predictor we study the mean value is zero and
+the variance is one (every column in the design/feature matrix).  You
+would then not get the same results, since we divide by the
+variance. The diagonal covariance matrix elements will then be one,
+while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our
+specific case.
 
 <h3 id="___sec19">Compute the sample covariance </h3>
 
 <p>
-Now we are going to use the mean centered data to compute the sample covariance of the data. Recall it is given by:
+Now we are going to use the mean centered data to compute the sample covariance of the data. 
 $$
 \begin{equation*}
 \Sigma_n = \frac{1}{n-1} \sum_{i=1}^n \bar{x}_i^T \bar{x}_i = \frac{1}{n-1} \sum_{i=1}^n (x_i - \mu_n)^T (x_i - \mu_n)
@@ -1085,11 +1080,29 @@ We can write our own code or simply use either the functionaly of <b>numpy</b> o
 <p>
 
 <!-- code=python (!bc pycod) typeset with pygments style "default" -->
-<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">print</span>(np<span style="color: #666666">.</span>cov(X<span style="color: #666666">.</span>T))
-<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>cov())
+<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>cov())
 <span style="color: #008000; font-weight: bold">print</span>(np<span style="color: #666666">.</span>cov(X_centered<span style="color: #666666">.</span>T))
 </pre></div>
 <p>
+Note that the way we define the covariance matrix here has a factor \( n-1 \) instead of \( n \).
+Our own code here is not very elegant and asks for improvements. 
+<p>
+
+<!-- code=python (!bc pycod) typeset with pygments style "default" -->
+<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># extract the relevant columns from the centered design matrix of dim n x 2                                                      x = X_centered[:,[0]]</span>
+y <span style="color: #666666">=</span> X_centered[:,[<span style="color: #666666">1</span>]]
+Cov <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #666666">2</span>,<span style="color: #666666">2</span>))
+Cov[<span style="color: #666666">0</span>,<span style="color: #666666">1</span>] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(x<span style="color: #666666">.</span>T<span style="color: #AA22FF">@y</span>)<span style="color: #666666">/</span>(n<span style="color: #666666">-1.0</span>)
+Cov[<span style="color: #666666">0</span>,<span style="color: #666666">0</span>] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(x<span style="color: #666666">.</span>T<span style="color: #AA22FF">@x</span>)<span style="color: #666666">/</span>(n<span style="color: #666666">-1.0</span>)
+Cov[<span style="color: #666666">1</span>,<span style="color: #666666">1</span>] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(y<span style="color: #666666">.</span>T<span style="color: #AA22FF">@y</span>)<span style="color: #666666">/</span>(n<span style="color: #666666">-1.0</span>)
+Cov[<span style="color: #666666">1</span>,<span style="color: #666666">0</span>]<span style="color: #666666">=</span> Cov[<span style="color: #666666">0</span>,<span style="color: #666666">1</span>]
+<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Centered covariance using own code&quot;</span>)
+<span style="color: #008000; font-weight: bold">print</span>(Cov)
+plt<span style="color: #666666">.</span>plot(x, y, <span style="color: #BA2121">&#39;x&#39;</span>)
+plt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&#39;equal&#39;</span>)
+plt<span style="color: #666666">.</span>show()
+</pre></div>
+<p>
 Depending on the number of points \( n \), we will get results that are close to the covariance values defined above.
 
 <h3 id="___sec20">Diagonalize the sample covariance matrix to obtain the principal components </h3>

doc/pub/DimRed/ipynb/DimRed.ipynb

@@ -10,7 +10,7 @@
     "<!-- Author: -->  \n",
     "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
     "\n",
-    "Date: **Jan 1, 2020**\n",
+    "Date: **Jan 2, 2020**\n",
     "\n",
     "Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
     "\n",
@@ -1111,17 +1111,13 @@
     "n = 100\n",
     "mean = (-1, 2)\n",
     "cov = [[4, 2], [2, 2]]\n",
-    "X = np.random.multivariate_normal(mean, cov, n)\n",
-    "# Print the X-matrix\n",
-    "print(X)"
+    "X = np.random.multivariate_normal(mean, cov, n)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Try to add to this code your own calculation of the covariance matrix.\n",
-    "\n",
     "Now we are going to implement the PCA algorithm. We will break it down into various substeps.\n",
     "\n",
     "### Compute the sample mean and center the data\n",
@@ -1161,7 +1157,7 @@
     "When you are done with these steps, print out $\\mu_n$ to verify it is\n",
     "close to $\\mu$ and plot your mean centered data to verify it is\n",
     "centered at the origin! Compare your code with the functionality from **Scikit-Learn** discussed above.\n",
-    "The following code elements perform these operations using **pandas** or using our own functionality for doing so. The latter, using **numpy** is rather simply through the **mean()** function."
+    "The following code elements perform these operations using **pandas** or using our own functionality for doing so. The latter, using **numpy** is rather simple through the **mean()** function."
    ]
   },
   {
@@ -1175,27 +1171,27 @@
     "df = pd.DataFrame(X)\n",
     "# Pandas does the centering for us\n",
     "df = df -df.mean()\n",
-    "display(df)\n",
-    "\n",
     "# we center it ourselves\n",
-    "X_centered = X - X.mean(axis=0)\n",
-    "# test that we get the same as Pandas\n",
-    "print(X_centered-df)"
+    "X_centered = X - X.mean(axis=0)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Alternatively, you could also have used the functions we discussed earlier for scaling the data set. \n",
-    "That is, we could have used the **StandardScaler** function in **Scikit-Learn**, a function which \n",
-    "ensures that for each feature/predictor we study the mean value is\n",
-    "zero and the variance is one (every column in the design/feature\n",
-    "matrix).  You would then not get the same results, since we divide by the variance. For diagonal covariance matrix elements will then be one, while the non-diagonal ones will be divided by $2\\sqrt{2}$ for our specific case.\n",
+    "Alternatively, we could use the functions we discussed\n",
+    "earlier for scaling the data set.  That is, we could have used the\n",
+    "**StandardScaler** function in **Scikit-Learn**, a function which ensures\n",
+    "that for each feature/predictor we study the mean value is zero and\n",
+    "the variance is one (every column in the design/feature matrix).  You\n",
+    "would then not get the same results, since we divide by the\n",
+    "variance. The diagonal covariance matrix elements will then be one,\n",
+    "while the non-diagonal ones need to be divided by $2\\sqrt{2}$ for our\n",
+    "specific case.\n",
     "\n",
     "### Compute the sample covariance\n",
     "\n",
-    "Now we are going to use the mean centered data to compute the sample covariance of the data. Recall it is given by:"
+    "Now we are going to use the mean centered data to compute the sample covariance of the data."
    ]
   },
   {
@@ -1223,11 +1219,40 @@
    },
    "outputs": [],
    "source": [
-    "print(np.cov(X.T))\n",
     "print(df.cov())\n",
     "print(np.cov(X_centered.T))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note that the way we define the covariance matrix here has a factor $n-1$ instead of $n$.\n",
+    "Our own code here is not very elegant and asks for improvements."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# extract the relevant columns from the centered design matrix of dim n x 2                                                      x = X_centered[:,[0]]\n",
+    "y = X_centered[:,[1]]\n",
+    "Cov = np.zeros((2,2))\n",
+    "Cov[0,1] = np.sum(x.T@y)/(n-1.0)\n",
+    "Cov[0,0] = np.sum(x.T@x)/(n-1.0)\n",
+    "Cov[1,1] = np.sum(y.T@y)/(n-1.0)\n",
+    "Cov[1,0]= Cov[0,1]\n",
+    "print(\"Centered covariance using own code\")\n",
+    "print(Cov)\n",
+    "plt.plot(x, y, 'x')\n",
+    "plt.axis('equal')\n",
+    "plt.show()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -1275,7 +1300,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 15,
    "metadata": {
     "collapsed": false
    },
@@ -1586,7 +1611,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": 16,
    "metadata": {
     "collapsed": false
    },
@@ -1633,7 +1658,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 17,
    "metadata": {
     "collapsed": false
    },
@@ -1657,7 +1682,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": 18,
    "metadata": {
     "collapsed": false
    },
@@ -1681,7 +1706,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": 19,
    "metadata": {
     "collapsed": false
    },
@@ -1705,7 +1730,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": 20,
    "metadata": {
     "collapsed": false
    },
@@ -1759,7 +1784,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
+   "execution_count": 21,
    "metadata": {
     "collapsed": false
    },
@@ -1782,7 +1807,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": 22,
    "metadata": {
     "collapsed": false
    },
@@ -1829,7 +1854,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": 23,
    "metadata": {
     "collapsed": false
    },

doc/src/DimRed/DimRed.do.txt

@@ -786,13 +786,8 @@ n = 100
 mean = (-1, 2)
 cov = [[4, 2], [2, 2]]
 X = np.random.multivariate_normal(mean, cov, n)
-# Print the X-matrix
-print(X)
 !ec
 
-
-Try to add to this code your own calculation of the covariance matrix.
-
 Now we are going to implement the PCA algorithm. We will break it down into various substeps.
 
 === Compute the sample mean and center the data ===
@@ -812,27 +807,28 @@ and the mean-centered data $\bar{X} = \{ \bar{x}_1, \ldots, \bar{x}_n \}$ takes
 When you are done with these steps, print out $\mu_n$ to verify it is
 close to $\mu$ and plot your mean centered data to verify it is
 centered at the origin! Compare your code with the functionality from _Scikit-Learn_ discussed above.
-The following code elements perform these operations using _pandas_ or using our own functionality for doing so. The latter, using _numpy_ is rather simply through the _mean()_ function. 
+The following code elements perform these operations using _pandas_ or using our own functionality for doing so. The latter, using _numpy_ is rather simple through the _mean()_ function. 
 !bc pycod
 df = pd.DataFrame(X)
 # Pandas does the centering for us
 df = df -df.mean()
-display(df)
-
 # we center it ourselves
 X_centered = X - X.mean(axis=0)
-# test that we get the same as Pandas
-print(X_centered-df)
 !ec
-Alternatively, you could also have used the functions we discussed earlier for scaling the data set. 
-That is, we could have used the _StandardScaler_ function in _Scikit-Learn_, a function which 
-ensures that for each feature/predictor we study the mean value is
-zero and the variance is one (every column in the design/feature
-matrix).  You would then not get the same results, since we divide by the variance. For diagonal covariance matrix elements will then be one, while the non-diagonal ones will be divided by $2\sqrt{2}$ for our specific case.
+
+Alternatively, we could use the functions we discussed
+earlier for scaling the data set.  That is, we could have used the
+_StandardScaler_ function in _Scikit-Learn_, a function which ensures
+that for each feature/predictor we study the mean value is zero and
+the variance is one (every column in the design/feature matrix).  You
+would then not get the same results, since we divide by the
+variance. The diagonal covariance matrix elements will then be one,
+while the non-diagonal ones need to be divided by $2\sqrt{2}$ for our
+specific case.
 
 === Compute the sample covariance ===
 
-Now we are going to use the mean centered data to compute the sample covariance of the data. Recall it is given by:
+Now we are going to use the mean centered data to compute the sample covariance of the data. 
 !bt
 \begin{equation*}
 \Sigma_n = \frac{1}{n-1} \sum_{i=1}^n \bar{x}_i^T \bar{x}_i = \frac{1}{n-1} \sum_{i=1}^n (x_i - \mu_n)^T (x_i - \mu_n)
@@ -841,10 +837,26 @@ Now we are going to use the mean centered data to compute the sample covariance
 where the data points $x_i \in \mathbb{R}^p$ (here in this example $p = 2$) are column vectors and $x^T$ is the transpose of $x$.
 We can write our own code or simply use either the functionaly of _numpy_ or that of _pandas_, as follows
 !bc pycod
-print(np.cov(X.T))
 print(df.cov())
 print(np.cov(X_centered.T))
 !ec
+Note that the way we define the covariance matrix here has a factor $n-1$ instead of $n$.
+Our own code here is not very elegant and asks for improvements. 
+!bc pycod
+# extract the relevant columns from the centered design matrix of dim n x 2                                                      x = X_centered[:,[0]]
+y = X_centered[:,[1]]
+Cov = np.zeros((2,2))
+Cov[0,1] = np.sum(x.T@y)/(n-1.0)
+Cov[0,0] = np.sum(x.T@x)/(n-1.0)
+Cov[1,1] = np.sum(y.T@y)/(n-1.0)
+Cov[1,0]= Cov[0,1]
+print("Centered covariance using own code")
+print(Cov)
+plt.plot(x, y, 'x')
+plt.axis('equal')
+plt.show()
+!ec
+
 Depending on the number of points $n$, we will get results that are close to the covariance values defined above.
 
 

doc/src/DimRed/Programs/PCAsimple.py

@@ -2,28 +2,47 @@ import numpy as np
 import pandas as pd
 from IPython.display import display
 import matplotlib.pyplot as plt
-from sklearn.model_selection import  train_test_split 
-from sklearn.preprocessing import MinMaxScaler, StandardScaler
 
-
-n = 10000
+n = 1000
 mean = (-1, 2)
 cov = [[4, 2], [2, 2]]
-print(cov)
 X = np.random.multivariate_normal(mean, cov, n)
-print(np.cov(X.T))
 
 df = pd.DataFrame(X)
 # Pandas does the centering for us
 df = df -df.mean()
-correlation_matrix = df.cov()
-print(correlation_matrix)
+print("Centered covariance with Pandas")
+covarianceX = df.cov()
+
+print(covarianceX)
 
 # we center it ourselves
 X_centered = X - X.mean(axis=0)
+print("Centered covariance using numpy")
 print(np.cov(X_centered.T))
-#print("test that we get the same as Pandas")
-#print(X_centered-X_train_scaled)
+# extract the relevant columns from the centered design matrix
+x = X_centered[:,[0]]
+y = X_centered[:,[1]]
+Cov = np.zeros((2,2))
+cov_xy = np.sum(x.T@y)/(n-1.0)
+cov_xx = np.sum(x.T@x)/(n-1.0)
+cov_yy = np.sum(y.T@y)/(n-1.0)
+
+Cov[0,0]= cov_xx
+Cov[1,1]= cov_yy
+Cov[0,1]= cov_xy
+Cov[1,0]= Cov[0,1]
+print("Centered covariance using own code")
+print(Cov)
+
+plt.plot(x, y, 'x')
+plt.axis('equal')
+plt.show()
+
+
+
+
+"""
 #Now we do an SVD
 U, s, V = np.linalg.svd(X_centered)
 c1 = V.T[:, 0]
@@ -38,6 +57,6 @@ print("Check that we get the same")
 print(X2D-X2Dsl)
 
 print(pca.components_.T[:, 0])
-
+"""