week43 update

2020-10-23 05:48:11 +02:00
parent ec5fe9b414
commit 83da23092b
14 changed files with 173 additions and 470 deletions
@@ -323,7 +323,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Oct 22, 2020</h4></center> <!-- date -->
+<center><h4>Oct 23, 2020</h4></center> <!-- date -->
 <br>
 <p>

@@ -305,8 +305,8 @@ MathJax.Hub.Config({
 <!-- !split -->

 <ul>
-<li> Thursday: Wrapping up Recurrent Neural Networks and solving differential equations.</li> 
-<li> Friday: Principal Component Analysis and Dimensionality Reduction</li>
+<li> Thursday: Wrapping up Recurrent Neural Networks and solving differential equations. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober22.mp4?vrtx=view-as-webpage" target="_self">Video of Lecture October 22</a></li>
+<li> Friday: Principal Component Analysis and Dimensionality Reduction. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober23.mp4?vrtx=view-as-webpage" target="_self">Video of Lecture October 23</a></li>
 </ul>

 We will also study the usage of <a href="https://www.youtube.com/watch?v=fRf4l5qaX1M&ab_channel=AlexSmola" target="_self">Autograd</a> in computing gradients for deep learning. For the documentation of Autograd and examples see the lectures slides from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week40/html/week40.html" target="_self">week 40</a> and the <a href="https://github.com/HIPS/autograd" target="_self">Autograd doucmentation</a>.
@@ -309,11 +309,11 @@ MathJax.Hub.Config({
 <p>
 The principal component analysis deals with the problem of fitting a
 low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than
-the totaldimension \( D \) of the problem at hand (our data
+the total dimension \( D \) of the problem at hand (our data
 set). Mathematically it can be formulated as a statistical problem or
 a geometric problem.  In our discussion of the theorem for the
-classical PCA, we will stay with a statistical approach. This is also
-what set the scene historically which for the PCA.
+classical PCA, we will stay with a statistical approach. 
+Historically, the PCA was first formulated in a statistical setting in order to estimate the principal component of a multivariate random variable.

 <p>
 We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition) 
@@ -324,6 +324,9 @@ We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see
 <li> If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.</li> 
 </ul>

+A good read is for example <a href="https://www.springer.com/gp/book/9780387878102" target="_self">Vidal, Ma and Sastry</a>.
+
+<p>
 <p>
 <!-- navigation buttons at the bottom of the page -->
 <ul class="pagination">
@@ -356,13 +356,10 @@ covariance_matrix <span style="color: #666666">=</span> Xpd<span style="color: #
 <p>
 We note here that the covariance is zero for the first rows and
 columns since all matrix elements in the design matrix were set to one
-(we are fitting the function in terms of a polynomial of degree \( n \)).
-
-<p>
-This means that the variance for these elements will be zero and will
-cause problems when we set up the correlation matrix.  We can simply
+(we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept
+and wee can simply
 drop these elements and construct a correlation
-matrix without these elements.
+matrix without them.

 <p>
 <p>
@@ -331,9 +331,6 @@ low-dimensional encoding of the data is then given by a set of vectors
 orthogonal projection of the data onto the columns spanned by the
 eigenvectors of the covariance(correlations matrix).

-<p>
-The proof which follows will be updated by mid January 2020.
-
 <p>
 <p>
 <!-- navigation buttons at the bottom of the page -->
@@ -353,7 +353,7 @@ the Singular Value Decomposition theorem. For categorical data, see
 chapter 12.4 and discussion therein.

 <p>
-Additional part of the proof for the other eigenvectors will be added by mid January 2020.
+For more details, see for example <a href="https://www.springer.com/gp/book/9780387878102" target="_self">Vidal, Ma and Sastry, chapter 2</a>.

 <p>
 <p>
@@ -307,7 +307,7 @@ MathJax.Hub.Config({
 <h2 id="___sec68" class="anchor">Geometric Interpretation and link with Singular Value Decomposition </h2>

 <p>
-This material will be added by mid January 2020.
+For a detailed demonstration of the geometric interpretation, see <a href="https://www.springer.com/gp/book/9780387878102" target="_self">Vidal, Ma and Sastry, section 2.1.2</a>.

 <p>
 <p>
@@ -323,7 +323,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Oct 22, 2020</h4></center> <!-- date -->
+<center><h4>Oct 23, 2020</h4></center> <!-- date -->
 <br>
 <p>

@@ -148,7 +148,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>&nbsp;<br>
-<center><h4>Oct 22, 2020</h4></center> <!-- date -->
+<center><h4>Oct 23, 2020</h4></center> <!-- date -->
 <br>
 <p>

@@ -161,8 +161,8 @@ MathJax.Hub.Config({
 <section>

 <ul>
-<p><li> Thursday: Wrapping up Recurrent Neural Networks and solving differential equations.</li> 
-<p><li> Friday: Principal Component Analysis and Dimensionality Reduction</li>
+<p><li> Thursday: Wrapping up Recurrent Neural Networks and solving differential equations. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober22.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture October 22</a></li>
+<p><li> Friday: Principal Component Analysis and Dimensionality Reduction. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober23.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture October 23</a></li>
 </ul>
 <p>

@@ -2953,11 +2953,11 @@ $$
 <p>
 The principal component analysis deals with the problem of fitting a
 low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than
-the totaldimension \( D \) of the problem at hand (our data
+the total dimension \( D \) of the problem at hand (our data
 set). Mathematically it can be formulated as a statistical problem or
 a geometric problem.  In our discussion of the theorem for the
-classical PCA, we will stay with a statistical approach. This is also
-what set the scene historically which for the PCA.
+classical PCA, we will stay with a statistical approach. 
+Historically, the PCA was first formulated in a statistical setting in order to estimate the principal component of a multivariate random variable.

 <p>
 We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition) 
@@ -2967,6 +2967,9 @@ We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see
 <p><li> We may want to ask the following question: Are there fewer intrinsic variables (say \( d < < p \)) that still approximately describe the data?</li>
 <p><li> If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.</li> 
 </ul>
+<p>
+
+A good read is for example <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry</a>.
 </section>


@@ -3285,13 +3288,10 @@ covariance_matrix = Xpd.cov()
 <p>
 We note here that the covariance is zero for the first rows and
 columns since all matrix elements in the design matrix were set to one
-(we are fitting the function in terms of a polynomial of degree \( n \)).
-
-<p>
-This means that the variance for these elements will be zero and will
-cause problems when we set up the correlation matrix.  We can simply
+(we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept
+and wee can simply
 drop these elements and construct a correlation
-matrix without these elements.
+matrix without them.
 </section>


@@ -3654,9 +3654,6 @@ low-dimensional encoding of the data is then given by a set of vectors
 \( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the
 orthogonal projection of the data onto the columns spanned by the
 eigenvectors of the covariance(correlations matrix).
-
-<p>
-The proof which follows will be updated by mid January 2020.
 </section>


@@ -3798,7 +3795,7 @@ the Singular Value Decomposition theorem. For categorical data, see
 chapter 12.4 and discussion therein.

 <p>
-Additional part of the proof for the other eigenvectors will be added by mid January 2020.
+For more details, see for example <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry, chapter 2</a>.
 </section>


@@ -3806,7 +3803,7 @@ Additional part of the proof for the other eigenvectors will be added by mid Jan
 <h2 id="___sec68">Geometric Interpretation and link with Singular Value Decomposition </h2>

 <p>
-This material will be added by mid January 2020.
+For a detailed demonstration of the geometric interpretation, see <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry, section 2.1.2</a>.
 </section>


@@ -235,14 +235,14 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Oct 22, 2020</h4></center> <!-- date -->
+<center><h4>Oct 23, 2020</h4></center> <!-- date -->
 <br>
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

 <ul>
-<li> Thursday: Wrapping up Recurrent Neural Networks and solving differential equations.</li> 
-<li> Friday: Principal Component Analysis and Dimensionality Reduction</li>
+<li> Thursday: Wrapping up Recurrent Neural Networks and solving differential equations. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober22.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture October 22</a></li>
+<li> Friday: Principal Component Analysis and Dimensionality Reduction. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober23.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture October 23</a></li>
 </ul>

 We will also study the usage of <a href="https://www.youtube.com/watch?v=fRf4l5qaX1M&ab_channel=AlexSmola" target="_blank">Autograd</a> in computing gradients for deep learning. For the documentation of Autograd and examples see the lectures slides from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week40/html/week40.html" target="_blank">week 40</a> and the <a href="https://github.com/HIPS/autograd" target="_blank">Autograd doucmentation</a>.
@@ -2893,11 +2893,11 @@ $$
 <p>
 The principal component analysis deals with the problem of fitting a
 low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than
-the totaldimension \( D \) of the problem at hand (our data
+the total dimension \( D \) of the problem at hand (our data
 set). Mathematically it can be formulated as a statistical problem or
 a geometric problem.  In our discussion of the theorem for the
-classical PCA, we will stay with a statistical approach. This is also
-what set the scene historically which for the PCA.
+classical PCA, we will stay with a statistical approach. 
+Historically, the PCA was first formulated in a statistical setting in order to estimate the principal component of a multivariate random variable.

 <p>
 We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition) 
@@ -2908,6 +2908,9 @@ We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see
 <li> If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.</li> 
 </ul>

+A good read is for example <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry</a>.
+
+<p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

 <h2 id="___sec51">Introducing the Covariance and Correlation functions  </h2>
@@ -3199,13 +3202,10 @@ covariance_matrix = Xpd.cov()
 <p>
 We note here that the covariance is zero for the first rows and
 columns since all matrix elements in the design matrix were set to one
-(we are fitting the function in terms of a polynomial of degree \( n \)).
-
-<p>
-This means that the variance for these elements will be zero and will
-cause problems when we set up the correlation matrix.  We can simply
+(we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept
+and wee can simply
 drop these elements and construct a correlation
-matrix without these elements.
+matrix without them.

 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -3538,9 +3538,6 @@ low-dimensional encoding of the data is then given by a set of vectors
 orthogonal projection of the data onto the columns spanned by the
 eigenvectors of the covariance(correlations matrix).

-<p>
-The proof which follows will be updated by mid January 2020.
-
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

@@ -3657,7 +3654,7 @@ the Singular Value Decomposition theorem. For categorical data, see
 chapter 12.4 and discussion therein.

 <p>
-Additional part of the proof for the other eigenvectors will be added by mid January 2020.
+For more details, see for example <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry, chapter 2</a>.

 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -3665,7 +3662,7 @@ Additional part of the proof for the other eigenvectors will be added by mid Jan
 <h2 id="___sec68">Geometric Interpretation and link with Singular Value Decomposition </h2>

 <p>
-This material will be added by mid January 2020.
+For a detailed demonstration of the geometric interpretation, see <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry, section 2.1.2</a>.

 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -240,14 +240,14 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Oct 22, 2020</h4></center> <!-- date -->
+<center><h4>Oct 23, 2020</h4></center> <!-- date -->
 <br>
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

 <ul>
-<li> Thursday: Wrapping up Recurrent Neural Networks and solving differential equations.</li> 
-<li> Friday: Principal Component Analysis and Dimensionality Reduction</li>
+<li> Thursday: Wrapping up Recurrent Neural Networks and solving differential equations. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober22.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture October 22</a></li>
+<li> Friday: Principal Component Analysis and Dimensionality Reduction. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober23.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture October 23</a></li>
 </ul>

 We will also study the usage of <a href="https://www.youtube.com/watch?v=fRf4l5qaX1M&ab_channel=AlexSmola" target="_blank">Autograd</a> in computing gradients for deep learning. For the documentation of Autograd and examples see the lectures slides from <a href="https://compphysics.github.io/MachineLearning/doc/pub/week40/html/week40.html" target="_blank">week 40</a> and the <a href="https://github.com/HIPS/autograd" target="_blank">Autograd doucmentation</a>.
@@ -2898,11 +2898,11 @@ $$
 <p>
 The principal component analysis deals with the problem of fitting a
 low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than
-the totaldimension \( D \) of the problem at hand (our data
+the total dimension \( D \) of the problem at hand (our data
 set). Mathematically it can be formulated as a statistical problem or
 a geometric problem.  In our discussion of the theorem for the
-classical PCA, we will stay with a statistical approach. This is also
-what set the scene historically which for the PCA.
+classical PCA, we will stay with a statistical approach. 
+Historically, the PCA was first formulated in a statistical setting in order to estimate the principal component of a multivariate random variable.

 <p>
 We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition) 
@@ -2913,6 +2913,9 @@ We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see
 <li> If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.</li> 
 </ul>

+A good read is for example <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry</a>.
+
+<p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

 <h2 id="___sec51">Introducing the Covariance and Correlation functions  </h2>
@@ -3204,13 +3207,10 @@ covariance_matrix <span style="color: #666666">=</span> Xpd<span style="color: #
 <p>
 We note here that the covariance is zero for the first rows and
 columns since all matrix elements in the design matrix were set to one
-(we are fitting the function in terms of a polynomial of degree \( n \)).
-
-<p>
-This means that the variance for these elements will be zero and will
-cause problems when we set up the correlation matrix.  We can simply
+(we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept
+and wee can simply
 drop these elements and construct a correlation
-matrix without these elements.
+matrix without them.

 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -3543,9 +3543,6 @@ low-dimensional encoding of the data is then given by a set of vectors
 orthogonal projection of the data onto the columns spanned by the
 eigenvectors of the covariance(correlations matrix).

-<p>
-The proof which follows will be updated by mid January 2020.
-
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

@@ -3662,7 +3659,7 @@ the Singular Value Decomposition theorem. For categorical data, see
 chapter 12.4 and discussion therein.

 <p>
-Additional part of the proof for the other eigenvectors will be added by mid January 2020.
+For more details, see for example <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry, chapter 2</a>.

 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -3670,7 +3667,7 @@ Additional part of the proof for the other eigenvectors will be added by mid Jan
 <h2 id="___sec68">Geometric Interpretation and link with Singular Value Decomposition </h2>

 <p>
-This material will be added by mid January 2020.
+For a detailed demonstration of the geometric interpretation, see <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry, section 2.1.2</a>.

 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2411,17 +2411,19 @@ o "Introduction to Partial Differential Equations by A. Tveito, R. Winther":"htt

 The principal component analysis deals with the problem of fitting a
 low-dimensional affine subspace $S$ of dimension $d$ much smaller than
-the totaldimension $D$ of the problem at hand (our data
+the total dimension $D$ of the problem at hand (our data
 set). Mathematically it can be formulated as a statistical problem or
 a geometric problem.  In our discussion of the theorem for the
-classical PCA, we will stay with a statistical approach. This is also
-what set the scene historically which for the PCA.
+classical PCA, we will stay with a statistical approach. 
+Historically, the PCA was first formulated in a statistical setting in order to estimate the principal component of a multivariate random variable.

 We have a data set defined by a design/feature matrix $\bm{X}$ (see below for its definition) 
 * Each data point is determined by $p$ extrinsic (measurement) variables
 * We may want to ask the following question: Are there fewer intrinsic variables (say $d << p$) that still approximately describe the data?
 * If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do. 

+A good read is for example "Vidal, Ma and Sastry":"https://www.springer.com/gp/book/9780387878102".
+

 !split
 ===== Introducing the Covariance and Correlation functions  =====
@@ -2706,12 +2708,10 @@ print(covariance_matrix)

 We note here that the covariance is zero for the first rows and
 columns since all matrix elements in the design matrix were set to one
-(we are fitting the function in terms of a polynomial of degree $n$).
-
-This means that the variance for these elements will be zero and will
-cause problems when we set up the correlation matrix.  We can simply
+(we are fitting the function in terms of a polynomial of degree $n$). We would however not include the intercept
+and wee can simply
 drop these elements and construct a correlation
-matrix without these elements. 
+matrix without them. 


 !split
@@ -3013,7 +3013,7 @@ $\bm{z}_i$ with at most $l$ vectors, with $l << p$, defined by the
 orthogonal projection of the data onto the columns spanned by the
 eigenvectors of the covariance(correlations matrix).

-The proof which follows will be updated by mid January 2020.
+

 !split
 ===== Proof of the PCA Theorem  =====
@@ -3132,12 +3132,12 @@ discussion in chapter 12.2 of Murphy's text has also a nice link with
 the Singular Value Decomposition theorem. For categorical data, see
 chapter 12.4 and discussion therein.

-Additional part of the proof for the other eigenvectors will be added by mid January 2020.
+For more details, see for example "Vidal, Ma and Sastry, chapter 2":"https://www.springer.com/gp/book/9780387878102".

 !split
 ===== Geometric Interpretation and link with Singular Value Decomposition =====

-This material will be added by mid January 2020.
+For a detailed demonstration of the geometric interpretation, see "Vidal, Ma and Sastry, section 2.1.2":"https://www.springer.com/gp/book/9780387878102".


 !split