The covariance takes values between zero and infinity and may thus @@ -2411,7 +2416,7 @@ $$
With these definitions, we can now rewrite our \( 2\times 2 \) -correaltion/covariance matrix in terms of a moe general design/feature +correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) @@ -2454,14 +2459,12 @@ the exact mean values. The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +
+Note that this assumes you have the features as the rows, and the inputs as columns, that is
$$
-\boldsymbol{W} = \begin{bmatrix} x_0 & y_0 \\
- x_1 & y_1 \\
- x_2 & y_2\\
- \dots & \dots \\
- x_{n-2} & y_{n-2}\\
- x_{n-1} & y_{n-1} &
+\boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\
+ y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\
\end{bmatrix},
$$
@@ -2550,6 +2553,7 @@ x = np.random.normal(size=n)
x = x - np.mean(x)
y = 4+3*x+np.random.normal(size=n)
y = y - np.mean(y)
+# Note that we transpose the matrix in order to stay with our ordering n x p
X = (np.vstack((x, y))).T
print(X)
Xpd = pd.DataFrame(X)
@@ -2678,6 +2682,61 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t
+We saw earlier that +
+$$
+\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
+$$
+
+
+Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix
+
+$$
+\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix},
+$$
+
+
+where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is
+
+
+$$
+\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\
+ 0 & \sigma_1 & 0 & \dots & 0 & 0 \\
+ 0 & 0 & \sigma_2 & \dots & 0 & 0 \
+ 0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\
+ 0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\
+\end{bmatrix},
+$$
+
+
+meaning we can write
+
+$$
+\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T.
+$$
+
+
+Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get
+
+$$
+\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2.
+$$
+
+
+This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)
+with eigenvalues given by the singular values squared, that is
+
+$$
+\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2.
+$$
+
+
+
+
+
The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly @@ -2424,7 +2434,7 @@ $$
With these definitions, we can now rewrite our \( 2\times 2 \) -correaltion/covariance matrix in terms of a moe general design/feature +correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) @@ -2463,13 +2473,11 @@ the exact mean values. The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +
+Note that this assumes you have the features as the rows, and the inputs as columns, that is $$ -\boldsymbol{W} = \begin{bmatrix} x_0 & y_0 \\ - x_1 & y_1 \\ - x_2 & y_2\\ - \dots & \dots \\ - x_{n-2} & y_{n-2}\\ - x_{n-1} & y_{n-1} & +\boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\ + y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\ \end{bmatrix}, $$ @@ -2556,6 +2564,7 @@ x = np.random.normal(size=n) x = x - np.mean(x) y = 4+3*x+np.random.normal(size=n) y = y - np.mean(y) +# Note that we transpose the matrix in order to stay with our ordering n x p X = (np.vstack((x, y))).T print(X) Xpd = pd.DataFrame(X) @@ -2675,6 +2684,49 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t
+
+We saw earlier that +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix +$$ +\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, +$$ + +where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is + +$$ +\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ + 0 & \sigma_1 & 0 & \dots & 0 & 0 \\ + 0 & 0 & \sigma_2 & \dots & 0 & 0 \ + 0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\ + 0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\ +\end{bmatrix}, +$$ + +meaning we can write +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. +$$ + +Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. +$$ + +This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) +with eigenvalues given by the singular values squared, that is +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. +$$ + +
+
+
diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index 4c69e39eb..b261f20e9 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -229,6 +229,10 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'introducing-the-covariance-and-correlation-functions'), + ('Covariance and Correlation Matrix', + 2, + None, + 'covariance-and-correlation-matrix'), ('Correlation Function and Design/Feature Matrix', 2, None, @@ -250,6 +254,7 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'rewriting-the-covariance-and-or-correlation-matrix'), + ('Linking with the SVD', 2, None, 'linking-with-the-svd'), ('Ridge and LASSO Regression', 2, None, @@ -2369,6 +2374,11 @@ of the population standard deviation. If you use a library like Scikit-Learn or nunmpy's function calculate the covariance, this quantity will be computed with a factor \( 1/(n-1) \). +
+
+
+
The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly @@ -2429,7 +2439,7 @@ $$
With these definitions, we can now rewrite our \( 2\times 2 \) -correaltion/covariance matrix in terms of a moe general design/feature +correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) @@ -2468,13 +2478,11 @@ the exact mean values. The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +
+Note that this assumes you have the features as the rows, and the inputs as columns, that is $$ -\boldsymbol{W} = \begin{bmatrix} x_0 & y_0 \\ - x_1 & y_1 \\ - x_2 & y_2\\ - \dots & \dots \\ - x_{n-2} & y_{n-2}\\ - x_{n-1} & y_{n-1} & +\boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\ + y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\ \end{bmatrix}, $$ @@ -2561,6 +2569,7 @@ x = np.r x = x - np.mean(x) y = 4+3*x+np.random.normal(size=n) y = y - np.mean(y) +# Note that we transpose the matrix in order to stay with our ordering n x p X = (np.vstack((x, y))).T print(X) Xpd = pd.DataFrame(X) @@ -2680,6 +2689,49 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t
+
+We saw earlier that +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix +$$ +\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, +$$ + +where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is + +$$ +\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ + 0 & \sigma_1 & 0 & \dots & 0 & 0 \\ + 0 & 0 & \sigma_2 & \dots & 0 & 0 \ + 0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\ + 0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\ +\end{bmatrix}, +$$ + +meaning we can write +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. +$$ + +Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. +$$ + +This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) +with eigenvalues given by the singular values squared, that is +$$ +\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. +$$ + +
+
+
diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 21138d176..0978c9ee2 100644 Binary files a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz and b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz differ diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb index a931e3f06..b78df7984 100644 --- a/doc/pub/week35/ipynb/week35.ipynb +++ b/doc/pub/week35/ipynb/week35.ipynb @@ -2992,7 +2992,7 @@ "**Scikit-Learn** or **nunmpy's** function calculate the covariance, this\n", "quantity will be computed with a factor $1/(n-1)$.\n", "\n", - "\n", + "## Covariance and Correlation Matrix\n", "\n", "The covariance takes values between zero and infinity and may thus\n", "lead to problems with loss of numerical precision for particularly\n", @@ -3098,7 +3098,7 @@ "metadata": {}, "source": [ "With these definitions, we can now rewrite our $2\\times 2$\n", - "correaltion/covariance matrix in terms of a moe general design/feature\n", + "correlation/covariance matrix in terms of a moe general design/feature\n", "matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$. This leads to a $p\\times p$\n", "covariance matrix for the vectors $\\boldsymbol{x}_i$ with $i=0,1,\\dots,p-1$" ] @@ -3153,7 +3153,9 @@ "the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have\n", "the exact mean values. The following simple function uses the\n", "**np.vstack** function which takes each vector of dimension $1\\times n$\n", - "and produces a $2\\times n$ matrix $\\boldsymbol{W}$" + "and produces a $2\\times n$ matrix $\\boldsymbol{W}$\n", + "\n", + "Note that this assumes you have the features as the rows, and the inputs as columns, that is" ] }, { @@ -3161,12 +3163,8 @@ "metadata": {}, "source": [ "$$\n", - "\\boldsymbol{W} = \\begin{bmatrix} x_0 & y_0 \\\\\n", - " x_1 & y_1 \\\\\n", - " x_2 & y_2\\\\\n", - " \\dots & \\dots \\\\\n", - " x_{n-2} & y_{n-2}\\\\\n", - " x_{n-1} & y_{n-1} & \n", + "\\boldsymbol{W} = \\begin{bmatrix} x_0 & x_1 & x_2 & \\dots & x_{n-2} & x_{n-1} \\\\\n", + " y_0 & y_1 & y_2 & \\dots & y_{n-2} & y_{n-1} \\\\\n", " \\end{bmatrix},\n", "$$" ] @@ -3279,6 +3277,7 @@ "x = x - np.mean(x)\n", "y = 4+3*x+np.random.normal(size=n)\n", "y = y - np.mean(y)\n", + "# Note that we transpose the matrix in order to stay with our ordering n x p\n", "X = (np.vstack((x, y))).T\n", "print(X)\n", "Xpd = pd.DataFrame(X)\n", @@ -3445,9 +3444,110 @@ "It is easy to generalize this to a matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$.\n", "\n", "\n", + "## Linking with the SVD\n", "\n", - "\n", - "\n", + "We saw earlier that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined earlier the matrix" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where the tilde-matrix $\\tilde{\\boldsymbol{\\Sigma}}$ is a matrix of dimension $p\\times p$ containing only the singular values $\\sigma_i$, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\tilde{\\boldsymbol{\\Sigma}}=\\begin{bmatrix} \\sigma_0 & 0 & 0 & \\dots & 0 & 0 \\\\\n", + " 0 & \\sigma_1 & 0 & \\dots & 0 & 0 \\\\\n", + "\t\t\t\t 0 & 0 & \\sigma_2 & \\dots & 0 & 0 \\\n", + "\t\t\t\t 0 & 0 & 0 & \\dots & \\sigma_{p-2} & 0 \\\\\n", + "\t\t\t\t 0 & 0 & 0 & \\dots & 0 & \\sigma_{p-1} \\\\\n", + "\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "meaning we can write" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2\\boldsymbol{V}^T.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Multiplying from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This means the vectors $\\boldsymbol{v}_i$ of the orthogonal matrix $\\boldsymbol{V}$ are the eigenvectors of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$\n", + "with eigenvalues given by the singular values squared, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ "## Ridge and LASSO Regression\n", "\n", "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index 6331b028a..9216eccec 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -1870,7 +1870,8 @@ of the population standard deviation. If you use a library like _Scikit-Learn_ or _nunmpy's_ function calculate the covariance, this quantity will be computed with a factor $1/(n-1)$. - +!split +===== Covariance and Correlation Matrix ===== The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly @@ -1933,7 +1934,7 @@ with a given vector !et With these definitions, we can now rewrite our $2\times 2$ -correaltion/covariance matrix in terms of a moe general design/feature +correlation/covariance matrix in terms of a moe general design/feature matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. This leads to a $p\times p$ covariance matrix for the vectors $\bm{x}_i$ with $i=0,1,\dots,p-1$ @@ -1974,15 +1975,11 @@ the exact mean values. The following simple function uses the _np.vstack_ function which takes each vector of dimension $1\times n$ and produces a $2\times n$ matrix $\bm{W}$ - +Note that this assumes you have the features as the rows, and the inputs as columns, that is !bt \[ -\bm{W} = \begin{bmatrix} x_0 & y_0 \\ - x_1 & y_1 \\ - x_2 & y_2\\ - \dots & \dots \\ - x_{n-2} & y_{n-2}\\ - x_{n-1} & y_{n-1} & +\bm{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\ + y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\ \end{bmatrix}, \] !et @@ -2057,6 +2054,7 @@ x = np.random.normal(size=n) x = x - np.mean(x) y = 4+3*x+np.random.normal(size=n) y = y - np.mean(y) +# Note that we transpose the matrix in order to stay with our ordering n x p X = (np.vstack((x, y))).T print(X) Xpd = pd.DataFrame(X) @@ -2171,6 +2169,55 @@ where we wrote $$\bm{C}[\bm{x}_0,\bm{x}_1] = \bm{C}[\bm{x}]$$ to indicate that t It is easy to generalize this to a matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. +!split +===== Linking with the SVD ===== + +We saw earlier that +!bt +\[ +\bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T=\bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T. +\] +!et +Since the matrices here have dimension $p\times p$, with $p$ corresponding to the singular values, we defined earlier the matrix +!bt +\[ +\bm{\Sigma}^T\bm{\Sigma} = \begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\bm{\Sigma}} \\ \bm{0}\\ \end{bmatrix}, +\] +!et +where the tilde-matrix $\tilde{\bm{\Sigma}}$ is a matrix of dimension $p\times p$ containing only the singular values $\sigma_i$, that is + +!bt +\[ +\tilde{\bm{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ + 0 & \sigma_1 & 0 & \dots & 0 & 0 \\ + 0 & 0 & \sigma_2 & \dots & 0 & 0 \ + 0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\ + 0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\ +\end{bmatrix}, +\] +!et +meaning we can write +!bt +\[ +\bm{X}^T\bm{X}=\bm{V}\tilde{\bm{\Sigma}}^2\bm{V}^T. +\] +!et +Multiplying from the right with $\bm{V}$ (using the orthogonality of $\bm{V}$) we get +!bt +\[ +\left(\bm{X}^T\bm{X}\right)\bm{V}=\bm{V}\tilde{\bm{\Sigma}}^2. +\] +!et +This means the vectors $\bm{v}_i$ of the orthogonal matrix $\bm{V}$ are the eigenvectors of the matrix $\bm{X}^T\bm{X}$ +with eigenvalues given by the singular values squared, that is +!bt +\[ +\left(\bm{X}^T\bm{X}\right)\bm{v}_i=\bm{v}_i\sigma_i^2. +\] +!et + + +