+We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition) + +
-The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function +The covariance takes values between zero and infinity and may thus +lead to problems with loss of numerical precision for particularly +large values. It is common to scale the covariance matrix by +introducing instead the correlation matrix defined via the so-called +correlation function + $$ corr[\boldsymbol{x},\boldsymbol{y}]=\frac{cov[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{var[\boldsymbol{x}]\var\boldsymbol{y}]}}. $$ -The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}] \in [-1,1] \). This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) and \( \boldsymbol{y} \) as +
+The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}]
+\in [-1,1] \). This avoids eventual problems with too large values. We
+can then define the correlation matrix for the two vectors \( \boldsymbol{x} \)
+and \( \boldsymbol{y} \) as
+
$$
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & corr[\boldsymbol{x},\boldsymbol{y}] \\
corr[\boldsymbol{y},\boldsymbol{x}] & 1 \\
diff --git a/doc/pub/DimRed/html/DimRed-reveal.html b/doc/pub/DimRed/html/DimRed-reveal.html
index 20496dcee..2513c9634 100644
--- a/doc/pub/DimRed/html/DimRed-reveal.html
+++ b/doc/pub/DimRed/html/DimRed-reveal.html
@@ -509,6 +509,15 @@ applications.
+We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)
+
+Basic ideas of the Principal Component Analysis (PCA)
+
+
+
-The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function +The covariance takes values between zero and infinity and may thus +lead to problems with loss of numerical precision for particularly +large values. It is common to scale the covariance matrix by +introducing instead the correlation matrix defined via the so-called +correlation function +
$$
corr[\boldsymbol{x},\boldsymbol{y}]=\frac{cov[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{var[\boldsymbol{x}]\var\boldsymbol{y}]}}.
$$
-The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}] \in [-1,1] \). This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) and \( \boldsymbol{y} \) as
+
+The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}] +\in [-1,1] \). This avoids eventual problems with too large values. We +can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) +and \( \boldsymbol{y} \) as +
$$
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & corr[\boldsymbol{x},\boldsymbol{y}] \\
@@ -609,9 +628,28 @@ $$
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \)
$$
-\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} var[\boldsymbol{x}_0] & cov[\boldsymbol{x}_0,\boldsymbol{x}_1] \\
- cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{x}_{p-1}] \\
- \end{bmatrix},
+\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix}
+var[\boldsymbol{x}_0] & cov[\boldsymbol{x}_0,\boldsymbol{x}_1] & cov[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
+cov[\boldsymbol{x}_1,\boldsymbol{x}_0] & var[\boldsymbol{x}_1] & cov[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
+cov[\boldsymbol{x}_2,\boldsymbol{x}_0] & cov[\boldsymbol{x}_2,\boldsymbol{x}_1] & var[\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
+\dots & \dots & \dots & \dots & \dots & \dots \\
+\dots & \dots & \dots & \dots & \dots & \dots \\
+cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & var[\boldsymbol{x}_{p-1}]\\
+\end{bmatrix},
+$$
+
+
+and the correlation matrix
+
+$$
+\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix}
+1 & corr[\boldsymbol{x}_0,\boldsymbol{x}_1] & corr[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & corr[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
+corr[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & corr[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & corr[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
+corr[\boldsymbol{x}_2,\boldsymbol{x}_0] & corr[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & corr[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
+\dots & \dots & \dots & \dots & \dots & \dots \\
+\dots & \dots & \dots & \dots & \dots & \dots \\
+corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\
+\end{bmatrix},
$$
diff --git a/doc/pub/DimRed/html/DimRed-solarized.html b/doc/pub/DimRed/html/DimRed-solarized.html
index 6cf01e88c..dbbe0c5ef 100644
--- a/doc/pub/DimRed/html/DimRed-solarized.html
+++ b/doc/pub/DimRed/html/DimRed-solarized.html
@@ -508,6 +508,14 @@ applications.
+We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition) + +
-The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function +The covariance takes values between zero and infinity and may thus +lead to problems with loss of numerical precision for particularly +large values. It is common to scale the covariance matrix by +introducing instead the correlation matrix defined via the so-called +correlation function + $$ corr[\boldsymbol{x},\boldsymbol{y}]=\frac{cov[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{var[\boldsymbol{x}]\var\boldsymbol{y}]}}. $$ -The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}] \in [-1,1] \). This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) and \( \boldsymbol{y} \) as +
+The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}] +\in [-1,1] \). This avoids eventual problems with too large values. We +can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) +and \( \boldsymbol{y} \) as + $$ \boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & corr[\boldsymbol{x},\boldsymbol{y}] \\ corr[\boldsymbol{y},\boldsymbol{x}] & 1 \\ @@ -587,9 +605,26 @@ $$
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \) $$ -\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} var[\boldsymbol{x}_0] & cov[\boldsymbol{x}_0,\boldsymbol{x}_1] \\ - cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{x}_{p-1}] \\ - \end{bmatrix}, +\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} +var[\boldsymbol{x}_0] & cov[\boldsymbol{x}_0,\boldsymbol{x}_1] & cov[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +cov[\boldsymbol{x}_1,\boldsymbol{x}_0] & var[\boldsymbol{x}_1] & cov[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +cov[\boldsymbol{x}_2,\boldsymbol{x}_0] & cov[\boldsymbol{x}_2,\boldsymbol{x}_1] & var[\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & var[\boldsymbol{x}_{p-1}]\\ +\end{bmatrix}, +$$ + +and the correlation matrix +$$ +\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} +1 & corr[\boldsymbol{x}_0,\boldsymbol{x}_1] & corr[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & corr[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +corr[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & corr[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & corr[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +corr[\boldsymbol{x}_2,\boldsymbol{x}_0] & corr[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & corr[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\ +\end{bmatrix}, $$
diff --git a/doc/pub/DimRed/html/DimRed.html b/doc/pub/DimRed/html/DimRed.html index ddbbd9055..8478d5b7f 100644 --- a/doc/pub/DimRed/html/DimRed.html +++ b/doc/pub/DimRed/html/DimRed.html @@ -513,6 +513,14 @@ applications.
+We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition) + +
-The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function +The covariance takes values between zero and infinity and may thus +lead to problems with loss of numerical precision for particularly +large values. It is common to scale the covariance matrix by +introducing instead the correlation matrix defined via the so-called +correlation function + $$ corr[\boldsymbol{x},\boldsymbol{y}]=\frac{cov[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{var[\boldsymbol{x}]\var\boldsymbol{y}]}}. $$ -The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}] \in [-1,1] \). This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) and \( \boldsymbol{y} \) as +
+The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}] +\in [-1,1] \). This avoids eventual problems with too large values. We +can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) +and \( \boldsymbol{y} \) as + $$ \boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & corr[\boldsymbol{x},\boldsymbol{y}] \\ corr[\boldsymbol{y},\boldsymbol{x}] & 1 \\ @@ -592,9 +610,26 @@ $$
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \) $$ -\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} var[\boldsymbol{x}_0] & cov[\boldsymbol{x}_0,\boldsymbol{x}_1] \\ - cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{x}_{p-1}] \\ - \end{bmatrix}, +\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} +var[\boldsymbol{x}_0] & cov[\boldsymbol{x}_0,\boldsymbol{x}_1] & cov[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +cov[\boldsymbol{x}_1,\boldsymbol{x}_0] & var[\boldsymbol{x}_1] & cov[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +cov[\boldsymbol{x}_2,\boldsymbol{x}_0] & cov[\boldsymbol{x}_2,\boldsymbol{x}_1] & var[\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & var[\boldsymbol{x}_{p-1}]\\ +\end{bmatrix}, +$$ + +and the correlation matrix +$$ +\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} +1 & corr[\boldsymbol{x}_0,\boldsymbol{x}_1] & corr[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & corr[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ +corr[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & corr[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & corr[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\ +corr[\boldsymbol{x}_2,\boldsymbol{x}_0] & corr[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & corr[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\ +\end{bmatrix}, $$
diff --git a/doc/pub/DimRed/ipynb/DimRed.ipynb b/doc/pub/DimRed/ipynb/DimRed.ipynb index e764465eb..802dd81f3 100644 --- a/doc/pub/DimRed/ipynb/DimRed.ipynb +++ b/doc/pub/DimRed/ipynb/DimRed.ipynb @@ -402,6 +402,12 @@ "\n", "## Basic ideas of the Principal Component Analysis (PCA)\n", "\n", + "We have a data set defined by a design/feature matrix $\\boldsymbol{X}$ (see below for its definition) \n", + "* So each data point is determined by $p$ extrinsic (measurement) variables\n", + "\n", + "* We may want to ask the following question: Are there fewer intrinsic variables (say $d << p$) that still approximately describe the data?\n", + "\n", + "* If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do\n", "\n", "## Introducing the Covariance and Correlation functions\n", "\n", @@ -474,7 +480,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function" + "The covariance takes values between zero and infinity and may thus\n", + "lead to problems with loss of numerical precision for particularly\n", + "large values. It is common to scale the covariance matrix by\n", + "introducing instead the correlation matrix defined via the so-called\n", + "correlation function" ] }, { @@ -490,7 +500,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The correlation function is then given by values $corr[\\boldsymbol{x},\\boldsymbol{y}] \\in [-1,1]$. This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors $\\boldsymbol{x}$ and $\\boldsymbol{y}$ as" + "The correlation function is then given by values $corr[\\boldsymbol{x},\\boldsymbol{y}]\n", + "\\in [-1,1]$. This avoids eventual problems with too large values. We\n", + "can then define the correlation matrix for the two vectors $\\boldsymbol{x}$\n", + "and $\\boldsymbol{y}$ as" ] }, { @@ -577,9 +590,37 @@ "metadata": {}, "source": [ "$$\n", - "\\boldsymbol{C}[\\boldsymbol{x}] = \\begin{bmatrix} var[\\boldsymbol{x}_0] & cov[\\boldsymbol{x}_0,\\boldsymbol{x}_1] \\\\\n", - " cov[\\boldsymbol{x},\\boldsymbol{y}] & var[\\boldsymbol{x}_{p-1}] \\\\\n", - " \\end{bmatrix},\n", + "\\boldsymbol{C}[\\boldsymbol{x}] = \\begin{bmatrix}\n", + "var[\\boldsymbol{x}_0] & cov[\\boldsymbol{x}_0,\\boldsymbol{x}_1] & cov[\\boldsymbol{x}_0,\\boldsymbol{x}_2] & \\dots & \\dots & cov[\\boldsymbol{x}_0,\\boldsymbol{x}_{p-1}]\\\\\n", + "cov[\\boldsymbol{x}_1,\\boldsymbol{x}_0] & var[\\boldsymbol{x}_1] & cov[\\boldsymbol{x}_1,\\boldsymbol{x}_2] & \\dots & \\dots & cov[\\boldsymbol{x}_1,\\boldsymbol{x}_{p-1}]\\\\\n", + "cov[\\boldsymbol{x}_2,\\boldsymbol{x}_0] & cov[\\boldsymbol{x}_2,\\boldsymbol{x}_1] & var[\\boldsymbol{x}_2] & \\dots & \\dots & cov[\\boldsymbol{x}_2,\\boldsymbol{x}_{p-1}]\\\\\n", + "\\dots & \\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", + "\\dots & \\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", + "cov[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_0] & cov[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_1] & cov[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_{2}] & \\dots & \\dots & var[\\boldsymbol{x}_{p-1}]\\\\\n", + "\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and the correlation matrix" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{K}[\\boldsymbol{x}] = \\begin{bmatrix}\n", + "1 & corr[\\boldsymbol{x}_0,\\boldsymbol{x}_1] & corr[\\boldsymbol{x}_0,\\boldsymbol{x}_2] & \\dots & \\dots & corr[\\boldsymbol{x}_0,\\boldsymbol{x}_{p-1}]\\\\\n", + "corr[\\boldsymbol{x}_1,\\boldsymbol{x}_0] & 1 & corr[\\boldsymbol{x}_1,\\boldsymbol{x}_2] & \\dots & \\dots & corr[\\boldsymbol{x}_1,\\boldsymbol{x}_{p-1}]\\\\\n", + "corr[\\boldsymbol{x}_2,\\boldsymbol{x}_0] & corr[\\boldsymbol{x}_2,\\boldsymbol{x}_1] & 1 & \\dots & \\dots & corr[\\boldsymbol{x}_2,\\boldsymbol{x}_{p-1}]\\\\\n", + "\\dots & \\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", + "\\dots & \\dots & \\dots & \\dots & \\dots & \\dots \\\\\n", + "corr[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_0] & corr[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_1] & corr[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_{2}] & \\dots & \\dots & 1\\\\\n", + "\\end{bmatrix},\n", "$$" ] }, diff --git a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz index 559fe806e..990230aad 100644 Binary files a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz and b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz differ diff --git a/doc/pub/DimRed/pdf/DimRed-minted.pdf b/doc/pub/DimRed/pdf/DimRed-minted.pdf index 463adcf4d..50d765d8b 100644 Binary files a/doc/pub/DimRed/pdf/DimRed-minted.pdf and b/doc/pub/DimRed/pdf/DimRed-minted.pdf differ diff --git a/doc/src/DimRed/DimRed.do.txt b/doc/src/DimRed/DimRed.do.txt index a66916083..8d74941d7 100644 --- a/doc/src/DimRed/DimRed.do.txt +++ b/doc/src/DimRed/DimRed.do.txt @@ -331,6 +331,11 @@ applications. !split ===== Basic ideas of the Principal Component Analysis (PCA) ===== +We have a data set defined by a design/feature matrix $\bm{X}$ (see below for its definition) +* So each data point is determined by $p$ extrinsic (measurement) variables +* We may want to ask the following question: Are there fewer intrinsic variables (say $d << p$) that still approximately describe the data? +* If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do + !split ===== Introducing the Covariance and Correlation functions ===== @@ -365,13 +370,23 @@ we can rewrite the covariance matrix in this case as \] !et -The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function +The covariance takes values between zero and infinity and may thus +lead to problems with loss of numerical precision for particularly +large values. It is common to scale the covariance matrix by +introducing instead the correlation matrix defined via the so-called +correlation function + !bt \[ corr[\bm{x},\bm{y}]=\frac{cov[\bm{x},\bm{y}]}{\sqrt{var[\bm{x}]\var\bm{y}]}}. \] !et -The correlation function is then given by values $corr[\bm{x},\bm{y}] \in [-1,1]$. This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors $\bm{x}$ and $\bm{y}$ as + +The correlation function is then given by values $corr[\bm{x},\bm{y}] +\in [-1,1]$. This avoids eventual problems with too large values. We +can then define the correlation matrix for the two vectors $\bm{x}$ +and $\bm{y}$ as + !bt \[ \bm{K}[\bm{x},\bm{y}] = \begin{bmatrix} 1 & corr[\bm{x},\bm{y}] \\ @@ -416,11 +431,30 @@ with a given vector With these definitions, we can now rewrite our $2\times 2$ correaltion/covariance matrix in terms of a moe general design/feature matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. This leads to a $p\times p$ covariance matrix for the vectors $\bm{x}_i$ with $i =0,1,\dots,p-1$ !bt \[ -\bm{C}[\bm{x}] = \begin{bmatrix} var[\bm{x}_0] & cov[\bm{x}_0,\bm{x}_1] \\ - cov[\bm{x},\bm{y}] & var[\bm{x}_{p-1}] \\ - \end{bmatrix}, +\bm{C}[\bm{x}] = \begin{bmatrix} +var[\bm{x}_0] & cov[\bm{x}_0,\bm{x}_1] & cov[\bm{x}_0,\bm{x}_2] & \dots & \dots & cov[\bm{x}_0,\bm{x}_{p-1}]\\ +cov[\bm{x}_1,\bm{x}_0] & var[\bm{x}_1] & cov[\bm{x}_1,\bm{x}_2] & \dots & \dots & cov[\bm{x}_1,\bm{x}_{p-1}]\\ +cov[\bm{x}_2,\bm{x}_0] & cov[\bm{x}_2,\bm{x}_1] & var[\bm{x}_2] & \dots & \dots & cov[\bm{x}_2,\bm{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +cov[\bm{x}_{p-1},\bm{x}_0] & cov[\bm{x}_{p-1},\bm{x}_1] & cov[\bm{x}_{p-1},\bm{x}_{2}] & \dots & \dots & var[\bm{x}_{p-1}]\\ +\end{bmatrix}, \] !et +and the correlation matrix +!bt +\[ +\bm{K}[\bm{x}] = \begin{bmatrix} +1 & corr[\bm{x}_0,\bm{x}_1] & corr[\bm{x}_0,\bm{x}_2] & \dots & \dots & corr[\bm{x}_0,\bm{x}_{p-1}]\\ +corr[\bm{x}_1,\bm{x}_0] & 1 & corr[\bm{x}_1,\bm{x}_2] & \dots & \dots & corr[\bm{x}_1,\bm{x}_{p-1}]\\ +corr[\bm{x}_2,\bm{x}_0] & corr[\bm{x}_2,\bm{x}_1] & 1 & \dots & \dots & corr[\bm{x}_2,\bm{x}_{p-1}]\\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +\dots & \dots & \dots & \dots & \dots & \dots \\ +corr[\bm{x}_{p-1},\bm{x}_0] & corr[\bm{x}_{p-1},\bm{x}_1] & corr[\bm{x}_{p-1},\bm{x}_{2}] & \dots & \dots & 1\\ +\end{bmatrix}, +\] +!et +