lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

typos in dim red

Browse Source
This commit is contained in:
mhjensen
2019-10-25 05:09:28 +02:00
parent 63659da7d8
commit 20d45a6142
15 changed files with 174 additions and 171 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/DecisionTrees/ipynb/DecisionTrees.ipynb
+72 -65
View File
File diff suppressed because one or more lines are too long
doc/pub/DimRed/html/._DimRed-bs000.html
+1 -1
View File
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 24, 2019</h4></center> <!-- date -->
<center><h4>Oct 25, 2019</h4></center> <!-- date -->
<br>
<p>
doc/pub/DimRed/html/._DimRed-bs010.html
+1 -1
View File
@@ -203,7 +203,7 @@ with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/featu
entries \( n \) being the row elements.
We can rewrite the design/feature matrix in terms of its column vectors as
$$
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_0 & \boldsymbol{x}_0 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix},
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix},
$$
with a given vector
doc/pub/DimRed/html/._DimRed-bs015.html
+1 -1
View File
@@ -208,7 +208,7 @@ If we then compute the expectation value
$$
\mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T=\begin{bmatrix}
x_{00}^2+x_{01}^2 & x_{00}x_{10}+x_{01}x_{11}\\
x_{10}x_{00}+x_{01}x_{11} & x_{10}^2+x_{11}^2\\
x_{10}x_{00}+x_{11}x_{01} & x_{10}^2+x_{11}^2\\
\end{bmatrix},
$$
doc/pub/DimRed/html/._DimRed-bs018.html
+2 -2
View File
@@ -193,10 +193,10 @@ each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
<p>
We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as
$$
J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2,
J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}}_i)^2,
$$
with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
<p>
doc/pub/DimRed/html/._DimRed-bs024.html
+12 -12
View File
@@ -185,7 +185,8 @@ MathJax.Hub.Config({
<!-- !split -->
<h2 id="___sec23" class="anchor">Back to the Cancer Data </h2>
We can now repeat the above but applied to real data, in this case our breat cancer data.
We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -197,31 +198,30 @@ We can now repeat the above but applied to real data, in this case our breat can
cancer <span style="color: #666666">=</span> load_breast_cancer()
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(cancer<span style="color: #666666">.</span>data,cancer<span style="color: #666666">.</span>target,random_state<span style="color: #666666">=0</span>)
<span style="color: #008000; font-weight: bold">print</span>(X_train<span style="color: #666666">.</span>shape)
<span style="color: #008000; font-weight: bold">print</span>(X_test<span style="color: #666666">.</span>shape)
logreg <span style="color: #666666">=</span> LogisticRegression()
logreg<span style="color: #666666">.</span>fit(X_train, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy from Logistic Regression: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test,y_test)))
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> MinMaxScaler, StandardScaler
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Train set accuracy from Logistic Regression: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_train,y_train)))
<span style="color: #408080; font-style: italic"># We scale the data</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
scaler <span style="color: #666666">=</span> StandardScaler()
scaler<span style="color: #666666">.</span>fit(X_train)
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
<span style="color: #408080; font-style: italic"># Then perform again a log reg fit</span>
logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test_scaled,y_test)))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Train set accuracy scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_train_scaled,y_train)))
<span style="color: #408080; font-style: italic">#thereafter we do a PCA with Scikit-learn</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.decomposition</span> <span style="color: #008000; font-weight: bold">import</span> PCA
pca <span style="color: #666666">=</span> PCA(n_components <span style="color: #666666">=</span> <span style="color: #666666">2</span>)
X2D_train <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>fit_transform(X_train_scaled)
X2D_test <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>fit_transform(X_test_scaled)
<span style="color: #408080; font-style: italic"># and finally compute the log reg fit and the score on the training data </span>
logreg<span style="color: #666666">.</span>fit(X2D_train,y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X2D_test,y_test)))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Train set accuracy scaled and PCA data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X2D_train,y_train)))
</pre></div>
<p>
We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
doc/pub/DimRed/html/DimRed-bs.html
+1 -1
View File
@@ -203,7 +203,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 24, 2019</h4></center> <!-- date -->
<center><h4>Oct 25, 2019</h4></center> <!-- date -->
<br>
<p>
doc/pub/DimRed/html/DimRed-reveal.html
+16 -17
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Oct 24, 2019</h4></center> <!-- date -->
<center><h4>Oct 25, 2019</h4></center> <!-- date -->
<br>
<p>
@@ -617,7 +617,7 @@ entries \( n \) being the row elements.
We can rewrite the design/feature matrix in terms of its column vectors as
<p>&nbsp;<br>
$$
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_0 & \boldsymbol{x}_0 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix},
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix},
$$
<p>&nbsp;<br>
@@ -870,7 +870,7 @@ If we then compute the expectation value
$$
\mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T=\begin{bmatrix}
x_{00}^2+x_{01}^2 & x_{00}x_{10}+x_{01}x_{11}\\
x_{10}x_{00}+x_{01}x_{11} & x_{10}^2+x_{11}^2\\
x_{10}x_{00}+x_{11}x_{01} & x_{10}^2+x_{11}^2\\
\end{bmatrix},
$$
<p>&nbsp;<br>
@@ -999,11 +999,11 @@ each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as
<p>&nbsp;<br>
$$
J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2,
J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}}_i)^2,
$$
<p>&nbsp;<br>
with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
<p>
@@ -1238,7 +1238,8 @@ variance that lies along the axis of each principal component.
<section>
<h2 id="___sec23">Back to the Cancer Data </h2>
We can now repeat the above but applied to real data, in this case our breat cancer data.
We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -1250,31 +1251,29 @@ We can now repeat the above but applied to real data, in this case our breat can
cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=<span style="color: #B452CD">0</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(X_train.shape)
<span style="color: #8B008B; font-weight: bold">print</span>(X_test.shape)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Test set accuracy from Logistic Regression: {:.2f}&quot;</span>.format(logreg.score(X_test,y_test)))
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> MinMaxScaler, StandardScaler
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Train set accuracy from Logistic Regression: {:.2f}&quot;</span>.format(logreg.score(X_train,y_train)))
<span style="color: #228B22"># We scale the data</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
<span style="color: #228B22"># Then perform again a log reg fit</span>
logreg.fit(X_train_scaled, y_train)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Test set accuracy scaled data: {:.2f}&quot;</span>.format(logreg.score(X_test_scaled,y_test)))
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Train set accuracy scaled data: {:.2f}&quot;</span>.format(logreg.score(X_train_scaled,y_train)))
<span style="color: #228B22">#thereafter we do a PCA with Scikit-learn</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.decomposition</span> <span style="color: #8B008B; font-weight: bold">import</span> PCA
pca = PCA(n_components = <span style="color: #B452CD">2</span>)
X2D_train = pca.fit_transform(X_train_scaled)
X2D_test = pca.fit_transform(X_test_scaled)
<span style="color: #228B22"># and finally compute the log reg fit and the score on the training data </span>
logreg.fit(X2D_train,y_train)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Test set accuracy scaled data: {:.2f}&quot;</span>.format(logreg.score(X2D_test,y_test)))
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Train set accuracy scaled and PCA data: {:.2f}&quot;</span>.format(logreg.score(X2D_train,y_train)))
</pre></div>
<p>
We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.
</section>
doc/pub/DimRed/html/DimRed-solarized.html
+17 -17
View File
@@ -163,7 +163,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 24, 2019</h4></center> <!-- date -->
<center><h4>Oct 25, 2019</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -613,7 +613,7 @@ with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/featu
entries \( n \) being the row elements.
We can rewrite the design/feature matrix in terms of its column vectors as
$$
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_0 & \boldsymbol{x}_0 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix},
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix},
$$
with a given vector
@@ -851,7 +851,7 @@ If we then compute the expectation value
$$
\mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T=\begin{bmatrix}
x_{00}^2+x_{01}^2 & x_{00}x_{10}+x_{01}x_{11}\\
x_{10}x_{00}+x_{01}x_{11} & x_{10}^2+x_{11}^2\\
x_{10}x_{00}+x_{11}x_{01} & x_{10}^2+x_{11}^2\\
\end{bmatrix},
$$
@@ -964,10 +964,10 @@ each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
<p>
We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as
$$
J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2,
J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}}_i)^2,
$$
with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
<p>
@@ -1177,7 +1177,8 @@ variance that lies along the axis of each principal component.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">Back to the Cancer Data </h2>
We can now repeat the above but applied to real data, in this case our breat cancer data.
We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -1189,31 +1190,30 @@ We can now repeat the above but applied to real data, in this case our breat can
cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=<span style="color: #B452CD">0</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(X_train.shape)
<span style="color: #8B008B; font-weight: bold">print</span>(X_test.shape)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Test set accuracy from Logistic Regression: {:.2f}&quot;</span>.format(logreg.score(X_test,y_test)))
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> MinMaxScaler, StandardScaler
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Train set accuracy from Logistic Regression: {:.2f}&quot;</span>.format(logreg.score(X_train,y_train)))
<span style="color: #228B22"># We scale the data</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
<span style="color: #228B22"># Then perform again a log reg fit</span>
logreg.fit(X_train_scaled, y_train)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Test set accuracy scaled data: {:.2f}&quot;</span>.format(logreg.score(X_test_scaled,y_test)))
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Train set accuracy scaled data: {:.2f}&quot;</span>.format(logreg.score(X_train_scaled,y_train)))
<span style="color: #228B22">#thereafter we do a PCA with Scikit-learn</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.decomposition</span> <span style="color: #8B008B; font-weight: bold">import</span> PCA
pca = PCA(n_components = <span style="color: #B452CD">2</span>)
X2D_train = pca.fit_transform(X_train_scaled)
X2D_test = pca.fit_transform(X_test_scaled)
<span style="color: #228B22"># and finally compute the log reg fit and the score on the training data </span>
logreg.fit(X2D_train,y_train)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Test set accuracy scaled data: {:.2f}&quot;</span>.format(logreg.score(X2D_test,y_test)))
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Train set accuracy scaled and PCA data: {:.2f}&quot;</span>.format(logreg.score(X2D_train,y_train)))
</pre></div>
<p>
We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/DimRed/html/DimRed.html
+17 -17
View File
@@ -168,7 +168,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 24, 2019</h4></center> <!-- date -->
<center><h4>Oct 25, 2019</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -618,7 +618,7 @@ with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/featu
entries \( n \) being the row elements.
We can rewrite the design/feature matrix in terms of its column vectors as
$$
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_0 & \boldsymbol{x}_0 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix},
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix},
$$
with a given vector
@@ -856,7 +856,7 @@ If we then compute the expectation value
$$
\mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T=\begin{bmatrix}
x_{00}^2+x_{01}^2 & x_{00}x_{10}+x_{01}x_{11}\\
x_{10}x_{00}+x_{01}x_{11} & x_{10}^2+x_{11}^2\\
x_{10}x_{00}+x_{11}x_{01} & x_{10}^2+x_{11}^2\\
\end{bmatrix},
$$
@@ -969,10 +969,10 @@ each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
<p>
We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as
$$
J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2,
J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}}_i)^2,
$$
with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
<p>
@@ -1182,7 +1182,8 @@ variance that lies along the axis of each principal component.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">Back to the Cancer Data </h2>
We can now repeat the above but applied to real data, in this case our breat cancer data.
We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -1194,31 +1195,30 @@ We can now repeat the above but applied to real data, in this case our breat can
cancer <span style="color: #666666">=</span> load_breast_cancer()
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(cancer<span style="color: #666666">.</span>data,cancer<span style="color: #666666">.</span>target,random_state<span style="color: #666666">=0</span>)
<span style="color: #008000; font-weight: bold">print</span>(X_train<span style="color: #666666">.</span>shape)
<span style="color: #008000; font-weight: bold">print</span>(X_test<span style="color: #666666">.</span>shape)
logreg <span style="color: #666666">=</span> LogisticRegression()
logreg<span style="color: #666666">.</span>fit(X_train, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy from Logistic Regression: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test,y_test)))
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> MinMaxScaler, StandardScaler
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Train set accuracy from Logistic Regression: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_train,y_train)))
<span style="color: #408080; font-style: italic"># We scale the data</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
scaler <span style="color: #666666">=</span> StandardScaler()
scaler<span style="color: #666666">.</span>fit(X_train)
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
<span style="color: #408080; font-style: italic"># Then perform again a log reg fit</span>
logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test_scaled,y_test)))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Train set accuracy scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_train_scaled,y_train)))
<span style="color: #408080; font-style: italic">#thereafter we do a PCA with Scikit-learn</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.decomposition</span> <span style="color: #008000; font-weight: bold">import</span> PCA
pca <span style="color: #666666">=</span> PCA(n_components <span style="color: #666666">=</span> <span style="color: #666666">2</span>)
X2D_train <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>fit_transform(X_train_scaled)
X2D_test <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>fit_transform(X_test_scaled)
<span style="color: #408080; font-style: italic"># and finally compute the log reg fit and the score on the training data </span>
logreg<span style="color: #666666">.</span>fit(X2D_train,y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X2D_test,y_test)))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Train set accuracy scaled and PCA data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X2D_train,y_train)))
</pre></div>
<p>
We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/DimRed/ipynb/DimRed.ipynb
+16 -17
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Oct 24, 2019**\n",
"Date: **Oct 25, 2019**\n",
"\n",
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -561,7 +561,7 @@
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix} \\boldsymbol{x}_0 & \\boldsymbol{x}_0 & \\boldsymbol{x}_0 & \\dots & \\dots & \\boldsymbol{x}_{p-1}\\end{bmatrix},\n",
"\\boldsymbol{X}=\\begin{bmatrix} \\boldsymbol{x}_0 & \\boldsymbol{x}_1 & \\boldsymbol{x}_2 & \\dots & \\dots & \\boldsymbol{x}_{p-1}\\end{bmatrix},\n",
"$$"
]
},
@@ -894,7 +894,7 @@
"$$\n",
"\\mathbb{E}[\\boldsymbol{X}\\boldsymbol{X}^T] = \\frac{1}{n}\\boldsymbol{X}\\boldsymbol{X}^T=\\begin{bmatrix}\n",
"x_{00}^2+x_{01}^2 & x_{00}x_{10}+x_{01}x_{11}\\\\\n",
"x_{10}x_{00}+x_{01}x_{11} & x_{10}^2+x_{11}^2\\\\\n",
"x_{10}x_{00}+x_{11}x_{01} & x_{10}^2+x_{11}^2\\\\\n",
"\\end{bmatrix},\n",
"$$"
]
@@ -1064,7 +1064,7 @@
"metadata": {},
"source": [
"$$\n",
"J(\\boldsymbol{W},\\boldsymbol{Z}) = \\frac{1}{n}\\sum_i (\\boldsymbol{x}_i - \\overline{\\boldsymbol{x}_i})^2,\n",
"J(\\boldsymbol{W},\\boldsymbol{Z}) = \\frac{1}{n}\\sum_i (\\boldsymbol{x}_i - \\overline{\\boldsymbol{x}}_i)^2,\n",
"$$"
]
},
@@ -1072,7 +1072,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"with $\\overline{\\boldsymbol{x}_i} = \\boldsymbol{W}\\boldsymbol{z}_i$, where $\\boldsymbol{z}_i$ is a row vector with dimension ${\\mathbb{R}}^{n}$ of the matrix\n",
"with $\\overline{\\boldsymbol{x}}_i = \\boldsymbol{W}\\boldsymbol{z}_i$, where $\\boldsymbol{z}_i$ is a row vector with dimension ${\\mathbb{R}}^{n}$ of the matrix\n",
"$\\boldsymbol{Z}\\in{\\mathbb{R}}^{p\\times n}$. When doing PCA we want to reduce this dimensionality. \n",
"\n",
"The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\\boldsymbol{W}=\\boldsymbol{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $\\boldsymbol{z}_i$ with at most $l$ vectors, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).\n",
@@ -1433,7 +1433,8 @@
"variance that lies along the axis of each principal component. \n",
"\n",
"## Back to the Cancer Data\n",
"We can now repeat the above but applied to real data, in this case our breat cancer data."
"We can now repeat the above but applied to real data, in this case our breast cancer data.\n",
"Here we compute performance scores on the training data using logistic regression."
]
},
{
@@ -1452,36 +1453,34 @@
"cancer = load_breast_cancer()\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)\n",
"print(X_train.shape)\n",
"print(X_test.shape)\n",
"\n",
"logreg = LogisticRegression()\n",
"logreg.fit(X_train, y_train)\n",
"print(\"Test set accuracy from Logistic Regression: {:.2f}\".format(logreg.score(X_test,y_test)))\n",
"\n",
"from sklearn.preprocessing import MinMaxScaler, StandardScaler\n",
"print(\"Train set accuracy from Logistic Regression: {:.2f}\".format(logreg.score(X_train,y_train)))\n",
"# We scale the data\n",
"from sklearn.preprocessing import StandardScaler\n",
"scaler = StandardScaler()\n",
"scaler.fit(X_train)\n",
"X_train_scaled = scaler.transform(X_train)\n",
"X_test_scaled = scaler.transform(X_test)\n",
"\n",
"# Then perform again a log reg fit\n",
"logreg.fit(X_train_scaled, y_train)\n",
"print(\"Test set accuracy scaled data: {:.2f}\".format(logreg.score(X_test_scaled,y_test)))\n",
"\n",
"print(\"Train set accuracy scaled data: {:.2f}\".format(logreg.score(X_train_scaled,y_train)))\n",
"#thereafter we do a PCA with Scikit-learn\n",
"from sklearn.decomposition import PCA\n",
"pca = PCA(n_components = 2)\n",
"X2D_train = pca.fit_transform(X_train_scaled)\n",
"X2D_test = pca.fit_transform(X_test_scaled)\n",
"\n",
"# and finally compute the log reg fit and the score on the training data\t\n",
"logreg.fit(X2D_train,y_train)\n",
"print(\"Test set accuracy scaled data: {:.2f}\".format(logreg.score(X2D_test,y_test)))"
"print(\"Train set accuracy scaled and PCA data: {:.2f}\".format(logreg.score(X2D_train,y_train)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see that our training data after the PCA decomposition has a performance similar to the non-scaled data. \n",
"\n",
"## More on the PCA\n",
"\n",
"Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to\n",
doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/DimRed/pdf/DimRed-minted.pdf
BIN
View File
Binary file not shown.
doc/src/DimRed/DimRed.do.txt
+15 -16
View File
@@ -421,7 +421,7 @@ entries $n$ being the row elements.
We can rewrite the design/feature matrix in terms of its column vectors as
!bt
\[
\bm{X}=\begin{bmatrix} \bm{x}_0 & \bm{x}_0 & \bm{x}_0 & \dots & \dots & \bm{x}_{p-1}\end{bmatrix},
\bm{X}=\begin{bmatrix} \bm{x}_0 & \bm{x}_1 & \bm{x}_2 & \dots & \dots & \bm{x}_{p-1}\end{bmatrix},
\]
!et
with a given vector
@@ -649,7 +649,7 @@ If we then compute the expectation value
\[
\mathbb{E}[\bm{X}\bm{X}^T] = \frac{1}{n}\bm{X}\bm{X}^T=\begin{bmatrix}
x_{00}^2+x_{01}^2 & x_{00}x_{10}+x_{01}x_{11}\\
x_{10}x_{00}+x_{01}x_{11} & x_{10}^2+x_{11}^2\\
x_{10}x_{00}+x_{11}x_{01} & x_{10}^2+x_{11}^2\\
\end{bmatrix},
\]
!et
@@ -750,10 +750,10 @@ each with dimension $\bm{x}\in {\mathbb{R}}^{n}$.
We assume also that we have an orthogonal transformation $\bm{W}\in {\mathbb{R}}^{p\times p}$. We define the reconstruction error (which is similar to the mean squared error we have seen before) as
!bt
\[
J(\bm{W},\bm{Z}) = \frac{1}{n}\sum_i (\bm{x}_i - \overline{\bm{x}_i})^2,
J(\bm{W},\bm{Z}) = \frac{1}{n}\sum_i (\bm{x}_i - \overline{\bm{x}}_i)^2,
\]
!et
with $\overline{\bm{x}_i} = \bm{W}\bm{z}_i$, where $\bm{z}_i$ is a row vector with dimension ${\mathbb{R}}^{n}$ of the matrix
with $\overline{\bm{x}}_i = \bm{W}\bm{z}_i$, where $\bm{z}_i$ is a row vector with dimension ${\mathbb{R}}^{n}$ of the matrix
$\bm{Z}\in{\mathbb{R}}^{p\times n}$. When doing PCA we want to reduce this dimensionality.
The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\bm{W}=\bm{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $\bm{z}_i$ with at most $l$ vectors, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
@@ -951,7 +951,8 @@ variance that lies along the axis of each principal component.
!split
===== Back to the Cancer Data =====
We can now repeat the above but applied to real data, in this case our breat cancer data.
We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.
!bc pycod
import matplotlib.pyplot as plt
import numpy as np
@@ -961,32 +962,30 @@ from sklearn.linear_model import LogisticRegression
cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
print(X_train.shape)
print(X_test.shape)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
print("Test set accuracy from Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
from sklearn.preprocessing import MinMaxScaler, StandardScaler
print("Train set accuracy from Logistic Regression: {:.2f}".format(logreg.score(X_train,y_train)))
# We scale the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Then perform again a log reg fit
logreg.fit(X_train_scaled, y_train)
print("Test set accuracy scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
print("Train set accuracy scaled data: {:.2f}".format(logreg.score(X_train_scaled,y_train)))
#thereafter we do a PCA with Scikit-learn
from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
X2D_train = pca.fit_transform(X_train_scaled)
X2D_test = pca.fit_transform(X_test_scaled)
# and finally compute the log reg fit and the score on the training data
logreg.fit(X2D_train,y_train)
print("Test set accuracy scaled data: {:.2f}".format(logreg.score(X2D_test,y_test)))
print("Train set accuracy scaled and PCA data: {:.2f}".format(logreg.score(X2D_train,y_train)))
!ec
We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.
!split
===== More on the PCA =====
doc/src/DimRed/PCAcancer.py
+3 -4
View File
@@ -23,7 +23,7 @@ print(X_test.shape)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
print("Test set accuracy from Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
print("Train set accuracy from Logistic Regression: {:.2f}".format(logreg.score(X_train,y_train)))
from sklearn.preprocessing import MinMaxScaler, StandardScaler
scaler = StandardScaler()
@@ -32,13 +32,12 @@ X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
logreg.fit(X_train_scaled, y_train)
print("Test set accuracy scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
print("Train set accuracy scaled data: {:.2f}".format(logreg.score(X_train_scaled,y_train)))
#thereafter we do a PCA with Scikit-learn
from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
X2D_train = pca.fit_transform(X_train_scaled)
X2D_test = pca.fit_transform(X_test_scaled)
logreg.fit(X2D_train,y_train)
print("Test set accuracy scaled data: {:.2f}".format(logreg.score(X2D_test,y_test)))
print("Train set accuracy scaled and PCA data: {:.2f}".format(logreg.score(X2D_train,y_train)))