dim red with new examples

2019-10-14 07:33:51 +02:00
parent 3d0d0cab05
commit e2f09224e6
19 changed files with 1118 additions and 406 deletions
@@ -71,15 +71,23 @@ div { text-align: justify; text-justify: inter-word; }
               None,
               '___sec0'),
              ('Preprocessing our data', 2, None, '___sec1'),
-              ('Simple preprocessing examples', 2, None, '___sec2'),
-              ('Principal Component Analysis', 2, None, '___sec3'),
-              ('PCA and scikit-learn', 2, None, '___sec4'),
-              ('More on the PCA', 2, None, '___sec5'),
-              ('Incremental PCA', 2, None, '___sec6'),
-              ('Randomized PCA', 2, None, '___sec7'),
-              ('Kernel PCA', 2, None, '___sec8'),
-              ('LLE', 2, None, '___sec9'),
-              ('Other techniques', 2, None, '___sec10')]}
+              ('Simple preprocessing examples, Franke function and regression',
+               2,
+               None,
+               '___sec2'),
+              ('Simple preprocessing examples, breast cancer data and '
+               'classification',
+               2,
+               None,
+               '___sec3'),
+              ('Principal Component Analysis', 2, None, '___sec4'),
+              ('PCA and scikit-learn', 2, None, '___sec5'),
+              ('More on the PCA', 2, None, '___sec6'),
+              ('Incremental PCA', 2, None, '___sec7'),
+              ('Randomized PCA', 2, None, '___sec8'),
+              ('Kernel PCA', 2, None, '___sec9'),
+              ('LLE', 2, None, '___sec10'),
+              ('Other techniques', 2, None, '___sec11')]}
 end of tocinfo -->

 <body>
@@ -121,7 +129,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Oct 12, 2019</h4></center> <!-- date -->
+<center><h4>Oct 14, 2019</h4></center> <!-- date -->
 <br>
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -173,11 +181,113 @@ This scaling has the drawback that it does not ensure that we have a particular
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec2">Simple preprocessing examples </h2>
+<h2 id="___sec2">Simple preprocessing examples, Franke function and regression </h2>

 <p>
-We show here how we can use a simple regression case (our nuclear binding energies discussed earlier).
-Rescaling our data with different
+
+<!-- code=python (!bc pycod) typeset with pygments style "default" -->
+<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Common imports</span>
+<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">os</span>
+<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
+<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
+<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
+<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">skl</span>
+<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> mean_squared_error
+<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span>  train_test_split
+<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> MinMaxScaler, StandardScaler, Normalizer
+<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.svm</span> <span style="color: #008000; font-weight: bold">import</span> SVR
+
+<span style="color: #408080; font-style: italic"># Where to save the figures and data files</span>
+PROJECT_ROOT_DIR <span style="color: #666666">=</span> <span style="color: #BA2121">&quot;Results&quot;</span>
+FIGURE_ID <span style="color: #666666">=</span> <span style="color: #BA2121">&quot;Results/FigureFiles&quot;</span>
+DATA_ID <span style="color: #666666">=</span> <span style="color: #BA2121">&quot;DataFiles/&quot;</span>
+
+<span style="color: #008000; font-weight: bold">if</span> <span style="color: #AA22FF; font-weight: bold">not</span> os<span style="color: #666666">.</span>path<span style="color: #666666">.</span>exists(PROJECT_ROOT_DIR):
+    os<span style="color: #666666">.</span>mkdir(PROJECT_ROOT_DIR)
+
+<span style="color: #008000; font-weight: bold">if</span> <span style="color: #AA22FF; font-weight: bold">not</span> os<span style="color: #666666">.</span>path<span style="color: #666666">.</span>exists(FIGURE_ID):
+    os<span style="color: #666666">.</span>makedirs(FIGURE_ID)
+
+<span style="color: #008000; font-weight: bold">if</span> <span style="color: #AA22FF; font-weight: bold">not</span> os<span style="color: #666666">.</span>path<span style="color: #666666">.</span>exists(DATA_ID):
+    os<span style="color: #666666">.</span>makedirs(DATA_ID)
+
+<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">image_path</span>(fig_id):
+    <span style="color: #008000; font-weight: bold">return</span> os<span style="color: #666666">.</span>path<span style="color: #666666">.</span>join(FIGURE_ID, fig_id)
+
+<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">data_path</span>(dat_id):
+    <span style="color: #008000; font-weight: bold">return</span> os<span style="color: #666666">.</span>path<span style="color: #666666">.</span>join(DATA_ID, dat_id)
+
+<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">save_fig</span>(fig_id):
+    plt<span style="color: #666666">.</span>savefig(image_path(fig_id) <span style="color: #666666">+</span> <span style="color: #BA2121">&quot;.png&quot;</span>, format<span style="color: #666666">=</span><span style="color: #BA2121">&#39;png&#39;</span>)
+
+
+<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">FrankeFunction</span>(x,y):
+	term1 <span style="color: #666666">=</span> <span style="color: #666666">0.75*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(<span style="color: #666666">0.25*</span>(<span style="color: #666666">9*</span>x<span style="color: #666666">-2</span>)<span style="color: #666666">**2</span>) <span style="color: #666666">-</span> <span style="color: #666666">0.25*</span>((<span style="color: #666666">9*</span>y<span style="color: #666666">-2</span>)<span style="color: #666666">**2</span>))
+	term2 <span style="color: #666666">=</span> <span style="color: #666666">0.75*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>((<span style="color: #666666">9*</span>x<span style="color: #666666">+1</span>)<span style="color: #666666">**2</span>)<span style="color: #666666">/49.0</span> <span style="color: #666666">-</span> <span style="color: #666666">0.1*</span>(<span style="color: #666666">9*</span>y<span style="color: #666666">+1</span>))
+	term3 <span style="color: #666666">=</span> <span style="color: #666666">0.5*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(<span style="color: #666666">9*</span>x<span style="color: #666666">-7</span>)<span style="color: #666666">**2/4.0</span> <span style="color: #666666">-</span> <span style="color: #666666">0.25*</span>((<span style="color: #666666">9*</span>y<span style="color: #666666">-3</span>)<span style="color: #666666">**2</span>))
+	term4 <span style="color: #666666">=</span> <span style="color: #666666">-0.2*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(<span style="color: #666666">9*</span>x<span style="color: #666666">-4</span>)<span style="color: #666666">**2</span> <span style="color: #666666">-</span> (<span style="color: #666666">9*</span>y<span style="color: #666666">-7</span>)<span style="color: #666666">**2</span>)
+	<span style="color: #008000; font-weight: bold">return</span> term1 <span style="color: #666666">+</span> term2 <span style="color: #666666">+</span> term3 <span style="color: #666666">+</span> term4
+
+
+<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">create_X</span>(x, y, n ):
+	<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">len</span>(x<span style="color: #666666">.</span>shape) <span style="color: #666666">&gt;</span> <span style="color: #666666">1</span>:
+		x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ravel(x)
+		y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ravel(y)
+
+	N <span style="color: #666666">=</span> <span style="color: #008000">len</span>(x)
+	l <span style="color: #666666">=</span> <span style="color: #008000">int</span>((n<span style="color: #666666">+1</span>)<span style="color: #666666">*</span>(n<span style="color: #666666">+2</span>)<span style="color: #666666">/2</span>)		<span style="color: #408080; font-style: italic"># Number of elements in beta</span>
+	X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ones((N,l))
+
+	<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">1</span>,n<span style="color: #666666">+1</span>):
+		q <span style="color: #666666">=</span> <span style="color: #008000">int</span>((i)<span style="color: #666666">*</span>(i<span style="color: #666666">+1</span>)<span style="color: #666666">/2</span>)
+		<span style="color: #008000; font-weight: bold">for</span> k <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(i<span style="color: #666666">+1</span>):
+			X[:,q<span style="color: #666666">+</span>k] <span style="color: #666666">=</span> (x<span style="color: #666666">**</span>(i<span style="color: #666666">-</span>k))<span style="color: #666666">*</span>(y<span style="color: #666666">**</span>k)
+
+	<span style="color: #008000; font-weight: bold">return</span> X
+
+
+<span style="color: #408080; font-style: italic"># Making meshgrid of datapoints and compute Franke&#39;s function</span>
+n <span style="color: #666666">=</span> <span style="color: #666666">5</span>
+N <span style="color: #666666">=</span> <span style="color: #666666">1000</span>
+x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sort(np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>uniform(<span style="color: #666666">0</span>, <span style="color: #666666">1</span>, N))
+y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sort(np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>uniform(<span style="color: #666666">0</span>, <span style="color: #666666">1</span>, N))
+z <span style="color: #666666">=</span> FrankeFunction(x, y)
+X <span style="color: #666666">=</span> create_X(x, y, n<span style="color: #666666">=</span>n)    
+<span style="color: #408080; font-style: italic"># split in training and test data</span>
+X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(X,z,test_size<span style="color: #666666">=0.2</span>)
+
+
+svm <span style="color: #666666">=</span> SVR(gamma<span style="color: #666666">=</span><span style="color: #BA2121">&#39;auto&#39;</span>,C<span style="color: #666666">=10.0</span>)
+svm<span style="color: #666666">.</span>fit(X_train, y_train)
+
+<span style="color: #408080; font-style: italic"># The mean squared error and R2 score</span>
+<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;MSE before scaling: {:.2f}&quot;</span><span style="color: #666666">.</span>format(mean_squared_error(svm<span style="color: #666666">.</span>predict(X_test), y_test)))
+<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;R2 score before scaling {:.2f}&quot;</span><span style="color: #666666">.</span>format(svm<span style="color: #666666">.</span>score(X_test,y_test)))
+
+scaler <span style="color: #666666">=</span> StandardScaler()
+scaler<span style="color: #666666">.</span>fit(X_train)
+X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
+X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
+
+<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Feature min values before scaling:</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121"> {}&quot;</span><span style="color: #666666">.</span>format(X_train<span style="color: #666666">.</span>min(axis<span style="color: #666666">=0</span>)))
+<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Feature max values before scaling:</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121"> {}&quot;</span><span style="color: #666666">.</span>format(X_train<span style="color: #666666">.</span>max(axis<span style="color: #666666">=0</span>)))
+
+<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Feature min values after scaling:</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121"> {}&quot;</span><span style="color: #666666">.</span>format(X_train_scaled<span style="color: #666666">.</span>min(axis<span style="color: #666666">=0</span>)))
+<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Feature max values after scaling:</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121"> {}&quot;</span><span style="color: #666666">.</span>format(X_train_scaled<span style="color: #666666">.</span>max(axis<span style="color: #666666">=0</span>)))
+
+svm <span style="color: #666666">=</span> SVR(gamma<span style="color: #666666">=</span><span style="color: #BA2121">&#39;auto&#39;</span>,C<span style="color: #666666">=10.0</span>)
+svm<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
+
+<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;MSE after  scaling: {:.2f}&quot;</span><span style="color: #666666">.</span>format(mean_squared_error(svm<span style="color: #666666">.</span>predict(X_test_scaled), y_test)))
+<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;R2 score for  scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(svm<span style="color: #666666">.</span>score(X_test_scaled,y_test)))
+</pre></div>
+<p>
+<!-- !split --><br><br><br><br><br><br><br><br><br><br>
+
+<h2 id="___sec3">Simple preprocessing examples, breast cancer data and classification </h2>
+
+<p>
+We show here how we can use a simple regression case on the breast cancer data using support vector machine as algorithm for classification

 <p>

@@ -225,7 +335,7 @@ svm<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec3">Principal Component Analysis </h2>
+<h2 id="___sec4">Principal Component Analysis </h2>
 <div class="alert alert-block alert-block alert-text-normal">
 <b></b>
 <p>
@@ -261,7 +371,7 @@ X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666"
 <p>
 <!-- !split  -->

-<h2 id="___sec4">PCA and scikit-learn </h2>
+<h2 id="___sec5">PCA and scikit-learn </h2>

 <p>
 Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -292,7 +402,7 @@ More material to come here.
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec5">More on the PCA </h2>
+<h2 id="___sec6">More on the PCA </h2>
 Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
 choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
 Unless, of course, you are reducing dimensionality for data visualization &#8212; in that case you will
@@ -320,7 +430,7 @@ X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec6">Incremental PCA </h2>
+<h2 id="___sec7">Incremental PCA </h2>
 One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
 memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
 been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
@@ -330,7 +440,7 @@ instances arrive).
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec7">Randomized PCA </h2>
+<h2 id="___sec8">Randomized PCA </h2>

 <p>
 Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -345,7 +455,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec8">Kernel PCA </h2>
+<h2 id="___sec9">Kernel PCA </h2>
 <div class="alert alert-block alert-block alert-text-normal">
 <b></b>
 <p>
@@ -374,7 +484,7 @@ X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #6666
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec9">LLE </h2>
+<h2 id="___sec10">LLE </h2>

 <p>
 Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -386,7 +496,7 @@ these local relationships are best preserved (more details shortly).
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

-<h2 id="___sec10">Other techniques </h2>
+<h2 id="___sec11">Other techniques </h2>

 <p>
 There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.