update

2023-11-16 07:11:29 +01:00
parent 64bbb5b739
commit 982e47382d
50 changed files with 2961 additions and 2404 deletions
@@ -372,7 +372,12 @@ const thebe_selector_output = ".output, .cell_output"
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="project2.html">
-   Project 2 on Machine Learning, deadline November 13 (Midnight)
+   Project 2 on Machine Learning, deadline November 17 (Midnight)
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="project3.html">
+   Project 3 on Machine Learning, deadline December 18 (midnight), 2023
  </a>
 </li>
 </ul>
@@ -461,16 +466,6 @@ const thebe_selector_output = ".output, .cell_output"
   Basics of a tree
  </a>
 </li>
- <li class="toc-h2 nav-item toc-entry">
-  <a class="reference internal nav-link" href="#a-sketch-of-a-tree-regression-problem">
-   A Sketch of a Tree, Regression problem
-  </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
-  <a class="reference internal nav-link" href="#a-sketch-of-a-tree-classification-problem">
-   A Sketch of a Tree, Classification  problem
-  </a>
- </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#a-typical-decision-tree-with-its-pertinent-jargon-classification-problem">
   A typical Decision Tree with its pertinent Jargon, Classification Problem
@@ -801,16 +796,6 @@ const thebe_selector_output = ".output, .cell_output"
   Basics of a tree
  </a>
 </li>
- <li class="toc-h2 nav-item toc-entry">
-  <a class="reference internal nav-link" href="#a-sketch-of-a-tree-regression-problem">
-   A Sketch of a Tree, Regression problem
-  </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
-  <a class="reference internal nav-link" href="#a-sketch-of-a-tree-classification-problem">
-   A Sketch of a Tree, Classification  problem
-  </a>
- </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#a-typical-decision-tree-with-its-pertinent-jargon-classification-problem">
   A typical Decision Tree with its pertinent Jargon, Classification Problem
@@ -1172,14 +1157,6 @@ learned the underlying structure of the training data and hence can,
 given some assumptions, make predictions about the target feature value
 (class) of unseen query instances.</p>
 </div>
-<div class="section" id="a-sketch-of-a-tree-regression-problem">
-<h2>A Sketch of a Tree, Regression problem<a class="headerlink" href="#a-sketch-of-a-tree-regression-problem" title="Permalink to this headline">¶</a></h2>
-<p><a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2022/NotesNov32022.pdf">See handwritten notes November 3</a></p>
-<!-- FIGURE: [DataFiles/Regsimpletree.png, width=600 frac=0.8] --></div>
-<div class="section" id="a-sketch-of-a-tree-classification-problem">
-<h2>A Sketch of a Tree, Classification  problem<a class="headerlink" href="#a-sketch-of-a-tree-classification-problem" title="Permalink to this headline">¶</a></h2>
-<p><a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2022/NotesNov32022.pdf">See handwritten notes November 3</a></p>
-<!-- FIGURE: [DataFiles/Classimpletree.png, width=600 frac=0.8] --></div>
 <div class="section" id="a-typical-decision-tree-with-its-pertinent-jargon-classification-problem">
 <h2>A typical Decision Tree with its pertinent Jargon, Classification Problem<a class="headerlink" href="#a-typical-decision-tree-with-its-pertinent-jargon-classification-problem" title="Permalink to this headline">¶</a></h2>
 <!-- dom:FIGURE: [DataFiles/cancer.png, width=600 frac=0.8] -->
@@ -1311,13 +1288,13 @@ predicting the target features of query instances is as follows:</p>
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>2nd degree coefficients:
-zero power:  4.888883015934703
-first power:  -0.10815559091341771
-second power:  0.0005603761549585715
+zero power:  3.7228270501360416
+first power:  0.125304962667421
+second power:  -0.0008187895061620525
 </pre></div>
 </div>
-<img alt="_images/week46_11_1.png" src="_images/week46_11_1.png" />
-<img alt="_images/week46_11_2.png" src="_images/week46_11_2.png" />
+<img alt="_images/week46_9_1.png" src="_images/week46_9_1.png" />
+<img alt="_images/week46_9_2.png" src="_images/week46_9_2.png" />
 </div>
 </div>
 </div>
@@ -1735,7 +1712,7 @@ s = -\sum_{k=1}^K p_{mk}\log{p_{mk}}.
 Text(0.9230769230769231, 0.4166666666666667, &#39;gini = 0.0\nsamples = 43\nvalue = [0, 0, 43]&#39;)]
 </pre></div>
 </div>
-<img alt="_images/week46_43_1.png" src="_images/week46_43_1.png" />
+<img alt="_images/week46_41_1.png" src="_images/week46_41_1.png" />
 </div>
 </div>
 </div>
@@ -1950,16 +1927,117 @@ these binary classes, they can easily be split into ones and zeros.</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
-<span class="ne">FileNotFoundError</span><span class="g g-Whitespace">                         </span>Traceback (most recent call last)
-<span class="nn">Input In [6],</span> in <span class="ni">&lt;cell line: 37&gt;</span><span class="nt">()</span>
-<span class="g g-Whitespace">     </span><span class="mi">34</span> <span class="k">def</span> <span class="nf">save_fig</span><span class="p">(</span><span class="n">fig_id</span><span class="p">):</span>
-<span class="g g-Whitespace">     </span><span class="mi">35</span>     <span class="n">plt</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="n">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">)</span> <span class="o">+</span> <span class="s2">&quot;.png&quot;</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="s1">&#39;png&#39;</span><span class="p">)</span>
-<span class="ne">---&gt; </span><span class="mi">37</span> <span class="n">infile</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">data_path</span><span class="p">(</span><span class="s2">&quot;grades.csv&quot;</span><span class="p">),</span><span class="s1">&#39;r&#39;</span><span class="p">)</span>
-<span class="g g-Whitespace">     </span><span class="mi">39</span> <span class="c1"># Read the experimental data with Pandas</span>
-<span class="g g-Whitespace">     </span><span class="mi">40</span> <span class="kn">from</span> <span class="nn">IPython.display</span> <span class="kn">import</span> <span class="n">display</span>
+<div class="output text_html"><div>
+<style scoped>
+    .dataframe tbody tr th:only-of-type {
+        vertical-align: middle;
+    }

-<span class="ne">FileNotFoundError</span>: [Errno 2] No such file or directory: &#39;DataFiles/grades.csv&#39;
+    .dataframe tbody tr th {
+        vertical-align: top;
+    }
+
+    .dataframe thead th {
+        text-align: right;
+    }
+</style>
+<table border="1" class="dataframe">
+  <thead>
+    <tr style="text-align: right;">
+      <th></th>
+      <th>Grade Trend</th>
+      <th>Hours slept</th>
+      <th>Hours Studied</th>
+      <th>Grade</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <th>0</th>
+      <td>1</td>
+      <td>0</td>
+      <td>1</td>
+      <td>1</td>
+    </tr>
+    <tr>
+      <th>1</th>
+      <td>0</td>
+      <td>1</td>
+      <td>0</td>
+      <td>0</td>
+    </tr>
+    <tr>
+      <th>2</th>
+      <td>1</td>
+      <td>0</td>
+      <td>1</td>
+      <td>1</td>
+    </tr>
+    <tr>
+      <th>3</th>
+      <td>1</td>
+      <td>1</td>
+      <td>1</td>
+      <td>1</td>
+    </tr>
+    <tr>
+      <th>4</th>
+      <td>0</td>
+      <td>0</td>
+      <td>1</td>
+      <td>0</td>
+    </tr>
+    <tr>
+      <th>5</th>
+      <td>1</td>
+      <td>0</td>
+      <td>0</td>
+      <td>0</td>
+    </tr>
+    <tr>
+      <th>6</th>
+      <td>0</td>
+      <td>1</td>
+      <td>1</td>
+      <td>0</td>
+    </tr>
+    <tr>
+      <th>7</th>
+      <td>0</td>
+      <td>0</td>
+      <td>1</td>
+      <td>0</td>
+    </tr>
+    <tr>
+      <th>8</th>
+      <td>1</td>
+      <td>0</td>
+      <td>0</td>
+      <td>0</td>
+    </tr>
+    <tr>
+      <th>9</th>
+      <td>1</td>
+      <td>1</td>
+      <td>1</td>
+      <td>1</td>
+    </tr>
+  </tbody>
+</table>
+</div></div><div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[1 0 1]
+ [0 1 0]
+ [1 0 1]
+ [1 1 1]
+ [0 0 1]
+ [1 0 0]
+ [0 1 1]
+ [0 0 1]
+ [1 0 0]
+ [1 1 1]]
+Train set accuracy with Decision Tree: 1.00
+</pre></div>
+</div>
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0
 </pre></div>
 </div>
 </div>
@@ -2073,6 +2151,65 @@ humidity and weak and strong for wind.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>  (0, 0)	1.0
+  (0, 7)	1.0
+  (0, 9)	1.0
+  (0, 13)	1.0
+  (1, 3)	1.0
+  (1, 5)	1.0
+  (1, 8)	1.0
+  (1, 12)	1.0
+  (2, 3)	1.0
+  (2, 5)	1.0
+  (2, 8)	1.0
+  (2, 11)	1.0
+  (3, 1)	1.0
+  (3, 5)	1.0
+  (3, 8)	1.0
+  (3, 12)	1.0
+  (4, 2)	1.0
+  (4, 6)	1.0
+  (4, 8)	1.0
+  (4, 12)	1.0
+  (5, 2)	1.0
+  (5, 4)	1.0
+  (5, 10)	1.0
+  (5, 12)	1.0
+  (6, 2)	1.0
+  :	:
+  (8, 12)	1.0
+  (9, 3)	1.0
+  (9, 4)	1.0
+  (9, 10)	1.0
+  (9, 12)	1.0
+  (10, 2)	1.0
+  (10, 6)	1.0
+  (10, 10)	1.0
+  (10, 12)	1.0
+  (11, 3)	1.0
+  (11, 6)	1.0
+  (11, 10)	1.0
+  (11, 11)	1.0
+  (12, 1)	1.0
+  (12, 6)	1.0
+  (12, 8)	1.0
+  (12, 11)	1.0
+  (13, 1)	1.0
+  (13, 5)	1.0
+  (13, 10)	1.0
+  (13, 12)	1.0
+  (14, 2)	1.0
+  (14, 6)	1.0
+  (14, 8)	1.0
+  (14, 11)	1.0
+Train set accuracy with Decision Tree: 0.73
+</pre></div>
+</div>
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0
+</pre></div>
+</div>
+</div>
 </div>
 </div>
 <div class="section" id="computing-the-gini-factor">
@@ -2147,6 +2284,67 @@ algorithm ID3.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>X1 &lt; 0.000 Gini=0.408
+X1 &lt; 0.000 Gini=0.408
+X1 &lt; 1.000 Gini=0.394
+X1 &lt; 2.000 Gini=0.394
+X1 &lt; 2.000 Gini=0.394
+X1 &lt; 2.000 Gini=0.394
+X1 &lt; 1.000 Gini=0.394
+X1 &lt; 0.000 Gini=0.408
+X1 &lt; 0.000 Gini=0.408
+X1 &lt; 2.000 Gini=0.394
+X1 &lt; 0.000 Gini=0.408
+X1 &lt; 1.000 Gini=0.394
+X1 &lt; 1.000 Gini=0.394
+X1 &lt; 2.000 Gini=0.394
+X2 &lt; 0.000 Gini=0.408
+X2 &lt; 0.000 Gini=0.408
+X2 &lt; 0.000 Gini=0.408
+X2 &lt; 1.000 Gini=0.407
+X2 &lt; 2.000 Gini=0.407
+X2 &lt; 2.000 Gini=0.407
+X2 &lt; 2.000 Gini=0.407
+X2 &lt; 1.000 Gini=0.407
+X2 &lt; 2.000 Gini=0.407
+X2 &lt; 1.000 Gini=0.407
+X2 &lt; 1.000 Gini=0.407
+X2 &lt; 1.000 Gini=0.407
+X2 &lt; 0.000 Gini=0.408
+X2 &lt; 1.000 Gini=0.407
+X3 &lt; 0.000 Gini=0.408
+X3 &lt; 0.000 Gini=0.408
+X3 &lt; 0.000 Gini=0.408
+X3 &lt; 0.000 Gini=0.408
+X3 &lt; 1.000 Gini=0.367
+X3 &lt; 1.000 Gini=0.367
+X3 &lt; 1.000 Gini=0.367
+X3 &lt; 0.000 Gini=0.408
+X3 &lt; 1.000 Gini=0.367
+X3 &lt; 1.000 Gini=0.367
+X3 &lt; 1.000 Gini=0.367
+X3 &lt; 0.000 Gini=0.408
+X3 &lt; 1.000 Gini=0.367
+X3 &lt; 0.000 Gini=0.408
+X4 &lt; 0.000 Gini=0.408
+X4 &lt; 1.000 Gini=0.405
+X4 &lt; 0.000 Gini=0.408
+X4 &lt; 0.000 Gini=0.408
+X4 &lt; 0.000 Gini=0.408
+X4 &lt; 1.000 Gini=0.405
+X4 &lt; 1.000 Gini=0.405
+X4 &lt; 0.000 Gini=0.408
+X4 &lt; 0.000 Gini=0.408
+X4 &lt; 0.000 Gini=0.408
+X4 &lt; 1.000 Gini=0.405
+X4 &lt; 1.000 Gini=0.405
+X4 &lt; 0.000 Gini=0.408
+X4 &lt; 1.000 Gini=0.405
+Split: [X3 &lt; 1.000]
+</pre></div>
+</div>
+</div>
 </div>
 </div>
 <div class="section" id="regression-trees">
@@ -2172,6 +2370,11 @@ algorithm ID3.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>DecisionTreeRegressor(max_depth=2, random_state=42)
+</pre></div>
+</div>
+</div>
 </div>
 </div>
 <div class="section" id="final-regressor-code">
@@ -2219,6 +2422,9 @@ algorithm ID3.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<img alt="_images/week46_72_0.png" src="_images/week46_72_0.png" />
+</div>
 </div>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
@@ -2253,6 +2459,9 @@ algorithm ID3.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<img alt="_images/week46_73_0.png" src="_images/week46_73_0.png" />
+</div>
 </div>
 </div>
 <div class="section" id="pros-and-cons-of-trees-pros">
@@ -2415,6 +2624,9 @@ numbers kicking in.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<img alt="_images/week46_83_0.png" src="_images/week46_83_0.png" />
+</div>
 </div>
 </div>
 <div class="section" id="using-the-voting-classifier">
@@ -2467,6 +2679,18 @@ numbers kicking in.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>LogisticRegression 0.864
+RandomForestClassifier 0.872
+SVC 0.888
+VotingClassifier 0.896
+LogisticRegression 0.864
+RandomForestClassifier 0.872
+SVC 0.888
+VotingClassifier 0.912
+</pre></div>
+</div>
+</div>
 </div>
 </div>
 <div class="section" id="voting-and-bagging">
@@ -2494,6 +2718,13 @@ numbers kicking in.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>VotingClassifier(estimators=[(&#39;lr&#39;, LogisticRegression(random_state=42)),
+                             (&#39;rf&#39;, RandomForestClassifier(random_state=42)),
+                             (&#39;svc&#39;, SVC(random_state=42))])
+</pre></div>
+</div>
+</div>
 </div>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
@@ -2506,6 +2737,16 @@ numbers kicking in.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>LogisticRegression 0.864
+RandomForestClassifier 0.896
+SVC 0.896
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>VotingClassifier 0.912
+</pre></div>
+</div>
+</div>
 </div>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
@@ -2520,6 +2761,14 @@ numbers kicking in.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>VotingClassifier(estimators=[(&#39;lr&#39;, LogisticRegression(random_state=42)),
+                             (&#39;rf&#39;, RandomForestClassifier(random_state=42)),
+                             (&#39;svc&#39;, SVC(probability=True, random_state=42))],
+                 voting=&#39;soft&#39;)
+</pre></div>
+</div>
+</div>
 </div>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
@@ -2532,6 +2781,16 @@ numbers kicking in.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>LogisticRegression 0.864
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>RandomForestClassifier 0.896
+SVC 0.896
+VotingClassifier 0.92
+</pre></div>
+</div>
+</div>
 </div>
 </div>
 <div class="section" id="bagging">
@@ -2636,6 +2895,49 @@ a decision tree wth different depths and perform a bootstrap aggregate (in this
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 1
+Error: 0.06380941468319971
+Bias^2: 0.05160313473529168
+Var: 0.01220627994790804
+0.06380941468319971 &gt;= 0.05160313473529168 + 0.01220627994790804 = 0.06380941468319971
+Polynomial degree: 2
+Error: 0.043464037468677004
+Bias^2: 0.02659851591375224
+Var: 0.01686552155492476
+0.043464037468677004 &gt;= 0.02659851591375224 + 0.01686552155492476 = 0.043464037468677
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 3
+Error: 0.020716391693769383
+Bias^2: 0.01159033914386312
+Var: 0.00912605254990626
+0.020716391693769383 &gt;= 0.01159033914386312 + 0.00912605254990626 = 0.02071639169376938
+Polynomial degree: 4
+Error: 0.02063627410934057
+Bias^2: 0.0117496656370668
+Var: 0.008886608472273775
+0.02063627410934057 &gt;= 0.0117496656370668 + 0.008886608472273775 = 0.020636274109340574
+Polynomial degree: 5
+Error: 0.02087627881701288
+Bias^2: 0.01349183949256158
+Var: 0.007384439324451296
+0.02087627881701288 &gt;= 0.01349183949256158 + 0.007384439324451296 = 0.020876278817012876
+Polynomial degree: 6
+Error: 0.02069601123831537
+Bias^2: 0.013918526350129823
+Var: 0.0067774848881855445
+0.02069601123831537 &gt;= 0.013918526350129823 + 0.0067774848881855445 = 0.020696011238315368
+Polynomial degree: 7
+Error: 0.022964339924731444
+Bias^2: 0.01550381208433455
+Var: 0.007460527840396904
+0.022964339924731444 &gt;= 0.01550381208433455 + 0.007460527840396904 = 0.022964339924731455
+Simple tree: 0.5148389267750961
+</pre></div>
+</div>
+<img alt="_images/week46_94_2.png" src="_images/week46_94_2.png" />
+</div>
 </div>
 </div>
 <div class="section" id="random-forests">
@@ -2756,6 +3058,23 @@ this setting.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>(426, 30)
+(143, 30)
+Test set accuracy Logistic Regression with scaled data: 0.96
+Test set accuracy SVM with scaled data: 0.96
+Test set accuracy with Decision Trees and scaled data: 0.87
+</pre></div>
+</div>
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.93333333 0.73333333 0.93333333 1.         1.         0.92857143
+ 1.         0.92857143 0.92857143 0.92857143]
+Test set accuracy with Random Forests and scaled data: 0.98
+</pre></div>
+</div>
+<img alt="_images/week46_100_2.png" src="_images/week46_100_2.png" />
+<img alt="_images/week46_100_3.png" src="_images/week46_100_3.png" />
+<img alt="_images/week46_100_4.png" src="_images/week46_100_4.png" />
+</div>
 </div>
 <p>Recall that the cumulative gains curve shows the percentage of the
 overall number of cases in a given category <em>gained</em> by targeting a
@@ -2787,6 +3106,11 @@ discrimination threshold is varied. It plots the true positive rate against the
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.9790209790209791
+</pre></div>
+</div>
+</div>
 </div>
 </div>
 <div class="section" id="boosting-a-bird-s-eye-view">
@@ -3049,6 +3373,11 @@ observations that are missed in the previous iterations.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<img alt="_images/week46_161_0.png" src="_images/week46_161_0.png" />
+<img alt="_images/week46_161_1.png" src="_images/week46_161_1.png" />
+<img alt="_images/week46_161_2.png" src="_images/week46_161_2.png" />
+</div>
 </div>
 </div>
 </div>