update

2023-08-28 10:35:01 +02:00
parent b4954d5316
commit 707c8e08f3
15 changed files with 1891 additions and 1772 deletions
@@ -1137,8 +1137,8 @@ doconce format html week35.do.txt --no_mako -->
 <h3>Reading recommendations:<a class="headerlink" href="#reading-recommendations" title="Permalink to this headline">¶</a></h3>
 <ol class="simple">
 <li><p>See lecture notes for week 35 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a></p></li>
-<li><p>Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics</p></li>
-<li><p>Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4</p></li>
+<li><p>Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)</p></li>
+<li><p>Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4 (on relevance for the discussion of linear regression).</p></li>
 </ol>
 </div>
 </div>
@@ -1278,14 +1278,13 @@ will treat <span class="math notranslate nohighlight">\(y_i\)</span> as our exac
 \]</div>
 <p>We note also that since our design matrix is defined as <span class="math notranslate nohighlight">\(\boldsymbol{X}\in
 {\mathbb{R}}^{n\times p}\)</span>, the product <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X} \in
-{\mathbb{R}}^{p\times p}\)</span>.  In the above case we have that <span class="math notranslate nohighlight">\(p \ll n\)</span>,
-in our case <span class="math notranslate nohighlight">\(p=5\)</span> meaning that we end up with inverting a small
+{\mathbb{R}}^{p\times p}\)</span>.  In most cases we have that <span class="math notranslate nohighlight">\(p \ll n\)</span>. In our example case below we have <span class="math notranslate nohighlight">\(p=5\)</span> meaning. We end up with inverting a small
 <span class="math notranslate nohighlight">\(5\times 5\)</span> matrix. This is a rather common situation, in many cases we end up with low-dimensional
 matrices to invert. The methods discussed here and for many other
 supervised learning algorithms like classification with logistic
 regression or support vector machines, exhibit dimensionalities which
 allow for the usage of direct linear algebra methods such as <strong>LU</strong> decomposition or <strong>Singular Value Decomposition</strong> (SVD) for finding the inverse of the matrix
-<span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>.</p>
+<span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>.  This is discussed on Thursday this week.</p>
 <p><strong>Small question</strong>: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix  <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>? What kind of problems can we expect?</p>
 </div>
 <div class="section" id="some-useful-matrix-and-vector-expressions">
@@ -1453,7 +1452,7 @@ C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\bold
 \]</div>
 <p>We note that the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> does not depend on the unknown parameters defined by the vector <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.
 We are now interested in minimizing the cost function with respect to the unknown parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
-<p>The mean squared error is a scalar and if we use the results from the last example, we define a new vector</p>
+<p>The mean squared error is a scalar and if we use the results from example three above, we can define a new vector</p>
 <div class="math notranslate nohighlight">
 \[
 \boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
@@ -1553,8 +1552,8 @@ We assume our data can represented by a fourth-order polynomial. For the <span c
 \[
 \tilde{y}_i = \beta_0+\beta_1x_i+\beta_2x_i^2+\beta_3x_i^3+\beta_4x_i^4.
 \]</div>
-<p>we have five predictors/features. The first is the intercept <span class="math notranslate nohighlight">\(\beta_0\)</span>. The other terms are <span class="math notranslate nohighlight">\(\beta_i\)</span> with <span class="math notranslate nohighlight">\(i=1,2,3,4\)</span>. Furthermore we have <span class="math notranslate nohighlight">\(n\)</span> entries for each predictor. It means that our design matrix is a
-<span class="math notranslate nohighlight">\(p\times n\)</span> matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
+<p>we have five predictors/features. The first is the intercept <span class="math notranslate nohighlight">\(\beta_0\)</span>. The other terms are <span class="math notranslate nohighlight">\(\beta_i\)</span> with <span class="math notranslate nohighlight">\(i=1,2,3,4\)</span>. Furthermore we have <span class="math notranslate nohighlight">\(n\)</span> entries for each predictor. It means that our design matrix is an
+<span class="math notranslate nohighlight">\(n\times p\)</span> matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
 </div>
 <div class="section" id="own-code-for-ordinary-least-squares">
 <h2>Own code for Ordinary Least Squares<a class="headerlink" href="#own-code-for-ordinary-least-squares" title="Permalink to this headline">¶</a></h2>
@@ -1562,25 +1561,24 @@ We assume our data can represented by a fourth-order polynomial. For the <span c
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># matrix inversion to find beta</span>
+<span class="c1"># First we set up the data</span>
+<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
+<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
+<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">5</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mf">0.1</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
+<span class="c1"># and then the design matrix X including the intercept</span>
+<span class="c1">#  The design matrix now as function of a fourth-order polynomial</span>
+<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">),</span><span class="mi">5</span><span class="p">))</span>
+<span class="n">X</span><span class="p">[:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mf">1.0</span>
+<span class="n">X</span><span class="p">[:,</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span>
+<span class="n">X</span><span class="p">[:,</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="mi">2</span>
+<span class="n">X</span><span class="p">[:,</span><span class="mi">3</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="mi">3</span>
+<span class="n">X</span><span class="p">[:,</span><span class="mi">4</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="mi">4</span>
 <span class="n">beta</span> <span class="o">=</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span><span class="p">)</span> <span class="o">@</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="p">)</span> <span class="o">@</span> <span class="n">y</span>
 <span class="c1"># and then make the prediction</span>
 <span class="n">ytilde</span> <span class="o">=</span> <span class="n">X</span> <span class="o">@</span> <span class="n">beta</span>
 </pre></div>
 </div>
 </div>
-<div class="cell_output docutils container">
-<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
-<span class="ne">NameError</span><span class="g g-Whitespace">                                 </span>Traceback (most recent call last)
-<span class="nn">Input In [1],</span> in <span class="ni">&lt;cell line: 2&gt;</span><span class="nt">()</span>
-<span class="g g-Whitespace">      </span><span class="mi">1</span> <span class="c1"># matrix inversion to find beta</span>
-<span class="ne">----&gt; </span><span class="mi">2</span> <span class="n">beta</span> <span class="o">=</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span><span class="p">)</span> <span class="o">@</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="p">)</span> <span class="o">@</span> <span class="n">y</span>
-<span class="g g-Whitespace">      </span><span class="mi">3</span> <span class="c1"># and then make the prediction</span>
-<span class="g g-Whitespace">      </span><span class="mi">4</span> <span class="n">ytilde</span> <span class="o">=</span> <span class="n">X</span> <span class="o">@</span> <span class="n">beta</span>
-
-<span class="ne">NameError</span>: name &#39;np&#39; is not defined
-</pre></div>
-</div>
-</div>
 </div>
 <p>Alternatively, you can use the least squares functionality in <strong>Numpy</strong> as</p>
 <div class="cell docutils container">
@@ -1590,6 +1588,17 @@ We assume our data can represented by a fourth-order polynomial. For the <span c
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
+<span class="ne">NameError</span><span class="g g-Whitespace">                                 </span>Traceback (most recent call last)
+<span class="nn">Input In [2],</span> in <span class="ni">&lt;cell line: 1&gt;</span><span class="nt">()</span>
+<span class="ne">----&gt; </span><span class="mi">1</span> <span class="n">fit</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">lstsq</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Energies</span><span class="p">,</span> <span class="n">rcond</span> <span class="o">=</span><span class="kc">None</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
+<span class="g g-Whitespace">      </span><span class="mi">2</span> <span class="n">ytildenp</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">fit</span><span class="p">,</span><span class="n">X</span><span class="o">.</span><span class="n">T</span><span class="p">)</span>
+
+<span class="ne">NameError</span>: name &#39;Energies&#39; is not defined
+</pre></div>
+</div>
+</div>
 </div>
 </div>
 <div class="section" id="adding-error-analysis-and-training-set-up">