corrected typo
This commit is contained in:
Morten Hjorth-Jensen
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
@@ -2,7 +2,7 @@
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a15180da",
|
||||
"id": "3080bd6e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -14,7 +14,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ac77b923",
|
||||
"id": "2121a646",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -27,7 +27,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e8b90270",
|
||||
"id": "5fdd1312",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -43,25 +43,26 @@
|
||||
"derivatives) of the chosen cost functions. Vectors are always written\n",
|
||||
"as boldfaced lower case letters and matrices as upper case boldfaced\n",
|
||||
"letters. You will find useful the notes from week 35 on derivatives of vectors and matrices.\n",
|
||||
"See also the textbook of Faisal at al, chapter 5 and in particular sections 5.3-5.5 at <https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MathMLbook.pdf>\n",
|
||||
"\n",
|
||||
"Show that"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2e2e8f1a",
|
||||
"id": "5bd583f7",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\frac{\\partial (\\boldsymbol{b}^T\\boldsymbol{a})}{\\partial \\boldsymbol{a}} = \\boldsymbol{b},\n",
|
||||
"\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1cdc68da",
|
||||
"id": "74d2af02",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -71,7 +72,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b87f9fdd",
|
||||
"id": "cc982163",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -83,7 +84,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6ab53932",
|
||||
"id": "2f2c4da8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -93,7 +94,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1a7a8ec2",
|
||||
"id": "75abd105",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -105,7 +106,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3bdb1514",
|
||||
"id": "ed48f425",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -114,13 +115,14 @@
|
||||
"\n",
|
||||
"The second derivative of the mean squared error is then proportional to the so-called Hessian matrix $\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}$.\n",
|
||||
"\n",
|
||||
"**Hint**: In these exercises it is always useful to write out with summation indices the various quantities.\n",
|
||||
"**Hint**: In these exercises it is always useful to write out with summation indices the various quantities. Take also a look at the weekly slides from week 35 and the various examples included there.\n",
|
||||
"\n",
|
||||
"As an example, consider the function"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3e54eca9",
|
||||
"id": "329edf30",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -132,7 +134,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3db49a1c",
|
||||
"id": "01f1090f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -142,7 +144,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5d4d3a0e",
|
||||
"id": "8c71ed48",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -154,7 +156,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "90eecd1c",
|
||||
"id": "4e8012af",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -164,7 +166,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7eb7605f",
|
||||
"id": "a1e0c123",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -176,7 +178,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b30dd90f",
|
||||
"id": "22a65aed",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -186,7 +188,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1167ec1f",
|
||||
"id": "e56f21e8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -198,7 +200,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8f309e9e",
|
||||
"id": "48394062",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -216,7 +218,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "2df1063f",
|
||||
"id": "bb67a97a",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -230,7 +232,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0cb75ec5",
|
||||
"id": "d60db901",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -244,7 +246,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f0c36bdb",
|
||||
"id": "274328a7",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -257,7 +259,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8c3ce778",
|
||||
"id": "59dce0f1",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -268,7 +270,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f861d243",
|
||||
"id": "1e1f8911",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -280,7 +282,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "51500e4b",
|
||||
"id": "ef31124f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -290,7 +292,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fda9dadd",
|
||||
"id": "46ac2123",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -302,7 +304,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6b691499",
|
||||
"id": "edf840af",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -313,7 +315,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b6f10ab9",
|
||||
"id": "31059187",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -333,7 +335,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "285159ae",
|
||||
"id": "f7382860",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -349,7 +351,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c2840fc9",
|
||||
"id": "a10412bd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -359,7 +361,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9dbda275",
|
||||
"id": "8a9f0b89",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -370,7 +372,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "824dba6f",
|
||||
"id": "df5c7f67",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -382,7 +384,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a3f059cf",
|
||||
"id": "e9b0fa39",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -414,11 +414,12 @@ analytical expressions is extremely helpful in case we have simpler
|
||||
derivatives as well as when we analyze various properties (like second
|
||||
derivatives) of the chosen cost functions. Vectors are always written
|
||||
as boldfaced lower case letters and matrices as upper case boldfaced
|
||||
letters. You will find useful the notes from week 35 on derivatives of vectors and matrices.</p>
|
||||
letters. You will find useful the notes from week 35 on derivatives of vectors and matrices.
|
||||
See also the textbook of Faisal at al, chapter 5 and in particular sections 5.3-5.5 at <a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MathMLbook.pdf">https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MathMLbook.pdf</a></p>
|
||||
<p>Show that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial (\boldsymbol{b}^T\boldsymbol{a})}{\partial \boldsymbol{a}} = \boldsymbol{b},
|
||||
\frac{\partial (\boldsymbol{a}^T\boldsymbol{x})}{\partial \boldsymbol{x}} = \boldsymbol{a}^T,
|
||||
\]</div>
|
||||
<p>and</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
@@ -432,8 +433,8 @@ letters. You will find useful the notes from week 35 on derivatives of vectors a
|
||||
\]</div>
|
||||
<p>and finally find the second derivative of this function with respect to the vector <span class="math notranslate nohighlight">\(\boldsymbol{s}\)</span>. If we replace the vector <span class="math notranslate nohighlight">\(\boldsymbol{s}\)</span> with the unknown parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> used to define the ordinary least squares method, we end up with the equations that determine these parameters. The matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> is then the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> here has to be replaced with the outputs <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>.</p>
|
||||
<p>The second derivative of the mean squared error is then proportional to the so-called Hessian matrix <span class="math notranslate nohighlight">\(\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}\)</span>.</p>
|
||||
<p><strong>Hint</strong>: In these exercises it is always useful to write out with summation indices the various quantities.
|
||||
As an example, consider the function</p>
|
||||
<p><strong>Hint</strong>: In these exercises it is always useful to write out with summation indices the various quantities. Take also a look at the weekly slides from week 35 and the various examples included there.</p>
|
||||
<p>As an example, consider the function</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
f(\boldsymbol{x}) =\boldsymbol{A}\boldsymbol{x},
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1345,13 +1345,13 @@ In order to find the derivative of <span class="math notranslate nohighlight">\(
|
||||
\[
|
||||
\alpha = \boldsymbol{z}^T\boldsymbol{x},
|
||||
\]</div>
|
||||
<p>which means that (using our previous example) we have</p>
|
||||
<p>which means that (using our previous example and keeping track of our definition of the derivative of a scalar) we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial \alpha}{\partial \boldsymbol{x}} = \boldsymbol{z}=\boldsymbol{A}^T\boldsymbol{y}.
|
||||
\frac{\partial \alpha}{\partial \boldsymbol{x}} = \frac{\partial \boldsymbol{z}^T\boldsymbol{x}}{\partial \boldsymbol{x}}=\boldsymbol{z}^T=\boldsymbol{A}^T\boldsymbol{y}.
|
||||
\]</div>
|
||||
<p>Note that the resulting vector elements are the same for <span class="math notranslate nohighlight">\(\boldsymbol{z}^T\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{z}\)</span>, the only difference is that one is just the transpose of the other.</p>
|
||||
<p>Since <span class="math notranslate nohighlight">\(\alpha\)</span> is a scalar we have <span class="math notranslate nohighlight">\(\alpha =\alpha^T=\boldsymbol{x}^T\boldsymbol{A}^T\boldsymbol{y}\)</span>. Defining now <span class="math notranslate nohighlight">\(\boldsymbol{z}=\boldsymbol{x}^T\boldsymbol{A}^T\)</span> we find that</p>
|
||||
<p>Since <span class="math notranslate nohighlight">\(\alpha\)</span> is a scalar we have <span class="math notranslate nohighlight">\(\alpha =\alpha^T=\boldsymbol{x}^T\boldsymbol{A}^T\boldsymbol{y}\)</span>. Defining now <span class="math notranslate nohighlight">\(\boldsymbol{z}^T=\boldsymbol{x}^T\boldsymbol{A}^T\)</span> we find that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial \alpha}{\partial \boldsymbol{y}} = \boldsymbol{z}^T=\boldsymbol{x}^T\boldsymbol{A}^T.
|
||||
@@ -1471,7 +1471,11 @@ C(\boldsymbol{\beta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
|
||||
<p>We list here some other useful relations we may encounter (recall that vectors are defined by boldfaced low-key letters)</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial (\boldsymbol{b}^T\boldsymbol{a})}{\partial \boldsymbol{a}} = \boldsymbol{b},
|
||||
\frac{\partial (\boldsymbol{x}^T\boldsymbol{a})}{\partial \boldsymbol{x}} = \boldsymbol{a}^T,
|
||||
\]</div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial (\boldsymbol{a}^T\boldsymbol{x})}{\partial \boldsymbol{x}} = \boldsymbol{a}^T,
|
||||
\]</div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
@@ -1593,7 +1597,7 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.995597266739957
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.9949937802432685
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1610,7 +1614,7 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.007984802498580442
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.01027999506395317
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1625,23 +1629,23 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.00320321 0.00245758 0.05081566 0.0019973 0.00120436 0.0178182
|
||||
0.00900007 0.00408859 0.00048748 0.00486475 0.01992842 0.00354541
|
||||
0.02097611 0.02124814 0.05100557 0.03342039 0.01004759 0.03045198
|
||||
0.01334173 0.02570599 0.01152362 0.0055851 0.01331589 0.00670173
|
||||
0.00111016 0.00674891 0.02190496 0.01318991 0.01012542 0.00536771
|
||||
0.01056529 0.03104011 0.02194347 0.00570653 0.02260804 0.00449913
|
||||
0.01584232 0.00856436 0.01271496 0.02217472 0.00201728 0.00743189
|
||||
0.03465345 0.01961887 0.00071763 0.05253796 0.00713411 0.02597965
|
||||
0.00044036 0.04210928 0.04276324 0.01934278 0.03413554 0.03117346
|
||||
0.01696872 0.0103092 0.05179687 0.03101773 0.00419078 0.04059438
|
||||
0.0681795 0.02798066 0.01269201 0.00019982 0.00546406 0.01333519
|
||||
0.00925952 0.06136355 0.05846851 0.03697739 0.04446751 0.0707621
|
||||
0.02017735 0.01044641 0.06713371 0.01791265 0.05612456 0.02375823
|
||||
0.0050523 0.03568016 0.00771754 0.01959569 0.00679037 0.01420455
|
||||
0.09201618 0.01115073 0.00372262 0.03688621 0.05250129 0.00520339
|
||||
0.00423753 0.00267063 0.0575829 0.00228698 0.00253137 0.0280728
|
||||
0.01264352 0.01788241 0.07061848 0.00521152]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.00436097 0.05486516 0.0284569 0.03042532 0.05255806 0.00279707
|
||||
0.02052907 0.02230304 0.04049356 0.04875464 0.0571849 0.01328032
|
||||
0.00537238 0.05368812 0.01718359 0.00919685 0.01370514 0.01506857
|
||||
0.03391617 0.01736839 0.01644373 0.01853101 0.00043401 0.0501397
|
||||
0.01655376 0.01410563 0.00248515 0.06472468 0.01025732 0.04921476
|
||||
0.00377783 0.01632149 0.0267125 0.05363498 0.00438551 0.00650452
|
||||
0.02410491 0.05852079 0.02783602 0.05441837 0.009371 0.01651438
|
||||
0.01887299 0.04798291 0.01970507 0.05486379 0.01445124 0.06584783
|
||||
0.00363331 0.02765621 0.05439747 0.02002135 0.00886469 0.01777143
|
||||
0.03649344 0.03241934 0.02896611 0.00248403 0.06486283 0.01642653
|
||||
0.04926154 0.00057042 0.0008744 0.02936993 0.03010304 0.01815639
|
||||
0.04037945 0.00553874 0.00223388 0.01474277 0.06839199 0.0306628
|
||||
0.01057262 0.04008416 0.0391324 0.0350905 0.03760958 0.0193714
|
||||
0.0201182 0.00557755 0.04383596 0.02646579 0.02413829 0.00946466
|
||||
0.01335463 0.0125267 0.00971419 0.06107268 0.02539264 0.0227732
|
||||
0.00396906 0.02333106 0.00192092 0.02267534 0.00241645 0.00636841
|
||||
0.02635071 0.03800193 0.04967168 0.00836194]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1710,15 +1714,15 @@ but now splitting the data into a training set and a test set.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 2.032142 0.27650047 3.22178183 2.62592637 -1.1157376 ]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 1.96008541 0.31597257 4.77260857 -0.39452621 0.29176118]
|
||||
Training R2
|
||||
0.994910550643694
|
||||
0.9958819974114181
|
||||
Training MSE
|
||||
0.011016261190269798
|
||||
0.00866641755559425
|
||||
Test R2
|
||||
0.9949909770903465
|
||||
0.9901527242749735
|
||||
Test MSE
|
||||
0.007328424743352142
|
||||
0.020056422754215136
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2276,7 +2280,7 @@ the aims is to reproduce Figure 2.11 of <a class="reference external" href="http
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/week35_146_0.png" src="_images/week35_146_0.png" />
|
||||
<img alt="_images/week35_147_0.png" src="_images/week35_147_0.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2677,7 +2681,7 @@ MSE with Sklearn intercept
|
||||
0.004113634617443131
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/week35_181_1.png" src="_images/week35_181_1.png" />
|
||||
<img alt="_images/week35_182_1.png" src="_images/week35_182_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>The intercept is the value of our output/target variable
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a15180da",
|
||||
"id": "3080bd6e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -14,7 +14,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ac77b923",
|
||||
"id": "2121a646",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -27,7 +27,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e8b90270",
|
||||
"id": "5fdd1312",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -43,25 +43,26 @@
|
||||
"derivatives) of the chosen cost functions. Vectors are always written\n",
|
||||
"as boldfaced lower case letters and matrices as upper case boldfaced\n",
|
||||
"letters. You will find useful the notes from week 35 on derivatives of vectors and matrices.\n",
|
||||
"See also the textbook of Faisal at al, chapter 5 and in particular sections 5.3-5.5 at <https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MathMLbook.pdf>\n",
|
||||
"\n",
|
||||
"Show that"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2e2e8f1a",
|
||||
"id": "5bd583f7",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\frac{\\partial (\\boldsymbol{b}^T\\boldsymbol{a})}{\\partial \\boldsymbol{a}} = \\boldsymbol{b},\n",
|
||||
"\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1cdc68da",
|
||||
"id": "74d2af02",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -71,7 +72,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b87f9fdd",
|
||||
"id": "cc982163",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -83,7 +84,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6ab53932",
|
||||
"id": "2f2c4da8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -93,7 +94,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1a7a8ec2",
|
||||
"id": "75abd105",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -105,7 +106,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3bdb1514",
|
||||
"id": "ed48f425",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -114,13 +115,14 @@
|
||||
"\n",
|
||||
"The second derivative of the mean squared error is then proportional to the so-called Hessian matrix $\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}$.\n",
|
||||
"\n",
|
||||
"**Hint**: In these exercises it is always useful to write out with summation indices the various quantities.\n",
|
||||
"**Hint**: In these exercises it is always useful to write out with summation indices the various quantities. Take also a look at the weekly slides from week 35 and the various examples included there.\n",
|
||||
"\n",
|
||||
"As an example, consider the function"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3e54eca9",
|
||||
"id": "329edf30",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -132,7 +134,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3db49a1c",
|
||||
"id": "01f1090f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -142,7 +144,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5d4d3a0e",
|
||||
"id": "8c71ed48",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -154,7 +156,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "90eecd1c",
|
||||
"id": "4e8012af",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -164,7 +166,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7eb7605f",
|
||||
"id": "a1e0c123",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -176,7 +178,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b30dd90f",
|
||||
"id": "22a65aed",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -186,7 +188,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1167ec1f",
|
||||
"id": "e56f21e8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -198,7 +200,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8f309e9e",
|
||||
"id": "48394062",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -216,7 +218,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "2df1063f",
|
||||
"id": "bb67a97a",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -230,7 +232,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0cb75ec5",
|
||||
"id": "d60db901",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -244,7 +246,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f0c36bdb",
|
||||
"id": "274328a7",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -257,7 +259,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8c3ce778",
|
||||
"id": "59dce0f1",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -268,7 +270,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f861d243",
|
||||
"id": "1e1f8911",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -280,7 +282,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "51500e4b",
|
||||
"id": "ef31124f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -290,7 +292,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fda9dadd",
|
||||
"id": "46ac2123",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -302,7 +304,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6b691499",
|
||||
"id": "edf840af",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -313,7 +315,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b6f10ab9",
|
||||
"id": "31059187",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -333,7 +335,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "285159ae",
|
||||
"id": "f7382860",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -349,7 +351,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c2840fc9",
|
||||
"id": "a10412bd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -359,7 +361,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9dbda275",
|
||||
"id": "8a9f0b89",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -370,7 +372,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "824dba6f",
|
||||
"id": "df5c7f67",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -382,7 +384,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a3f059cf",
|
||||
"id": "e9b0fa39",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
|
||||
@@ -21,11 +21,12 @@
|
||||
# derivatives) of the chosen cost functions. Vectors are always written
|
||||
# as boldfaced lower case letters and matrices as upper case boldfaced
|
||||
# letters. You will find useful the notes from week 35 on derivatives of vectors and matrices.
|
||||
# See also the textbook of Faisal at al, chapter 5 and in particular sections 5.3-5.5 at <https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MathMLbook.pdf>
|
||||
#
|
||||
# Show that
|
||||
|
||||
# $$
|
||||
# \frac{\partial (\boldsymbol{b}^T\boldsymbol{a})}{\partial \boldsymbol{a}} = \boldsymbol{b},
|
||||
# \frac{\partial (\boldsymbol{a}^T\boldsymbol{x})}{\partial \boldsymbol{x}} = \boldsymbol{a}^T,
|
||||
# $$
|
||||
|
||||
# and
|
||||
@@ -44,7 +45,8 @@
|
||||
#
|
||||
# The second derivative of the mean squared error is then proportional to the so-called Hessian matrix $\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}$.
|
||||
#
|
||||
# **Hint**: In these exercises it is always useful to write out with summation indices the various quantities.
|
||||
# **Hint**: In these exercises it is always useful to write out with summation indices the various quantities. Take also a look at the weekly slides from week 35 and the various examples included there.
|
||||
#
|
||||
# As an example, consider the function
|
||||
|
||||
# $$
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -291,15 +291,15 @@
|
||||
# \alpha = \boldsymbol{z}^T\boldsymbol{x},
|
||||
# $$
|
||||
|
||||
# which means that (using our previous example) we have
|
||||
# which means that (using our previous example and keeping track of our definition of the derivative of a scalar) we have
|
||||
|
||||
# $$
|
||||
# \frac{\partial \alpha}{\partial \boldsymbol{x}} = \boldsymbol{z}=\boldsymbol{A}^T\boldsymbol{y}.
|
||||
# \frac{\partial \alpha}{\partial \boldsymbol{x}} = \frac{\partial \boldsymbol{z}^T\boldsymbol{x}}{\partial \boldsymbol{x}}=\boldsymbol{z}^T=\boldsymbol{A}^T\boldsymbol{y}.
|
||||
# $$
|
||||
|
||||
# Note that the resulting vector elements are the same for $\boldsymbol{z}^T$ and $\boldsymbol{z}$, the only difference is that one is just the transpose of the other.
|
||||
#
|
||||
# Since $\alpha$ is a scalar we have $\alpha =\alpha^T=\boldsymbol{x}^T\boldsymbol{A}^T\boldsymbol{y}$. Defining now $\boldsymbol{z}=\boldsymbol{x}^T\boldsymbol{A}^T$ we find that
|
||||
# Since $\alpha$ is a scalar we have $\alpha =\alpha^T=\boldsymbol{x}^T\boldsymbol{A}^T\boldsymbol{y}$. Defining now $\boldsymbol{z}^T=\boldsymbol{x}^T\boldsymbol{A}^T$ we find that
|
||||
|
||||
# $$
|
||||
# \frac{\partial \alpha}{\partial \boldsymbol{y}} = \boldsymbol{z}^T=\boldsymbol{x}^T\boldsymbol{A}^T.
|
||||
@@ -436,7 +436,11 @@
|
||||
# We list here some other useful relations we may encounter (recall that vectors are defined by boldfaced low-key letters)
|
||||
|
||||
# $$
|
||||
# \frac{\partial (\boldsymbol{b}^T\boldsymbol{a})}{\partial \boldsymbol{a}} = \boldsymbol{b},
|
||||
# \frac{\partial (\boldsymbol{x}^T\boldsymbol{a})}{\partial \boldsymbol{x}} = \boldsymbol{a}^T,
|
||||
# $$
|
||||
|
||||
# $$
|
||||
# \frac{\partial (\boldsymbol{a}^T\boldsymbol{x})}{\partial \boldsymbol{x}} = \boldsymbol{a}^T,
|
||||
# $$
|
||||
|
||||
# $$
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
Reference in New Issue
Block a user