diff --git a/doc/LectureNotes/exercisesweek35.ipynb b/doc/LectureNotes/exercisesweek35.ipynb
index e9e4b1b70..4c5dfdacf 100644
--- a/doc/LectureNotes/exercisesweek35.ipynb
+++ b/doc/LectureNotes/exercisesweek35.ipynb
@@ -142,11 +142,11 @@
"source": [
"
\n",
"\n",
+ "We define $\\alpha \\equiv a^T x = \\sum_j a_j x_j$. Then\n",
+ "\n",
"$$\n",
- "\\frac{\\partial (a^T x)}{\\partial x} = \\frac{\\partial}{\\partial x_j} (a_{ji} x_j) = a_{ji} = a^T\n",
+ "\\frac{\\partial \\alpha}{\\partial x} = \\frac{\\partial}{\\partial x_j} (a_j x_j) = a_j = a^T\n",
"$$\n",
- "where $ji = 0$ since we have column-vectors.\n",
- "Thus $a_{ji} = a_{j0} = a^T$ a row vector.\n",
"\n",
"
"
]
@@ -170,26 +170,19 @@
"source": [
"\n",
"\n",
- "Using differentials,\n",
- "\n",
+ "If we define $\\phi \\equiv a^T A a$ which is evidently a scalar (thus $\\phi = \\phi^T$), we can write it as:\n",
"$$\n",
- "\\mathrm{d}f\n",
- "= (\\mathrm{d}\\mathbf{a})^T\\mathbf{A}\\mathbf{a}+\\mathbf{a}^T\\mathbf{A}\\,\\mathrm{d}\\mathbf{a}\n",
- "= (\\mathrm{d}\\mathbf{a})^T(\\mathbf{A}+\\mathbf{A}^T)\\mathbf{a}.\n",
+ "\\phi = \\sum_{i, j} a_i A_{ij} a_j\n",
"$$\n",
- "\n",
- "By definition $\\mathrm{d}f = \\big(\\tfrac{\\partial f}{\\partial \\mathbf{a}}\\big)^T \\mathrm{d}\\mathbf{a}$, hence\n",
- "\n",
+ "Using the scalar property we can rewrite the problem as\n",
"$$\n",
- "\\frac{\\partial (\\mathbf{a}^T \\mathbf{A}\\mathbf{a})}{\\partial \\mathbf{a}}\n",
- "= (\\mathbf{A}+\\mathbf{A}^T)\\mathbf{a}.\n",
+ "\\frac{\\partial \\phi}{\\partial a} = \\frac{\\partial}{\\partial a} a^T A a = \\frac{\\partial \\phi^T}{\\partial a} = \\frac{\\partial}{\\partial a}a^T A^T a\n",
"$$\n",
- "\n",
- "If you represent gradients as row vectors, equivalently\n",
- "\n",
+ "Now evaluating\n",
"$$\n",
- "\\frac{\\partial (\\mathbf{a}^T \\mathbf{A}\\mathbf{a})}{\\partial \\mathbf{a}}=\\mathbf{a}^T(\\mathbf{A}+\\mathbf{A}^T).\n",
+ "\\frac{\\partial \\phi}{\\partial a_k} = \\sum_{i,j} (\\frac{\\partial a_i}{\\partial a_k} A_{ij} a_j + a_j A_{ij} \\frac{\\partial a_j}{\\partial a_k}) = \\sum_{i,j} (\\delta_{ik} A_{ij} a_j + a_j A_{ij} \\delta_{jk}) = \\sum_{j} A_{kj} a_j + \\sum_{i} A_{ik} a_i = a^T (A + A^T)\n",
"$$\n",
+ "with the Kronecker-Delta $\\delta_{xy} = \\begin{cases} 0 & x \\neq y \\\\ 1 & x = y \\end{cases}$. As the derivative of two components can be written as $\\frac{\\partial a_x}{\\partial a_y} = \\delta_{xy}$.\n",
"\n",
"
"
]
@@ -289,7 +282,7 @@
"&= -2 (x-As)^T A\n",
"\\end{aligned}\n",
"$$\n",
- "\n",
+ "We can make the last step due to the fact, that $(x-As)^T(x-As) \\equiv \\gamma$ is a scalar and thus $\\gamma = \\gamma^T$. Therefore the term for the derivative of $\\gamma^T$ has to be equivalent and we can derive the last equivalence.\n",
""
]
},
@@ -1062,7 +1055,7 @@
"metadata": {},
"source": [
"\n",
- "According to the graph the models ability to generalize increases with the degree of the polynomial. This is indicated by the testing MSE decreasing with the increase in polynomial degree. The difference to the graph from Bickel et al stems from the fact, that we use an exponential function which in turn can be written as an infinite polynomial function. Thus we approach the source of the data with an increasing degree of the polynomial inputs. Thus in this case the models generalization ability increases with the degree.\n",
+ "According to the graph the models ability to generalize increases with the degree of the polynomial. This is indicated by the testing MSE decreasing with the increase in polynomial degree. The difference to the graph from Bickel et al stems from the fact, that we use an exponential function which in turn can be written as an infinite polynomial function. Thus we approach the source of the data with an increasing degree of the polynomial inputs. Thus in this case the models generalization ability increases with the degree. We don't see the typical features of overfitting normally arriving with an increase in parameters of the model since our ratios between dataset size, noise in the training data and parameter number is still in favor of good generalizbility. With a further increase in noise or parameter number or a decrease in dataset size we could observe a decrease in model performance with resepect to the testing set. Our model would be overfit.\n",
"\n",
"
"
]