update week 35

doc/pub/week35/ipynb/week35.ipynb

@@ -841,6 +841,46 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "## Meet the Hessian Matrix\n",
+    "\n",
+    "A very important matrix we will meet again and again in Machine Learning is the Hessian.\n",
+    "It is given by the second derivative of the cost function with respect to the parameter $\\beta$. Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} =\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\left[-\\frac{2}{n}\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right]=\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The Hessian matrix plays an important role and is defined here as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters\n",
+    "$\\hat{\\boldsymbol{\\beta}}$. Furthermore, we will see later this week that is (beside $1/n$) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.\n",
+    "\n",
+    "**Linear algebra question:**  Can we use the Hessian matrix to say something about properties of the cost function (our optmization  problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). \n",
+    "\n",
     "## Interpretations and optimizing our parameters\n",
     "The residuals $\\boldsymbol{\\epsilon}$ are in turn given by"
    ]
@@ -2360,6 +2400,21 @@
     "in the program terminating due to a singular matrix.\n",
     "\n",
     "\n",
+    "## Note about SVD Calculations\n",
+    "\n",
+    "The $U$, $S$, and $V$ matrices returned from the **svd()** function\n",
+    "cannot be multiplied directly.\n",
+    "\n",
+    "As you can see from the code, the $S$\n",
+    "vector must be converted into a diagonal matrix. This may cause a \n",
+    "as\n",
+    "the size of the matrices do not fit the rules of matrix\n",
+    "multiplication, where the number of columns in a matrix must match the\n",
+    "number of rows in the subsequent matrix.\n",
+    "\n",
+    "If you wish to include the zero singular values, you will need to resize the matrices. More about this later.\n",
+    "\n",
+    "\n",
     "\n",
     "## Mathematical Properties\n",
     "\n",