added steepest descent

2019-11-12 00:06:51 +01:00
parent 18239e1a21
commit 3d5793a7b1
10 changed files with 173 additions and 6 deletions
@@ -293,7 +293,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Nov 11, 2019</h4></center> <!-- date -->
+<center><h4>Nov 12, 2019</h4></center> <!-- date -->
 <br>
 <p>

@@ -284,6 +284,28 @@ $$
 (\hat{\boldsymbol{f}})  \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
 $$

+<p>
+We define a real function \( h_m(x) \) that defines our final function \( f_M(x) \) as
+$$
+f_M(x) = \sum_{m=0}^M h_m(x).
+$$
+
+<p>
+In the steepest decent approach we approximate \( h_m(x) = -\rho_m g_m(x) \), where \( \rho_m \) is a scalar and \( g_m(x) \) the gradient defined as
+$$
+g_m(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{m-1}(x_i)}.
+$$
+
+<p>
+With the new gradient we can update \( f_m(x) = f_{m-1}(x) -\rho_m g_m(x) \). Using the above squared-error function we see that
+the gradient is \( g_m(x_i) = -2(y_i-f(x_i)) \).
+
+<p>
+Choosing \( f_0(x)=0 \) we obtain \( g_m(x) = -2y_i \) and inserting this into the minimization problem for the cost function we have
+$$
+(\rho_1)  \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
+$$
+
 <p>
 <p>
 <!-- navigation buttons at the bottom of the page -->
@@ -293,7 +293,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Nov 11, 2019</h4></center> <!-- date -->
+<center><h4>Nov 12, 2019</h4></center> <!-- date -->
 <br>
 <p>

@@ -148,7 +148,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>&nbsp;<br>
-<center><h4>Nov 11, 2019</h4></center> <!-- date -->
+<center><h4>Nov 12, 2019</h4></center> <!-- date -->
 <br>
 <p>

@@ -2381,6 +2381,34 @@ $$
 (\hat{\boldsymbol{f}})  \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
 $$
 <p>&nbsp;<br>
+
+<p>
+We define a real function \( h_m(x) \) that defines our final function \( f_M(x) \) as
+<p>&nbsp;<br>
+$$
+f_M(x) = \sum_{m=0}^M h_m(x).
+$$
+<p>&nbsp;<br>
+
+<p>
+In the steepest decent approach we approximate \( h_m(x) = -\rho_m g_m(x) \), where \( \rho_m \) is a scalar and \( g_m(x) \) the gradient defined as
+<p>&nbsp;<br>
+$$
+g_m(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{m-1}(x_i)}.
+$$
+<p>&nbsp;<br>
+
+<p>
+With the new gradient we can update \( f_m(x) = f_{m-1}(x) -\rho_m g_m(x) \). Using the above squared-error function we see that
+the gradient is \( g_m(x_i) = -2(y_i-f(x_i)) \).
+
+<p>
+Choosing \( f_0(x)=0 \) we obtain \( g_m(x) = -2y_i \) and inserting this into the minimization problem for the cost function we have
+<p>&nbsp;<br>
+$$
+(\rho_1)  \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
+$$
+<p>&nbsp;<br>
 </section>


@@ -221,7 +221,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Nov 11, 2019</h4></center> <!-- date -->
+<center><h4>Nov 12, 2019</h4></center> <!-- date -->
 <br>
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2348,6 +2348,28 @@ $$
 (\hat{\boldsymbol{f}})  \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
 $$

+<p>
+We define a real function \( h_m(x) \) that defines our final function \( f_M(x) \) as
+$$
+f_M(x) = \sum_{m=0}^M h_m(x).
+$$
+
+<p>
+In the steepest decent approach we approximate \( h_m(x) = -\rho_m g_m(x) \), where \( \rho_m \) is a scalar and \( g_m(x) \) the gradient defined as
+$$
+g_m(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{m-1}(x_i)}.
+$$
+
+<p>
+With the new gradient we can update \( f_m(x) = f_{m-1}(x) -\rho_m g_m(x) \). Using the above squared-error function we see that
+the gradient is \( g_m(x_i) = -2(y_i-f(x_i)) \).
+
+<p>
+Choosing \( f_0(x)=0 \) we obtain \( g_m(x) = -2y_i \) and inserting this into the minimization problem for the cost function we have
+$$
+(\rho_1)  \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
+$$
+
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

@@ -226,7 +226,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Nov 11, 2019</h4></center> <!-- date -->
+<center><h4>Nov 12, 2019</h4></center> <!-- date -->
 <br>
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2353,6 +2353,28 @@ $$
 (\hat{\boldsymbol{f}})  \mathrm{argmin}_{\boldsymbol{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
 $$

+<p>
+We define a real function \( h_m(x) \) that defines our final function \( f_M(x) \) as
+$$
+f_M(x) = \sum_{m=0}^M h_m(x).
+$$
+
+<p>
+In the steepest decent approach we approximate \( h_m(x) = -\rho_m g_m(x) \), where \( \rho_m \) is a scalar and \( g_m(x) \) the gradient defined as
+$$
+g_m(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{m-1}(x_i)}.
+$$
+
+<p>
+With the new gradient we can update \( f_m(x) = f_{m-1}(x) -\rho_m g_m(x) \). Using the above squared-error function we see that
+the gradient is \( g_m(x_i) = -2(y_i-f(x_i)) \).
+
+<p>
+Choosing \( f_0(x)=0 \) we obtain \( g_m(x) = -2y_i \) and inserting this into the minimization problem for the cost function we have
+$$
+(\rho_1)  \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
+$$
+
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

@@ -10,7 +10,7 @@
    "<!-- Author: -->  \n",
    "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
    "\n",
-    "Date: **Nov 11, 2019**\n",
+    "Date: **Nov 12, 2019**\n",
    "\n",
    "Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
    "\n",
@@ -2580,6 +2580,57 @@
    "$$"
   ]
  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We define a real function $h_m(x)$ that defines our final function $f_M(x)$ as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "f_M(x) = \\sum_{m=0}^M h_m(x).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the steepest decent approach we approximate $h_m(x) = -\\rho_m g_m(x)$, where $\\rho_m$ is a scalar and $g_m(x)$ the gradient defined as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "g_m(x_i) = \\left[ \\frac{\\partial {\\cal L}(y_i, f(x_i))}{\\partial f(x_i)}\\right]_{f(x_i)=f_{m-1}(x_i)}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "With the new gradient we can update $f_m(x) = f_{m-1}(x) -\\rho_m g_m(x)$. Using the above squared-error function we see that\n",
+    "the gradient is $g_m(x_i) = -2(y_i-f(x_i))$.\n",
+    "\n",
+    "Choosing $f_0(x)=0$ we obtain $g_m(x) = -2y_i$ and inserting this into the minimization problem for the cost function we have"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "(\\rho_1)  \\mathrm{argmin}_{\\rho}\\hspace{0.1cm} \\sum_{i=0}^{n-1}(y_i+2\\rho y_i)^2.\n",
+    "$$"
+   ]
+  },
  {
   "cell_type": "markdown",
   "metadata": {},
@@ -1960,7 +1960,29 @@ This means that for every iteration, we need to optimize
 \]
 !et

+We define a real function $h_m(x)$ that defines our final function $f_M(x)$ as
+!bt
+\[
+f_M(x) = \sum_{m=0}^M h_m(x).
+\]
+!et

+In the steepest decent approach we approximate $h_m(x) = -\rho_m g_m(x)$, where $\rho_m$ is a scalar and $g_m(x)$ the gradient defined as
+!bt
+\[
+g_m(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{m-1}(x_i)}.
+\]
+!et
+
+With the new gradient we can update $f_m(x) = f_{m-1}(x) -\rho_m g_m(x)$. Using the above squared-error function we see that
+the gradient is $g_m(x_i) = -2(y_i-f(x_i))$.
+
+Choosing $f_0(x)=0$ we obtain $g_m(x) = -2y_i$ and inserting this into the minimization problem for the cost function we have
+!bt
+\[
+(\rho_1)  \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
+\]
+!et


 !split