-In general, the update of some parameters \( \vec \omega \) given a cost -function defined by some weights \( \vec \omega \), \( c(x, \vec \omega) \), +In general, the update of some parameters \( \hat{\omega} \) given a cost +function defined by some weights \( \hat{\omega} \), \( c(x, \hat{\omega}) \), goes as follows: $$ -\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega), +\hat{\omega}_{\mathrm{new} } = \hat{\omega} - \lambda \nabla_{\hat{\omega}} c(x, \hat{\omega}), $$
-for a number of iterations or until $ \big|\big| \vec -\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some +for a number of iterations or until \( \big|\big| \hat{\omega}_{\mathrm{new} } - \hat{\omega} \big|\big| \) +is smaller than some given tolerance.
The value of \( \lambda \) decides how large steps the algorithm must take -in the direction of $ \nabla_{\vec \omega} c(x, \vec \omega)$. The -notatation \( \nabla_{\vec \omega} \) denotes the gradient with respect to -the elements in \( \vec \omega \). +in the direction of $ \nabla_{\hat{\omega}} c(x, \hat{\omega})$. The +notatation \( \nabla_{\hat{\omega}} \) denotes the gradient with respect to +the elements in \( \hat{\omega} \).
diff --git a/doc/pub/NeuralNet/html/._NeuralNet-bs102.html b/doc/pub/NeuralNet/html/._NeuralNet-bs102.html index 68b801209..9a6b42535 100644 --- a/doc/pub/NeuralNet/html/._NeuralNet-bs102.html +++ b/doc/pub/NeuralNet/html/._NeuralNet-bs102.html @@ -372,8 +372,7 @@ The feedforward step is similar to as for the neural netowork, but now consideri The \( i \)-th neuron at layer \( l \) recieves the result \( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer \( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in -\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec -w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are +\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( w_{i,j}^{(l), \ \mathrm{hidden}} \) with as many weigths as there are elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias \( b_i^{(l), \ \mathrm{hidden} } \): diff --git a/doc/pub/NeuralNet/html/NeuralNet-reveal.html b/doc/pub/NeuralNet/html/NeuralNet-reveal.html index da0e17bf7..26dec382d 100644 --- a/doc/pub/NeuralNet/html/NeuralNet-reveal.html +++ b/doc/pub/NeuralNet/html/NeuralNet-reveal.html @@ -4238,26 +4238,26 @@ The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.
-In general, the update of some parameters \( \vec \omega \) given a cost -function defined by some weights \( \vec \omega \), \( c(x, \vec \omega) \), +In general, the update of some parameters \( \hat{\omega} \) given a cost +function defined by some weights \( \hat{\omega} \), \( c(x, \hat{\omega}) \), goes as follows:
$$
-\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
+\hat{\omega}_{\mathrm{new} } = \hat{\omega} - \lambda \nabla_{\hat{\omega}} c(x, \hat{\omega}),
$$
-for a number of iterations or until $ \big|\big| \vec -\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some +for a number of iterations or until \( \big|\big| \hat{\omega}_{\mathrm{new} } - \hat{\omega} \big|\big| \) +is smaller than some given tolerance.
The value of \( \lambda \) decides how large steps the algorithm must take -in the direction of $ \nabla_{\vec \omega} c(x, \vec \omega)$. The -notatation \( \nabla_{\vec \omega} \) denotes the gradient with respect to -the elements in \( \vec \omega \). +in the direction of $ \nabla_{\hat{\omega}} c(x, \hat{\omega})$. The +notatation \( \nabla_{\hat{\omega}} \) denotes the gradient with respect to +the elements in \( \hat{\omega} \). @@ -4353,8 +4353,7 @@ The feedforward step is similar to as for the neural netowork, but now consideri The \( i \)-th neuron at layer \( l \) recieves the result \( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer \( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in -\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec -w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are +\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( w_{i,j}^{(l), \ \mathrm{hidden}} \) with as many weigths as there are elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias \( b_i^{(l), \ \mathrm{hidden} } \): diff --git a/doc/pub/NeuralNet/html/NeuralNet-solarized.html b/doc/pub/NeuralNet/html/NeuralNet-solarized.html index bfbe755ff..419531dea 100644 --- a/doc/pub/NeuralNet/html/NeuralNet-solarized.html +++ b/doc/pub/NeuralNet/html/NeuralNet-solarized.html @@ -4090,24 +4090,24 @@ The idea of the gradient descent algorithm is to update parameters in direction where the cost function decreases goes to a minimum.
-In general, the update of some parameters \( \vec \omega \) given a cost -function defined by some weights \( \vec \omega \), \( c(x, \vec \omega) \), +In general, the update of some parameters \( \hat{\omega} \) given a cost +function defined by some weights \( \hat{\omega} \), \( c(x, \hat{\omega}) \), goes as follows: $$ -\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega), +\hat{\omega}_{\mathrm{new} } = \hat{\omega} - \lambda \nabla_{\hat{\omega}} c(x, \hat{\omega}), $$
-for a number of iterations or until $ \big|\big| \vec -\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some +for a number of iterations or until \( \big|\big| \hat{\omega}_{\mathrm{new} } - \hat{\omega} \big|\big| \) +is smaller than some given tolerance.
The value of \( \lambda \) decides how large steps the algorithm must take -in the direction of $ \nabla_{\vec \omega} c(x, \vec \omega)$. The -notatation \( \nabla_{\vec \omega} \) denotes the gradient with respect to -the elements in \( \vec \omega \). +in the direction of $ \nabla_{\hat{\omega}} c(x, \hat{\omega})$. The +notatation \( \nabla_{\hat{\omega}} \) denotes the gradient with respect to +the elements in \( \hat{\omega} \).
@@ -4200,8 +4200,7 @@ The feedforward step is similar to as for the neural netowork, but now consideri
The \( i \)-th neuron at layer \( l \) recieves the result
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer
\( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in
-\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec
-w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are
+\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( w_{i,j}^{(l), \ \mathrm{hidden}} \) with as many weigths as there are
elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
\( b_i^{(l), \ \mathrm{hidden} } \):
diff --git a/doc/pub/NeuralNet/html/NeuralNet.html b/doc/pub/NeuralNet/html/NeuralNet.html
index e4b6df910..ff0d73330 100644
--- a/doc/pub/NeuralNet/html/NeuralNet.html
+++ b/doc/pub/NeuralNet/html/NeuralNet.html
@@ -4095,24 +4095,24 @@ The idea of the gradient descent algorithm is to update parameters in
direction where the cost function decreases goes to a minimum.
-In general, the update of some parameters \( \vec \omega \) given a cost -function defined by some weights \( \vec \omega \), \( c(x, \vec \omega) \), +In general, the update of some parameters \( \hat{\omega} \) given a cost +function defined by some weights \( \hat{\omega} \), \( c(x, \hat{\omega}) \), goes as follows: $$ -\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega), +\hat{\omega}_{\mathrm{new} } = \hat{\omega} - \lambda \nabla_{\hat{\omega}} c(x, \hat{\omega}), $$
-for a number of iterations or until $ \big|\big| \vec -\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some +for a number of iterations or until \( \big|\big| \hat{\omega}_{\mathrm{new} } - \hat{\omega} \big|\big| \) +is smaller than some given tolerance.
The value of \( \lambda \) decides how large steps the algorithm must take -in the direction of $ \nabla_{\vec \omega} c(x, \vec \omega)$. The -notatation \( \nabla_{\vec \omega} \) denotes the gradient with respect to -the elements in \( \vec \omega \). +in the direction of $ \nabla_{\hat{\omega}} c(x, \hat{\omega})$. The +notatation \( \nabla_{\hat{\omega}} \) denotes the gradient with respect to +the elements in \( \hat{\omega} \).
@@ -4205,8 +4205,7 @@ The feedforward step is similar to as for the neural netowork, but now consideri
The \( i \)-th neuron at layer \( l \) recieves the result
\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) from the \( j \)-th neuron at layer
\( l-1 \). The \( i \)-th neuron at layer \( l \) weights all of the elements in
-\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( \vec
-w_{i,j}^{(l), \ \mathrm{hidden} } \) with as many weigths as there are
+\( \hat{x}_j^{(l-1),\mathrm{hidden} } \) with a weight vector \( w_{i,j}^{(l), \ \mathrm{hidden}} \) with as many weigths as there are
elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
\( b_i^{(l), \ \mathrm{hidden} } \):
diff --git a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
index 07dc89694..dad165cbf 100644
--- a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
+++ b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb
@@ -4545,8 +4545,8 @@
"The idea of the gradient descent algorithm is to update parameters in\n",
"direction where the cost function decreases goes to a minimum.\n",
"\n",
- "In general, the update of some parameters $\\vec \\omega$ given a cost\n",
- "function defined by some weights $\\vec \\omega$, $c(x, \\vec \\omega)$,\n",
+ "In general, the update of some parameters $\\hat{\\omega}$ given a cost\n",
+ "function defined by some weights $\\hat{\\omega}$, $c(x, \\hat{\\omega})$,\n",
"goes as follows:"
]
},
@@ -4555,7 +4555,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\vec \\omega_{\\mathrm{new} } = \\vec \\omega - \\lambda \\nabla_{\\vec \\omega} c(x, \\vec \\omega),\n",
+ "\\hat{\\omega}_{\\mathrm{new} } = \\hat{\\omega} - \\lambda \\nabla_{\\hat{\\omega}} c(x, \\hat{\\omega}),\n",
"$$"
]
},
@@ -4563,14 +4563,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "for a number of iterations or until $ \\big|\\big| \\vec\n",
- "\\omega_{\\mathrm{new} } - \\vec \\omega \\big|\\big|$ is smaller than some\n",
+ "for a number of iterations or until $\\big|\\big| \\hat{\\omega}_{\\mathrm{new} } - \\hat{\\omega} \\big|\\big|$ \n",
+ "is smaller than some\n",
"given tolerance.\n",
"\n",
"The value of $\\lambda$ decides how large steps the algorithm must take\n",
- "in the direction of $ \\nabla_{\\vec \\omega} c(x, \\vec \\omega)$. The\n",
- "notatation $\\nabla_{\\vec \\omega}$ denotes the gradient with respect to\n",
- "the elements in $\\vec \\omega$.\n",
+ "in the direction of $ \\nabla_{\\hat{\\omega}} c(x, \\hat{\\omega})$. The\n",
+ "notatation $\\nabla_{\\hat{\\omega}}$ denotes the gradient with respect to\n",
+ "the elements in $\\hat{\\omega}$.\n",
"\n",
"\n",
"## More on GD and cost function\n",
@@ -4671,8 +4671,7 @@
"The $i$-th neuron at layer $l$ recieves the result\n",
"$\\hat{x}_j^{(l-1),\\mathrm{hidden} }$ from the $j$-th neuron at layer\n",
"$l-1$. The $i$-th neuron at layer $l$ weights all of the elements in\n",
- "$\\hat{x}_j^{(l-1),\\mathrm{hidden} }$ with a weight vector $\\vec\n",
- "w_{i,j}^{(l), \\ \\mathrm{hidden} }$ with as many weigths as there are\n",
+ "$\\hat{x}_j^{(l-1),\\mathrm{hidden} }$ with a weight vector $w_{i,j}^{(l), \\ \\mathrm{hidden}}$ with as many weigths as there are\n",
"elements in$\\hat{x}_j^{(l-1),\\mathrm{hidden} }$, and adds a bias\n",
"$b_i^{(l), \\ \\mathrm{hidden} }$:"
]
diff --git a/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz b/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz
index 67b12625b..3f80b06c9 100644
Binary files a/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz and b/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz differ
diff --git a/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf b/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf
index 95ce82474..1ae904c28 100644
Binary files a/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf and b/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf differ
diff --git a/doc/src/NeuralNet/NeuralNet.do.txt b/doc/src/NeuralNet/NeuralNet.do.txt
index 2eb2616b9..958c631f5 100644
--- a/doc/src/NeuralNet/NeuralNet.do.txt
+++ b/doc/src/NeuralNet/NeuralNet.do.txt
@@ -3436,24 +3436,24 @@ def cost_function(P, x):
The idea of the gradient descent algorithm is to update parameters in
direction where the cost function decreases goes to a minimum.
-In general, the update of some parameters $\vec \omega$ given a cost
-function defined by some weights $\vec \omega$, $c(x, \vec \omega)$,
+In general, the update of some parameters $\hat{\omega}$ given a cost
+function defined by some weights $\hat{\omega}$, $c(x, \hat{\omega})$,
goes as follows:
!bt
\[
-\vec \omega_{\mathrm{new} } = \vec \omega - \lambda \nabla_{\vec \omega} c(x, \vec \omega),
+\hat{\omega}_{\mathrm{new} } = \hat{\omega} - \lambda \nabla_{\hat{\omega}} c(x, \hat{\omega}),
\]
!et
-for a number of iterations or until $ \big|\big| \vec
-\omega_{\mathrm{new} } - \vec \omega \big|\big|$ is smaller than some
+for a number of iterations or until $\big|\big| \hat{\omega}_{\mathrm{new} } - \hat{\omega} \big|\big|$
+is smaller than some
given tolerance.
The value of $\lambda$ decides how large steps the algorithm must take
-in the direction of $ \nabla_{\vec \omega} c(x, \vec \omega)$. The
-notatation $\nabla_{\vec \omega}$ denotes the gradient with respect to
-the elements in $\vec \omega$.
+in the direction of $ \nabla_{\hat{\omega}} c(x, \hat{\omega})$. The
+notatation $\nabla_{\hat{\omega}}$ denotes the gradient with respect to
+the elements in $\hat{\omega}$.
!split
@@ -3536,8 +3536,7 @@ The feedforward step is similar to as for the neural netowork, but now consideri
The $i$-th neuron at layer $l$ recieves the result
$\hat{x}_j^{(l-1),\mathrm{hidden} }$ from the $j$-th neuron at layer
$l-1$. The $i$-th neuron at layer $l$ weights all of the elements in
-$\hat{x}_j^{(l-1),\mathrm{hidden} }$ with a weight vector $\vec
-w_{i,j}^{(l), \ \mathrm{hidden} }$ with as many weigths as there are
+$\hat{x}_j^{(l-1),\mathrm{hidden} }$ with a weight vector $w_{i,j}^{(l), \ \mathrm{hidden}}$ with as many weigths as there are
elements in$\hat{x}_j^{(l-1),\mathrm{hidden} }$, and adds a bias
$b_i^{(l), \ \mathrm{hidden} }$: