To solve this problem, you may want to use a variant of the ReLU function, such as the leaky ReLU discussed before or the so-called exponential linear unit (ELU) function $$ -ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z > 0,\\ z & z \le 0.\end{array}\right. +ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z < 0,\\ z & z \ge 0.\end{array}\right. $$
diff --git a/doc/pub/NeuralNet/html/NeuralNet-reveal.html b/doc/pub/NeuralNet/html/NeuralNet-reveal.html index 8fc3067d1..1022e6c03 100644 --- a/doc/pub/NeuralNet/html/NeuralNet-reveal.html +++ b/doc/pub/NeuralNet/html/NeuralNet-reveal.html @@ -2873,7 +2873,7 @@ since the gradient of the ReLU function is 0 when its input is negative. To solve this problem, you may want to use a variant of the ReLU function, such as the leaky ReLU discussed before or the so-called exponential linear unit (ELU) function
$$
-ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z > 0,\\ z & z \le 0.\end{array}\right.
+ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z < 0,\\ z & z \ge 0.\end{array}\right.
$$
diff --git a/doc/pub/NeuralNet/html/NeuralNet-solarized.html b/doc/pub/NeuralNet/html/NeuralNet-solarized.html
index 4fe68b031..5aa97c64b 100644
--- a/doc/pub/NeuralNet/html/NeuralNet-solarized.html
+++ b/doc/pub/NeuralNet/html/NeuralNet-solarized.html
@@ -2731,7 +2731,7 @@ since the gradient of the ReLU function is 0 when its input is negative.
To solve this problem, you may want to use a variant of the ReLU function, such as the leaky ReLU discussed before or the so-called exponential linear unit (ELU) function $$ -ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z > 0,\\ z & z \le 0.\end{array}\right. +ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z < 0,\\ z & z \ge 0.\end{array}\right. $$
diff --git a/doc/pub/NeuralNet/html/NeuralNet.html b/doc/pub/NeuralNet/html/NeuralNet.html index a7600b8c4..e468079e7 100644 --- a/doc/pub/NeuralNet/html/NeuralNet.html +++ b/doc/pub/NeuralNet/html/NeuralNet.html @@ -2736,7 +2736,7 @@ since the gradient of the ReLU function is 0 when its input is negative.
To solve this problem, you may want to use a variant of the ReLU function, such as the leaky ReLU discussed before or the so-called exponential linear unit (ELU) function $$ -ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z > 0,\\ z & z \le 0.\end{array}\right. +ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z < 0,\\ z & z \ge 0.\end{array}\right. $$
diff --git a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb index d347caee3..e9142f874 100644 --- a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb +++ b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb @@ -3021,7 +3021,7 @@ "metadata": {}, "source": [ "$$\n", - "ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z > 0,\\\\ z & z \\le 0.\\end{array}\\right.\n", + "ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n", "$$" ] }, diff --git a/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz b/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz index b76609507..e55a4b7de 100644 Binary files a/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz and b/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz differ diff --git a/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf b/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf index 1630dbd4c..96c52cb51 100644 Binary files a/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf and b/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf differ diff --git a/doc/src/NeuralNet/NeuralNet.do.txt b/doc/src/NeuralNet/NeuralNet.do.txt index 750133368..e4f4f12f7 100644 --- a/doc/src/NeuralNet/NeuralNet.do.txt +++ b/doc/src/NeuralNet/NeuralNet.do.txt @@ -2263,7 +2263,7 @@ since the gradient of the ReLU function is 0 when its input is negative. To solve this problem, you may want to use a variant of the ReLU function, such as the leaky ReLU discussed before or the so-called exponential linear unit (ELU) function !bt \[ -ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z > 0,\\ z & z \le 0.\end{array}\right. +ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z < 0,\\ z & z \ge 0.\end{array}\right. \] !et