35
36
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs027.html b/doc/pub/week40/html/._week40-bs027.html
index 44ef4bc74..6e3b758a8 100644
--- a/doc/pub/week40/html/._week40-bs027.html
+++ b/doc/pub/week40/html/._week40-bs027.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -378,7 +380,7 @@ Note that if n is equal to zero or one, Autograd will give an error message. Thi
36
37
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs028.html b/doc/pub/week40/html/._week40-bs028.html
index fa615d1f2..9e4f97dac 100644
--- a/doc/pub/week40/html/._week40-bs028.html
+++ b/doc/pub/week40/html/._week40-bs028.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -366,7 +368,7 @@ Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The
37
38
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs029.html b/doc/pub/week40/html/._week40-bs029.html
index 52d49881d..257779594 100644
--- a/doc/pub/week40/html/._week40-bs029.html
+++ b/doc/pub/week40/html/._week40-bs029.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -382,7 +384,7 @@ x = np.a
38
39
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs030.html b/doc/pub/week40/html/._week40-bs030.html
index 264505d63..a9c9d8dde 100644
--- a/doc/pub/week40/html/._week40-bs030.html
+++ b/doc/pub/week40/html/._week40-bs030.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -353,7 +355,7 @@ a /=b
39
40
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs031.html b/doc/pub/week40/html/._week40-bs031.html
index 163a9b137..168bb8e69 100644
--- a/doc/pub/week40/html/._week40-bs031.html
+++ b/doc/pub/week40/html/._week40-bs031.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -351,7 +353,7 @@ MathJax.Hub.Config({
40
41
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs032.html b/doc/pub/week40/html/._week40-bs032.html
index 0dfa5ba0f..2c50d316b 100644
--- a/doc/pub/week40/html/._week40-bs032.html
+++ b/doc/pub/week40/html/._week40-bs032.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -354,7 +356,7 @@ a weight variable.
41
42
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs033.html b/doc/pub/week40/html/._week40-bs033.html
index c7cd7a3ca..4eed41089 100644
--- a/doc/pub/week40/html/._week40-bs033.html
+++ b/doc/pub/week40/html/._week40-bs033.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -403,7 +405,7 @@ humanities to life science and medicine.
42
43
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs034.html b/doc/pub/week40/html/._week40-bs034.html
index f6ee62dcf..16337bc06 100644
--- a/doc/pub/week40/html/._week40-bs034.html
+++ b/doc/pub/week40/html/._week40-bs034.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -367,7 +369,7 @@ methods we discussed earlier.
43
44
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs035.html b/doc/pub/week40/html/._week40-bs035.html
index d704b0b2a..98195a9a6 100644
--- a/doc/pub/week40/html/._week40-bs035.html
+++ b/doc/pub/week40/html/._week40-bs035.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -358,7 +360,7 @@ to all nodes in the subsequent layer, making this a so-called
44
45
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs036.html b/doc/pub/week40/html/._week40-bs036.html
index 4f3f0e001..1b99889de 100644
--- a/doc/pub/week40/html/._week40-bs036.html
+++ b/doc/pub/week40/html/._week40-bs036.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -366,7 +368,7 @@ recognition.
45
46
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs037.html b/doc/pub/week40/html/._week40-bs037.html
index 9c716a195..a3f8bc102 100644
--- a/doc/pub/week40/html/._week40-bs037.html
+++ b/doc/pub/week40/html/._week40-bs037.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -357,7 +359,7 @@ especially well-suited for handwriting and speech recognition.
46
47
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs038.html b/doc/pub/week40/html/._week40-bs038.html
index f7ee2346f..f79c52e51 100644
--- a/doc/pub/week40/html/._week40-bs038.html
+++ b/doc/pub/week40/html/._week40-bs038.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -357,7 +359,7 @@ type of NN due the unusual activation functions.
47
48
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs039.html b/doc/pub/week40/html/._week40-bs039.html
index 3b218e9eb..8a0bf9246 100644
--- a/doc/pub/week40/html/._week40-bs039.html
+++ b/doc/pub/week40/html/._week40-bs039.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -354,7 +356,7 @@ Such networks are often called multilayer perceptrons (MLPs).
48
49
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs040.html b/doc/pub/week40/html/._week40-bs040.html
index f48a0744b..98c9b24af 100644
--- a/doc/pub/week40/html/._week40-bs040.html
+++ b/doc/pub/week40/html/._week40-bs040.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -358,7 +360,7 @@ as to not restrict the range of output values.
49
50
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs041.html b/doc/pub/week40/html/._week40-bs041.html
index 4e7607701..3fa8fae51 100644
--- a/doc/pub/week40/html/._week40-bs041.html
+++ b/doc/pub/week40/html/._week40-bs041.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -352,7 +354,7 @@ MathJax.Hub.Config({
50
51
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs042.html b/doc/pub/week40/html/._week40-bs042.html
index df7b11be8..847329d7c 100644
--- a/doc/pub/week40/html/._week40-bs042.html
+++ b/doc/pub/week40/html/._week40-bs042.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -385,7 +387,7 @@ What is happening here?
51
52
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs043.html b/doc/pub/week40/html/._week40-bs043.html
index 7f47db33a..41719af0c 100644
--- a/doc/pub/week40/html/._week40-bs043.html
+++ b/doc/pub/week40/html/._week40-bs043.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -401,7 +403,7 @@ Not exactly impressive, but somewhat better.
52
53
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs044.html b/doc/pub/week40/html/._week40-bs044.html
index 0f78c30a2..4189d3d6e 100644
--- a/doc/pub/week40/html/._week40-bs044.html
+++ b/doc/pub/week40/html/._week40-bs044.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,21 +319,20 @@ MathJax.Hub.Config({
-Mathematical model
+Adding Neural Networks
-The output \( y \) is produced via the activation function \( f \)
-$$
- y = f\left(\sum_{i=1}^n w_ix_i + b_i\right) = f(z),
-$$
-This function receives \( x_i \) as inputs.
-Here the activation \( z=(\sum_{i=1}^n w_ix_i+b_i) \).
-In an FFNN of such neurons, the inputs \( x_i \) are the outputs of
-the neurons in the preceding layer. Furthermore, an MLP is
-fully-connected, which means that each neuron receives a weighted sum
-of the outputs of all neurons in the previous layer.
+
+
# and now neural networks with Scikit-Learn and the XOR
+from sklearn.neural_network import MLPClassifier
+from sklearn.datasets import make_classification
+X, yXOR = make_classification(n_samples=100, random_state=1)
+FFNN = MLPClassifier(random_state=1, max_iter=300).fit(X, yXOR)
+FFNN.predict_proba(X)
+print(f"Test set accuracy with Feed Forward Neural Network for XOR gate:{FFNN.score(X, yXOR)}")
+
@@ -358,7 +359,7 @@ of the outputs of all neurons in the previous layer.
53
54
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs045.html b/doc/pub/week40/html/._week40-bs045.html
index af9355d77..3f304a915 100644
--- a/doc/pub/week40/html/._week40-bs045.html
+++ b/doc/pub/week40/html/._week40-bs045.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -320,46 +322,17 @@ MathJax.Hub.Config({
Mathematical model
-First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( z_i^1 \) of the input coordinates \( x_j \),
-
+The output \( y \) is produced via the activation function \( f \)
$$
-\begin{equation} z_i^1 = \sum_{j=1}^{M} w_{ij}^1 x_j + b_i^1
-\tag{7}
-\end{equation}
+ y = f\left(\sum_{i=1}^n w_ix_i + b_i\right) = f(z),
$$
-
-Here \( b_i \) is the so-called bias which is normally needed in
-case of zero activation weights or inputs. How to fix the biases and
-the weights will be discussed below. The value of \( z_i^1 \) is the
-argument to the activation function \( f_i \) of each node \( i \), The
-variable \( M \) stands for all possible inputs to a given node \( i \) in the
-first layer. We define the output \( y_i^1 \) of all neurons in layer 1 as
-
-$$
-\begin{equation}
- y_i^1 = f(z_i^1) = f\left(\sum_{j=1}^M w_{ij}^1 x_j + b_i^1\right)
-\tag{8}
-\end{equation}
-$$
-
-
-where we assume that all nodes in the same layer have identical
-activation functions, hence the notation \( f \). In general, we could assume in the more general case that different layers have different activation functions.
-In this case we would identify these functions with a superscript \( l \) for the \( l \)-th layer,
-
-$$
-\begin{equation}
- y_i^l = f^l(u_i^l) = f^l\left(\sum_{j=1}^{N_{l-1}} w_{ij}^l y_j^{l-1} + b_i^l\right)
-\tag{9}
-\end{equation}
-$$
-
-
-where \( N_l \) is the number of nodes in layer \( l \). When the output of
-all the nodes in the first hidden layer are computed, the values of
-the subsequent layer can be calculated and so forth until the output
-is obtained.
+This function receives \( x_i \) as inputs.
+Here the activation \( z=(\sum_{i=1}^n w_ix_i+b_i) \).
+In an FFNN of such neurons, the inputs \( x_i \) are the outputs of
+the neurons in the preceding layer. Furthermore, an MLP is
+fully-connected, which means that each neuron receives a weighted sum
+of the outputs of all neurons in the previous layer.
@@ -387,7 +360,7 @@ is obtained.
54
55
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs046.html b/doc/pub/week40/html/._week40-bs046.html
index b23d17a0b..e15d0450c 100644
--- a/doc/pub/week40/html/._week40-bs046.html
+++ b/doc/pub/week40/html/._week40-bs046.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -320,29 +322,47 @@ MathJax.Hub.Config({
Mathematical model
-The output of neuron \( i \) in layer 2 is thus,
+First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( z_i^1 \) of the input coordinates \( x_j \),
$$
-\begin{align}
- y_i^2 &= f^2\left(\sum_{j=1}^N w_{ij}^2 y_j^1 + b_i^2\right)
-\tag{10}\\
- &= f^2\left[\sum_{j=1}^N w_{ij}^2f^1\left(\sum_{k=1}^M w_{jk}^1 x_k + b_j^1\right) + b_i^2\right]
-\tag{11}
-\end{align}
+\begin{equation} z_i^1 = \sum_{j=1}^{M} w_{ij}^1 x_j + b_i^1
+\tag{7}
+\end{equation}
$$
-where we have substituted \( y_k^1 \) with the inputs \( x_k \). Finally, the ANN output reads
+
+Here \( b_i \) is the so-called bias which is normally needed in
+case of zero activation weights or inputs. How to fix the biases and
+the weights will be discussed below. The value of \( z_i^1 \) is the
+argument to the activation function \( f_i \) of each node \( i \), The
+variable \( M \) stands for all possible inputs to a given node \( i \) in the
+first layer. We define the output \( y_i^1 \) of all neurons in layer 1 as
$$
-\begin{align}
- y_i^3 &= f^3\left(\sum_{j=1}^N w_{ij}^3 y_j^2 + b_i^3\right)
-\tag{12}\\
- &= f_3\left[\sum_{j} w_{ij}^3 f^2\left(\sum_{k} w_{jk}^2 f^1\left(\sum_{m} w_{km}^1 x_m + b_k^1\right) + b_j^2\right)
- + b_1^3\right]
-\tag{13}
-\end{align}
+\begin{equation}
+ y_i^1 = f(z_i^1) = f\left(\sum_{j=1}^M w_{ij}^1 x_j + b_i^1\right)
+\tag{8}
+\end{equation}
$$
+
+where we assume that all nodes in the same layer have identical
+activation functions, hence the notation \( f \). In general, we could assume in the more general case that different layers have different activation functions.
+In this case we would identify these functions with a superscript \( l \) for the \( l \)-th layer,
+
+$$
+\begin{equation}
+ y_i^l = f^l(u_i^l) = f^l\left(\sum_{j=1}^{N_{l-1}} w_{ij}^l y_j^{l-1} + b_i^l\right)
+\tag{9}
+\end{equation}
+$$
+
+
+where \( N_l \) is the number of nodes in layer \( l \). When the output of
+all the nodes in the first hidden layer are computed, the values of
+the subsequent layer can be calculated and so forth until the output
+is obtained.
+
@@ -369,7 +389,7 @@ $$
55
56
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs047.html b/doc/pub/week40/html/._week40-bs047.html
index a6ef605c1..57141c9f1 100644
--- a/doc/pub/week40/html/._week40-bs047.html
+++ b/doc/pub/week40/html/._week40-bs047.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -320,19 +322,28 @@ MathJax.Hub.Config({
Mathematical model
-We can generalize this expression to an MLP with \( l \) hidden
-layers. The complete functional form is,
+The output of neuron \( i \) in layer 2 is thus,
$$
\begin{align}
-&y^{l+1}_i = f^{l+1}\left[\!\sum_{j=1}^{N_l} w_{ij}^3 f^l\left(\sum_{k=1}^{N_{l-1}}w_{jk}^{l-1}\left(\dots f^1\left(\sum_{n=1}^{N_0} w_{mn}^1 x_n+ b_m^1\right)\dots\right)+b_k^2\right)+b_1^3\right] &&
-\tag{14}
+ y_i^2 &= f^2\left(\sum_{j=1}^N w_{ij}^2 y_j^1 + b_i^2\right)
+\tag{10}\\
+ &= f^2\left[\sum_{j=1}^N w_{ij}^2f^1\left(\sum_{k=1}^M w_{jk}^1 x_k + b_j^1\right) + b_i^2\right]
+\tag{11}
\end{align}
$$
-
-which illustrates a basic property of MLPs: The only independent
-variables are the input values \( x_n \).
+where we have substituted \( y_k^1 \) with the inputs \( x_k \). Finally, the ANN output reads
+
+$$
+\begin{align}
+ y_i^3 &= f^3\left(\sum_{j=1}^N w_{ij}^3 y_j^2 + b_i^3\right)
+\tag{12}\\
+ &= f_3\left[\sum_{j} w_{ij}^3 f^2\left(\sum_{k} w_{jk}^2 f^1\left(\sum_{m} w_{km}^1 x_m + b_k^1\right) + b_j^2\right)
+ + b_1^3\right]
+\tag{13}
+\end{align}
+$$
@@ -360,7 +371,7 @@ variables are the input values \( x_n \).
56
57
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs048.html b/doc/pub/week40/html/._week40-bs048.html
index 59f3145c4..c3cb405c3 100644
--- a/doc/pub/week40/html/._week40-bs048.html
+++ b/doc/pub/week40/html/._week40-bs048.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -320,28 +322,19 @@ MathJax.Hub.Config({
Mathematical model
-This confirms that an MLP, despite its quite convoluted mathematical
-form, is nothing more than an analytic function, specifically a
-mapping of real-valued vectors \( \hat{x} \in \mathbb{R}^n \rightarrow
-\hat{y} \in \mathbb{R}^m \).
-
-
-Furthermore, the flexibility and universality of an MLP can be
-illustrated by realizing that the expression is essentially a nested
-sum of scaled activation functions of the form
+We can generalize this expression to an MLP with \( l \) hidden
+layers. The complete functional form is,
$$
-\begin{equation}
- f(x) = c_1 f(c_2 x + c_3) + c_4
-\tag{15}
-\end{equation}
+\begin{align}
+&y^{l+1}_i = f^{l+1}\left[\!\sum_{j=1}^{N_l} w_{ij}^3 f^l\left(\sum_{k=1}^{N_{l-1}}w_{jk}^{l-1}\left(\dots f^1\left(\sum_{n=1}^{N_0} w_{mn}^1 x_n+ b_m^1\right)\dots\right)+b_k^2\right)+b_1^3\right] &&
+\tag{14}
+\end{align}
$$
-where the parameters \( c_i \) are weights and biases. By adjusting these
-parameters, the activation functions can be shifted up and down or
-left and right, change slope or be rescaled which is the key to the
-flexibility of a neural network.
+which illustrates a basic property of MLPs: The only independent
+variables are the input values \( x_n \).
@@ -369,7 +362,7 @@ flexibility of a neural network.
57
58
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs049.html b/doc/pub/week40/html/._week40-bs049.html
index 40bb883f2..384602606 100644
--- a/doc/pub/week40/html/._week40-bs049.html
+++ b/doc/pub/week40/html/._week40-bs049.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,42 +319,32 @@ MathJax.Hub.Config({
-Matrix-vector notation
+Mathematical model
-We can introduce a more convenient notation for the activations in an A NN.
+This confirms that an MLP, despite its quite convoluted mathematical
+form, is nothing more than an analytic function, specifically a
+mapping of real-valued vectors \( \hat{x} \in \mathbb{R}^n \rightarrow
+\hat{y} \in \mathbb{R}^m \).
-Additionally, we can represent the biases and activations
-as layer-wise column vectors \( \hat{b}_l \) and \( \hat{y}_l \), so that the \( i \)-th element of each vector
-is the bias \( b_i^l \) and activation \( y_i^l \) of node \( i \) in layer \( l \) respectively.
+Furthermore, the flexibility and universality of an MLP can be
+illustrated by realizing that the expression is essentially a nested
+sum of scaled activation functions of the form
-
-We have that \( \mathrm{W}_l \) is an \( N_{l-1} \times N_l \) matrix, while \( \hat{b}_l \) and \( \hat{y}_l \) are \( N_l \times 1 \) column vectors.
-With this notation, the sum becomes a matrix-vector multiplication, and we can write
-the equation for the activations of hidden layer 2 (assuming three nodes for simplicity) as
$$
\begin{equation}
- \hat{y}_2 = f_2(\mathrm{W}_2 \hat{y}_{1} + \hat{b}_{2}) =
- f_2\left(\left[\begin{array}{ccc}
- w^2_{11} &w^2_{12} &w^2_{13} \\
- w^2_{21} &w^2_{22} &w^2_{23} \\
- w^2_{31} &w^2_{32} &w^2_{33} \\
- \end{array} \right] \cdot
- \left[\begin{array}{c}
- y^1_1 \\
- y^1_2 \\
- y^1_3 \\
- \end{array}\right] +
- \left[\begin{array}{c}
- b^2_1 \\
- b^2_2 \\
- b^2_3 \\
- \end{array}\right]\right).
-\tag{16}
+ f(x) = c_1 f(c_2 x + c_3) + c_4
+\tag{15}
\end{equation}
$$
+
+where the parameters \( c_i \) are weights and biases. By adjusting these
+parameters, the activation functions can be shifted up and down or
+left and right, change slope or be rescaled which is the key to the
+flexibility of a neural network.
+
@@ -379,7 +371,7 @@ $$
58
59
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs050.html b/doc/pub/week40/html/._week40-bs050.html
index 53bc94999..d9af18a2e 100644
--- a/doc/pub/week40/html/._week40-bs050.html
+++ b/doc/pub/week40/html/._week40-bs050.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,26 +319,42 @@ MathJax.Hub.Config({
-Matrix-vector notation and activation
+Matrix-vector notation
-The activation of node \( i \) in layer 2 is
+We can introduce a more convenient notation for the activations in an A NN.
+
+Additionally, we can represent the biases and activations
+as layer-wise column vectors \( \hat{b}_l \) and \( \hat{y}_l \), so that the \( i \)-th element of each vector
+is the bias \( b_i^l \) and activation \( y_i^l \) of node \( i \) in layer \( l \) respectively.
+
+
+We have that \( \mathrm{W}_l \) is an \( N_{l-1} \times N_l \) matrix, while \( \hat{b}_l \) and \( \hat{y}_l \) are \( N_l \times 1 \) column vectors.
+With this notation, the sum becomes a matrix-vector multiplication, and we can write
+the equation for the activations of hidden layer 2 (assuming three nodes for simplicity) as
$$
\begin{equation}
- y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) =
- f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right).
-\tag{17}
+ \hat{y}_2 = f_2(\mathrm{W}_2 \hat{y}_{1} + \hat{b}_{2}) =
+ f_2\left(\left[\begin{array}{ccc}
+ w^2_{11} &w^2_{12} &w^2_{13} \\
+ w^2_{21} &w^2_{22} &w^2_{23} \\
+ w^2_{31} &w^2_{32} &w^2_{33} \\
+ \end{array} \right] \cdot
+ \left[\begin{array}{c}
+ y^1_1 \\
+ y^1_2 \\
+ y^1_3 \\
+ \end{array}\right] +
+ \left[\begin{array}{c}
+ b^2_1 \\
+ b^2_2 \\
+ b^2_3 \\
+ \end{array}\right]\right).
+\tag{16}
\end{equation}
$$
-
-This is not just a convenient and compact notation, but also a useful
-and intuitive way to think about MLPs: The output is calculated by a
-series of matrix-vector multiplications and vector additions that are
-used as input to the activation functions. For each operation
-\( \mathrm{W}_l \hat{y}_{l-1} \) we move forward one layer.
-
@@ -363,7 +381,7 @@ used as input to the activation functions. For each operation
59
60
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs051.html b/doc/pub/week40/html/._week40-bs051.html
index 4c0ba376d..0d9716b32 100644
--- a/doc/pub/week40/html/._week40-bs051.html
+++ b/doc/pub/week40/html/._week40-bs051.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,21 +319,27 @@ MathJax.Hub.Config({
-Activation functions
+Matrix-vector notation and activation
-A property that characterizes a neural network, other than its
-connectivity, is the choice of activation function(s). As described
-in, the following restrictions are imposed on an activation function
-for a FFNN to fulfill the universal approximation theorem
+The activation of node \( i \) in layer 2 is
-
- - Non-constant
- - Bounded
- - Monotonically-increasing
- - Continuous
-
+$$
+\begin{equation}
+ y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) =
+ f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right).
+\tag{17}
+\end{equation}
+$$
+
+This is not just a convenient and compact notation, but also a useful
+and intuitive way to think about MLPs: The output is calculated by a
+series of matrix-vector multiplications and vector additions that are
+used as input to the activation functions. For each operation
+\( \mathrm{W}_l \hat{y}_{l-1} \) we move forward one layer.
+
+
diff --git a/doc/pub/week40/html/._week40-bs052.html b/doc/pub/week40/html/._week40-bs052.html
index 437439754..cad2e326f 100644
--- a/doc/pub/week40/html/._week40-bs052.html
+++ b/doc/pub/week40/html/._week40-bs052.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,29 +319,21 @@ MathJax.Hub.Config({
-Activation functions, Logistic and Hyperbolic ones
+Activation functions
-The second requirement excludes all linear functions. Furthermore, in
-a MLP with only linear activation functions, each layer simply
-performs a linear transformation of its inputs.
+A property that characterizes a neural network, other than its
+connectivity, is the choice of activation function(s). As described
+in, the following restrictions are imposed on an activation function
+for a FFNN to fulfill the universal approximation theorem
-
-Regardless of the number of layers, the output of the NN will be
-nothing but a linear function of the inputs. Thus we need to introduce
-some kind of non-linearity to the NN to be able to fit non-linear
-functions Typical examples are the logistic Sigmoid
+
+ - Non-constant
+ - Bounded
+ - Monotonically-increasing
+ - Continuous
+
-$$
- f(x) = \frac{1}{1 + e^{-x}},
-$$
-
-and the hyperbolic tangent function
-$$
- f(x) = \tanh(x)
-$$
-
-
diff --git a/doc/pub/week40/html/._week40-bs053.html b/doc/pub/week40/html/._week40-bs053.html
index 1aca87020..85f3c6505 100644
--- a/doc/pub/week40/html/._week40-bs053.html
+++ b/doc/pub/week40/html/._week40-bs053.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,90 +319,28 @@ MathJax.Hub.Config({
-Relevance
+Activation functions, Logistic and Hyperbolic ones
-The sigmoid function are more biologically plausible because the
-output of inactive neurons are zero. Such activation function are
-called one-sided. However, it has been shown that the hyperbolic
-tangent performs better than the sigmoid for training MLPs. has
-become the most popular for deep neural networks
+The second requirement excludes all linear functions. Furthermore, in
+a MLP with only linear activation functions, each layer simply
+performs a linear transformation of its inputs.
+Regardless of the number of layers, the output of the NN will be
+nothing but a linear function of the inputs. Thus we need to introduce
+some kind of non-linearity to the NN to be able to fit non-linear
+functions Typical examples are the logistic Sigmoid
-
-
"""The sigmoid function (or the logistic curve) is a
-function that takes any real number, z, and outputs a number (0,1).
-It is useful in neural networks for assigning weights on a relative scale.
-The value z is the weighted sum of parameters involved in the learning algorithm."""
+$$
+ f(x) = \frac{1}{1 + e^{-x}},
+$$
-import numpy
-import matplotlib.pyplot as plt
-import math as mt
+and the hyperbolic tangent function
+$$
+ f(x) = \tanh(x)
+$$
-z = numpy.arange(-5, 5, .1)
-sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))
-sigma = sigma_fn(z)
-
-fig = plt.figure()
-ax = fig.add_subplot(111)
-ax.plot(z, sigma)
-ax.set_ylim([-0.1, 1.1])
-ax.set_xlim([-5,5])
-ax.grid(True)
-ax.set_xlabel('z')
-ax.set_title('sigmoid function')
-
-plt.show()
-
-"""Step Function"""
-z = numpy.arange(-5, 5, .02)
-step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)
-step = step_fn(z)
-
-fig = plt.figure()
-ax = fig.add_subplot(111)
-ax.plot(z, step)
-ax.set_ylim([-0.5, 1.5])
-ax.set_xlim([-5,5])
-ax.grid(True)
-ax.set_xlabel('z')
-ax.set_title('step function')
-
-plt.show()
-
-"""Sine Function"""
-z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)
-t = numpy.sin(z)
-
-fig = plt.figure()
-ax = fig.add_subplot(111)
-ax.plot(z, t)
-ax.set_ylim([-1.0, 1.0])
-ax.set_xlim([-2*mt.pi,2*mt.pi])
-ax.grid(True)
-ax.set_xlabel('z')
-ax.set_title('sine function')
-
-plt.show()
-
-"""Plots a graph of the squashing function used by a rectified linear
-unit"""
-z = numpy.arange(-2, 2, .1)
-zero = numpy.zeros(len(z))
-y = numpy.max([zero, z], axis=0)
-
-fig = plt.figure()
-ax = fig.add_subplot(111)
-ax.plot(z, y)
-ax.set_ylim([-2.0, 2.0])
-ax.set_xlim([-2.0, 2.0])
-ax.grid(True)
-ax.set_xlabel('z')
-ax.set_title('Rectified linear unit')
-
-plt.show()
-
@@ -427,7 +367,7 @@ plt.show()
62
63
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs054.html b/doc/pub/week40/html/._week40-bs054.html
index 3ad7b914c..cfcdd1cbe 100644
--- a/doc/pub/week40/html/._week40-bs054.html
+++ b/doc/pub/week40/html/._week40-bs054.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,41 +319,90 @@ MathJax.Hub.Config({
-The multilayer perceptron (MLP)
+Relevance
-The multilayer perceptron is a very popular, and easy to implement approach, to deep learning. It consists of
-
-
-- A neural network with one or more layers of nodes between the input and the output nodes.
-- The multilayer network structure, or architecture, or topology, consists of an input layer, one or more hidden layers, and one output layer.
-- The input nodes pass values to the first hidden layer, its nodes pass the information on to the second and so on till we reach the output layer.
-
-
-As a convention it is normal to call a network with one layer of input units, one layer of hidden
-units and one layer of output units as a two-layer network. A network with two layers of hidden units is called a three-layer network etc etc.
+The sigmoid function are more biologically plausible because the
+output of inactive neurons are zero. Such activation function are
+called one-sided. However, it has been shown that the hyperbolic
+tangent performs better than the sigmoid for training MLPs. has
+become the most popular for deep neural networks
-For an MLP network there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.
-Hereafter we will call the various entities of a layer for nodes.
-There are also no connections within a single layer.
-
-The number of input nodes does not need to equal the number of output
-nodes. This applies also to the hidden layers. Each layer may have its
-own number of nodes and activation functions.
+
+
"""The sigmoid function (or the logistic curve) is a
+function that takes any real number, z, and outputs a number (0,1).
+It is useful in neural networks for assigning weights on a relative scale.
+The value z is the weighted sum of parameters involved in the learning algorithm."""
-
-The hidden layers have their name from the fact that they are not
-linked to observables and as we will see below when we define the
-so-called activation \( \hat{z} \), we can think of this as a basis
-expansion of the original inputs \( \hat{x} \). The difference however
-between neural networks and say linear regression is that now these
-basis functions (which will correspond to the weights in the network)
-are learned from data. This results in an important difference between
-neural networks and deep learning approaches on one side and methods
-like logistic regression or linear regression and their modifications on the other side.
+import numpy
+import matplotlib.pyplot as plt
+import math as mt
+z = numpy.arange(-5, 5, .1)
+sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))
+sigma = sigma_fn(z)
+
+fig = plt.figure()
+ax = fig.add_subplot(111)
+ax.plot(z, sigma)
+ax.set_ylim([-0.1, 1.1])
+ax.set_xlim([-5,5])
+ax.grid(True)
+ax.set_xlabel('z')
+ax.set_title('sigmoid function')
+
+plt.show()
+
+"""Step Function"""
+z = numpy.arange(-5, 5, .02)
+step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)
+step = step_fn(z)
+
+fig = plt.figure()
+ax = fig.add_subplot(111)
+ax.plot(z, step)
+ax.set_ylim([-0.5, 1.5])
+ax.set_xlim([-5,5])
+ax.grid(True)
+ax.set_xlabel('z')
+ax.set_title('step function')
+
+plt.show()
+
+"""Sine Function"""
+z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)
+t = numpy.sin(z)
+
+fig = plt.figure()
+ax = fig.add_subplot(111)
+ax.plot(z, t)
+ax.set_ylim([-1.0, 1.0])
+ax.set_xlim([-2*mt.pi,2*mt.pi])
+ax.grid(True)
+ax.set_xlabel('z')
+ax.set_title('sine function')
+
+plt.show()
+
+"""Plots a graph of the squashing function used by a rectified linear
+unit"""
+z = numpy.arange(-2, 2, .1)
+zero = numpy.zeros(len(z))
+y = numpy.max([zero, z], axis=0)
+
+fig = plt.figure()
+ax = fig.add_subplot(111)
+ax.plot(z, y)
+ax.set_ylim([-2.0, 2.0])
+ax.set_xlim([-2.0, 2.0])
+ax.grid(True)
+ax.set_xlabel('z')
+ax.set_title('Rectified linear unit')
+
+plt.show()
+
@@ -378,7 +429,7 @@ like logistic regression or linear regression and their modifications on the oth
63
64
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs055.html b/doc/pub/week40/html/._week40-bs055.html
index 49a427d66..b901bedb5 100644
--- a/doc/pub/week40/html/._week40-bs055.html
+++ b/doc/pub/week40/html/._week40-bs055.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,30 +319,40 @@ MathJax.Hub.Config({
-From one to many layers, the universal approximation theorem
+The multilayer perceptron (MLP)
-A neural network with only one layer, what we called the simple
-perceptron, is best suited if we have a standard binary model with
-clear (linear) boundaries between the outcomes. As such it could
-equally well be replaced by standard linear regression or logistic
-regression. Networks with one or more hidden layers approximate
-systems with more complex boundaries.
+The multilayer perceptron is a very popular, and easy to implement approach, to deep learning. It consists of
+
+
+- A neural network with one or more layers of nodes between the input and the output nodes.
+- The multilayer network structure, or architecture, or topology, consists of an input layer, one or more hidden layers, and one output layer.
+- The input nodes pass values to the first hidden layer, its nodes pass the information on to the second and so on till we reach the output layer.
+
+
+As a convention it is normal to call a network with one layer of input units, one layer of hidden
+units and one layer of output units as a two-layer network. A network with two layers of hidden units is called a three-layer network etc etc.
-As stated earlier,
-an important theorem in studies of neural networks, restated without
-proof here, is the universal approximation
-theorem.
+For an MLP network there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units.
+Hereafter we will call the various entities of a layer for nodes.
+There are also no connections within a single layer.
-It states that a feed-forward network with a single hidden layer
-containing a finite number of neurons can approximate continuous
-functions on compact subsets of real functions. The theorem thus
-states that simple neural networks can represent a wide variety of
-interesting functions when given appropriate parameters. It is the
-multilayer feedforward architecture itself which gives neural networks
-the potential of being universal approximators.
+The number of input nodes does not need to equal the number of output
+nodes. This applies also to the hidden layers. Each layer may have its
+own number of nodes and activation functions.
+
+
+The hidden layers have their name from the fact that they are not
+linked to observables and as we will see below when we define the
+so-called activation \( \hat{z} \), we can think of this as a basis
+expansion of the original inputs \( \hat{x} \). The difference however
+between neural networks and say linear regression is that now these
+basis functions (which will correspond to the weights in the network)
+are learned from data. This results in an important difference between
+neural networks and deep learning approaches on one side and methods
+like logistic regression or linear regression and their modifications on the other side.
@@ -367,6 +379,8 @@ the potential of being universal approximators.
63
64
65
+ ...
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs056.html b/doc/pub/week40/html/._week40-bs056.html
index bbe15a57a..7b9585409 100644
--- a/doc/pub/week40/html/._week40-bs056.html
+++ b/doc/pub/week40/html/._week40-bs056.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,33 +319,30 @@ MathJax.Hub.Config({
-Deriving the back propagation code for a multilayer perceptron model
+From one to many layers, the universal approximation theorem
-As we have seen now in a feed forward network, we can express the final output of our network in terms of basic matrix-vector multiplications.
-The unknowwn quantities are our weights \( w_{ij} \) and we need to find an algorithm for changing them so that our errors are as small as possible.
-This leads us to the famous back propagation algorithm.
+A neural network with only one layer, what we called the simple
+perceptron, is best suited if we have a standard binary model with
+clear (linear) boundaries between the outcomes. As such it could
+equally well be replaced by standard linear regression or logistic
+regression. Networks with one or more hidden layers approximate
+systems with more complex boundaries.
-The questions we want to ask are how do changes in the biases and the
-weights in our network change the cost function and how can we use the
-final output to modify the weights?
+As stated earlier,
+an important theorem in studies of neural networks, restated without
+proof here, is the universal approximation
+theorem.
-To derive these equations let us start with a plain regression problem
-and define our cost function as
-
-$$
-{\cal C}(\hat{W}) = \frac{1}{2}\sum_{i=1}^n\left(y_i - t_i\right)^2,
-$$
-
-
-where the $t_i$s are our \( n \) targets (the values we want to
-reproduce), while the outputs of the network after having propagated
-all inputs \( \hat{x} \) are given by \( y_i \). Below we will demonstrate
-how the basic equations arising from the back propagation algorithm
-can be modified in order to study classification problems with \( K \)
-classes.
+It states that a feed-forward network with a single hidden layer
+containing a finite number of neurons can approximate continuous
+functions on compact subsets of real functions. The theorem thus
+states that simple neural networks can represent a wide variety of
+interesting functions when given appropriate parameters. It is the
+multilayer feedforward architecture itself which gives neural networks
+the potential of being universal approximators.
@@ -369,6 +368,7 @@ classes.
63
64
65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs057.html b/doc/pub/week40/html/._week40-bs057.html
index 3394857f0..9a5fcc203 100644
--- a/doc/pub/week40/html/._week40-bs057.html
+++ b/doc/pub/week40/html/._week40-bs057.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,40 +319,33 @@ MathJax.Hub.Config({
-Definitions
+Deriving the back propagation code for a multilayer perceptron model
-With our definition of the targets \( \hat{t} \), the outputs of the
-network \( \hat{y} \) and the inputs \( \hat{x} \) we
-define now the activation \( z_j^l \) of node/neuron/unit \( j \) of the
-\( l \)-th layer as a function of the bias, the weights which add up from
-the previous layer \( l-1 \) and the forward passes/outputs
-\( \hat{a}^{l-1} \) from the previous layer as
+As we have seen now in a feed forward network, we can express the final output of our network in terms of basic matrix-vector multiplications.
+The unknowwn quantities are our weights \( w_{ij} \) and we need to find an algorithm for changing them so that our errors are as small as possible.
+This leads us to the famous back propagation algorithm.
+
+
+The questions we want to ask are how do changes in the biases and the
+weights in our network change the cost function and how can we use the
+final output to modify the weights?
+
+
+To derive these equations let us start with a plain regression problem
+and define our cost function as
$$
-z_j^l = \sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l,
+{\cal C}(\hat{W}) = \frac{1}{2}\sum_{i=1}^n\left(y_i - t_i\right)^2,
$$
-where \( b_k^l \) are the biases from layer \( l \). Here \( M_{l-1} \)
-represents the total number of nodes/neurons/units of layer \( l-1 \). The
-figure here illustrates this equation. We can rewrite this in a more
-compact form as the matrix-vector products we discussed earlier,
-
-$$
-\hat{z}^l = \left(\hat{W}^l\right)^T\hat{a}^{l-1}+\hat{b}^l.
-$$
-
-
-With the activation values \( \hat{z}^l \) we can in turn define the
-output of layer \( l \) as \( \hat{a}^l = f(\hat{z}^l) \) where \( f \) is our
-activation function. In the examples here we will use the sigmoid
-function discussed in our logistic regression lectures. We will also use the same activation function \( f \) for all layers
-and their nodes. It means we have
-
-$$
-a_j^l = f(z_j^l) = \frac{1}{1+\exp{-(z_j^l)}}.
-$$
+where the $t_i$s are our \( n \) targets (the values we want to
+reproduce), while the outputs of the network after having propagated
+all inputs \( \hat{x} \) are given by \( y_i \). Below we will demonstrate
+how the basic equations arising from the back propagation algorithm
+can be modified in order to study classification problems with \( K \)
+classes.
@@ -375,6 +370,7 @@ $$
63
64
65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs058.html b/doc/pub/week40/html/._week40-bs058.html
index e73f951e7..189ca1fbb 100644
--- a/doc/pub/week40/html/._week40-bs058.html
+++ b/doc/pub/week40/html/._week40-bs058.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,23 +319,39 @@ MathJax.Hub.Config({
-Derivatives and the chain rule
+Definitions
-From the definition of the activation \( z_j^l \) we have
-$$
-\frac{\partial z_j^l}{\partial w_{ij}^l} = a_i^{l-1},
-$$
+With our definition of the targets \( \hat{t} \), the outputs of the
+network \( \hat{y} \) and the inputs \( \hat{x} \) we
+define now the activation \( z_j^l \) of node/neuron/unit \( j \) of the
+\( l \)-th layer as a function of the bias, the weights which add up from
+the previous layer \( l-1 \) and the forward passes/outputs
+\( \hat{a}^{l-1} \) from the previous layer as
-and
$$
-\frac{\partial z_j^l}{\partial a_i^{l-1}} = w_{ji}^l.
+z_j^l = \sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l,
$$
-With our definition of the activation function we have that (note that this function depends only on \( z_j^l \))
+where \( b_k^l \) are the biases from layer \( l \). Here \( M_{l-1} \)
+represents the total number of nodes/neurons/units of layer \( l-1 \). The
+figure here illustrates this equation. We can rewrite this in a more
+compact form as the matrix-vector products we discussed earlier,
+
$$
-\frac{\partial a_j^l}{\partial z_j^{l}} = a_j^l(1-a_j^l)=f(z_j^l)(1-f(z_j^l)).
+\hat{z}^l = \left(\hat{W}^l\right)^T\hat{a}^{l-1}+\hat{b}^l.
+$$
+
+
+With the activation values \( \hat{z}^l \) we can in turn define the
+output of layer \( l \) as \( \hat{a}^l = f(\hat{z}^l) \) where \( f \) is our
+activation function. In the examples here we will use the sigmoid
+function discussed in our logistic regression lectures. We will also use the same activation function \( f \) for all layers
+and their nodes. It means we have
+
+$$
+a_j^l = f(z_j^l) = \frac{1}{1+\exp{-(z_j^l)}}.
$$
@@ -358,6 +376,7 @@ $$
63
64
65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs059.html b/doc/pub/week40/html/._week40-bs059.html
index a0b81b7ef..815d7bc3c 100644
--- a/doc/pub/week40/html/._week40-bs059.html
+++ b/doc/pub/week40/html/._week40-bs059.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,26 +319,23 @@ MathJax.Hub.Config({
-Derivative of the cost function
+Derivatives and the chain rule
-With these definitions we can now compute the derivative of the cost function in terms of the weights.
+From the definition of the activation \( z_j^l \) we have
+$$
+\frac{\partial z_j^l}{\partial w_{ij}^l} = a_i^{l-1},
+$$
+
+and
+$$
+\frac{\partial z_j^l}{\partial a_i^{l-1}} = w_{ji}^l.
+$$
-Let us specialize to the output layer \( l=L \). Our cost function is
+With our definition of the activation function we have that (note that this function depends only on \( z_j^l \))
$$
-{\cal C}(\hat{W^L}) = \frac{1}{2}\sum_{i=1}^n\left(y_i - t_i\right)^2=\frac{1}{2}\sum_{i=1}^n\left(a_i^L - t_i\right)^2,
-$$
-
-The derivative of this function with respect to the weights is
-
-$$
-\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L} = \left(a_j^L - t_j\right)\frac{\partial a_j^L}{\partial w_{jk}^{L}},
-$$
-
-The last partial derivative can easily be computed and reads (by applying the chain rule)
-$$
-\frac{\partial a_j^L}{\partial w_{jk}^{L}} = \frac{\partial a_j^L}{\partial z_{j}^{L}}\frac{\partial z_j^L}{\partial w_{jk}^{L}}=a_j^L(1-a_j^L)a_k^{L-1},
+\frac{\partial a_j^l}{\partial z_j^{l}} = a_j^l(1-a_j^l)=f(z_j^l)(1-f(z_j^l)).
$$
@@ -360,6 +359,7 @@ $$
63
64
65
+ 66
»
diff --git a/doc/pub/week40/html/._week40-bs060.html b/doc/pub/week40/html/._week40-bs060.html
index a081a5ff6..03c2aa9e9 100644
--- a/doc/pub/week40/html/._week40-bs060.html
+++ b/doc/pub/week40/html/._week40-bs060.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -317,51 +319,26 @@ MathJax.Hub.Config({
-Bringing it together, first back propagation equation
+Derivative of the cost function
-We have thus
-$$
-\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L} = \left(a_j^L - t_j\right)a_j^L(1-a_j^L)a_k^{L-1},
-$$
+With these definitions we can now compute the derivative of the cost function in terms of the weights.
-Defining
+Let us specialize to the output layer \( l=L \). Our cost function is
$$
-\delta_j^L = a_j^L(1-a_j^L)\left(a_j^L - t_j\right) = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)},
+{\cal C}(\hat{W^L}) = \frac{1}{2}\sum_{i=1}^n\left(y_i - t_i\right)^2=\frac{1}{2}\sum_{i=1}^n\left(a_i^L - t_i\right)^2,
$$
-and using the Hadamard product of two vectors we can write this as
-$$
-\hat{\delta}^L = f'(\hat{z}^L)\circ\frac{\partial {\cal C}}{\partial (\hat{a}^L)}.
-$$
-
-
-This is an important expression. The second term on the right handside
-measures how fast the cost function is changing as a function of the $j$th
-output activation. If, for example, the cost function doesn't depend
-much on a particular output node \( j \), then \( \delta_j^L \) will be small,
-which is what we would expect. The first term on the right, measures
-how fast the activation function \( f \) is changing at a given activation
-value \( z_j^L \).
-
-
-Notice that everything in the above equations is easily computed. In
-particular, we compute \( z_j^L \) while computing the behaviour of the
-network, and it is only a small additional overhead to compute
-\( f'(z^L_j) \). The exact form of the derivative with respect to the
-output depends on the form of the cost function.
-However, provided the cost function is known there should be little
-trouble in calculating
+The derivative of this function with respect to the weights is
$$
-\frac{\partial {\cal C}}{\partial (a_j^L)}
+\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L} = \left(a_j^L - t_j\right)\frac{\partial a_j^L}{\partial w_{jk}^{L}},
$$
-
-With the definition of \( \delta_j^L \) we have a more compact definition of the derivative of the cost function in terms of the weights, namely
+The last partial derivative can easily be computed and reads (by applying the chain rule)
$$
-\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L} = \delta_j^La_k^{L-1}.
+\frac{\partial a_j^L}{\partial w_{jk}^{L}} = \frac{\partial a_j^L}{\partial z_{j}^{L}}\frac{\partial z_j^L}{\partial w_{jk}^{L}}=a_j^L(1-a_j^L)a_k^{L-1},
$$
@@ -384,6 +361,7 @@ $$
63
64
65
+ 66
»
diff --git a/doc/pub/week40/html/week40-bs.html b/doc/pub/week40/html/week40-bs.html
index 9a5172107..62292850d 100644
--- a/doc/pub/week40/html/week40-bs.html
+++ b/doc/pub/week40/html/week40-bs.html
@@ -146,6 +146,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -281,27 +282,28 @@ MathJax.Hub.Config({
Illustration of a single perceptropn model and a multi-perceptron model
Examples of XOR, OR and AND gates
Does Logistic Regression do a better Job?
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Mathematical model
- Matrix-vector notation
- Matrix-vector notation and activation
- Activation functions
- Activation functions, Logistic and Hyperbolic ones
- Relevance
- The multilayer perceptron (MLP)
- From one to many layers, the universal approximation theorem
- Deriving the back propagation code for a multilayer perceptron model
- Definitions
- Derivatives and the chain rule
- Derivative of the cost function
- Bringing it together, first back propagation equation
- Derivatives in terms of \( z_j^L \)
- Bringing it together
- Final back propagating equation
- Setting up the Back propagation algorithm
+ Adding Neural Networks
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Mathematical model
+ Matrix-vector notation
+ Matrix-vector notation and activation
+ Activation functions
+ Activation functions, Logistic and Hyperbolic ones
+ Relevance
+ The multilayer perceptron (MLP)
+ From one to many layers, the universal approximation theorem
+ Deriving the back propagation code for a multilayer perceptron model
+ Definitions
+ Derivatives and the chain rule
+ Derivative of the cost function
+ Bringing it together, first back propagation equation
+ Derivatives in terms of \( z_j^L \)
+ Bringing it together
+ Final back propagating equation
+ Setting up the Back propagation algorithm
@@ -360,7 +362,7 @@ MathJax.Hub.Config({
9
10
...
- 65
+ 66
»
diff --git a/doc/pub/week40/html/week40-reveal.html b/doc/pub/week40/html/week40-reveal.html
index ec49e4af9..32a4d35b8 100644
--- a/doc/pub/week40/html/week40-reveal.html
+++ b/doc/pub/week40/html/week40-reveal.html
@@ -1544,6 +1544,24 @@ Not exactly impressive, but somewhat better.
+
+Adding Neural Networks
+
+
+
+
+
# and now neural networks with Scikit-Learn and the XOR
+
+from sklearn.neural_network import MLPClassifier
+from sklearn.datasets import make_classification
+X, yXOR = make_classification(n_samples=100, random_state=1)
+FFNN = MLPClassifier(random_state=1, max_iter=300).fit(X, yXOR)
+FFNN.predict_proba(X)
+print(f"Test set accuracy with Feed Forward Neural Network for XOR gate:{FFNN.score(X, yXOR)}")
+Mathematical model
diff --git a/doc/pub/week40/html/week40-solarized.html b/doc/pub/week40/html/week40-solarized.html
index e701356d7..bfcf09ac9 100644
--- a/doc/pub/week40/html/week40-solarized.html
+++ b/doc/pub/week40/html/week40-solarized.html
@@ -166,6 +166,7 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -1605,6 +1606,23 @@ Not exactly impressive, but somewhat better.
+
Adding Neural Networks
+
+
+
+
+
# and now neural networks with Scikit-Learn and the XOR
+
+from sklearn.neural_network import MLPClassifier
+from sklearn.datasets import make_classification
+X, yXOR = make_classification(n_samples=100, random_state=1)
+FFNN = MLPClassifier(random_state=1, max_iter=300).fit(X, yXOR)
+FFNN.predict_proba(X)
+print(f"Test set accuracy with Feed Forward Neural Network for XOR gate:{FFNN.score(X, yXOR)}")
+
+
+
Mathematical model
diff --git a/doc/pub/week40/html/week40.html b/doc/pub/week40/html/week40.html
index 7001ea050..8cdb7fdf3 100644
--- a/doc/pub/week40/html/week40.html
+++ b/doc/pub/week40/html/week40.html
@@ -171,6 +171,7 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'does-logistic-regression-do-a-better-job'),
+ ('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
('Mathematical model', 2, None, 'mathematical-model'),
@@ -1610,6 +1611,23 @@ Not exactly impressive, but somewhat better.
+
Adding Neural Networks
+
+
+
+
+
# and now neural networks with Scikit-Learn and the XOR
+
+from sklearn.neural_network import MLPClassifier
+from sklearn.datasets import make_classification
+X, yXOR = make_classification(n_samples=100, random_state=1)
+FFNN = MLPClassifier(random_state=1, max_iter=300).fit(X, yXOR)
+FFNN.predict_proba(X)
+print(f"Test set accuracy with Feed Forward Neural Network for XOR gate:{FFNN.score(X, yXOR)}")
+
+
+
Mathematical model
diff --git a/doc/pub/week40/ipynb/ipynb-week40-src.tar.gz b/doc/pub/week40/ipynb/ipynb-week40-src.tar.gz
index 617f08e7f..506cea50c 100644
Binary files a/doc/pub/week40/ipynb/ipynb-week40-src.tar.gz and b/doc/pub/week40/ipynb/ipynb-week40-src.tar.gz differ
diff --git a/doc/pub/week40/ipynb/week40.ipynb b/doc/pub/week40/ipynb/week40.ipynb
index 55eac8f6c..e1f728064 100644
--- a/doc/pub/week40/ipynb/week40.ipynb
+++ b/doc/pub/week40/ipynb/week40.ipynb
@@ -1661,9 +1661,33 @@
"source": [
"Not exactly impressive, but somewhat better.\n",
"\n",
+ "## Adding Neural Networks"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
+ "source": [
"\n",
+ "# and now neural networks with Scikit-Learn and the XOR\n",
"\n",
- "\n",
+ "from sklearn.neural_network import MLPClassifier\n",
+ "from sklearn.datasets import make_classification\n",
+ "X, yXOR = make_classification(n_samples=100, random_state=1)\n",
+ "FFNN = MLPClassifier(random_state=1, max_iter=300).fit(X, yXOR)\n",
+ "FFNN.predict_proba(X)\n",
+ "print(f\"Test set accuracy with Feed Forward Neural Network for XOR gate:{FFNN.score(X, yXOR)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
"## Mathematical model\n",
"\n",
"The output $y$ is produced via the activation function $f$"
diff --git a/doc/src/week40/week40.do.txt b/doc/src/week40/week40.do.txt
index cab7d74b7..d6de94ac3 100644
--- a/doc/src/week40/week40.do.txt
+++ b/doc/src/week40/week40.do.txt
@@ -1200,6 +1200,21 @@ print("Test set accuracy with Logistic Regression for AND gate: {:.2f}".format(l
Not exactly impressive, but somewhat better.
+!split
+===== Adding Neural Networks =====
+
+!bc pycod
+
+# and now neural networks with Scikit-Learn and the XOR
+
+from sklearn.neural_network import MLPClassifier
+from sklearn.datasets import make_classification
+X, yXOR = make_classification(n_samples=100, random_state=1)
+FFNN = MLPClassifier(random_state=1, max_iter=300).fit(X, yXOR)
+FFNN.predict_proba(X)
+print(f"Test set accuracy with Feed Forward Neural Network for XOR gate:{FFNN.score(X, yXOR)}")
+
+!ec