updating exercises

doc/src/week43/exercisesweek43.do.txt

@@ -89,6 +89,59 @@ o Train the network and study your results and compare with results obtained eit
 
 Everything you develop here can be used directly into the code for the project.
 
+
+!split
+=====  Setting up dimensionalities by hand =====
+
+It can be useful to test the dimensionalities for the network.  Let us assume we have performed an optimization for XOR gate and found that the weights for the hidden layer are given by
+!bt
+\bm{W_h}=\begin{bmatrix} 1 & 1 \\
+                       1 & 1 \end{bmatrix},
+!et
+
+Multiplying $\bm{X}$ and $\bm{W}$ gives
+
+!bt
+\bm{X}{W}_h=\begin{bmatrix} 0 & 0 \\
+                       1 & 1 \\
+		       1 & 1 \\
+		       2 & 2 \end{bmatrix},
+!et
+Assume also that the bias vector for the hidden layer is
+!bt
+\bm{b}_h=\begin{bmatrix} 0 \\
+                       -1\end{bmatrix},
+!et
+Adding it gives us the input to the activation function of the hidden layer
+!bt
+\bm{z}_h=\bm{X}\bm{W}_h+\bm{b}_h=\begin{bmatrix} 0 & -1 \\
+                       1 & 0 \\
+		       1 & 0 \\
+		       2 & 1 \end{bmatrix},
+!et
+
+Let us then assume that our activation function is the RELU function, which simply means that we take the max of $0$ and the elements of the input argument $\bm{z}_h$, that is we have
+!bt
+\bm{a}_h=\mathrm{RELU}(\bm{z}_h=\bm{X}\bm{W}_h+\bm{b}_h)=\begin{bmatrix} 0 & 0 \\
+                       1 & 0 \\
+		       1 & 0 \\
+		       2 & 1 \end{bmatrix},
+!et
+Assume also that the bias of the output layer is zero and that the weights of the output layer are
+!bt
+\bm{w}_o=\begin{bmatrix} 1 \\
+                       -2\end{bmatrix},
+!et
+and multiplying with $\bm{a}_h$ gives the output
+!bt
+\bm{a}_o=\bm{w}_h^T\begin{bmatrix} 0 & 0 \\
+                       1 & 0 \\
+		       1 & 0 \\
+		       2 & 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix},
+!et
+the wanted result.
+
+
 !split
 ===== Setting up the Neural Network =====
 
@@ -120,10 +173,6 @@ def feed_forward(X):
     probabilities = sigmoid(z_o)
     return probabilities
 
-# we obtain a prediction by taking the class with the highest likelihood
-def predict(X):
-    probabilities = feed_forward(X)
-    return np.argmax(probabilities, axis=1)
 
 # ensure the same random numbers appear every time
 np.random.seed(0)
@@ -141,7 +190,7 @@ yAND = np.array( [ 0, 0 ,0, 1])
 # Defining the neural network
 n_inputs, n_features = X.shape
 n_hidden_neurons = 2
-n_categories = 2
+n_categories = 1
 n_features = 2
 
 # we make the weights normally distributed using numpy.random.randn
@@ -157,10 +206,6 @@ output_bias = np.zeros(n_categories) + 0.01
 probabilities = feed_forward(X)
 print(probabilities)
 
-
-predictions = predict(X)
-print(predictions)
-
 !ec
 
 Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above.
@@ -190,10 +235,7 @@ yOR = np.array( [ 0, 1 ,1, 1])
 yAND = np.array( [ 0, 0 ,0, 1])
 
 # Defining the neural network
-n_inputs, n_features = X.shape
 n_hidden_neurons = 2
-n_categories = 2
-n_features = 2
 
 eta_vals = np.logspace(-5, 1, 7)
 lmbd_vals = np.logspace(-5, 1, 7)
@@ -1217,3 +1259,25 @@ scores = multiclass.fit(X, target, scheduler, epochs=1000)
 
 !ec
 
+
+
+!split
+===== Testing the XOR gate and other gates =====
+
+Let us now use our code to test the XOR gate.
+
+!bc pycod
+X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)
+
+# The XOR gate
+yXOR = np.array( [[ 0], [1] ,[1], [0]])
+
+input_nodes = X.shape[1]
+output_nodes = 1
+
+logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)
+logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights
+scheduler = Adam(eta=1e-1, rho=0.9, rho2=0.999)
+scores = logistic_regression.fit(X, yXOR, scheduler, epochs=1000)
+!ec
+Not bad, but the results depend strongly on the learning reate. Try different learning rates.