-
diff --git a/doc/pub/Splines/html/._Splines-bs001.html b/doc/pub/Splines/html/._Splines-bs001.html index 8eda56d7d..3d3be0f19 100644 --- a/doc/pub/Splines/html/._Splines-bs001.html +++ b/doc/pub/Splines/html/._Splines-bs001.html @@ -80,19 +80,19 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec25'), - ('Stochastic Gradient Descent', 2, None, '___sec26'), - ('Computation of gradients', 2, None, '___sec27'), - ('SGD example', 2, None, '___sec28'), - ('The gradient step', 2, None, '___sec29'), - ('Simple example code', 2, None, '___sec30'), - ('When do we stop?', 2, None, '___sec31'), - ('Slightly different approach', 2, None, '___sec32'), - ('Program for stochastic gradient', 2, None, '___sec33'), - ('Logistic Regression example', 2, None, '___sec34'), ('Using gradient descent methods, limitations', 2, None, - '___sec35'), + '___sec26'), + ('Stochastic Gradient Descent', 2, None, '___sec27'), + ('Computation of gradients', 2, None, '___sec28'), + ('SGD example', 2, None, '___sec29'), + ('The gradient step', 2, None, '___sec30'), + ('Simple example code', 2, None, '___sec31'), + ('When do we stop?', 2, None, '___sec32'), + ('Slightly different approach', 2, None, '___sec33'), + ('Program for stochastic gradient', 2, None, '___sec34'), + ('Logistic Regression example', 2, None, '___sec35'), ('Momentum based GD', 2, None, '___sec36'), ('More on momentum based approaches', 2, None, '___sec37'), ('Momentum parameter', 2, None, '___sec38'), @@ -216,16 +216,16 @@ MathJax.Hub.Config({
-Stochastic gradient descent (SGD) and variants thereof address some of -the shortcomings of the Gradient descent method discussed above. +
-The underlying idea of SGD comes from the observation that the cost -function, which we want to minimize, can almost always be written as a -sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \), -$$ -C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, -\mathbf{\beta}). -$$ - -
-This in turn means that the gradient can be -computed as a sum over \( i \)-gradients -$$ -\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}). -$$ +Stochastic gradient descent (SGD) and variants thereof address some of +the shortcomings of the Gradient descent method discussed above.
-Stochasticity/randomness is introduced by only taking the -gradient on a subset of the data called minibatches. If there are \( n \) -data points and the size of each minibatch is \( M \), there will be \( n/M \) -minibatches. We denote these minibatches by \( B_k \) where -\( k=1,\cdots,n/M \). +The underlying idea of SGD comes from the observation that the cost +function, which we want to minimize, can almost always be written as a +sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \), +$$ +C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, +\mathbf{\beta}). +$$
diff --git a/doc/pub/Splines/html/._Splines-bs029.html b/doc/pub/Splines/html/._Splines-bs029.html index 849377730..a9db9ad04 100644 --- a/doc/pub/Splines/html/._Splines-bs029.html +++ b/doc/pub/Splines/html/._Splines-bs029.html @@ -80,19 +80,19 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec25'), - ('Stochastic Gradient Descent', 2, None, '___sec26'), - ('Computation of gradients', 2, None, '___sec27'), - ('SGD example', 2, None, '___sec28'), - ('The gradient step', 2, None, '___sec29'), - ('Simple example code', 2, None, '___sec30'), - ('When do we stop?', 2, None, '___sec31'), - ('Slightly different approach', 2, None, '___sec32'), - ('Program for stochastic gradient', 2, None, '___sec33'), - ('Logistic Regression example', 2, None, '___sec34'), ('Using gradient descent methods, limitations', 2, None, - '___sec35'), + '___sec26'), + ('Stochastic Gradient Descent', 2, None, '___sec27'), + ('Computation of gradients', 2, None, '___sec28'), + ('SGD example', 2, None, '___sec29'), + ('The gradient step', 2, None, '___sec30'), + ('Simple example code', 2, None, '___sec31'), + ('When do we stop?', 2, None, '___sec32'), + ('Slightly different approach', 2, None, '___sec33'), + ('Program for stochastic gradient', 2, None, '___sec34'), + ('Logistic Regression example', 2, None, '___sec35'), ('Momentum based GD', 2, None, '___sec36'), ('More on momentum based approaches', 2, None, '___sec37'), ('Momentum parameter', 2, None, '___sec38'), @@ -216,16 +216,16 @@ MathJax.Hub.Config({
-The idea is now to approximate the gradient by replacing the sum over -all data points with a sum over the data points in one the minibatches -picked at random in each gradient descent step +This in turn means that the gradient can be +computed as a sum over \( i \)-gradients $$ -\nabla_{\beta} -C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta -c_i(\mathbf{x}_i, \mathbf{\beta}). +\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, +\mathbf{\beta}). $$ +
+Stochasticity/randomness is introduced by only taking the +gradient on a subset of the data called minibatches. If there are \( n \) +data points and the size of each minibatch is \( M \), there will be \( n/M \) +minibatches. We denote these minibatches by \( B_k \) where +\( k=1,\cdots,n/M \). +
diff --git a/doc/pub/Splines/html/._Splines-bs030.html b/doc/pub/Splines/html/._Splines-bs030.html index b82c0566a..1dd8af54e 100644 --- a/doc/pub/Splines/html/._Splines-bs030.html +++ b/doc/pub/Splines/html/._Splines-bs030.html @@ -80,19 +80,19 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec25'), - ('Stochastic Gradient Descent', 2, None, '___sec26'), - ('Computation of gradients', 2, None, '___sec27'), - ('SGD example', 2, None, '___sec28'), - ('The gradient step', 2, None, '___sec29'), - ('Simple example code', 2, None, '___sec30'), - ('When do we stop?', 2, None, '___sec31'), - ('Slightly different approach', 2, None, '___sec32'), - ('Program for stochastic gradient', 2, None, '___sec33'), - ('Logistic Regression example', 2, None, '___sec34'), ('Using gradient descent methods, limitations', 2, None, - '___sec35'), + '___sec26'), + ('Stochastic Gradient Descent', 2, None, '___sec27'), + ('Computation of gradients', 2, None, '___sec28'), + ('SGD example', 2, None, '___sec29'), + ('The gradient step', 2, None, '___sec30'), + ('Simple example code', 2, None, '___sec31'), + ('When do we stop?', 2, None, '___sec32'), + ('Slightly different approach', 2, None, '___sec33'), + ('Program for stochastic gradient', 2, None, '___sec34'), + ('Logistic Regression example', 2, None, '___sec35'), ('Momentum based GD', 2, None, '___sec36'), ('More on momentum based approaches', 2, None, '___sec37'), ('Momentum parameter', 2, None, '___sec38'), @@ -216,16 +216,16 @@ MathJax.Hub.Config({
-Thus a gradient descent step now looks like +The idea is now to approximate the gradient by replacing the sum over +all data points with a sum over the data points in one the minibatches +picked at random in each gradient descent step $$ -\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) +\nabla_{\beta} +C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, +\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta +c_i(\mathbf{x}_i, \mathbf{\beta}). $$ -
-where \( k \) is picked at random with equal -probability from \( [1,n/M] \). An iteration over the number of -minibathces (n/M) is commonly referred to as an epoch. Thus it is -typical to choose a number of epochs and for each epoch iterate over -the number of minibatches, as exemplified in the code below. -
diff --git a/doc/pub/Splines/html/._Splines-bs031.html b/doc/pub/Splines/html/._Splines-bs031.html index fb28e77b0..9a8783073 100644 --- a/doc/pub/Splines/html/._Splines-bs031.html +++ b/doc/pub/Splines/html/._Splines-bs031.html @@ -80,19 +80,19 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec25'), - ('Stochastic Gradient Descent', 2, None, '___sec26'), - ('Computation of gradients', 2, None, '___sec27'), - ('SGD example', 2, None, '___sec28'), - ('The gradient step', 2, None, '___sec29'), - ('Simple example code', 2, None, '___sec30'), - ('When do we stop?', 2, None, '___sec31'), - ('Slightly different approach', 2, None, '___sec32'), - ('Program for stochastic gradient', 2, None, '___sec33'), - ('Logistic Regression example', 2, None, '___sec34'), ('Using gradient descent methods, limitations', 2, None, - '___sec35'), + '___sec26'), + ('Stochastic Gradient Descent', 2, None, '___sec27'), + ('Computation of gradients', 2, None, '___sec28'), + ('SGD example', 2, None, '___sec29'), + ('The gradient step', 2, None, '___sec30'), + ('Simple example code', 2, None, '___sec31'), + ('When do we stop?', 2, None, '___sec32'), + ('Slightly different approach', 2, None, '___sec33'), + ('Program for stochastic gradient', 2, None, '___sec34'), + ('Logistic Regression example', 2, None, '___sec35'), ('Momentum based GD', 2, None, '___sec36'), ('More on momentum based approaches', 2, None, '___sec37'), ('Momentum parameter', 2, None, '___sec38'), @@ -216,16 +216,16 @@ MathJax.Hub.Config({
+Thus a gradient descent step now looks like +$$ +\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i, +\mathbf{\beta}) +$$ - -
import numpy as np
-
-n = 100 #100 datapoints
-M = 5 #size of each minibatch
-m = int(n/M) #number of minibatches
-n_epochs = 10 #number of epochs
-
-j = 0
-for epoch in range(1,n_epochs+1):
- for i in range(m):
- k = np.random.randint(m) #Pick the k-th minibatch at random
- #Compute the gradient using the data in minibatch Bk
- #Compute new suggestion for
- j += 1
--Taking the gradient only on a subset of the data has two important -benefits. First, it introduces randomness which decreases the chance -that our opmization scheme gets stuck in a local minima. Second, if -the size of the minibatches are small relative to the number of -datapoints (\( M < n \)), the computation of the gradient is much -cheaper since we sum over the datapoints in the \( k-th \) minibatch and not -all \( n \) datapoints. +where \( k \) is picked at random with equal +probability from \( [1,n/M] \). An iteration over the number of +minibathces (n/M) is commonly referred to as an epoch. Thus it is +typical to choose a number of epochs and for each epoch iterate over +the number of minibatches, as exemplified in the code below.
diff --git a/doc/pub/Splines/html/._Splines-bs032.html b/doc/pub/Splines/html/._Splines-bs032.html index bb59661a0..162c35e44 100644 --- a/doc/pub/Splines/html/._Splines-bs032.html +++ b/doc/pub/Splines/html/._Splines-bs032.html @@ -80,19 +80,19 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec25'), - ('Stochastic Gradient Descent', 2, None, '___sec26'), - ('Computation of gradients', 2, None, '___sec27'), - ('SGD example', 2, None, '___sec28'), - ('The gradient step', 2, None, '___sec29'), - ('Simple example code', 2, None, '___sec30'), - ('When do we stop?', 2, None, '___sec31'), - ('Slightly different approach', 2, None, '___sec32'), - ('Program for stochastic gradient', 2, None, '___sec33'), - ('Logistic Regression example', 2, None, '___sec34'), ('Using gradient descent methods, limitations', 2, None, - '___sec35'), + '___sec26'), + ('Stochastic Gradient Descent', 2, None, '___sec27'), + ('Computation of gradients', 2, None, '___sec28'), + ('SGD example', 2, None, '___sec29'), + ('The gradient step', 2, None, '___sec30'), + ('Simple example code', 2, None, '___sec31'), + ('When do we stop?', 2, None, '___sec32'), + ('Slightly different approach', 2, None, '___sec33'), + ('Program for stochastic gradient', 2, None, '___sec34'), + ('Logistic Regression example', 2, None, '___sec35'), ('Momentum based GD', 2, None, '___sec36'), ('More on momentum based approaches', 2, None, '___sec37'), ('Momentum parameter', 2, None, '___sec38'), @@ -216,16 +216,16 @@ MathJax.Hub.Config({
-A natural question is when do we stop the search for a new minimum? -One possibility is to compute the full gradient after a given number -of epochs and check if the norm of the gradient is smaller than some -threshold and stop if true. However, the condition that the gradient -is zero is valid also for local minima, so this would only tell us -that we are close to a local/global minimum. However, we could also -evaluate the cost function at this point, store the result and -continue the search. If the test kicks in at a later stage we can -compare the values of the cost function and keep the \( \beta \) that -gave the lowest value. + + +
import numpy as np
+
+n = 100 #100 datapoints
+M = 5 #size of each minibatch
+m = int(n/M) #number of minibatches
+n_epochs = 10 #number of epochs
+
+j = 0
+for epoch in range(1,n_epochs+1):
+ for i in range(m):
+ k = np.random.randint(m) #Pick the k-th minibatch at random
+ #Compute the gradient using the data in minibatch Bk
+ #Compute new suggestion for
+ j += 1
++Taking the gradient only on a subset of the data has two important +benefits. First, it introduces randomness which decreases the chance +that our opmization scheme gets stuck in a local minima. Second, if +the size of the minibatches are small relative to the number of +datapoints (\( M < n \)), the computation of the gradient is much +cheaper since we sum over the datapoints in the \( k-th \) minibatch and not +all \( n \) datapoints.
diff --git a/doc/pub/Splines/html/._Splines-bs033.html b/doc/pub/Splines/html/._Splines-bs033.html index 31e3bd5fb..f5daec5d2 100644 --- a/doc/pub/Splines/html/._Splines-bs033.html +++ b/doc/pub/Splines/html/._Splines-bs033.html @@ -80,19 +80,19 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec25'), - ('Stochastic Gradient Descent', 2, None, '___sec26'), - ('Computation of gradients', 2, None, '___sec27'), - ('SGD example', 2, None, '___sec28'), - ('The gradient step', 2, None, '___sec29'), - ('Simple example code', 2, None, '___sec30'), - ('When do we stop?', 2, None, '___sec31'), - ('Slightly different approach', 2, None, '___sec32'), - ('Program for stochastic gradient', 2, None, '___sec33'), - ('Logistic Regression example', 2, None, '___sec34'), ('Using gradient descent methods, limitations', 2, None, - '___sec35'), + '___sec26'), + ('Stochastic Gradient Descent', 2, None, '___sec27'), + ('Computation of gradients', 2, None, '___sec28'), + ('SGD example', 2, None, '___sec29'), + ('The gradient step', 2, None, '___sec30'), + ('Simple example code', 2, None, '___sec31'), + ('When do we stop?', 2, None, '___sec32'), + ('Slightly different approach', 2, None, '___sec33'), + ('Program for stochastic gradient', 2, None, '___sec34'), + ('Logistic Regression example', 2, None, '___sec35'), ('Momentum based GD', 2, None, '___sec36'), ('More on momentum based approaches', 2, None, '___sec37'), ('Momentum parameter', 2, None, '___sec38'), @@ -216,16 +216,16 @@ MathJax.Hub.Config({
-Another approach is to let the step length \( \gamma_j \) depend on the -number of epochs in such a way that it becomes very small after a -reasonable time such that we do not move at all. +A natural question is when do we stop the search for a new minimum? +One possibility is to compute the full gradient after a given number +of epochs and check if the norm of the gradient is smaller than some +threshold and stop if true. However, the condition that the gradient +is zero is valid also for local minima, so this would only tell us +that we are close to a local/global minimum. However, we could also +evaluate the cost function at this point, store the result and +continue the search. If the test kicks in at a later stage we can +compare the values of the cost function and keep the \( \beta \) that +gave the lowest value. -
-As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \). - -
-In this way we can fix the number of epochs, compute \( \beta \) and -evaluate the cost function at the end. Repeating the computation will -give a different result since the scheme is random by design. Then we -pick the final \( \beta \) that gives the lowest value of the cost -function. - -
- - -
import numpy as np
-
-def step_length(t,t0,t1):
- return t0/(t+t1)
-
-n = 100 #100 datapoints
-M = 5 #size of each minibatch
-m = int(n/M) #number of minibatches
-n_epochs = 500 #number of epochs
-t0 = 1.0
-t1 = 10
-
-gamma_j = t0/t1
-j = 0
-for epoch in range(1,n_epochs+1):
- for i in range(m):
- k = np.random.randint(m) #Pick the k-th minibatch at random
- #Compute the gradient using the data in minibatch Bk
- #Compute new suggestion for beta
- t = epoch*m+i
- gamma_j = step_length(t,t0,t1)
- j += 1
-
-print("gamma_j after %d epochs: %g" % (n_epochs,gamma_j))
-
diff --git a/doc/pub/Splines/html/._Splines-bs034.html b/doc/pub/Splines/html/._Splines-bs034.html index b28545fff..83fa01c74 100644 --- a/doc/pub/Splines/html/._Splines-bs034.html +++ b/doc/pub/Splines/html/._Splines-bs034.html @@ -80,19 +80,19 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec25'), - ('Stochastic Gradient Descent', 2, None, '___sec26'), - ('Computation of gradients', 2, None, '___sec27'), - ('SGD example', 2, None, '___sec28'), - ('The gradient step', 2, None, '___sec29'), - ('Simple example code', 2, None, '___sec30'), - ('When do we stop?', 2, None, '___sec31'), - ('Slightly different approach', 2, None, '___sec32'), - ('Program for stochastic gradient', 2, None, '___sec33'), - ('Logistic Regression example', 2, None, '___sec34'), ('Using gradient descent methods, limitations', 2, None, - '___sec35'), + '___sec26'), + ('Stochastic Gradient Descent', 2, None, '___sec27'), + ('Computation of gradients', 2, None, '___sec28'), + ('SGD example', 2, None, '___sec29'), + ('The gradient step', 2, None, '___sec30'), + ('Simple example code', 2, None, '___sec31'), + ('When do we stop?', 2, None, '___sec32'), + ('Slightly different approach', 2, None, '___sec33'), + ('Program for stochastic gradient', 2, None, '___sec34'), + ('Logistic Regression example', 2, None, '___sec35'), ('Momentum based GD', 2, None, '___sec36'), ('More on momentum based approaches', 2, None, '___sec37'), ('Momentum parameter', 2, None, '___sec38'), @@ -216,16 +216,16 @@ MathJax.Hub.Config({
+Another approach is to let the step length \( \gamma_j \) depend on the +number of epochs in such a way that it becomes very small after a +reasonable time such that we do not move at all. + +
+As an example, let \( e = 0,1,2,3,\cdots \) denote the current epoch and let \( t_0, t_1 > 0 \) be two fixed numbers. Furthermore, let \( t = e \cdot m + i \) where \( m \) is the number of minibatches and \( i=0,\cdots,m-1 \). Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length \( \gamma_j (0; t_0, t_1) = t_0/t_1 \) which decays in time \( t \). + +
+In this way we can fix the number of epochs, compute \( \beta \) and +evaluate the cost function at the end. Repeating the computation will +give a different result since the scheme is random by design. Then we +pick the final \( \beta \) that gives the lowest value of the cost +function.
-
# Importing various packages
-from math import exp, sqrt
-from random import random, seed
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.linear_model import SGDRegressor
+import numpy as np
-x = 2*np.random.rand(100,1)
-y = 4+3*x+np.random.randn(100,1)
-
-xb = np.c_[np.ones((100,1)), x]
-theta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
-print("Own inversion")
-print(theta_linreg)
-sgdreg = SGDRegressor(max_iter = 50, penalty=None, eta0=0.1)
-sgdreg.fit(x,y.ravel())
-print("sgdreg from scikit")
-print(sgdreg.intercept_, sgdreg.coef_)
-
-
-theta = np.random.randn(2,1)
-
-eta = 0.1
-Niterations = 1000
-m = 100
-
-for iter in range(Niterations):
- gradients = 2.0/m*xb.T.dot(xb.dot(theta)-y)
- theta -= eta*gradients
-print("theta frm own gd")
-print(theta)
-
-xnew = np.array([[0],[2]])
-xbnew = np.c_[np.ones((2,1)), xnew]
-ypredict = xbnew.dot(theta)
-ypredict2 = xbnew.dot(theta_linreg)
-
-
-n_epochs = 50
-t0, t1 = 5, 50
-m = 100
-def learning_schedule(t):
+def step_length(t,t0,t1):
return t0/(t+t1)
-theta = np.random.randn(2,1)
+n = 100 #100 datapoints
+M = 5 #size of each minibatch
+m = int(n/M) #number of minibatches
+n_epochs = 500 #number of epochs
+t0 = 1.0
+t1 = 10
-for epoch in range(n_epochs):
+gamma_j = t0/t1
+j = 0
+for epoch in range(1,n_epochs+1):
for i in range(m):
- random_index = np.random.randint(m)
- xi = xb[random_index:random_index+1]
- yi = y[random_index:random_index+1]
- gradients = 2 * xi.T.dot(xi.dot(theta)-yi)
- eta = learning_schedule(epoch*m+i)
- theta = theta - eta*gradients
-print("theta from own sdg")
-print(theta)
+ k = np.random.randint(m) #Pick the k-th minibatch at random
+ #Compute the gradient using the data in minibatch Bk
+ #Compute new suggestion for beta
+ t = epoch*m+i
+ gamma_j = step_length(t,t0,t1)
+ j += 1
-plt.plot(xnew, ypredict, "r-")
-plt.plot(xnew, ypredict2, "b-")
-plt.plot(x, y ,'ro')
-plt.axis([0,2.0,0, 15.0])
-plt.xlabel(r'$x$')
-plt.ylabel(r'$y$')
-plt.title(r'Random numbers ')
-plt.show()
+print("gamma_j after %d epochs: %g" % (n_epochs,gamma_j))
diff --git a/doc/pub/Splines/html/._Splines-bs035.html b/doc/pub/Splines/html/._Splines-bs035.html
index 3bc46f8a9..89f412e48 100644
--- a/doc/pub/Splines/html/._Splines-bs035.html
+++ b/doc/pub/Splines/html/._Splines-bs035.html
@@ -80,19 +80,19 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec25'),
- ('Stochastic Gradient Descent', 2, None, '___sec26'),
- ('Computation of gradients', 2, None, '___sec27'),
- ('SGD example', 2, None, '___sec28'),
- ('The gradient step', 2, None, '___sec29'),
- ('Simple example code', 2, None, '___sec30'),
- ('When do we stop?', 2, None, '___sec31'),
- ('Slightly different approach', 2, None, '___sec32'),
- ('Program for stochastic gradient', 2, None, '___sec33'),
- ('Logistic Regression example', 2, None, '___sec34'),
('Using gradient descent methods, limitations',
2,
None,
- '___sec35'),
+ '___sec26'),
+ ('Stochastic Gradient Descent', 2, None, '___sec27'),
+ ('Computation of gradients', 2, None, '___sec28'),
+ ('SGD example', 2, None, '___sec29'),
+ ('The gradient step', 2, None, '___sec30'),
+ ('Simple example code', 2, None, '___sec31'),
+ ('When do we stop?', 2, None, '___sec32'),
+ ('Slightly different approach', 2, None, '___sec33'),
+ ('Program for stochastic gradient', 2, None, '___sec34'),
+ ('Logistic Regression example', 2, None, '___sec35'),
('Momentum based GD', 2, None, '___sec36'),
('More on momentum based approaches', 2, None, '___sec37'),
('Momentum parameter', 2, None, '___sec38'),
@@ -216,16 +216,16 @@ MathJax.Hub.Config({
+ + +
# Importing various packages
+from math import exp, sqrt
+from random import random, seed
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.linear_model import SGDRegressor
+
+x = 2*np.random.rand(100,1)
+y = 4+3*x+np.random.randn(100,1)
+
+xb = np.c_[np.ones((100,1)), x]
+theta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
+print("Own inversion")
+print(theta_linreg)
+sgdreg = SGDRegressor(max_iter = 50, penalty=None, eta0=0.1)
+sgdreg.fit(x,y.ravel())
+print("sgdreg from scikit")
+print(sgdreg.intercept_, sgdreg.coef_)
+
+
+theta = np.random.randn(2,1)
+
+eta = 0.1
+Niterations = 1000
+m = 100
+
+for iter in range(Niterations):
+ gradients = 2.0/m*xb.T @ ((xb @ theta)-y)
+ theta -= eta*gradients
+print("theta frm own gd")
+print(theta)
+
+xnew = np.array([[0],[2]])
+xbnew = np.c_[np.ones((2,1)), xnew]
+ypredict = xbnew.dot(theta)
+ypredict2 = xbnew.dot(theta_linreg)
+
+
+n_epochs = 50
+t0, t1 = 5, 50
+m = 100
+def learning_schedule(t):
+ return t0/(t+t1)
+
+theta = np.random.randn(2,1)
+
+for epoch in range(n_epochs):
+ for i in range(m):
+ random_index = np.random.randint(m)
+ xi = xb[random_index:random_index+1]
+ yi = y[random_index:random_index+1]
+ gradients = 2 * xi.T @ ((xi @ theta)-yi)
+ eta = learning_schedule(epoch*m+i)
+ theta = theta - eta*gradients
+print("theta from own sdg")
+print(theta)
+
+plt.plot(xnew, ypredict, "r-")
+plt.plot(xnew, ypredict2, "b-")
+plt.plot(x, y ,'ro')
+plt.axis([0,2.0,0, 15.0])
+plt.xlabel(r'$x$')
+plt.ylabel(r'$y$')
+plt.title(r'Random numbers ')
+plt.show()
+
diff --git a/doc/pub/Splines/html/._Splines-bs036.html b/doc/pub/Splines/html/._Splines-bs036.html index 2e06ec99b..ffd5b17df 100644 --- a/doc/pub/Splines/html/._Splines-bs036.html +++ b/doc/pub/Splines/html/._Splines-bs036.html @@ -80,19 +80,19 @@ Automatically generated HTML file from DocOnce source 2, None, '___sec25'), - ('Stochastic Gradient Descent', 2, None, '___sec26'), - ('Computation of gradients', 2, None, '___sec27'), - ('SGD example', 2, None, '___sec28'), - ('The gradient step', 2, None, '___sec29'), - ('Simple example code', 2, None, '___sec30'), - ('When do we stop?', 2, None, '___sec31'), - ('Slightly different approach', 2, None, '___sec32'), - ('Program for stochastic gradient', 2, None, '___sec33'), - ('Logistic Regression example', 2, None, '___sec34'), ('Using gradient descent methods, limitations', 2, None, - '___sec35'), + '___sec26'), + ('Stochastic Gradient Descent', 2, None, '___sec27'), + ('Computation of gradients', 2, None, '___sec28'), + ('SGD example', 2, None, '___sec29'), + ('The gradient step', 2, None, '___sec30'), + ('Simple example code', 2, None, '___sec31'), + ('When do we stop?', 2, None, '___sec32'), + ('Slightly different approach', 2, None, '___sec33'), + ('Program for stochastic gradient', 2, None, '___sec34'), + ('Logistic Regression example', 2, None, '___sec35'), ('Momentum based GD', 2, None, '___sec36'), ('More on momentum based approaches', 2, None, '___sec37'), ('Momentum parameter', 2, None, '___sec38'), @@ -216,16 +216,16 @@ MathJax.Hub.Config({
-
diff --git a/doc/pub/Splines/html/Splines-reveal.html b/doc/pub/Splines/html/Splines-reveal.html index 2219d4572..176156e4d 100644 --- a/doc/pub/Splines/html/Splines-reveal.html +++ b/doc/pub/Splines/html/Splines-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({
-
@@ -905,7 +905,21 @@ plt.show()
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -925,7 +939,7 @@ $$
This in turn means that the gradient can be
@@ -947,7 +961,7 @@ minibatches. We denote these minibatches by \( B_k \) where
Thus a gradient descent step now looks like
@@ -994,7 +1008,7 @@ the number of minibatches, as exemplified in the code below.
@@ -1026,7 +1040,7 @@ all \( n \) datapoints.
A natural question is when do we stop the search for a new minimum?
@@ -1043,7 +1057,7 @@ gave the lowest value.
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -1094,7 +1108,7 @@ j = 0
@@ -1126,7 +1140,7 @@ Niterations = 1000
m = 100
for iter in range(Niterations):
- gradients = 2.0/m*xb.T.dot(xb.dot(theta)-y)
+ gradients = 2.0/m*xb.T @ ((xb @ theta)-y)
theta -= eta*gradients
print("theta frm own gd")
print(theta)
@@ -1150,7 +1164,7 @@ theta = np.random.randn(2,1]
yi = y[random_index:random_index+1]
- gradients = 2 * xi.T.dot(xi.dot(theta)-yi)
+ gradients = 2 * xi.T @ ((xi @ theta)-yi)
eta = learning_schedule(epoch*m+i)
theta = theta - eta*gradients
print("theta from own sdg")
@@ -1169,21 +1183,7 @@ plt.show()
-
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -924,7 +937,7 @@ $$
This in turn means that the gradient can be
@@ -944,7 +957,7 @@ minibatches. We denote these minibatches by \( B_k \) where
Thus a gradient descent step now looks like
@@ -987,7 +1000,7 @@ the number of minibatches, as exemplified in the code below.
@@ -1019,7 +1032,7 @@ all \( n \) datapoints.
A natural question is when do we stop the search for a new minimum?
@@ -1036,7 +1049,7 @@ gave the lowest value.
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -1084,7 +1097,7 @@ j = 0
@@ -1116,7 +1129,7 @@ Niterations = 1000
m = 100
for iter in range(Niterations):
- gradients = 2.0/m*xb.T.dot(xb.dot(theta)-y)
+ gradients = 2.0/m*xb.T @ ((xb @ theta)-y)
theta -= eta*gradients
print("theta frm own gd")
print(theta)
@@ -1140,7 +1153,7 @@ theta = np.random.randn(2,1]
yi = y[random_index:random_index+1]
- gradients = 2 * xi.T.dot(xi.dot(theta)-yi)
+ gradients = 2 * xi.T @ ((xi @ theta)-yi)
eta = learning_schedule(epoch*m+i)
theta = theta - eta*gradients
print("theta from own sdg")
@@ -1158,24 +1171,11 @@ plt.show()
diff --git a/doc/pub/Splines/html/Splines.html b/doc/pub/Splines/html/Splines.html
index 7a767d910..b39dd2177 100644
--- a/doc/pub/Splines/html/Splines.html
+++ b/doc/pub/Splines/html/Splines.html
@@ -105,19 +105,19 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec25'),
- ('Stochastic Gradient Descent', 2, None, '___sec26'),
- ('Computation of gradients', 2, None, '___sec27'),
- ('SGD example', 2, None, '___sec28'),
- ('The gradient step', 2, None, '___sec29'),
- ('Simple example code', 2, None, '___sec30'),
- ('When do we stop?', 2, None, '___sec31'),
- ('Slightly different approach', 2, None, '___sec32'),
- ('Program for stochastic gradient', 2, None, '___sec33'),
- ('Logistic Regression example', 2, None, '___sec34'),
('Using gradient descent methods, limitations',
2,
None,
- '___sec35'),
+ '___sec26'),
+ ('Stochastic Gradient Descent', 2, None, '___sec27'),
+ ('Computation of gradients', 2, None, '___sec28'),
+ ('SGD example', 2, None, '___sec29'),
+ ('The gradient step', 2, None, '___sec30'),
+ ('Simple example code', 2, None, '___sec31'),
+ ('When do we stop?', 2, None, '___sec32'),
+ ('Slightly different approach', 2, None, '___sec33'),
+ ('Program for stochastic gradient', 2, None, '___sec34'),
+ ('Logistic Regression example', 2, None, '___sec35'),
('Momentum based GD', 2, None, '___sec36'),
('More on momentum based approaches', 2, None, '___sec37'),
('Momentum parameter', 2, None, '___sec38'),
@@ -219,7 +219,7 @@ MathJax.Hub.Config({
-
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -929,7 +942,7 @@ $$
This in turn means that the gradient can be
@@ -949,7 +962,7 @@ minibatches. We denote these minibatches by \( B_k \) where
Thus a gradient descent step now looks like
@@ -992,7 +1005,7 @@ the number of minibatches, as exemplified in the code below.
@@ -1024,7 +1037,7 @@ all \( n \) datapoints.
A natural question is when do we stop the search for a new minimum?
@@ -1041,7 +1054,7 @@ gave the lowest value.
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -1089,7 +1102,7 @@ j = 0
@@ -1121,7 +1134,7 @@ Niterations = 1
m = 100
for iter in range(Niterations):
- gradients = 2.0/m*xb.T.dot(xb.dot(theta)-y)
+ gradients = 2.0/m*xb.T @ ((xb @ theta)-y)
theta -= eta*gradients
print("theta frm own gd")
print(theta)
@@ -1145,7 +1158,7 @@ theta = np.= np.random.randint(m)
xi = xb[random_index:random_index+1]
yi = y[random_index:random_index+1]
- gradients = 2 * xi.T.dot(xi.dot(theta)-yi)
+ gradients = 2 * xi.T @ ((xi @ theta)-yi)
eta = learning_schedule(epoch*m+i)
theta = theta - eta*gradients
print("theta from own sdg")
@@ -1163,24 +1176,11 @@ plt.show()
diff --git a/doc/pub/Splines/ipynb/Splines.ipynb b/doc/pub/Splines/ipynb/Splines.ipynb
index 6aefd3965..f3c394695 100644
--- a/doc/pub/Splines/ipynb/Splines.ipynb
+++ b/doc/pub/Splines/ipynb/Splines.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Sep 25, 2019**\n",
+ "Date: **Sep 26, 2019**\n",
"\n",
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -923,6 +923,20 @@
"cell_type": "markdown",
"metadata": {},
"source": [
+ "## Using gradient descent methods, limitations\n",
+ "\n",
+ "* **Gradient descent (GD) finds local minima of our function**. Since the GD algorithm is deterministic, if it converges, it will converge to a local minimum of our energy function. Because in ML we are often dealing with extremely rugged landscapes with many local minima, this can lead to poor performance.\n",
+ "\n",
+ "* **GD is sensitive to initial conditions**. One consequence of the local nature of GD is that initial conditions matter. Depending on where one starts, one will end up at a different local minima. Therefore, it is very important to think about how one initializes the training process. This is true for GD as well as more complicated variants of GD.\n",
+ "\n",
+ "* **Gradients are computationally expensive to calculate for large datasets**. In many cases in statistics and ML, the energy function is a sum of terms, with one term for each data point. For example, in linear regression, $E \\propto \\sum_{i=1}^n (y_i - \\mathbf{w}^T\\cdot\\mathbf{x}_i)^2$; for logistic regression, the square error is replaced by the cross entropy. To calculate the gradient we have to sum over *all* $n$ data points. Doing this at every GD step becomes extremely computationally expensive. An ingenious solution to this, is to calculate the gradients using small subsets of the data called \"mini batches\". This has the added benefit of introducing stochasticity into our algorithm.\n",
+ "\n",
+ "* **GD is very sensitive to choices of learning rates**. GD is extremely sensitive to the choice of learning rates. If the learning rate is very small, the training process take an extremely long time. For larger learning rates, GD can diverge and give poor results. Furthermore, depending on what the local landscape looks like, we have to modify the learning rates to ensure convergence. Ideally, we would *adaptively* choose the learning rates to match the landscape.\n",
+ "\n",
+ "* **GD treats all directions in parameter space uniformly.** Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive. \n",
+ "\n",
+ "* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.\n",
+ "\n",
"## Stochastic Gradient Descent\n",
"\n",
"Stochastic gradient descent (SGD) and variants thereof address some of\n",
@@ -1172,7 +1186,7 @@
"m = 100\n",
"\n",
"for iter in range(Niterations):\n",
- " gradients = 2.0/m*xb.T.dot(xb.dot(theta)-y)\n",
+ " gradients = 2.0/m*xb.T @ ((xb @ theta)-y)\n",
" theta -= eta*gradients\n",
"print(\"theta frm own gd\")\n",
"print(theta)\n",
@@ -1196,7 +1210,7 @@
" random_index = np.random.randint(m)\n",
" xi = xb[random_index:random_index+1]\n",
" yi = y[random_index:random_index+1]\n",
- " gradients = 2 * xi.T.dot(xi.dot(theta)-yi)\n",
+ " gradients = 2 * xi.T @ ((xi @ theta)-yi)\n",
" eta = learning_schedule(epoch*m+i)\n",
" theta = theta - eta*gradients\n",
"print(\"theta from own sdg\")\n",
@@ -1219,19 +1233,6 @@
"## Logistic Regression example\n",
"\n",
"\n",
- "## Using gradient descent methods, limitations\n",
- "\n",
- "* **Gradient descent (GD) finds local minima of our function**. Since the GD algorithm is deterministic, if it converges, it will converge to a local minimum of our energy function. Because in ML we are often dealing with extremely rugged landscapes with many local minima, this can lead to poor performance.\n",
- "\n",
- "* **GD is sensitive to initial conditions**. One consequence of the local nature of GD is that initial conditions matter. Depending on where one starts, one will end up at a different local minima. Therefore, it is very important to think about how one initializes the training process. This is true for GD as well as more complicated variants of GD.\n",
- "\n",
- "* **Gradients are computationally expensive to calculate for large datasets**. In many cases in statistics and ML, the energy function is a sum of terms, with one term for each data point. For example, in linear regression, $E \\propto \\sum_{i=1}^n (y_i - \\mathbf{w}^T\\cdot\\mathbf{x}_i)^2$; for logistic regression, the square error is replaced by the cross entropy. To calculate the gradient we have to sum over *all* $n$ data points. Doing this at every GD step becomes extremely computationally expensive. An ingenious solution to this, is to calculate the gradients using small subsets of the data called \"mini batches\". This has the added benefit of introducing stochasticity into our algorithm.\n",
- "\n",
- "* **GD is very sensitive to choices of learning rates**. GD is extremely sensitive to the choice of learning rates. If the learning rate is very small, the training process take an extremely long time. For larger learning rates, GD can diverge and give poor results. Furthermore, depending on what the local landscape looks like, we have to modify the learning rates to ensure convergence. Ideally, we would *adaptively* choose the learning rates to match the landscape.\n",
- "\n",
- "* **GD treats all directions in parameter space uniformly.** Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive. \n",
- "\n",
- "* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.\n",
"\n",
"## Momentum based GD\n",
"\n",
diff --git a/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz b/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz
index e82771d7c..63c898211 100644
Binary files a/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz and b/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz differ
diff --git a/doc/pub/Splines/pdf/Splines-minted.pdf b/doc/pub/Splines/pdf/Splines-minted.pdf
index 98e8f46c6..0b8b56091 100644
Binary files a/doc/pub/Splines/pdf/Splines-minted.pdf and b/doc/pub/Splines/pdf/Splines-minted.pdf differ
diff --git a/doc/src/LogisticRegression/moons.py b/doc/src/LogisticRegression/moons.py
new file mode 100644
index 000000000..8698a7a02
--- /dev/null
+++ b/doc/src/LogisticRegression/moons.py
@@ -0,0 +1,44 @@
+import numpy as np
+from sklearn import datasets, linear_model
+import matplotlib.pyplot as plt
+
+
+def generate_data():
+ np.random.seed(0)
+ X, y = datasets.make_moons(200, noise=0.20)
+ return X, y
+
+
+def visualize(X, y, clf):
+ plot_decision_boundary(lambda x: clf.predict(x), X, y)
+
+def plot_decision_boundary(pred_func, X, y):
+ # Set min and max values and give it some padding
+ x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
+ y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
+ h = 0.01
+ # Generate a grid of points with distance h between them
+ xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
+ # Predict the function value for the whole gid
+ Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
+ Z = Z.reshape(xx.shape)
+ # Plot the contour and training examples
+ plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
+ plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
+ plt.show()
+
+
+def classify(X, y):
+ clf = linear_model.LogisticRegressionCV()
+ clf.fit(X, y)
+ return clf
+
+
+def main():
+ X, y = generate_data()
+ # visualize(X, y)
+ clf = classify(X, y)
+ visualize(X, y, clf)
+
+if __name__ == "__main__":
+ main()
diff --git a/doc/src/Splines/Splines.do.txt b/doc/src/Splines/Splines.do.txt
index e2f94c8b1..cfabe9a4a 100644
--- a/doc/src/Splines/Splines.do.txt
+++ b/doc/src/Splines/Splines.do.txt
@@ -625,6 +625,21 @@ plt.show()
!ec
+!split
+===== Using gradient descent methods, limitations =====
+
+* _Gradient descent (GD) finds local minima of our function_. Since the GD algorithm is deterministic, if it converges, it will converge to a local minimum of our energy function. Because in ML we are often dealing with extremely rugged landscapes with many local minima, this can lead to poor performance.
+
+* _GD is sensitive to initial conditions_. One consequence of the local nature of GD is that initial conditions matter. Depending on where one starts, one will end up at a different local minima. Therefore, it is very important to think about how one initializes the training process. This is true for GD as well as more complicated variants of GD.
+
+* _Gradients are computationally expensive to calculate for large datasets_. In many cases in statistics and ML, the energy function is a sum of terms, with one term for each data point. For example, in linear regression, $E \propto \sum_{i=1}^n (y_i - \mathbf{w}^T\cdot\mathbf{x}_i)^2$; for logistic regression, the square error is replaced by the cross entropy. To calculate the gradient we have to sum over *all* $n$ data points. Doing this at every GD step becomes extremely computationally expensive. An ingenious solution to this, is to calculate the gradients using small subsets of the data called ``mini batches''. This has the added benefit of introducing stochasticity into our algorithm.
+
+* _GD is very sensitive to choices of learning rates_. GD is extremely sensitive to the choice of learning rates. If the learning rate is very small, the training process take an extremely long time. For larger learning rates, GD can diverge and give poor results. Furthermore, depending on what the local landscape looks like, we have to modify the learning rates to ensure convergence. Ideally, we would *adaptively* choose the learning rates to match the landscape.
+
+* _GD treats all directions in parameter space uniformly._ Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive.
+
+* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.
+
!split
@@ -820,7 +835,7 @@ Niterations = 1000
m = 100
for iter in range(Niterations):
- gradients = 2.0/m*xb.T.dot(xb.dot(theta)-y)
+ gradients = 2.0/m*xb.T @ ((xb @ theta)-y)
theta -= eta*gradients
print("theta frm own gd")
print(theta)
@@ -844,7 +859,7 @@ for epoch in range(n_epochs):
random_index = np.random.randint(m)
xi = xb[random_index:random_index+1]
yi = y[random_index:random_index+1]
- gradients = 2 * xi.T.dot(xi.dot(theta)-yi)
+ gradients = 2 * xi.T @ ((xi @ theta)-yi)
eta = learning_schedule(epoch*m+i)
theta = theta - eta*gradients
print("theta from own sdg")
@@ -866,21 +881,6 @@ plt.show()
===== Logistic Regression example =====
-!split
-===== Using gradient descent methods, limitations =====
-
-* _Gradient descent (GD) finds local minima of our function_. Since the GD algorithm is deterministic, if it converges, it will converge to a local minimum of our energy function. Because in ML we are often dealing with extremely rugged landscapes with many local minima, this can lead to poor performance.
-
-* _GD is sensitive to initial conditions_. One consequence of the local nature of GD is that initial conditions matter. Depending on where one starts, one will end up at a different local minima. Therefore, it is very important to think about how one initializes the training process. This is true for GD as well as more complicated variants of GD.
-
-* _Gradients are computationally expensive to calculate for large datasets_. In many cases in statistics and ML, the energy function is a sum of terms, with one term for each data point. For example, in linear regression, $E \propto \sum_{i=1}^n (y_i - \mathbf{w}^T\cdot\mathbf{x}_i)^2$; for logistic regression, the square error is replaced by the cross entropy. To calculate the gradient we have to sum over *all* $n$ data points. Doing this at every GD step becomes extremely computationally expensive. An ingenious solution to this, is to calculate the gradients using small subsets of the data called ``mini batches''. This has the added benefit of introducing stochasticity into our algorithm.
-
-* _GD is very sensitive to choices of learning rates_. GD is extremely sensitive to the choice of learning rates. If the learning rate is very small, the training process take an extremely long time. For larger learning rates, GD can diverge and give poor results. Furthermore, depending on what the local landscape looks like, we have to modify the learning rates to ensure convergence. Ideally, we would *adaptively* choose the learning rates to match the landscape.
-
-* _GD treats all directions in parameter space uniformly._ Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive.
-
-* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.
-
!split
===== Momentum based GD =====
diff --git a/doc/src/Splines/logsgd.py b/doc/src/Splines/logsgd.py
index e78691984..7adc63c96 100644
--- a/doc/src/Splines/logsgd.py
+++ b/doc/src/Splines/logsgd.py
@@ -1,34 +1,34 @@
from math import exp, sqrt
from random import random, seed
import numpy as np
+from sklearn import datasets, linear_model
import matplotlib.pyplot as plt
from sklearn.linear_model import SGDRegressor
-x = 2*np.random.rand(100,1)
-y = 4+3*x#+np.random.randn(100,1)
-
-xb = np.c_[np.ones((100,1)), x]
+np.random.seed(0)
+X, y = datasets.make_moons(200, noise=0.20)
+#X = np.c_[np.ones((100,1)), x]
sgdreg = SGDRegressor(max_iter = 50, penalty=None, eta0=0.1)
-sgdreg.fit(x,y.ravel())
+sgdreg.fit(X,y.ravel())
print("sgdreg from scikit")
print(sgdreg.intercept_, sgdreg.coef_)
-
+"""
theta = np.random.randn(2,1)
eta = 0.1
Niterations = 1000
m = 100
for iter in range(Niterations):
- gradients = 2.0/m*xb.T @ ((xb @ theta)-y)
+ gradients = 2.0/m*X.T @ ((X @ theta)-y)
theta -= eta*gradients
print("theta from own gd")
print(theta)
xnew = np.array([[0],[2]])
-xbnew = np.c_[np.ones((2,1)), xnew]
-ypredict = xbnew.dot(theta)
+Xnew = np.c_[np.ones((2,1)), xnew]
+ypredict = Xnew.dot(theta)
n_epochs = 50
t0, t1 = 5, 50
@@ -41,7 +41,7 @@ theta = np.random.randn(2,1)
for epoch in range(n_epochs):
for i in range(m):
random_index = np.random.randint(m)
- xi = xb[random_index:random_index+1]
+ xi = X[random_index:random_index+1]
yi = y[random_index:random_index+1]
gradients = 2 * xi.T @ ((xi @ theta)-yi)
eta = learning_schedule(epoch*m+i)
@@ -56,3 +56,4 @@ plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.title(r'Random numbers ')
plt.show()
+"""
Stochastic Gradient Descent
+Using gradient descent methods, limitations
+
+
+
+Stochastic Gradient Descent
Computation of gradients
+Computation of gradients
SGD example
+SGD example
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -973,7 +987,7 @@ $$
The gradient step
+The gradient step
Simple example code
+Simple example code
When do we stop?
+When do we stop?
Slightly different approach
+Slightly different approach
Program for stochastic gradient
+Program for stochastic gradient
Logistic Regression example
-Using gradient descent methods, limitations
-
-
-
+Logistic Regression example
Sep 25, 2019
Sep 26, 2019
@@ -906,7 +906,20 @@ plt.show()
-Stochastic Gradient Descent
+Using gradient descent methods, limitations
+
+
+
+
+
+
+Stochastic Gradient Descent
-Computation of gradients
+Computation of gradients
-SGD example
+SGD example
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -968,7 +981,7 @@ $$
-The gradient step
+The gradient step
-Simple example code
+Simple example code
-When do we stop?
+When do we stop?
-Slightly different approach
+Slightly different approach
-Program for stochastic gradient
+Program for stochastic gradient
-Logistic Regression example
+Logistic Regression example
-Using gradient descent methods, limitations
-
-
-
-
-
-
Momentum based GD
Sep 25, 2019
Sep 26, 2019
@@ -911,7 +911,20 @@ plt.show()
-Stochastic Gradient Descent
+Using gradient descent methods, limitations
+
+
+
+
+
+
+Stochastic Gradient Descent
-Computation of gradients
+Computation of gradients
-SGD example
+SGD example
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -973,7 +986,7 @@ $$
-The gradient step
+The gradient step
-Simple example code
+Simple example code
-When do we stop?
+When do we stop?
-Slightly different approach
+Slightly different approach
-Program for stochastic gradient
+Program for stochastic gradient
-Logistic Regression example
+Logistic Regression example
-Using gradient descent methods, limitations
-
-
-
-
-
-
Momentum based GD