fixed some typos in GD
This commit is contained in:
mhjensen
@@ -625,6 +625,21 @@ plt.show()
|
||||
|
||||
!ec
|
||||
|
||||
!split
|
||||
===== Using gradient descent methods, limitations =====
|
||||
|
||||
* _Gradient descent (GD) finds local minima of our function_. Since the GD algorithm is deterministic, if it converges, it will converge to a local minimum of our energy function. Because in ML we are often dealing with extremely rugged landscapes with many local minima, this can lead to poor performance.
|
||||
|
||||
* _GD is sensitive to initial conditions_. One consequence of the local nature of GD is that initial conditions matter. Depending on where one starts, one will end up at a different local minima. Therefore, it is very important to think about how one initializes the training process. This is true for GD as well as more complicated variants of GD.
|
||||
|
||||
* _Gradients are computationally expensive to calculate for large datasets_. In many cases in statistics and ML, the energy function is a sum of terms, with one term for each data point. For example, in linear regression, $E \propto \sum_{i=1}^n (y_i - \mathbf{w}^T\cdot\mathbf{x}_i)^2$; for logistic regression, the square error is replaced by the cross entropy. To calculate the gradient we have to sum over *all* $n$ data points. Doing this at every GD step becomes extremely computationally expensive. An ingenious solution to this, is to calculate the gradients using small subsets of the data called ``mini batches''. This has the added benefit of introducing stochasticity into our algorithm.
|
||||
|
||||
* _GD is very sensitive to choices of learning rates_. GD is extremely sensitive to the choice of learning rates. If the learning rate is very small, the training process take an extremely long time. For larger learning rates, GD can diverge and give poor results. Furthermore, depending on what the local landscape looks like, we have to modify the learning rates to ensure convergence. Ideally, we would *adaptively* choose the learning rates to match the landscape.
|
||||
|
||||
* _GD treats all directions in parameter space uniformly._ Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive.
|
||||
|
||||
* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.
|
||||
|
||||
|
||||
|
||||
!split
|
||||
@@ -820,7 +835,7 @@ Niterations = 1000
|
||||
m = 100
|
||||
|
||||
for iter in range(Niterations):
|
||||
gradients = 2.0/m*xb.T.dot(xb.dot(theta)-y)
|
||||
gradients = 2.0/m*xb.T @ ((xb @ theta)-y)
|
||||
theta -= eta*gradients
|
||||
print("theta frm own gd")
|
||||
print(theta)
|
||||
@@ -844,7 +859,7 @@ for epoch in range(n_epochs):
|
||||
random_index = np.random.randint(m)
|
||||
xi = xb[random_index:random_index+1]
|
||||
yi = y[random_index:random_index+1]
|
||||
gradients = 2 * xi.T.dot(xi.dot(theta)-yi)
|
||||
gradients = 2 * xi.T @ ((xi @ theta)-yi)
|
||||
eta = learning_schedule(epoch*m+i)
|
||||
theta = theta - eta*gradients
|
||||
print("theta from own sdg")
|
||||
@@ -866,21 +881,6 @@ plt.show()
|
||||
===== Logistic Regression example =====
|
||||
|
||||
|
||||
!split
|
||||
===== Using gradient descent methods, limitations =====
|
||||
|
||||
* _Gradient descent (GD) finds local minima of our function_. Since the GD algorithm is deterministic, if it converges, it will converge to a local minimum of our energy function. Because in ML we are often dealing with extremely rugged landscapes with many local minima, this can lead to poor performance.
|
||||
|
||||
* _GD is sensitive to initial conditions_. One consequence of the local nature of GD is that initial conditions matter. Depending on where one starts, one will end up at a different local minima. Therefore, it is very important to think about how one initializes the training process. This is true for GD as well as more complicated variants of GD.
|
||||
|
||||
* _Gradients are computationally expensive to calculate for large datasets_. In many cases in statistics and ML, the energy function is a sum of terms, with one term for each data point. For example, in linear regression, $E \propto \sum_{i=1}^n (y_i - \mathbf{w}^T\cdot\mathbf{x}_i)^2$; for logistic regression, the square error is replaced by the cross entropy. To calculate the gradient we have to sum over *all* $n$ data points. Doing this at every GD step becomes extremely computationally expensive. An ingenious solution to this, is to calculate the gradients using small subsets of the data called ``mini batches''. This has the added benefit of introducing stochasticity into our algorithm.
|
||||
|
||||
* _GD is very sensitive to choices of learning rates_. GD is extremely sensitive to the choice of learning rates. If the learning rate is very small, the training process take an extremely long time. For larger learning rates, GD can diverge and give poor results. Furthermore, depending on what the local landscape looks like, we have to modify the learning rates to ensure convergence. Ideally, we would *adaptively* choose the learning rates to match the landscape.
|
||||
|
||||
* _GD treats all directions in parameter space uniformly._ Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive.
|
||||
|
||||
* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.
|
||||
|
||||
|
||||
!split
|
||||
===== Momentum based GD =====
|
||||
|
||||
+11
-10
@@ -1,34 +1,34 @@
|
||||
from math import exp, sqrt
|
||||
from random import random, seed
|
||||
import numpy as np
|
||||
from sklearn import datasets, linear_model
|
||||
import matplotlib.pyplot as plt
|
||||
from sklearn.linear_model import SGDRegressor
|
||||
|
||||
x = 2*np.random.rand(100,1)
|
||||
y = 4+3*x#+np.random.randn(100,1)
|
||||
|
||||
xb = np.c_[np.ones((100,1)), x]
|
||||
np.random.seed(0)
|
||||
X, y = datasets.make_moons(200, noise=0.20)
|
||||
#X = np.c_[np.ones((100,1)), x]
|
||||
|
||||
sgdreg = SGDRegressor(max_iter = 50, penalty=None, eta0=0.1)
|
||||
sgdreg.fit(x,y.ravel())
|
||||
sgdreg.fit(X,y.ravel())
|
||||
print("sgdreg from scikit")
|
||||
print(sgdreg.intercept_, sgdreg.coef_)
|
||||
|
||||
|
||||
"""
|
||||
theta = np.random.randn(2,1)
|
||||
eta = 0.1
|
||||
Niterations = 1000
|
||||
m = 100
|
||||
|
||||
for iter in range(Niterations):
|
||||
gradients = 2.0/m*xb.T @ ((xb @ theta)-y)
|
||||
gradients = 2.0/m*X.T @ ((X @ theta)-y)
|
||||
theta -= eta*gradients
|
||||
print("theta from own gd")
|
||||
print(theta)
|
||||
|
||||
xnew = np.array([[0],[2]])
|
||||
xbnew = np.c_[np.ones((2,1)), xnew]
|
||||
ypredict = xbnew.dot(theta)
|
||||
Xnew = np.c_[np.ones((2,1)), xnew]
|
||||
ypredict = Xnew.dot(theta)
|
||||
|
||||
n_epochs = 50
|
||||
t0, t1 = 5, 50
|
||||
@@ -41,7 +41,7 @@ theta = np.random.randn(2,1)
|
||||
for epoch in range(n_epochs):
|
||||
for i in range(m):
|
||||
random_index = np.random.randint(m)
|
||||
xi = xb[random_index:random_index+1]
|
||||
xi = X[random_index:random_index+1]
|
||||
yi = y[random_index:random_index+1]
|
||||
gradients = 2 * xi.T @ ((xi @ theta)-yi)
|
||||
eta = learning_schedule(epoch*m+i)
|
||||
@@ -56,3 +56,4 @@ plt.xlabel(r'$x$')
|
||||
plt.ylabel(r'$y$')
|
||||
plt.title(r'Random numbers ')
|
||||
plt.show()
|
||||
"""
|
||||
|
||||
Reference in New Issue
Block a user