From d478ce2e57ddb56f41010048493d4763b4a09fbd Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Tue, 27 Sep 2022 09:29:38 +0200 Subject: [PATCH] update week39 --- doc/pub/week39/html/week39-bs.html | 46 +- doc/pub/week39/html/week39-reveal.html | 192 ++--- doc/pub/week39/html/week39-solarized.html | 204 ++--- doc/pub/week39/html/week39.html | 204 ++--- doc/pub/week39/ipynb/ipynb-week39-src.tar.gz | Bin 193 -> 192 bytes doc/pub/week39/ipynb/week39.ipynb | 836 ++++++++++--------- doc/src/week39/week39.do.txt | 168 ++-- 7 files changed, 830 insertions(+), 820 deletions(-) diff --git a/doc/pub/week39/html/week39-bs.html b/doc/pub/week39/html/week39-bs.html index 08bbbb758..83ee34d7e 100644 --- a/doc/pub/week39/html/week39-bs.html +++ b/doc/pub/week39/html/week39-bs.html @@ -153,6 +153,20 @@ doconce format html week39.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'using-gradient-descent-methods-limitations'), + ('Overview video on Stochastic Gradient Descent', + 2, + None, + 'overview-video-on-stochastic-gradient-descent'), + ('Batches and mini-batches', 2, None, 'batches-and-mini-batches'), + ('Stochastic Gradient Descent (SGD)', + 2, + None, + 'stochastic-gradient-descent-sgd'), + ('Stochastic Gradient Descent', + 2, + None, + 'stochastic-gradient-descent'), + ('Computation of gradients', 2, None, 'computation-of-gradients'), ('Stochastic Gradient Descent', 2, None, @@ -174,20 +188,6 @@ doconce format html week39.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'code-with-a-number-of-minibatches-which-varies'), - ('Overview video on Stochastic Gradient Descent', - 2, - None, - 'overview-video-on-stochastic-gradient-descent'), - ('Batches and mini-batches', 2, None, 'batches-and-mini-batches'), - ('Stochastic Gradient Descent (SGD)', - 2, - None, - 'stochastic-gradient-descent-sgd'), - ('Stochastic Gradient Descent', - 2, - None, - 'stochastic-gradient-descent'), - ('Computation of gradients', 2, None, 'computation-of-gradients'), ('SGD example', 2, None, 'sgd-example'), ('The gradient step', 2, None, 'the-gradient-step'), ('Simple example code', 2, None, 'simple-example-code'), @@ -325,20 +325,20 @@ MathJax.Hub.Config({
  • The Hessian matrix for Ridge Regression
  • Program example for gradient descent with Ridge Regression
  • Using gradient descent methods, limitations
    • -
  • Stochastic Gradient Descent
    • -
  • Computation of gradients
    • +
  • Overview video on Stochastic Gradient Descent
    • +
  • Batches and mini-batches
    • +
  • Stochastic Gradient Descent (SGD)
    • +
  • Stochastic Gradient Descent
    • +
  • Computation of gradients
    • +
  • Stochastic Gradient Descent
    • +
  • Computation of gradients
  • SGD example
  • The gradient step
  • Simple example code
  • When do we stop?
  • Slightly different approach
  • Program for stochastic gradient
    • -
  • Code with a Number of Minibatches which varies
    • -
  • Overview video on Stochastic Gradient Descent
    • -
  • Batches and mini-batches
    • -
  • Stochastic Gradient Descent (SGD)
    • -
  • Stochastic Gradient Descent
    • -
  • Computation of gradients
    • +
  • Code with a Number of Minibatches which varies
  • SGD example
  • The gradient step
  • Simple example code
    • @@ -396,7 +396,7 @@ MathJax.Hub.Config({
    -

    Sep 26, 2022

    +

    Sep 27, 2022


    diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html index dde9d35a3..387d4b785 100644 --- a/doc/pub/week39/html/week39-reveal.html +++ b/doc/pub/week39/html/week39-reveal.html @@ -184,7 +184,7 @@ MathJax.Hub.Config({
    -

    Sep 26, 2022

    +

    Sep 27, 2022


    @@ -204,12 +204,12 @@ MathJax.Hub.Config({

    See lecture notes for week 39. -For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. +For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.

    For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.

    -For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11. +For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11.

    These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning.

    @@ -222,6 +222,8 @@ For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5

    Optimization, the central part of any Machine Learning algortithm

    +

    The first few slides here are a repetition from last week.

    +

    Almost every problem in machine learning and data science starts with a dataset \( X \), a model \( g(\beta) \), which is a function of the parameters \( \beta \) and a cost function \( C(X, g(\beta)) \) that allows @@ -1625,6 +1627,97 @@ plt.show()

    +
    +

    Overview video on Stochastic Gradient Descent

    + +What is Stochastic Gradient Descent +
    + +
    +

    Batches and mini-batches

    + +

    In gradient descent we compute the cost function and its gradient for all data points we have.

    + +

    In large-scale applications such as the ILSVRC challenge, the +training data can have on order of millions of examples. Hence, it +seems wasteful to compute the full cost function over the entire +training set in order to perform only a single parameter update. A +very common approach to addressing this challenge is to compute the +gradient over batches of the training data. For example, in current +a typical batch could contain some thousand examples from +an entire training set of several millions. This batch is then used to +perform a parameter update. +

    +
    + +
    +

    Stochastic Gradient Descent (SGD)

    + +

    In stochastic gradient descent, the extreme case is the case where we +have only one batch, that is we include the whole data set. +

    + +

    This process is called Stochastic Gradient +Descent (SGD) (or also sometimes on-line gradient descent). This is +relatively less common to see because in practice due to vectorized +code optimizations it can be computationally much more efficient to +evaluate the gradient for 100 examples, than the gradient for one +example 100 times. Even though SGD technically refers to using a +single example at a time to evaluate the gradient, you will hear +people use the term SGD even when referring to mini-batch gradient +descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD +for “Batch gradient descent” are rare to see), where it is usually +assumed that mini-batches are used. The size of the mini-batch is a +hyperparameter but it is not very common to cross-validate or bootstrap it. It is +usually based on memory constraints (if any), or set to some value, +e.g. 32, 64 or 128. We use powers of 2 in practice because many +vectorized operation implementations work faster when their inputs are +sized in powers of 2. +

    + +

    In our notes with SGD we mean stochastic gradient descent with mini-batches.

    +
    + +
    +

    Stochastic Gradient Descent

    + +

    Stochastic gradient descent (SGD) and variants thereof address some of +the shortcomings of the Gradient descent method discussed above. +

    + +

    The underlying idea of SGD comes from the observation that the cost +function, which we want to minimize, can almost always be written as a +sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \), +

    +

     
    +$$ +C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, +\mathbf{\beta}). +$$ +

     
    +

    + +
    +

    Computation of gradients

    + +

    This in turn means that the gradient can be +computed as a sum over \( i \)-gradients +

    +

     
    +$$ +\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, +\mathbf{\beta}). +$$ +

     
    + +

    Stochasticity/randomness is introduced by only taking the +gradient on a subset of the data called minibatches. If there are \( n \) +data points and the size of each minibatch is \( M \), there will be \( n/M \) +minibatches. We denote these minibatches by \( B_k \) where +\( k=1,\cdots,n/M \). +

    +
    +

    Stochastic Gradient Descent

    @@ -2026,97 +2119,6 @@ plt.show()
    -
    -

    Overview video on Stochastic Gradient Descent

    - -What is Stochastic Gradient Descent -
    - -
    -

    Batches and mini-batches

    - -

    In gradient descent we compute the cost function and its gradient for all data points we have.

    - -

    In large-scale applications such as the ILSVRC challenge, the -training data can have on order of millions of examples. Hence, it -seems wasteful to compute the full cost function over the entire -training set in order to perform only a single parameter update. A -very common approach to addressing this challenge is to compute the -gradient over batches of the training data. For example, in current -a typical batch could contain some thousand examples from -an entire training set of several millions. This batch is then used to -perform a parameter update. -

    -
    - -
    -

    Stochastic Gradient Descent (SGD)

    - -

    In stochastic gradient descent, the extreme case is the case where we -have only one batch, that is we include the whole data set. -

    - -

    This process is called Stochastic Gradient -Descent (SGD) (or also sometimes on-line gradient descent). This is -relatively less common to see because in practice due to vectorized -code optimizations it can be computationally much more efficient to -evaluate the gradient for 100 examples, than the gradient for one -example 100 times. Even though SGD technically refers to using a -single example at a time to evaluate the gradient, you will hear -people use the term SGD even when referring to mini-batch gradient -descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD -for “Batch gradient descent” are rare to see), where it is usually -assumed that mini-batches are used. The size of the mini-batch is a -hyperparameter but it is not very common to cross-validate or bootstrap it. It is -usually based on memory constraints (if any), or set to some value, -e.g. 32, 64 or 128. We use powers of 2 in practice because many -vectorized operation implementations work faster when their inputs are -sized in powers of 2. -

    - -

    In our notes with SGD we mean stochastic gradient descent with mini-batches.

    -
    - -
    -

    Stochastic Gradient Descent

    - -

    Stochastic gradient descent (SGD) and variants thereof address some of -the shortcomings of the Gradient descent method discussed above. -

    - -

    The underlying idea of SGD comes from the observation that the cost -function, which we want to minimize, can almost always be written as a -sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \), -

    -

     
    -$$ -C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, -\mathbf{\beta}). -$$ -

     
    -

    - -
    -

    Computation of gradients

    - -

    This in turn means that the gradient can be -computed as a sum over \( i \)-gradients -

    -

     
    -$$ -\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}). -$$ -

     
    - -

    Stochasticity/randomness is introduced by only taking the -gradient on a subset of the data called minibatches. If there are \( n \) -data points and the size of each minibatch is \( M \), there will be \( n/M \) -minibatches. We denote these minibatches by \( B_k \) where -\( k=1,\cdots,n/M \). -

    -
    -

    SGD example

    @@ -2397,7 +2399,7 @@ plt.show()

    In the above code, we have use replacement in setting up the mini-batches. The discussion here may be -useful. More material will be added later. +useful.

    diff --git a/doc/pub/week39/html/week39-solarized.html b/doc/pub/week39/html/week39-solarized.html index a1b285df9..d6a46d51d 100644 --- a/doc/pub/week39/html/week39-solarized.html +++ b/doc/pub/week39/html/week39-solarized.html @@ -180,6 +180,20 @@ div.toc p,a { 2, None, 'using-gradient-descent-methods-limitations'), + ('Overview video on Stochastic Gradient Descent', + 2, + None, + 'overview-video-on-stochastic-gradient-descent'), + ('Batches and mini-batches', 2, None, 'batches-and-mini-batches'), + ('Stochastic Gradient Descent (SGD)', + 2, + None, + 'stochastic-gradient-descent-sgd'), + ('Stochastic Gradient Descent', + 2, + None, + 'stochastic-gradient-descent'), + ('Computation of gradients', 2, None, 'computation-of-gradients'), ('Stochastic Gradient Descent', 2, None, @@ -201,20 +215,6 @@ div.toc p,a { 2, None, 'code-with-a-number-of-minibatches-which-varies'), - ('Overview video on Stochastic Gradient Descent', - 2, - None, - 'overview-video-on-stochastic-gradient-descent'), - ('Batches and mini-batches', 2, None, 'batches-and-mini-batches'), - ('Stochastic Gradient Descent (SGD)', - 2, - None, - 'stochastic-gradient-descent-sgd'), - ('Stochastic Gradient Descent', - 2, - None, - 'stochastic-gradient-descent'), - ('Computation of gradients', 2, None, 'computation-of-gradients'), ('SGD example', 2, None, 'sgd-example'), ('The gradient step', 2, None, 'the-gradient-step'), ('Simple example code', 2, None, 'simple-example-code'), @@ -311,7 +311,7 @@ MathJax.Hub.Config({
    -

    Sep 26, 2022

    +

    Sep 27, 2022


    @@ -324,12 +324,12 @@ MathJax.Hub.Config({
  • Reading recommendations:

    • See lecture notes for week 39. -For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. +For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.

    For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.

    -For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11. +For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11.

    These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning.











    @@ -340,6 +340,8 @@ For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5









    Optimization, the central part of any Machine Learning algortithm

    +

    The first few slides here are a repetition from last week.

    +

    Almost every problem in machine learning and data science starts with a dataset \( X \), a model \( g(\beta) \), which is a function of the parameters \( \beta \) and a cost function \( C(X, g(\beta)) \) that allows @@ -1596,6 +1598,89 @@ plt.show()

  • GD treats all directions in parameter space uniformly. Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive.
  • GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.
    • +









    +

    Overview video on Stochastic Gradient Descent

    + +What is Stochastic Gradient Descent + +









    +

    Batches and mini-batches

    + +

    In gradient descent we compute the cost function and its gradient for all data points we have.

    + +

    In large-scale applications such as the ILSVRC challenge, the +training data can have on order of millions of examples. Hence, it +seems wasteful to compute the full cost function over the entire +training set in order to perform only a single parameter update. A +very common approach to addressing this challenge is to compute the +gradient over batches of the training data. For example, in current +a typical batch could contain some thousand examples from +an entire training set of several millions. This batch is then used to +perform a parameter update. +

    + +









    +

    Stochastic Gradient Descent (SGD)

    + +

    In stochastic gradient descent, the extreme case is the case where we +have only one batch, that is we include the whole data set. +

    + +

    This process is called Stochastic Gradient +Descent (SGD) (or also sometimes on-line gradient descent). This is +relatively less common to see because in practice due to vectorized +code optimizations it can be computationally much more efficient to +evaluate the gradient for 100 examples, than the gradient for one +example 100 times. Even though SGD technically refers to using a +single example at a time to evaluate the gradient, you will hear +people use the term SGD even when referring to mini-batch gradient +descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD +for “Batch gradient descent” are rare to see), where it is usually +assumed that mini-batches are used. The size of the mini-batch is a +hyperparameter but it is not very common to cross-validate or bootstrap it. It is +usually based on memory constraints (if any), or set to some value, +e.g. 32, 64 or 128. We use powers of 2 in practice because many +vectorized operation implementations work faster when their inputs are +sized in powers of 2. +

    + +

    In our notes with SGD we mean stochastic gradient descent with mini-batches.

    + +









    +

    Stochastic Gradient Descent

    + +

    Stochastic gradient descent (SGD) and variants thereof address some of +the shortcomings of the Gradient descent method discussed above. +

    + +

    The underlying idea of SGD comes from the observation that the cost +function, which we want to minimize, can almost always be written as a +sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \), +

    +$$ +C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, +\mathbf{\beta}). +$$ + + +









    +

    Computation of gradients

    + +

    This in turn means that the gradient can be +computed as a sum over \( i \)-gradients +

    +$$ +\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, +\mathbf{\beta}). +$$ + +

    Stochasticity/randomness is introduced by only taking the +gradient on a subset of the data called minibatches. If there are \( n \) +data points and the size of each minibatch is \( M \), there will be \( n/M \) +minibatches. We denote these minibatches by \( B_k \) where +\( k=1,\cdots,n/M \). +

    +









    Stochastic Gradient Descent

    @@ -1982,89 +2067,6 @@ plt.show() -









    -

    Overview video on Stochastic Gradient Descent

    - -What is Stochastic Gradient Descent - -









    -

    Batches and mini-batches

    - -

    In gradient descent we compute the cost function and its gradient for all data points we have.

    - -

    In large-scale applications such as the ILSVRC challenge, the -training data can have on order of millions of examples. Hence, it -seems wasteful to compute the full cost function over the entire -training set in order to perform only a single parameter update. A -very common approach to addressing this challenge is to compute the -gradient over batches of the training data. For example, in current -a typical batch could contain some thousand examples from -an entire training set of several millions. This batch is then used to -perform a parameter update. -

    - -









    -

    Stochastic Gradient Descent (SGD)

    - -

    In stochastic gradient descent, the extreme case is the case where we -have only one batch, that is we include the whole data set. -

    - -

    This process is called Stochastic Gradient -Descent (SGD) (or also sometimes on-line gradient descent). This is -relatively less common to see because in practice due to vectorized -code optimizations it can be computationally much more efficient to -evaluate the gradient for 100 examples, than the gradient for one -example 100 times. Even though SGD technically refers to using a -single example at a time to evaluate the gradient, you will hear -people use the term SGD even when referring to mini-batch gradient -descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD -for “Batch gradient descent” are rare to see), where it is usually -assumed that mini-batches are used. The size of the mini-batch is a -hyperparameter but it is not very common to cross-validate or bootstrap it. It is -usually based on memory constraints (if any), or set to some value, -e.g. 32, 64 or 128. We use powers of 2 in practice because many -vectorized operation implementations work faster when their inputs are -sized in powers of 2. -

    - -

    In our notes with SGD we mean stochastic gradient descent with mini-batches.

    - -









    -

    Stochastic Gradient Descent

    - -

    Stochastic gradient descent (SGD) and variants thereof address some of -the shortcomings of the Gradient descent method discussed above. -

    - -

    The underlying idea of SGD comes from the observation that the cost -function, which we want to minimize, can almost always be written as a -sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \), -

    -$$ -C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, -\mathbf{\beta}). -$$ - - -









    -

    Computation of gradients

    - -

    This in turn means that the gradient can be -computed as a sum over \( i \)-gradients -

    -$$ -\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}). -$$ - -

    Stochasticity/randomness is introduced by only taking the -gradient on a subset of the data called minibatches. If there are \( n \) -data points and the size of each minibatch is \( M \), there will be \( n/M \) -minibatches. We denote these minibatches by \( B_k \) where -\( k=1,\cdots,n/M \). -

    -









    SGD example

    @@ -2335,7 +2337,7 @@ plt.show()

    In the above code, we have use replacement in setting up the mini-batches. The discussion here may be -useful. More material will be added later. +useful.











    diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html index c817c9cf9..d1efb903c 100644 --- a/doc/pub/week39/html/week39.html +++ b/doc/pub/week39/html/week39.html @@ -257,6 +257,20 @@ div.toc p,a { 2, None, 'using-gradient-descent-methods-limitations'), + ('Overview video on Stochastic Gradient Descent', + 2, + None, + 'overview-video-on-stochastic-gradient-descent'), + ('Batches and mini-batches', 2, None, 'batches-and-mini-batches'), + ('Stochastic Gradient Descent (SGD)', + 2, + None, + 'stochastic-gradient-descent-sgd'), + ('Stochastic Gradient Descent', + 2, + None, + 'stochastic-gradient-descent'), + ('Computation of gradients', 2, None, 'computation-of-gradients'), ('Stochastic Gradient Descent', 2, None, @@ -278,20 +292,6 @@ div.toc p,a { 2, None, 'code-with-a-number-of-minibatches-which-varies'), - ('Overview video on Stochastic Gradient Descent', - 2, - None, - 'overview-video-on-stochastic-gradient-descent'), - ('Batches and mini-batches', 2, None, 'batches-and-mini-batches'), - ('Stochastic Gradient Descent (SGD)', - 2, - None, - 'stochastic-gradient-descent-sgd'), - ('Stochastic Gradient Descent', - 2, - None, - 'stochastic-gradient-descent'), - ('Computation of gradients', 2, None, 'computation-of-gradients'), ('SGD example', 2, None, 'sgd-example'), ('The gradient step', 2, None, 'the-gradient-step'), ('Simple example code', 2, None, 'simple-example-code'), @@ -388,7 +388,7 @@ MathJax.Hub.Config({
    -

    Sep 26, 2022

    +

    Sep 27, 2022


    @@ -401,12 +401,12 @@ MathJax.Hub.Config({
  • Reading recommendations:

    • See lecture notes for week 39. -For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. +For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.

    For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.

    -For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11. +For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11.

    These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning.











    @@ -417,6 +417,8 @@ For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5









    Optimization, the central part of any Machine Learning algortithm

    +

    The first few slides here are a repetition from last week.

    +

    Almost every problem in machine learning and data science starts with a dataset \( X \), a model \( g(\beta) \), which is a function of the parameters \( \beta \) and a cost function \( C(X, g(\beta)) \) that allows @@ -1673,6 +1675,89 @@ plt.show()

  • GD treats all directions in parameter space uniformly. Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive.
  • GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.
    • +









    +

    Overview video on Stochastic Gradient Descent

    + +What is Stochastic Gradient Descent + +









    +

    Batches and mini-batches

    + +

    In gradient descent we compute the cost function and its gradient for all data points we have.

    + +

    In large-scale applications such as the ILSVRC challenge, the +training data can have on order of millions of examples. Hence, it +seems wasteful to compute the full cost function over the entire +training set in order to perform only a single parameter update. A +very common approach to addressing this challenge is to compute the +gradient over batches of the training data. For example, in current +a typical batch could contain some thousand examples from +an entire training set of several millions. This batch is then used to +perform a parameter update. +

    + +









    +

    Stochastic Gradient Descent (SGD)

    + +

    In stochastic gradient descent, the extreme case is the case where we +have only one batch, that is we include the whole data set. +

    + +

    This process is called Stochastic Gradient +Descent (SGD) (or also sometimes on-line gradient descent). This is +relatively less common to see because in practice due to vectorized +code optimizations it can be computationally much more efficient to +evaluate the gradient for 100 examples, than the gradient for one +example 100 times. Even though SGD technically refers to using a +single example at a time to evaluate the gradient, you will hear +people use the term SGD even when referring to mini-batch gradient +descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD +for “Batch gradient descent” are rare to see), where it is usually +assumed that mini-batches are used. The size of the mini-batch is a +hyperparameter but it is not very common to cross-validate or bootstrap it. It is +usually based on memory constraints (if any), or set to some value, +e.g. 32, 64 or 128. We use powers of 2 in practice because many +vectorized operation implementations work faster when their inputs are +sized in powers of 2. +

    + +

    In our notes with SGD we mean stochastic gradient descent with mini-batches.

    + +









    +

    Stochastic Gradient Descent

    + +

    Stochastic gradient descent (SGD) and variants thereof address some of +the shortcomings of the Gradient descent method discussed above. +

    + +

    The underlying idea of SGD comes from the observation that the cost +function, which we want to minimize, can almost always be written as a +sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \), +

    +$$ +C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, +\mathbf{\beta}). +$$ + + +









    +

    Computation of gradients

    + +

    This in turn means that the gradient can be +computed as a sum over \( i \)-gradients +

    +$$ +\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, +\mathbf{\beta}). +$$ + +

    Stochasticity/randomness is introduced by only taking the +gradient on a subset of the data called minibatches. If there are \( n \) +data points and the size of each minibatch is \( M \), there will be \( n/M \) +minibatches. We denote these minibatches by \( B_k \) where +\( k=1,\cdots,n/M \). +

    +









    Stochastic Gradient Descent

    @@ -2059,89 +2144,6 @@ plt.show() -









    -

    Overview video on Stochastic Gradient Descent

    - -What is Stochastic Gradient Descent - -









    -

    Batches and mini-batches

    - -

    In gradient descent we compute the cost function and its gradient for all data points we have.

    - -

    In large-scale applications such as the ILSVRC challenge, the -training data can have on order of millions of examples. Hence, it -seems wasteful to compute the full cost function over the entire -training set in order to perform only a single parameter update. A -very common approach to addressing this challenge is to compute the -gradient over batches of the training data. For example, in current -a typical batch could contain some thousand examples from -an entire training set of several millions. This batch is then used to -perform a parameter update. -

    - -









    -

    Stochastic Gradient Descent (SGD)

    - -

    In stochastic gradient descent, the extreme case is the case where we -have only one batch, that is we include the whole data set. -

    - -

    This process is called Stochastic Gradient -Descent (SGD) (or also sometimes on-line gradient descent). This is -relatively less common to see because in practice due to vectorized -code optimizations it can be computationally much more efficient to -evaluate the gradient for 100 examples, than the gradient for one -example 100 times. Even though SGD technically refers to using a -single example at a time to evaluate the gradient, you will hear -people use the term SGD even when referring to mini-batch gradient -descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD -for “Batch gradient descent” are rare to see), where it is usually -assumed that mini-batches are used. The size of the mini-batch is a -hyperparameter but it is not very common to cross-validate or bootstrap it. It is -usually based on memory constraints (if any), or set to some value, -e.g. 32, 64 or 128. We use powers of 2 in practice because many -vectorized operation implementations work faster when their inputs are -sized in powers of 2. -

    - -

    In our notes with SGD we mean stochastic gradient descent with mini-batches.

    - -









    -

    Stochastic Gradient Descent

    - -

    Stochastic gradient descent (SGD) and variants thereof address some of -the shortcomings of the Gradient descent method discussed above. -

    - -

    The underlying idea of SGD comes from the observation that the cost -function, which we want to minimize, can almost always be written as a -sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \), -

    -$$ -C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, -\mathbf{\beta}). -$$ - - -









    -

    Computation of gradients

    - -

    This in turn means that the gradient can be -computed as a sum over \( i \)-gradients -

    -$$ -\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}). -$$ - -

    Stochasticity/randomness is introduced by only taking the -gradient on a subset of the data called minibatches. If there are \( n \) -data points and the size of each minibatch is \( M \), there will be \( n/M \) -minibatches. We denote these minibatches by \( B_k \) where -\( k=1,\cdots,n/M \). -

    -









    SGD example

    @@ -2412,7 +2414,7 @@ plt.show()

    In the above code, we have use replacement in setting up the mini-batches. The discussion here may be -useful. More material will be added later. +useful.











    diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz index a07bbc21cf42f32d01213b36c63bf63ce3b85a43..403824ba80d3e57fc7b3dccb7aa8641a92b844d8 100644 GIT binary patch literal 192 zcmV;x06+g9iwFQxrZQsy1MSaC3c@fD2H>uHia9~av*J>)3m1Zj7f5Mpqc*8YiuU&Q z0lHG$6cO@meuf!_nSHieZ?nYTeKZ?{P|6q#xk&k#h)neeV-A>6%s8Q#5I~r6f+9fk zo%GT=FKmBGU7e(MQooz)$I9}?E8Gu)&+(`BNXPUlb< u>4p|qd2N-IAanuHia9|^nl#0wU>7a~5igL^)W+JRCMnw6 z+Xv`MaZ^OdxA_@n7-kOHdb7(ScXz>J5JHSn7&B$blqjC-3C0OvN-&LMmLdiSlZ2oE z&~hiebk=b*oYGWhp{!8v=7zDd{IF+!1)ljQj+L~q-S@823Y2yt*SdzA5H(pvvh7t4 vg%);fff3hE8UbAOz>7jUsTIG3tZ~RrOh#00;m8On6v0 diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index 3759e5fe5..59aea6e97 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "0c90f76e", + "id": "624309b8", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "7d3b7765", + "id": "047a991a", "metadata": { "editable": true }, @@ -22,14 +22,14 @@ "# Week 39: Optimization and Gradient Methods\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", "\n", - "Date: **Sep 26, 2022**\n", + "Date: **Sep 27, 2022**\n", "\n", "Copyright 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license" ] }, { "cell_type": "markdown", - "id": "48b123e9", + "id": "3001fadb", "metadata": { "editable": true }, @@ -43,17 +43,17 @@ "* Reading recommendations:\n", "\n", "See [lecture notes for week 39](https://compphysics.github.io/MachineLearning/doc/web/course.html).\n", - "For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. \n", + "For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. \n", "\n", "For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.\n", "\n", - "**For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11**.\n", + "**For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11**.\n", "These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning." ] }, { "cell_type": "markdown", - "id": "2743d44d", + "id": "b30c33fc", "metadata": { "editable": true }, @@ -65,13 +65,15 @@ }, { "cell_type": "markdown", - "id": "cf6792f0", + "id": "73508bc3", "metadata": { "editable": true }, "source": [ "## Optimization, the central part of any Machine Learning algortithm\n", "\n", + "The first few slides here are a repetition from last week. \n", + "\n", "Almost every problem in machine learning and data science starts with\n", "a dataset $X$, a model $g(\\beta)$, which is a function of the\n", "parameters $\\beta$ and a cost function $C(X, g(\\beta))$ that allows\n", @@ -84,7 +86,7 @@ }, { "cell_type": "markdown", - "id": "1b31a0ae", + "id": "db45f384", "metadata": { "editable": true }, @@ -101,7 +103,7 @@ }, { "cell_type": "markdown", - "id": "b58ee8fa", + "id": "1af235be", "metadata": { "editable": true }, @@ -116,7 +118,7 @@ }, { "cell_type": "markdown", - "id": "45110669", + "id": "9adc53cb", "metadata": { "editable": true }, @@ -126,7 +128,7 @@ }, { "cell_type": "markdown", - "id": "fc1eb384", + "id": "77a0472d", "metadata": { "editable": true }, @@ -142,7 +144,7 @@ }, { "cell_type": "markdown", - "id": "b404bd87", + "id": "7feaa413", "metadata": { "editable": true }, @@ -154,7 +156,7 @@ }, { "cell_type": "markdown", - "id": "bd45f8e5", + "id": "e0e070a9", "metadata": { "editable": true }, @@ -165,7 +167,7 @@ }, { "cell_type": "markdown", - "id": "ae2bb8a6", + "id": "f490b0e0", "metadata": { "editable": true }, @@ -177,7 +179,7 @@ }, { "cell_type": "markdown", - "id": "c30608ab", + "id": "02387ee2", "metadata": { "editable": true }, @@ -187,7 +189,7 @@ }, { "cell_type": "markdown", - "id": "2ffc2104", + "id": "a8fe4a21", "metadata": { "editable": true }, @@ -201,7 +203,7 @@ }, { "cell_type": "markdown", - "id": "d2c7a263", + "id": "eb736dda", "metadata": { "editable": true }, @@ -213,7 +215,7 @@ }, { "cell_type": "markdown", - "id": "4967733a", + "id": "df8cb898", "metadata": { "editable": true }, @@ -223,7 +225,7 @@ }, { "cell_type": "markdown", - "id": "66498b60", + "id": "fd44dff8", "metadata": { "editable": true }, @@ -235,7 +237,7 @@ }, { "cell_type": "markdown", - "id": "3f907c8c", + "id": "4e1ebc91", "metadata": { "editable": true }, @@ -247,7 +249,7 @@ }, { "cell_type": "markdown", - "id": "dfc5152e", + "id": "90ad7450", "metadata": { "editable": true }, @@ -267,7 +269,7 @@ }, { "cell_type": "markdown", - "id": "66cc5141", + "id": "aee0363b", "metadata": { "editable": true }, @@ -283,7 +285,7 @@ }, { "cell_type": "markdown", - "id": "75995b34", + "id": "a51e8a41", "metadata": { "editable": true }, @@ -299,7 +301,7 @@ }, { "cell_type": "markdown", - "id": "629dee63", + "id": "c8d8d8e5", "metadata": { "editable": true }, @@ -310,7 +312,7 @@ }, { "cell_type": "markdown", - "id": "19696110", + "id": "5640b0b9", "metadata": { "editable": true }, @@ -322,7 +324,7 @@ }, { "cell_type": "markdown", - "id": "24dfad4f", + "id": "29b131b5", "metadata": { "editable": true }, @@ -332,7 +334,7 @@ }, { "cell_type": "markdown", - "id": "9778ef8c", + "id": "85009a37", "metadata": { "editable": true }, @@ -344,7 +346,7 @@ }, { "cell_type": "markdown", - "id": "6cb54622", + "id": "f831bc88", "metadata": { "editable": true }, @@ -354,7 +356,7 @@ }, { "cell_type": "markdown", - "id": "594889e0", + "id": "0fc71a7f", "metadata": { "editable": true }, @@ -366,7 +368,7 @@ }, { "cell_type": "markdown", - "id": "808509d5", + "id": "51ffa803", "metadata": { "editable": true }, @@ -388,7 +390,7 @@ }, { "cell_type": "markdown", - "id": "f56b9913", + "id": "a646a713", "metadata": { "editable": true }, @@ -401,7 +403,7 @@ }, { "cell_type": "markdown", - "id": "a893728d", + "id": "ad162944", "metadata": { "editable": true }, @@ -414,7 +416,7 @@ }, { "cell_type": "markdown", - "id": "3012662d", + "id": "bf72a75b", "metadata": { "editable": true }, @@ -424,7 +426,7 @@ }, { "cell_type": "markdown", - "id": "75f33e97", + "id": "c3e6fdc5", "metadata": { "editable": true }, @@ -442,7 +444,7 @@ }, { "cell_type": "markdown", - "id": "ccbcdb59", + "id": "29a7c03b", "metadata": { "editable": true }, @@ -452,7 +454,7 @@ }, { "cell_type": "markdown", - "id": "7cab2062", + "id": "60fa3964", "metadata": { "editable": true }, @@ -467,7 +469,7 @@ }, { "cell_type": "markdown", - "id": "2cbcaa24", + "id": "02bd34d4", "metadata": { "editable": true }, @@ -477,7 +479,7 @@ }, { "cell_type": "markdown", - "id": "a6e6b703", + "id": "5c2e7766", "metadata": { "editable": true }, @@ -491,7 +493,7 @@ }, { "cell_type": "markdown", - "id": "590ea58c", + "id": "ffd135f1", "metadata": { "editable": true }, @@ -501,7 +503,7 @@ }, { "cell_type": "markdown", - "id": "7740caa6", + "id": "098c35e0", "metadata": { "editable": true }, @@ -515,7 +517,7 @@ }, { "cell_type": "markdown", - "id": "5d507076", + "id": "38bbd32e", "metadata": { "editable": true }, @@ -530,7 +532,7 @@ }, { "cell_type": "markdown", - "id": "80756c72", + "id": "db7d1c20", "metadata": { "editable": true }, @@ -547,7 +549,7 @@ }, { "cell_type": "markdown", - "id": "8d7ca446", + "id": "ec78a232", "metadata": { "editable": true }, @@ -559,7 +561,7 @@ }, { "cell_type": "markdown", - "id": "f1ee3436", + "id": "9d79aa59", "metadata": { "editable": true }, @@ -573,7 +575,7 @@ }, { "cell_type": "markdown", - "id": "8515a36e", + "id": "d352745c", "metadata": { "editable": true }, @@ -588,7 +590,7 @@ }, { "cell_type": "markdown", - "id": "f760f5b5", + "id": "df488a82", "metadata": { "editable": true }, @@ -600,7 +602,7 @@ }, { "cell_type": "markdown", - "id": "7b4dab8b", + "id": "22875ea7", "metadata": { "editable": true }, @@ -611,7 +613,7 @@ }, { "cell_type": "markdown", - "id": "97fb0ea4", + "id": "d19f30cb", "metadata": { "editable": true }, @@ -639,7 +641,7 @@ }, { "cell_type": "markdown", - "id": "cac49404", + "id": "f1b4a5e4", "metadata": { "editable": true }, @@ -661,7 +663,7 @@ }, { "cell_type": "markdown", - "id": "e0711342", + "id": "64ee4acd", "metadata": { "editable": true }, @@ -683,7 +685,7 @@ }, { "cell_type": "markdown", - "id": "a4955ff1", + "id": "7939950d", "metadata": { "editable": true }, @@ -695,7 +697,7 @@ }, { "cell_type": "markdown", - "id": "7be3cf94", + "id": "9ebd13ec", "metadata": { "editable": true }, @@ -732,7 +734,7 @@ }, { "cell_type": "markdown", - "id": "55ba4799", + "id": "e1e93067", "metadata": { "editable": true }, @@ -760,7 +762,7 @@ }, { "cell_type": "markdown", - "id": "8ee1c52a", + "id": "67e2eeb8", "metadata": { "editable": true }, @@ -790,7 +792,7 @@ }, { "cell_type": "markdown", - "id": "cf766c34", + "id": "cdce400d", "metadata": { "editable": true }, @@ -810,7 +812,7 @@ }, { "cell_type": "markdown", - "id": "ff2cc9ee", + "id": "00f0b878", "metadata": { "editable": true }, @@ -822,7 +824,7 @@ }, { "cell_type": "markdown", - "id": "5ff1ecb0", + "id": "b067183e", "metadata": { "editable": true }, @@ -832,7 +834,7 @@ }, { "cell_type": "markdown", - "id": "0c285859", + "id": "023447f2", "metadata": { "editable": true }, @@ -844,7 +846,7 @@ }, { "cell_type": "markdown", - "id": "59a1c170", + "id": "1a29b74d", "metadata": { "editable": true }, @@ -856,7 +858,7 @@ }, { "cell_type": "markdown", - "id": "a3333271", + "id": "c4088348", "metadata": { "editable": true }, @@ -868,7 +870,7 @@ }, { "cell_type": "markdown", - "id": "29b34934", + "id": "3f5cd556", "metadata": { "editable": true }, @@ -880,7 +882,7 @@ }, { "cell_type": "markdown", - "id": "19434cf2", + "id": "49eb9918", "metadata": { "editable": true }, @@ -891,7 +893,7 @@ }, { "cell_type": "markdown", - "id": "3c6ee22f", + "id": "294579e5", "metadata": { "editable": true }, @@ -904,7 +906,7 @@ }, { "cell_type": "markdown", - "id": "ae76233d", + "id": "4e35f70e", "metadata": { "editable": true }, @@ -916,7 +918,7 @@ }, { "cell_type": "markdown", - "id": "f5642412", + "id": "b6055cf1", "metadata": { "editable": true }, @@ -926,7 +928,7 @@ }, { "cell_type": "markdown", - "id": "c41f7b67", + "id": "e4bf26af", "metadata": { "editable": true }, @@ -938,7 +940,7 @@ }, { "cell_type": "markdown", - "id": "c95fd204", + "id": "08110157", "metadata": { "editable": true }, @@ -948,7 +950,7 @@ }, { "cell_type": "markdown", - "id": "6768c8eb", + "id": "d0699ac3", "metadata": { "editable": true }, @@ -959,7 +961,7 @@ }, { "cell_type": "markdown", - "id": "ab55615c", + "id": "19e28e76", "metadata": { "editable": true }, @@ -971,7 +973,7 @@ }, { "cell_type": "markdown", - "id": "ff0cffb2", + "id": "61503485", "metadata": { "editable": true }, @@ -983,7 +985,7 @@ }, { "cell_type": "markdown", - "id": "fdebf121", + "id": "d54ae35a", "metadata": { "editable": true }, @@ -995,7 +997,7 @@ }, { "cell_type": "markdown", - "id": "ac8ba078", + "id": "2a75f182", "metadata": { "editable": true }, @@ -1006,7 +1008,7 @@ }, { "cell_type": "markdown", - "id": "8bb0b0d8", + "id": "e76669ca", "metadata": { "editable": true }, @@ -1017,7 +1019,7 @@ }, { "cell_type": "markdown", - "id": "87cb14dd", + "id": "ddbe6b99", "metadata": { "editable": true }, @@ -1029,7 +1031,7 @@ }, { "cell_type": "markdown", - "id": "2fb15341", + "id": "0ab65253", "metadata": { "editable": true }, @@ -1039,7 +1041,7 @@ }, { "cell_type": "markdown", - "id": "c7df213c", + "id": "343d865c", "metadata": { "editable": true }, @@ -1051,7 +1053,7 @@ }, { "cell_type": "markdown", - "id": "4a027e90", + "id": "69e8fa9d", "metadata": { "editable": true }, @@ -1061,7 +1063,7 @@ }, { "cell_type": "markdown", - "id": "31ee0279", + "id": "260f0bd1", "metadata": { "editable": true }, @@ -1073,7 +1075,7 @@ }, { "cell_type": "markdown", - "id": "bea2d497", + "id": "28d654ef", "metadata": { "editable": true }, @@ -1083,7 +1085,7 @@ }, { "cell_type": "markdown", - "id": "92f0417a", + "id": "f595dbe2", "metadata": { "editable": true }, @@ -1095,7 +1097,7 @@ }, { "cell_type": "markdown", - "id": "a1edf262", + "id": "c84110d2", "metadata": { "editable": true }, @@ -1105,7 +1107,7 @@ }, { "cell_type": "markdown", - "id": "65395c91", + "id": "738d0376", "metadata": { "editable": true }, @@ -1117,7 +1119,7 @@ }, { "cell_type": "markdown", - "id": "53a82d75", + "id": "697bcfaf", "metadata": { "editable": true }, @@ -1128,7 +1130,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "6c2e556c", + "id": "e714b8eb", "metadata": { "collapsed": false, "editable": true @@ -1161,7 +1163,7 @@ }, { "cell_type": "markdown", - "id": "d190cd0d", + "id": "cd9916f9", "metadata": { "editable": true }, @@ -1172,7 +1174,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "1d8d0df6", + "id": "b8644e9d", "metadata": { "collapsed": false, "editable": true @@ -1186,7 +1188,7 @@ }, { "cell_type": "markdown", - "id": "4579d59a", + "id": "99c41a8f", "metadata": { "editable": true }, @@ -1197,7 +1199,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "cead5258", + "id": "ab68472c", "metadata": { "collapsed": false, "editable": true @@ -1210,7 +1212,7 @@ }, { "cell_type": "markdown", - "id": "50a4c9db", + "id": "71ac5c37", "metadata": { "editable": true }, @@ -1221,7 +1223,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "b70d152f", + "id": "8454f18e", "metadata": { "collapsed": false, "editable": true @@ -1239,7 +1241,7 @@ }, { "cell_type": "markdown", - "id": "d3f6888d", + "id": "429802dd", "metadata": { "editable": true }, @@ -1250,7 +1252,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "2a016f4d", + "id": "974e38db", "metadata": { "collapsed": false, "editable": true @@ -1265,7 +1267,7 @@ }, { "cell_type": "markdown", - "id": "32e58c12", + "id": "bd44034d", "metadata": { "editable": true }, @@ -1275,7 +1277,7 @@ }, { "cell_type": "markdown", - "id": "d3da915f", + "id": "6aa2a802", "metadata": { "editable": true }, @@ -1289,7 +1291,7 @@ }, { "cell_type": "markdown", - "id": "f88e8467", + "id": "b21a338d", "metadata": { "editable": true }, @@ -1301,7 +1303,7 @@ }, { "cell_type": "markdown", - "id": "4e0cc2a4", + "id": "36de2035", "metadata": { "editable": true }, @@ -1312,7 +1314,7 @@ }, { "cell_type": "markdown", - "id": "354fc5e3", + "id": "96460d24", "metadata": { "editable": true }, @@ -1324,7 +1326,7 @@ }, { "cell_type": "markdown", - "id": "c213ab2d", + "id": "bcf8f093", "metadata": { "editable": true }, @@ -1335,7 +1337,7 @@ }, { "cell_type": "markdown", - "id": "32f8d9df", + "id": "f25ef4c8", "metadata": { "editable": true }, @@ -1346,7 +1348,7 @@ }, { "cell_type": "markdown", - "id": "c3dc3ab8", + "id": "081d1da3", "metadata": { "editable": true }, @@ -1358,7 +1360,7 @@ }, { "cell_type": "markdown", - "id": "78d86146", + "id": "aa6122e9", "metadata": { "editable": true }, @@ -1368,7 +1370,7 @@ }, { "cell_type": "markdown", - "id": "35aac193", + "id": "ddd0620e", "metadata": { "editable": true }, @@ -1380,7 +1382,7 @@ }, { "cell_type": "markdown", - "id": "bf46eda9", + "id": "53d704bd", "metadata": { "editable": true }, @@ -1392,7 +1394,7 @@ }, { "cell_type": "markdown", - "id": "1d836e8b", + "id": "53101dcb", "metadata": { "editable": true }, @@ -1404,7 +1406,7 @@ }, { "cell_type": "markdown", - "id": "de26406e", + "id": "803ccc03", "metadata": { "editable": true }, @@ -1416,7 +1418,7 @@ }, { "cell_type": "markdown", - "id": "8ec45440", + "id": "abe3b5b0", "metadata": { "editable": true }, @@ -1427,7 +1429,7 @@ }, { "cell_type": "markdown", - "id": "26ae50c9", + "id": "6bf05de5", "metadata": { "editable": true }, @@ -1439,7 +1441,7 @@ }, { "cell_type": "markdown", - "id": "251ff0be", + "id": "1253bcaf", "metadata": { "editable": true }, @@ -1449,7 +1451,7 @@ }, { "cell_type": "markdown", - "id": "90a0c0d7", + "id": "f7671b5f", "metadata": { "editable": true }, @@ -1461,7 +1463,7 @@ }, { "cell_type": "markdown", - "id": "d437b7b6", + "id": "087861cd", "metadata": { "editable": true }, @@ -1471,7 +1473,7 @@ }, { "cell_type": "markdown", - "id": "17fea24a", + "id": "3859c9e2", "metadata": { "editable": true }, @@ -1483,7 +1485,7 @@ }, { "cell_type": "markdown", - "id": "ba05db63", + "id": "c36443ac", "metadata": { "editable": true }, @@ -1503,7 +1505,7 @@ }, { "cell_type": "markdown", - "id": "bc80d1cd", + "id": "4b135acf", "metadata": { "editable": true }, @@ -1515,7 +1517,7 @@ }, { "cell_type": "markdown", - "id": "522c2f42", + "id": "00556f1d", "metadata": { "editable": true }, @@ -1525,7 +1527,7 @@ }, { "cell_type": "markdown", - "id": "b779c16e", + "id": "153a1d22", "metadata": { "editable": true }, @@ -1537,7 +1539,7 @@ }, { "cell_type": "markdown", - "id": "850dde03", + "id": "a3c51787", "metadata": { "editable": true }, @@ -1547,7 +1549,7 @@ }, { "cell_type": "markdown", - "id": "49181750", + "id": "7bc48291", "metadata": { "editable": true }, @@ -1558,7 +1560,7 @@ }, { "cell_type": "markdown", - "id": "46671a60", + "id": "a431587b", "metadata": { "editable": true }, @@ -1570,7 +1572,7 @@ }, { "cell_type": "markdown", - "id": "acee18bb", + "id": "0d322f94", "metadata": { "editable": true }, @@ -1582,7 +1584,7 @@ }, { "cell_type": "markdown", - "id": "4b02ac33", + "id": "9382ecc2", "metadata": { "editable": true }, @@ -1594,7 +1596,7 @@ }, { "cell_type": "markdown", - "id": "652272ff", + "id": "66d7625d", "metadata": { "editable": true }, @@ -1607,7 +1609,7 @@ }, { "cell_type": "markdown", - "id": "a30e9915", + "id": "e82b8692", "metadata": { "editable": true }, @@ -1618,7 +1620,7 @@ }, { "cell_type": "markdown", - "id": "5de05d05", + "id": "05180d7b", "metadata": { "editable": true }, @@ -1630,7 +1632,7 @@ }, { "cell_type": "markdown", - "id": "48612da5", + "id": "52a090f3", "metadata": { "editable": true }, @@ -1646,7 +1648,7 @@ }, { "cell_type": "markdown", - "id": "d52a08d9", + "id": "ded09ab0", "metadata": { "editable": true }, @@ -1658,7 +1660,7 @@ }, { "cell_type": "markdown", - "id": "79a68599", + "id": "d92bad7b", "metadata": { "editable": true }, @@ -1669,7 +1671,7 @@ }, { "cell_type": "markdown", - "id": "cf2b78b7", + "id": "a25cad33", "metadata": { "editable": true }, @@ -1681,7 +1683,7 @@ }, { "cell_type": "markdown", - "id": "15df9c04", + "id": "51681d94", "metadata": { "editable": true }, @@ -1691,7 +1693,7 @@ }, { "cell_type": "markdown", - "id": "4c96a39c", + "id": "d0d7c11f", "metadata": { "editable": true }, @@ -1703,7 +1705,7 @@ }, { "cell_type": "markdown", - "id": "bf37e76a", + "id": "470b62a2", "metadata": { "editable": true }, @@ -1713,7 +1715,7 @@ }, { "cell_type": "markdown", - "id": "9ceef519", + "id": "597cd7c3", "metadata": { "editable": true }, @@ -1725,7 +1727,7 @@ }, { "cell_type": "markdown", - "id": "29b8b7c2", + "id": "2d289b36", "metadata": { "editable": true }, @@ -1735,7 +1737,7 @@ }, { "cell_type": "markdown", - "id": "0e165432", + "id": "676a6a8e", "metadata": { "editable": true }, @@ -1747,7 +1749,7 @@ }, { "cell_type": "markdown", - "id": "89a6a6fd", + "id": "fb6b2368", "metadata": { "editable": true }, @@ -1771,7 +1773,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "32fbee0d", + "id": "c8ff1895", "metadata": { "collapsed": false, "editable": true @@ -1784,7 +1786,7 @@ }, { "cell_type": "markdown", - "id": "213f9bb0", + "id": "ba8abbc0", "metadata": { "editable": true }, @@ -1795,7 +1797,7 @@ }, { "cell_type": "markdown", - "id": "974d4f13", + "id": "681fb1ee", "metadata": { "editable": true }, @@ -1807,7 +1809,7 @@ }, { "cell_type": "markdown", - "id": "9dc2eda8", + "id": "3b4b2655", "metadata": { "editable": true }, @@ -1817,7 +1819,7 @@ }, { "cell_type": "markdown", - "id": "c4ffa97a", + "id": "3dd7cb44", "metadata": { "editable": true }, @@ -1829,7 +1831,7 @@ }, { "cell_type": "markdown", - "id": "a688e175", + "id": "963c2869", "metadata": { "editable": true }, @@ -1843,7 +1845,7 @@ }, { "cell_type": "markdown", - "id": "ec7738bd", + "id": "3faae97a", "metadata": { "editable": true }, @@ -1859,7 +1861,7 @@ }, { "cell_type": "markdown", - "id": "54aa68e9", + "id": "f4be1f8f", "metadata": { "editable": true }, @@ -1869,7 +1871,7 @@ }, { "cell_type": "markdown", - "id": "a995e317", + "id": "a62c2f65", "metadata": { "editable": true }, @@ -1881,7 +1883,7 @@ }, { "cell_type": "markdown", - "id": "16ac0b3c", + "id": "bd482d3d", "metadata": { "editable": true }, @@ -1891,7 +1893,7 @@ }, { "cell_type": "markdown", - "id": "dd7615cb", + "id": "e03f97ee", "metadata": { "editable": true }, @@ -1903,7 +1905,7 @@ }, { "cell_type": "markdown", - "id": "173dec79", + "id": "59c746ff", "metadata": { "editable": true }, @@ -1917,7 +1919,7 @@ }, { "cell_type": "markdown", - "id": "fa9dcb45", + "id": "ff621a81", "metadata": { "editable": true }, @@ -1927,7 +1929,7 @@ }, { "cell_type": "markdown", - "id": "2039954a", + "id": "229284c8", "metadata": { "editable": true }, @@ -1938,7 +1940,7 @@ }, { "cell_type": "markdown", - "id": "14d3da82", + "id": "a5632b50", "metadata": { "editable": true }, @@ -1953,7 +1955,7 @@ }, { "cell_type": "markdown", - "id": "77c8694e", + "id": "44f5596d", "metadata": { "editable": true }, @@ -1963,7 +1965,7 @@ }, { "cell_type": "markdown", - "id": "310e14ea", + "id": "92164631", "metadata": { "editable": true }, @@ -1975,7 +1977,7 @@ }, { "cell_type": "markdown", - "id": "808a35b5", + "id": "a48fce93", "metadata": { "editable": true }, @@ -1987,7 +1989,7 @@ }, { "cell_type": "markdown", - "id": "1bb5911f", + "id": "8bbca853", "metadata": { "editable": true }, @@ -2002,7 +2004,7 @@ }, { "cell_type": "markdown", - "id": "13895bc9", + "id": "3c9fd447", "metadata": { "editable": true }, @@ -2015,7 +2017,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "54d45402", + "id": "65e8d8f5", "metadata": { "collapsed": false, "editable": true @@ -2072,7 +2074,7 @@ }, { "cell_type": "markdown", - "id": "8cef05f3", + "id": "321fd022", "metadata": { "editable": true }, @@ -2083,7 +2085,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "0216fdb8", + "id": "5847a934", "metadata": { "collapsed": false, "editable": true @@ -2110,7 +2112,7 @@ }, { "cell_type": "markdown", - "id": "55783fbe", + "id": "68c9136e", "metadata": { "editable": true }, @@ -2122,7 +2124,7 @@ }, { "cell_type": "markdown", - "id": "2a1d53b1", + "id": "0c4580c8", "metadata": { "editable": true }, @@ -2134,7 +2136,7 @@ }, { "cell_type": "markdown", - "id": "7e34a319", + "id": "5e578724", "metadata": { "editable": true }, @@ -2144,7 +2146,7 @@ }, { "cell_type": "markdown", - "id": "f04601ab", + "id": "3bacb190", "metadata": { "editable": true }, @@ -2158,7 +2160,7 @@ }, { "cell_type": "markdown", - "id": "1703d690", + "id": "028de774", "metadata": { "editable": true }, @@ -2168,7 +2170,7 @@ }, { "cell_type": "markdown", - "id": "2c088c07", + "id": "5f75aef9", "metadata": { "editable": true }, @@ -2180,7 +2182,7 @@ }, { "cell_type": "markdown", - "id": "0a8eff61", + "id": "be235465", "metadata": { "editable": true }, @@ -2191,7 +2193,7 @@ }, { "cell_type": "markdown", - "id": "11650d7e", + "id": "cee3e15d", "metadata": { "editable": true }, @@ -2206,7 +2208,7 @@ }, { "cell_type": "markdown", - "id": "9f2bcd4a", + "id": "06592b0e", "metadata": { "editable": true }, @@ -2220,7 +2222,7 @@ }, { "cell_type": "markdown", - "id": "9de5fa7a", + "id": "fbaae00b", "metadata": { "editable": true }, @@ -2231,7 +2233,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "f659a82f", + "id": "c85edc81", "metadata": { "collapsed": false, "editable": true @@ -2292,7 +2294,7 @@ }, { "cell_type": "markdown", - "id": "7d8b24e4", + "id": "4d80fe88", "metadata": { "editable": true }, @@ -2314,7 +2316,73 @@ }, { "cell_type": "markdown", - "id": "e6c62008", + "id": "cba6bc16", + "metadata": { + "editable": true + }, + "source": [ + "## Overview video on Stochastic Gradient Descent\n", + "\n", + "[What is Stochastic Gradient Descent](https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer)" + ] + }, + { + "cell_type": "markdown", + "id": "314db380", + "metadata": { + "editable": true + }, + "source": [ + "## Batches and mini-batches\n", + "\n", + "In gradient descent we compute the cost function and its gradient for all data points we have.\n", + "\n", + "In large-scale applications such as the [ILSVRC challenge](https://www.image-net.org/challenges/LSVRC/), the\n", + "training data can have on order of millions of examples. Hence, it\n", + "seems wasteful to compute the full cost function over the entire\n", + "training set in order to perform only a single parameter update. A\n", + "very common approach to addressing this challenge is to compute the\n", + "gradient over batches of the training data. For example, in current\n", + "a typical batch could contain some thousand examples from\n", + "an entire training set of several millions. This batch is then used to\n", + "perform a parameter update." + ] + }, + { + "cell_type": "markdown", + "id": "a3bd5dc9", + "metadata": { + "editable": true + }, + "source": [ + "## Stochastic Gradient Descent (SGD)\n", + "\n", + "In stochastic gradient descent, the extreme case is the case where we\n", + "have only one batch, that is we include the whole data set.\n", + "\n", + "This process is called Stochastic Gradient\n", + "Descent (SGD) (or also sometimes on-line gradient descent). This is\n", + "relatively less common to see because in practice due to vectorized\n", + "code optimizations it can be computationally much more efficient to\n", + "evaluate the gradient for 100 examples, than the gradient for one\n", + "example 100 times. Even though SGD technically refers to using a\n", + "single example at a time to evaluate the gradient, you will hear\n", + "people use the term SGD even when referring to mini-batch gradient\n", + "descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD\n", + "for “Batch gradient descent” are rare to see), where it is usually\n", + "assumed that mini-batches are used. The size of the mini-batch is a\n", + "hyperparameter but it is not very common to cross-validate or bootstrap it. It is\n", + "usually based on memory constraints (if any), or set to some value,\n", + "e.g. 32, 64 or 128. We use powers of 2 in practice because many\n", + "vectorized operation implementations work faster when their inputs are\n", + "sized in powers of 2.\n", + "\n", + "In our notes with SGD we mean stochastic gradient descent with mini-batches." + ] + }, + { + "cell_type": "markdown", + "id": "a3a4ccdd", "metadata": { "editable": true }, @@ -2331,7 +2399,7 @@ }, { "cell_type": "markdown", - "id": "92140b0e", + "id": "bcd2d9da", "metadata": { "editable": true }, @@ -2344,7 +2412,7 @@ }, { "cell_type": "markdown", - "id": "e7bbda8e", + "id": "e8e31492", "metadata": { "editable": true }, @@ -2357,7 +2425,7 @@ }, { "cell_type": "markdown", - "id": "fc80afa3", + "id": "98a46f0a", "metadata": { "editable": true }, @@ -2370,7 +2438,7 @@ }, { "cell_type": "markdown", - "id": "92119c94", + "id": "842a65e9", "metadata": { "editable": true }, @@ -2384,7 +2452,77 @@ }, { "cell_type": "markdown", - "id": "f1b03d25", + "id": "6076c141", + "metadata": { + "editable": true + }, + "source": [ + "## Stochastic Gradient Descent\n", + "\n", + "Stochastic gradient descent (SGD) and variants thereof address some of\n", + "the shortcomings of the Gradient descent method discussed above.\n", + "\n", + "The underlying idea of SGD comes from the observation that the cost\n", + "function, which we want to minimize, can almost always be written as a\n", + "sum over $n$ data points $\\{\\mathbf{x}_i\\}_{i=1}^n$," + ] + }, + { + "cell_type": "markdown", + "id": "9059febd", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\beta}).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "da676e64", + "metadata": { + "editable": true + }, + "source": [ + "## Computation of gradients\n", + "\n", + "This in turn means that the gradient can be\n", + "computed as a sum over $i$-gradients" + ] + }, + { + "cell_type": "markdown", + "id": "8ec48f17", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\beta}).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "57cafc9b", + "metadata": { + "editable": true + }, + "source": [ + "Stochasticity/randomness is introduced by only taking the\n", + "gradient on a subset of the data called minibatches. If there are $n$\n", + "data points and the size of each minibatch is $M$, there will be $n/M$\n", + "minibatches. We denote these minibatches by $B_k$ where\n", + "$k=1,\\cdots,n/M$." + ] + }, + { + "cell_type": "markdown", + "id": "44578e02", "metadata": { "editable": true }, @@ -2406,7 +2544,7 @@ }, { "cell_type": "markdown", - "id": "1a086625", + "id": "41e35cd9", "metadata": { "editable": true }, @@ -2421,7 +2559,7 @@ }, { "cell_type": "markdown", - "id": "3f4df101", + "id": "26cfd6d9", "metadata": { "editable": true }, @@ -2433,7 +2571,7 @@ }, { "cell_type": "markdown", - "id": "b818a10b", + "id": "fecc043f", "metadata": { "editable": true }, @@ -2446,7 +2584,7 @@ }, { "cell_type": "markdown", - "id": "2d824978", + "id": "f7dad62a", "metadata": { "editable": true }, @@ -2460,7 +2598,7 @@ }, { "cell_type": "markdown", - "id": "73b9b526", + "id": "0a1122c5", "metadata": { "editable": true }, @@ -2471,7 +2609,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "edd945e4", + "id": "e0b0f1e4", "metadata": { "collapsed": false, "editable": true @@ -2496,7 +2634,7 @@ }, { "cell_type": "markdown", - "id": "0b15596f", + "id": "b03d1b70", "metadata": { "editable": true }, @@ -2512,7 +2650,7 @@ }, { "cell_type": "markdown", - "id": "fcce29ec", + "id": "81392e39", "metadata": { "editable": true }, @@ -2533,7 +2671,7 @@ }, { "cell_type": "markdown", - "id": "16bb99e0", + "id": "a35c98fa", "metadata": { "editable": true }, @@ -2556,7 +2694,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "27672e88", + "id": "faa43d57", "metadata": { "collapsed": false, "editable": true @@ -2591,7 +2729,7 @@ }, { "cell_type": "markdown", - "id": "7535b5d0", + "id": "026cdc3d", "metadata": { "editable": true }, @@ -2602,7 +2740,7 @@ { "cell_type": "code", "execution_count": 12, - "id": "a9b982e8", + "id": "36d689f3", "metadata": { "collapsed": false, "editable": true @@ -2678,7 +2816,7 @@ }, { "cell_type": "markdown", - "id": "651c38bc", + "id": "e352b052", "metadata": { "editable": true }, @@ -2688,7 +2826,7 @@ }, { "cell_type": "markdown", - "id": "bb5af5d5", + "id": "6df2d658", "metadata": { "editable": true }, @@ -2701,7 +2839,7 @@ { "cell_type": "code", "execution_count": 13, - "id": "7bc8398a", + "id": "ef94f697", "metadata": { "collapsed": false, "editable": true @@ -2777,143 +2915,7 @@ }, { "cell_type": "markdown", - "id": "ecd51ae9", - "metadata": { - "editable": true - }, - "source": [ - "## Overview video on Stochastic Gradient Descent\n", - "\n", - "[What is Stochastic Gradient Descent](https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer)" - ] - }, - { - "cell_type": "markdown", - "id": "3086a2dc", - "metadata": { - "editable": true - }, - "source": [ - "## Batches and mini-batches\n", - "\n", - "In gradient descent we compute the cost function and its gradient for all data points we have.\n", - "\n", - "In large-scale applications such as the [ILSVRC challenge](https://www.image-net.org/challenges/LSVRC/), the\n", - "training data can have on order of millions of examples. Hence, it\n", - "seems wasteful to compute the full cost function over the entire\n", - "training set in order to perform only a single parameter update. A\n", - "very common approach to addressing this challenge is to compute the\n", - "gradient over batches of the training data. For example, in current\n", - "a typical batch could contain some thousand examples from\n", - "an entire training set of several millions. This batch is then used to\n", - "perform a parameter update." - ] - }, - { - "cell_type": "markdown", - "id": "7085f8fd", - "metadata": { - "editable": true - }, - "source": [ - "## Stochastic Gradient Descent (SGD)\n", - "\n", - "In stochastic gradient descent, the extreme case is the case where we\n", - "have only one batch, that is we include the whole data set.\n", - "\n", - "This process is called Stochastic Gradient\n", - "Descent (SGD) (or also sometimes on-line gradient descent). This is\n", - "relatively less common to see because in practice due to vectorized\n", - "code optimizations it can be computationally much more efficient to\n", - "evaluate the gradient for 100 examples, than the gradient for one\n", - "example 100 times. Even though SGD technically refers to using a\n", - "single example at a time to evaluate the gradient, you will hear\n", - "people use the term SGD even when referring to mini-batch gradient\n", - "descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD\n", - "for “Batch gradient descent” are rare to see), where it is usually\n", - "assumed that mini-batches are used. The size of the mini-batch is a\n", - "hyperparameter but it is not very common to cross-validate or bootstrap it. It is\n", - "usually based on memory constraints (if any), or set to some value,\n", - "e.g. 32, 64 or 128. We use powers of 2 in practice because many\n", - "vectorized operation implementations work faster when their inputs are\n", - "sized in powers of 2.\n", - "\n", - "In our notes with SGD we mean stochastic gradient descent with mini-batches." - ] - }, - { - "cell_type": "markdown", - "id": "b6331bd3", - "metadata": { - "editable": true - }, - "source": [ - "## Stochastic Gradient Descent\n", - "\n", - "Stochastic gradient descent (SGD) and variants thereof address some of\n", - "the shortcomings of the Gradient descent method discussed above.\n", - "\n", - "The underlying idea of SGD comes from the observation that the cost\n", - "function, which we want to minimize, can almost always be written as a\n", - "sum over $n$ data points $\\{\\mathbf{x}_i\\}_{i=1}^n$," - ] - }, - { - "cell_type": "markdown", - "id": "9bfebbb2", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}).\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "238a4fc9", - "metadata": { - "editable": true - }, - "source": [ - "## Computation of gradients\n", - "\n", - "This in turn means that the gradient can be\n", - "computed as a sum over $i$-gradients" - ] - }, - { - "cell_type": "markdown", - "id": "9ef691a3", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}).\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "80f76175", - "metadata": { - "editable": true - }, - "source": [ - "Stochasticity/randomness is introduced by only taking the\n", - "gradient on a subset of the data called minibatches. If there are $n$\n", - "data points and the size of each minibatch is $M$, there will be $n/M$\n", - "minibatches. We denote these minibatches by $B_k$ where\n", - "$k=1,\\cdots,n/M$." - ] - }, - { - "cell_type": "markdown", - "id": "051b67fc", + "id": "1a493e7e", "metadata": { "editable": true }, @@ -2936,7 +2938,7 @@ }, { "cell_type": "markdown", - "id": "7f566d2a", + "id": "df7909bc", "metadata": { "editable": true }, @@ -2951,7 +2953,7 @@ }, { "cell_type": "markdown", - "id": "3f7eb9ac", + "id": "afdf2e10", "metadata": { "editable": true }, @@ -2963,7 +2965,7 @@ }, { "cell_type": "markdown", - "id": "6dda77e6", + "id": "c2e37d69", "metadata": { "editable": true }, @@ -2976,7 +2978,7 @@ }, { "cell_type": "markdown", - "id": "94adb2ca", + "id": "6a125356", "metadata": { "editable": true }, @@ -2990,7 +2992,7 @@ }, { "cell_type": "markdown", - "id": "630ab10d", + "id": "abaf8e3d", "metadata": { "editable": true }, @@ -3001,7 +3003,7 @@ { "cell_type": "code", "execution_count": 14, - "id": "3d1a3474", + "id": "4628460a", "metadata": { "collapsed": false, "editable": true @@ -3026,7 +3028,7 @@ }, { "cell_type": "markdown", - "id": "cb182f3b", + "id": "ec7e3114", "metadata": { "editable": true }, @@ -3042,7 +3044,7 @@ }, { "cell_type": "markdown", - "id": "44aa3a50", + "id": "8edd25f6", "metadata": { "editable": true }, @@ -3063,7 +3065,7 @@ }, { "cell_type": "markdown", - "id": "ebd5f637", + "id": "4a05e627", "metadata": { "editable": true }, @@ -3086,7 +3088,7 @@ { "cell_type": "code", "execution_count": 15, - "id": "78e602eb", + "id": "c016a06b", "metadata": { "collapsed": false, "editable": true @@ -3121,7 +3123,7 @@ }, { "cell_type": "markdown", - "id": "d3d5fdfc", + "id": "547b28ad", "metadata": { "editable": true }, @@ -3131,7 +3133,7 @@ }, { "cell_type": "markdown", - "id": "2d89fdc5", + "id": "ca68cd6e", "metadata": { "editable": true }, @@ -3142,7 +3144,7 @@ { "cell_type": "code", "execution_count": 16, - "id": "5bd4ddd0", + "id": "1890a370", "metadata": { "collapsed": false, "editable": true @@ -3222,7 +3224,7 @@ }, { "cell_type": "markdown", - "id": "906ca8a3", + "id": "49371ea1", "metadata": { "editable": true }, @@ -3232,12 +3234,12 @@ "In the above code, we have use replacement in setting up the\n", "mini-batches. The discussion\n", "[here](https://sebastianraschka.com/faq/docs/sgd-methods.html) may be\n", - "useful. More material will be added later." + "useful." ] }, { "cell_type": "markdown", - "id": "4a184020", + "id": "6f44130c", "metadata": { "editable": true }, @@ -3252,7 +3254,7 @@ }, { "cell_type": "markdown", - "id": "02d4f63f", + "id": "3dfe1344", "metadata": { "editable": true }, @@ -3264,7 +3266,7 @@ }, { "cell_type": "markdown", - "id": "8c5143d0", + "id": "7438ccdb", "metadata": { "editable": true }, @@ -3282,7 +3284,7 @@ }, { "cell_type": "markdown", - "id": "037205df", + "id": "f6c3290d", "metadata": { "editable": true }, @@ -3301,7 +3303,7 @@ }, { "cell_type": "markdown", - "id": "e3dcc6e8", + "id": "3e3bf9cd", "metadata": { "editable": true }, @@ -3313,7 +3315,7 @@ }, { "cell_type": "markdown", - "id": "d9a353dd", + "id": "30fb1856", "metadata": { "editable": true }, @@ -3323,7 +3325,7 @@ }, { "cell_type": "markdown", - "id": "473055d4", + "id": "59802bd4", "metadata": { "editable": true }, @@ -3339,7 +3341,7 @@ }, { "cell_type": "markdown", - "id": "35e67283", + "id": "4a93e7c9", "metadata": { "editable": true }, @@ -3351,7 +3353,7 @@ }, { "cell_type": "markdown", - "id": "bf92c54d", + "id": "2bd5f401", "metadata": { "editable": true }, @@ -3361,7 +3363,7 @@ }, { "cell_type": "markdown", - "id": "f2541cad", + "id": "8a70082f", "metadata": { "editable": true }, @@ -3373,7 +3375,7 @@ }, { "cell_type": "markdown", - "id": "1520d357", + "id": "18808d48", "metadata": { "editable": true }, @@ -3383,7 +3385,7 @@ }, { "cell_type": "markdown", - "id": "102449f7", + "id": "26406ac8", "metadata": { "editable": true }, @@ -3395,7 +3397,7 @@ }, { "cell_type": "markdown", - "id": "0fbe4820", + "id": "c462f769", "metadata": { "editable": true }, @@ -3411,7 +3413,7 @@ }, { "cell_type": "markdown", - "id": "b163b029", + "id": "6bf678e7", "metadata": { "editable": true }, @@ -3423,7 +3425,7 @@ }, { "cell_type": "markdown", - "id": "17f8c54e", + "id": "4cdbd09d", "metadata": { "editable": true }, @@ -3456,7 +3458,7 @@ }, { "cell_type": "markdown", - "id": "c5e2273f", + "id": "5d252515", "metadata": { "editable": true }, @@ -3468,7 +3470,7 @@ }, { "cell_type": "markdown", - "id": "96ee109a", + "id": "596d58a2", "metadata": { "editable": true }, @@ -3486,7 +3488,7 @@ }, { "cell_type": "markdown", - "id": "eb0b51e1", + "id": "50e01c2e", "metadata": { "editable": true }, @@ -3496,7 +3498,7 @@ }, { "cell_type": "markdown", - "id": "1140b00b", + "id": "386f9167", "metadata": { "editable": true }, @@ -3527,7 +3529,7 @@ }, { "cell_type": "markdown", - "id": "12e8dbaa", + "id": "709b2114", "metadata": { "editable": true }, @@ -3542,7 +3544,7 @@ }, { "cell_type": "markdown", - "id": "05867c79", + "id": "116ee962", "metadata": { "editable": true }, @@ -3560,7 +3562,7 @@ }, { "cell_type": "markdown", - "id": "47c4750f", + "id": "fe810dcf", "metadata": { "editable": true }, @@ -3572,7 +3574,7 @@ }, { "cell_type": "markdown", - "id": "2f18c06e", + "id": "cd81ad67", "metadata": { "editable": true }, @@ -3584,7 +3586,7 @@ }, { "cell_type": "markdown", - "id": "d00446d9", + "id": "06498abc", "metadata": { "editable": true }, @@ -3602,7 +3604,7 @@ }, { "cell_type": "markdown", - "id": "bc5f3776", + "id": "8a3c0446", "metadata": { "editable": true }, @@ -3625,7 +3627,7 @@ }, { "cell_type": "markdown", - "id": "f3071815", + "id": "82aeaa28", "metadata": { "editable": true }, @@ -3643,7 +3645,7 @@ }, { "cell_type": "markdown", - "id": "aac9b1f4", + "id": "8c71ca82", "metadata": { "editable": true }, @@ -3655,7 +3657,7 @@ }, { "cell_type": "markdown", - "id": "14b3cf69", + "id": "f0b27db1", "metadata": { "editable": true }, @@ -3667,7 +3669,7 @@ }, { "cell_type": "markdown", - "id": "1762335e", + "id": "5e3da9e9", "metadata": { "editable": true }, @@ -3679,7 +3681,7 @@ }, { "cell_type": "markdown", - "id": "07235a77", + "id": "6c5c06b4", "metadata": { "editable": true }, @@ -3691,7 +3693,7 @@ }, { "cell_type": "markdown", - "id": "5060ad05", + "id": "3b247467", "metadata": { "editable": true }, @@ -3703,7 +3705,7 @@ }, { "cell_type": "markdown", - "id": "7b4d96d7", + "id": "22079281", "metadata": { "editable": true }, @@ -3720,7 +3722,7 @@ }, { "cell_type": "markdown", - "id": "a4c384f5", + "id": "a24eece5", "metadata": { "editable": true }, @@ -3739,7 +3741,7 @@ }, { "cell_type": "markdown", - "id": "2c3d8429", + "id": "634c6f26", "metadata": { "editable": true }, @@ -3751,7 +3753,7 @@ }, { "cell_type": "markdown", - "id": "b5857328", + "id": "71ef4fbf", "metadata": { "editable": true }, @@ -3771,7 +3773,7 @@ }, { "cell_type": "markdown", - "id": "06b5c926", + "id": "40720b38", "metadata": { "editable": true }, @@ -3809,7 +3811,7 @@ }, { "cell_type": "markdown", - "id": "721bf584", + "id": "5c07021c", "metadata": { "editable": true }, @@ -3821,7 +3823,7 @@ }, { "cell_type": "markdown", - "id": "49d791e1", + "id": "ae7a6c62", "metadata": { "editable": true }, @@ -3831,7 +3833,7 @@ }, { "cell_type": "markdown", - "id": "9c676f72", + "id": "5b9164e2", "metadata": { "editable": true }, @@ -3843,7 +3845,7 @@ }, { "cell_type": "markdown", - "id": "2055fb52", + "id": "d56054c0", "metadata": { "editable": true }, @@ -3854,7 +3856,7 @@ { "cell_type": "code", "execution_count": 17, - "id": "eca0da75", + "id": "15d3d863", "metadata": { "collapsed": false, "editable": true @@ -3899,7 +3901,7 @@ }, { "cell_type": "markdown", - "id": "5dae5aba", + "id": "5ab6d83d", "metadata": { "editable": true }, @@ -3916,7 +3918,7 @@ { "cell_type": "code", "execution_count": 18, - "id": "d0790e9f", + "id": "751d1395", "metadata": { "collapsed": false, "editable": true @@ -3944,7 +3946,7 @@ }, { "cell_type": "markdown", - "id": "84ee3863", + "id": "838d2638", "metadata": { "editable": true }, @@ -3959,7 +3961,7 @@ { "cell_type": "code", "execution_count": 19, - "id": "51d09008", + "id": "2241609f", "metadata": { "collapsed": false, "editable": true @@ -4003,7 +4005,7 @@ }, { "cell_type": "markdown", - "id": "2d8e4407", + "id": "b8a70461", "metadata": { "editable": true }, @@ -4013,7 +4015,7 @@ }, { "cell_type": "markdown", - "id": "30ad2970", + "id": "a5bc4872", "metadata": { "editable": true }, @@ -4024,7 +4026,7 @@ { "cell_type": "code", "execution_count": 20, - "id": "8e8e65cf", + "id": "33f0541e", "metadata": { "collapsed": false, "editable": true @@ -4052,7 +4054,7 @@ }, { "cell_type": "markdown", - "id": "3aca489a", + "id": "50e2ba5d", "metadata": { "editable": true }, @@ -4067,7 +4069,7 @@ }, { "cell_type": "markdown", - "id": "6c20bd49", + "id": "fe1a3058", "metadata": { "editable": true }, @@ -4078,7 +4080,7 @@ { "cell_type": "code", "execution_count": 21, - "id": "d002c399", + "id": "fda91b69", "metadata": { "collapsed": false, "editable": true @@ -4106,7 +4108,7 @@ }, { "cell_type": "markdown", - "id": "a933898e", + "id": "1a48d07c", "metadata": { "editable": true }, @@ -4117,7 +4119,7 @@ { "cell_type": "code", "execution_count": 22, - "id": "387235f0", + "id": "6c8baf8d", "metadata": { "collapsed": false, "editable": true @@ -4142,7 +4144,7 @@ }, { "cell_type": "markdown", - "id": "3cc26abd", + "id": "3ac7c6c0", "metadata": { "editable": true }, @@ -4153,7 +4155,7 @@ { "cell_type": "code", "execution_count": 23, - "id": "b1dd069d", + "id": "ac60cd6c", "metadata": { "collapsed": false, "editable": true @@ -4189,7 +4191,7 @@ { "cell_type": "code", "execution_count": 24, - "id": "baf90cb4", + "id": "59735281", "metadata": { "collapsed": false, "editable": true @@ -4209,7 +4211,7 @@ }, { "cell_type": "markdown", - "id": "0eff5efe", + "id": "b247b6ad", "metadata": { "editable": true }, @@ -4220,7 +4222,7 @@ { "cell_type": "code", "execution_count": 25, - "id": "37b54f99", + "id": "da05505a", "metadata": { "collapsed": false, "editable": true @@ -4258,7 +4260,7 @@ }, { "cell_type": "markdown", - "id": "3fd3588a", + "id": "feb7935f", "metadata": { "editable": true }, @@ -4268,7 +4270,7 @@ }, { "cell_type": "markdown", - "id": "bba43464", + "id": "714f2b8d", "metadata": { "editable": true }, @@ -4282,7 +4284,7 @@ { "cell_type": "code", "execution_count": 26, - "id": "534ccb45", + "id": "0ac7f4e6", "metadata": { "collapsed": false, "editable": true @@ -4304,7 +4306,7 @@ }, { "cell_type": "markdown", - "id": "97bdc2ce", + "id": "2db47b66", "metadata": { "editable": true }, @@ -4314,7 +4316,7 @@ }, { "cell_type": "markdown", - "id": "a914c4e0", + "id": "0c29b75a", "metadata": { "editable": true }, @@ -4325,7 +4327,7 @@ { "cell_type": "code", "execution_count": 27, - "id": "b00a4fd5", + "id": "6cfc0ef8", "metadata": { "collapsed": false, "editable": true @@ -4347,7 +4349,7 @@ }, { "cell_type": "markdown", - "id": "665e48d5", + "id": "ee6bf2f1", "metadata": { "editable": true }, @@ -4360,7 +4362,7 @@ { "cell_type": "code", "execution_count": 28, - "id": "a33a6436", + "id": "37c14206", "metadata": { "collapsed": false, "editable": true @@ -4385,7 +4387,7 @@ }, { "cell_type": "markdown", - "id": "a564dc5a", + "id": "645d9ee0", "metadata": { "editable": true }, @@ -4397,7 +4399,7 @@ { "cell_type": "code", "execution_count": 29, - "id": "eab6652e", + "id": "27ffa613", "metadata": { "collapsed": false, "editable": true @@ -4412,7 +4414,7 @@ }, { "cell_type": "markdown", - "id": "74647197", + "id": "32628932", "metadata": { "editable": true }, @@ -4427,7 +4429,7 @@ { "cell_type": "code", "execution_count": 30, - "id": "264a6c11", + "id": "47a1833d", "metadata": { "collapsed": false, "editable": true @@ -4487,7 +4489,7 @@ }, { "cell_type": "markdown", - "id": "e9cf7f42", + "id": "d4dde158", "metadata": { "editable": true }, @@ -4499,7 +4501,7 @@ { "cell_type": "code", "execution_count": 31, - "id": "86d059a4", + "id": "b86c1477", "metadata": { "collapsed": false, "editable": true @@ -4583,7 +4585,7 @@ }, { "cell_type": "markdown", - "id": "0d1ea9d0", + "id": "27576d79", "metadata": { "editable": true }, @@ -4594,7 +4596,7 @@ { "cell_type": "code", "execution_count": 32, - "id": "4dc2352d", + "id": "c6d0db97", "metadata": { "collapsed": false, "editable": true diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt index 6d0bd00af..c41d0a431 100644 --- a/doc/src/week39/week39.do.txt +++ b/doc/src/week39/week39.do.txt @@ -11,11 +11,11 @@ DATE: today * Reading recommendations: See "lecture notes for week 39":"https://compphysics.github.io/MachineLearning/doc/web/course.html". -For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. +For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well. For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text. -_For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11_. +_For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11_. These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. !split @@ -28,6 +28,8 @@ These sections summarize neatly what we have done till now and point to what is !split ===== Optimization, the central part of any Machine Learning algortithm ===== +The first few slides here are a repetition from last week. + Almost every problem in machine learning and data science starts with a dataset $X$, a model $g(\beta)$, which is a function of the parameters $\beta$ and a cost function $C(X, g(\beta))$ that allows @@ -1068,6 +1070,86 @@ plt.show() * GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points. +!split +===== Overview video on Stochastic Gradient Descent ===== + +"What is Stochastic Gradient Descent":"https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer" + + +!split +===== Batches and mini-batches ===== + +In gradient descent we compute the cost function and its gradient for all data points we have. + +In large-scale applications such as the "ILSVRC challenge":"https://www.image-net.org/challenges/LSVRC/", the +training data can have on order of millions of examples. Hence, it +seems wasteful to compute the full cost function over the entire +training set in order to perform only a single parameter update. A +very common approach to addressing this challenge is to compute the +gradient over batches of the training data. For example, in current +a typical batch could contain some thousand examples from +an entire training set of several millions. This batch is then used to +perform a parameter update. + +!split +===== Stochastic Gradient Descent (SGD) ===== + +In stochastic gradient descent, the extreme case is the case where we +have only one batch, that is we include the whole data set. + +This process is called Stochastic Gradient +Descent (SGD) (or also sometimes on-line gradient descent). This is +relatively less common to see because in practice due to vectorized +code optimizations it can be computationally much more efficient to +evaluate the gradient for 100 examples, than the gradient for one +example 100 times. Even though SGD technically refers to using a +single example at a time to evaluate the gradient, you will hear +people use the term SGD even when referring to mini-batch gradient +descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD +for “Batch gradient descent” are rare to see), where it is usually +assumed that mini-batches are used. The size of the mini-batch is a +hyperparameter but it is not very common to cross-validate or bootstrap it. It is +usually based on memory constraints (if any), or set to some value, +e.g. 32, 64 or 128. We use powers of 2 in practice because many +vectorized operation implementations work faster when their inputs are +sized in powers of 2. + +In our notes with SGD we mean stochastic gradient descent with mini-batches. + + +!split +===== Stochastic Gradient Descent ===== + +Stochastic gradient descent (SGD) and variants thereof address some of +the shortcomings of the Gradient descent method discussed above. + +The underlying idea of SGD comes from the observation that the cost +function, which we want to minimize, can almost always be written as a +sum over $n$ data points $\{\mathbf{x}_i\}_{i=1}^n$, +!bt +\[ +C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, +\mathbf{\beta}). +\] +!et + +!split +===== Computation of gradients ===== + +This in turn means that the gradient can be +computed as a sum over $i$-gradients +!bt +\[ +\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, +\mathbf{\beta}). +\] +!et + +Stochasticity/randomness is introduced by only taking the +gradient on a subset of the data called minibatches. If there are $n$ +data points and the size of each minibatch is $M$, there will be $n/M$ +minibatches. We denote these minibatches by $B_k$ where +$k=1,\cdots,n/M$. !split @@ -1385,86 +1467,6 @@ plt.show() -!split -===== Overview video on Stochastic Gradient Descent ===== - -"What is Stochastic Gradient Descent":"https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer" - - -!split -===== Batches and mini-batches ===== - -In gradient descent we compute the cost function and its gradient for all data points we have. - -In large-scale applications such as the "ILSVRC challenge":"https://www.image-net.org/challenges/LSVRC/", the -training data can have on order of millions of examples. Hence, it -seems wasteful to compute the full cost function over the entire -training set in order to perform only a single parameter update. A -very common approach to addressing this challenge is to compute the -gradient over batches of the training data. For example, in current -a typical batch could contain some thousand examples from -an entire training set of several millions. This batch is then used to -perform a parameter update. - -!split -===== Stochastic Gradient Descent (SGD) ===== - -In stochastic gradient descent, the extreme case is the case where we -have only one batch, that is we include the whole data set. - -This process is called Stochastic Gradient -Descent (SGD) (or also sometimes on-line gradient descent). This is -relatively less common to see because in practice due to vectorized -code optimizations it can be computationally much more efficient to -evaluate the gradient for 100 examples, than the gradient for one -example 100 times. Even though SGD technically refers to using a -single example at a time to evaluate the gradient, you will hear -people use the term SGD even when referring to mini-batch gradient -descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD -for “Batch gradient descent” are rare to see), where it is usually -assumed that mini-batches are used. The size of the mini-batch is a -hyperparameter but it is not very common to cross-validate or bootstrap it. It is -usually based on memory constraints (if any), or set to some value, -e.g. 32, 64 or 128. We use powers of 2 in practice because many -vectorized operation implementations work faster when their inputs are -sized in powers of 2. - -In our notes with SGD we mean stochastic gradient descent with mini-batches. - - -!split -===== Stochastic Gradient Descent ===== - -Stochastic gradient descent (SGD) and variants thereof address some of -the shortcomings of the Gradient descent method discussed above. - -The underlying idea of SGD comes from the observation that the cost -function, which we want to minimize, can almost always be written as a -sum over $n$ data points $\{\mathbf{x}_i\}_{i=1}^n$, -!bt -\[ -C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, -\mathbf{\beta}). -\] -!et - -!split -===== Computation of gradients ===== - -This in turn means that the gradient can be -computed as a sum over $i$-gradients -!bt -\[ -\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}). -\] -!et - -Stochasticity/randomness is introduced by only taking the -gradient on a subset of the data called minibatches. If there are $n$ -data points and the size of each minibatch is $M$, there will be $n/M$ -minibatches. We denote these minibatches by $B_k$ where -$k=1,\cdots,n/M$. !split ===== SGD example ===== @@ -1677,7 +1679,7 @@ plt.show() In the above code, we have use replacement in setting up the mini-batches. The discussion "here":"https://sebastianraschka.com/faq/docs/sgd-methods.html" may be -useful. More material will be added later. +useful. !split