lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week39

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2022-09-27 09:29:38 +02:00
parent 878f01db99
commit d478ce2e57
7 changed files with 830 additions and 820 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week39/html/week39-bs.html
+23 -23
View File
@@ -153,6 +153,20 @@ doconce format html week39.do.txt --html_style=bootstrap --pygments_html_style=d
2,
None,
'using-gradient-descent-methods-limitations'),
('Overview video on Stochastic Gradient Descent',
2,
None,
'overview-video-on-stochastic-gradient-descent'),
('Batches and mini-batches', 2, None, 'batches-and-mini-batches'),
('Stochastic Gradient Descent (SGD)',
2,
None,
'stochastic-gradient-descent-sgd'),
('Stochastic Gradient Descent',
2,
None,
'stochastic-gradient-descent'),
('Computation of gradients', 2, None, 'computation-of-gradients'),
('Stochastic Gradient Descent',
2,
None,
@@ -174,20 +188,6 @@ doconce format html week39.do.txt --html_style=bootstrap --pygments_html_style=d
2,
None,
'code-with-a-number-of-minibatches-which-varies'),
('Overview video on Stochastic Gradient Descent',
2,
None,
'overview-video-on-stochastic-gradient-descent'),
('Batches and mini-batches', 2, None, 'batches-and-mini-batches'),
('Stochastic Gradient Descent (SGD)',
2,
None,
'stochastic-gradient-descent-sgd'),
('Stochastic Gradient Descent',
2,
None,
'stochastic-gradient-descent'),
('Computation of gradients', 2, None, 'computation-of-gradients'),
('SGD example', 2, None, 'sgd-example'),
('The gradient step', 2, None, 'the-gradient-step'),
('Simple example code', 2, None, 'simple-example-code'),
@@ -325,20 +325,20 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week39-bs042.html#the-hessian-matrix-for-ridge-regression" style="font-size: 80%;">The Hessian matrix for Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs043.html#program-example-for-gradient-descent-with-ridge-regression" style="font-size: 80%;">Program example for gradient descent with Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs044.html#using-gradient-descent-methods-limitations" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs057.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs058.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs045.html#overview-video-on-stochastic-gradient-descent" style="font-size: 80%;">Overview video on Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs046.html#batches-and-mini-batches" style="font-size: 80%;">Batches and mini-batches</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs047.html#stochastic-gradient-descent-sgd" style="font-size: 80%;">Stochastic Gradient Descent (SGD)</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs050.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs051.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs050.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs051.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs059.html#sgd-example" style="font-size: 80%;">SGD example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs060.html#the-gradient-step" style="font-size: 80%;">The gradient step</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs061.html#simple-example-code" style="font-size: 80%;">Simple example code</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs062.html#when-do-we-stop" style="font-size: 80%;">When do we stop?</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs063.html#slightly-different-approach" style="font-size: 80%;">Slightly different approach</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs064.html#program-for-stochastic-gradient" style="font-size: 80%;">Program for stochastic gradient</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs053.html#code-with-a-number-of-minibatches-which-varies" style="font-size: 80%;">Code with a Number of Minibatches which varies</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs054.html#overview-video-on-stochastic-gradient-descent" style="font-size: 80%;">Overview video on Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs055.html#batches-and-mini-batches" style="font-size: 80%;">Batches and mini-batches</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs056.html#stochastic-gradient-descent-sgd" style="font-size: 80%;">Stochastic Gradient Descent (SGD)</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs057.html#stochastic-gradient-descent" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs058.html#computation-of-gradients" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs058.html#code-with-a-number-of-minibatches-which-varies" style="font-size: 80%;">Code with a Number of Minibatches which varies</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs059.html#sgd-example" style="font-size: 80%;">SGD example</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs060.html#the-gradient-step" style="font-size: 80%;">The gradient step</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs061.html#simple-example-code" style="font-size: 80%;">Simple example code</a></li>
@@ -396,7 +396,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Sep 26, 2022</h4>
<h4>Sep 27, 2022</h4>
</center> <!-- date -->
<br>
doc/pub/week39/html/week39-reveal.html
+97 -95
View File
@@ -184,7 +184,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Sep 26, 2022</h4>
<h4>Sep 27, 2022</h4>
</center> <!-- date -->
<br>
@@ -204,12 +204,12 @@ MathJax.Hub.Config({
</ul>
<p>
<p>See <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_blank">lecture notes for week 39</a>.
For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
</p>
<p>For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.</p>
<b>For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11</b>.
<b>For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11</b>.
<p>These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. </p>
</section>
@@ -222,6 +222,8 @@ For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5
<section>
<h2 id="optimization-the-central-part-of-any-machine-learning-algortithm">Optimization, the central part of any Machine Learning algortithm </h2>
<p>The first few slides here are a repetition from last week. </p>
<p>Almost every problem in machine learning and data science starts with
a dataset \( X \), a model \( g(\beta) \), which is a function of the
parameters \( \beta \) and a cost function \( C(X, g(\beta)) \) that allows
@@ -1625,6 +1627,97 @@ plt.show()
</ul>
</section>
<section>
<h2 id="overview-video-on-stochastic-gradient-descent">Overview video on Stochastic Gradient Descent </h2>
<a href="https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer" target="_blank">What is Stochastic Gradient Descent</a>
</section>
<section>
<h2 id="batches-and-mini-batches">Batches and mini-batches </h2>
<p>In gradient descent we compute the cost function and its gradient for all data points we have.</p>
<p>In large-scale applications such as the <a href="https://www.image-net.org/challenges/LSVRC/" target="_blank">ILSVRC challenge</a>, the
training data can have on order of millions of examples. Hence, it
seems wasteful to compute the full cost function over the entire
training set in order to perform only a single parameter update. A
very common approach to addressing this challenge is to compute the
gradient over batches of the training data. For example, in current
a typical batch could contain some thousand examples from
an entire training set of several millions. This batch is then used to
perform a parameter update.
</p>
</section>
<section>
<h2 id="stochastic-gradient-descent-sgd">Stochastic Gradient Descent (SGD) </h2>
<p>In stochastic gradient descent, the extreme case is the case where we
have only one batch, that is we include the whole data set.
</p>
<p>This process is called Stochastic Gradient
Descent (SGD) (or also sometimes on-line gradient descent). This is
relatively less common to see because in practice due to vectorized
code optimizations it can be computationally much more efficient to
evaluate the gradient for 100 examples, than the gradient for one
example 100 times. Even though SGD technically refers to using a
single example at a time to evaluate the gradient, you will hear
people use the term SGD even when referring to mini-batch gradient
descent (i.e. mentions of MGD for &#8220;Minibatch Gradient Descent&#8221;, or BGD
for &#8220;Batch gradient descent&#8221; are rare to see), where it is usually
assumed that mini-batches are used. The size of the mini-batch is a
hyperparameter but it is not very common to cross-validate or bootstrap it. It is
usually based on memory constraints (if any), or set to some value,
e.g. 32, 64 or 128. We use powers of 2 in practice because many
vectorized operation implementations work faster when their inputs are
sized in powers of 2.
</p>
<p>In our notes with SGD we mean stochastic gradient descent with mini-batches.</p>
</section>
<section>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>Stochastic gradient descent (SGD) and variants thereof address some of
the shortcomings of the Gradient descent method discussed above.
</p>
<p>The underlying idea of SGD comes from the observation that the cost
function, which we want to minimize, can almost always be written as a
sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
</p>
<p>&nbsp;<br>
$$
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="computation-of-gradients">Computation of gradients </h2>
<p>This in turn means that the gradient can be
computed as a sum over \( i \)-gradients
</p>
<p>&nbsp;<br>
$$
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$
<p>&nbsp;<br>
<p>Stochasticity/randomness is introduced by only taking the
gradient on a subset of the data called minibatches. If there are \( n \)
data points and the size of each minibatch is \( M \), there will be \( n/M \)
minibatches. We denote these minibatches by \( B_k \) where
\( k=1,\cdots,n/M \).
</p>
</section>
<section>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
@@ -2026,97 +2119,6 @@ plt.show()
</div>
</section>
<section>
<h2 id="overview-video-on-stochastic-gradient-descent">Overview video on Stochastic Gradient Descent </h2>
<a href="https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer" target="_blank">What is Stochastic Gradient Descent</a>
</section>
<section>
<h2 id="batches-and-mini-batches">Batches and mini-batches </h2>
<p>In gradient descent we compute the cost function and its gradient for all data points we have.</p>
<p>In large-scale applications such as the <a href="https://www.image-net.org/challenges/LSVRC/" target="_blank">ILSVRC challenge</a>, the
training data can have on order of millions of examples. Hence, it
seems wasteful to compute the full cost function over the entire
training set in order to perform only a single parameter update. A
very common approach to addressing this challenge is to compute the
gradient over batches of the training data. For example, in current
a typical batch could contain some thousand examples from
an entire training set of several millions. This batch is then used to
perform a parameter update.
</p>
</section>
<section>
<h2 id="stochastic-gradient-descent-sgd">Stochastic Gradient Descent (SGD) </h2>
<p>In stochastic gradient descent, the extreme case is the case where we
have only one batch, that is we include the whole data set.
</p>
<p>This process is called Stochastic Gradient
Descent (SGD) (or also sometimes on-line gradient descent). This is
relatively less common to see because in practice due to vectorized
code optimizations it can be computationally much more efficient to
evaluate the gradient for 100 examples, than the gradient for one
example 100 times. Even though SGD technically refers to using a
single example at a time to evaluate the gradient, you will hear
people use the term SGD even when referring to mini-batch gradient
descent (i.e. mentions of MGD for &#8220;Minibatch Gradient Descent&#8221;, or BGD
for &#8220;Batch gradient descent&#8221; are rare to see), where it is usually
assumed that mini-batches are used. The size of the mini-batch is a
hyperparameter but it is not very common to cross-validate or bootstrap it. It is
usually based on memory constraints (if any), or set to some value,
e.g. 32, 64 or 128. We use powers of 2 in practice because many
vectorized operation implementations work faster when their inputs are
sized in powers of 2.
</p>
<p>In our notes with SGD we mean stochastic gradient descent with mini-batches.</p>
</section>
<section>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>Stochastic gradient descent (SGD) and variants thereof address some of
the shortcomings of the Gradient descent method discussed above.
</p>
<p>The underlying idea of SGD comes from the observation that the cost
function, which we want to minimize, can almost always be written as a
sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
</p>
<p>&nbsp;<br>
$$
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="computation-of-gradients">Computation of gradients </h2>
<p>This in turn means that the gradient can be
computed as a sum over \( i \)-gradients
</p>
<p>&nbsp;<br>
$$
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$
<p>&nbsp;<br>
<p>Stochasticity/randomness is introduced by only taking the
gradient on a subset of the data called minibatches. If there are \( n \)
data points and the size of each minibatch is \( M \), there will be \( n/M \)
minibatches. We denote these minibatches by \( B_k \) where
\( k=1,\cdots,n/M \).
</p>
</section>
<section>
<h2 id="sgd-example">SGD example </h2>
@@ -2397,7 +2399,7 @@ plt.show()
<p>In the above code, we have use replacement in setting up the
mini-batches. The discussion
<a href="https://sebastianraschka.com/faq/docs/sgd-methods.html" target="_blank">here</a> may be
useful. More material will be added later.
useful.
</p>
</section>
doc/pub/week39/html/week39-solarized.html
+103 -101
View File
@@ -180,6 +180,20 @@ div.toc p,a {
2,
None,
'using-gradient-descent-methods-limitations'),
('Overview video on Stochastic Gradient Descent',
2,
None,
'overview-video-on-stochastic-gradient-descent'),
('Batches and mini-batches', 2, None, 'batches-and-mini-batches'),
('Stochastic Gradient Descent (SGD)',
2,
None,
'stochastic-gradient-descent-sgd'),
('Stochastic Gradient Descent',
2,
None,
'stochastic-gradient-descent'),
('Computation of gradients', 2, None, 'computation-of-gradients'),
('Stochastic Gradient Descent',
2,
None,
@@ -201,20 +215,6 @@ div.toc p,a {
2,
None,
'code-with-a-number-of-minibatches-which-varies'),
('Overview video on Stochastic Gradient Descent',
2,
None,
'overview-video-on-stochastic-gradient-descent'),
('Batches and mini-batches', 2, None, 'batches-and-mini-batches'),
('Stochastic Gradient Descent (SGD)',
2,
None,
'stochastic-gradient-descent-sgd'),
('Stochastic Gradient Descent',
2,
None,
'stochastic-gradient-descent'),
('Computation of gradients', 2, None, 'computation-of-gradients'),
('SGD example', 2, None, 'sgd-example'),
('The gradient step', 2, None, 'the-gradient-step'),
('Simple example code', 2, None, 'simple-example-code'),
@@ -311,7 +311,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Sep 26, 2022</h4>
<h4>Sep 27, 2022</h4>
</center> <!-- date -->
<br>
@@ -324,12 +324,12 @@ MathJax.Hub.Config({
<li> Reading recommendations:</li>
</ul>
<p>See <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_blank">lecture notes for week 39</a>.
For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
</p>
<p>For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.</p>
<b>For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11</b>.
<b>For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11</b>.
<p>These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. </p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -340,6 +340,8 @@ For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="optimization-the-central-part-of-any-machine-learning-algortithm">Optimization, the central part of any Machine Learning algortithm </h2>
<p>The first few slides here are a repetition from last week. </p>
<p>Almost every problem in machine learning and data science starts with
a dataset \( X \), a model \( g(\beta) \), which is a function of the
parameters \( \beta \) and a cost function \( C(X, g(\beta)) \) that allows
@@ -1596,6 +1598,89 @@ plt.show()
<li> <b>GD treats all directions in parameter space uniformly.</b> Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive.</li>
<li> GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="overview-video-on-stochastic-gradient-descent">Overview video on Stochastic Gradient Descent </h2>
<a href="https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer" target="_blank">What is Stochastic Gradient Descent</a>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="batches-and-mini-batches">Batches and mini-batches </h2>
<p>In gradient descent we compute the cost function and its gradient for all data points we have.</p>
<p>In large-scale applications such as the <a href="https://www.image-net.org/challenges/LSVRC/" target="_blank">ILSVRC challenge</a>, the
training data can have on order of millions of examples. Hence, it
seems wasteful to compute the full cost function over the entire
training set in order to perform only a single parameter update. A
very common approach to addressing this challenge is to compute the
gradient over batches of the training data. For example, in current
a typical batch could contain some thousand examples from
an entire training set of several millions. This batch is then used to
perform a parameter update.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="stochastic-gradient-descent-sgd">Stochastic Gradient Descent (SGD) </h2>
<p>In stochastic gradient descent, the extreme case is the case where we
have only one batch, that is we include the whole data set.
</p>
<p>This process is called Stochastic Gradient
Descent (SGD) (or also sometimes on-line gradient descent). This is
relatively less common to see because in practice due to vectorized
code optimizations it can be computationally much more efficient to
evaluate the gradient for 100 examples, than the gradient for one
example 100 times. Even though SGD technically refers to using a
single example at a time to evaluate the gradient, you will hear
people use the term SGD even when referring to mini-batch gradient
descent (i.e. mentions of MGD for &#8220;Minibatch Gradient Descent&#8221;, or BGD
for &#8220;Batch gradient descent&#8221; are rare to see), where it is usually
assumed that mini-batches are used. The size of the mini-batch is a
hyperparameter but it is not very common to cross-validate or bootstrap it. It is
usually based on memory constraints (if any), or set to some value,
e.g. 32, 64 or 128. We use powers of 2 in practice because many
vectorized operation implementations work faster when their inputs are
sized in powers of 2.
</p>
<p>In our notes with SGD we mean stochastic gradient descent with mini-batches.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>Stochastic gradient descent (SGD) and variants thereof address some of
the shortcomings of the Gradient descent method discussed above.
</p>
<p>The underlying idea of SGD comes from the observation that the cost
function, which we want to minimize, can almost always be written as a
sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
</p>
$$
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="computation-of-gradients">Computation of gradients </h2>
<p>This in turn means that the gradient can be
computed as a sum over \( i \)-gradients
</p>
$$
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$
<p>Stochasticity/randomness is introduced by only taking the
gradient on a subset of the data called minibatches. If there are \( n \)
data points and the size of each minibatch is \( M \), there will be \( n/M \)
minibatches. We denote these minibatches by \( B_k \) where
\( k=1,\cdots,n/M \).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
@@ -1982,89 +2067,6 @@ plt.show()
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="overview-video-on-stochastic-gradient-descent">Overview video on Stochastic Gradient Descent </h2>
<a href="https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer" target="_blank">What is Stochastic Gradient Descent</a>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="batches-and-mini-batches">Batches and mini-batches </h2>
<p>In gradient descent we compute the cost function and its gradient for all data points we have.</p>
<p>In large-scale applications such as the <a href="https://www.image-net.org/challenges/LSVRC/" target="_blank">ILSVRC challenge</a>, the
training data can have on order of millions of examples. Hence, it
seems wasteful to compute the full cost function over the entire
training set in order to perform only a single parameter update. A
very common approach to addressing this challenge is to compute the
gradient over batches of the training data. For example, in current
a typical batch could contain some thousand examples from
an entire training set of several millions. This batch is then used to
perform a parameter update.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="stochastic-gradient-descent-sgd">Stochastic Gradient Descent (SGD) </h2>
<p>In stochastic gradient descent, the extreme case is the case where we
have only one batch, that is we include the whole data set.
</p>
<p>This process is called Stochastic Gradient
Descent (SGD) (or also sometimes on-line gradient descent). This is
relatively less common to see because in practice due to vectorized
code optimizations it can be computationally much more efficient to
evaluate the gradient for 100 examples, than the gradient for one
example 100 times. Even though SGD technically refers to using a
single example at a time to evaluate the gradient, you will hear
people use the term SGD even when referring to mini-batch gradient
descent (i.e. mentions of MGD for &#8220;Minibatch Gradient Descent&#8221;, or BGD
for &#8220;Batch gradient descent&#8221; are rare to see), where it is usually
assumed that mini-batches are used. The size of the mini-batch is a
hyperparameter but it is not very common to cross-validate or bootstrap it. It is
usually based on memory constraints (if any), or set to some value,
e.g. 32, 64 or 128. We use powers of 2 in practice because many
vectorized operation implementations work faster when their inputs are
sized in powers of 2.
</p>
<p>In our notes with SGD we mean stochastic gradient descent with mini-batches.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>Stochastic gradient descent (SGD) and variants thereof address some of
the shortcomings of the Gradient descent method discussed above.
</p>
<p>The underlying idea of SGD comes from the observation that the cost
function, which we want to minimize, can almost always be written as a
sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
</p>
$$
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="computation-of-gradients">Computation of gradients </h2>
<p>This in turn means that the gradient can be
computed as a sum over \( i \)-gradients
</p>
$$
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$
<p>Stochasticity/randomness is introduced by only taking the
gradient on a subset of the data called minibatches. If there are \( n \)
data points and the size of each minibatch is \( M \), there will be \( n/M \)
minibatches. We denote these minibatches by \( B_k \) where
\( k=1,\cdots,n/M \).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="sgd-example">SGD example </h2>
@@ -2335,7 +2337,7 @@ plt.show()
<p>In the above code, we have use replacement in setting up the
mini-batches. The discussion
<a href="https://sebastianraschka.com/faq/docs/sgd-methods.html" target="_blank">here</a> may be
useful. More material will be added later.
useful.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week39/html/week39.html
+103 -101
View File
@@ -257,6 +257,20 @@ div.toc p,a {
2,
None,
'using-gradient-descent-methods-limitations'),
('Overview video on Stochastic Gradient Descent',
2,
None,
'overview-video-on-stochastic-gradient-descent'),
('Batches and mini-batches', 2, None, 'batches-and-mini-batches'),
('Stochastic Gradient Descent (SGD)',
2,
None,
'stochastic-gradient-descent-sgd'),
('Stochastic Gradient Descent',
2,
None,
'stochastic-gradient-descent'),
('Computation of gradients', 2, None, 'computation-of-gradients'),
('Stochastic Gradient Descent',
2,
None,
@@ -278,20 +292,6 @@ div.toc p,a {
2,
None,
'code-with-a-number-of-minibatches-which-varies'),
('Overview video on Stochastic Gradient Descent',
2,
None,
'overview-video-on-stochastic-gradient-descent'),
('Batches and mini-batches', 2, None, 'batches-and-mini-batches'),
('Stochastic Gradient Descent (SGD)',
2,
None,
'stochastic-gradient-descent-sgd'),
('Stochastic Gradient Descent',
2,
None,
'stochastic-gradient-descent'),
('Computation of gradients', 2, None, 'computation-of-gradients'),
('SGD example', 2, None, 'sgd-example'),
('The gradient step', 2, None, 'the-gradient-step'),
('Simple example code', 2, None, 'simple-example-code'),
@@ -388,7 +388,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Sep 26, 2022</h4>
<h4>Sep 27, 2022</h4>
</center> <!-- date -->
<br>
@@ -401,12 +401,12 @@ MathJax.Hub.Config({
<li> Reading recommendations:</li>
</ul>
<p>See <a href="https://compphysics.github.io/MachineLearning/doc/web/course.html" target="_blank">lecture notes for week 39</a>.
For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
</p>
<p>For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.</p>
<b>For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11</b>.
<b>For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11</b>.
<p>These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning. </p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -417,6 +417,8 @@ For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="optimization-the-central-part-of-any-machine-learning-algortithm">Optimization, the central part of any Machine Learning algortithm </h2>
<p>The first few slides here are a repetition from last week. </p>
<p>Almost every problem in machine learning and data science starts with
a dataset \( X \), a model \( g(\beta) \), which is a function of the
parameters \( \beta \) and a cost function \( C(X, g(\beta)) \) that allows
@@ -1673,6 +1675,89 @@ plt<span style="color: #666666">.</span>show()
<li> <b>GD treats all directions in parameter space uniformly.</b> Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive.</li>
<li> GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="overview-video-on-stochastic-gradient-descent">Overview video on Stochastic Gradient Descent </h2>
<a href="https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer" target="_blank">What is Stochastic Gradient Descent</a>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="batches-and-mini-batches">Batches and mini-batches </h2>
<p>In gradient descent we compute the cost function and its gradient for all data points we have.</p>
<p>In large-scale applications such as the <a href="https://www.image-net.org/challenges/LSVRC/" target="_blank">ILSVRC challenge</a>, the
training data can have on order of millions of examples. Hence, it
seems wasteful to compute the full cost function over the entire
training set in order to perform only a single parameter update. A
very common approach to addressing this challenge is to compute the
gradient over batches of the training data. For example, in current
a typical batch could contain some thousand examples from
an entire training set of several millions. This batch is then used to
perform a parameter update.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="stochastic-gradient-descent-sgd">Stochastic Gradient Descent (SGD) </h2>
<p>In stochastic gradient descent, the extreme case is the case where we
have only one batch, that is we include the whole data set.
</p>
<p>This process is called Stochastic Gradient
Descent (SGD) (or also sometimes on-line gradient descent). This is
relatively less common to see because in practice due to vectorized
code optimizations it can be computationally much more efficient to
evaluate the gradient for 100 examples, than the gradient for one
example 100 times. Even though SGD technically refers to using a
single example at a time to evaluate the gradient, you will hear
people use the term SGD even when referring to mini-batch gradient
descent (i.e. mentions of MGD for &#8220;Minibatch Gradient Descent&#8221;, or BGD
for &#8220;Batch gradient descent&#8221; are rare to see), where it is usually
assumed that mini-batches are used. The size of the mini-batch is a
hyperparameter but it is not very common to cross-validate or bootstrap it. It is
usually based on memory constraints (if any), or set to some value,
e.g. 32, 64 or 128. We use powers of 2 in practice because many
vectorized operation implementations work faster when their inputs are
sized in powers of 2.
</p>
<p>In our notes with SGD we mean stochastic gradient descent with mini-batches.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>Stochastic gradient descent (SGD) and variants thereof address some of
the shortcomings of the Gradient descent method discussed above.
</p>
<p>The underlying idea of SGD comes from the observation that the cost
function, which we want to minimize, can almost always be written as a
sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
</p>
$$
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="computation-of-gradients">Computation of gradients </h2>
<p>This in turn means that the gradient can be
computed as a sum over \( i \)-gradients
</p>
$$
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$
<p>Stochasticity/randomness is introduced by only taking the
gradient on a subset of the data called minibatches. If there are \( n \)
data points and the size of each minibatch is \( M \), there will be \( n/M \)
minibatches. We denote these minibatches by \( B_k \) where
\( k=1,\cdots,n/M \).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
@@ -2059,89 +2144,6 @@ plt.show()
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="overview-video-on-stochastic-gradient-descent">Overview video on Stochastic Gradient Descent </h2>
<a href="https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer" target="_blank">What is Stochastic Gradient Descent</a>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="batches-and-mini-batches">Batches and mini-batches </h2>
<p>In gradient descent we compute the cost function and its gradient for all data points we have.</p>
<p>In large-scale applications such as the <a href="https://www.image-net.org/challenges/LSVRC/" target="_blank">ILSVRC challenge</a>, the
training data can have on order of millions of examples. Hence, it
seems wasteful to compute the full cost function over the entire
training set in order to perform only a single parameter update. A
very common approach to addressing this challenge is to compute the
gradient over batches of the training data. For example, in current
a typical batch could contain some thousand examples from
an entire training set of several millions. This batch is then used to
perform a parameter update.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="stochastic-gradient-descent-sgd">Stochastic Gradient Descent (SGD) </h2>
<p>In stochastic gradient descent, the extreme case is the case where we
have only one batch, that is we include the whole data set.
</p>
<p>This process is called Stochastic Gradient
Descent (SGD) (or also sometimes on-line gradient descent). This is
relatively less common to see because in practice due to vectorized
code optimizations it can be computationally much more efficient to
evaluate the gradient for 100 examples, than the gradient for one
example 100 times. Even though SGD technically refers to using a
single example at a time to evaluate the gradient, you will hear
people use the term SGD even when referring to mini-batch gradient
descent (i.e. mentions of MGD for &#8220;Minibatch Gradient Descent&#8221;, or BGD
for &#8220;Batch gradient descent&#8221; are rare to see), where it is usually
assumed that mini-batches are used. The size of the mini-batch is a
hyperparameter but it is not very common to cross-validate or bootstrap it. It is
usually based on memory constraints (if any), or set to some value,
e.g. 32, 64 or 128. We use powers of 2 in practice because many
vectorized operation implementations work faster when their inputs are
sized in powers of 2.
</p>
<p>In our notes with SGD we mean stochastic gradient descent with mini-batches.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="stochastic-gradient-descent">Stochastic Gradient Descent </h2>
<p>Stochastic gradient descent (SGD) and variants thereof address some of
the shortcomings of the Gradient descent method discussed above.
</p>
<p>The underlying idea of SGD comes from the observation that the cost
function, which we want to minimize, can almost always be written as a
sum over \( n \) data points \( \{\mathbf{x}_i\}_{i=1}^n \),
</p>
$$
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="computation-of-gradients">Computation of gradients </h2>
<p>This in turn means that the gradient can be
computed as a sum over \( i \)-gradients
</p>
$$
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$
<p>Stochasticity/randomness is introduced by only taking the
gradient on a subset of the data called minibatches. If there are \( n \)
data points and the size of each minibatch is \( M \), there will be \( n/M \)
minibatches. We denote these minibatches by \( B_k \) where
\( k=1,\cdots,n/M \).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="sgd-example">SGD example </h2>
@@ -2412,7 +2414,7 @@ plt<span style="color: #666666">.</span>show()
<p>In the above code, we have use replacement in setting up the
mini-batches. The discussion
<a href="https://sebastianraschka.com/faq/docs/sgd-methods.html" target="_blank">here</a> may be
useful. More material will be added later.
useful.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week39/ipynb/ipynb-week39-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week39/ipynb/week39.ipynb
+419 -417
View File
File diff suppressed because it is too large Load Diff
doc/src/week39/week39.do.txt
+85 -83
View File
@@ -11,11 +11,11 @@ DATE: today
* Reading recommendations:
See "lecture notes for week 39":"https://compphysics.github.io/MachineLearning/doc/web/course.html".
For a good discussion on gradient methods, see Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
For a good discussion on gradient methods, we would like to recommend Goodfellow et al section 4.3-4.5 and chapter 8. We will come back to the latter chapter in our discussion of Neural networks as well.
For Stochastic Gradient Descent, we recommend chapter 4 of Geron's text.
_For project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11_.
_For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11_.
These sections summarize neatly what we have done till now and point to what is coming with respect to deep learning.
!split
@@ -28,6 +28,8 @@ These sections summarize neatly what we have done till now and point to what is
!split
===== Optimization, the central part of any Machine Learning algortithm =====
The first few slides here are a repetition from last week.
Almost every problem in machine learning and data science starts with
a dataset $X$, a model $g(\beta)$, which is a function of the
parameters $\beta$ and a cost function $C(X, g(\beta))$ that allows
@@ -1068,6 +1070,86 @@ plt.show()
* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.
!split
===== Overview video on Stochastic Gradient Descent =====
"What is Stochastic Gradient Descent":"https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer"
!split
===== Batches and mini-batches =====
In gradient descent we compute the cost function and its gradient for all data points we have.
In large-scale applications such as the "ILSVRC challenge":"https://www.image-net.org/challenges/LSVRC/", the
training data can have on order of millions of examples. Hence, it
seems wasteful to compute the full cost function over the entire
training set in order to perform only a single parameter update. A
very common approach to addressing this challenge is to compute the
gradient over batches of the training data. For example, in current
a typical batch could contain some thousand examples from
an entire training set of several millions. This batch is then used to
perform a parameter update.
!split
===== Stochastic Gradient Descent (SGD) =====
In stochastic gradient descent, the extreme case is the case where we
have only one batch, that is we include the whole data set.
This process is called Stochastic Gradient
Descent (SGD) (or also sometimes on-line gradient descent). This is
relatively less common to see because in practice due to vectorized
code optimizations it can be computationally much more efficient to
evaluate the gradient for 100 examples, than the gradient for one
example 100 times. Even though SGD technically refers to using a
single example at a time to evaluate the gradient, you will hear
people use the term SGD even when referring to mini-batch gradient
descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD
for “Batch gradient descent” are rare to see), where it is usually
assumed that mini-batches are used. The size of the mini-batch is a
hyperparameter but it is not very common to cross-validate or bootstrap it. It is
usually based on memory constraints (if any), or set to some value,
e.g. 32, 64 or 128. We use powers of 2 in practice because many
vectorized operation implementations work faster when their inputs are
sized in powers of 2.
In our notes with SGD we mean stochastic gradient descent with mini-batches.
!split
===== Stochastic Gradient Descent =====
Stochastic gradient descent (SGD) and variants thereof address some of
the shortcomings of the Gradient descent method discussed above.
The underlying idea of SGD comes from the observation that the cost
function, which we want to minimize, can almost always be written as a
sum over $n$ data points $\{\mathbf{x}_i\}_{i=1}^n$,
!bt
\[
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
\]
!et
!split
===== Computation of gradients =====
This in turn means that the gradient can be
computed as a sum over $i$-gradients
!bt
\[
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
\]
!et
Stochasticity/randomness is introduced by only taking the
gradient on a subset of the data called minibatches. If there are $n$
data points and the size of each minibatch is $M$, there will be $n/M$
minibatches. We denote these minibatches by $B_k$ where
$k=1,\cdots,n/M$.
!split
@@ -1385,86 +1467,6 @@ plt.show()
!split
===== Overview video on Stochastic Gradient Descent =====
"What is Stochastic Gradient Descent":"https://www.youtube.com/watch?v=vMh0zPT0tLI&ab_channel=StatQuestwithJoshStarmer"
!split
===== Batches and mini-batches =====
In gradient descent we compute the cost function and its gradient for all data points we have.
In large-scale applications such as the "ILSVRC challenge":"https://www.image-net.org/challenges/LSVRC/", the
training data can have on order of millions of examples. Hence, it
seems wasteful to compute the full cost function over the entire
training set in order to perform only a single parameter update. A
very common approach to addressing this challenge is to compute the
gradient over batches of the training data. For example, in current
a typical batch could contain some thousand examples from
an entire training set of several millions. This batch is then used to
perform a parameter update.
!split
===== Stochastic Gradient Descent (SGD) =====
In stochastic gradient descent, the extreme case is the case where we
have only one batch, that is we include the whole data set.
This process is called Stochastic Gradient
Descent (SGD) (or also sometimes on-line gradient descent). This is
relatively less common to see because in practice due to vectorized
code optimizations it can be computationally much more efficient to
evaluate the gradient for 100 examples, than the gradient for one
example 100 times. Even though SGD technically refers to using a
single example at a time to evaluate the gradient, you will hear
people use the term SGD even when referring to mini-batch gradient
descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD
for “Batch gradient descent” are rare to see), where it is usually
assumed that mini-batches are used. The size of the mini-batch is a
hyperparameter but it is not very common to cross-validate or bootstrap it. It is
usually based on memory constraints (if any), or set to some value,
e.g. 32, 64 or 128. We use powers of 2 in practice because many
vectorized operation implementations work faster when their inputs are
sized in powers of 2.
In our notes with SGD we mean stochastic gradient descent with mini-batches.
!split
===== Stochastic Gradient Descent =====
Stochastic gradient descent (SGD) and variants thereof address some of
the shortcomings of the Gradient descent method discussed above.
The underlying idea of SGD comes from the observation that the cost
function, which we want to minimize, can almost always be written as a
sum over $n$ data points $\{\mathbf{x}_i\}_{i=1}^n$,
!bt
\[
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
\]
!et
!split
===== Computation of gradients =====
This in turn means that the gradient can be
computed as a sum over $i$-gradients
!bt
\[
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
\]
!et
Stochasticity/randomness is introduced by only taking the
gradient on a subset of the data called minibatches. If there are $n$
data points and the size of each minibatch is $M$, there will be $n/M$
minibatches. We denote these minibatches by $B_k$ where
$k=1,\cdots,n/M$.
!split
===== SGD example =====
@@ -1677,7 +1679,7 @@ plt.show()
In the above code, we have use replacement in setting up the
mini-batches. The discussion
"here":"https://sebastianraschka.com/faq/docs/sgd-methods.html" may be
useful. More material will be added later.
useful.
!split