lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating splines

Browse Source
This commit is contained in:
mhjensen
2018-10-12 04:55:15 +02:00
parent 74e29912f4
commit c71c008945
83 changed files with 14530 additions and 4991 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/Splines/html/._Splines-bs049.html
+112 -86
View File
@@ -69,74 +69,86 @@ Automatically generated HTML file from DocOnce source
('Steepest descent method', 2, None, '___sec19'),
('Steepest descent method', 2, None, '___sec20'),
('Final expressions', 2, None, '___sec21'),
('Code examples for steepest descent', 2, None, '___sec22'),
('Simple codes for steepest descent and conjugate gradient '
'using a $2\\times 2$ matrix, in c++, Python code to come',
2,
None,
'___sec22'),
'___sec23'),
('The routine for the steepest descent method',
2,
None,
'___sec23'),
('Steepest descent example', 2, None, '___sec24'),
('Conjugate gradient', 2, None, '___sec25'),
('Revisiting our first homework', 2, None, '___sec26'),
('Gradient descent example', 2, None, '___sec27'),
('The derivative of the cost/loss function', 2, None, '___sec28'),
('The Hessian matrix', 2, None, '___sec29'),
('Simple program', 2, None, '___sec30'),
('Gradient Descent Example', 2, None, '___sec31'),
'___sec24'),
('Steepest descent example', 2, None, '___sec25'),
('Conjugate gradient', 2, None, '___sec26'),
('Revisiting our first homework', 2, None, '___sec27'),
('Gradient descent example', 2, None, '___sec28'),
('The derivative of the cost/loss function', 2, None, '___sec29'),
('The Hessian matrix', 2, None, '___sec30'),
('Simple program', 2, None, '___sec31'),
('Gradient Descent Example', 2, None, '___sec32'),
('And a corresponding example using _scikit-learn_',
2,
None,
'___sec32'),
('Gradient descent and Ridge', 2, None, '___sec33'),
('Automatic differentiation', 2, None, '___sec34'),
('Using autograd', 2, None, '___sec35'),
('Autograd with more complicated functions', 2, None, '___sec36'),
'___sec33'),
('Gradient descent and Ridge', 2, None, '___sec34'),
('Automatic differentiation', 2, None, '___sec35'),
('Using autograd', 2, None, '___sec36'),
('Autograd with more complicated functions', 2, None, '___sec37'),
('More complicated functions using the elements of their '
'arguments directly',
2,
None,
'___sec37'),
'___sec38'),
('Functions using mathematical functions from Numpy',
2,
None,
'___sec38'),
('More autograd', 2, None, '___sec39'),
('And with loops', 2, None, '___sec40'),
('Using recursion', 2, None, '___sec41'),
('Unsupported functions', 2, None, '___sec42'),
'___sec39'),
('More autograd', 2, None, '___sec40'),
('And with loops', 2, None, '___sec41'),
('Using recursion', 2, None, '___sec42'),
('Unsupported functions', 2, None, '___sec43'),
('The syntax a.dot(b) when finding the dot product',
2,
None,
'___sec43'),
('Recommended to avoid', 2, None, '___sec44'),
('Stochastic Gradient Descent', 2, None, '___sec45'),
('Computation of gradients', 2, None, '___sec46'),
('SGD example', 2, None, '___sec47'),
('The gradient step', 2, None, '___sec48'),
('Simple example code', 2, None, '___sec49'),
('When do we stop?', 2, None, '___sec50'),
('Slightly different approach', 2, None, '___sec51'),
('Program for stochastic gradient', 2, None, '___sec52'),
('Momentum based methods', 2, None, '___sec53'),
('Conjugate gradient method', 2, None, '___sec54'),
'___sec44'),
('Recommended to avoid', 2, None, '___sec45'),
('Stochastic Gradient Descent', 2, None, '___sec46'),
('Computation of gradients', 2, None, '___sec47'),
('SGD example', 2, None, '___sec48'),
('The gradient step', 2, None, '___sec49'),
('Simple example code', 2, None, '___sec50'),
('When do we stop?', 2, None, '___sec51'),
('Slightly different approach', 2, None, '___sec52'),
('Program for stochastic gradient', 2, None, '___sec53'),
('Momentum based methods', 2, None, '___sec54'),
('Conjugate gradient method', 2, None, '___sec55'),
('Conjugate gradient method', 2, None, '___sec56'),
('Conjugate gradient method', 2, None, '___sec57'),
('Conjugate gradient method and iterations', 2, None, '___sec58'),
('Conjugate gradient method', 2, None, '___sec59'),
('Conjugate gradient method', 2, None, '___sec58'),
('Conjugate gradient method and iterations', 2, None, '___sec59'),
('Conjugate gradient method', 2, None, '___sec60'),
('Conjugate gradient method', 2, None, '___sec61'),
('Conjugate gradient method', 2, None, '___sec62'),
('Simple implementation of the Conjugate gradient algorithm',
2,
None,
'___sec62'),
'___sec63'),
('Broyden–Fletcher–Goldfarb–Shanno algorithm',
2,
None,
'___sec63')]}
'___sec64'),
('Using gradient descent methods, limitations',
2,
None,
'___sec65'),
('Momentum based GD', 2, None, '___sec66'),
('More on momentum based approaches', 2, None, '___sec67'),
('Momentum parameter', 2, None, '___sec68'),
('Second moment of the gradient', 2, None, '___sec69'),
('RMS prop', 2, None, '___sec70'),
('ADAM optimizer', 2, None, '___sec71'),
('Practical tips', 2, None, '___sec72')]}
end of tocinfo -->
<body>
@@ -196,48 +208,57 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._Splines-bs020.html#___sec19" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs021.html#___sec20" style="font-size: 80%;">Steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs022.html#___sec21" style="font-size: 80%;">Final expressions</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">The routine for the steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">Steepest descent example</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Conjugate gradient</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Revisiting our first homework</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Gradient descent example</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The Hessian matrix</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">Simple program</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Gradient Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">Gradient descent and Ridge</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Automatic differentiation</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Using autograd</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Autograd with more complicated functions</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">More autograd</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">And with loops</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">Using recursion</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Unsupported functions</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">Recommended to avoid</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">SGD example</a></li>
<!-- navigation toc: --> <li><a href="#___sec48" style="font-size: 80%;">The gradient step</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">Simple example code</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">When do we stop?</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">Slightly different approach</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs053.html#___sec52" style="font-size: 80%;">Program for stochastic gradient</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs054.html#___sec53" style="font-size: 80%;">Momentum based methods</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs055.html#___sec54" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs023.html#___sec22" style="font-size: 80%;">Code examples for steepest descent</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs024.html#___sec23" style="font-size: 80%;">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs025.html#___sec24" style="font-size: 80%;">The routine for the steepest descent method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs026.html#___sec25" style="font-size: 80%;">Steepest descent example</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs027.html#___sec26" style="font-size: 80%;">Conjugate gradient</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs028.html#___sec27" style="font-size: 80%;">Revisiting our first homework</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs029.html#___sec28" style="font-size: 80%;">Gradient descent example</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs030.html#___sec29" style="font-size: 80%;">The derivative of the cost/loss function</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs031.html#___sec30" style="font-size: 80%;">The Hessian matrix</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs032.html#___sec31" style="font-size: 80%;">Simple program</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs033.html#___sec32" style="font-size: 80%;">Gradient Descent Example</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs034.html#___sec33" style="font-size: 80%;">And a corresponding example using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs035.html#___sec34" style="font-size: 80%;">Gradient descent and Ridge</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs036.html#___sec35" style="font-size: 80%;">Automatic differentiation</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs037.html#___sec36" style="font-size: 80%;">Using autograd</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs038.html#___sec37" style="font-size: 80%;">Autograd with more complicated functions</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs039.html#___sec38" style="font-size: 80%;">More complicated functions using the elements of their arguments directly</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs040.html#___sec39" style="font-size: 80%;">Functions using mathematical functions from Numpy</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs041.html#___sec40" style="font-size: 80%;">More autograd</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs042.html#___sec41" style="font-size: 80%;">And with loops</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs043.html#___sec42" style="font-size: 80%;">Using recursion</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs044.html#___sec43" style="font-size: 80%;">Unsupported functions</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs045.html#___sec44" style="font-size: 80%;">The syntax a.dot(b) when finding the dot product</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs046.html#___sec45" style="font-size: 80%;">Recommended to avoid</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs047.html#___sec46" style="font-size: 80%;">Stochastic Gradient Descent</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs048.html#___sec47" style="font-size: 80%;">Computation of gradients</a></li>
<!-- navigation toc: --> <li><a href="#___sec48" style="font-size: 80%;">SGD example</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs050.html#___sec49" style="font-size: 80%;">The gradient step</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs051.html#___sec50" style="font-size: 80%;">Simple example code</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs052.html#___sec51" style="font-size: 80%;">When do we stop?</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs053.html#___sec52" style="font-size: 80%;">Slightly different approach</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs054.html#___sec53" style="font-size: 80%;">Program for stochastic gradient</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs055.html#___sec54" style="font-size: 80%;">Momentum based methods</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs056.html#___sec55" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs057.html#___sec56" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs058.html#___sec57" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs059.html#___sec58" style="font-size: 80%;">Conjugate gradient method and iterations</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs060.html#___sec59" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs059.html#___sec58" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs060.html#___sec59" style="font-size: 80%;">Conjugate gradient method and iterations</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs061.html#___sec60" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs062.html#___sec61" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs063.html#___sec62" style="font-size: 80%;">Simple implementation of the Conjugate gradient algorithm</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs064.html#___sec63" style="font-size: 80%;">Broyden–Fletcher–Goldfarb–Shanno algorithm</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs063.html#___sec62" style="font-size: 80%;">Conjugate gradient method</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs064.html#___sec63" style="font-size: 80%;">Simple implementation of the Conjugate gradient algorithm</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs065.html#___sec64" style="font-size: 80%;">Broyden–Fletcher–Goldfarb–Shanno algorithm</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs066.html#___sec65" style="font-size: 80%;">Using gradient descent methods, limitations</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs067.html#___sec66" style="font-size: 80%;">Momentum based GD</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs068.html#___sec67" style="font-size: 80%;">More on momentum based approaches</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs069.html#___sec68" style="font-size: 80%;">Momentum parameter</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs070.html#___sec69" style="font-size: 80%;">Second moment of the gradient</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs071.html#___sec70" style="font-size: 80%;">RMS prop</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs072.html#___sec71" style="font-size: 80%;">ADAM optimizer</a></li>
<!-- navigation toc: --> <li><a href="._Splines-bs073.html#___sec72" style="font-size: 80%;">Practical tips</a></li>
</ul>
</li>
@@ -253,22 +274,27 @@ MathJax.Hub.Config({
<a name="part0049"></a>
<!-- !split -->
<h2 id="___sec48" class="anchor">The gradient step </h2>
<h2 id="___sec48" class="anchor">SGD example </h2>
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
\( B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 =
(\mathbf{x}_9,\mathbf{x}_{10}) \). Note that if you choose \( M=1 \) you
have only a single batch with all data points and on the other extreme,
you may choose \( M=n \) resulting in a minibatch for each datapoint, i.e
\( B_k = \mathbf{x}_k \).
<p>
Thus a gradient descent step now looks like
The idea is now to approximate the gradient by replacing the sum over
all data points with a sum over the data points in one the minibatches
picked at random in each gradient descent step
$$
\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta})
\nabla_{\beta}
C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
c_i(\mathbf{x}_i, \mathbf{\beta}).
$$
<p>
where \( k \) is picked at random with equal
probability from \( [1,n/M] \). An iteration over the number of
minibathces (n/M) is commonly referred to as an epoch. Thus it is
typical to choose a number of epochs and for each epoch iterate over
the number of minibatches, as exemplified in the code below.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
@@ -295,7 +321,7 @@ the number of minibatches, as exemplified in the code below.
<li><a href="._Splines-bs057.html">58</a></li>
<li><a href="._Splines-bs058.html">59</a></li>
<li><a href="">...</a></li>
<li><a href="._Splines-bs064.html">65</a></li>
<li><a href="._Splines-bs073.html">74</a></li>
<li><a href="._Splines-bs050.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->