lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week 48

Browse Source
This commit is contained in:
mhjensen
2020-11-27 06:44:20 +01:00
parent 0ec6719302
commit 6397ae1385
22 changed files with 185 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week48/html/._week48-bs000.html
+1 -1
View File
@@ -284,7 +284,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 26, 2020</h4></center> <!-- date -->
<center><h4>Nov 27, 2020</h4></center> <!-- date -->
<br>
<p>
doc/pub/week48/html/._week48-bs001.html
+2 -2
View File
@@ -268,8 +268,8 @@ MathJax.Hub.Config({
<h2 id="___sec0" class="anchor">Overview of week 48 </h2>
<ul>
<li> <b>Thursday</b>: Support Vector Machines: Kernels, Classification and Regression</li>
<li> <b>Friday</b>: Summary of course with perspectives for future studies</li>
<li> <b>Thursday</b>: Support Vector Machines: Kernels, Classification and Regression. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember26.mp4?vrtx=view-as-webpage" target="_self">Video of Lecture</a>.</li>
<li> <b>Friday</b>: Summary of course with perspectives for future studies. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember27.mp4?vrtx=view-as-webpage" target="_self">Video of Lecture</a>.</li>
</ul>
Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion.
doc/pub/week48/html/._week48-bs012.html
+1 -1
View File
@@ -272,7 +272,7 @@ If we use Python as programming language and wish to venture beyond
<b>scikit-learn</b>, <b>tensorflow</b> and similar software which makes our
lives so much easier, we need to dive into the wonderful world of
quadratic programming. We can, if we wish, solve the minimization
problem using say standard gradient methods or conjugate gradient
problem using standard gradient methods or conjugate gradient
methods. However, these methods tend to exhibit a rather slow
converge. So, welcome to the promised land of quadratic programming.
doc/pub/week48/html/._week48-bs015.html
+1 -1
View File
@@ -280,7 +280,7 @@ $$
<ol>
<li> With a given kernel we can thus define the matrix \( \boldsymbol{P} \).</li>
<li> The matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \). Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up.</li>
<li> The vector \( \boldsymbol{q} \) has all elements equal -1.</li>
<li> The vector \( \boldsymbol{q} \) has all elements equal to \( -1 \).</li>
<li> The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \).</li>
<li> To set up the matrix \( \boldsymbol{G} \) we note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into \( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).</li>
</ol>
doc/pub/week48/html/._week48-bs021.html
+3 -3
View File
@@ -268,16 +268,16 @@ MathJax.Hub.Config({
<h2 id="___sec21" class="anchor">Statistical analysis and optimization of data </h2>
<p>
The following topics be covered
The following topics have been discussed:
<ol>
<li> Basic concepts, expectation values, variance, covariance, correlation functions and errors;</li>
<li> Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;</li>
<li> Central elements from linear algebra</li>
<li> Central elements from linear algebra, matrix inversion and SVD</li>
<li> Gradient methods for data optimization</li>
<li> Estimation of errors using cross-validation, bootstrapping and jackknife methods;</li>
<li> Practical optimization using Singular-value decomposition and least squares for parameterizing data.</li>
<li> Principal Component Analysis.</li>
<li> Principal Component Analysis to reduce the number of features.</li>
</ol>
<p>
doc/pub/week48/html/._week48-bs023.html
+1 -1
View File
@@ -281,7 +281,7 @@ ethical conduct is emphasized throughout the course.
<ul>
<li> Understand linear methods for regression and classification;</li>
<li> Learn about neural network;</li>
<li> Learn about baggin, boosting and trees</li>
<li> Learn about bagging, boosting and trees</li>
<li> Support vector machines</li>
<li> Learn about basic data analysis;</li>
<li> Be capable of extending the acquired knowledge to other systems and cases;</li>
doc/pub/week48/html/._week48-bs025.html
+1
View File
@@ -276,6 +276,7 @@ Where to find recent results:
<li> <b>ICLR</b>: <a href="https://openreview.net/group?id=ICLR.cc/2018/Conference#accepted-oral-papers" target="_self">International Conference on Learning Representations</a></li>
<li> <b>ICML</b>: International Conference on Machine Learning</li>
<li> <a href="http://www.jmlr.org/papers/v19/" target="_self">Journal of Machine Learning Research</a></li>
<li> <a href="https://arxiv.org/list/cs.LG/recent" target="_self">Follow ML on ArXiv</a></li>
</ol>
<p>
doc/pub/week48/html/._week48-bs026.html
+3 -3
View File
@@ -268,11 +268,11 @@ MathJax.Hub.Config({
<h2 id="___sec26" class="anchor">Starting your Machine Learning Project </h2>
<ol>
<li> Identify problem type: classification, generation, regression</li>
<li> Identify problem type: classification, regression</li>
<li> Consider your data carefully</li>
<li> Choose a simple model that fits 1. and 2.</li>
<li> Consider your data carefully again&#8230; data representation</li>
<li> Based on results, feedback loop to earliest possible point</li>
<li> Consider your data carefully again! Think of data representation more carefully.</li>
<li> Based on your results, feedback loop to earliest possible point</li>
</ol>
<p>
doc/pub/week48/html/._week48-bs030.html
+4 -1
View File
@@ -279,11 +279,14 @@ MathJax.Hub.Config({
<ul>
<li> RMSProp</li>
<li> Adam</li>
<li> and more</li>
</ul>
</ol>
Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft classifiers, depths of trees and many other. Need to explore a large set of hyperparameters and regularization methods.
Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft
classifiers, depths of trees and many other. Need to explore a large
set of hyperparameters and regularization methods.
<p>
<p>
doc/pub/week48/html/._week48-bs031.html
+2 -2
View File
@@ -271,8 +271,8 @@ MathJax.Hub.Config({
When do we resample?
<ol>
<li> Bootstrap</li>
<li> Cross-validation</li>
<li> <a href="https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A" target="_self">Bootstrap</a></li>
<li> <a href="https://www.youtube.com/watch?v=fSytzGwwBVw&ab_channel=StatQuestwithJoshStarmer" target="_self">Cross-validation</a></li>
<li> Jackknife and many other</li>
</ol>
doc/pub/week48/html/._week48-bs044.html
+1 -1
View File
@@ -268,7 +268,7 @@ MathJax.Hub.Config({
<h2 id="___sec44" class="anchor">Joint distribution </h2>
<p>
The restricted Boltzmann machine is described by a Bolztmann distribution
The restricted Boltzmann machine is described by a Boltzmann distribution
$$
\begin{align}
P_{rbm}(\mathbf{x},\mathbf{h}) = \frac{1}{Z} e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})},
doc/pub/week48/html/._week48-bs047.html
+3
View File
@@ -281,6 +281,9 @@ Other types of units include:
<li> Rectified linear units</li>
</ol>
To read more, see <a href="https://github.com/CompPhysics/ComputationalPhysics2/blob/gh-pages/doc/pub/notebook2/ipynb/notebook2.ipynb" target="_self">Lectures on Boltzmann machines in Physics</a>.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
doc/pub/week48/html/._week48-bs050.html
+2
View File
@@ -297,6 +297,8 @@ learning.
<p>
See also A. Geron's textbook, chapter 16.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
doc/pub/week48/html/._week48-bs056.html
+3
View File
@@ -287,6 +287,9 @@ still very optimistic about the future of machine learning. As we look
forward to the future, here are what we think the research hotspots in
the next ten years will be.
<p>
See the article on <a href="https://www.frontiersin.org/articles/10.3389/frai.2020.00025/full" target="_self">Discovery of Physics From Data: Universal Laws and Discrepancies</a>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
doc/pub/week48/html/._week48-bs057.html
+4 -1
View File
@@ -285,7 +285,10 @@ contrast, rational humans tend to reply on clear and trustworthy
causality relations obtained via logical reasoning on real and clear
facts. It is one of the core goals of explainable machine learning to
transition from solving problems by data correlation to solving
problems by logical reasoning. Bayesian Machine Learning is one of the exciting research directions in this field.
problems by logical reasoning.
<p>
<b>Bayesian Machine Learning is one of the exciting research directions in this field</b>.
<p>
<p>
doc/pub/week48/html/week48-bs.html
+1 -1
View File
@@ -284,7 +284,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 26, 2020</h4></center> <!-- date -->
<center><h4>Nov 27, 2020</h4></center> <!-- date -->
<br>
<p>
doc/pub/week48/html/week48-reveal.html
+31 -17
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Nov 26, 2020</h4></center> <!-- date -->
<center><h4>Nov 27, 2020</h4></center> <!-- date -->
<br>
<p>
@@ -162,8 +162,8 @@ MathJax.Hub.Config({
<h2 id="___sec0">Overview of week 48 </h2>
<ul>
<p><li> <b>Thursday</b>: Support Vector Machines: Kernels, Classification and Regression</li>
<p><li> <b>Friday</b>: Summary of course with perspectives for future studies</li>
<p><li> <b>Thursday</b>: Support Vector Machines: Kernels, Classification and Regression. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember26.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a>.</li>
<p><li> <b>Friday</b>: Summary of course with perspectives for future studies. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember27.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a>.</li>
</ul>
<p>
@@ -677,7 +677,7 @@ If we use Python as programming language and wish to venture beyond
<b>scikit-learn</b>, <b>tensorflow</b> and similar software which makes our
lives so much easier, we need to dive into the wonderful world of
quadratic programming. We can, if we wish, solve the minimization
problem using say standard gradient methods or conjugate gradient
problem using standard gradient methods or conjugate gradient
methods. However, these methods tend to exhibit a rather slow
converge. So, welcome to the promised land of quadratic programming.
@@ -816,7 +816,7 @@ $$
<ol>
<p><li> With a given kernel we can thus define the matrix \( \boldsymbol{P} \).</li>
<p><li> The matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \). Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up.</li>
<p><li> The vector \( \boldsymbol{q} \) has all elements equal -1.</li>
<p><li> The vector \( \boldsymbol{q} \) has all elements equal to \( -1 \).</li>
<p><li> The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \).</li>
<p><li> To set up the matrix \( \boldsymbol{G} \) we note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into \( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).</li>
</ol>
@@ -904,16 +904,16 @@ The course has two central parts
<h2 id="___sec21">Statistical analysis and optimization of data </h2>
<p>
The following topics be covered
The following topics have been discussed:
<ol>
<p><li> Basic concepts, expectation values, variance, covariance, correlation functions and errors;</li>
<p><li> Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;</li>
<p><li> Central elements from linear algebra</li>
<p><li> Central elements from linear algebra, matrix inversion and SVD</li>
<p><li> Gradient methods for data optimization</li>
<p><li> Estimation of errors using cross-validation, bootstrapping and jackknife methods;</li>
<p><li> Practical optimization using Singular-value decomposition and least squares for parameterizing data.</li>
<p><li> Principal Component Analysis.</li>
<p><li> Principal Component Analysis to reduce the number of features.</li>
</ol>
</section>
@@ -977,7 +977,7 @@ ethical conduct is emphasized throughout the course.
<ul>
<p><li> Understand linear methods for regression and classification;</li>
<p><li> Learn about neural network;</li>
<p><li> Learn about baggin, boosting and trees</li>
<p><li> Learn about bagging, boosting and trees</li>
<p><li> Support vector machines</li>
<p><li> Learn about basic data analysis;</li>
<p><li> Be capable of extending the acquired knowledge to other systems and cases;</li>
@@ -1017,6 +1017,7 @@ Where to find recent results:
<p><li> <b>ICLR</b>: <a href="https://openreview.net/group?id=ICLR.cc/2018/Conference#accepted-oral-papers" target="_blank">International Conference on Learning Representations</a></li>
<p><li> <b>ICML</b>: International Conference on Machine Learning</li>
<p><li> <a href="http://www.jmlr.org/papers/v19/" target="_blank">Journal of Machine Learning Research</a></li>
<p><li> <a href="https://arxiv.org/list/cs.LG/recent" target="_blank">Follow ML on ArXiv</a></li>
</ol>
</section>
@@ -1025,11 +1026,11 @@ Where to find recent results:
<h2 id="___sec26">Starting your Machine Learning Project </h2>
<ol>
<p><li> Identify problem type: classification, generation, regression</li>
<p><li> Identify problem type: classification, regression</li>
<p><li> Consider your data carefully</li>
<p><li> Choose a simple model that fits 1. and 2.</li>
<p><li> Consider your data carefully again&#8230; data representation</li>
<p><li> Based on results, feedback loop to earliest possible point</li>
<p><li> Consider your data carefully again! Think of data representation more carefully.</li>
<p><li> Based on your results, feedback loop to earliest possible point</li>
</ol>
</section>
@@ -1114,12 +1115,16 @@ Where to find recent results:
<p><li> RMSProp</li>
<p><li> Adam</li>
<p><li> and more</li>
</ul>
<p>
</ol>
<p>
Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft classifiers, depths of trees and many other. Need to explore a large set of hyperparameters and regularization methods.
Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft
classifiers, depths of trees and many other. Need to explore a large
set of hyperparameters and regularization methods.
</section>
@@ -1130,8 +1135,8 @@ Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft classifie
When do we resample?
<ol>
<p><li> Bootstrap</li>
<p><li> Cross-validation</li>
<p><li> <a href="https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A" target="_blank">Bootstrap</a></li>
<p><li> <a href="https://www.youtube.com/watch?v=fSytzGwwBVw&ab_channel=StatQuestwithJoshStarmer" target="_blank">Cross-validation</a></li>
<p><li> Jackknife and many other</li>
</ol>
</section>
@@ -1380,7 +1385,7 @@ over (integrated out).
<h2 id="___sec44">Joint distribution </h2>
<p>
The restricted Boltzmann machine is described by a Bolztmann distribution
The restricted Boltzmann machine is described by a Boltzmann distribution
<p>&nbsp;<br>
$$
\begin{align}
@@ -1488,6 +1493,9 @@ Other types of units include:
<p><li> Binomial units</li>
<p><li> Rectified linear units</li>
</ol>
<p>
To read more, see <a href="https://github.com/CompPhysics/ComputationalPhysics2/blob/gh-pages/doc/pub/notebook2/ipynb/notebook2.ipynb" target="_blank">Lectures on Boltzmann machines in Physics</a>.
</section>
@@ -1687,6 +1695,9 @@ models for data fitting? Although there are many challenges, we are
still very optimistic about the future of machine learning. As we look
forward to the future, here are what we think the research hotspots in
the next ten years will be.
<p>
See the article on <a href="https://www.frontiersin.org/articles/10.3389/frai.2020.00025/full" target="_blank">Discovery of Physics From Data: Universal Laws and Discrepancies</a>
</section>
@@ -1711,7 +1722,10 @@ contrast, rational humans tend to reply on clear and trustworthy
causality relations obtained via logical reasoning on real and clear
facts. It is one of the core goals of explainable machine learning to
transition from solving problems by data correlation to solving
problems by logical reasoning. Bayesian Machine Learning is one of the exciting research directions in this field.
problems by logical reasoning.
<p>
<b>Bayesian Machine Learning is one of the exciting research directions in this field</b>.
</section>
doc/pub/week48/html/week48-solarized.html
+32 -17
View File
@@ -209,7 +209,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 26, 2020</h4></center> <!-- date -->
<center><h4>Nov 27, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -217,8 +217,8 @@ MathJax.Hub.Config({
<h2 id="___sec0">Overview of week 48 </h2>
<ul>
<li> <b>Thursday</b>: Support Vector Machines: Kernels, Classification and Regression</li>
<li> <b>Friday</b>: Summary of course with perspectives for future studies</li>
<li> <b>Thursday</b>: Support Vector Machines: Kernels, Classification and Regression. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember26.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a>.</li>
<li> <b>Friday</b>: Summary of course with perspectives for future studies. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember27.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a>.</li>
</ul>
Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion.
@@ -706,7 +706,7 @@ If we use Python as programming language and wish to venture beyond
<b>scikit-learn</b>, <b>tensorflow</b> and similar software which makes our
lives so much easier, we need to dive into the wonderful world of
quadratic programming. We can, if we wish, solve the minimization
problem using say standard gradient methods or conjugate gradient
problem using standard gradient methods or conjugate gradient
methods. However, these methods tend to exhibit a rather slow
converge. So, welcome to the promised land of quadratic programming.
@@ -828,7 +828,7 @@ $$
<ol>
<li> With a given kernel we can thus define the matrix \( \boldsymbol{P} \).</li>
<li> The matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \). Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up.</li>
<li> The vector \( \boldsymbol{q} \) has all elements equal -1.</li>
<li> The vector \( \boldsymbol{q} \) has all elements equal to \( -1 \).</li>
<li> The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \).</li>
<li> To set up the matrix \( \boldsymbol{G} \) we note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into \( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).</li>
</ol>
@@ -912,16 +912,16 @@ The course has two central parts
<h2 id="___sec21">Statistical analysis and optimization of data </h2>
<p>
The following topics be covered
The following topics have been discussed:
<ol>
<li> Basic concepts, expectation values, variance, covariance, correlation functions and errors;</li>
<li> Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;</li>
<li> Central elements from linear algebra</li>
<li> Central elements from linear algebra, matrix inversion and SVD</li>
<li> Gradient methods for data optimization</li>
<li> Estimation of errors using cross-validation, bootstrapping and jackknife methods;</li>
<li> Practical optimization using Singular-value decomposition and least squares for parameterizing data.</li>
<li> Principal Component Analysis.</li>
<li> Principal Component Analysis to reduce the number of features.</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -986,7 +986,7 @@ ethical conduct is emphasized throughout the course.
<ul>
<li> Understand linear methods for regression and classification;</li>
<li> Learn about neural network;</li>
<li> Learn about baggin, boosting and trees</li>
<li> Learn about bagging, boosting and trees</li>
<li> Support vector machines</li>
<li> Learn about basic data analysis;</li>
<li> Be capable of extending the acquired knowledge to other systems and cases;</li>
@@ -1024,6 +1024,7 @@ Where to find recent results:
<li> <b>ICLR</b>: <a href="https://openreview.net/group?id=ICLR.cc/2018/Conference#accepted-oral-papers" target="_blank">International Conference on Learning Representations</a></li>
<li> <b>ICML</b>: International Conference on Machine Learning</li>
<li> <a href="http://www.jmlr.org/papers/v19/" target="_blank">Journal of Machine Learning Research</a></li>
<li> <a href="https://arxiv.org/list/cs.LG/recent" target="_blank">Follow ML on ArXiv</a></li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1031,11 +1032,11 @@ Where to find recent results:
<h2 id="___sec26">Starting your Machine Learning Project </h2>
<ol>
<li> Identify problem type: classification, generation, regression</li>
<li> Identify problem type: classification, regression</li>
<li> Consider your data carefully</li>
<li> Choose a simple model that fits 1. and 2.</li>
<li> Consider your data carefully again&#8230; data representation</li>
<li> Based on results, feedback loop to earliest possible point</li>
<li> Consider your data carefully again! Think of data representation more carefully.</li>
<li> Based on your results, feedback loop to earliest possible point</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1113,11 +1114,14 @@ Where to find recent results:
<ul>
<li> RMSProp</li>
<li> Adam</li>
<li> and more</li>
</ul>
</ol>
Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft classifiers, depths of trees and many other. Need to explore a large set of hyperparameters and regularization methods.
Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft
classifiers, depths of trees and many other. Need to explore a large
set of hyperparameters and regularization methods.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1128,8 +1132,8 @@ Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft classifie
When do we resample?
<ol>
<li> Bootstrap</li>
<li> Cross-validation</li>
<li> <a href="https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A" target="_blank">Bootstrap</a></li>
<li> <a href="https://www.youtube.com/watch?v=fSytzGwwBVw&ab_channel=StatQuestwithJoshStarmer" target="_blank">Cross-validation</a></li>
<li> Jackknife and many other</li>
</ol>
@@ -1371,7 +1375,7 @@ over (integrated out).
<h2 id="___sec44">Joint distribution </h2>
<p>
The restricted Boltzmann machine is described by a Bolztmann distribution
The restricted Boltzmann machine is described by a Boltzmann distribution
$$
\begin{align}
P_{rbm}(\mathbf{x},\mathbf{h}) = \frac{1}{Z} e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})},
@@ -1474,6 +1478,9 @@ Other types of units include:
<li> Rectified linear units</li>
</ol>
To read more, see <a href="https://github.com/CompPhysics/ComputationalPhysics2/blob/gh-pages/doc/pub/notebook2/ipynb/notebook2.ipynb" target="_blank">Lectures on Boltzmann machines in Physics</a>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec48">Autoencoders: Overarching view </h2>
@@ -1568,6 +1575,8 @@ learning.
<p>
See also A. Geron's textbook, chapter 16.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec51">Transfer learning </h2>
@@ -1670,6 +1679,9 @@ still very optimistic about the future of machine learning. As we look
forward to the future, here are what we think the research hotspots in
the next ten years will be.
<p>
See the article on <a href="https://www.frontiersin.org/articles/10.3389/frai.2020.00025/full" target="_blank">Discovery of Physics From Data: Universal Laws and Discrepancies</a>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1693,7 +1705,10 @@ contrast, rational humans tend to reply on clear and trustworthy
causality relations obtained via logical reasoning on real and clear
facts. It is one of the core goals of explainable machine learning to
transition from solving problems by data correlation to solving
problems by logical reasoning. Bayesian Machine Learning is one of the exciting research directions in this field.
problems by logical reasoning.
<p>
<b>Bayesian Machine Learning is one of the exciting research directions in this field</b>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week48/html/week48.html
+32 -17
View File
@@ -214,7 +214,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 26, 2020</h4></center> <!-- date -->
<center><h4>Nov 27, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -222,8 +222,8 @@ MathJax.Hub.Config({
<h2 id="___sec0">Overview of week 48 </h2>
<ul>
<li> <b>Thursday</b>: Support Vector Machines: Kernels, Classification and Regression</li>
<li> <b>Friday</b>: Summary of course with perspectives for future studies</li>
<li> <b>Thursday</b>: Support Vector Machines: Kernels, Classification and Regression. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember26.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a>.</li>
<li> <b>Friday</b>: Summary of course with perspectives for future studies. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember27.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a>.</li>
</ul>
Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion.
@@ -711,7 +711,7 @@ If we use Python as programming language and wish to venture beyond
<b>scikit-learn</b>, <b>tensorflow</b> and similar software which makes our
lives so much easier, we need to dive into the wonderful world of
quadratic programming. We can, if we wish, solve the minimization
problem using say standard gradient methods or conjugate gradient
problem using standard gradient methods or conjugate gradient
methods. However, these methods tend to exhibit a rather slow
converge. So, welcome to the promised land of quadratic programming.
@@ -833,7 +833,7 @@ $$
<ol>
<li> With a given kernel we can thus define the matrix \( \boldsymbol{P} \).</li>
<li> The matrix \( \boldsymbol{P} \) has matrix elements \( p_{ij}=y_iy_jK(\boldsymbol{x}_i,\boldsymbol{x}_j) \). Given a kernel \( K \) and the targets \( y_i \) this matrix is easy to set up.</li>
<li> The vector \( \boldsymbol{q} \) has all elements equal -1.</li>
<li> The vector \( \boldsymbol{q} \) has all elements equal to \( -1 \).</li>
<li> The constraint \( \boldsymbol{y}^T\boldsymbol{\lambda}=0 \) leads to \( f=0 \) and \( \boldsymbol{A}=\boldsymbol{y} \).</li>
<li> To set up the matrix \( \boldsymbol{G} \) we note that the inequalities \( 0\leq \lambda_i \leq C \) can be split up into \( 0\leq \lambda_i \) and \( \lambda_i \leq C \). These two inequalities define then the matrix \( \boldsymbol{G} \) and the vector \( \boldsymbol{h} \).</li>
</ol>
@@ -917,16 +917,16 @@ The course has two central parts
<h2 id="___sec21">Statistical analysis and optimization of data </h2>
<p>
The following topics be covered
The following topics have been discussed:
<ol>
<li> Basic concepts, expectation values, variance, covariance, correlation functions and errors;</li>
<li> Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;</li>
<li> Central elements from linear algebra</li>
<li> Central elements from linear algebra, matrix inversion and SVD</li>
<li> Gradient methods for data optimization</li>
<li> Estimation of errors using cross-validation, bootstrapping and jackknife methods;</li>
<li> Practical optimization using Singular-value decomposition and least squares for parameterizing data.</li>
<li> Principal Component Analysis.</li>
<li> Principal Component Analysis to reduce the number of features.</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -991,7 +991,7 @@ ethical conduct is emphasized throughout the course.
<ul>
<li> Understand linear methods for regression and classification;</li>
<li> Learn about neural network;</li>
<li> Learn about baggin, boosting and trees</li>
<li> Learn about bagging, boosting and trees</li>
<li> Support vector machines</li>
<li> Learn about basic data analysis;</li>
<li> Be capable of extending the acquired knowledge to other systems and cases;</li>
@@ -1029,6 +1029,7 @@ Where to find recent results:
<li> <b>ICLR</b>: <a href="https://openreview.net/group?id=ICLR.cc/2018/Conference#accepted-oral-papers" target="_blank">International Conference on Learning Representations</a></li>
<li> <b>ICML</b>: International Conference on Machine Learning</li>
<li> <a href="http://www.jmlr.org/papers/v19/" target="_blank">Journal of Machine Learning Research</a></li>
<li> <a href="https://arxiv.org/list/cs.LG/recent" target="_blank">Follow ML on ArXiv</a></li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1036,11 +1037,11 @@ Where to find recent results:
<h2 id="___sec26">Starting your Machine Learning Project </h2>
<ol>
<li> Identify problem type: classification, generation, regression</li>
<li> Identify problem type: classification, regression</li>
<li> Consider your data carefully</li>
<li> Choose a simple model that fits 1. and 2.</li>
<li> Consider your data carefully again&#8230; data representation</li>
<li> Based on results, feedback loop to earliest possible point</li>
<li> Consider your data carefully again! Think of data representation more carefully.</li>
<li> Based on your results, feedback loop to earliest possible point</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1118,11 +1119,14 @@ Where to find recent results:
<ul>
<li> RMSProp</li>
<li> Adam</li>
<li> and more</li>
</ul>
</ol>
Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft classifiers, depths of trees and many other. Need to explore a large set of hyperparameters and regularization methods.
Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft
classifiers, depths of trees and many other. Need to explore a large
set of hyperparameters and regularization methods.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1133,8 +1137,8 @@ Which regularization and hyperparameters? \( L_1 \) or \( L_2 \), soft classifie
When do we resample?
<ol>
<li> Bootstrap</li>
<li> Cross-validation</li>
<li> <a href="https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A" target="_blank">Bootstrap</a></li>
<li> <a href="https://www.youtube.com/watch?v=fSytzGwwBVw&ab_channel=StatQuestwithJoshStarmer" target="_blank">Cross-validation</a></li>
<li> Jackknife and many other</li>
</ol>
@@ -1376,7 +1380,7 @@ over (integrated out).
<h2 id="___sec44">Joint distribution </h2>
<p>
The restricted Boltzmann machine is described by a Bolztmann distribution
The restricted Boltzmann machine is described by a Boltzmann distribution
$$
\begin{align}
P_{rbm}(\mathbf{x},\mathbf{h}) = \frac{1}{Z} e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})},
@@ -1479,6 +1483,9 @@ Other types of units include:
<li> Rectified linear units</li>
</ol>
To read more, see <a href="https://github.com/CompPhysics/ComputationalPhysics2/blob/gh-pages/doc/pub/notebook2/ipynb/notebook2.ipynb" target="_blank">Lectures on Boltzmann machines in Physics</a>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec48">Autoencoders: Overarching view </h2>
@@ -1573,6 +1580,8 @@ learning.
<p>
See also A. Geron's textbook, chapter 16.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec51">Transfer learning </h2>
@@ -1675,6 +1684,9 @@ still very optimistic about the future of machine learning. As we look
forward to the future, here are what we think the research hotspots in
the next ten years will be.
<p>
See the article on <a href="https://www.frontiersin.org/articles/10.3389/frai.2020.00025/full" target="_blank">Discovery of Physics From Data: Universal Laws and Discrepancies</a>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1698,7 +1710,10 @@ contrast, rational humans tend to reply on clear and trustworthy
causality relations obtained via logical reasoning on real and clear
facts. It is one of the core goals of explainable machine learning to
transition from solving problems by data correlation to solving
problems by logical reasoning. Bayesian Machine Learning is one of the exciting research directions in this field.
problems by logical reasoning.
<p>
<b>Bayesian Machine Learning is one of the exciting research directions in this field</b>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week48/ipynb/ipynb-week48-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week48/ipynb/week48.ipynb
+31 -17
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Nov 26, 2020**\n",
"Date: **Nov 27, 2020**\n",
"\n",
"Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -18,9 +18,9 @@
"\n",
"## Overview of week 48\n",
"\n",
"* **Thursday**: Support Vector Machines: Kernels, Classification and Regression\n",
"* **Thursday**: Support Vector Machines: Kernels, Classification and Regression. [Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember26.mp4?vrtx=view-as-webpage).\n",
"\n",
"* **Friday**: Summary of course with perspectives for future studies\n",
"* **Friday**: Summary of course with perspectives for future studies. [Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember27.mp4?vrtx=view-as-webpage).\n",
"\n",
"Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion.\n",
"\n",
@@ -603,7 +603,7 @@
"**scikit-learn**, **tensorflow** and similar software which makes our\n",
"lives so much easier, we need to dive into the wonderful world of\n",
"quadratic programming. We can, if we wish, solve the minimization\n",
"problem using say standard gradient methods or conjugate gradient\n",
"problem using standard gradient methods or conjugate gradient\n",
"methods. However, these methods tend to exhibit a rather slow\n",
"converge. So, welcome to the promised land of quadratic programming.\n",
"\n",
@@ -823,7 +823,7 @@
"\n",
"2. The matrix $\\boldsymbol{P}$ has matrix elements $p_{ij}=y_iy_jK(\\boldsymbol{x}_i,\\boldsymbol{x}_j)$. Given a kernel $K$ and the targets $y_i$ this matrix is easy to set up.\n",
"\n",
"3. The vector $\\boldsymbol{q}$ has all elements equal -1.\n",
"3. The vector $\\boldsymbol{q}$ has all elements equal to $-1$.\n",
"\n",
"4. The constraint $\\boldsymbol{y}^T\\boldsymbol{\\lambda}=0$ leads to $f=0$ and $\\boldsymbol{A}=\\boldsymbol{y}$.\n",
"\n",
@@ -906,12 +906,12 @@
"\n",
"## Statistical analysis and optimization of data\n",
"\n",
"The following topics be covered\n",
"The following topics have been discussed:\n",
"1. Basic concepts, expectation values, variance, covariance, correlation functions and errors;\n",
"\n",
"2. Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;\n",
"\n",
"3. Central elements from linear algebra\n",
"3. Central elements from linear algebra, matrix inversion and SVD\n",
"\n",
"4. Gradient methods for data optimization\n",
"\n",
@@ -919,7 +919,7 @@
"\n",
"6. Practical optimization using Singular-value decomposition and least squares for parameterizing data.\n",
"\n",
"7. Principal Component Analysis.\n",
"7. Principal Component Analysis to reduce the number of features.\n",
"\n",
"## Machine learning\n",
"\n",
@@ -980,7 +980,7 @@
"\n",
"* Learn about neural network;\n",
"\n",
"* Learn about baggin, boosting and trees\n",
"* Learn about bagging, boosting and trees\n",
"\n",
"* Support vector machines\n",
"\n",
@@ -1021,17 +1021,19 @@
"\n",
"5. [Journal of Machine Learning Research](http://www.jmlr.org/papers/v19/) \n",
"\n",
"6. [Follow ML on ArXiv](https://arxiv.org/list/cs.LG/recent)\n",
"\n",
"## Starting your Machine Learning Project\n",
"\n",
"1. Identify problem type: classification, generation, regression\n",
"1. Identify problem type: classification, regression\n",
"\n",
"2. Consider your data carefully\n",
"\n",
"3. Choose a simple model that fits 1. and 2.\n",
"\n",
"4. Consider your data carefully again… data representation\n",
"4. Consider your data carefully again! Think of data representation more carefully.\n",
"\n",
"5. Based on results, feedback loop to earliest possible point\n",
"5. Based on your results, feedback loop to earliest possible point\n",
"\n",
"## Choose a Model and Algorithm\n",
"\n",
@@ -1092,17 +1094,21 @@
"\n",
" * Adam\n",
"\n",
" * and more\n",
"\n",
"Which regularization and hyperparameters? $L_1$ or $L_2$, soft classifiers, depths of trees and many other. Need to explore a large set of hyperparameters and regularization methods. \n",
"\n",
"Which regularization and hyperparameters? $L_1$ or $L_2$, soft\n",
"classifiers, depths of trees and many other. Need to explore a large\n",
"set of hyperparameters and regularization methods.\n",
"\n",
"\n",
"## Resampling\n",
"\n",
"When do we resample?\n",
"\n",
"1. Bootstrap\n",
"1. [Bootstrap](https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A)\n",
"\n",
"2. Cross-validation\n",
"2. [Cross-validation](https://www.youtube.com/watch?v=fSytzGwwBVw&ab_channel=StatQuestwithJoshStarmer)\n",
"\n",
"3. Jackknife and many other\n",
"\n",
@@ -1286,7 +1292,7 @@
"\n",
"## Joint distribution\n",
"\n",
"The restricted Boltzmann machine is described by a Bolztmann distribution"
"The restricted Boltzmann machine is described by a Boltzmann distribution"
]
},
{
@@ -1436,6 +1442,9 @@
"\n",
"4. Rectified linear units\n",
"\n",
"To read more, see [Lectures on Boltzmann machines in Physics](https://github.com/CompPhysics/ComputationalPhysics2/blob/gh-pages/doc/pub/notebook2/ipynb/notebook2.ipynb).\n",
"\n",
"\n",
"## Autoencoders: Overarching view\n",
"\n",
"Autoencoders are artificial neural networks capable of learning\n",
@@ -1508,6 +1517,7 @@
"[Lecture on Reinforcement Learning](https://www.youtube.com/watch?v=FgzM3zpZ55o&ab_channel=stanfordonline).\n",
"\n",
"See also A. Geron's textbook, chapter 16.\n",
"\n",
"## Transfer learning\n",
"\n",
"The goal of transfer learning is to transfer the model or knowledge\n",
@@ -1585,6 +1595,7 @@
"forward to the future, here are what we think the research hotspots in\n",
"the next ten years will be.\n",
"\n",
"See the article on [Discovery of Physics From Data: Universal Laws and Discrepancies](https://www.frontiersin.org/articles/10.3389/frai.2020.00025/full)\n",
"\n",
"## Explainable machine learning\n",
"\n",
@@ -1603,7 +1614,9 @@
"causality relations obtained via logical reasoning on real and clear\n",
"facts. It is one of the core goals of explainable machine learning to\n",
"transition from solving problems by data correlation to solving\n",
"problems by logical reasoning. Bayesian Machine Learning is one of the exciting research directions in this field.\n",
"problems by logical reasoning.\n",
"\n",
"**Bayesian Machine Learning is one of the exciting research directions in this field**.\n",
"\n",
"## Quantum machine learning\n",
"\n",
@@ -1626,6 +1639,7 @@
"\n",
"[Lecture on Quantum ML](https://www.youtube.com/watch?v=Xh9pUu3-WxM&ab_channel=InstituteforPure%26AppliedMathematics%28IPAM%29).\n",
"\n",
"\n",
"[Read interview with Maria Schuld on her work on Quantum Machine Learning](https://physics.aps.org/articles/v13/179?utm_campaign=weekly&utm_medium=email&utm_source=emailalert). See also [her recent textbook](https://www.springer.com/gp/book/9783319964232). \n",
"\n",
"\n",
doc/src/week48/week48.do.txt
+26 -17
View File
@@ -5,8 +5,8 @@ DATE: today
!split
===== Overview of week 48 =====
* _Thursday_: Support Vector Machines: Kernels, Classification and Regression
* _Friday_: Summary of course with perspectives for future studies
* _Thursday_: Support Vector Machines: Kernels, Classification and Regression. "Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember26.mp4?vrtx=view-as-webpage".
* _Friday_: Summary of course with perspectives for future studies. "Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureNovember27.mp4?vrtx=view-as-webpage".
Geron's chapter 5. Chapter 12 (sections 12.1-12.3 are the most relevant ones) of Hastie et al contains also a good discussion.
@@ -466,7 +466,7 @@ If we use Python as programming language and wish to venture beyond
_scikit-learn_, _tensorflow_ and similar software which makes our
lives so much easier, we need to dive into the wonderful world of
quadratic programming. We can, if we wish, solve the minimization
problem using say standard gradient methods or conjugate gradient
problem using standard gradient methods or conjugate gradient
methods. However, these methods tend to exhibit a rather slow
converge. So, welcome to the promised land of quadratic programming.
@@ -582,7 +582,7 @@ We have the general problem
o With a given kernel we can thus define the matrix $\bm{P}$.
o The matrix $\bm{P}$ has matrix elements $p_{ij}=y_iy_jK(\bm{x}_i,\bm{x}_j)$. Given a kernel $K$ and the targets $y_i$ this matrix is easy to set up.
o The vector $\bm{q}$ has all elements equal -1.
o The vector $\bm{q}$ has all elements equal to $-1$.
o The constraint $\bm{y}^T\bm{\lambda}=0$ leads to $f=0$ and $\bm{A}=\bm{y}$.
o To set up the matrix $\bm{G}$ we note that the inequalities $0\leq \lambda_i \leq C$ can be split up into $0\leq \lambda_i$ and $\lambda_i \leq C$. These two inequalities define then the matrix $\bm{G}$ and the vector $\bm{h}$.
@@ -652,14 +652,14 @@ o Machine learning
!split
===== Statistical analysis and optimization of data =====
The following topics be covered
The following topics have been discussed:
o Basic concepts, expectation values, variance, covariance, correlation functions and errors;
o Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;
o Central elements from linear algebra
o Central elements from linear algebra, matrix inversion and SVD
o Gradient methods for data optimization
o Estimation of errors using cross-validation, bootstrapping and jackknife methods;
o Practical optimization using Singular-value decomposition and least squares for parameterizing data.
o Principal Component Analysis.
o Principal Component Analysis to reduce the number of features.
!split
===== Machine learning =====
@@ -700,7 +700,7 @@ ethical conduct is emphasized throughout the course.
* Understand linear methods for regression and classification;
* Learn about neural network;
* Learn about baggin, boosting and trees
* Learn about bagging, boosting and trees
* Support vector machines
* Learn about basic data analysis;
* Be capable of extending the acquired knowledge to other systems and cases;
@@ -732,17 +732,17 @@ o _NIPS_: "Neural Information Processing Systems":"https://papers.nips.cc"
o _ICLR_: "International Conference on Learning Representations":"https://openreview.net/group?id=ICLR.cc/2018/Conference#accepted-oral-papers"
o _ICML_: International Conference on Machine Learning
o "Journal of Machine Learning Research":"http://www.jmlr.org/papers/v19/"
o "Follow ML on ArXiv":"https://arxiv.org/list/cs.LG/recent"
!split
===== Starting your Machine Learning Project =====
o Identify problem type: classification, generation, regression
o Identify problem type: classification, regression
o Consider your data carefully
o Choose a simple model that fits 1. and 2.
o Consider your data carefully again… data representation
o Based on results, feedback loop to earliest possible point
o Consider your data carefully again! Think of data representation more carefully.
o Based on your results, feedback loop to earliest possible point
@@ -786,8 +786,11 @@ o Stochastic gradient descent + momentum
o State-of-the-art approaches:
* RMSProp
* Adam
* and more
Which regularization and hyperparameters? $L_1$ or $L_2$, soft classifiers, depths of trees and many other. Need to explore a large set of hyperparameters and regularization methods.
Which regularization and hyperparameters? $L_1$ or $L_2$, soft
classifiers, depths of trees and many other. Need to explore a large
set of hyperparameters and regularization methods.
!split
@@ -795,8 +798,8 @@ Which regularization and hyperparameters? $L_1$ or $L_2$, soft classifiers, dept
When do we resample?
o Bootstrap
o Cross-validation
o "Bootstrap":"https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A"
o "Cross-validation":"https://www.youtube.com/watch?v=fSytzGwwBVw&ab_channel=StatQuestwithJoshStarmer"
o Jackknife and many other
@@ -972,7 +975,7 @@ _The network parameters, to be optimized/learned_:
!split
===== Joint distribution =====
The restricted Boltzmann machine is described by a Bolztmann distribution
The restricted Boltzmann machine is described by a Boltzmann distribution
!bt
\begin{align}
P_{rbm}(\mathbf{x},\mathbf{h}) = \frac{1}{Z} e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})},
@@ -1047,6 +1050,7 @@ Other types of units include:
o Binomial units
o Rectified linear units
To read more, see "Lectures on Boltzmann machines in Physics":"https://github.com/CompPhysics/ComputationalPhysics2/blob/gh-pages/doc/pub/notebook2/ipynb/notebook2.ipynb".
!split
@@ -1124,6 +1128,7 @@ learning.
"Lecture on Reinforcement Learning":"https://www.youtube.com/watch?v=FgzM3zpZ55o&ab_channel=stanfordonline".
See also A. Geron's textbook, chapter 16.
!split
===== Transfer learning =====
@@ -1207,6 +1212,7 @@ still very optimistic about the future of machine learning. As we look
forward to the future, here are what we think the research hotspots in
the next ten years will be.
See the article on "Discovery of Physics From Data: Universal Laws and Discrepancies":"https://www.frontiersin.org/articles/10.3389/frai.2020.00025/full"
!split
===== Explainable machine learning =====
@@ -1226,7 +1232,9 @@ contrast, rational humans tend to reply on clear and trustworthy
causality relations obtained via logical reasoning on real and clear
facts. It is one of the core goals of explainable machine learning to
transition from solving problems by data correlation to solving
problems by logical reasoning. Bayesian Machine Learning is one of the exciting research directions in this field.
problems by logical reasoning.
_Bayesian Machine Learning is one of the exciting research directions in this field_.
!split
===== Quantum machine learning =====
@@ -1250,6 +1258,7 @@ computing systems.
"Lecture on Quantum ML":"https://www.youtube.com/watch?v=Xh9pUu3-WxM&ab_channel=InstituteforPure%26AppliedMathematics%28IPAM%29".
"Read interview with Maria Schuld on her work on Quantum Machine Learning":"https://physics.aps.org/articles/v13/179?utm_campaign=weekly&utm_medium=email&utm_source=emailalert". See also "her recent textbook":"https://www.springer.com/gp/book/9783319964232".