lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Created chapter on Log regression

Browse Source
This commit is contained in:
mhjensen
2018-09-17 10:05:14 +02:00
parent 18c26eed14
commit bf5278f406
16 changed files with 1903 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/LogReg/html/._LogReg-bs000.html
+170
View File
@@ -0,0 +1,170 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Logistic Regression">
<title>Data Analysis and Machine Learning: Logistic Regression</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Logistic Regression', 2, None, '___sec0'),
('Basics', 2, None, '___sec1'),
('Linear classifier', 2, None, '___sec2'),
('Some selected properties', 2, None, '___sec3'),
('The cross-entropy as a cost function for logistic regression',
2,
None,
'___sec4'),
('Maximum likelihood', 2, None, '___sec5'),
('Minimizing the cross entropy', 2, None, '___sec6')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="LogReg-bs.html">Data Analysis and Machine Learning: Logistic Regression</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._LogReg-bs001.html#___sec0" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs002.html#___sec1" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs003.html#___sec2" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs004.html#___sec3" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs005.html#___sec4" style="font-size: 80%;">The cross-entropy as a cost function for logistic regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs006.html#___sec5" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs008.html#___sec6" style="font-size: 80%;">Minimizing the cross entropy</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0000"></a>
<!-- ------------------- main content ---------------------- -->
<div class="jumbotron">
<center><h1>Data Analysis and Machine Learning: Logistic Regression</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
<center>
<b>Morten Hjorth-Jensen</b> [1, 2]
</center>
<p>
<!-- institution(s) -->
<center>[1] <b>Department of Physics, University of Oslo</b></center>
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 17, 2018</h4></center> <!-- date -->
<br>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li class="active"><a href="._LogReg-bs000.html">1</a></li>
<li><a href="._LogReg-bs001.html">2</a></li>
<li><a href="._LogReg-bs002.html">3</a></li>
<li><a href="._LogReg-bs003.html">4</a></li>
<li><a href="._LogReg-bs004.html">5</a></li>
<li><a href="._LogReg-bs005.html">6</a></li>
<li><a href="._LogReg-bs006.html">7</a></li>
<li><a href="._LogReg-bs007.html">8</a></li>
<li><a href="._LogReg-bs008.html">9</a></li>
<li><a href="._LogReg-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</body>
</html>
doc/pub/LogReg/html/._LogReg-bs001.html
+176
View File
@@ -0,0 +1,176 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Logistic Regression">
<title>Data Analysis and Machine Learning: Logistic Regression</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Logistic Regression', 2, None, '___sec0'),
('Basics', 2, None, '___sec1'),
('Linear classifier', 2, None, '___sec2'),
('Some selected properties', 2, None, '___sec3'),
('The cross-entropy as a cost function for logistic regression',
2,
None,
'___sec4'),
('Maximum likelihood', 2, None, '___sec5'),
('Minimizing the cross entropy', 2, None, '___sec6')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="LogReg-bs.html">Data Analysis and Machine Learning: Logistic Regression</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs002.html#___sec1" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs003.html#___sec2" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs004.html#___sec3" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs005.html#___sec4" style="font-size: 80%;">The cross-entropy as a cost function for logistic regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs006.html#___sec5" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs008.html#___sec6" style="font-size: 80%;">Minimizing the cross entropy</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0001"></a>
<!-- !split -->
</div> <!-- end jumbotron -->
<h2 id="___sec0" class="anchor">Logistic Regression </h2>
<p>
So far we have focused on learning from datasets for which there is a
<b>continuous</b> output. In linear regression we have been
concerned with learning the coefficients of a polynomial to predict
the response of a continuous variable \( y_i \) on unseen data based on
its independent variables \( {\bf x}_i \).
<p>
Classification problems,
however, are concerned with outcomes taking the form of discrete
variables (i.e. categories). For example, we may want to detect if
there's a cat or a dog in an image. Or given a specific system,
we'd like to identify its state, say whether it is an ordered or disordered system (typical situation in solid state physics).
(e.g. ordered/disordered).
<p>
<b>Logistic regression deals with binary, dichotomous outcomes (e.g. True or
False, Success or Failure, etc.). It is worth noting that logistic
regression is also commonly used in modern supervised Deep Learning
models</b>, as we will see later.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._LogReg-bs000.html">&laquo;</a></li>
<li><a href="._LogReg-bs000.html">1</a></li>
<li class="active"><a href="._LogReg-bs001.html">2</a></li>
<li><a href="._LogReg-bs002.html">3</a></li>
<li><a href="._LogReg-bs003.html">4</a></li>
<li><a href="._LogReg-bs004.html">5</a></li>
<li><a href="._LogReg-bs005.html">6</a></li>
<li><a href="._LogReg-bs006.html">7</a></li>
<li><a href="._LogReg-bs007.html">8</a></li>
<li><a href="._LogReg-bs008.html">9</a></li>
<li><a href="._LogReg-bs002.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/LogReg/html/._LogReg-bs002.html
+164
View File
@@ -0,0 +1,164 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Logistic Regression">
<title>Data Analysis and Machine Learning: Logistic Regression</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Logistic Regression', 2, None, '___sec0'),
('Basics', 2, None, '___sec1'),
('Linear classifier', 2, None, '___sec2'),
('Some selected properties', 2, None, '___sec3'),
('The cross-entropy as a cost function for logistic regression',
2,
None,
'___sec4'),
('Maximum likelihood', 2, None, '___sec5'),
('Minimizing the cross entropy', 2, None, '___sec6')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="LogReg-bs.html">Data Analysis and Machine Learning: Logistic Regression</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._LogReg-bs001.html#___sec0" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs003.html#___sec2" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs004.html#___sec3" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs005.html#___sec4" style="font-size: 80%;">The cross-entropy as a cost function for logistic regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs006.html#___sec5" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs008.html#___sec6" style="font-size: 80%;">Minimizing the cross entropy</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0002"></a>
<!-- !split -->
<h2 id="___sec1" class="anchor">Basics </h2>
<p>
We consider the case where the dependent variables \( y_i\in\mathbb{Z} \)
are discrete and only take values from \( m=0,\dots,M-1 \) (i.e. \( M \)
classes).
<p>
The goal is to predict the
output classes from the design matrix \( X\in\mathbb{R}^{n\times p} \)
made of \( n \) samples, each of which bears \( p \) features. The
primary goal is to identify the classes to which new unseen samples
belong.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._LogReg-bs001.html">&laquo;</a></li>
<li><a href="._LogReg-bs000.html">1</a></li>
<li><a href="._LogReg-bs001.html">2</a></li>
<li class="active"><a href="._LogReg-bs002.html">3</a></li>
<li><a href="._LogReg-bs003.html">4</a></li>
<li><a href="._LogReg-bs004.html">5</a></li>
<li><a href="._LogReg-bs005.html">6</a></li>
<li><a href="._LogReg-bs006.html">7</a></li>
<li><a href="._LogReg-bs007.html">8</a></li>
<li><a href="._LogReg-bs008.html">9</a></li>
<li><a href="._LogReg-bs003.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/LogReg/html/._LogReg-bs003.html
+164
View File
@@ -0,0 +1,164 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Logistic Regression">
<title>Data Analysis and Machine Learning: Logistic Regression</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Logistic Regression', 2, None, '___sec0'),
('Basics', 2, None, '___sec1'),
('Linear classifier', 2, None, '___sec2'),
('Some selected properties', 2, None, '___sec3'),
('The cross-entropy as a cost function for logistic regression',
2,
None,
'___sec4'),
('Maximum likelihood', 2, None, '___sec5'),
('Minimizing the cross entropy', 2, None, '___sec6')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="LogReg-bs.html">Data Analysis and Machine Learning: Logistic Regression</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._LogReg-bs001.html#___sec0" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs002.html#___sec1" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs004.html#___sec3" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs005.html#___sec4" style="font-size: 80%;">The cross-entropy as a cost function for logistic regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs006.html#___sec5" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs008.html#___sec6" style="font-size: 80%;">Minimizing the cross entropy</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0003"></a>
<!-- !split -->
<h2 id="___sec2" class="anchor">Linear classifier </h2>
<p>
Let us start by considering a slightly simpler classifier: a linear classifier that categorizes examples using a weighted linear-combination of the features and an additive offset
$$
\begin{equation}
s_i = \boldsymbol{x}_i^T\boldsymbol{w} + b_0 \equiv \mathbf{x}_i^T\mathbf{w},
\tag{1}
\end{equation}
$$
where we use the short-hand notation
\( \mathbf{x}_i = (1,\boldsymbol{x}_i) \) and \( \mathbf{w}_i = (b_0,\boldsymbol{w}_i) \).
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._LogReg-bs002.html">&laquo;</a></li>
<li><a href="._LogReg-bs000.html">1</a></li>
<li><a href="._LogReg-bs001.html">2</a></li>
<li><a href="._LogReg-bs002.html">3</a></li>
<li class="active"><a href="._LogReg-bs003.html">4</a></li>
<li><a href="._LogReg-bs004.html">5</a></li>
<li><a href="._LogReg-bs005.html">6</a></li>
<li><a href="._LogReg-bs006.html">7</a></li>
<li><a href="._LogReg-bs007.html">8</a></li>
<li><a href="._LogReg-bs008.html">9</a></li>
<li><a href="._LogReg-bs004.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/LogReg/html/._LogReg-bs004.html
+174
View File
@@ -0,0 +1,174 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Logistic Regression">
<title>Data Analysis and Machine Learning: Logistic Regression</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Logistic Regression', 2, None, '___sec0'),
('Basics', 2, None, '___sec1'),
('Linear classifier', 2, None, '___sec2'),
('Some selected properties', 2, None, '___sec3'),
('The cross-entropy as a cost function for logistic regression',
2,
None,
'___sec4'),
('Maximum likelihood', 2, None, '___sec5'),
('Minimizing the cross entropy', 2, None, '___sec6')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="LogReg-bs.html">Data Analysis and Machine Learning: Logistic Regression</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._LogReg-bs001.html#___sec0" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs002.html#___sec1" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs003.html#___sec2" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs005.html#___sec4" style="font-size: 80%;">The cross-entropy as a cost function for logistic regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs006.html#___sec5" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs008.html#___sec6" style="font-size: 80%;">Minimizing the cross entropy</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0004"></a>
<!-- !split -->
<h2 id="___sec3" class="anchor">Some selected properties </h2>
<p>
This function takes values on the entire real axis. In the case of
logistic regression, however, the labels \( y_i \) are discrete
variables. One simple way to get a discrete output is to have sign
functions that map the output of a linear regressor to \( \{0,1\} \),
\( f(s_i)=sign(s_i)=1 \) if \( s_i\ge 0 \) and 0 if otherwise. Indeed,
this is commonly known as the &quot;perceptron" in the machine learning
literature. This model is extremely simple, and it is favorable in
many cases (e.g. noisy data) to have a ``soft" classifier that outputs
the probability of a given category. For example, given
\( \mathbf{x}_i \), the classifier outputs the probability of being in
category \( m \). One such function is the logistic (or sigmoid) function:
$$
\begin{equation}
f(s) = \frac{1}{1+\mathrm e^{-s}}.
\tag{2}
\end{equation}
$$
Note that \( 1-f(s)= f(-s) \), which will be useful shortly.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._LogReg-bs003.html">&laquo;</a></li>
<li><a href="._LogReg-bs000.html">1</a></li>
<li><a href="._LogReg-bs001.html">2</a></li>
<li><a href="._LogReg-bs002.html">3</a></li>
<li><a href="._LogReg-bs003.html">4</a></li>
<li class="active"><a href="._LogReg-bs004.html">5</a></li>
<li><a href="._LogReg-bs005.html">6</a></li>
<li><a href="._LogReg-bs006.html">7</a></li>
<li><a href="._LogReg-bs007.html">8</a></li>
<li><a href="._LogReg-bs008.html">9</a></li>
<li><a href="._LogReg-bs005.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/LogReg/html/._LogReg-bs005.html
+170
View File
@@ -0,0 +1,170 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Logistic Regression">
<title>Data Analysis and Machine Learning: Logistic Regression</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Logistic Regression', 2, None, '___sec0'),
('Basics', 2, None, '___sec1'),
('Linear classifier', 2, None, '___sec2'),
('Some selected properties', 2, None, '___sec3'),
('The cross-entropy as a cost function for logistic regression',
2,
None,
'___sec4'),
('Maximum likelihood', 2, None, '___sec5'),
('Minimizing the cross entropy', 2, None, '___sec6')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="LogReg-bs.html">Data Analysis and Machine Learning: Logistic Regression</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._LogReg-bs001.html#___sec0" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs002.html#___sec1" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs003.html#___sec2" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs004.html#___sec3" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">The cross-entropy as a cost function for logistic regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs006.html#___sec5" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs008.html#___sec6" style="font-size: 80%;">Minimizing the cross entropy</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0005"></a>
<!-- !split -->
<h2 id="___sec4" class="anchor">The cross-entropy as a cost function for logistic regression </h2>
<p>
The perceptron is an example of a ``hard classification&quot;: each datapoint is deterministically assigned to a category (i.e \( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a &quot;soft&quot; classifier that outputs the probability of a given category rather than a single value. For example, given \( \mathbf{x}_i \), the classifier outputs the probability of being in category \( m \).
Logistic regression is the most canonical example of a soft classifier. In logistic regression, the probability that a data point \( \boldsymbol{x}_i \) belongs to a category \( y_i=\{0,1\} \) is is given by
$$
\begin{eqnarray}
P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& \frac{1}{1+\mathrm{e}^{-\mathbf{x}^T_i\mathbf{w}}},\nonumber\\
P(y_i=0|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& 1 - P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)},
\end{eqnarray}
$$
where \( \boldsymbol{\theta}=\mathbf{w} \) are the weights we wish to learn from the data.
<p>
Notice that in terms of the logistic function, we can write
$$
P(y_i=1) =f(\mathbf{x}_i^T\mathbf{w})=1-P(y_i=0).
$$
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._LogReg-bs004.html">&laquo;</a></li>
<li><a href="._LogReg-bs000.html">1</a></li>
<li><a href="._LogReg-bs001.html">2</a></li>
<li><a href="._LogReg-bs002.html">3</a></li>
<li><a href="._LogReg-bs003.html">4</a></li>
<li><a href="._LogReg-bs004.html">5</a></li>
<li class="active"><a href="._LogReg-bs005.html">6</a></li>
<li><a href="._LogReg-bs006.html">7</a></li>
<li><a href="._LogReg-bs007.html">8</a></li>
<li><a href="._LogReg-bs008.html">9</a></li>
<li><a href="._LogReg-bs006.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/LogReg/html/._LogReg-bs006.html
+174
View File
@@ -0,0 +1,174 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Logistic Regression">
<title>Data Analysis and Machine Learning: Logistic Regression</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Logistic Regression', 2, None, '___sec0'),
('Basics', 2, None, '___sec1'),
('Linear classifier', 2, None, '___sec2'),
('Some selected properties', 2, None, '___sec3'),
('The cross-entropy as a cost function for logistic regression',
2,
None,
'___sec4'),
('Maximum likelihood', 2, None, '___sec5'),
('Minimizing the cross entropy', 2, None, '___sec6')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="LogReg-bs.html">Data Analysis and Machine Learning: Logistic Regression</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._LogReg-bs001.html#___sec0" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs002.html#___sec1" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs003.html#___sec2" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs004.html#___sec3" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs005.html#___sec4" style="font-size: 80%;">The cross-entropy as a cost function for logistic regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs008.html#___sec6" style="font-size: 80%;">Minimizing the cross entropy</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0006"></a>
<!-- !split -->
<h2 id="___sec5" class="anchor">Maximum likelihood </h2>
<p>
We now define the cost function for logistic regression using Maximum
Likelihood Estimation (MLE). Recall, that in MLE we choose parameters
to maximize the probability of seeing the observed data. Consider a
dataset \( \mathcal{D}=\{(y_i,\boldsymbol{x}_i)\} \) with binary labels
\( y_i\in\{0,1\} \) where the data points are drawn independently. The
likelihood of the seeing the data under our model is just:
$$
\begin{align}
P(\mathcal{D}|\mathbf{w})& = \prod_{i=1}^n \left[f(\mathbf{x}_i^T\mathbf{w})\right]^{y_i}\left[1-f(\mathbf{x}_i^T\mathbf{w})\right]^{1-y_i}\nonumber \\
\tag{3}
\end{align}
$$
from which we can readily compute the log-likelihood:
$$
\begin{equation}
l(\mathbf{w}) = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
\tag{4}
\end{equation}
$$
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._LogReg-bs005.html">&laquo;</a></li>
<li><a href="._LogReg-bs000.html">1</a></li>
<li><a href="._LogReg-bs001.html">2</a></li>
<li><a href="._LogReg-bs002.html">3</a></li>
<li><a href="._LogReg-bs003.html">4</a></li>
<li><a href="._LogReg-bs004.html">5</a></li>
<li><a href="._LogReg-bs005.html">6</a></li>
<li class="active"><a href="._LogReg-bs006.html">7</a></li>
<li><a href="._LogReg-bs007.html">8</a></li>
<li><a href="._LogReg-bs008.html">9</a></li>
<li><a href="._LogReg-bs007.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/LogReg/html/._LogReg-bs007.html
+165
View File
@@ -0,0 +1,165 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Logistic Regression">
<title>Data Analysis and Machine Learning: Logistic Regression</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Logistic Regression', 2, None, '___sec0'),
('Basics', 2, None, '___sec1'),
('Linear classifier', 2, None, '___sec2'),
('Some selected properties', 2, None, '___sec3'),
('The cross-entropy as a cost function for logistic regression',
2,
None,
'___sec4'),
('Maximum likelihood', 2, None, '___sec5'),
('Minimizing the cross entropy', 2, None, '___sec6')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="LogReg-bs.html">Data Analysis and Machine Learning: Logistic Regression</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._LogReg-bs001.html#___sec0" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs002.html#___sec1" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs003.html#___sec2" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs004.html#___sec3" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs005.html#___sec4" style="font-size: 80%;">The cross-entropy as a cost function for logistic regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs006.html#___sec5" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs008.html#___sec6" style="font-size: 80%;">Minimizing the cross entropy</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0007"></a>
<!-- !split -->
The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \theta \)
$$
\hat{\mathbf{w}} = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
$$
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
$$
\begin{eqnarray}
\mathcal{C}(\mathbf{w}) &=& - l(\mathbf{w}) \\
&=& \sum_{i=1}^n -y_i\log f(\mathbf{x}_i^T\mathbf{w}) - (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].\nonumber
\end{eqnarray}
$$
This equation is known in statistics as the \emph{cross entropy}. Finally, we note that just as in linear regression,
in practice we usually supplement the cross-entropy with additional regularization terms, usually \( L_1 \) and \( L_2 \) regularization as we did for Ridge and Lasso regression.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._LogReg-bs006.html">&laquo;</a></li>
<li><a href="._LogReg-bs000.html">1</a></li>
<li><a href="._LogReg-bs001.html">2</a></li>
<li><a href="._LogReg-bs002.html">3</a></li>
<li><a href="._LogReg-bs003.html">4</a></li>
<li><a href="._LogReg-bs004.html">5</a></li>
<li><a href="._LogReg-bs005.html">6</a></li>
<li><a href="._LogReg-bs006.html">7</a></li>
<li class="active"><a href="._LogReg-bs007.html">8</a></li>
<li><a href="._LogReg-bs008.html">9</a></li>
<li><a href="._LogReg-bs008.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/LogReg/html/._LogReg-bs008.html
+169
View File
@@ -0,0 +1,169 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Logistic Regression">
<title>Data Analysis and Machine Learning: Logistic Regression</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Logistic Regression', 2, None, '___sec0'),
('Basics', 2, None, '___sec1'),
('Linear classifier', 2, None, '___sec2'),
('Some selected properties', 2, None, '___sec3'),
('The cross-entropy as a cost function for logistic regression',
2,
None,
'___sec4'),
('Maximum likelihood', 2, None, '___sec5'),
('Minimizing the cross entropy', 2, None, '___sec6')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="LogReg-bs.html">Data Analysis and Machine Learning: Logistic Regression</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._LogReg-bs001.html#___sec0" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs002.html#___sec1" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs003.html#___sec2" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs004.html#___sec3" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs005.html#___sec4" style="font-size: 80%;">The cross-entropy as a cost function for logistic regression</a></li>
<!-- navigation toc: --> <li><a href="._LogReg-bs006.html#___sec5" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">Minimizing the cross entropy</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0008"></a>
<!-- !split -->
<h2 id="___sec6" class="anchor">Minimizing the cross entropy </h2>
<p>
The cross entropy is a convex function of the weights \( \mathbf{w} \) and,
therefore, any local minimizer is a global minimizer. Minimizing this
cost function leads to the following equation
$$
\begin{equation}
\boldsymbol{0}=\boldsymbol{\nabla} \mathcal{C}(\mathbf{w}) = \sum_{i=1}^n\left[f(\mathbf{x}_i^T\mathbf{w})-y_i\right]\mathbf{x}_i,
\tag{5}
\end{equation}
$$
<p>
where we made use of the logistic function identity \( \partial_z f(z) =
f(z)[1-f(z)] \). This equation defines a transcendental equation for
\( \mathbf{w} \), the solution of which, unlike linear regression, cannot
be written in a closed form.
Here we need gradient descent methods!
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._LogReg-bs007.html">&laquo;</a></li>
<li><a href="._LogReg-bs000.html">1</a></li>
<li><a href="._LogReg-bs001.html">2</a></li>
<li><a href="._LogReg-bs002.html">3</a></li>
<li><a href="._LogReg-bs003.html">4</a></li>
<li><a href="._LogReg-bs004.html">5</a></li>
<li><a href="._LogReg-bs005.html">6</a></li>
<li><a href="._LogReg-bs006.html">7</a></li>
<li><a href="._LogReg-bs007.html">8</a></li>
<li class="active"><a href="._LogReg-bs008.html">9</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/src/LogisticRegression/LogReg.do.txt
+153
View File
@@ -0,0 +1,153 @@
TITLE: Data Analysis and Machine Learning: Logistic Regression
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
!split
===== Logistic Regression =====
So far we have focused on learning from datasets for which there is a
_continuous_ output. In linear regression we have been
concerned with learning the coefficients of a polynomial to predict
the response of a continuous variable $y_i$ on unseen data based on
its independent variables ${\bf x}_i$.
Classification problems,
however, are concerned with outcomes taking the form of discrete
variables (i.e. categories). For example, we may want to detect if
there's a cat or a dog in an image. Or given a specific system,
we'd like to identify its state, say whether it is an ordered or disordered system (typical situation in solid state physics).
(e.g. ordered/disordered).
_Logistic regression deals with binary, dichotomous outcomes (e.g. True or
False, Success or Failure, etc.). It is worth noting that logistic
regression is also commonly used in modern supervised Deep Learning
models_, as we will see later.
!split
===== Basics =====
We consider the case where the dependent variables $y_i\in\mathbb{Z}$
are discrete and only take values from $m=0,\dots,M-1$ (i.e. $M$
classes).
The goal is to predict the
output classes from the design matrix $X\in\mathbb{R}^{n\times p}$
made of $n$ samples, each of which bears $p$ features. The
primary goal is to identify the classes to which new unseen samples
belong.
!split
===== Linear classifier =====
Let us start by considering a slightly simpler classifier: a linear classifier that categorizes examples using a weighted linear-combination of the features and an additive offset
!bt
\begin{equation}
s_i = \boldsymbol{x}_i^T\boldsymbol{w} + b_0 \equiv \mathbf{x}_i^T\mathbf{w},
\end{equation}
!et
where we use the short-hand notation
$\mathbf{x}_i = (1,\boldsymbol{x}_i)$ and $\mathbf{w}_i = (b_0,\boldsymbol{w}_i)$.
!split
===== Some selected properties =====
This function takes values on the entire real axis. In the case of
logistic regression, however, the labels $y_i$ are discrete
variables. One simple way to get a discrete output is to have sign
functions that map the output of a linear regressor to $\{0,1\}$,
$f(s_i)=sign(s_i)=1$ if $s_i\ge 0$ and 0 if otherwise. Indeed,
this is commonly known as the ``perceptron" in the machine learning
literature. This model is extremely simple, and it is favorable in
many cases (e.g. noisy data) to have a ``soft" classifier that outputs
the probability of a given category. For example, given
$\mathbf{x}_i$, the classifier outputs the probability of being in
category $m$. One such function is the logistic (or sigmoid) function:
!bt
\begin{equation}
f(s) = \frac{1}{1+\mathrm e^{-s}}.
\label{eq:log_fun}
\end{equation}
!et
Note that $1-f(s)= f(-s)$, which will be useful shortly.
!split
===== The cross-entropy as a cost function for logistic regression =====
The perceptron is an example of a ``hard classification'': each datapoint is deterministically assigned to a category (i.e $y_i=0$ or $y_i=1$). In many cases, it is favorable to have a ``soft'' classifier that outputs the probability of a given category rather than a single value. For example, given $\mathbf{x}_i$, the classifier outputs the probability of being in category $m$.
Logistic regression is the most canonical example of a soft classifier. In logistic regression, the probability that a data point $\boldsymbol{x}_i$ belongs to a category $y_i=\{0,1\}$ is is given by
!bt
\begin{eqnarray}
P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& \frac{1}{1+\mathrm{e}^{-\mathbf{x}^T_i\mathbf{w}}},\nonumber\\
P(y_i=0|\boldsymbol{x}_i,\boldsymbol{\theta)} &=& 1 - P(y_i=1|\boldsymbol{x}_i,\boldsymbol{\theta)},
\end{eqnarray}
!et
where $\boldsymbol{\theta}=\mathbf{w}$ are the weights we wish to learn from the data.
Notice that in terms of the logistic function, we can write
!bt
\[
P(y_i=1) =f(\mathbf{x}_i^T\mathbf{w})=1-P(y_i=0).
\]
!et
!split
===== Maximum likelihood =====
We now define the cost function for logistic regression using Maximum
Likelihood Estimation (MLE). Recall, that in MLE we choose parameters
to maximize the probability of seeing the observed data. Consider a
dataset $\mathcal{D}=\{(y_i,\boldsymbol{x}_i)\}$ with binary labels
$y_i\in\{0,1\}$ where the data points are drawn independently. The
likelihood of the seeing the data under our model is just:
!bt
\begin{align}
P(\mathcal{D}|\mathbf{w})& = \prod_{i=1}^n \left[f(\mathbf{x}_i^T\mathbf{w})\right]^{y_i}\left[1-f(\mathbf{x}_i^T\mathbf{w})\right]^{1-y_i}\nonumber \\
\end{align}
!et
from which we can readily compute the log-likelihood:
!bt
\begin{equation}
l(\mathbf{w}) = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
\end{equation}
!et
!split
The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\theta$
!bt
\[
\hat{\mathbf{w}} = \sum_{i=1}^n y_i\log f(\mathbf{x}_i^T\mathbf{w}) + (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].
\]
!et
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
!bt
\begin{eqnarray}
\mathcal{C}(\mathbf{w}) &=& - l(\mathbf{w}) \\
&=& \sum_{i=1}^n -y_i\log f(\mathbf{x}_i^T\mathbf{w}) - (1-y_i)\log\left[1-f(\mathbf{x}_i^T\mathbf{w})\right].\nonumber
\end{eqnarray}
!et
This equation is known in statistics as the \emph{cross entropy}. Finally, we note that just as in linear regression,
in practice we usually supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.
!split
===== Minimizing the cross entropy =====
The cross entropy is a convex function of the weights $\mathbf{w}$ and,
therefore, any local minimizer is a global minimizer. Minimizing this
cost function leads to the following equation
!bt
\begin{equation}
\boldsymbol{0}=\boldsymbol{\nabla} \mathcal{C}(\mathbf{w}) = \sum_{i=1}^n\left[f(\mathbf{x}_i^T\mathbf{w})-y_i\right]\mathbf{x}_i,
\end{equation}
!et
where we made use of the logistic function identity $\partial_z f(z) =
f(z)[1-f(z)]$. This equation defines a transcendental equation for
$\mathbf{w}$, the solution of which, unlike linear regression, cannot
be written in a closed form.
Here we need gradient descent methods!
doc/src/LogisticRegression/beamerthemered_plain.sty
+12
View File
@@ -0,0 +1,12 @@
\mode<presentation>
\usecolortheme[rgb={0.8, 0.2, 0}]{structure}
\usefonttheme[onlysmall]{structurebold}
\setbeamertemplate{navigation symbols}{}
%\setbeamertemplate{footline}[frame number]
\usepackage{tikz}
\usetikzlibrary{arrows,shapes,backgrounds,decorations,mindmap}
\mode
<all>
doc/src/LogisticRegression/beamerthemered_shadow.sty
+15
View File
@@ -0,0 +1,15 @@
\mode<presentation>
\useoutertheme{smoothbars}
\useinnertheme[shadow=true]{rounded}
\usecolortheme{orchid}
\usecolortheme{whale}
\usecolortheme[rgb={0.7, 0.2, 0}]{structure} % (darker red)
\useoutertheme{shadow}
\usefonttheme[onlysmall]{structurebold}
\setbeamercolor{title}{use=structure,fg=white,bg=structure.fg}
\setbeamerfont{block title}{size={}}
\mode
<all>
doc/src/LogisticRegression/clean.sh
Executable
+3
View File
@@ -0,0 +1,3 @@
#!/bin/sh
doconce clean
rm -rf *.pdf *.tex ipynb*.tar.gz *.html ._*.html *~ reveal.js Trash README.txt
doc/src/LogisticRegression/make.sh
Executable
+118
View File
@@ -0,0 +1,118 @@
#!/bin/sh
set -x
function system {
"$@"
if [ $? -ne 0 ]; then
echo "make.sh: unsuccessful command $@"
echo "abort!"
exit 1
fi
}
if [ $# -eq 0 ]; then
echo 'bash make.sh slides1|slides2'
exit 1
fi
name=$1
rm -f *.tar.gz
opt="--encoding=utf-8"
# Note: Makefile examples contain constructions like ${PROG} which
# looks like Mako constructions, but they are not. Use --no_mako
# to turn off Mako processing.
opt="--no_mako"
rm -f *.aux
html=${name}-reveal
system doconce format html $name --pygments_html_style=perldoc --keep_pygments_html_bg --html_links_in_new_window --html_output=$html $opt
system doconce slides_html $html reveal --html_slide_theme=beige
# Plain HTML documents
html=${name}-solarized
system doconce format html $name --pygments_html_style=perldoc --html_style=solarized3 --html_links_in_new_window --html_output=$html $opt
system doconce split_html $html.html --method=space10
html=${name}
system doconce format html $name --pygments_html_style=default --html_style=bloodish --html_links_in_new_window --html_output=$html $opt
system doconce split_html $html.html --method=space10
# Bootstrap style
html=${name}-bs
system doconce format html $name --html_style=bootstrap --pygments_html_style=default --html_admon=bootstrap_panel --html_output=$html $opt
system doconce split_html $html.html --method=split --pagination --nav_button=bottom
# IPython notebook
system doconce format ipynb $name $opt
# LaTeX Beamer slides
beamertheme=red_plain
system doconce format pdflatex $name --latex_title_layout=beamer --latex_table_format=footnotesize $opt
system doconce ptex2tex $name envir=minted
# Add special packages
doconce subst "% Add user's preamble" "\g<1>\n\\usepackage{simplewick}" $name.tex
system doconce slides_beamer $name --beamer_slide_theme=$beamertheme
system pdflatex -shell-escape ${name}
system pdflatex -shell-escape ${name}
cp $name.pdf ${name}-beamer.pdf
cp $name.tex ${name}-beamer.tex
# Handouts
system doconce format pdflatex $name --latex_title_layout=beamer --latex_table_format=footnotesize $opt
system doconce ptex2tex $name envir=minted
# Add special packages
doconce subst "% Add user's preamble" "\g<1>\n\\usepackage{simplewick}" $name.tex
system doconce slides_beamer $name --beamer_slide_theme=red_shadow --handout
system pdflatex -shell-escape $name
pdflatex -shell-escape $name
pdflatex -shell-escape $name
pdfnup --nup 2x3 --frame true --delta "1cm 1cm" --scale 0.9 --outfile ${name}-beamer-handouts2x3.pdf ${name}.pdf
rm -f ${name}.pdf
# Ordinary plain LaTeX document
rm -f *.aux # important after beamer
system doconce format pdflatex $name --minted_latex_style=trac --latex_admon=paragraph $opt
system doconce ptex2tex $name envir=minted
# Add special packages
doconce subst "% Add user's preamble" "\g<1>\n\\usepackage{simplewick}" $name.tex
doconce replace 'section{' 'section*{' $name.tex
pdflatex -shell-escape $name
pdflatex -shell-escape $name
mv -f $name.pdf ${name}-minted.pdf
cp $name.tex ${name}-plain-minted.tex
# Publish
dest=../../pub
if [ ! -d $dest/$name ]; then
mkdir $dest/$name
mkdir $dest/$name/pdf
mkdir $dest/$name/html
mkdir $dest/$name/ipynb
fi
cp ${name}*.pdf $dest/$name/pdf
cp -r ${name}*.html ._${name}*.html reveal.js $dest/$name/html
# Figures: cannot just copy link, need to physically copy the files
if [ -d fig-${name} ]; then
if [ ! -d $dest/$name/html/fig-$name ]; then
mkdir $dest/$name/html/fig-$name
fi
cp -r fig-${name}/* $dest/$name/html/fig-$name
fi
cp ${name}.ipynb $dest/$name/ipynb
ipynb_tarfile=ipynb-${name}-src.tar.gz
if [ ! -f ${ipynb_tarfile} ]; then
cat > README.txt <<EOF
This IPython notebook ${name}.ipynb does not require any additional
programs.
EOF
tar czf ${ipynb_tarfile} README.txt
fi
cp ${ipynb_tarfile} $dest/$name/ipynb
doc/web/course.do.txt
+2 -1
View File
@@ -10,7 +10,7 @@ AUTHOR: "Morten Hjorth-Jensen":"http://mhjgit.github.io/info/doc/web/" at Depart
<%
pub_url = 'https://compphysics.github.io/MachineLearning/doc/pub'
published = ['Intro2Course', 'Introduction', 'How2ReadData', 'Linalg', 'Statistics', 'Splines', 'Regression', 'NeuralNet', 'Bayesian', 'DecisionTrees', 'svm', 'BM',]
published = ['Intro2Course', 'Introduction', 'How2ReadData', 'Linalg', 'Statistics', 'Splines', 'Regression', 'LogReg', 'NeuralNet', 'Bayesian', 'DecisionTrees', 'svm', 'BM',]
chapters = {
'Intro2Course': 'Basic introduction to the course with schedule etc',
'Introduction': 'Introduction to Data Analysis and Machine Learning',
@@ -19,6 +19,7 @@ chapters = {
'Statistics': 'Monte Carlo methods and elements of probability theory',
'Splines': 'Splines and Gradient methods',
'Regression': 'Regression Methods',
'LogReg': 'Logistic Regression',
'NeuralNet': 'Neural Networks',
'Bayesian': 'Elements of Bayesian theory',
'DecisionTrees': 'Decision trees, from simple to random ones',
doc/web/course.html
+74 -44
View File
@@ -84,41 +84,42 @@ div { text-align: justify; text-justify: inter-word; }
'___sec4'),
('Splines and Gradient methods', 2, None, '___sec5'),
('Regression Methods', 2, None, '___sec6'),
('Neural Networks', 2, None, '___sec7'),
('Elements of Bayesian theory', 2, None, '___sec8'),
('Logistic Regression', 2, None, '___sec7'),
('Neural Networks', 2, None, '___sec8'),
('Elements of Bayesian theory', 2, None, '___sec9'),
('Decision trees, from simple to random ones',
2,
None,
'___sec9'),
('Support Vector Machines', 2, None, '___sec10'),
'___sec10'),
('Support Vector Machines', 2, None, '___sec11'),
('Unsupervised Learning, Boltzmann Machines',
2,
None,
'___sec11'),
('Python and Scikit Learn, a short guide', 2, None, '___sec12'),
('Teach yourself C++', 2, None, '___sec13'),
('Projects and Exercises Fall 2018', 2, None, '___sec14'),
('First homework set, week 35', 3, None, '___sec15'),
('Second homework set, week 36', 3, None, '___sec16'),
('Project 1, Deadline October 1', 3, None, '___sec17'),
('Project 2, Deadline November 5', 3, None, '___sec18'),
('Project 3, Deadline November 30', 3, None, '___sec19'),
('Course content', 3, None, '___sec20'),
('Learning outcomes', 2, None, '___sec21'),
('Prerequisites', 2, None, '___sec22'),
('The course has two central parts', 2, None, '___sec23'),
'___sec12'),
('Python and Scikit Learn, a short guide', 2, None, '___sec13'),
('Teach yourself C++', 2, None, '___sec14'),
('Projects and Exercises Fall 2018', 2, None, '___sec15'),
('First homework set, week 35', 3, None, '___sec16'),
('Second homework set, week 36', 3, None, '___sec17'),
('Project 1, Deadline October 1', 3, None, '___sec18'),
('Project 2, Deadline November 5', 3, None, '___sec19'),
('Project 3, Deadline November 30', 3, None, '___sec20'),
('Course content', 3, None, '___sec21'),
('Learning outcomes', 2, None, '___sec22'),
('Prerequisites', 2, None, '___sec23'),
('The course has two central parts', 2, None, '___sec24'),
('Statistical analysis and optimization of data',
3,
None,
'___sec24'),
('Machine learning', 3, None, '___sec25'),
('Recommended textbooks', 2, None, '___sec26'),
'___sec25'),
('Machine learning', 3, None, '___sec26'),
('Recommended textbooks', 2, None, '___sec27'),
('"Other '
'textbooks":"https://github.com/CompPhysics/MachineLearning/tree/master/doc/Textbooks"',
2,
None,
'___sec27'),
('Teaching schedule Fall 2018', 2, None, '___sec28')]}
'___sec28'),
('Teaching schedule Fall 2018', 2, None, '___sec29')]}
end of tocinfo -->
<body>
@@ -384,7 +385,36 @@ formulas in HTML or ipython notebook files.
</ul>
<h2 id="___sec7">Neural Networks </h2>
<h2 id="___sec7">Logistic Regression </h2>
<ul>
<li> LaTeX PDF:</li>
<ul>
<li> For printing:</li>
<ul>
<li> <a href="https://compphysics.github.io/MachineLearning/doc/pub/LogReg/pdf/LogReg-minted.pdf" target="_self">Standard one-page format</a></li>
</ul>
</ul>
<li> HTML:</li>
<ul>
<li> <a href="https://compphysics.github.io/MachineLearning/doc/pub/LogReg/html/LogReg.html" target="_self">Plain html</a></li>
<li> <a href="https://compphysics.github.io/MachineLearning/doc/pub/LogReg/html/LogReg-bs.html" target="_self">Bootstrap slide style, easy for reading on mobile devices</a></li>
</ul>
<li> Jupyter notebook:</li>
<ul>
<li> <a href="https://compphysics.github.io/MachineLearning/doc/pub/LogReg/ipynb/LogReg.ipynb" target="_self">ipynb file</a></li>
</ul>
</ul>
<h2 id="___sec8">Neural Networks </h2>
<ul>
<li> LaTeX PDF:</li>
@@ -413,7 +443,7 @@ formulas in HTML or ipython notebook files.
</ul>
<h2 id="___sec8">Elements of Bayesian theory </h2>
<h2 id="___sec9">Elements of Bayesian theory </h2>
<ul>
<li> LaTeX PDF:</li>
@@ -442,7 +472,7 @@ formulas in HTML or ipython notebook files.
</ul>
<h2 id="___sec9">Decision trees, from simple to random ones </h2>
<h2 id="___sec10">Decision trees, from simple to random ones </h2>
<ul>
<li> LaTeX PDF:</li>
@@ -471,7 +501,7 @@ formulas in HTML or ipython notebook files.
</ul>
<h2 id="___sec10">Support Vector Machines </h2>
<h2 id="___sec11">Support Vector Machines </h2>
<ul>
<li> LaTeX PDF:</li>
@@ -500,7 +530,7 @@ formulas in HTML or ipython notebook files.
</ul>
<h2 id="___sec11">Unsupervised Learning, Boltzmann Machines </h2>
<h2 id="___sec12">Unsupervised Learning, Boltzmann Machines </h2>
<ul>
<li> LaTeX PDF:</li>
@@ -531,7 +561,7 @@ formulas in HTML or ipython notebook files.
<!-- !split -->
<h2 id="___sec12">Python and Scikit Learn, a short guide </h2>
<h2 id="___sec13">Python and Scikit Learn, a short guide </h2>
<ul>
<li> HTML format only:</li>
@@ -544,7 +574,7 @@ formulas in HTML or ipython notebook files.
<!-- !split -->
<h2 id="___sec13">Teach yourself C++ </h2>
<h2 id="___sec14">Teach yourself C++ </h2>
<ul>
<li> HTML format only:</li>
@@ -557,9 +587,9 @@ formulas in HTML or ipython notebook files.
<!-- !split -->
<h2 id="___sec14">Projects and Exercises Fall 2018 </h2>
<h2 id="___sec15">Projects and Exercises Fall 2018 </h2>
<h3 id="___sec15">First homework set, week 35 </h3>
<h3 id="___sec16">First homework set, week 35 </h3>
<ul>
<li> LaTeX and PDF:</li>
@@ -578,7 +608,7 @@ formulas in HTML or ipython notebook files.
</ul>
<h3 id="___sec16">Second homework set, week 36 </h3>
<h3 id="___sec17">Second homework set, week 36 </h3>
<ul>
<li> LaTeX and PDF:</li>
@@ -597,7 +627,7 @@ formulas in HTML or ipython notebook files.
</ul>
<h3 id="___sec17">Project 1, Deadline October 1 </h3>
<h3 id="___sec18">Project 1, Deadline October 1 </h3>
<ul>
<li> LaTeX and PDF:</li>
@@ -616,7 +646,7 @@ formulas in HTML or ipython notebook files.
</ul>
<h3 id="___sec18">Project 2, Deadline November 5 </h3>
<h3 id="___sec19">Project 2, Deadline November 5 </h3>
<ul>
<li> LaTeX and PDF:</li>
@@ -635,7 +665,7 @@ formulas in HTML or ipython notebook files.
</ul>
<h3 id="___sec19">Project 3, Deadline November 30 </h3>
<h3 id="___sec20">Project 3, Deadline November 30 </h3>
<ul>
<li> LaTeX and PDF:</li>
@@ -654,7 +684,7 @@ formulas in HTML or ipython notebook files.
</ul>
<h3 id="___sec20">Course content </h3>
<h3 id="___sec21">Course content </h3>
<p>
Probability theory and statistical methods play a central role in science. Nowadays we are
@@ -673,7 +703,7 @@ tools of probability theory, the aim of this course is to expose you to central
This course covers thus topics like Monte Carlo methods and Markov chains, Bayesian statistics, error estimates, various linear methods, optimization of data and error analysis and central algorithms in machine learning.
The course has several numerical projects and numerical exercises that are meant to illustrate the theory.
<h2 id="___sec21">Learning outcomes </h2>
<h2 id="___sec22">Learning outcomes </h2>
<p>
The course introduces a variety of central algorithms and methods
@@ -690,19 +720,19 @@ essential for studies of data analysis and machine learning. The course is proje
<li> Work on numerical projects to illustrate the theory. The projects play a central role and students are expected to know modern programming languages like Python or C++.</li>
</ul>
<h2 id="___sec22">Prerequisites </h2>
<h2 id="___sec23">Prerequisites </h2>
<p>
Basic knowledge in programming and numerics. Required courses are the equivalents to the University of Oslo mathematics courses MAT1100, MAT1110, MAT1120 and at least one of the corresponding computing and programming courses INF1000/INF1110 or MAT-INF1100/MAT-INF1100L/BIOS1100/KJM-INF1xxx.
<h2 id="___sec23">The course has two central parts </h2>
<h2 id="___sec24">The course has two central parts </h2>
<ol>
<li> Statistical analysis and optimization of data</li>
<li> Machine learning</li>
</ol>
<h3 id="___sec24">Statistical analysis and optimization of data </h3>
<h3 id="___sec25">Statistical analysis and optimization of data </h3>
<p>
The following topics will be covered
@@ -719,7 +749,7 @@ The following topics will be covered
<li> Practical optimization using Singular-value decomposition and least squares for parameterizing data.</li>
</ul>
<h3 id="___sec25">Machine learning </h3>
<h3 id="___sec26">Machine learning </h3>
<p>
The following topics will be covered
@@ -735,14 +765,14 @@ The following topics will be covered
All the above topics will be supported by examples, hands-on exercises and project work.
<h2 id="___sec26">Recommended textbooks </h2>
<h2 id="___sec27">Recommended textbooks </h2>
<ul>
<li> <a href="https://www.springer.com/gp/book/9780387848570" target="_self">Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer</a></li>
<li> <a href="http://shop.oreilly.com/product/0636920052289.do" target="_self">Aurelien Geron, Hands&#8209;On Machine Learning with Scikit&#8209;Learn and TensorFlow, O'Reilly</a></li>
</ul>
<h2 id="___sec27"><a href="https://github.com/CompPhysics/MachineLearning/tree/master/doc/Textbooks" target="_self">Other textbooks</a> </h2>
<h2 id="___sec28"><a href="https://github.com/CompPhysics/MachineLearning/tree/master/doc/Textbooks" target="_self">Other textbooks</a> </h2>
<p>
<b>General learning book on statistical analysis</b>:
@@ -764,7 +794,7 @@ All the above topics will be supported by examples, hands-on exercises and proje
<!-- !split -->
<h2 id="___sec28">Teaching schedule Fall 2018 </h2>
<h2 id="___sec29">Teaching schedule Fall 2018 </h2>
<p>
Acronyms for textbooks and references to chapter