lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on logreg

Browse Source
This commit is contained in:
mhjensen
2020-09-16 10:57:03 +02:00
parent 8b7b993103
commit 25b62ff618
27 changed files with 1472 additions and 1411 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week38/html/._week38-bs000.html
+45 -41
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -186,7 +190,7 @@ MathJax.Hub.Config({
<li><a href="._week38-bs008.html">9</a></li>
<li><a href="._week38-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs001.html
+48 -46
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,13 +147,11 @@ MathJax.Hub.Config({
<a name="part0001"></a>
<!-- !split -->
<h2 id="___sec0" class="anchor">To do for log reg </h2>
<h2 id="___sec0" class="anchor">Plans for week 38 </h2>
<ul>
<li> Develop code for log reg step by step, with link to gradient descent part</li>
<li> show how to read and set up design matrix</li>
<li> use breast cancer data as example</li>
<li> develop other classification examples, pulsar example</li>
<li> Thursday: Summary of regression methods and discussion of project 1. We revisit also cross-validation and bootstrap as resampling techniques with examples</li>
<li> Friday: Logistic Regression</li>
</ul>
<p>
@@ -168,7 +170,7 @@ MathJax.Hub.Config({
<li><a href="._week38-bs009.html">10</a></li>
<li><a href="._week38-bs010.html">11</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs002.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs002.html
+47 -56
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -141,22 +145,9 @@ MathJax.Hub.Config({
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0002"></a>
<!-- !split -->
<!-- !split -->
<h2 id="___sec1" class="anchor">Logistic Regression </h2>
<p>
In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable \( y_i \) is based on some
independent variables \( \hat{x}_i \). Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
parameters \( \hat{\beta} \) to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.
<h2 id="___sec1" class="anchor">Thursday: </h2>
<p>
<p>
@@ -176,7 +167,7 @@ simple recipe for fitting our data.
<li><a href="._week38-bs010.html">11</a></li>
<li><a href="._week38-bs011.html">12</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs003.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs003.html
+47 -61
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -141,27 +145,9 @@ MathJax.Hub.Config({
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0003"></a>
<!-- !split -->
<!-- !split -->
<h2 id="___sec2" class="anchor">Classification problems </h2>
<p>
Classification problems, however, are concerned with outcomes taking
the form of discrete variables (i.e. categories). We may for example,
on the basis of DNA sequencing for a number of patients, like to find
out which mutations are important for a certain disease; or based on
scans of various patients' brains, figure out if there is a tumor or
not; or given a specific physical system, we'd like to identify its
state, say whether it is an ordered or disordered system (typical
situation in solid state physics); or classify the status of a
patient, whether she/he has a stroke or not and many other similar
situations.
<p>
The most common situation we encounter when we apply logistic
regression is that of two possible outcomes, normally denoted as a
binary outcome, true or false, positive or negative, success or
failure etc.
<h2 id="___sec2" class="anchor">Friday: Intro to Logistic Regression </h2>
<p>
<p>
@@ -182,7 +168,7 @@ failure etc.
<li><a href="._week38-bs011.html">12</a></li>
<li><a href="._week38-bs012.html">13</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs004.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs004.html
+58 -57
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -141,25 +145,22 @@ MathJax.Hub.Config({
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0004"></a>
<!-- !split -->
<!-- !split -->
<h2 id="___sec3" class="anchor">Optimization and Deep learning </h2>
<h2 id="___sec3" class="anchor">Logistic Regression </h2>
<p>
Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters \( \hat{\beta} \). The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
family of gradient descent methods. The latter are the working horses
of basically all modern machine learning algorithms.
<p>
We note also that many of the topics discussed here on logistic
regression are also commonly used in modern supervised Deep Learning
models, as we will see later.
In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable \( y_i \) is based on some
independent variables \( \hat{x}_i \). Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
parameters \( \hat{\beta} \) to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.
<p>
<p>
@@ -181,7 +182,7 @@ models, as we will see later.
<li><a href="._week38-bs012.html">13</a></li>
<li><a href="._week38-bs013.html">14</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs005.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs005.html
+60 -60
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,29 +147,25 @@ MathJax.Hub.Config({
<a name="part0005"></a>
<!-- !split -->
<h2 id="___sec4" class="anchor">Basics </h2>
<h2 id="___sec4" class="anchor">Classification problems </h2>
<p>
We consider the case where the dependent variables, also called the
responses or the outcomes, \( y_i \) are discrete and only take values
from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
Classification problems, however, are concerned with outcomes taking
the form of discrete variables (i.e. categories). We may for example,
on the basis of DNA sequencing for a number of patients, like to find
out which mutations are important for a certain disease; or based on
scans of various patients' brains, figure out if there is a tumor or
not; or given a specific physical system, we'd like to identify its
state, say whether it is an ordered or disordered system (typical
situation in solid state physics); or classify the status of a
patient, whether she/he has a stroke or not and many other similar
situations.
<p>
The goal is to predict the
output classes from the design matrix \( \hat{X}\in\mathbb{R}^{n\times p} \)
made of \( n \) samples, each of which carries \( p \) features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.
<p>
Let us specialize to the case of two classes only, with outputs
\( y_i=0 \) and \( y_i=1 \). Our outcomes could represent the status of a
credit card user that could default or not on her/his credit card
debt. That is
$$
y_i = \begin{bmatrix} 0 & \mathrm{no}\\ 1 & \mathrm{yes} \end{bmatrix}.
$$
The most common situation we encounter when we apply logistic
regression is that of two possible outcomes, normally denoted as a
binary outcome, true or false, positive or negative, success or
failure etc.
<p>
<p>
@@ -188,7 +188,7 @@ $$
<li><a href="._week38-bs013.html">14</a></li>
<li><a href="._week38-bs014.html">15</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs006.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs006.html
+58 -57
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,26 +147,23 @@ MathJax.Hub.Config({
<a name="part0006"></a>
<!-- !split -->
<h2 id="___sec5" class="anchor">Linear classifier </h2>
<h2 id="___sec5" class="anchor">Optimization and Deep learning </h2>
<p>
Before moving to the logistic model, let us try to use our linear
regression model to classify these two outcomes. We could for example
fit a linear model to the default case if \( y_i > 0.5 \) and the no
default case \( y_i \leq 0.5 \).
Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters \( \hat{\beta} \). The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
family of gradient descent methods. The latter are the working horses
of basically all modern machine learning algorithms.
<p>
We would then have our
weighted linear combination, namely
$$
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon},
\tag{1}
\end{equation}
$$
where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our
\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators/predictors.
We note also that many of the topics discussed here on logistic
regression are also commonly used in modern supervised Deep Learning
models, as we will see later.
<p>
<p>
@@ -186,7 +187,7 @@ where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \
<li><a href="._week38-bs014.html">15</a></li>
<li><a href="._week38-bs015.html">16</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs007.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs007.html
+65 -56
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -141,26 +145,31 @@ MathJax.Hub.Config({
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0007"></a>
<!-- !split -->
<!-- !split -->
<h2 id="___sec6" class="anchor">Some selected properties </h2>
<h2 id="___sec6" class="anchor">Basics </h2>
<p>
The main problem with our function is that it takes values on the
entire real axis. In the case of logistic regression, however, the
labels \( y_i \) are discrete variables. A typical example is the credit
card data discussed below here, where we can set the state of
defaulting the debt to \( y_i=1 \) and not to \( y_i=0 \) for one the persons
in the data set (see the full example below).
We consider the case where the dependent variables, also called the
responses or the outcomes, \( y_i \) are discrete and only take values
from \( k=0,\dots,K-1 \) (i.e. \( K \) classes).
<p>
One simple way to get a discrete output is to have sign
functions that map the output of a linear regressor to values \( \{0,1\} \),
\( f(s_i)=sign(s_i)=1 \) if \( s_i\ge 0 \) and 0 if otherwise.
We will encounter this model in our first demonstration of neural networks. Historically it is called the &quot;perceptron" model in the machine learning
literature. This model is extremely simple. However, in many cases it is more
favorable to use a ``soft" classifier that outputs
the probability of a given category. This leads us to the logistic function.
The goal is to predict the
output classes from the design matrix \( \hat{X}\in\mathbb{R}^{n\times p} \)
made of \( n \) samples, each of which carries \( p \) features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.
<p>
Let us specialize to the case of two classes only, with outputs
\( y_i=0 \) and \( y_i=1 \). Our outcomes could represent the status of a
credit card user that could default or not on her/his credit card
debt. That is
$$
y_i = \begin{bmatrix} 0 & \mathrm{no}\\ 1 & \mathrm{yes} \end{bmatrix}.
$$
<p>
<p>
@@ -185,7 +194,7 @@ the probability of a given category. This leads us to the logistic function.
<li><a href="._week38-bs015.html">16</a></li>
<li><a href="._week38-bs016.html">17</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs008.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs008.html
+60 -54
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,24 +147,26 @@ MathJax.Hub.Config({
<a name="part0008"></a>
<!-- !split -->
<h2 id="___sec7" class="anchor">The logistic function </h2>
<h2 id="___sec7" class="anchor">Linear classifier </h2>
<p>
The perceptron is an example of a ``hard classification&quot; model. We
will encounter this model when we discuss neural networks as
well. Each datapoint is deterministically assigned to a category (i.e
\( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a &quot;soft&quot;
classifier that outputs the probability of a given category rather
than a single value. For example, given \( x_i \), the classifier
outputs the probability of being in a category \( k \). Logistic regression
is the most common example of a so-called soft classifier. In logistic
regression, the probability that a data point \( x_i \)
belongs to a category \( y_i=\{0,1\} \) is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,
Before moving to the logistic model, let us try to use our linear
regression model to classify these two outcomes. We could for example
fit a linear model to the default case if \( y_i > 0.5 \) and the no
default case \( y_i \leq 0.5 \).
<p>
We would then have our
weighted linear combination, namely
$$
p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon},
\tag{1}
\end{equation}
$$
Note that \( 1-p(t)= p(-t) \).
where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \) is our
\( n\times p \) design matrix and \( \hat{\beta} \) represents our estimators/predictors.
<p>
<p>
@@ -186,7 +192,7 @@ Note that \( 1-p(t)= p(-t) \).
<li><a href="._week38-bs016.html">17</a></li>
<li><a href="._week38-bs017.html">18</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs009.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs009.html
+59 -99
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,69 +147,25 @@ MathJax.Hub.Config({
<a name="part0009"></a>
<!-- !split -->
<h2 id="___sec8" class="anchor">Examples of likelihood functions used in logistic regression and nueral networks </h2>
<h2 id="___sec8" class="anchor">Some selected properties </h2>
<p>
The following code plots the logistic function, the step function and other functions we will encounter from here and on.
The main problem with our function is that it takes values on the
entire real axis. In the case of logistic regression, however, the
labels \( y_i \) are discrete variables. A typical example is the credit
card data discussed below here, where we can set the state of
defaulting the debt to \( y_i=1 \) and not to \( y_i=0 \) for one the persons
in the data set (see the full example below).
<p>
One simple way to get a discrete output is to have sign
functions that map the output of a linear regressor to values \( \{0,1\} \),
\( f(s_i)=sign(s_i)=1 \) if \( s_i\ge 0 \) and 0 if otherwise.
We will encounter this model in our first demonstration of neural networks. Historically it is called the &quot;perceptron" model in the machine learning
literature. This model is extremely simple. However, in many cases it is more
favorable to use a ``soft" classifier that outputs
the probability of a given category. This leads us to the logistic function.
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;The sigmoid function (or the logistic curve) is a</span>
<span style="color: #BA2121; font-style: italic">function that takes any real number, z, and outputs a number (0,1).</span>
<span style="color: #BA2121; font-style: italic">It is useful in neural networks for assigning weights on a relative scale.</span>
<span style="color: #BA2121; font-style: italic">The value z is the weighted sum of parameters involved in the learning algorithm.&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">math</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">mt</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-5</span>, <span style="color: #666666">5</span>, <span style="color: #666666">.1</span>)
sigma_fn <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>vectorize(<span style="color: #008000; font-weight: bold">lambda</span> z: <span style="color: #666666">1/</span>(<span style="color: #666666">1+</span>numpy<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>z)))
sigma <span style="color: #666666">=</span> sigma_fn(z)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, sigma)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-5</span>,<span style="color: #666666">5</span>])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;sigmoid function&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Step Function&quot;&quot;&quot;</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-5</span>, <span style="color: #666666">5</span>, <span style="color: #666666">.02</span>)
step_fn <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>vectorize(<span style="color: #008000; font-weight: bold">lambda</span> z: <span style="color: #666666">1.0</span> <span style="color: #008000; font-weight: bold">if</span> z <span style="color: #666666">&gt;=</span> <span style="color: #666666">0.0</span> <span style="color: #008000; font-weight: bold">else</span> <span style="color: #666666">0.0</span>)
step <span style="color: #666666">=</span> step_fn(z)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, step)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-0.5</span>, <span style="color: #666666">1.5</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-5</span>,<span style="color: #666666">5</span>])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;step function&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;tanh Function&quot;&quot;&quot;</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-2*</span>mt<span style="color: #666666">.</span>pi, <span style="color: #666666">2*</span>mt<span style="color: #666666">.</span>pi, <span style="color: #666666">0.1</span>)
t <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>tanh(z)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, t)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-1.0</span>, <span style="color: #666666">1.0</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-2*</span>mt<span style="color: #666666">.</span>pi,<span style="color: #666666">2*</span>mt<span style="color: #666666">.</span>pi])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;tanh function&#39;</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
@@ -231,7 +191,7 @@ plt<span style="color: #666666">.</span>show()
<li><a href="._week38-bs017.html">18</a></li>
<li><a href="._week38-bs018.html">19</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs010.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs010.html
+58 -54
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,24 +147,24 @@ MathJax.Hub.Config({
<a name="part0010"></a>
<!-- !split -->
<h2 id="___sec9" class="anchor">Two parameters </h2>
<h2 id="___sec9" class="anchor">The logistic function </h2>
<p>
We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
The perceptron is an example of a ``hard classification&quot; model. We
will encounter this model when we discuss neural networks as
well. Each datapoint is deterministically assigned to a category (i.e
\( y_i=0 \) or \( y_i=1 \)). In many cases, it is favorable to have a &quot;soft&quot;
classifier that outputs the probability of a given category rather
than a single value. For example, given \( x_i \), the classifier
outputs the probability of being in a category \( k \). Logistic regression
is the most common example of a so-called soft classifier. In logistic
regression, the probability that a data point \( x_i \)
belongs to a category \( y_i=\{0,1\} \) is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,
$$
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
\end{align*}
p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
$$
where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
<p>
Note that we used
$$
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
$$
Note that \( 1-p(t)= p(-t) \).
<p>
<p>
@@ -188,7 +192,7 @@ $$
<li><a href="._week38-bs018.html">19</a></li>
<li><a href="._week38-bs019.html">20</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs011.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs011.html
+106 -57
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -141,28 +145,71 @@ MathJax.Hub.Config({
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0011"></a>
<!-- !split -->
<!-- !split -->
<h2 id="___sec10" class="anchor">Maximum likelihood </h2>
<h2 id="___sec10" class="anchor">Examples of likelihood functions used in logistic regression and nueral networks </h2>
<p>
In order to define the total likelihood for all possible outcomes from a
dataset \( \mathcal{D}=\{(y_i,x_i)\} \), with the binary labels
\( y_i\in\{0,1\} \) and where the data points are drawn independently, we use the so-called <a href="https://en.wikipedia.org/wiki/Maximum_likelihood_estimation" target="_self">Maximum Likelihood Estimation</a> (MLE) principle.
We aim thus at maximizing
the probability of seeing the observed data. We can then approximate the
likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is
$$
\begin{align*}
P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
$$
The following code plots the logistic function, the step function and other functions we will encounter from here and on.
from which we obtain the log-likelihood and our <b>cost/loss</b> function
$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
$$
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;The sigmoid function (or the logistic curve) is a</span>
<span style="color: #BA2121; font-style: italic">function that takes any real number, z, and outputs a number (0,1).</span>
<span style="color: #BA2121; font-style: italic">It is useful in neural networks for assigning weights on a relative scale.</span>
<span style="color: #BA2121; font-style: italic">The value z is the weighted sum of parameters involved in the learning algorithm.&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">math</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">mt</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-5</span>, <span style="color: #666666">5</span>, <span style="color: #666666">.1</span>)
sigma_fn <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>vectorize(<span style="color: #008000; font-weight: bold">lambda</span> z: <span style="color: #666666">1/</span>(<span style="color: #666666">1+</span>numpy<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>z)))
sigma <span style="color: #666666">=</span> sigma_fn(z)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, sigma)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-5</span>,<span style="color: #666666">5</span>])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;sigmoid function&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Step Function&quot;&quot;&quot;</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-5</span>, <span style="color: #666666">5</span>, <span style="color: #666666">.02</span>)
step_fn <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>vectorize(<span style="color: #008000; font-weight: bold">lambda</span> z: <span style="color: #666666">1.0</span> <span style="color: #008000; font-weight: bold">if</span> z <span style="color: #666666">&gt;=</span> <span style="color: #666666">0.0</span> <span style="color: #008000; font-weight: bold">else</span> <span style="color: #666666">0.0</span>)
step <span style="color: #666666">=</span> step_fn(z)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, step)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-0.5</span>, <span style="color: #666666">1.5</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-5</span>,<span style="color: #666666">5</span>])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;step function&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;tanh Function&quot;&quot;&quot;</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-2*</span>mt<span style="color: #666666">.</span>pi, <span style="color: #666666">2*</span>mt<span style="color: #666666">.</span>pi, <span style="color: #666666">0.1</span>)
t <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>tanh(z)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, t)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-1.0</span>, <span style="color: #666666">1.0</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-2*</span>mt<span style="color: #666666">.</span>pi,<span style="color: #666666">2*</span>mt<span style="color: #666666">.</span>pi])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;tanh function&#39;</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
@@ -188,6 +235,8 @@ $$
<li><a href="._week38-bs018.html">19</a></li>
<li><a href="._week38-bs019.html">20</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs012.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs012.html
+57 -49
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,24 +147,25 @@ MathJax.Hub.Config({
<a name="part0012"></a>
<!-- !split -->
<h2 id="___sec11" class="anchor">The cost function rewritten </h2>
<h2 id="___sec11" class="anchor">Two parameters </h2>
<p>
Reordering the logarithms, we can rewrite the <b>cost/loss</b> function as
We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
\end{align*}
$$
where \( \hat{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
<p>
The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \beta \).
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
Note that we used
$$
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
$$
This equation is known in statistics as the <b>cross entropy</b>. Finally, we note that just as in linear regression,
in practice we often supplement the cross-entropy with additional regularization terms, usually \( L_1 \) and \( L_2 \) regularization as we did for Ridge and Lasso regression.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
@@ -185,6 +190,9 @@ in practice we often supplement the cross-entropy with additional regularization
<li><a href="._week38-bs018.html">19</a></li>
<li><a href="._week38-bs019.html">20</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs021.html">22</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs013.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs013.html
+59 -52
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -141,25 +145,26 @@ MathJax.Hub.Config({
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0013"></a>
<!-- !split -->
<!-- !split -->
<h2 id="___sec12" class="anchor">Minimizing the cross entropy </h2>
<h2 id="___sec12" class="anchor">Maximum likelihood </h2>
<p>
The cross entropy is a convex function of the weights \( \hat{\beta} \) and,
therefore, any local minimizer is a global minimizer.
<p>
Minimizing this
cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain
In order to define the total likelihood for all possible outcomes from a
dataset \( \mathcal{D}=\{(y_i,x_i)\} \), with the binary labels
\( y_i\in\{0,1\} \) and where the data points are drawn independently, we use the so-called <a href="https://en.wikipedia.org/wiki/Maximum_likelihood_estimation" target="_self">Maximum Likelihood Estimation</a> (MLE) principle.
We aim thus at maximizing
the probability of seeing the observed data. We can then approximate the
likelihood in terms of the product of the individual probabilities of a specific outcome \( y_i \), that is
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
\begin{align*}
P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
$$
and
from which we obtain the log-likelihood and our <b>cost/loss</b> function
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
$$
<p>
@@ -185,6 +190,8 @@ $$
<li><a href="._week38-bs018.html">19</a></li>
<li><a href="._week38-bs019.html">20</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs021.html">22</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs014.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs014.html
+56 -52
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,25 +147,23 @@ MathJax.Hub.Config({
<a name="part0014"></a>
<!-- !split -->
<h2 id="___sec13" class="anchor">A more compact expression </h2>
<h2 id="___sec13" class="anchor">The cost function rewritten </h2>
<p>
Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an
\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a
vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first
derivative of cost function as
Reordering the logarithms, we can rewrite the <b>cost/loss</b> function as
$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$
<p>
If we in addition define a diagonal matrix \( \hat{W} \) with elements
\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as
The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to \( \beta \).
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that
$$
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$
$$
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
$$
This equation is known in statistics as the <b>cross entropy</b>. Finally, we note that just as in linear regression,
in practice we often supplement the cross-entropy with additional regularization terms, usually \( L_1 \) and \( L_2 \) regularization as we did for Ridge and Lasso regression.
<p>
<p>
@@ -185,6 +187,8 @@ $$
<li><a href="._week38-bs018.html">19</a></li>
<li><a href="._week38-bs019.html">20</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs021.html">22</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs015.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs015.html
+57 -45
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,17 +147,23 @@ MathJax.Hub.Config({
<a name="part0015"></a>
<!-- !split -->
<h2 id="___sec14" class="anchor">Extending to more predictors </h2>
<h2 id="___sec14" class="anchor">Minimizing the cross entropy </h2>
<p>
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors
The cross entropy is a convex function of the weights \( \hat{\beta} \) and,
therefore, any local minimizer is a global minimizer.
<p>
Minimizing this
cost function with respect to the two parameters \( \beta_0 \) and \( \beta_1 \) we obtain
$$
\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
$$
Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
and
$$
p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$
<p>
@@ -177,6 +187,8 @@ $$
<li><a href="._week38-bs018.html">19</a></li>
<li><a href="._week38-bs019.html">20</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs021.html">22</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs016.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs016.html
+58 -57
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,30 +147,25 @@ MathJax.Hub.Config({
<a name="part0016"></a>
<!-- !split -->
<h2 id="___sec15" class="anchor">Including more classes </h2>
<h2 id="___sec15" class="anchor">A more compact expression </h2>
<p>
Till now we have mainly focused on two classes, the so-called binary
system. Suppose we wish to extend to \( K \) classes. Let us for the sake
of simplicity assume we have only two predictors. We have then
following model
Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an
\( n\times p \) matrix \( \hat{X} \) which contains the \( x_i \) values and a
vector \( \hat{p} \) of fitted probabilities \( p(y_i\vert x_i,\hat{\beta}) \). We can rewrite in a more compact form the first
derivative of cost function as
$$
\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1,
$$
$$
\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
$$
and so on till the class \( C=K-1 \) class
$$
\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
$$
<p>
and the model is specified in term of \( K-1 \) so-called log-odds or
<b>logit</b> transformations.
If we in addition define a diagonal matrix \( \hat{W} \) with elements
\( p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta}) \), we can obtain a compact expression of the second derivative as
$$
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
$$
<p>
<p>
@@ -188,6 +187,8 @@ and the model is specified in term of \( K-1 \) so-called log-odds or
<li><a href="._week38-bs018.html">19</a></li>
<li><a href="._week38-bs019.html">20</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs021.html">22</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs017.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs017.html
+51 -69
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,43 +147,19 @@ MathJax.Hub.Config({
<a name="part0017"></a>
<!-- !split -->
<h2 id="___sec16" class="anchor">More classes </h2>
<h2 id="___sec16" class="anchor">Extending to more predictors </h2>
<p>
In our discussion of neural networks we will encounter the above again
in terms of a slightly modified function, the so-called <b>Softmax</b> function.
<p>
The softmax function is used in various multiclass classification
methods, such as multinomial logistic regression (also known as
softmax regression), multiclass linear discriminant analysis, naive
Bayes classifiers, and artificial neural networks. Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of \( K \) distinct linear functions,
and the predicted probability for the \( k \)-th class given a sample
vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two
predictors):
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors
$$
p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
$$
It is easy to extend to more predictors. The final class is
Here we defined \( \hat{x}=[1,x_1,x_2,\dots,x_p] \) and \( \hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p] \) leading to
$$
p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
$$
<p>
and they sum to one. Our earlier discussions were all specialized to
the case with two classes only. It is easy to see from the above that
what we derived earlier is compatible with these equations.
<p>
To find the optimal parameters we would typically use a gradient
descent method. Newton's method and gradient descent methods are
discussed in the material on <a href="https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html" target="_self">optimization
methods</a>.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
@@ -199,6 +179,8 @@ methods</a>.
<li><a href="._week38-bs018.html">19</a></li>
<li><a href="._week38-bs019.html">20</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs021.html">22</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs018.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs018.html
+65 -83
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,55 +147,31 @@ MathJax.Hub.Config({
<a name="part0018"></a>
<!-- !split -->
<h2 id="___sec17" class="anchor">A simple classification problem </h2>
<h2 id="___sec17" class="anchor">Including more classes </h2>
<p>
Till now we have mainly focused on two classes, the so-called binary
system. Suppose we wish to extend to \( K \) classes. Let us for the sake
of simplicity assume we have only two predictors. We have then
following model
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets, linear_model
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
$$
\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1,
$$
$$
\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
$$
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">generate_data</span>():
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">0</span>)
X, y <span style="color: #666666">=</span> datasets<span style="color: #666666">.</span>make_moons(<span style="color: #666666">200</span>, noise<span style="color: #666666">=0.20</span>)
<span style="color: #008000; font-weight: bold">return</span> X, y
and so on till the class \( C=K-1 \) class
$$
\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
$$
<p>
and the model is specified in term of \( K-1 \) so-called log-odds or
<b>logit</b> transformations.
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">visualize</span>(X, y, clf):
plot_decision_boundary(<span style="color: #008000; font-weight: bold">lambda</span> x: clf<span style="color: #666666">.</span>predict(x), X, y)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">plot_decision_boundary</span>(pred_func, X, y):
<span style="color: #408080; font-style: italic"># Set min and max values and give it some padding</span>
x_min, x_max <span style="color: #666666">=</span> X[:, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>min() <span style="color: #666666">-</span> <span style="color: #666666">.5</span>, X[:, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>max() <span style="color: #666666">+</span> <span style="color: #666666">.5</span>
y_min, y_max <span style="color: #666666">=</span> X[:, <span style="color: #666666">1</span>]<span style="color: #666666">.</span>min() <span style="color: #666666">-</span> <span style="color: #666666">.5</span>, X[:, <span style="color: #666666">1</span>]<span style="color: #666666">.</span>max() <span style="color: #666666">+</span> <span style="color: #666666">.5</span>
h <span style="color: #666666">=</span> <span style="color: #666666">0.01</span>
<span style="color: #408080; font-style: italic"># Generate a grid of points with distance h between them</span>
xx, yy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>meshgrid(np<span style="color: #666666">.</span>arange(x_min, x_max, h), np<span style="color: #666666">.</span>arange(y_min, y_max, h))
<span style="color: #408080; font-style: italic"># Predict the function value for the whole gid</span>
Z <span style="color: #666666">=</span> pred_func(np<span style="color: #666666">.</span>c_[xx<span style="color: #666666">.</span>ravel(), yy<span style="color: #666666">.</span>ravel()])
Z <span style="color: #666666">=</span> Z<span style="color: #666666">.</span>reshape(xx<span style="color: #666666">.</span>shape)
<span style="color: #408080; font-style: italic"># Plot the contour and training examples</span>
plt<span style="color: #666666">.</span>contourf(xx, yy, Z, cmap<span style="color: #666666">=</span>plt<span style="color: #666666">.</span>cm<span style="color: #666666">.</span>Spectral)
plt<span style="color: #666666">.</span>scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], c<span style="color: #666666">=</span>y, cmap<span style="color: #666666">=</span>plt<span style="color: #666666">.</span>cm<span style="color: #666666">.</span>Spectral)
plt<span style="color: #666666">.</span>show()
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">classify</span>(X, y):
clf <span style="color: #666666">=</span> linear_model<span style="color: #666666">.</span>LogisticRegressionCV()
clf<span style="color: #666666">.</span>fit(X, y)
<span style="color: #008000; font-weight: bold">return</span> clf
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">main</span>():
X, y <span style="color: #666666">=</span> generate_data()
<span style="color: #408080; font-style: italic"># visualize(X, y)</span>
clf <span style="color: #666666">=</span> classify(X, y)
visualize(X, y, clf)
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #19177C">__name__</span> <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;__main__&quot;</span>:
main()
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
@@ -210,6 +190,8 @@ MathJax.Hub.Config({
<li class="active"><a href="._week38-bs018.html">19</a></li>
<li><a href="._week38-bs019.html">20</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs021.html">22</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs019.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs019.html
+79 -66
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,36 +147,43 @@ MathJax.Hub.Config({
<a name="part0019"></a>
<!-- !split -->
<h2 id="___sec18" class="anchor">Cancer Data again now with Decision Trees and other Methods </h2>
<h2 id="___sec18" class="anchor">More classes </h2>
<p>
In our discussion of neural networks we will encounter the above again
in terms of a slightly modified function, the so-called <b>Softmax</b> function.
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> load_breast_cancer
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LogisticRegression
<p>
The softmax function is used in various multiclass classification
methods, such as multinomial logistic regression (also known as
softmax regression), multiclass linear discriminant analysis, naive
Bayes classifiers, and artificial neural networks. Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of \( K \) distinct linear functions,
and the predicted probability for the \( k \)-th class given a sample
vector \( \hat{x} \) and a weighting vector \( \hat{\beta} \) is (with two
predictors):
<span style="color: #408080; font-style: italic"># Load the data</span>
cancer <span style="color: #666666">=</span> load_breast_cancer()
$$
p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
$$
It is easy to extend to more predictors. The final class is
$$
p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
$$
<p>
and they sum to one. Our earlier discussions were all specialized to
the case with two classes only. It is easy to see from the above that
what we derived earlier is compatible with these equations.
<p>
To find the optimal parameters we would typically use a gradient
descent method. Newton's method and gradient descent methods are
discussed in the material on <a href="https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html" target="_self">optimization
methods</a>.
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(cancer<span style="color: #666666">.</span>data,cancer<span style="color: #666666">.</span>target,random_state<span style="color: #666666">=0</span>)
<span style="color: #008000; font-weight: bold">print</span>(X_train<span style="color: #666666">.</span>shape)
<span style="color: #008000; font-weight: bold">print</span>(X_test<span style="color: #666666">.</span>shape)
<span style="color: #408080; font-style: italic"># Logistic Regression</span>
logreg <span style="color: #666666">=</span> LogisticRegression(solver<span style="color: #666666">=</span><span style="color: #BA2121">&#39;lbfgs&#39;</span>)
logreg<span style="color: #666666">.</span>fit(X_train, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy with Logistic Regression: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test,y_test)))
<span style="color: #408080; font-style: italic">#now scale the data</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
scaler <span style="color: #666666">=</span> StandardScaler()
scaler<span style="color: #666666">.</span>fit(X_train)
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
<span style="color: #408080; font-style: italic"># Logistic Regression</span>
logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy Logistic Regression with scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test_scaled,y_test)))
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
@@ -190,6 +201,8 @@ logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<li><a href="._week38-bs018.html">19</a></li>
<li class="active"><a href="._week38-bs019.html">20</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs021.html">22</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs020.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/._week38-bs020.html
+88 -82
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -143,57 +147,56 @@ MathJax.Hub.Config({
<a name="part0020"></a>
<!-- !split -->
<h2 id="___sec19" class="anchor">Other measures in classification studies: Cancer Data again </h2>
<h2 id="___sec19" class="anchor">A simple classification problem </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> load_breast_cancer
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LogisticRegression
<span style="color: #408080; font-style: italic"># Load the data</span>
cancer <span style="color: #666666">=</span> load_breast_cancer()
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(cancer<span style="color: #666666">.</span>data,cancer<span style="color: #666666">.</span>target,random_state<span style="color: #666666">=0</span>)
<span style="color: #008000; font-weight: bold">print</span>(X_train<span style="color: #666666">.</span>shape)
<span style="color: #008000; font-weight: bold">print</span>(X_test<span style="color: #666666">.</span>shape)
<span style="color: #408080; font-style: italic"># Logistic Regression</span>
logreg <span style="color: #666666">=</span> LogisticRegression(solver<span style="color: #666666">=</span><span style="color: #BA2121">&#39;lbfgs&#39;</span>)
logreg<span style="color: #666666">.</span>fit(X_train, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy with Logistic Regression: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test,y_test)))
<span style="color: #408080; font-style: italic">#now scale the data</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
scaler <span style="color: #666666">=</span> StandardScaler()
scaler<span style="color: #666666">.</span>fit(X_train)
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
<span style="color: #408080; font-style: italic"># Logistic Regression</span>
logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy Logistic Regression with scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test_scaled,y_test)))
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets, linear_model
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> LabelEncoder
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> cross_validate
<span style="color: #408080; font-style: italic">#Cross validation</span>
accuracy <span style="color: #666666">=</span> cross_validate(logreg,X_test_scaled,y_test,cv<span style="color: #666666">=10</span>)[<span style="color: #BA2121">&#39;test_score&#39;</span>]
<span style="color: #008000; font-weight: bold">print</span>(accuracy)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy with Logistic Regression and scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test_scaled,y_test)))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">generate_data</span>():
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">0</span>)
X, y <span style="color: #666666">=</span> datasets<span style="color: #666666">.</span>make_moons(<span style="color: #666666">200</span>, noise<span style="color: #666666">=0.20</span>)
<span style="color: #008000; font-weight: bold">return</span> X, y
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scikitplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">skplt</span>
y_pred <span style="color: #666666">=</span> logreg<span style="color: #666666">.</span>predict(X_test_scaled)
skplt<span style="color: #666666">.</span>metrics<span style="color: #666666">.</span>plot_confusion_matrix(y_test, y_pred, normalize<span style="color: #666666">=</span><span style="color: #008000">True</span>)
plt<span style="color: #666666">.</span>show()
y_probas <span style="color: #666666">=</span> logreg<span style="color: #666666">.</span>predict_proba(X_test_scaled)
skplt<span style="color: #666666">.</span>metrics<span style="color: #666666">.</span>plot_roc(y_test, y_probas)
plt<span style="color: #666666">.</span>show()
skplt<span style="color: #666666">.</span>metrics<span style="color: #666666">.</span>plot_cumulative_gain(y_test, y_probas)
plt<span style="color: #666666">.</span>show()
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">visualize</span>(X, y, clf):
plot_decision_boundary(<span style="color: #008000; font-weight: bold">lambda</span> x: clf<span style="color: #666666">.</span>predict(x), X, y)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">plot_decision_boundary</span>(pred_func, X, y):
<span style="color: #408080; font-style: italic"># Set min and max values and give it some padding</span>
x_min, x_max <span style="color: #666666">=</span> X[:, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>min() <span style="color: #666666">-</span> <span style="color: #666666">.5</span>, X[:, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>max() <span style="color: #666666">+</span> <span style="color: #666666">.5</span>
y_min, y_max <span style="color: #666666">=</span> X[:, <span style="color: #666666">1</span>]<span style="color: #666666">.</span>min() <span style="color: #666666">-</span> <span style="color: #666666">.5</span>, X[:, <span style="color: #666666">1</span>]<span style="color: #666666">.</span>max() <span style="color: #666666">+</span> <span style="color: #666666">.5</span>
h <span style="color: #666666">=</span> <span style="color: #666666">0.01</span>
<span style="color: #408080; font-style: italic"># Generate a grid of points with distance h between them</span>
xx, yy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>meshgrid(np<span style="color: #666666">.</span>arange(x_min, x_max, h), np<span style="color: #666666">.</span>arange(y_min, y_max, h))
<span style="color: #408080; font-style: italic"># Predict the function value for the whole gid</span>
Z <span style="color: #666666">=</span> pred_func(np<span style="color: #666666">.</span>c_[xx<span style="color: #666666">.</span>ravel(), yy<span style="color: #666666">.</span>ravel()])
Z <span style="color: #666666">=</span> Z<span style="color: #666666">.</span>reshape(xx<span style="color: #666666">.</span>shape)
<span style="color: #408080; font-style: italic"># Plot the contour and training examples</span>
plt<span style="color: #666666">.</span>contourf(xx, yy, Z, cmap<span style="color: #666666">=</span>plt<span style="color: #666666">.</span>cm<span style="color: #666666">.</span>Spectral)
plt<span style="color: #666666">.</span>scatter(X[:, <span style="color: #666666">0</span>], X[:, <span style="color: #666666">1</span>], c<span style="color: #666666">=</span>y, cmap<span style="color: #666666">=</span>plt<span style="color: #666666">.</span>cm<span style="color: #666666">.</span>Spectral)
plt<span style="color: #666666">.</span>show()
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">classify</span>(X, y):
clf <span style="color: #666666">=</span> linear_model<span style="color: #666666">.</span>LogisticRegressionCV()
clf<span style="color: #666666">.</span>fit(X, y)
<span style="color: #008000; font-weight: bold">return</span> clf
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">main</span>():
X, y <span style="color: #666666">=</span> generate_data()
<span style="color: #408080; font-style: italic"># visualize(X, y)</span>
clf <span style="color: #666666">=</span> classify(X, y)
visualize(X, y, clf)
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #19177C">__name__</span> <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;__main__&quot;</span>:
main()
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
@@ -209,6 +212,9 @@ plt<span style="color: #666666">.</span>show()
<li><a href="._week38-bs018.html">19</a></li>
<li><a href="._week38-bs019.html">20</a></li>
<li class="active"><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs021.html">22</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs021.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/week38-bs.html
+45 -41
View File
@@ -41,36 +41,38 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -108,26 +110,28 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">To do for log reg</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs001.html#___sec0" style="font-size: 80%;">Plans for week 38</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs002.html#___sec1" style="font-size: 80%;">Thursday:</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs003.html#___sec2" style="font-size: 80%;">Friday: Intro to Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs004.html#___sec3" style="font-size: 80%;">Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs005.html#___sec4" style="font-size: 80%;">Classification problems</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs006.html#___sec5" style="font-size: 80%;">Optimization and Deep learning</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs007.html#___sec6" style="font-size: 80%;">Basics</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs008.html#___sec7" style="font-size: 80%;">Linear classifier</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs009.html#___sec8" style="font-size: 80%;">Some selected properties</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs010.html#___sec9" style="font-size: 80%;">The logistic function</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs011.html#___sec10" style="font-size: 80%;">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs012.html#___sec11" style="font-size: 80%;">Two parameters</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs013.html#___sec12" style="font-size: 80%;">Maximum likelihood</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs014.html#___sec13" style="font-size: 80%;">The cost function rewritten</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs015.html#___sec14" style="font-size: 80%;">Minimizing the cross entropy</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs016.html#___sec15" style="font-size: 80%;">A more compact expression</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs017.html#___sec16" style="font-size: 80%;">Extending to more predictors</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs018.html#___sec17" style="font-size: 80%;">Including more classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs019.html#___sec18" style="font-size: 80%;">More classes</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs020.html#___sec19" style="font-size: 80%;">A simple classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs021.html#___sec20" style="font-size: 80%;">Cancer Data again now with Decision Trees and other Methods</a></li>
<!-- navigation toc: --> <li><a href="._week38-bs022.html#___sec21" style="font-size: 80%;">Other measures in classification studies: Cancer Data again</a></li>
</ul>
</li>
@@ -186,7 +190,7 @@ MathJax.Hub.Config({
<li><a href="._week38-bs008.html">9</a></li>
<li><a href="._week38-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week38-bs020.html">21</a></li>
<li><a href="._week38-bs022.html">23</a></li>
<li><a href="._week38-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week38/html/week38-reveal.html
+32 -24
View File
@@ -159,19 +159,27 @@ MathJax.Hub.Config({
<section>
<h2 id="___sec0">To do for log reg </h2>
<h2 id="___sec0">Plans for week 38 </h2>
<ul>
<p><li> Develop code for log reg step by step, with link to gradient descent part</li>
<p><li> show how to read and set up design matrix</li>
<p><li> use breast cancer data as example</li>
<p><li> develop other classification examples, pulsar example</li>
<p><li> Thursday: Summary of regression methods and discussion of project 1. We revisit also cross-validation and bootstrap as resampling techniques with examples</li>
<p><li> Friday: Logistic Regression</li>
</ul>
</section>
<section>
<h2 id="___sec1">Logistic Regression </h2>
<h2 id="___sec1">Thursday: </h2>
</section>
<section>
<h2 id="___sec2">Friday: Intro to Logistic Regression </h2>
</section>
<section>
<h2 id="___sec3">Logistic Regression </h2>
<p>
In linear regression our main interest was centered on learning the
@@ -189,7 +197,7 @@ simple recipe for fitting our data.
<section>
<h2 id="___sec2">Classification problems </h2>
<h2 id="___sec4">Classification problems </h2>
<p>
Classification problems, however, are concerned with outcomes taking
@@ -212,7 +220,7 @@ failure etc.
<section>
<h2 id="___sec3">Optimization and Deep learning </h2>
<h2 id="___sec5">Optimization and Deep learning </h2>
<p>
Logistic regression will also serve as our stepping stone towards
@@ -233,7 +241,7 @@ models, as we will see later.
<section>
<h2 id="___sec4">Basics </h2>
<h2 id="___sec6">Basics </h2>
<p>
We consider the case where the dependent variables, also called the
@@ -262,7 +270,7 @@ $$
<section>
<h2 id="___sec5">Linear classifier </h2>
<h2 id="___sec7">Linear classifier </h2>
<p>
Before moving to the logistic model, let us try to use our linear
@@ -288,7 +296,7 @@ where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \
<section>
<h2 id="___sec6">Some selected properties </h2>
<h2 id="___sec8">Some selected properties </h2>
<p>
The main problem with our function is that it takes values on the
@@ -310,7 +318,7 @@ the probability of a given category. This leads us to the logistic function.
<section>
<h2 id="___sec7">The logistic function </h2>
<h2 id="___sec9">The logistic function </h2>
<p>
The perceptron is an example of a ``hard classification&quot; model. We
@@ -334,7 +342,7 @@ Note that \( 1-p(t)= p(-t) \).
<section>
<h2 id="___sec8">Examples of likelihood functions used in logistic regression and nueral networks </h2>
<h2 id="___sec10">Examples of likelihood functions used in logistic regression and nueral networks </h2>
<p>
The following code plots the logistic function, the step function and other functions we will encounter from here and on.
@@ -401,7 +409,7 @@ plt.show()
<section>
<h2 id="___sec9">Two parameters </h2>
<h2 id="___sec11">Two parameters </h2>
<p>
We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
@@ -427,7 +435,7 @@ $$
<section>
<h2 id="___sec10">Maximum likelihood </h2>
<h2 id="___sec12">Maximum likelihood </h2>
<p>
In order to define the total likelihood for all possible outcomes from a
@@ -454,7 +462,7 @@ $$
<section>
<h2 id="___sec11">The cost function rewritten </h2>
<h2 id="___sec13">The cost function rewritten </h2>
<p>
Reordering the logarithms, we can rewrite the <b>cost/loss</b> function as
@@ -479,7 +487,7 @@ in practice we often supplement the cross-entropy with additional regularization
<section>
<h2 id="___sec12">Minimizing the cross entropy </h2>
<h2 id="___sec14">Minimizing the cross entropy </h2>
<p>
The cross entropy is a convex function of the weights \( \hat{\beta} \) and,
@@ -505,7 +513,7 @@ $$
<section>
<h2 id="___sec13">A more compact expression </h2>
<h2 id="___sec15">A more compact expression </h2>
<p>
Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an
@@ -532,7 +540,7 @@ $$
<section>
<h2 id="___sec14">Extending to more predictors </h2>
<h2 id="___sec16">Extending to more predictors </h2>
<p>
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors
@@ -552,7 +560,7 @@ $$
<section>
<h2 id="___sec15">Including more classes </h2>
<h2 id="___sec17">Including more classes </h2>
<p>
Till now we have mainly focused on two classes, the so-called binary
@@ -586,7 +594,7 @@ and the model is specified in term of \( K-1 \) so-called log-odds or
<section>
<h2 id="___sec16">More classes </h2>
<h2 id="___sec18">More classes </h2>
<p>
In our discussion of neural networks we will encounter the above again
@@ -630,7 +638,7 @@ methods</a>.
<section>
<h2 id="___sec17">A simple classification problem </h2>
<h2 id="___sec19">A simple classification problem </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -683,7 +691,7 @@ methods</a>.
<section>
<h2 id="___sec18">Cancer Data again now with Decision Trees and other Methods </h2>
<h2 id="___sec20">Cancer Data again now with Decision Trees and other Methods </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -717,7 +725,7 @@ logreg.fit(X_train_scaled, y_train)
<section>
<h2 id="___sec19">Other measures in classification studies: Cancer Data again </h2>
<h2 id="___sec21">Other measures in classification studies: Cancer Data again </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
doc/pub/week38/html/week38-solarized.html
+54 -44
View File
@@ -35,36 +35,38 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -111,18 +113,26 @@ MathJax.Hub.Config({
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">To do for log reg </h2>
<h2 id="___sec0">Plans for week 38 </h2>
<ul>
<li> Develop code for log reg step by step, with link to gradient descent part</li>
<li> show how to read and set up design matrix</li>
<li> use breast cancer data as example</li>
<li> develop other classification examples, pulsar example</li>
<li> Thursday: Summary of regression methods and discussion of project 1. We revisit also cross-validation and bootstrap as resampling techniques with examples</li>
<li> Friday: Logistic Regression</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Thursday: </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Friday: Intro to Logistic Regression </h2>
<p>
<!-- !split -->
<h2 id="___sec1">Logistic Regression </h2>
<h2 id="___sec3">Logistic Regression </h2>
<p>
In linear regression our main interest was centered on learning the
@@ -140,7 +150,7 @@ simple recipe for fitting our data.
<p>
<!-- !split -->
<h2 id="___sec2">Classification problems </h2>
<h2 id="___sec4">Classification problems </h2>
<p>
Classification problems, however, are concerned with outcomes taking
@@ -163,7 +173,7 @@ failure etc.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Optimization and Deep learning </h2>
<h2 id="___sec5">Optimization and Deep learning </h2>
<p>
Logistic regression will also serve as our stepping stone towards
@@ -184,7 +194,7 @@ models, as we will see later.
<p>
<!-- !split -->
<h2 id="___sec4">Basics </h2>
<h2 id="___sec6">Basics </h2>
<p>
We consider the case where the dependent variables, also called the
@@ -211,7 +221,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Linear classifier </h2>
<h2 id="___sec7">Linear classifier </h2>
<p>
Before moving to the logistic model, let us try to use our linear
@@ -235,7 +245,7 @@ where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Some selected properties </h2>
<h2 id="___sec8">Some selected properties </h2>
<p>
The main problem with our function is that it takes values on the
@@ -257,7 +267,7 @@ the probability of a given category. This leads us to the logistic function.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">The logistic function </h2>
<h2 id="___sec9">The logistic function </h2>
<p>
The perceptron is an example of a ``hard classification&quot; model. We
@@ -279,7 +289,7 @@ Note that \( 1-p(t)= p(-t) \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Examples of likelihood functions used in logistic regression and nueral networks </h2>
<h2 id="___sec10">Examples of likelihood functions used in logistic regression and nueral networks </h2>
<p>
The following code plots the logistic function, the step function and other functions we will encounter from here and on.
@@ -345,7 +355,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Two parameters </h2>
<h2 id="___sec11">Two parameters </h2>
<p>
We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
@@ -367,7 +377,7 @@ $$
<p>
<!-- !split -->
<h2 id="___sec10">Maximum likelihood </h2>
<h2 id="___sec12">Maximum likelihood </h2>
<p>
In order to define the total likelihood for all possible outcomes from a
@@ -390,7 +400,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">The cost function rewritten </h2>
<h2 id="___sec13">The cost function rewritten </h2>
<p>
Reordering the logarithms, we can rewrite the <b>cost/loss</b> function as
@@ -411,7 +421,7 @@ in practice we often supplement the cross-entropy with additional regularization
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Minimizing the cross entropy </h2>
<h2 id="___sec14">Minimizing the cross entropy </h2>
<p>
The cross entropy is a convex function of the weights \( \hat{\beta} \) and,
@@ -433,7 +443,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">A more compact expression </h2>
<h2 id="___sec15">A more compact expression </h2>
<p>
Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an
@@ -456,7 +466,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">Extending to more predictors </h2>
<h2 id="___sec16">Extending to more predictors </h2>
<p>
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors
@@ -472,7 +482,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Including more classes </h2>
<h2 id="___sec17">Including more classes </h2>
<p>
Till now we have mainly focused on two classes, the so-called binary
@@ -500,7 +510,7 @@ and the model is specified in term of \( K-1 \) so-called log-odds or
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">More classes </h2>
<h2 id="___sec18">More classes </h2>
<p>
In our discussion of neural networks we will encounter the above again
@@ -540,7 +550,7 @@ methods</a>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">A simple classification problem </h2>
<h2 id="___sec19">A simple classification problem </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -592,7 +602,7 @@ methods</a>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec18">Cancer Data again now with Decision Trees and other Methods </h2>
<h2 id="___sec20">Cancer Data again now with Decision Trees and other Methods </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -625,7 +635,7 @@ logreg.fit(X_train_scaled, y_train)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Other measures in classification studies: Cancer Data again </h2>
<h2 id="___sec21">Other measures in classification studies: Cancer Data again </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
doc/pub/week38/html/week38.html
+54 -44
View File
@@ -40,36 +40,38 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('To do for log reg', 2, None, '___sec0'),
('Logistic Regression', 2, None, '___sec1'),
('Classification problems', 2, None, '___sec2'),
('Optimization and Deep learning', 2, None, '___sec3'),
('Basics', 2, None, '___sec4'),
('Linear classifier', 2, None, '___sec5'),
('Some selected properties', 2, None, '___sec6'),
('The logistic function', 2, None, '___sec7'),
'sections': [('Plans for week 38', 2, None, '___sec0'),
('Thursday:', 2, None, '___sec1'),
('Friday: Intro to Logistic Regression', 2, None, '___sec2'),
('Logistic Regression', 2, None, '___sec3'),
('Classification problems', 2, None, '___sec4'),
('Optimization and Deep learning', 2, None, '___sec5'),
('Basics', 2, None, '___sec6'),
('Linear classifier', 2, None, '___sec7'),
('Some selected properties', 2, None, '___sec8'),
('The logistic function', 2, None, '___sec9'),
('Examples of likelihood functions used in logistic regression '
'and nueral networks',
2,
None,
'___sec8'),
('Two parameters', 2, None, '___sec9'),
('Maximum likelihood', 2, None, '___sec10'),
('The cost function rewritten', 2, None, '___sec11'),
('Minimizing the cross entropy', 2, None, '___sec12'),
('A more compact expression', 2, None, '___sec13'),
('Extending to more predictors', 2, None, '___sec14'),
('Including more classes', 2, None, '___sec15'),
('More classes', 2, None, '___sec16'),
('A simple classification problem', 2, None, '___sec17'),
'___sec10'),
('Two parameters', 2, None, '___sec11'),
('Maximum likelihood', 2, None, '___sec12'),
('The cost function rewritten', 2, None, '___sec13'),
('Minimizing the cross entropy', 2, None, '___sec14'),
('A more compact expression', 2, None, '___sec15'),
('Extending to more predictors', 2, None, '___sec16'),
('Including more classes', 2, None, '___sec17'),
('More classes', 2, None, '___sec18'),
('A simple classification problem', 2, None, '___sec19'),
('Cancer Data again now with Decision Trees and other Methods',
2,
None,
'___sec18'),
'___sec20'),
('Other measures in classification studies: Cancer Data again',
2,
None,
'___sec19')]}
'___sec21')]}
end of tocinfo -->
<body>
@@ -116,18 +118,26 @@ MathJax.Hub.Config({
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">To do for log reg </h2>
<h2 id="___sec0">Plans for week 38 </h2>
<ul>
<li> Develop code for log reg step by step, with link to gradient descent part</li>
<li> show how to read and set up design matrix</li>
<li> use breast cancer data as example</li>
<li> develop other classification examples, pulsar example</li>
<li> Thursday: Summary of regression methods and discussion of project 1. We revisit also cross-validation and bootstrap as resampling techniques with examples</li>
<li> Friday: Logistic Regression</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Thursday: </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Friday: Intro to Logistic Regression </h2>
<p>
<!-- !split -->
<h2 id="___sec1">Logistic Regression </h2>
<h2 id="___sec3">Logistic Regression </h2>
<p>
In linear regression our main interest was centered on learning the
@@ -145,7 +155,7 @@ simple recipe for fitting our data.
<p>
<!-- !split -->
<h2 id="___sec2">Classification problems </h2>
<h2 id="___sec4">Classification problems </h2>
<p>
Classification problems, however, are concerned with outcomes taking
@@ -168,7 +178,7 @@ failure etc.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Optimization and Deep learning </h2>
<h2 id="___sec5">Optimization and Deep learning </h2>
<p>
Logistic regression will also serve as our stepping stone towards
@@ -189,7 +199,7 @@ models, as we will see later.
<p>
<!-- !split -->
<h2 id="___sec4">Basics </h2>
<h2 id="___sec6">Basics </h2>
<p>
We consider the case where the dependent variables, also called the
@@ -216,7 +226,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Linear classifier </h2>
<h2 id="___sec7">Linear classifier </h2>
<p>
Before moving to the logistic model, let us try to use our linear
@@ -240,7 +250,7 @@ where \( \hat{y} \) is a vector representing the possible outcomes, \( \hat{X} \
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Some selected properties </h2>
<h2 id="___sec8">Some selected properties </h2>
<p>
The main problem with our function is that it takes values on the
@@ -262,7 +272,7 @@ the probability of a given category. This leads us to the logistic function.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">The logistic function </h2>
<h2 id="___sec9">The logistic function </h2>
<p>
The perceptron is an example of a ``hard classification&quot; model. We
@@ -284,7 +294,7 @@ Note that \( 1-p(t)= p(-t) \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Examples of likelihood functions used in logistic regression and nueral networks </h2>
<h2 id="___sec10">Examples of likelihood functions used in logistic regression and nueral networks </h2>
<p>
The following code plots the logistic function, the step function and other functions we will encounter from here and on.
@@ -350,7 +360,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Two parameters </h2>
<h2 id="___sec11">Two parameters </h2>
<p>
We assume now that we have two classes with \( y_i \) either \( 0 \) or \( 1 \). Furthermore we assume also that we have only two parameters \( \beta \) in our fitting of the Sigmoid function, that is we define probabilities
@@ -372,7 +382,7 @@ $$
<p>
<!-- !split -->
<h2 id="___sec10">Maximum likelihood </h2>
<h2 id="___sec12">Maximum likelihood </h2>
<p>
In order to define the total likelihood for all possible outcomes from a
@@ -395,7 +405,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">The cost function rewritten </h2>
<h2 id="___sec13">The cost function rewritten </h2>
<p>
Reordering the logarithms, we can rewrite the <b>cost/loss</b> function as
@@ -416,7 +426,7 @@ in practice we often supplement the cross-entropy with additional regularization
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Minimizing the cross entropy </h2>
<h2 id="___sec14">Minimizing the cross entropy </h2>
<p>
The cross entropy is a convex function of the weights \( \hat{\beta} \) and,
@@ -438,7 +448,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">A more compact expression </h2>
<h2 id="___sec15">A more compact expression </h2>
<p>
Let us now define a vector \( \hat{y} \) with \( n \) elements \( y_i \), an
@@ -461,7 +471,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">Extending to more predictors </h2>
<h2 id="___sec16">Extending to more predictors </h2>
<p>
Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with \( p \) predictors
@@ -477,7 +487,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Including more classes </h2>
<h2 id="___sec17">Including more classes </h2>
<p>
Till now we have mainly focused on two classes, the so-called binary
@@ -505,7 +515,7 @@ and the model is specified in term of \( K-1 \) so-called log-odds or
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">More classes </h2>
<h2 id="___sec18">More classes </h2>
<p>
In our discussion of neural networks we will encounter the above again
@@ -545,7 +555,7 @@ methods</a>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">A simple classification problem </h2>
<h2 id="___sec19">A simple classification problem </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -597,7 +607,7 @@ methods</a>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec18">Cancer Data again now with Decision Trees and other Methods </h2>
<h2 id="___sec20">Cancer Data again now with Decision Trees and other Methods </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -630,7 +640,7 @@ logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Other measures in classification studies: Cancer Data again </h2>
<h2 id="___sec21">Other measures in classification studies: Cancer Data again </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
doc/pub/week38/ipynb/ipynb-week38-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week38/ipynb/week38.ipynb
+6 -5
View File
@@ -17,15 +17,16 @@
"\n",
"\n",
"\n",
"## To do for log reg\n",
"## Plans for week 38\n",
"\n",
"* Develop code for log reg step by step, with link to gradient descent part\n",
"* Thursday: Summary of regression methods and discussion of project 1. We revisit also cross-validation and bootstrap as resampling techniques with examples\n",
"\n",
"* show how to read and set up design matrix\n",
"* Friday: Logistic Regression\n",
"\n",
"* use breast cancer data as example\n",
"## Thursday:\n",
"\n",
"## Friday: Intro to Logistic Regression\n",
"\n",
"* develop other classification examples, pulsar example\n",
"\n",
"<!-- !split -->\n",
"## Logistic Regression\n",