lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on files

Browse Source
This commit is contained in:
mhjensen
2020-09-16 18:50:58 +02:00
parent 4a35f83493
commit 99b3e89e74
35 changed files with 14538 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/Projects/2020/Project1/html/._Project1-bs000.html
+4 -6
View File
@@ -165,7 +165,7 @@ MathJax.Hub.Config({
<center><b>Department of Physics, University of Oslo, Norway</b></center>
<br>
<p>
<center><h4>Sep 7, 2020</h4></center> <!-- date -->
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<br>
<p>
</div> <!-- end jumbotron -->
@@ -404,18 +404,16 @@ Note also that when you calculate the bias, in all applications you don't know t
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
data.
<p>
Implement the \( k \)-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
from the test folds. You can compare your own code with that from
<b>Scikit-Learn</b> if needed.
<p>
Compare the MSE you get from your cross-validation code with the one you got from your <b>bootstrap</b> code.
Compare the MSE you get from your cross-validation code with the one you got from your <b>bootstrap</b> code. Comment your results. Try \( 5-10 \) folds.
You can also compare your own cross-validation code with the one provided by <b>Scikit-Learn</b>.
<h3 id="___sec4" class="anchor">Part d): Ridge Regression on the Franke function with resampling </h3>
doc/Projects/2020/Project1/html/Project1-bs.html
+4 -6
View File
@@ -165,7 +165,7 @@ MathJax.Hub.Config({
<center><b>Department of Physics, University of Oslo, Norway</b></center>
<br>
<p>
<center><h4>Sep 7, 2020</h4></center> <!-- date -->
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<br>
<p>
</div> <!-- end jumbotron -->
@@ -404,18 +404,16 @@ Note also that when you calculate the bias, in all applications you don't know t
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
data.
<p>
Implement the \( k \)-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
from the test folds. You can compare your own code with that from
<b>Scikit-Learn</b> if needed.
<p>
Compare the MSE you get from your cross-validation code with the one you got from your <b>bootstrap</b> code.
Compare the MSE you get from your cross-validation code with the one you got from your <b>bootstrap</b> code. Comment your results. Try \( 5-10 \) folds.
You can also compare your own cross-validation code with the one provided by <b>Scikit-Learn</b>.
<h3 id="___sec4" class="anchor">Part d): Ridge Regression on the Franke function with resampling </h3>
doc/Projects/2020/Project1/html/Project1.html
+4 -6
View File
@@ -122,7 +122,7 @@ MathJax.Hub.Config({
<center><b>Department of Physics, University of Oslo, Norway</b></center>
<br>
<p>
<center><h4>Sep 7, 2020</h4></center> <!-- date -->
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<br>
<h2 id="___sec0">Regression analysis and resampling methods </h2>
@@ -359,18 +359,16 @@ Note also that when you calculate the bias, in all applications you don't know t
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
data.
<p>
Implement the \( k \)-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
from the test folds. You can compare your own code with that from
<b>Scikit-Learn</b> if needed.
<p>
Compare the MSE you get from your cross-validation code with the one you got from your <b>bootstrap</b> code.
Compare the MSE you get from your cross-validation code with the one you got from your <b>bootstrap</b> code. Comment your results. Try \( 5-10 \) folds.
You can also compare your own cross-validation code with the one provided by <b>Scikit-Learn</b>.
<h3 id="___sec4">Part d): Ridge Regression on the Franke function with resampling </h3>
doc/Projects/2020/Project1/ipynb/Project1.ipynb
+5 -9
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**[Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html)**, Department of Physics, University of Oslo, Norway\n",
"\n",
"Date: **Sep 7, 2020**\n",
"Date: **Sep 16, 2020**\n",
"\n",
"Copyright 1999-2020, [Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html). Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -318,19 +318,15 @@
"The aim here is to write your own code for another widely popular\n",
"resampling technique, the so-called cross-validation method. Again,\n",
"before you start with cross-validation approach, you should scale your\n",
"data and split it in test and training data as you did earlier.\n",
"Perform a resampling of the data where you split the data in training\n",
"data and test data using for example\n",
"\n",
"\n",
"data.\n",
"\n",
"Implement the $k$-fold cross-validation algorithm (write your own\n",
"code) and evaluate again the MSE function resulting\n",
"from the test data. You can compare your own code with that from\n",
"from the test folds. You can compare your own code with that from\n",
"**Scikit-Learn** if needed. \n",
"\n",
"Compare the MSE you get from your cross-validation code with the one you got from your **bootstrap** code.\n",
"You can also compare your own cross-validation code with the one provided by **Scikit-Learn**.\n",
"Compare the MSE you get from your cross-validation code with the one you got from your **bootstrap** code. Comment your results. Try $5-10$ folds. \n",
"You can also compare your own cross-validation code with the one provided by **Scikit-Learn**. \n",
"\n",
"\n",
"### Part d): Ridge Regression on the Franke function with resampling\n",
doc/Projects/2020/Project1/ipynb/ipynb-Project1-src.tar.gz
BIN
View File
Binary file not shown.
doc/Projects/2020/Project1/pdf/Project1.aux
+25
View File
@@ -0,0 +1,25 @@
\relax
\providecommand\hyper@newdestlabel[2]{}
\providecommand\HyperFirstAtBeginDocument{\AtBeginDocument}
\HyperFirstAtBeginDocument{\ifx\hyper@anchor\@undefined
\global\let\oldcontentsline\contentsline
\gdef\contentsline#1#2#3#4{\oldcontentsline{#1}{#2}{#3}}
\global\let\oldnewlabel\newlabel
\gdef\newlabel#1#2{\newlabelxx{#1}#2}
\gdef\newlabelxx#1#2#3#4#5#6{\oldnewlabel{#1}{{#2}{#3}}}
\AtEndDocument{\ifx\hyper@anchor\@undefined
\let\contentsline\oldcontentsline
\let\newlabel\oldnewlabel
\fi}
\fi}
\global\let\hyper@last\relax
\gdef\HyperFirstAtBeginDocument#1{#1}
\providecommand\HyField@AuxAddToFields[1]{}
\providecommand\HyField@AuxAddToCoFields[2]{}
\@writefile{toc}{\contentsline {paragraph}{Part a): Ordinary Least Square (OLS) on the Franke function.}{3}{section*.2}}
\@writefile{toc}{\contentsline {paragraph}{Part b): Bias-variance trade-off and resamplng techniques.}{4}{section*.3}}
\@writefile{toc}{\contentsline {paragraph}{Part c) Cross-validation as resampling techniques, adding more complexity.}{5}{section*.4}}
\@writefile{toc}{\contentsline {paragraph}{Part d): Ridge Regression on the Franke function with resampling.}{5}{section*.5}}
\@writefile{toc}{\contentsline {paragraph}{Part e): Lasso Regression on the Franke function with resampling.}{5}{section*.6}}
\@writefile{toc}{\contentsline {paragraph}{Part f): Introducing real data and preparing the data analysis.}{5}{section*.7}}
\@writefile{toc}{\contentsline {paragraph}{Part g) OLS, Ridge and Lasso regression with resampling.}{6}{section*.8}}
doc/Projects/2020/Project1/pdf/Project1.idx
View File
doc/Projects/2020/Project1/pdf/Project1.log
+851
View File
@@ -0,0 +1,851 @@
This is pdfTeX, Version 3.14159265-2.6-1.40.16 (TeX Live 2015/Debian) (preloaded format=pdflatex 2018.10.20) 16 SEP 2020 17:14
entering extended mode
restricted \write18 enabled.
%&-line parsing enabled.
**Project1
(./Project1.tex
LaTeX2e <2016/02/01>
Babel <3.9q> and hyphenation patterns for 81 language(s) loaded.
(/usr/share/texlive/texmf-dist/tex/latex/base/article.cls
Document Class: article 2014/09/29 v1.4h Standard LaTeX document class
(/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo
File: size10.clo 2014/09/29 v1.4h Standard LaTeX file (size option)
)
\c@part=\count79
\c@section=\count80
\c@subsection=\count81
\c@subsubsection=\count82
\c@paragraph=\count83
\c@subparagraph=\count84
\c@figure=\count85
\c@table=\count86
\abovecaptionskip=\skip41
\belowcaptionskip=\skip42
\bibindent=\dimen102
)
(/usr/share/texlive/texmf-dist/tex/latex/relsize/relsize.sty
Package: relsize 2013/03/29 ver 4.1
)
(/usr/share/texlive/texmf-dist/tex/latex/base/makeidx.sty
Package: makeidx 2014/09/29 v1.0m Standard LaTeX package
)
(/usr/share/texlive/texmf-dist/tex/latex/graphics/color.sty
Package: color 2016/01/03 v1.1b Standard LaTeX Color (DPC)
(/usr/share/texlive/texmf-dist/tex/latex/latexconfig/color.cfg
File: color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive
)
Package color Info: Driver file: pdftex.def on input line 143.
(/usr/share/texlive/texmf-dist/tex/latex/pdftex-def/pdftex.def
File: pdftex.def 2011/05/27 v0.06d Graphics/color for pdfTeX
(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/infwarerr.sty
Package: infwarerr 2010/04/08 v1.3 Providing info/warning/error messages (HO)
)
(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/ltxcmds.sty
Package: ltxcmds 2011/11/09 v1.22 LaTeX kernel commands for general use (HO)
)
\Gread@gobject=\count87
))
(/usr/share/texlive/texmf-dist/tex/latex/setspace/setspace.sty
Package: setspace 2011/12/19 v6.7a set line spacing
)
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsmath.sty
Package: amsmath 2016/03/03 v2.15a AMS math features
\@mathmargin=\skip43
For additional information on amsmath, use the `?' option.
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amstext.sty
Package: amstext 2000/06/29 v2.01 AMS text
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsgen.sty
File: amsgen.sty 1999/11/30 v2.0 generic functions
\@emptytoks=\toks14
\ex@=\dimen103
))
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsbsy.sty
Package: amsbsy 1999/11/29 v1.2d Bold Symbols
\pmbraise@=\dimen104
)
(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsopn.sty
Package: amsopn 1999/12/14 v2.01 operator names
)
\inf@bad=\count88
LaTeX Info: Redefining \frac on input line 199.
\uproot@=\count89
\leftroot@=\count90
LaTeX Info: Redefining \overline on input line 297.
\classnum@=\count91
\DOTSCASE@=\count92
LaTeX Info: Redefining \ldots on input line 394.
LaTeX Info: Redefining \dots on input line 397.
LaTeX Info: Redefining \cdots on input line 518.
\Mathstrutbox@=\box26
\strutbox@=\box27
\big@size=\dimen105
LaTeX Font Info: Redeclaring font encoding OML on input line 630.
LaTeX Font Info: Redeclaring font encoding OMS on input line 631.
\macc@depth=\count93
\c@MaxMatrixCols=\count94
\dotsspace@=\muskip10
\c@parentequation=\count95
\dspbrk@lvl=\count96
\tag@help=\toks15
\row@=\count97
\column@=\count98
\maxfields@=\count99
\andhelp@=\toks16
\eqnshift@=\dimen106
\alignsep@=\dimen107
\tagshift@=\dimen108
\tagwidth@=\dimen109
\totwidth@=\dimen110
\lineht@=\dimen111
\@envbody=\toks17
\multlinegap=\skip44
\multlinetaggap=\skip45
\mathdisplay@stack=\toks18
LaTeX Info: Redefining \[ on input line 2735.
LaTeX Info: Redefining \] on input line 2736.
)
(/usr/share/texlive/texmf-dist/tex/latex/amsfonts/amsfonts.sty
Package: amsfonts 2013/01/14 v3.01 Basic AMSFonts support
\symAMSa=\mathgroup4
\symAMSb=\mathgroup5
LaTeX Font Info: Overwriting math alphabet `\mathfrak' in version `bold'
(Font) U/euf/m/n --> U/euf/b/n on input line 106.
)
(/usr/share/texlive/texmf-dist/tex/latex/amsfonts/amssymb.sty
Package: amssymb 2013/01/14 v3.01 AMS font symbols
)
(/usr/share/texlive/texmf-dist/tex/latex/xcolor/xcolor.sty
Package: xcolor 2007/01/21 v2.11 LaTeX color extensions (UK)
(/usr/share/texlive/texmf-dist/tex/latex/latexconfig/color.cfg
File: color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive
)
Package xcolor Info: Driver file: pdftex.def on input line 225.
(/usr/share/texlive/texmf-dist/tex/latex/colortbl/colortbl.sty
Package: colortbl 2012/02/13 v1.0a Color table columns (DPC)
(/usr/share/texlive/texmf-dist/tex/latex/tools/array.sty
Package: array 2014/10/28 v2.4c Tabular extension package (FMi)
\col@sep=\dimen112
\extrarowheight=\dimen113
\NC@list=\toks19
\extratabsurround=\skip46
\backup@length=\skip47
)
\everycr=\toks20
\minrowclearance=\skip48
)
LaTeX Info: Redefining \color on input line 702.
\rownum=\count100
Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1337.
Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1341.
Package xcolor Info: Model `RGB' extended on input line 1353.
Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1355.
Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1356.
Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1357.
Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1358.
Package xcolor Info: Model `Gray' substituted by `gray' on input line 1359.
Package xcolor Info: Model `wave' substituted by `hsb' on input line 1360.
)
(/usr/share/texlive/texmf-dist/tex/latex/tools/bm.sty
Package: bm 2014/10/28 v1.1c Bold Symbol Support (DPC/FMi)
\symboldoperators=\mathgroup6
\symboldletters=\mathgroup7
\symboldsymbols=\mathgroup8
LaTeX Font Info: Redeclaring math alphabet \mathbf on input line 141.
LaTeX Info: Redefining \bm on input line 207.
)
(/usr/share/texlive/texmf-dist/tex/latex/ltablex/ltablex.sty
Package: ltablex 2014/08/13 v1.1 Modified tabularx
(/usr/share/texlive/texmf-dist/tex/latex/tools/longtable.sty
Package: longtable 2014/10/28 v4.11 Multi-page Table package (DPC)
\LTleft=\skip49
\LTright=\skip50
\LTpre=\skip51
\LTpost=\skip52
\LTchunksize=\count101
\LTcapwidth=\dimen114
\LT@head=\box28
\LT@firsthead=\box29
\LT@foot=\box30
\LT@lastfoot=\box31
\LT@cols=\count102
\LT@rows=\count103
\c@LT@tables=\count104
\c@LT@chunks=\count105
\LT@p@ftn=\toks21
)
(/usr/share/texlive/texmf-dist/tex/latex/tools/tabularx.sty
Package: tabularx 2014/10/28 v2.10 `tabularx' package (DPC)
\TX@col@width=\dimen115
\TX@old@table=\dimen116
\TX@old@col=\dimen117
\TX@target=\dimen118
\TX@delta=\dimen119
\TX@cols=\count106
\TX@ftn=\toks22
))
(/usr/share/texlive/texmf-dist/tex/latex/microtype/microtype.sty
Package: microtype 2013/05/23 v2.5a Micro-typographical refinements (RS)
(/usr/share/texlive/texmf-dist/tex/latex/graphics/keyval.sty
Package: keyval 2014/10/28 v1.15 key=value parser (DPC)
\KV@toks@=\toks23
)
\MT@toks=\toks24
\MT@count=\count107
LaTeX Info: Redefining \textls on input line 766.
\MT@outer@kern=\dimen120
LaTeX Info: Redefining \textmicrotypecontext on input line 1285.
\MT@listname@count=\count108
(/usr/share/texlive/texmf-dist/tex/latex/microtype/microtype-pdftex.def
File: microtype-pdftex.def 2013/05/23 v2.5a Definitions specific to pdftex (RS)
LaTeX Info: Redefining \lsstyle on input line 915.
LaTeX Info: Redefining \lslig on input line 915.
\MT@outer@space=\skip53
)
Package microtype Info: Loading configuration file microtype.cfg.
(/usr/share/texlive/texmf-dist/tex/latex/microtype/microtype.cfg
File: microtype.cfg 2013/05/23 v2.5a microtype main configuration file (RS)
))
(/usr/share/texlive/texmf-dist/tex/latex/graphics/graphicx.sty
Package: graphicx 2014/10/28 v1.0g Enhanced LaTeX Graphics (DPC,SPQR)
(/usr/share/texlive/texmf-dist/tex/latex/graphics/graphics.sty
Package: graphics 2016/01/03 v1.0q Standard LaTeX Graphics (DPC,SPQR)
(/usr/share/texlive/texmf-dist/tex/latex/graphics/trig.sty
Package: trig 2016/01/03 v1.10 sin cos tan (DPC)
)
(/usr/share/texlive/texmf-dist/tex/latex/latexconfig/graphics.cfg
File: graphics.cfg 2010/04/23 v1.9 graphics configuration of TeX Live
)
Package graphics Info: Driver file: pdftex.def on input line 95.
)
\Gin@req@height=\dimen121
\Gin@req@width=\dimen122
)
(/usr/share/texlive/texmf-dist/tex/latex/fancyvrb/fancyvrb.sty
Package: fancyvrb 2008/02/07
Style option: `fancyvrb' v2.7a, with DG/SPQR fixes, and firstline=lastline fix
<2008/02/07> (tvz)
\FV@CodeLineNo=\count109
\FV@InFile=\read1
\FV@TabBox=\box32
\c@FancyVerbLine=\count110
\FV@StepNumber=\count111
\FV@OutFile=\write3
) (/usr/share/texlive/texmf-dist/tex/latex/base/fontenc.sty
Package: fontenc 2005/09/27 v1.99g Standard LaTeX package
(/usr/share/texlive/texmf-dist/tex/latex/base/t1enc.def
File: t1enc.def 2005/09/27 v1.99g Standard LaTeX file
LaTeX Font Info: Redeclaring font encoding T1 on input line 48.
))
(/usr/share/texlive/texmf-dist/tex/latex/ucs/ucs.sty
Package: ucs 2013/05/11 v2.2 UCS: Unicode input support
(/usr/share/texlive/texmf-dist/tex/latex/ucs/data/uni-global.def
File: uni-global.def 2013/05/13 UCS: Unicode global data
)
\uc@secondtry=\count112
\uc@combtoks=\toks25
\uc@combtoksb=\toks26
\uc@temptokena=\toks27
)
(/usr/share/texlive/texmf-dist/tex/latex/base/inputenc.sty
Package: inputenc 2015/03/17 v1.2c Input encoding file
\inpenc@prehook=\toks28
\inpenc@posthook=\toks29
(/usr/share/texlive/texmf-dist/tex/latex/ucs/utf8x.def
File: utf8x.def 2004/10/17 UCS: Input encoding UTF-8
))
(/usr/share/texmf/tex/latex/lm/lmodern.sty
Package: lmodern 2009/10/30 v1.6 Latin Modern Fonts
LaTeX Font Info: Overwriting symbol font `operators' in version `normal'
(Font) OT1/cmr/m/n --> OT1/lmr/m/n on input line 22.
LaTeX Font Info: Overwriting symbol font `letters' in version `normal'
(Font) OML/cmm/m/it --> OML/lmm/m/it on input line 23.
LaTeX Font Info: Overwriting symbol font `symbols' in version `normal'
(Font) OMS/cmsy/m/n --> OMS/lmsy/m/n on input line 24.
LaTeX Font Info: Overwriting symbol font `largesymbols' in version `normal'
(Font) OMX/cmex/m/n --> OMX/lmex/m/n on input line 25.
LaTeX Font Info: Overwriting symbol font `operators' in version `bold'
(Font) OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 26.
LaTeX Font Info: Overwriting symbol font `letters' in version `bold'
(Font) OML/cmm/b/it --> OML/lmm/b/it on input line 27.
LaTeX Font Info: Overwriting symbol font `symbols' in version `bold'
(Font) OMS/cmsy/b/n --> OMS/lmsy/b/n on input line 28.
LaTeX Font Info: Overwriting symbol font `largesymbols' in version `bold'
(Font) OMX/cmex/m/n --> OMX/lmex/m/n on input line 29.
LaTeX Font Info: Overwriting math alphabet `\mathbf' in version `normal'
(Font) OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 31.
LaTeX Font Info: Overwriting math alphabet `\mathsf' in version `normal'
(Font) OT1/cmss/m/n --> OT1/lmss/m/n on input line 32.
LaTeX Font Info: Overwriting math alphabet `\mathit' in version `normal'
(Font) OT1/cmr/m/it --> OT1/lmr/m/it on input line 33.
LaTeX Font Info: Overwriting math alphabet `\mathtt' in version `normal'
(Font) OT1/cmtt/m/n --> OT1/lmtt/m/n on input line 34.
LaTeX Font Info: Overwriting math alphabet `\mathbf' in version `bold'
(Font) OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 35.
LaTeX Font Info: Overwriting math alphabet `\mathsf' in version `bold'
(Font) OT1/cmss/bx/n --> OT1/lmss/bx/n on input line 36.
LaTeX Font Info: Overwriting math alphabet `\mathit' in version `bold'
(Font) OT1/cmr/bx/it --> OT1/lmr/bx/it on input line 37.
LaTeX Font Info: Overwriting math alphabet `\mathtt' in version `bold'
(Font) OT1/cmtt/m/n --> OT1/lmtt/m/n on input line 38.
)
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/hyperref.sty
Package: hyperref 2012/11/06 v6.83m Hypertext links for LaTeX
(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/hobsub-hyperref.sty
Package: hobsub-hyperref 2012/05/28 v1.13 Bundle oberdiek, subset hyperref (HO)
(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/hobsub-generic.sty
Package: hobsub-generic 2012/05/28 v1.13 Bundle oberdiek, subset generic (HO)
Package: hobsub 2012/05/28 v1.13 Construct package bundles (HO)
Package hobsub Info: Skipping package `infwarerr' (already loaded).
Package hobsub Info: Skipping package `ltxcmds' (already loaded).
Package: ifluatex 2010/03/01 v1.3 Provides the ifluatex switch (HO)
Package ifluatex Info: LuaTeX not detected.
Package: ifvtex 2010/03/01 v1.5 Detect VTeX and its facilities (HO)
Package ifvtex Info: VTeX not detected.
Package: intcalc 2007/09/27 v1.1 Expandable calculations with integers (HO)
Package: ifpdf 2011/01/30 v2.3 Provides the ifpdf switch (HO)
Package ifpdf Info: pdfTeX in PDF mode is detected.
Package: etexcmds 2011/02/16 v1.5 Avoid name clashes with e-TeX commands (HO)
Package etexcmds Info: Could not find \expanded.
(etexcmds) That can mean that you are not using pdfTeX 1.50 or
(etexcmds) that some package has redefined \expanded.
(etexcmds) In the latter case, load this package earlier.
Package: kvsetkeys 2012/04/25 v1.16 Key value parser (HO)
Package: kvdefinekeys 2011/04/07 v1.3 Define keys (HO)
Package: pdftexcmds 2011/11/29 v0.20 Utility functions of pdfTeX for LuaTeX (HO
)
Package pdftexcmds Info: LuaTeX not detected.
Package pdftexcmds Info: \pdf@primitive is available.
Package pdftexcmds Info: \pdf@ifprimitive is available.
Package pdftexcmds Info: \pdfdraftmode found.
Package: pdfescape 2011/11/25 v1.13 Implements pdfTeX's escape features (HO)
Package: bigintcalc 2012/04/08 v1.3 Expandable calculations on big integers (HO
)
Package: bitset 2011/01/30 v1.1 Handle bit-vector datatype (HO)
Package: uniquecounter 2011/01/30 v1.2 Provide unlimited unique counter (HO)
)
Package hobsub Info: Skipping package `hobsub' (already loaded).
Package: letltxmacro 2010/09/02 v1.4 Let assignment for LaTeX macros (HO)
Package: hopatch 2012/05/28 v1.2 Wrapper for package hooks (HO)
Package: xcolor-patch 2011/01/30 xcolor patch
Package: atveryend 2011/06/30 v1.8 Hooks at the very end of document (HO)
Package atveryend Info: \enddocument detected (standard20110627).
Package: atbegshi 2011/10/05 v1.16 At begin shipout hook (HO)
Package: refcount 2011/10/16 v3.4 Data extraction from label references (HO)
Package: hycolor 2011/01/30 v1.7 Color options for hyperref/bookmark (HO)
)
(/usr/share/texlive/texmf-dist/tex/generic/ifxetex/ifxetex.sty
Package: ifxetex 2010/09/12 v0.6 Provides ifxetex conditional
)
(/usr/share/texlive/texmf-dist/tex/latex/oberdiek/auxhook.sty
Package: auxhook 2011/03/04 v1.3 Hooks for auxiliary files (HO)
)
(/usr/share/texlive/texmf-dist/tex/latex/oberdiek/kvoptions.sty
Package: kvoptions 2011/06/30 v3.11 Key value format for package options (HO)
)
\@linkdim=\dimen123
\Hy@linkcounter=\count113
\Hy@pagecounter=\count114
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/pd1enc.def
File: pd1enc.def 2012/11/06 v6.83m Hyperref: PDFDocEncoding definition (HO)
)
\Hy@SavedSpaceFactor=\count115
(/usr/share/texlive/texmf-dist/tex/latex/latexconfig/hyperref.cfg
File: hyperref.cfg 2002/06/06 v1.2 hyperref configuration of TeXLive
)
Package hyperref Info: Option `final' set `true' on input line 4319.
Package hyperref Info: Hyper figures OFF on input line 4443.
Package hyperref Info: Link nesting OFF on input line 4448.
Package hyperref Info: Hyper index ON on input line 4451.
Package hyperref Info: Plain pages OFF on input line 4458.
Package hyperref Info: Backreferencing OFF on input line 4463.
Package hyperref Info: Implicit mode ON; LaTeX internals redefined.
Package hyperref Info: Bookmarks ON on input line 4688.
\c@Hy@tempcnt=\count116
(/usr/share/texlive/texmf-dist/tex/latex/url/url.sty
\Urlmuskip=\muskip11
Package: url 2013/09/16 ver 3.4 Verb mode for urls, etc.
)
LaTeX Info: Redefining \url on input line 5041.
\XeTeXLinkMargin=\dimen124
\Fld@menulength=\count117
\Field@Width=\dimen125
\Fld@charsize=\dimen126
Package hyperref Info: Hyper figures OFF on input line 6295.
Package hyperref Info: Link nesting OFF on input line 6300.
Package hyperref Info: Hyper index ON on input line 6303.
Package hyperref Info: backreferencing OFF on input line 6310.
Package hyperref Info: Link coloring OFF on input line 6315.
Package hyperref Info: Link coloring with OCG OFF on input line 6320.
Package hyperref Info: PDF/A mode OFF on input line 6325.
LaTeX Info: Redefining \ref on input line 6365.
LaTeX Info: Redefining \pageref on input line 6369.
\Hy@abspage=\count118
\c@Item=\count119
\c@Hfootnote=\count120
)
Package hyperref Message: Driver (autodetected): hpdftex.
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/hpdftex.def
File: hpdftex.def 2012/11/06 v6.83m Hyperref driver for pdfTeX
\Fld@listcount=\count121
\c@bookmark@seq@number=\count122
(/usr/share/texlive/texmf-dist/tex/latex/oberdiek/rerunfilecheck.sty
Package: rerunfilecheck 2011/04/15 v1.7 Rerun checks for auxiliary files (HO)
Package uniquecounter Info: New unique counter `rerunfilecheck' on input line 2
82.
)
\Hy@SectionHShift=\skip54
)
Package hyperref Info: Option `breaklinks' set `true' on input line 46.
Package hyperref Info: Option `colorlinks' set `true' on input line 46.
Package hyperref Info: Option `pdfmenubar' set `true' on input line 46.
Package hyperref Info: Option `pdftoolbar' set `true' on input line 46.
(/usr/share/texlive/texmf-dist/tex/latex/fancyhdr/fancyhdr.sty
\fancy@headwidth=\skip55
\f@ncyO@elh=\skip56
\f@ncyO@erh=\skip57
\f@ncyO@olh=\skip58
\f@ncyO@orh=\skip59
\f@ncyO@elf=\skip60
\f@ncyO@erf=\skip61
\f@ncyO@olf=\skip62
\f@ncyO@orf=\skip63
)
Package Fancyhdr Warning: \fancyfoot's `E' option without twoside option is use
less on input line 55.
\@indexfile=\write4
\openout4 = `Project1.idx'.
Writing index file Project1.idx
(/usr/share/texlive/texmf-dist/tex/latex/idxlayout/idxlayout.sty
Package: idxlayout 2012/03/30 v0.4d Configurable index layout
(/usr/share/texlive/texmf-dist/tex/latex/etoolbox/etoolbox.sty
Package: etoolbox 2015/08/02 v2.2a e-TeX tools for LaTeX (JAW)
\etb@tempcnta=\count123
)
(/usr/share/texlive/texmf-dist/tex/latex/tools/multicol.sty
Package: multicol 2015/08/19 v1.8n multicolumn formatting (FMi)
\c@tracingmulticols=\count124
\mult@box=\box33
\multicol@leftmargin=\dimen127
\c@unbalance=\count125
\c@collectmore=\count126
\doublecol@number=\count127
\multicoltolerance=\count128
\multicolpretolerance=\count129
\full@width=\dimen128
\page@free=\dimen129
\premulticols=\dimen130
\postmulticols=\dimen131
\multicolsep=\skip64
\multicolbaselineskip=\skip65
\partial@page=\box34
\last@line=\box35
\maxbalancingoverflow=\dimen132
\mult@rightbox=\box36
\mult@grightbox=\box37
\mult@gfirstbox=\box38
\mult@firstbox=\box39
\@tempa=\box40
\@tempa=\box41
\@tempa=\box42
\@tempa=\box43
\@tempa=\box44
\@tempa=\box45
\@tempa=\box46
\@tempa=\box47
\@tempa=\box48
\@tempa=\box49
\@tempa=\box50
\@tempa=\box51
\@tempa=\box52
\@tempa=\box53
\@tempa=\box54
\@tempa=\box55
\@tempa=\box56
\c@columnbadness=\count130
\c@finalcolumnbadness=\count131
\last@try=\dimen133
\multicolovershoot=\dimen134
\multicolundershoot=\dimen135
\mult@nat@firstbox=\box57
\colbreak@box=\box58
\mc@col@check@num=\count132
)
\c@idxcols=\count133
\indexcolsep=\skip66
\indexrule=\skip67
\ila@indentunit=\skip68
\ila@initsep=\skip69
)
(/usr/share/texlive/texmf-dist/tex/latex/tocbibind/tocbibind.sty
Package: tocbibind 2010/10/13 v1.5k extra ToC listings
Package tocbibind Info: The document has section divisions on input line 50.
Package tocbibind Note: Using section or other style headings.
) (/usr/share/texlive/texmf-dist/tex/latex/ms/ragged2e.sty
Package: ragged2e 2009/05/21 v2.1 ragged2e Package (MS)
(/usr/share/texlive/texmf-dist/tex/latex/ms/everysel.sty
Package: everysel 2011/10/28 v1.2 EverySelectfont Package (MS)
)
\CenteringLeftskip=\skip70
\RaggedLeftLeftskip=\skip71
\RaggedRightLeftskip=\skip72
\CenteringRightskip=\skip73
\RaggedLeftRightskip=\skip74
\RaggedRightRightskip=\skip75
\CenteringParfillskip=\skip76
\RaggedLeftParfillskip=\skip77
\RaggedRightParfillskip=\skip78
\JustifyingParfillskip=\skip79
\CenteringParindent=\skip80
\RaggedLeftParindent=\skip81
\RaggedRightParindent=\skip82
\JustifyingParindent=\skip83
)
No file Project1.aux.
\openout1 = `Project1.aux'.
LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 90.
LaTeX Font Info: ... okay on input line 90.
LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 90.
LaTeX Font Info: ... okay on input line 90.
LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 90.
LaTeX Font Info: ... okay on input line 90.
LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 90.
LaTeX Font Info: ... okay on input line 90.
LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 90.
LaTeX Font Info: ... okay on input line 90.
LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 90.
LaTeX Font Info: ... okay on input line 90.
LaTeX Font Info: Checking defaults for PD1/pdf/m/n on input line 90.
LaTeX Font Info: ... okay on input line 90.
LaTeX Font Info: Try loading font information for T1+lmr on input line 90.
(/usr/share/texmf/tex/latex/lm/t1lmr.fd
File: t1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern
)
(/usr/share/texlive/texmf-dist/tex/context/base/supp-pdf.mkii
[Loading MPS to PDF converter (version 2006.09.02).]
\scratchcounter=\count134
\scratchdimen=\dimen136
\scratchbox=\box59
\nofMPsegments=\count135
\nofMParguments=\count136
\everyMPshowfont=\toks30
\MPscratchCnt=\count137
\MPscratchDim=\dimen137
\MPnumerator=\count138
\makeMPintoPDFobject=\count139
\everyMPtoPDFconversion=\toks31
)
LaTeX Info: Redefining \microtypecontext on input line 90.
Package microtype Info: Generating PDF output.
Package microtype Info: Character protrusion enabled (level 2).
Package microtype Info: Using default protrusion set `alltext'.
Package microtype Info: Automatic font expansion enabled (level 2),
(microtype) stretch: 20, shrink: 20, step: 1, non-selected.
Package microtype Info: Using default expansion set `basictext'.
Package microtype Info: No adjustment of tracking.
Package microtype Info: No adjustment of interword spacing.
Package microtype Info: No adjustment of character kerning.
(/usr/share/texlive/texmf-dist/tex/latex/microtype/mt-cmr.cfg
File: mt-cmr.cfg 2013/05/19 v2.2 microtype config. file: Computer Modern Roman
(RS)
)
(/usr/share/texlive/texmf-dist/tex/latex/oberdiek/epstopdf-base.sty
Package: epstopdf-base 2010/02/09 v2.5 Base part for package epstopdf
(/usr/share/texlive/texmf-dist/tex/latex/oberdiek/grfext.sty
Package: grfext 2010/08/19 v1.1 Manage graphics extensions (HO)
)
Package grfext Info: Graphics extension search list:
(grfext) [.png,.pdf,.jpg,.mps,.jpeg,.jbig2,.jb2,.PNG,.PDF,.JPG,.JPE
G,.JBIG2,.JB2,.eps]
(grfext) \AppendGraphicsExtensions on input line 452.
(/usr/share/texlive/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg
File: epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Liv
e
))
(/usr/share/texlive/texmf-dist/tex/latex/ucs/ucsencs.def
File: ucsencs.def 2011/01/21 Fixes to fontencodings LGR, T3
)
\AtBeginShipoutBox=\box60
Package hyperref Info: Link coloring ON on input line 90.
(/usr/share/texlive/texmf-dist/tex/latex/hyperref/nameref.sty
Package: nameref 2012/10/27 v2.43 Cross-referencing by name of section
(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/gettitlestring.sty
Package: gettitlestring 2010/12/03 v1.4 Cleanup title references (HO)
)
\c@section@level=\count140
)
LaTeX Info: Redefining \ref on input line 90.
LaTeX Info: Redefining \pageref on input line 90.
LaTeX Info: Redefining \nameref on input line 90.
\@outlinefile=\write5
\openout5 = `Project1.out'.
ABD: EverySelectfont initializing macros
LaTeX Info: Redefining \selectfont on input line 90.
LaTeX Font Info: Try loading font information for OT1+lmr on input line 155.
(/usr/share/texmf/tex/latex/lm/ot1lmr.fd
File: ot1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern
)
LaTeX Font Info: Try loading font information for OML+lmm on input line 155.
(/usr/share/texmf/tex/latex/lm/omllmm.fd
File: omllmm.fd 2009/10/30 v1.6 Font defs for Latin Modern
)
LaTeX Font Info: Try loading font information for OMS+lmsy on input line 155
.
(/usr/share/texmf/tex/latex/lm/omslmsy.fd
File: omslmsy.fd 2009/10/30 v1.6 Font defs for Latin Modern
)
LaTeX Font Info: Try loading font information for OMX+lmex on input line 155
.
(/usr/share/texmf/tex/latex/lm/omxlmex.fd
File: omxlmex.fd 2009/10/30 v1.6 Font defs for Latin Modern
)
LaTeX Font Info: External font `lmex10' loaded for size
(Font) <10> on input line 155.
LaTeX Font Info: External font `lmex10' loaded for size
(Font) <7> on input line 155.
LaTeX Font Info: External font `lmex10' loaded for size
(Font) <5> on input line 155.
LaTeX Font Info: Try loading font information for U+msa on input line 155.
(/usr/share/texlive/texmf-dist/tex/latex/amsfonts/umsa.fd
File: umsa.fd 2013/01/14 v3.01 AMS symbols A
)
(/usr/share/texlive/texmf-dist/tex/latex/microtype/mt-msa.cfg
File: mt-msa.cfg 2006/02/04 v1.1 microtype config. file: AMS symbols (a) (RS)
)
LaTeX Font Info: Try loading font information for U+msb on input line 155.
(/usr/share/texlive/texmf-dist/tex/latex/amsfonts/umsb.fd
File: umsb.fd 2013/01/14 v3.01 AMS symbols B
)
(/usr/share/texlive/texmf-dist/tex/latex/microtype/mt-msb.cfg
File: mt-msb.cfg 2005/06/01 v1.0 microtype config. file: AMS symbols (b) (RS)
)
LaTeX Font Info: Try loading font information for OMS+lmr on input line 174.
(/usr/share/texmf/tex/latex/lm/omslmr.fd
File: omslmr.fd 2009/10/30 v1.6 Font defs for Latin Modern
)
LaTeX Font Info: Font shape `OMS/lmr/m/n' in size <8> not available
(Font) Font shape `OMS/lmsy/m/n' tried instead on input line 174.
[1
{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map}]
LaTeX Font Info: Try loading font information for T1+lmtt on input line 181.
(/usr/share/texmf/tex/latex/lm/t1lmtt.fd
File: t1lmtt.fd 2009/10/30 v1.6 Font defs for Latin Modern
)
Package microtype Info: Loading generic settings for font family
(microtype) `lmtt' (encoding: T1).
(microtype) For optimal results, create family-specific settings.
(microtype) See the microtype manual for details.
[2] [3] [4]
[5] (/usr/share/texlive/texmf-dist/tex/latex/ucs/data/uni-0.def
File: uni-0.def 2013/05/13 UCS: Unicode data U+0000..U+00FF
) [6]
LaTeX Font Info: Font shape `OMS/lmr/m/n' in size <10> not available
(Font) Font shape `OMS/lmsy/m/n' tried instead on input line 452.
[7]
[8]
Package atveryend Info: Empty hook `BeforeClearDocument' on input line 551.
[9]
Package atveryend Info: Empty hook `AfterLastShipout' on input line 551.
(./Project1.aux)
Package atveryend Info: Executing hook `AtVeryEndDocument' on input line 551.
*File List*
article.cls 2014/09/29 v1.4h Standard LaTeX document class
size10.clo 2014/09/29 v1.4h Standard LaTeX file (size option)
relsize.sty 2013/03/29 ver 4.1
makeidx.sty 2014/09/29 v1.0m Standard LaTeX package
color.sty 1999/02/16
color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive
pdftex.def 2011/05/27 v0.06d Graphics/color for pdfTeX
infwarerr.sty 2010/04/08 v1.3 Providing info/warning/error messages (HO)
ltxcmds.sty 2011/11/09 v1.22 LaTeX kernel commands for general use (HO)
setspace.sty 2011/12/19 v6.7a set line spacing
amsmath.sty 2016/03/03 v2.15a AMS math features
amstext.sty 2000/06/29 v2.01 AMS text
amsgen.sty 1999/11/30 v2.0 generic functions
amsbsy.sty 1999/11/29 v1.2d Bold Symbols
amsopn.sty 1999/12/14 v2.01 operator names
amsfonts.sty 2013/01/14 v3.01 Basic AMSFonts support
amssymb.sty 2013/01/14 v3.01 AMS font symbols
xcolor.sty 2007/01/21 v2.11 LaTeX color extensions (UK)
color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive
colortbl.sty 2012/02/13 v1.0a Color table columns (DPC)
array.sty 2014/10/28 v2.4c Tabular extension package (FMi)
bm.sty 2014/10/28 v1.1c Bold Symbol Support (DPC/FMi)
ltablex.sty 2014/08/13 v1.1 Modified tabularx
longtable.sty 2014/10/28 v4.11 Multi-page Table package (DPC)
tabularx.sty 2014/10/28 v2.10 `tabularx' package (DPC)
microtype.sty 2013/05/23 v2.5a Micro-typographical refinements (RS)
keyval.sty 2014/10/28 v1.15 key=value parser (DPC)
microtype-pdftex.def 2013/05/23 v2.5a Definitions specific to pdftex (RS)
microtype.cfg 2013/05/23 v2.5a microtype main configuration file (RS)
graphicx.sty 2014/10/28 v1.0g Enhanced LaTeX Graphics (DPC,SPQR)
graphics.sty 2016/01/03 v1.0q Standard LaTeX Graphics (DPC,SPQR)
trig.sty 2016/01/03 v1.10 sin cos tan (DPC)
graphics.cfg 2010/04/23 v1.9 graphics configuration of TeX Live
fancyvrb.sty 2008/02/07
fontenc.sty
t1enc.def 2005/09/27 v1.99g Standard LaTeX file
ucs.sty 2013/05/11 v2.2 UCS: Unicode input support
uni-global.def 2013/05/13 UCS: Unicode global data
inputenc.sty 2015/03/17 v1.2c Input encoding file
utf8x.def 2004/10/17 UCS: Input encoding UTF-8
lmodern.sty 2009/10/30 v1.6 Latin Modern Fonts
hyperref.sty 2012/11/06 v6.83m Hypertext links for LaTeX
hobsub-hyperref.sty 2012/05/28 v1.13 Bundle oberdiek, subset hyperref (HO)
hobsub-generic.sty 2012/05/28 v1.13 Bundle oberdiek, subset generic (HO)
hobsub.sty 2012/05/28 v1.13 Construct package bundles (HO)
ifluatex.sty 2010/03/01 v1.3 Provides the ifluatex switch (HO)
ifvtex.sty 2010/03/01 v1.5 Detect VTeX and its facilities (HO)
intcalc.sty 2007/09/27 v1.1 Expandable calculations with integers (HO)
ifpdf.sty 2011/01/30 v2.3 Provides the ifpdf switch (HO)
etexcmds.sty 2011/02/16 v1.5 Avoid name clashes with e-TeX commands (HO)
kvsetkeys.sty 2012/04/25 v1.16 Key value parser (HO)
kvdefinekeys.sty 2011/04/07 v1.3 Define keys (HO)
pdftexcmds.sty 2011/11/29 v0.20 Utility functions of pdfTeX for LuaTeX (HO)
pdfescape.sty 2011/11/25 v1.13 Implements pdfTeX's escape features (HO)
bigintcalc.sty 2012/04/08 v1.3 Expandable calculations on big integers (HO)
bitset.sty 2011/01/30 v1.1 Handle bit-vector datatype (HO)
uniquecounter.sty 2011/01/30 v1.2 Provide unlimited unique counter (HO)
letltxmacro.sty 2010/09/02 v1.4 Let assignment for LaTeX macros (HO)
hopatch.sty 2012/05/28 v1.2 Wrapper for package hooks (HO)
xcolor-patch.sty 2011/01/30 xcolor patch
atveryend.sty 2011/06/30 v1.8 Hooks at the very end of document (HO)
atbegshi.sty 2011/10/05 v1.16 At begin shipout hook (HO)
refcount.sty 2011/10/16 v3.4 Data extraction from label references (HO)
hycolor.sty 2011/01/30 v1.7 Color options for hyperref/bookmark (HO)
ifxetex.sty 2010/09/12 v0.6 Provides ifxetex conditional
auxhook.sty 2011/03/04 v1.3 Hooks for auxiliary files (HO)
kvoptions.sty 2011/06/30 v3.11 Key value format for package options (HO)
pd1enc.def 2012/11/06 v6.83m Hyperref: PDFDocEncoding definition (HO)
hyperref.cfg 2002/06/06 v1.2 hyperref configuration of TeXLive
url.sty 2013/09/16 ver 3.4 Verb mode for urls, etc.
hpdftex.def 2012/11/06 v6.83m Hyperref driver for pdfTeX
rerunfilecheck.sty 2011/04/15 v1.7 Rerun checks for auxiliary files (HO)
fancyhdr.sty
idxlayout.sty 2012/03/30 v0.4d Configurable index layout
etoolbox.sty 2015/08/02 v2.2a e-TeX tools for LaTeX (JAW)
multicol.sty 2015/08/19 v1.8n multicolumn formatting (FMi)
tocbibind.sty 2010/10/13 v1.5k extra ToC listings
ragged2e.sty 2009/05/21 v2.1 ragged2e Package (MS)
everysel.sty 2011/10/28 v1.2 EverySelectfont Package (MS)
t1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern
supp-pdf.mkii
mt-cmr.cfg 2013/05/19 v2.2 microtype config. file: Computer Modern Roman (
RS)
epstopdf-base.sty 2010/02/09 v2.5 Base part for package epstopdf
grfext.sty 2010/08/19 v1.1 Manage graphics extensions (HO)
epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Live
ucsencs.def 2011/01/21 Fixes to fontencodings LGR, T3
nameref.sty 2012/10/27 v2.43 Cross-referencing by name of section
gettitlestring.sty 2010/12/03 v1.4 Cleanup title references (HO)
ot1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern
omllmm.fd 2009/10/30 v1.6 Font defs for Latin Modern
omslmsy.fd 2009/10/30 v1.6 Font defs for Latin Modern
omxlmex.fd 2009/10/30 v1.6 Font defs for Latin Modern
umsa.fd 2013/01/14 v3.01 AMS symbols A
mt-msa.cfg 2006/02/04 v1.1 microtype config. file: AMS symbols (a) (RS)
umsb.fd 2013/01/14 v3.01 AMS symbols B
mt-msb.cfg 2005/06/01 v1.0 microtype config. file: AMS symbols (b) (RS)
omslmr.fd 2009/10/30 v1.6 Font defs for Latin Modern
t1lmtt.fd 2009/10/30 v1.6 Font defs for Latin Modern
uni-0.def 2013/05/13 UCS: Unicode data U+0000..U+00FF
***********
Package atveryend Info: Executing hook `AtEndAfterFileList' on input line 551.
Package rerunfilecheck Warning: File `Project1.out' has changed.
(rerunfilecheck) Rerun to get outlines right
(rerunfilecheck) or use package `bookmark'.
Package rerunfilecheck Info: Checksums for `Project1.out':
(rerunfilecheck) Before: <no file>
(rerunfilecheck) After: D41D8CD98F00B204E9800998ECF8427E;0.
Package atveryend Info: Empty hook `AtVeryVeryEnd' on input line 551.
)
Here is how much of TeX's memory you used:
10632 strings out of 493029
155761 string characters out of 6136233
291311 words of memory out of 5000000
13835 multiletter control sequences out of 15000+600000
66983 words of font info for 101 fonts, out of 8000000 for 9000
1141 hyphenation exceptions out of 8191
41i,15n,43p,1001b,479s stack positions out of 5000i,500n,10000p,200000b,80000s
{/usr/share/texmf/fonts/enc/dvips/lm/lm-rm.enc}{/usr/share/texmf/fonts/enc/dv
ips/lm/lm-mathsy.enc}{/usr/share/texmf/fonts/enc/dvips/lm/lm-mathit.enc}{/usr/s
hare/texmf/fonts/enc/dvips/lm/lm-ec.enc}{/usr/share/texmf/fonts/enc/dvips/lm/lm
-mathex.enc}</usr/share/texlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmbx1
0.pfb></usr/share/texlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmmib10.pfb
></usr/share/texmf/fonts/type1/public/lm/lmbx10.pfb></usr/share/texmf/fonts/typ
e1/public/lm/lmbx12.pfb></usr/share/texmf/fonts/type1/public/lm/lmex10.pfb></us
r/share/texmf/fonts/type1/public/lm/lmmi10.pfb></usr/share/texmf/fonts/type1/pu
blic/lm/lmmi7.pfb></usr/share/texmf/fonts/type1/public/lm/lmr10.pfb></usr/share
/texmf/fonts/type1/public/lm/lmr7.pfb></usr/share/texmf/fonts/type1/public/lm/l
mr8.pfb></usr/share/texmf/fonts/type1/public/lm/lmr9.pfb></usr/share/texmf/font
s/type1/public/lm/lmri10.pfb></usr/share/texmf/fonts/type1/public/lm/lmsy10.pfb
></usr/share/texmf/fonts/type1/public/lm/lmsy7.pfb></usr/share/texmf/fonts/type
1/public/lm/lmsy8.pfb></usr/share/texmf/fonts/type1/public/lm/lmtt10.pfb></usr/
share/texlive/texmf-dist/fonts/type1/public/amsfonts/symbols/msbm10.pfb>
Output written on Project1.pdf (9 pages, 281972 bytes).
PDF statistics:
168 PDF objects out of 1000 (max. 8388607)
138 compressed objects within 2 object streams
29 named destinations out of 1000 (max. 500000)
19969 words of extra memory for PDF output out of 20736 (max. 10000000)
doc/Projects/2020/Project1/pdf/Project1.out
View File
doc/Projects/2020/Project1/pdf/Project1.p.tex
+5 -9
View File
@@ -155,7 +155,7 @@ Project 1 on Machine Learning, deadline October 5, 2020
% --- begin date ---
\begin{center}
Sep 7, 2020
Sep 16, 2020
\end{center}
% --- end date ---
@@ -370,19 +370,15 @@ Note also that when you calculate the bias, in all applications you don't know t
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
data.
Implement the $k$-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
from the test folds. You can compare your own code with that from
\textbf{Scikit-Learn} if needed.
Compare the MSE you get from your cross-validation code with the one you got from your \textbf{bootstrap} code.
You can also compare your own cross-validation code with the one provided by \textbf{Scikit-Learn}.
Compare the MSE you get from your cross-validation code with the one you got from your \textbf{bootstrap} code. Comment your results. Try $5-10$ folds.
You can also compare your own cross-validation code with the one provided by \textbf{Scikit-Learn}.
\paragraph{Part d): Ridge Regression on the Franke function with resampling.}
doc/Projects/2020/Project1/pdf/Project1.pdf
BIN
View File
Binary file not shown.
doc/Projects/2020/Project1/pdf/Project1.tex
+6 -10
View File
@@ -125,7 +125,7 @@ Project 1 on Machine Learning, deadline October 5, 2020
% --- begin date ---
\begin{center}
Sep 7, 2020
Sep 16, 2020
\end{center}
% --- end date ---
@@ -227,7 +227,7 @@ We will generate our own dataset for a function
$\mathrm{FrankeFunction}(x,y)$ with $x,y \in [0,1]$. The function
$f(x,y)$ is the Franke function. You should explore also the addition
of an added stochastic noise to this function using the normal
distribution $\cal{N}(0,1)$.
distribution $N(0,1)$.
\emph{Write your own code} (using either a matrix inversion or a singular
value decomposition from e.g., \textbf{numpy} ) or use your code from
@@ -340,19 +340,15 @@ Note also that when you calculate the bias, in all applications you don't know t
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
data.
Implement the $k$-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
from the test folds. You can compare your own code with that from
\textbf{Scikit-Learn} if needed.
Compare the MSE you get from your cross-validation code with the one you got from your \textbf{bootstrap} code.
You can also compare your own cross-validation code with the one provided by \textbf{Scikit-Learn}.
Compare the MSE you get from your cross-validation code with the one you got from your \textbf{bootstrap} code. Comment your results. Try $5-10$ folds.
You can also compare your own cross-validation code with the one provided by \textbf{Scikit-Learn}.
\paragraph{Part d): Ridge Regression on the Franke function with resampling.}
doc/Projects/2020/hw2/ipynb/hw2.ipynb
+740
View File
@@ -0,0 +1,740 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- dom:TITLE: Homework 2, weeks 36 and 37 -->\n",
"# Homework 2, weeks 36 and 37\n",
"<!-- dom:AUTHOR: [Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html) at Department of Physics, University of Oslo, Norway -->\n",
"<!-- Author: --> \n",
"**[Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html)**, Department of Physics, University of Oslo, Norway\n",
"\n",
"Date: **Sep 8, 2020**\n",
"\n",
"Copyright 1999-2020, [Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html). Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"<!-- --- begin exercise --- -->\n",
"\n",
"## Exercise 1: Adding Ridge and Lasso Regression\n",
"\n",
"This exercise is a continuation of exercise 3 from exercise set 1 (week 35). We will\n",
"use the same function to generate our data set, still staying with a\n",
"simple function $y(x)$ which we want to fit using linear regression,\n",
"but now extending the analysis to include the Ridge and the Lasso\n",
"regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods, see the [regression slides](https://compphysics.github.io/MachineLearning/doc/pub/Regression/html/Regression-bs.html)). \n",
"\n",
"We will thus again generate our own dataset for a function $y(x)$ where \n",
"$x \\in [0,1]$ and defined by random numbers computed with the uniform\n",
"distribution. The function $y$ is a quadratic polynomial in $x$ with\n",
"added stochastic noise according to the normal distribution $\\cal{N}(0,1)$.\n",
"\n",
"The following simple Python instructions define our $x$ and $y$ values (with 100 data points)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"x = np.random.rand(100)\n",
"y = 2.0+5*x*x+0.1*np.random.randn(100)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**a)**\n",
"Write your own code for the Ridge method (see chapter 3.4 of Hastie *et al.*, equations (3.43) and (3.44)) and compute the parametrization for different values of $\\lambda$. Compare and analyze your results with those from exercise 3. Study the dependence on $\\lambda$ while also varying the strength of the noise in your expression for $y(x)$.\n",
"\n",
"\n",
"<!-- --- begin solution of exercise --- -->\n",
"**Solution.**\n",
"The code here allows you to perform your own Ridge calculation and perform calculations for various values of the regularization parameter $\\lambda$. This program can easily be extended upon."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
"import os\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.preprocessing import StandardScaler\n",
"\n",
"def R2(y_data, y_model):\n",
" return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)\n",
"def MSE(y_data,y_model):\n",
" n = np.size(y_model)\n",
" return np.sum((y_data-y_model)**2)/n\n",
"\n",
"\n",
"# A seed just to ensure that the random numbers are the same for every run.\n",
"# Useful for eventual debugging.\n",
"np.random.seed(3155)\n",
"\n",
"x = np.random.rand(100)\n",
"y = 2.0+5*x*x+0.1*np.random.randn(100)\n",
"\n",
"# number of features p (here degree of polynomial\n",
"p = 3\n",
"# The design matrix now as function of a given polynomial\n",
"X = np.zeros((len(x),p))\n",
"X[:,0] = 1.0\n",
"X[:,1] = x\n",
"X[:,2] = x*x\n",
"# We split the data in test and training data\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
"scaler = StandardScaler()\n",
"scaler.fit(X_train)\n",
"X_train_scaled = scaler.transform(X_train)\n",
"X_test_scaled = scaler.transform(X_test)\n",
"\n",
"# matrix inversion to find beta\n",
"OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
"print(OLSbeta)\n",
"# and then make the prediction\n",
"ytildeOLS = X_train @ OLSbeta\n",
"print(\"Training R2 for OLS\")\n",
"print(R2(y_train,ytildeOLS))\n",
"print(\"Training MSE for OLS\")\n",
"print(MSE(y_train,ytildeOLS))\n",
"ypredictOLS = X_test @ OLSbeta\n",
"print(\"Test R2 for OLS\")\n",
"print(R2(y_test,ypredictOLS))\n",
"print(\"Test MSE OLS\")\n",
"print(MSE(y_test,ypredictOLS))\n",
"\n",
"# Repeat now for Ridge regression and various values of the regularization parameter\n",
"I = np.eye(p,p)\n",
"# Decide which values of lambda to use\n",
"nlambdas = 20\n",
"MSEPredict = np.zeros(nlambdas)\n",
"MSETrain = np.zeros(nlambdas)\n",
"lambdas = np.logspace(-4, 1, nlambdas)\n",
"for i in range(nlambdas):\n",
" lmb = lambdas[i]\n",
" Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
" # and then make the prediction\n",
" ytildeRidge = X_train @ Ridgebeta\n",
" ypredictRidge = X_test @ Ridgebeta\n",
" MSEPredict[i] = MSE(y_test,ypredictRidge)\n",
" MSETrain[i] = MSE(y_train,ytildeRidge)\n",
"# Now plot the results\n",
"plt.figure()\n",
"plt.plot(np.log10(lambdas), MSETrain, label = 'MSE Ridge train')\n",
"plt.plot(np.log10(lambdas), MSEPredict, 'r--', label = 'MSE Ridge Test')\n",
"plt.xlabel('log10(lambda)')\n",
"plt.ylabel('MSE')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- --- end solution of exercise --- -->\n",
"\n",
"**b)**\n",
"Repeat the above but using the functionality of **Scikit-Learn**. Compare your code with the results from **Scikit-Learn**. Remember to run with the same random numbers for generating $x$ and $y$.\n",
"\n",
"\n",
"<!-- --- begin solution of exercise --- -->\n",
"**Solution.**\n",
"To use **scikit-learn** with Ridge, we simply need to add the relevant function **Ridge()**, as done in the code here."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.preprocessing import StandardScaler\n",
"import sklearn.linear_model as skl\n",
"\n",
"def R2(y_data, y_model):\n",
" return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)\n",
"def MSE(y_data,y_model):\n",
" n = np.size(y_model)\n",
" return np.sum((y_data-y_model)**2)/n\n",
"\n",
"\n",
"# A seed just to ensure that the random numbers are the same for every run.\n",
"# Useful for eventual debugging.\n",
"np.random.seed(3155)\n",
"\n",
"x = np.random.rand(100)\n",
"y = 2.0+5*x*x+0.1*np.random.randn(100)\n",
"\n",
"# number of features p (here degree of polynomial\n",
"p = 3\n",
"# The design matrix now as function of a given polynomial\n",
"X = np.zeros((len(x),p))\n",
"X[:,0] = 1.0\n",
"X[:,1] = x\n",
"X[:,2] = x*x\n",
"# We split the data in test and training data\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
"scaler = StandardScaler()\n",
"scaler.fit(X_train)\n",
"X_train_scaled = scaler.transform(X_train)\n",
"X_test_scaled = scaler.transform(X_test)\n",
"\n",
"# matrix inversion to find beta\n",
"OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
"print(OLSbeta)\n",
"# and then make the prediction\n",
"ytildeOLS = X_train @ OLSbeta\n",
"print(\"Training R2 for OLS\")\n",
"print(R2(y_train,ytildeOLS))\n",
"print(\"Training MSE for OLS\")\n",
"print(MSE(y_train,ytildeOLS))\n",
"ypredictOLS = X_test @ OLSbeta\n",
"print(\"Test R2 for OLS\")\n",
"print(R2(y_test,ypredictOLS))\n",
"print(\"Test MSE OLS\")\n",
"print(MSE(y_test,ypredictOLS))\n",
"\n",
"# Repeat now for Ridge regression and various values of the regularization parameter\n",
"I = np.eye(p,p)\n",
"# Decide which values of lambda to use\n",
"nlambdas = 100\n",
"MSEPredict = np.zeros(nlambdas)\n",
"MSEPredictSKL = np.zeros(nlambdas)\n",
"MSETrain = np.zeros(nlambdas)\n",
"lambdas = np.logspace(-4, 0, nlambdas)\n",
"for i in range(nlambdas):\n",
" lmb = lambdas[i]\n",
" # add ridge\n",
" clf_ridge = skl.Ridge(alpha=lmb).fit(X_train, y_train)\n",
" yridge = clf_ridge.predict(X_test)\n",
" Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
" # and then make the prediction\n",
" ytildeRidge = X_train @ Ridgebeta\n",
" ypredictRidge = X_test @ Ridgebeta\n",
" MSEPredict[i] = MSE(y_test,ypredictRidge)\n",
" MSEPredictSKL[i] = MSE(y_test,yridge)\n",
" MSETrain[i] = MSE(y_train,ytildeRidge)\n",
"#then plot the results\n",
"plt.figure()\n",
"plt.plot(np.log10(lambdas), MSETrain, label = 'MSE Ridge train')\n",
"plt.plot(np.log10(lambdas), MSEPredict, 'r--', label = 'MSE Ridge Test')\n",
"plt.plot(np.log10(lambdas), MSEPredictSKL, 'g--', label = 'MSE Ridge sickit-learn Test')\n",
"plt.xlabel('log10(lambda)')\n",
"plt.ylabel('MSE')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- --- end solution of exercise --- -->\n",
"\n",
"**c)**\n",
"Our next step is to study the variance of the parameters $\\beta_1$ and $\\beta_2$ (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function or using **Scikit-Learn** to find the parameters $\\beta$. From your results calculate the variance of these parameters (recall that this is equal to the diagonal elements of the matrix $(\\hat{X}^T\\hat{X})+\\lambda\\hat{I})^{-1}$). Discuss the results of these variances as functions of $\\lambda$. In particular, try to link your discussion with the discussion in Hastie *et al.* and their figures 3.10 and 3.11. **Scikit-Learn** may not provide the variance of the parameters $\\beta$. This needs to be checked. With your own code you can however do so.\n",
"\n",
"\n",
"<!-- --- begin solution of exercise --- -->\n",
"**Solution.**"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.preprocessing import StandardScaler\n",
"import sklearn.linear_model as skl\n",
"\n",
"def R2(y_data, y_model):\n",
" return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)\n",
"def MSE(y_data,y_model):\n",
" n = np.size(y_model)\n",
" return np.sum((y_data-y_model)**2)/n\n",
"\n",
"\n",
"# A seed just to ensure that the random numbers are the same for every run.\n",
"# Useful for eventual debugging.\n",
"np.random.seed(3155)\n",
"\n",
"x = np.random.rand(100)\n",
"y = 2.0+5*x*x+0.1*np.random.randn(100)\n",
"\n",
"# number of features p (here degree of polynomial\n",
"p = 3\n",
"# The design matrix now as function of a given polynomial\n",
"X = np.zeros((len(x),p))\n",
"X[:,0] = 1.0\n",
"X[:,1] = x\n",
"X[:,2] = x*x\n",
"# We split the data in test and training data\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
"scaler = StandardScaler()\n",
"scaler.fit(X_train)\n",
"X_train_scaled = scaler.transform(X_train)\n",
"X_test_scaled = scaler.transform(X_test)\n",
"\n",
"# matrix inversion to find beta\n",
"OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
"print(OLSbeta)\n",
"# The variance is given by the inverse of the matrix X^TX\n",
"print(np.linalg.inv(X_train.T @ X_train))\n",
"\n",
"# Repeat now for Ridge regression and various values of the regularization parameter\n",
"I = np.eye(p,p)\n",
"# Decide which values of lambda to use\n",
"nlambdas = 10\n",
"MSEPredict = np.zeros(nlambdas)\n",
"MSEPredictSKL = np.zeros(nlambdas)\n",
"MSETrain = np.zeros(nlambdas)\n",
"lambdas = np.logspace(-4, 0, nlambdas)\n",
"for i in range(nlambdas):\n",
" lmb = lambdas[i]\n",
" Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
" print(np.linalg.inv(X_train.T @ X_train+lmb*I))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- --- end solution of exercise --- -->\n",
"\n",
"**d)**\n",
"Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie *et al.*. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of **scikit-learn**. We recommend the latter since we have not yet discussed how to solve the Lasso equations numerically. Also, you do not need to compute the variance of the parameters $\\beta$ but you can extract their values and study their behavior as functions of the regularization parameter $\\lambda$.\n",
"\n",
"\n",
"<!-- --- begin solution of exercise --- -->\n",
"**Solution.**"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.preprocessing import StandardScaler\n",
"import sklearn.linear_model as skl\n",
"#from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",
"def R2(y_data, y_model):\n",
" return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)\n",
"def MSE(y_data,y_model):\n",
" n = np.size(y_model)\n",
" return np.sum((y_data-y_model)**2)/n\n",
"\n",
"\n",
"# A seed just to ensure that the random numbers are the same for every run.\n",
"# Useful for eventual debugging.\n",
"np.random.seed(3155)\n",
"\n",
"x = np.random.rand(100)\n",
"y = 2.0+5*x*x+0.1*np.random.randn(100)\n",
"\n",
"# number of features p (here degree of polynomial\n",
"p = 3\n",
"# The design matrix now as function of a given polynomial\n",
"X = np.zeros((len(x),p))\n",
"X[:,0] = 1.0\n",
"X[:,1] = x\n",
"X[:,2] = x*x\n",
"# We split the data in test and training data\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
"scaler = StandardScaler()\n",
"scaler.fit(X_train)\n",
"X_train_scaled = scaler.transform(X_train)\n",
"X_test_scaled = scaler.transform(X_test)\n",
"\n",
"# matrix inversion to find beta\n",
"OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n",
"print(OLSbeta)\n",
"# and then make the prediction\n",
"ytildeOLS = X_train @ OLSbeta\n",
"print(\"Training R2 for OLS\")\n",
"print(R2(y_train,ytildeOLS))\n",
"print(\"Training MSE for OLS\")\n",
"print(MSE(y_train,ytildeOLS))\n",
"ypredictOLS = X_test @ OLSbeta\n",
"print(\"Test R2 for OLS\")\n",
"print(R2(y_test,ypredictOLS))\n",
"print(\"Test MSE OLS\")\n",
"print(MSE(y_test,ypredictOLS))\n",
"\n",
"# Repeat now for Ridge regression and various values of the regularization parameter\n",
"I = np.eye(p,p)\n",
"# Decide which values of lambda to use\n",
"nlambdas = 100\n",
"MSEPredictLasso = np.zeros(nlambdas)\n",
"MSEPredictRidge = np.zeros(nlambdas)\n",
"lambdas = np.logspace(-4, 0, nlambdas)\n",
"for i in range(nlambdas):\n",
" lmb = lambdas[i]\n",
" # add ridge\n",
" clf_ridge = skl.Ridge(alpha=lmb).fit(X_train, y_train)\n",
" clf_lasso = skl.Lasso(alpha=lmb).fit(X_train, y_train)\n",
" yridge = clf_ridge.predict(X_test)\n",
" ylasso = clf_lasso.predict(X_test)\n",
" MSEPredictLasso[i] = MSE(y_test,ylasso)\n",
" MSEPredictRidge[i] = MSE(y_test,yridge)\n",
"#then plot the results\n",
"plt.figure()\n",
"plt.plot(np.log10(lambdas), MSEPredictRidge, 'r--', label = 'MSE Ridge Test')\n",
"plt.plot(np.log10(lambdas), MSEPredictLasso, 'g--', label = 'MSE Lasso Test')\n",
"plt.xlabel('log10(lambda)')\n",
"plt.ylabel('MSE')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- --- end solution of exercise --- -->\n",
"\n",
"**e)**\n",
"Finally, using **Scikit-Learn** or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n",
"\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and the $R^2$ score function.\n",
"If $\\tilde{\\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where we have defined the mean value of $\\hat{y}$ as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\bar{y} = \\frac{1}{n} \\sum_{i=0}^{n - 1} y_i.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Discuss these quantities as functions of the variable $\\lambda$ in the Ridge and Lasso regression methods.\n",
"\n",
"\n",
"<!-- --- begin solution of exercise --- -->\n",
"**Solution.**\n",
"These results can all be studied with the codes we have above. These scores are included in the codes above.\n",
"\n",
"<!-- --- end solution of exercise --- -->\n",
"\n",
"\n",
"<!-- --- end exercise --- -->\n",
"\n",
"\n",
"\n",
"\n",
"<!-- --- begin exercise --- -->\n",
"\n",
"## Exercise 2: Normalizing our data\n",
"\n",
"A much used approach before starting to train the data is to preprocess our\n",
"data. Normally the data may need a rescaling and/or may be sensitive\n",
"to extreme values. Scaling the data renders our inputs much more\n",
"suitable for the algorithms we want to employ.\n",
"\n",
"**Scikit-Learn** has several functions which allow us to rescale the\n",
"data, normally resulting in much better results in terms of various\n",
"accuracy scores. The **StandardScaler** function in **Scikit-Learn**\n",
"ensures that for each feature/predictor we study the mean value is\n",
"zero and the variance is one (every column in the design/feature\n",
"matrix). This scaling has the drawback that it does not ensure that\n",
"we have a particular maximum or minimum in our data set. Another\n",
"function included in **Scikit-Learn** is the **MinMaxScaler** which\n",
"ensures that all features are exactly between $0$ and $1$. The\n",
"\n",
"\n",
"The **Normalizer** scales each data\n",
"point such that the feature vector has a euclidean length of one. In other words, it\n",
"projects a data point on the circle (or sphere in the case of higher dimensions) with a\n",
"radius of 1. This means every data point is scaled by a different number (by the\n",
"inverse of it’s length).\n",
"This normalization is often used when only the direction (or angle) of the data matters,\n",
"not the length of the feature vector.\n",
"\n",
"The **RobustScaler** works similarly to the StandardScaler in that it\n",
"ensures statistical properties for each feature that guarantee that\n",
"they are on the same scale. However, the RobustScaler uses the median\n",
"and quartiles, instead of mean and variance. This makes the\n",
"RobustScaler ignore data points that are very different from the rest\n",
"(like measurement errors). These odd data points are also called\n",
"outliers, and might often lead to trouble for other scaling\n",
"techniques.\n",
"\n",
"\n",
"It also common to split the data in a **training** set and a **testing** set. A typical split is to use $80\\%$ of the data for training and the rest\n",
"for testing. This can be done as follows with our design matrix $\\boldsymbol{X}$ and data $\\boldsymbol{y}$ (remember to import **scikit-learn**)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# split in training and test data\n",
"X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we can use the standard scaler to scale our data as"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"scaler = StandardScaler()\n",
"scaler.fit(X_train)\n",
"X_train_scaled = scaler.transform(X_train)\n",
"X_test_scaled = scaler.transform(X_test)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this exercise we want you to to compute the MSE for the training\n",
"data and the test data as function of the complexity of a polynomial,\n",
"that is the degree of a given polynomial. We want you also to compute the $R2$ score as function of the complexity of the model for both training data and test data. You should also run the calculation with and without scaling. \n",
"\n",
"One of \n",
"the aims is to reproduce Figure 2.11 of [Hastie et al](https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf).\n",
"We will also use Ridge and Lasso regression. \n",
"\n",
"\n",
"Our data is defined by $x\\in [-3,3]$ with a total of for example $100$ data points."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"np.random.seed()\n",
"n = 100\n",
"maxdegree = 14\n",
"# Make data set.\n",
"x = np.linspace(-3, 3, n).reshape(-1, 1)\n",
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where $y$ is the function we want to fit with a given polynomial.\n",
"\n",
"\n",
"**a)**\n",
"Write a first code which sets up a design matrix $X$ defined by a fifth-order polynomial. Scale your data and split it in training and test data.\n",
"\n",
"\n",
"<!-- --- begin solution of exercise --- -->\n",
"**Solution.**"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",
"from sklearn.preprocessing import PolynomialFeatures\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.pipeline import make_pipeline\n",
"\n",
"\n",
"np.random.seed(2018)\n",
"n = 50\n",
"maxdegree = 5\n",
"# Make data set.\n",
"x = np.linspace(-3, 3, n).reshape(-1, 1)\n",
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)\n",
"TestError = np.zeros(maxdegree)\n",
"TrainError = np.zeros(maxdegree)\n",
"polydegree = np.zeros(maxdegree)\n",
"x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)\n",
"scaler = StandardScaler()\n",
"scaler.fit(X_train)\n",
"x_train_scaled = scaler.transform(x_train)\n",
"x_test_scaled = scaler.transform(x_test)\n",
"\n",
"for degree in range(maxdegree):\n",
" model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))\n",
" clf = model.fit(x_train_scale,y_train)\n",
" y_fit = clf.predict(x_train_scaled)\n",
" y_pred = clf.predict(x_test_scaled) \n",
" polydegree[degree] = degree\n",
" TestError[degree] = np.mean( np.mean((y_test - y_pred)**2) )\n",
" TrainError[degree] = np.mean( np.mean((y_train - y_fit)**2) )\n",
"\n",
"plt.plot(polydegree, TestError, label='Test Error')\n",
"plt.plot(polydegree, TrainError, label='Train Error')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- --- end solution of exercise --- -->\n",
"\n",
"**b)**\n",
"Perform an ordinary least squares and compute the means squared error and the $R2$ factor for the training data and the test data, with and without scaling.\n",
"\n",
"\n",
"<!-- --- begin solution of exercise --- -->\n",
"**Solution.**\n",
"This requires a simple extension to the above code where you simply add a statement calling the $R2$ function included in the same code.\n",
"\n",
"<!-- --- end solution of exercise --- -->\n",
"\n",
"**c)**\n",
"Add now a model which allows you to make polynomials up to degree $15$. Perform a standard OLS fitting of the training data and compute the MSE and $R2$ for the training and test data and plot both test and training data MSE and $R2$ as functions of the polynomial degree. Compare what you see with Figure 2.11 of Hastie et al. Comment your results. For which polynomial degree do you find an optimal MSE (smallest value)?\n",
"\n",
"\n",
"<!-- --- begin solution of exercise --- -->\n",
"**Solution.**\n",
"Here you simply need to change the degree of the polynomial in the above code to $n=15$.\n",
"\n",
"<!-- --- end solution of exercise --- -->\n",
"\n",
"**d)**\n",
"Repeat part (2c) but now using Ridge regressions with various hyperparameters $\\lambda$. Make the same plots for the optimal $\\lambda$ value for each polynomial degree. Compare these results with those from the standard OLS approach.\n",
"\n",
"\n",
"<!-- --- begin solution of exercise --- -->\n",
"**Solution.**\n",
"Here you need to add for example the same loop over the parameters $\\lambda$ as you did in the first exercise, that is add"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"nlambdas = 100\n",
"MSEPredictRidge = np.zeros(nlambdas)\n",
"lambdas = np.logspace(-4, 0, nlambdas)\n",
"for i in range(nlambdas):\n",
" lmb = lambdas[i]\n",
" # add ridge\n",
" clf_ridge = skl.Ridge(alpha=lmb).fit(X_train_scaled, y_train)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The plotting functionality of the first exercise can be reused here as well.\n",
"\n",
"<!-- --- end solution of exercise --- -->\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"<!-- --- end exercise --- -->"
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 2
}
doc/Projects/2020/hw2/pdf/hw2.idx
View File
doc/src/Projects/2020/Project1/Project1.do.txt
+4 -8
View File
@@ -230,19 +230,15 @@ Note also that when you calculate the bias, in all applications you don't know t
The aim here is to write your own code for another widely popular
resampling technique, the so-called cross-validation method. Again,
before you start with cross-validation approach, you should scale your
data and split it in test and training data as you did earlier.
Perform a resampling of the data where you split the data in training
data and test data using for example
data.
Implement the $k$-fold cross-validation algorithm (write your own
code) and evaluate again the MSE function resulting
from the test data. You can compare your own code with that from
from the test folds. You can compare your own code with that from
_Scikit-Learn_ if needed.
Compare the MSE you get from your cross-validation code with the one you got from your _bootstrap_ code.
You can also compare your own cross-validation code with the one provided by _Scikit-Learn_.
Compare the MSE you get from your cross-validation code with the one you got from your _bootstrap_ code. Comment your results. Try $5-10$ folds.
You can also compare your own cross-validation code with the one provided by _Scikit-Learn_.
=== Part d): Ridge Regression on the Franke function with resampling ===
doc/src/week39/week39.do.txt
+1989
View File
File diff suppressed because it is too large Load Diff
doc/src/week40/week40.do.txt
+2454
View File
File diff suppressed because it is too large Load Diff
doc/src/week41/figslides/cnn.jpeg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

doc/src/week41/figslides/nn.jpeg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 67 KiB

doc/src/week41/week41.do.txt
+608
View File
@@ -0,0 +1,608 @@
TITLE: Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
!split
===== Convolutional Neural Networks (recognizing images) =====
Convolutional neural networks (CNNs) were developed during the last
decade of the previous century, with a focus on character recognition
tasks. Nowadays, CNNs are a central element in the spectacular success
of dee learning methods. The success in for example image
classifications have made them a central tool for most machine
learning practitioners.
CNNs are very similar to ordinary Neural Networks.
They are made up of neurons that have learnable weights and
biases. Each neuron receives some inputs, performs a dot product and
optionally follows it with a non-linearity. The whole network still
expresses a single differentiable score function: from the raw image
pixels on one end to class scores at the other. And they still have a
loss function (for example Softmax) on the last (fully-connected) layer
and all the tips/tricks we developed for learning regular Neural
Networks still apply (back propagation, gradient descent etc etc).
What is the difference? _CNN architectures make the explicit assumption that
the inputs are images, which allows us to encode certain properties
into the architecture. These then make the forward function more
efficient to implement and vastly reduce the amount of parameters in
the network._
Here we provide only a superficial overview, for the more interested, we recommend highly the course
"IN5400 – Machine Learning for Image Analysis":"https://www.uio.no/studier/emner/matnat/ifi/IN5400/index-eng.html"
and the slides of "CS231":"http://cs231n.github.io/convolutional-networks/".
Another good read is the article here URL:"https://arxiv.org/pdf/1603.07285.pdf".
!split
===== Regular NNs don’t scale well to full images =====
As an example, consider
an image of size $32\times 32\times 3$ (32 wide, 32 high, 3 color channels), so a
single fully-connected neuron in a first hidden layer of a regular
Neural Network would have $32\times 32\times 3 = 3072$ weights. This amount still
seems manageable, but clearly this fully-connected structure does not
scale to larger images. For example, an image of more respectable
size, say $200\times 200\times 3$, would lead to neurons that have
$200\times 200\times 3 = 120,000$ weights.
We could have
several such neurons, and the parameters would add up quickly! Clearly,
this full connectivity is wasteful and the huge number of parameters
would quickly lead to possible overfitting.
FIGURE: [figslides/nn.jpeg, width=500 frac=0.6] A regular 3-layer Neural Network.
!split
===== 3D volumes of neurons =====
Convolutional Neural Networks take advantage of the fact that the
input consists of images and they constrain the architecture in a more
sensible way.
In particular, unlike a regular Neural Network, the
layers of a CNN have neurons arranged in 3 dimensions: width,
height, depth. (Note that the word depth here refers to the third
dimension of an activation volume, not to the depth of a full Neural
Network, which can refer to the total number of layers in a network.)
To understand it better, the above example of an image
with an input volume of
activations has dimensions $32\times 32\times 3$ (width, height,
depth respectively).
The neurons in a layer will
only be connected to a small region of the layer before it, instead of
all of the neurons in a fully-connected manner. Moreover, the final
output layer could for this specific image have dimensions $1\times 1 \times 10$,
because by the
end of the CNN architecture we will reduce the full image into a
single vector of class scores, arranged along the depth
dimension.
FIGURE: [figslides/cnn.jpeg, width=500 frac=0.6] A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels).
!split
===== Layers used to build CNNs =====
A simple CNN is a sequence of layers, and every layer of a CNN
transforms one volume of activations to another through a
differentiable function. We use three main types of layers to build
CNN architectures: Convolutional Layer, Pooling Layer, and
Fully-Connected Layer (exactly as seen in regular Neural Networks). We
will stack these layers to form a full CNN architecture.
A simple CNN for image classification could have the architecture:
* _INPUT_ ($32\times 32 \times 3$) will hold the raw pixel values of the image, in this case an image of width 32, height 32, and with three color channels R,G,B.
* _CONV_ (convolutional )layer will compute the output of neurons that are connected to local regions in the input, each computing a dot product between their weights and a small region they are connected to in the input volume. This may result in volume such as $[32\times 32\times 12]$ if we decided to use 12 filters.
* _RELU_ layer will apply an elementwise activation function, such as the $max(0,x)$ thresholding at zero. This leaves the size of the volume unchanged ($[32\times 32\times 12]$).
* _POOL_ (pooling) layer will perform a downsampling operation along the spatial dimensions (width, height), resulting in volume such as $[16\times 16\times 12]$.
* _FC_ (i.e. fully-connected) layer will compute the class scores, resulting in volume of size $[1\times 1\times 10]$, where each of the 10 numbers correspond to a class score, such as among the 10 categories of the MNIST images we considered above . As with ordinary Neural Networks and as the name implies, each neuron in this layer will be connected to all the numbers in the previous volume.
!split
===== Transforming images =====
CNNs transform the original image layer by layer from the original
pixel values to the final class scores.
Observe that some layers contain
parameters and other don’t. In particular, the CNN layers perform
transformations that are a function of not only the activations in the
input volume, but also of the parameters (the weights and biases of
the neurons). On the other hand, the RELU/POOL layers will implement a
fixed function. The parameters in the CONV/FC layers will be trained
with gradient descent so that the class scores that the CNN computes
are consistent with the labels in the training set for each image.
!split
===== CNNs in brief =====
In summary:
* A CNN architecture is in the simplest case a list of Layers that transform the image volume into an output volume (e.g. holding the class scores)
* There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular)
* Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function
* Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don’t)
* Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn’t)
For more material on convolutional networks, we strongly recommend
the course
"IN5400 – Machine Learning for Image Analysis":"https://www.uio.no/studier/emner/matnat/ifi/IN5400/index-eng.html"
and the slides of "CS231":"http://cs231n.github.io/convolutional-networks/" which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). "Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs":"http://neuralnetworksanddeeplearning.com/chap6.html".
!split
===== CNNs in more detail, building convolutional neural networks in Tensorflow and Keras =====
As discussed above, CNNs are neural networks built from the assumption that the inputs
to the network are 2D images. This is important because the number of features or pixels in images
grows very fast with the image size, and an enormous number of weights and biases are needed in order to build an accurate network.
As before, we still have our input, a hidden layer and an output. What's novel about convolutional networks
are the _convolutional_ and _pooling_ layers stacked in pairs between the input and the hidden layer.
In addition, the data is no longer represented as a 2D feature matrix, instead each input is a number of 2D
matrices, typically 1 for each color dimension (Red, Green, Blue).
!split
===== Setting it up =====
It means that to represent the entire
dataset of images, we require a 4D matrix or _tensor_. This tensor has the dimensions:
!bt
\[
(n_{inputs},\, n_{pixels, width},\, n_{pixels, height},\, depth) .
\]
!et
!split
===== The MNIST dataset again =====
The MNIST dataset consists of grayscale images with a pixel size of
$28\times 28$, meaning we require $28 \times 28 = 724$ weights to each
neuron in the first hidden layer.
If we were to analyze images of size $128\times 128$ we would require
$128 \times 128 = 16384$ weights to each neuron. Even worse if we were
dealing with color images, as most images are, we have an image matrix
of size $128\times 128$ for each color dimension (Red, Green, Blue),
meaning 3 times the number of weights $= 49152$ are required for every
single neuron in the first hidden layer.
!split
===== Strong correlations =====
Images typically have strong local correlations, meaning that a small
part of the image varies little from its neighboring regions. If for
example we have an image of a blue car, we can roughly assume that a
small blue part of the image is surrounded by other blue regions.
Therefore, instead of connecting every single pixel to a neuron in the
first hidden layer, as we have previously done with deep neural
networks, we can instead connect each neuron to a small part of the
image (in all 3 RGB depth dimensions). The size of each small area is
fixed, and known as a "receptive":"https://en.wikipedia.org/wiki/Receptive_field".
!split
===== Layers of a CNN =====
The layers of a convolutional neural network arrange neurons in 3D: width, height and depth.
The input image is typically a square matrix of depth 3.
A _convolution_ is performed on the image which outputs
a 3D volume of neurons. The weights to the input are arranged in a number of 2D matrices, known as _filters_.
Each filter slides along the input image, taking the dot product
between each small part of the image and the filter, in all depth
dimensions. This is then passed through a non-linear function,
typically the _Rectified Linear (ReLu)_ function, which serves as the
activation of the neurons in the first convolutional layer. This is
further passed through a _pooling layer_, which reduces the size of the
convolutional layer, e.g. by taking the maximum or average across some
small regions, and this serves as input to the next convolutional
layer.
!split
===== Systematic reduction =====
By systematically reducing the size of the input volume, through
convolution and pooling, the network should create representations of
small parts of the input, and then from them assemble representations
of larger areas. The final pooling layer is flattened to serve as
input to a hidden layer, such that each neuron in the final pooling
layer is connected to every single neuron in the hidden layer. This
then serves as input to the output layer, e.g. a softmax output for
classification.
!split
===== Prerequisites: Collect and pre-process data =====
!bc pycod
# import necessary packages
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
# ensure the same random numbers appear every time
np.random.seed(0)
# display images in notebook
%matplotlib inline
plt.rcParams['figure.figsize'] = (12,12)
# download MNIST dataset
digits = datasets.load_digits()
# define inputs and labels
inputs = digits.images
labels = digits.target
# RGB images have a depth of 3
# our images are grayscale so they should have a depth of 1
inputs = inputs[:,:,:,np.newaxis]
print("inputs = (n_inputs, pixel_width, pixel_height, depth) = " + str(inputs.shape))
print("labels = (n_inputs) = " + str(labels.shape))
# choose some random images to display
n_inputs = len(inputs)
indices = np.arange(n_inputs)
random_indices = np.random.choice(indices, size=5)
for i, image in enumerate(digits.images[random_indices]):
plt.subplot(1, 5, i+1)
plt.axis('off')
plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
plt.title("Label: %d" % digits.target[random_indices[i]])
plt.show()
!ec
!split
===== Importing Keras and Tensorflow =====
!bc pycod
from keras.utils import to_categorical
from sklearn.model_selection import train_test_split
# representation of labels
labels = to_categorical(labels)
# split into train and test data
# one-liner from scikit-learn library
train_size = 0.8
test_size = 1 - train_size
X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,
test_size=test_size)
!ec
!split
===== Using TensorFlow backend =====
We need to define model and architecture and choose cost function and optmizer.
!bc pycid
import tensorflow as tf
class ConvolutionalNeuralNetworkTensorflow:
def __init__(
self,
X_train,
Y_train,
X_test,
Y_test,
n_filters=10,
n_neurons_connected=50,
n_categories=10,
receptive_field=3,
stride=1,
padding=1,
epochs=10,
batch_size=100,
eta=0.1,
lmbd=0.0):
self.global_step = tf.Variable(0, dtype=tf.int32, trainable=False, name='global_step')
self.X_train = X_train
self.Y_train = Y_train
self.X_test = X_test
self.Y_test = Y_test
self.n_inputs, self.input_width, self.input_height, self.depth = X_train.shape
self.n_filters = n_filters
self.n_downsampled = int(self.input_width*self.input_height*n_filters / 4)
self.n_neurons_connected = n_neurons_connected
self.n_categories = n_categories
self.receptive_field = receptive_field
self.stride = stride
self.strides = [stride, stride, stride, stride]
self.padding = padding
self.epochs = epochs
self.batch_size = batch_size
self.iterations = self.n_inputs // self.batch_size
self.eta = eta
self.lmbd = lmbd
self.create_placeholders()
self.create_CNN()
self.create_loss()
self.create_optimiser()
self.create_accuracy()
def create_placeholders(self):
with tf.name_scope('data'):
self.X = tf.placeholder(tf.float32, shape=(None, self.input_width, self.input_height, self.depth), name='X_data')
self.Y = tf.placeholder(tf.float32, shape=(None, self.n_categories), name='Y_data')
def create_CNN(self):
with tf.name_scope('CNN'):
# Convolutional layer
self.W_conv = self.weight_variable([self.receptive_field, self.receptive_field, self.depth, self.n_filters], name='conv', dtype=tf.float32)
b_conv = self.weight_variable([self.n_filters], name='conv', dtype=tf.float32)
z_conv = tf.nn.conv2d(self.X, self.W_conv, self.strides, padding='SAME', name='conv') + b_conv
a_conv = tf.nn.relu(z_conv)
# 2x2 max pooling
a_pool = tf.nn.max_pool(a_conv, [1, 2, 2, 1], [1, 2, 2, 1], padding='SAME', name='pool')
# Fully connected layer
a_pool_flat = tf.reshape(a_pool, [-1, self.n_downsampled])
self.W_fc = self.weight_variable([self.n_downsampled, self.n_neurons_connected], name='fc', dtype=tf.float32)
b_fc = self.bias_variable([self.n_neurons_connected], name='fc', dtype=tf.float32)
a_fc = tf.nn.relu(tf.matmul(a_pool_flat, self.W_fc) + b_fc)
# Output layer
self.W_out = self.weight_variable([self.n_neurons_connected, self.n_categories], name='out', dtype=tf.float32)
b_out = self.bias_variable([self.n_categories], name='out', dtype=tf.float32)
self.z_out = tf.matmul(a_fc, self.W_out) + b_out
def create_loss(self):
with tf.name_scope('loss'):
softmax_loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=self.Y, logits=self.z_out))
regularizer_loss_conv = tf.nn.l2_loss(self.W_conv)
regularizer_loss_fc = tf.nn.l2_loss(self.W_fc)
regularizer_loss_out = tf.nn.l2_loss(self.W_out)
regularizer_loss = self.lmbd*(regularizer_loss_conv + regularizer_loss_fc + regularizer_loss_out)
self.loss = softmax_loss + regularizer_loss
def create_accuracy(self):
with tf.name_scope('accuracy'):
probabilities = tf.nn.softmax(self.z_out)
predictions = tf.argmax(probabilities, 1)
labels = tf.argmax(self.Y, 1)
correct_predictions = tf.equal(predictions, labels)
correct_predictions = tf.cast(correct_predictions, tf.float32)
self.accuracy = tf.reduce_mean(correct_predictions)
def create_optimiser(self):
with tf.name_scope('optimizer'):
self.optimizer = tf.train.GradientDescentOptimizer(learning_rate=self.eta).minimize(self.loss, global_step=self.global_step)
def weight_variable(self, shape, name='', dtype=tf.float32):
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial, name=name, dtype=dtype)
def bias_variable(self, shape, name='', dtype=tf.float32):
initial = tf.constant(0.1, shape=shape)
return tf.Variable(initial, name=name, dtype=dtype)
def fit(self):
data_indices = np.arange(self.n_inputs)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(self.epochs):
for j in range(self.iterations):
chosen_datapoints = np.random.choice(data_indices, size=self.batch_size, replace=False)
batch_X, batch_Y = self.X_train[chosen_datapoints], self.Y_train[chosen_datapoints]
sess.run([CNN.loss, CNN.optimizer],
feed_dict={CNN.X: batch_X,
CNN.Y: batch_Y})
accuracy = sess.run(CNN.accuracy,
feed_dict={CNN.X: batch_X,
CNN.Y: batch_Y})
step = sess.run(CNN.global_step)
self.train_loss, self.train_accuracy = sess.run([CNN.loss, CNN.accuracy],
feed_dict={CNN.X: self.X_train,
CNN.Y: self.Y_train})
self.test_loss, self.test_accuracy = sess.run([CNN.loss, CNN.accuracy],
feed_dict={CNN.X: self.X_test,
CNN.Y: self.Y_test})
!ec
!split
===== Train the model =====
We need now to train the model, evaluate it and test its performance on test data, and eventually include hyperparameters.
!bc pycod
epochs = 100
batch_size = 100
n_filters = 10
n_neurons_connected = 50
n_categories = 10
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
CNN_tf = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
for i, eta in enumerate(eta_vals):
for j, lmbd in enumerate(lmbd_vals):
CNN = ConvolutionalNeuralNetworkTensorflow(X_train, Y_train, X_test, Y_test,
n_filters=n_filters, n_neurons_connected=n_neurons_connected,
n_categories=n_categories, epochs=epochs, batch_size=batch_size,
eta=eta, lmbd=lmbd)
CNN.fit()
print("Learning rate = ", eta)
print("Lambda = ", lmbd)
print("Test accuracy: %.3f" % CNN.test_accuracy)
print()
CNN_tf[i][j] = CNN
!ec
!split
===== Visualizing the results =====
!bc pycod
# visual representation of grid search
# uses seaborn heatmap, could probably do this in matplotlib
import seaborn as sns
sns.set()
train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
for i in range(len(eta_vals)):
for j in range(len(lmbd_vals)):
CNN = CNN_tf[i][j]
train_accuracy[i][j] = CNN.train_accuracy
test_accuracy[i][j] = CNN.test_accuracy
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Training Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Test Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
!ec
!split
===== Running with Keras =====
!bc pycod
from keras.models import Sequential
from keras.layers.convolutional import Conv2D
from keras.layers.convolutional import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
from keras.regularizers import l2
from keras.optimizers import SGD
def create_convolutional_neural_network_keras(input_shape, receptive_field,
n_filters, n_neurons_connected, n_categories,
eta, lmbd):
model = Sequential()
model.add(Conv2D(n_filters, (receptive_field, receptive_field), input_shape=input_shape, padding='same',
activation='relu', kernel_regularizer=l2(lmbd)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(n_neurons_connected, activation='relu', kernel_regularizer=l2(lmbd)))
model.add(Dense(n_categories, activation='softmax', kernel_regularizer=l2(lmbd)))
sgd = SGD(lr=eta)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
return model
epochs = 100
batch_size = 100
input_shape = X_train.shape[1:4]
receptive_field = 3
n_filters = 10
n_neurons_connected = 50
n_categories = 10
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
!ec
!split
===== Final part =====
!bc pycod
CNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
for i, eta in enumerate(eta_vals):
for j, lmbd in enumerate(lmbd_vals):
CNN = create_convolutional_neural_network_keras(input_shape, receptive_field,
n_filters, n_neurons_connected, n_categories,
eta, lmbd)
CNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0)
scores = CNN.evaluate(X_test, Y_test)
CNN_keras[i][j] = CNN
print("Learning rate = ", eta)
print("Lambda = ", lmbd)
print("Test accuracy: %.3f" % scores[1])
print()
!ec
!split
===== Final visualization =====
!bc
# visual representation of grid search
# uses seaborn heatmap, could probably do this in matplotlib
import seaborn as sns
sns.set()
train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
for i in range(len(eta_vals)):
for j in range(len(lmbd_vals)):
CNN = CNN_keras[i][j]
train_accuracy[i][j] = CNN.evaluate(X_train, Y_train)[1]
test_accuracy[i][j] = CNN.evaluate(X_test, Y_test)[1]
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Training Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Test Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
!ec
!split
===== Fun links =====
o "Self-Driving cars using a convolutional neural network":"https://arxiv.org/abs/1604.07316"
o "Abstract art using convolutional neural networks":"https://deepdreamgenerator.com/"
doc/src/week42/week42.do.txt
+616
View File
@@ -0,0 +1,616 @@
TITLE: Convolutional Neural Networks
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
!split
===== To do list =====
* add material about the mathematics, with more explanations
* update codes to tensorflow 2
* add more elaborated examples
* update keras codes and think of pytorch examples?
* Example on pollen cases https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0229751
* Example on nuclear physics experiments
!split
===== Convolutional Neural Networks (recognizing images) =====
Convolutional neural networks (CNNs) were developed during the last
decade of the previous century, with a focus on character recognition
tasks. Nowadays, CNNs are a central element in the spectacular success
of dee learning methods. The success in for example image
classifications have made them a central tool for most machine
learning practitioners.
CNNs are very similar to ordinary Neural Networks.
They are made up of neurons that have learnable weights and
biases. Each neuron receives some inputs, performs a dot product and
optionally follows it with a non-linearity. The whole network still
expresses a single differentiable score function: from the raw image
pixels on one end to class scores at the other. And they still have a
loss function (for example Softmax) on the last (fully-connected) layer
and all the tips/tricks we developed for learning regular Neural
Networks still apply (back propagation, gradient descent etc etc).
What is the difference? _CNN architectures make the explicit assumption that
the inputs are images, which allows us to encode certain properties
into the architecture. These then make the forward function more
efficient to implement and vastly reduce the amount of parameters in
the network._
Here we provide only a superficial overview, for the more interested, we recommend highly the course
"IN5400 – Machine Learning for Image Analysis":"https://www.uio.no/studier/emner/matnat/ifi/IN5400/index-eng.html"
and the slides of "CS231":"http://cs231n.github.io/convolutional-networks/".
Another good read is the article here URL:"https://arxiv.org/pdf/1603.07285.pdf".
!split
===== Regular NNs don’t scale well to full images =====
As an example, consider
an image of size $32\times 32\times 3$ (32 wide, 32 high, 3 color channels), so a
single fully-connected neuron in a first hidden layer of a regular
Neural Network would have $32\times 32\times 3 = 3072$ weights. This amount still
seems manageable, but clearly this fully-connected structure does not
scale to larger images. For example, an image of more respectable
size, say $200\times 200\times 3$, would lead to neurons that have
$200\times 200\times 3 = 120,000$ weights.
We could have
several such neurons, and the parameters would add up quickly! Clearly,
this full connectivity is wasteful and the huge number of parameters
would quickly lead to possible overfitting.
FIGURE: [figslides/nn.jpeg, width=500 frac=0.6] A regular 3-layer Neural Network.
!split
===== 3D volumes of neurons =====
Convolutional Neural Networks take advantage of the fact that the
input consists of images and they constrain the architecture in a more
sensible way.
In particular, unlike a regular Neural Network, the
layers of a CNN have neurons arranged in 3 dimensions: width,
height, depth. (Note that the word depth here refers to the third
dimension of an activation volume, not to the depth of a full Neural
Network, which can refer to the total number of layers in a network.)
To understand it better, the above example of an image
with an input volume of
activations has dimensions $32\times 32\times 3$ (width, height,
depth respectively).
The neurons in a layer will
only be connected to a small region of the layer before it, instead of
all of the neurons in a fully-connected manner. Moreover, the final
output layer could for this specific image have dimensions $1\times 1 \times 10$,
because by the
end of the CNN architecture we will reduce the full image into a
single vector of class scores, arranged along the depth
dimension.
FIGURE: [figslides/cnn.jpeg, width=500 frac=0.6] A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels).
!split
===== Layers used to build CNNs =====
A simple CNN is a sequence of layers, and every layer of a CNN
transforms one volume of activations to another through a
differentiable function. We use three main types of layers to build
CNN architectures: Convolutional Layer, Pooling Layer, and
Fully-Connected Layer (exactly as seen in regular Neural Networks). We
will stack these layers to form a full CNN architecture.
A simple CNN for image classification could have the architecture:
* _INPUT_ ($32\times 32 \times 3$) will hold the raw pixel values of the image, in this case an image of width 32, height 32, and with three color channels R,G,B.
* _CONV_ (convolutional )layer will compute the output of neurons that are connected to local regions in the input, each computing a dot product between their weights and a small region they are connected to in the input volume. This may result in volume such as $[32\times 32\times 12]$ if we decided to use 12 filters.
* _RELU_ layer will apply an elementwise activation function, such as the $max(0,x)$ thresholding at zero. This leaves the size of the volume unchanged ($[32\times 32\times 12]$).
* _POOL_ (pooling) layer will perform a downsampling operation along the spatial dimensions (width, height), resulting in volume such as $[16\times 16\times 12]$.
* _FC_ (i.e. fully-connected) layer will compute the class scores, resulting in volume of size $[1\times 1\times 10]$, where each of the 10 numbers correspond to a class score, such as among the 10 categories of the MNIST images we considered above . As with ordinary Neural Networks and as the name implies, each neuron in this layer will be connected to all the numbers in the previous volume.
!split
===== Transforming images =====
CNNs transform the original image layer by layer from the original
pixel values to the final class scores.
Observe that some layers contain
parameters and other don’t. In particular, the CNN layers perform
transformations that are a function of not only the activations in the
input volume, but also of the parameters (the weights and biases of
the neurons). On the other hand, the RELU/POOL layers will implement a
fixed function. The parameters in the CONV/FC layers will be trained
with gradient descent so that the class scores that the CNN computes
are consistent with the labels in the training set for each image.
!split
===== CNNs in brief =====
In summary:
* A CNN architecture is in the simplest case a list of Layers that transform the image volume into an output volume (e.g. holding the class scores)
* There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular)
* Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function
* Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don’t)
* Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn’t)
For more material on convolutional networks, we strongly recommend
the course
"IN5400 – Machine Learning for Image Analysis":"https://www.uio.no/studier/emner/matnat/ifi/IN5400/index-eng.html"
and the slides of "CS231":"http://cs231n.github.io/convolutional-networks/" which is taught at Stanford University (consistently ranked as one of the top computer science programs in the world). "Michael Nielsen's book is a must read, in particular chapter 6 which deals with CNNs":"http://neuralnetworksanddeeplearning.com/chap6.html".
!split
===== CNNs in more detail, building convolutional neural networks in Tensorflow and Keras =====
As discussed above, CNNs are neural networks built from the assumption that the inputs
to the network are 2D images. This is important because the number of features or pixels in images
grows very fast with the image size, and an enormous number of weights and biases are needed in order to build an accurate network.
As before, we still have our input, a hidden layer and an output. What's novel about convolutional networks
are the _convolutional_ and _pooling_ layers stacked in pairs between the input and the hidden layer.
In addition, the data is no longer represented as a 2D feature matrix, instead each input is a number of 2D
matrices, typically 1 for each color dimension (Red, Green, Blue).
!split
===== Setting it up =====
It means that to represent the entire
dataset of images, we require a 4D matrix or _tensor_. This tensor has the dimensions:
!bt
\[
(n_{inputs},\, n_{pixels, width},\, n_{pixels, height},\, depth) .
\]
!et
!split
===== The MNIST dataset again =====
The MNIST dataset consists of grayscale images with a pixel size of
$28\times 28$, meaning we require $28 \times 28 = 724$ weights to each
neuron in the first hidden layer.
If we were to analyze images of size $128\times 128$ we would require
$128 \times 128 = 16384$ weights to each neuron. Even worse if we were
dealing with color images, as most images are, we have an image matrix
of size $128\times 128$ for each color dimension (Red, Green, Blue),
meaning 3 times the number of weights $= 49152$ are required for every
single neuron in the first hidden layer.
!split
===== Strong correlations =====
Images typically have strong local correlations, meaning that a small
part of the image varies little from its neighboring regions. If for
example we have an image of a blue car, we can roughly assume that a
small blue part of the image is surrounded by other blue regions.
Therefore, instead of connecting every single pixel to a neuron in the
first hidden layer, as we have previously done with deep neural
networks, we can instead connect each neuron to a small part of the
image (in all 3 RGB depth dimensions). The size of each small area is
fixed, and known as a "receptive":"https://en.wikipedia.org/wiki/Receptive_field".
!split
===== Layers of a CNN =====
The layers of a convolutional neural network arrange neurons in 3D: width, height and depth.
The input image is typically a square matrix of depth 3.
A _convolution_ is performed on the image which outputs
a 3D volume of neurons. The weights to the input are arranged in a number of 2D matrices, known as _filters_.
Each filter slides along the input image, taking the dot product
between each small part of the image and the filter, in all depth
dimensions. This is then passed through a non-linear function,
typically the _Rectified Linear (ReLu)_ function, which serves as the
activation of the neurons in the first convolutional layer. This is
further passed through a _pooling layer_, which reduces the size of the
convolutional layer, e.g. by taking the maximum or average across some
small regions, and this serves as input to the next convolutional
layer.
!split
===== Systematic reduction =====
By systematically reducing the size of the input volume, through
convolution and pooling, the network should create representations of
small parts of the input, and then from them assemble representations
of larger areas. The final pooling layer is flattened to serve as
input to a hidden layer, such that each neuron in the final pooling
layer is connected to every single neuron in the hidden layer. This
then serves as input to the output layer, e.g. a softmax output for
classification.
!split
===== Prerequisites: Collect and pre-process data =====
!bc pycod
# import necessary packages
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
# ensure the same random numbers appear every time
np.random.seed(0)
# display images in notebook
%matplotlib inline
plt.rcParams['figure.figsize'] = (12,12)
# download MNIST dataset
digits = datasets.load_digits()
# define inputs and labels
inputs = digits.images
labels = digits.target
# RGB images have a depth of 3
# our images are grayscale so they should have a depth of 1
inputs = inputs[:,:,:,np.newaxis]
print("inputs = (n_inputs, pixel_width, pixel_height, depth) = " + str(inputs.shape))
print("labels = (n_inputs) = " + str(labels.shape))
# choose some random images to display
n_inputs = len(inputs)
indices = np.arange(n_inputs)
random_indices = np.random.choice(indices, size=5)
for i, image in enumerate(digits.images[random_indices]):
plt.subplot(1, 5, i+1)
plt.axis('off')
plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
plt.title("Label: %d" % digits.target[random_indices[i]])
plt.show()
!ec
!split
===== Importing Keras and Tensorflow =====
!bc pycod
from keras.utils import to_categorical
from sklearn.model_selection import train_test_split
# representation of labels
labels = to_categorical(labels)
# split into train and test data
# one-liner from scikit-learn library
train_size = 0.8
test_size = 1 - train_size
X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,
test_size=test_size)
!ec
!split
===== Using TensorFlow backend =====
We need to define model and architecture and choose cost function and optmizer.
!bc pycid
import tensorflow as tf
class ConvolutionalNeuralNetworkTensorflow:
def __init__(
self,
X_train,
Y_train,
X_test,
Y_test,
n_filters=10,
n_neurons_connected=50,
n_categories=10,
receptive_field=3,
stride=1,
padding=1,
epochs=10,
batch_size=100,
eta=0.1,
lmbd=0.0):
self.global_step = tf.Variable(0, dtype=tf.int32, trainable=False, name='global_step')
self.X_train = X_train
self.Y_train = Y_train
self.X_test = X_test
self.Y_test = Y_test
self.n_inputs, self.input_width, self.input_height, self.depth = X_train.shape
self.n_filters = n_filters
self.n_downsampled = int(self.input_width*self.input_height*n_filters / 4)
self.n_neurons_connected = n_neurons_connected
self.n_categories = n_categories
self.receptive_field = receptive_field
self.stride = stride
self.strides = [stride, stride, stride, stride]
self.padding = padding
self.epochs = epochs
self.batch_size = batch_size
self.iterations = self.n_inputs // self.batch_size
self.eta = eta
self.lmbd = lmbd
self.create_placeholders()
self.create_CNN()
self.create_loss()
self.create_optimiser()
self.create_accuracy()
def create_placeholders(self):
with tf.name_scope('data'):
self.X = tf.placeholder(tf.float32, shape=(None, self.input_width, self.input_height, self.depth), name='X_data')
self.Y = tf.placeholder(tf.float32, shape=(None, self.n_categories), name='Y_data')
def create_CNN(self):
with tf.name_scope('CNN'):
# Convolutional layer
self.W_conv = self.weight_variable([self.receptive_field, self.receptive_field, self.depth, self.n_filters], name='conv', dtype=tf.float32)
b_conv = self.weight_variable([self.n_filters], name='conv', dtype=tf.float32)
z_conv = tf.nn.conv2d(self.X, self.W_conv, self.strides, padding='SAME', name='conv') + b_conv
a_conv = tf.nn.relu(z_conv)
# 2x2 max pooling
a_pool = tf.nn.max_pool(a_conv, [1, 2, 2, 1], [1, 2, 2, 1], padding='SAME', name='pool')
# Fully connected layer
a_pool_flat = tf.reshape(a_pool, [-1, self.n_downsampled])
self.W_fc = self.weight_variable([self.n_downsampled, self.n_neurons_connected], name='fc', dtype=tf.float32)
b_fc = self.bias_variable([self.n_neurons_connected], name='fc', dtype=tf.float32)
a_fc = tf.nn.relu(tf.matmul(a_pool_flat, self.W_fc) + b_fc)
# Output layer
self.W_out = self.weight_variable([self.n_neurons_connected, self.n_categories], name='out', dtype=tf.float32)
b_out = self.bias_variable([self.n_categories], name='out', dtype=tf.float32)
self.z_out = tf.matmul(a_fc, self.W_out) + b_out
def create_loss(self):
with tf.name_scope('loss'):
softmax_loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=self.Y, logits=self.z_out))
regularizer_loss_conv = tf.nn.l2_loss(self.W_conv)
regularizer_loss_fc = tf.nn.l2_loss(self.W_fc)
regularizer_loss_out = tf.nn.l2_loss(self.W_out)
regularizer_loss = self.lmbd*(regularizer_loss_conv + regularizer_loss_fc + regularizer_loss_out)
self.loss = softmax_loss + regularizer_loss
def create_accuracy(self):
with tf.name_scope('accuracy'):
probabilities = tf.nn.softmax(self.z_out)
predictions = tf.argmax(probabilities, 1)
labels = tf.argmax(self.Y, 1)
correct_predictions = tf.equal(predictions, labels)
correct_predictions = tf.cast(correct_predictions, tf.float32)
self.accuracy = tf.reduce_mean(correct_predictions)
def create_optimiser(self):
with tf.name_scope('optimizer'):
self.optimizer = tf.train.GradientDescentOptimizer(learning_rate=self.eta).minimize(self.loss, global_step=self.global_step)
def weight_variable(self, shape, name='', dtype=tf.float32):
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial, name=name, dtype=dtype)
def bias_variable(self, shape, name='', dtype=tf.float32):
initial = tf.constant(0.1, shape=shape)
return tf.Variable(initial, name=name, dtype=dtype)
def fit(self):
data_indices = np.arange(self.n_inputs)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(self.epochs):
for j in range(self.iterations):
chosen_datapoints = np.random.choice(data_indices, size=self.batch_size, replace=False)
batch_X, batch_Y = self.X_train[chosen_datapoints], self.Y_train[chosen_datapoints]
sess.run([CNN.loss, CNN.optimizer],
feed_dict={CNN.X: batch_X,
CNN.Y: batch_Y})
accuracy = sess.run(CNN.accuracy,
feed_dict={CNN.X: batch_X,
CNN.Y: batch_Y})
step = sess.run(CNN.global_step)
self.train_loss, self.train_accuracy = sess.run([CNN.loss, CNN.accuracy],
feed_dict={CNN.X: self.X_train,
CNN.Y: self.Y_train})
self.test_loss, self.test_accuracy = sess.run([CNN.loss, CNN.accuracy],
feed_dict={CNN.X: self.X_test,
CNN.Y: self.Y_test})
!ec
!split
===== Train the model =====
We need now to train the model, evaluate it and test its performance on test data, and eventually include hyperparameters.
!bc pycod
epochs = 100
batch_size = 100
n_filters = 10
n_neurons_connected = 50
n_categories = 10
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
CNN_tf = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
for i, eta in enumerate(eta_vals):
for j, lmbd in enumerate(lmbd_vals):
CNN = ConvolutionalNeuralNetworkTensorflow(X_train, Y_train, X_test, Y_test,
n_filters=n_filters, n_neurons_connected=n_neurons_connected,
n_categories=n_categories, epochs=epochs, batch_size=batch_size,
eta=eta, lmbd=lmbd)
CNN.fit()
print("Learning rate = ", eta)
print("Lambda = ", lmbd)
print("Test accuracy: %.3f" % CNN.test_accuracy)
print()
CNN_tf[i][j] = CNN
!ec
!split
===== Visualizing the results =====
!bc pycod
# visual representation of grid search
# uses seaborn heatmap, could probably do this in matplotlib
import seaborn as sns
sns.set()
train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
for i in range(len(eta_vals)):
for j in range(len(lmbd_vals)):
CNN = CNN_tf[i][j]
train_accuracy[i][j] = CNN.train_accuracy
test_accuracy[i][j] = CNN.test_accuracy
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Training Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Test Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
!ec
!split
===== Running with Keras =====
!bc pycod
from keras.models import Sequential
from keras.layers.convolutional import Conv2D
from keras.layers.convolutional import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
from keras.regularizers import l2
from keras.optimizers import SGD
def create_convolutional_neural_network_keras(input_shape, receptive_field,
n_filters, n_neurons_connected, n_categories,
eta, lmbd):
model = Sequential()
model.add(Conv2D(n_filters, (receptive_field, receptive_field), input_shape=input_shape, padding='same',
activation='relu', kernel_regularizer=l2(lmbd)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(n_neurons_connected, activation='relu', kernel_regularizer=l2(lmbd)))
model.add(Dense(n_categories, activation='softmax', kernel_regularizer=l2(lmbd)))
sgd = SGD(lr=eta)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
return model
epochs = 100
batch_size = 100
input_shape = X_train.shape[1:4]
receptive_field = 3
n_filters = 10
n_neurons_connected = 50
n_categories = 10
eta_vals = np.logspace(-5, 1, 7)
lmbd_vals = np.logspace(-5, 1, 7)
!ec
!split
===== Final part =====
!bc pycod
CNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
for i, eta in enumerate(eta_vals):
for j, lmbd in enumerate(lmbd_vals):
CNN = create_convolutional_neural_network_keras(input_shape, receptive_field,
n_filters, n_neurons_connected, n_categories,
eta, lmbd)
CNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0)
scores = CNN.evaluate(X_test, Y_test)
CNN_keras[i][j] = CNN
print("Learning rate = ", eta)
print("Lambda = ", lmbd)
print("Test accuracy: %.3f" % scores[1])
print()
!ec
!split
===== Final visualization =====
!bc
# visual representation of grid search
# uses seaborn heatmap, could probably do this in matplotlib
import seaborn as sns
sns.set()
train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
for i in range(len(eta_vals)):
for j in range(len(lmbd_vals)):
CNN = CNN_keras[i][j]
train_accuracy[i][j] = CNN.evaluate(X_train, Y_train)[1]
test_accuracy[i][j] = CNN.evaluate(X_test, Y_test)[1]
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Training Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
fig, ax = plt.subplots(figsize = (10, 10))
sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
ax.set_title("Test Accuracy")
ax.set_ylabel("$\eta$")
ax.set_xlabel("$\lambda$")
plt.show()
!ec
!split
===== Fun links =====
o "Self-Driving cars using a convolutional neural network":"https://arxiv.org/abs/1604.07316"
o "Abstract art using convolutional neural networks":"https://deepdreamgenerator.com/"
doc/src/week43/week43.do.txt
+1010
View File
File diff suppressed because it is too large Load Diff
doc/src/week44/DataFiles/bank.csv
+1372
View File
File diff suppressed because it is too large Load Diff
doc/src/week44/DataFiles/cancer.dot
+57
View File
@@ -0,0 +1,57 @@
digraph Tree {
node [shape=box, style="filled, rounded", color="black", fontname=helvetica] ;
edge [fontname=helvetica] ;
0 [label="worst perimeter <= 106.05\ngini = 0.465\nsamples = 426\nvalue = [[269, 157]\n[157, 269]]", fillcolor="#e5813908"] ;
1 [label="worst concave points <= 0.159\ngini = 0.067\nsamples = 259\nvalue = [[250, 9]\n[9, 250]]", fillcolor="#e58139db"] ;
0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ;
2 [label="worst concave points <= 0.135\ngini = 0.031\nsamples = 253\nvalue = [[249, 4]\n[4, 249]]", fillcolor="#e58139ee"] ;
1 -> 2 ;
3 [label="radius error <= 0.643\ngini = 0.008\nsamples = 242\nvalue = [[241, 1]\n[1, 241]]", fillcolor="#e58139fb"] ;
2 -> 3 ;
4 [label="gini = 0.0\nsamples = 239\nvalue = [[239, 0]\n[0, 239]]", fillcolor="#e58139ff"] ;
3 -> 4 ;
5 [label="worst symmetry <= 0.208\ngini = 0.444\nsamples = 3\nvalue = [[2, 1]\n[1, 2]]", fillcolor="#e5813913"] ;
3 -> 5 ;
6 [label="gini = 0.0\nsamples = 1\nvalue = [[0, 1]\n[1, 0]]", fillcolor="#e58139ff"] ;
5 -> 6 ;
7 [label="gini = 0.0\nsamples = 2\nvalue = [[2, 0]\n[0, 2]]", fillcolor="#e58139ff"] ;
5 -> 7 ;
8 [label="worst texture <= 29.455\ngini = 0.397\nsamples = 11\nvalue = [[8, 3]\n[3, 8]]", fillcolor="#e581392c"] ;
2 -> 8 ;
9 [label="gini = 0.0\nsamples = 8\nvalue = [[8, 0]\n[0, 8]]", fillcolor="#e58139ff"] ;
8 -> 9 ;
10 [label="gini = 0.0\nsamples = 3\nvalue = [[0, 3]\n[3, 0]]", fillcolor="#e58139ff"] ;
8 -> 10 ;
11 [label="mean texture <= 16.22\ngini = 0.278\nsamples = 6\nvalue = [[1, 5]\n[5, 1]]", fillcolor="#e581396b"] ;
1 -> 11 ;
12 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139ff"] ;
11 -> 12 ;
13 [label="gini = 0.0\nsamples = 5\nvalue = [[0, 5]\n[5, 0]]", fillcolor="#e58139ff"] ;
11 -> 13 ;
14 [label="worst texture <= 20.645\ngini = 0.202\nsamples = 167\nvalue = [[19, 148]\n[148, 19]]", fillcolor="#e5813994"] ;
0 -> 14 [labeldistance=2.5, labelangle=-45, headlabel="False"] ;
15 [label="worst radius <= 17.74\ngini = 0.375\nsamples = 16\nvalue = [[12, 4]\n[4, 12]]", fillcolor="#e5813938"] ;
14 -> 15 ;
16 [label="gini = 0.0\nsamples = 11\nvalue = [[11, 0]\n[0, 11]]", fillcolor="#e58139ff"] ;
15 -> 16 ;
17 [label="mean texture <= 13.745\ngini = 0.32\nsamples = 5\nvalue = [[1, 4]\n[4, 1]]", fillcolor="#e5813955"] ;
15 -> 17 ;
18 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139ff"] ;
17 -> 18 ;
19 [label="gini = 0.0\nsamples = 4\nvalue = [[0, 4]\n[4, 0]]", fillcolor="#e58139ff"] ;
17 -> 19 ;
20 [label="mean concave points <= 0.049\ngini = 0.088\nsamples = 151\nvalue = [[7, 144]\n[144, 7]]", fillcolor="#e58139d0"] ;
14 -> 20 ;
21 [label="concave points error <= 0.01\ngini = 0.48\nsamples = 15\nvalue = [[6, 9]\n[9, 6]]", fillcolor="#e5813900"] ;
20 -> 21 ;
22 [label="gini = 0.0\nsamples = 9\nvalue = [[0, 9]\n[9, 0]]", fillcolor="#e58139ff"] ;
21 -> 22 ;
23 [label="gini = 0.0\nsamples = 6\nvalue = [[6, 0]\n[0, 6]]", fillcolor="#e58139ff"] ;
21 -> 23 ;
24 [label="worst smoothness <= 0.096\ngini = 0.015\nsamples = 136\nvalue = [[1, 135]\n[135, 1]]", fillcolor="#e58139f7"] ;
20 -> 24 ;
25 [label="gini = 0.0\nsamples = 1\nvalue = [[1, 0]\n[0, 1]]", fillcolor="#e58139ff"] ;
24 -> 25 ;
26 [label="gini = 0.0\nsamples = 135\nvalue = [[0, 135]\n[135, 0]]", fillcolor="#e58139ff"] ;
24 -> 26 ;
}
doc/src/week44/DataFiles/cancer.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 216 KiB

doc/src/week44/DataFiles/ensembleoverview.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB

doc/src/week44/DataFiles/ride.csv
+15
View File
@@ -0,0 +1,15 @@
Outlook,Temperature,Humidity,Wind,Ride
0,0,0,0,0
0,0,0,1,1
1,0,0,0,1
2,1,0,0,1
2,2,1,0,1
2,2,1,1,0
1,2,1,1,1
0,1,0,0,0
0,2,1,0,1
2,1,1,0,1
0,1,1,1,1
1,1,0,1,1
1,0,1,0,1
2,1,0,1,0
1 Outlook Temperature Humidity Wind Ride
2 0 0 0 0 0
3 0 0 0 1 1
4 1 0 0 0 1
5 2 1 0 0 1
6 2 2 1 0 1
7 2 2 1 1 0
8 1 2 1 1 1
9 0 1 0 0 0
10 0 2 1 0 1
11 2 1 1 0 1
12 0 1 1 1 1
13 1 1 0 1 1
14 1 0 1 0 1
15 2 1 0 1 0
doc/src/week44/DataFiles/ride.csv~
+15
View File
@@ -0,0 +1,15 @@
Outlook,Temperature,Humidity,Wind,Ride
Sunny,Hot,High,Weak,0
Sunny,Hot,High,Strong,1
Overcast,Hot,High,Weak,1
Rain,Mild,High,Weak,1
Rain,Cool,Normal,Weak,1
Rain,Cool,Normal,Strong,0
Overcast,Cool,Normal,Strong,1
Sunny,Mild,High,Weak,0
Sunny,Cool,Normal,Weak,1
Rain,Mild,Normal,Weak,1
Sunny,Mild,Normal,Strong,1
Overcast,Mild,High,Strong,1
Overcast,Hot,Normal,Weak,1
Rain,Mild,High,Strong,0
doc/src/week44/DataFiles/ride.dot
+13
View File
@@ -0,0 +1,13 @@
digraph Tree {
node [shape=box, style="filled, rounded", color="black", fontname=helvetica] ;
edge [fontname=helvetica] ;
0 [label="X[7] <= 0.5\ngini = 0.48\nsamples = 15\nvalue = [4, 10, 1]", fillcolor="#39e5818b"] ;
1 [label="X[1] <= 0.5\ngini = 0.408\nsamples = 14\nvalue = [4, 10, 0]", fillcolor="#39e58199"] ;
0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ;
2 [label="gini = 0.48\nsamples = 10\nvalue = [4, 6, 0]", fillcolor="#39e58155"] ;
1 -> 2 ;
3 [label="gini = 0.0\nsamples = 4\nvalue = [0, 4, 0]", fillcolor="#39e581ff"] ;
1 -> 3 ;
4 [label="gini = 0.0\nsamples = 1\nvalue = [0, 0, 1]", fillcolor="#8139e5ff"] ;
0 -> 4 [labeldistance=2.5, labelangle=-45, headlabel="False"] ;
}
doc/src/week44/DataFiles/rideclass.csv
+15
View File
@@ -0,0 +1,15 @@
Day,Outlook,Temperature,Humidity,Wind,Ride
1,Sunny,Hot,High,Weak,0
2,Sunny,Hot,High,Strong,1
3,Overcast,Hot,High,Weak,1
4,Rain,Mild,High,Weak,1
5,Rain,Cool,Normal,Weak,1
6,Rain,Cool,Normal,Strong,0
7,Overcast,Cool,Normal,Strong,1
8,Sunny,Mild,High,Weak,0
9,Sunny,Cool,Normal,Weak,1
10,Rain,Mild,Normal,Weak,1
11,Sunny,Mild,Normal,Strong,1
12,Overcast,Mild,High,Strong,1
13,Overcast,Hot,Normal,Weak,1
14,Rain,Mild,High,Strong,0
1 Day Outlook Temperature Humidity Wind Ride
2 1 Sunny Hot High Weak 0
3 2 Sunny Hot High Strong 1
4 3 Overcast Hot High Weak 1
5 4 Rain Mild High Weak 1
6 5 Rain Cool Normal Weak 1
7 6 Rain Cool Normal Strong 0
8 7 Overcast Cool Normal Strong 1
9 8 Sunny Mild High Weak 0
10 9 Sunny Cool Normal Weak 1
11 10 Rain Mild Normal Weak 1
12 11 Sunny Mild Normal Strong 1
13 12 Overcast Mild High Strong 1
14 13 Overcast Hot Normal Weak 1
15 14 Rain Mild High Strong 0
doc/src/week44/DataFiles/zoo.csv
+101
View File
@@ -0,0 +1,101 @@
aardvark,1,0,0,1,0,0,1,1,1,1,0,0,4,0,0,1,1
antelope,1,0,0,1,0,0,0,1,1,1,0,0,4,1,0,1,1
bass,0,0,1,0,0,1,1,1,1,0,0,1,0,1,0,0,4
bear,1,0,0,1,0,0,1,1,1,1,0,0,4,0,0,1,1
boar,1,0,0,1,0,0,1,1,1,1,0,0,4,1,0,1,1
buffalo,1,0,0,1,0,0,0,1,1,1,0,0,4,1,0,1,1
calf,1,0,0,1,0,0,0,1,1,1,0,0,4,1,1,1,1
carp,0,0,1,0,0,1,0,1,1,0,0,1,0,1,1,0,4
catfish,0,0,1,0,0,1,1,1,1,0,0,1,0,1,0,0,4
cavy,1,0,0,1,0,0,0,1,1,1,0,0,4,0,1,0,1
cheetah,1,0,0,1,0,0,1,1,1,1,0,0,4,1,0,1,1
chicken,0,1,1,0,1,0,0,0,1,1,0,0,2,1,1,0,2
chub,0,0,1,0,0,1,1,1,1,0,0,1,0,1,0,0,4
clam,0,0,1,0,0,0,1,0,0,0,0,0,0,0,0,0,7
crab,0,0,1,0,0,1,1,0,0,0,0,0,4,0,0,0,7
crayfish,0,0,1,0,0,1,1,0,0,0,0,0,6,0,0,0,7
crow,0,1,1,0,1,0,1,0,1,1,0,0,2,1,0,0,2
deer,1,0,0,1,0,0,0,1,1,1,0,0,4,1,0,1,1
dogfish,0,0,1,0,0,1,1,1,1,0,0,1,0,1,0,1,4
dolphin,0,0,0,1,0,1,1,1,1,1,0,1,0,1,0,1,1
dove,0,1,1,0,1,0,0,0,1,1,0,0,2,1,1,0,2
duck,0,1,1,0,1,1,0,0,1,1,0,0,2,1,0,0,2
elephant,1,0,0,1,0,0,0,1,1,1,0,0,4,1,0,1,1
flamingo,0,1,1,0,1,0,0,0,1,1,0,0,2,1,0,1,2
flea,0,0,1,0,0,0,0,0,0,1,0,0,6,0,0,0,6
frog,0,0,1,0,0,1,1,1,1,1,0,0,4,0,0,0,5
frog,0,0,1,0,0,1,1,1,1,1,1,0,4,0,0,0,5
fruitbat,1,0,0,1,1,0,0,1,1,1,0,0,2,1,0,0,1
giraffe,1,0,0,1,0,0,0,1,1,1,0,0,4,1,0,1,1
girl,1,0,0,1,0,0,1,1,1,1,0,0,2,0,1,1,1
gnat,0,0,1,0,1,0,0,0,0,1,0,0,6,0,0,0,6
goat,1,0,0,1,0,0,0,1,1,1,0,0,4,1,1,1,1
gorilla,1,0,0,1,0,0,0,1,1,1,0,0,2,0,0,1,1
gull,0,1,1,0,1,1,1,0,1,1,0,0,2,1,0,0,2
haddock,0,0,1,0,0,1,0,1,1,0,0,1,0,1,0,0,4
hamster,1,0,0,1,0,0,0,1,1,1,0,0,4,1,1,0,1
hare,1,0,0,1,0,0,0,1,1,1,0,0,4,1,0,0,1
hawk,0,1,1,0,1,0,1,0,1,1,0,0,2,1,0,0,2
herring,0,0,1,0,0,1,1,1,1,0,0,1,0,1,0,0,4
honeybee,1,0,1,0,1,0,0,0,0,1,1,0,6,0,1,0,6
housefly,1,0,1,0,1,0,0,0,0,1,0,0,6,0,0,0,6
kiwi,0,1,1,0,0,0,1,0,1,1,0,0,2,1,0,0,2
ladybird,0,0,1,0,1,0,1,0,0,1,0,0,6,0,0,0,6
lark,0,1,1,0,1,0,0,0,1,1,0,0,2,1,0,0,2
leopard,1,0,0,1,0,0,1,1,1,1,0,0,4,1,0,1,1
lion,1,0,0,1,0,0,1,1,1,1,0,0,4,1,0,1,1
lobster,0,0,1,0,0,1,1,0,0,0,0,0,6,0,0,0,7
lynx,1,0,0,1,0,0,1,1,1,1,0,0,4,1,0,1,1
mink,1,0,0,1,0,1,1,1,1,1,0,0,4,1,0,1,1
mole,1,0,0,1,0,0,1,1,1,1,0,0,4,1,0,0,1
mongoose,1,0,0,1,0,0,1,1,1,1,0,0,4,1,0,1,1
moth,1,0,1,0,1,0,0,0,0,1,0,0,6,0,0,0,6
newt,0,0,1,0,0,1,1,1,1,1,0,0,4,1,0,0,5
octopus,0,0,1,0,0,1,1,0,0,0,0,0,8,0,0,1,7
opossum,1,0,0,1,0,0,1,1,1,1,0,0,4,1,0,0,1
oryx,1,0,0,1,0,0,0,1,1,1,0,0,4,1,0,1,1
ostrich,0,1,1,0,0,0,0,0,1,1,0,0,2,1,0,1,2
parakeet,0,1,1,0,1,0,0,0,1,1,0,0,2,1,1,0,2
penguin,0,1,1,0,0,1,1,0,1,1,0,0,2,1,0,1,2
pheasant,0,1,1,0,1,0,0,0,1,1,0,0,2,1,0,0,2
pike,0,0,1,0,0,1,1,1,1,0,0,1,0,1,0,1,4
piranha,0,0,1,0,0,1,1,1,1,0,0,1,0,1,0,0,4
pitviper,0,0,1,0,0,0,1,1,1,1,1,0,0,1,0,0,3
platypus,1,0,1,1,0,1,1,0,1,1,0,0,4,1,0,1,1
polecat,1,0,0,1,0,0,1,1,1,1,0,0,4,1,0,1,1
pony,1,0,0,1,0,0,0,1,1,1,0,0,4,1,1,1,1
porpoise,0,0,0,1,0,1,1,1,1,1,0,1,0,1,0,1,1
puma,1,0,0,1,0,0,1,1,1,1,0,0,4,1,0,1,1
pussycat,1,0,0,1,0,0,1,1,1,1,0,0,4,1,1,1,1
raccoon,1,0,0,1,0,0,1,1,1,1,0,0,4,1,0,1,1
reindeer,1,0,0,1,0,0,0,1,1,1,0,0,4,1,1,1,1
rhea,0,1,1,0,0,0,1,0,1,1,0,0,2,1,0,1,2
scorpion,0,0,0,0,0,0,1,0,0,1,1,0,8,1,0,0,7
seahorse,0,0,1,0,0,1,0,1,1,0,0,1,0,1,0,0,4
seal,1,0,0,1,0,1,1,1,1,1,0,1,0,0,0,1,1
sealion,1,0,0,1,0,1,1,1,1,1,0,1,2,1,0,1,1
seasnake,0,0,0,0,0,1,1,1,1,0,1,0,0,1,0,0,3
seawasp,0,0,1,0,0,1,1,0,0,0,1,0,0,0,0,0,7
skimmer,0,1,1,0,1,1,1,0,1,1,0,0,2,1,0,0,2
skua,0,1,1,0,1,1,1,0,1,1,0,0,2,1,0,0,2
slowworm,0,0,1,0,0,0,1,1,1,1,0,0,0,1,0,0,3
slug,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,7
sole,0,0,1,0,0,1,0,1,1,0,0,1,0,1,0,0,4
sparrow,0,1,1,0,1,0,0,0,1,1,0,0,2,1,0,0,2
squirrel,1,0,0,1,0,0,0,1,1,1,0,0,2,1,0,0,1
starfish,0,0,1,0,0,1,1,0,0,0,0,0,5,0,0,0,7
stingray,0,0,1,0,0,1,1,1,1,0,1,1,0,1,0,1,4
swan,0,1,1,0,1,1,0,0,1,1,0,0,2,1,0,1,2
termite,0,0,1,0,0,0,0,0,0,1,0,0,6,0,0,0,6
toad,0,0,1,0,0,1,0,1,1,1,0,0,4,0,0,0,5
tortoise,0,0,1,0,0,0,0,0,1,1,0,0,4,1,0,1,3
tuatara,0,0,1,0,0,0,1,1,1,1,0,0,4,1,0,0,3
tuna,0,0,1,0,0,1,1,1,1,0,0,1,0,1,0,1,4
vampire,1,0,0,1,1,0,0,1,1,1,0,0,2,1,0,0,1
vole,1,0,0,1,0,0,0,1,1,1,0,0,4,1,0,0,1
vulture,0,1,1,0,1,0,1,0,1,1,0,0,2,1,0,1,2
wallaby,1,0,0,1,0,0,0,1,1,1,0,0,2,1,0,1,1
wasp,1,0,1,0,1,0,0,0,0,1,1,0,6,0,0,0,6
wolf,1,0,0,1,0,0,1,1,1,1,0,0,4,1,0,1,1
worm,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,7
wren,0,1,1,0,1,0,0,0,1,1,0,0,2,1,0,0,2
1 aardvark 1 0 0 1 0 0 1 1 1 1 0 0 4 0 0 1 1
2 antelope 1 0 0 1 0 0 0 1 1 1 0 0 4 1 0 1 1
3 bass 0 0 1 0 0 1 1 1 1 0 0 1 0 1 0 0 4
4 bear 1 0 0 1 0 0 1 1 1 1 0 0 4 0 0 1 1
5 boar 1 0 0 1 0 0 1 1 1 1 0 0 4 1 0 1 1
6 buffalo 1 0 0 1 0 0 0 1 1 1 0 0 4 1 0 1 1
7 calf 1 0 0 1 0 0 0 1 1 1 0 0 4 1 1 1 1
8 carp 0 0 1 0 0 1 0 1 1 0 0 1 0 1 1 0 4
9 catfish 0 0 1 0 0 1 1 1 1 0 0 1 0 1 0 0 4
10 cavy 1 0 0 1 0 0 0 1 1 1 0 0 4 0 1 0 1
11 cheetah 1 0 0 1 0 0 1 1 1 1 0 0 4 1 0 1 1
12 chicken 0 1 1 0 1 0 0 0 1 1 0 0 2 1 1 0 2
13 chub 0 0 1 0 0 1 1 1 1 0 0 1 0 1 0 0 4
14 clam 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 7
15 crab 0 0 1 0 0 1 1 0 0 0 0 0 4 0 0 0 7
16 crayfish 0 0 1 0 0 1 1 0 0 0 0 0 6 0 0 0 7
17 crow 0 1 1 0 1 0 1 0 1 1 0 0 2 1 0 0 2
18 deer 1 0 0 1 0 0 0 1 1 1 0 0 4 1 0 1 1
19 dogfish 0 0 1 0 0 1 1 1 1 0 0 1 0 1 0 1 4
20 dolphin 0 0 0 1 0 1 1 1 1 1 0 1 0 1 0 1 1
21 dove 0 1 1 0 1 0 0 0 1 1 0 0 2 1 1 0 2
22 duck 0 1 1 0 1 1 0 0 1 1 0 0 2 1 0 0 2
23 elephant 1 0 0 1 0 0 0 1 1 1 0 0 4 1 0 1 1
24 flamingo 0 1 1 0 1 0 0 0 1 1 0 0 2 1 0 1 2
25 flea 0 0 1 0 0 0 0 0 0 1 0 0 6 0 0 0 6
26 frog 0 0 1 0 0 1 1 1 1 1 0 0 4 0 0 0 5
27 frog 0 0 1 0 0 1 1 1 1 1 1 0 4 0 0 0 5
28 fruitbat 1 0 0 1 1 0 0 1 1 1 0 0 2 1 0 0 1
29 giraffe 1 0 0 1 0 0 0 1 1 1 0 0 4 1 0 1 1
30 girl 1 0 0 1 0 0 1 1 1 1 0 0 2 0 1 1 1
31 gnat 0 0 1 0 1 0 0 0 0 1 0 0 6 0 0 0 6
32 goat 1 0 0 1 0 0 0 1 1 1 0 0 4 1 1 1 1
33 gorilla 1 0 0 1 0 0 0 1 1 1 0 0 2 0 0 1 1
34 gull 0 1 1 0 1 1 1 0 1 1 0 0 2 1 0 0 2
35 haddock 0 0 1 0 0 1 0 1 1 0 0 1 0 1 0 0 4
36 hamster 1 0 0 1 0 0 0 1 1 1 0 0 4 1 1 0 1
37 hare 1 0 0 1 0 0 0 1 1 1 0 0 4 1 0 0 1
38 hawk 0 1 1 0 1 0 1 0 1 1 0 0 2 1 0 0 2
39 herring 0 0 1 0 0 1 1 1 1 0 0 1 0 1 0 0 4
40 honeybee 1 0 1 0 1 0 0 0 0 1 1 0 6 0 1 0 6
41 housefly 1 0 1 0 1 0 0 0 0 1 0 0 6 0 0 0 6
42 kiwi 0 1 1 0 0 0 1 0 1 1 0 0 2 1 0 0 2
43 ladybird 0 0 1 0 1 0 1 0 0 1 0 0 6 0 0 0 6
44 lark 0 1 1 0 1 0 0 0 1 1 0 0 2 1 0 0 2
45 leopard 1 0 0 1 0 0 1 1 1 1 0 0 4 1 0 1 1
46 lion 1 0 0 1 0 0 1 1 1 1 0 0 4 1 0 1 1
47 lobster 0 0 1 0 0 1 1 0 0 0 0 0 6 0 0 0 7
48 lynx 1 0 0 1 0 0 1 1 1 1 0 0 4 1 0 1 1
49 mink 1 0 0 1 0 1 1 1 1 1 0 0 4 1 0 1 1
50 mole 1 0 0 1 0 0 1 1 1 1 0 0 4 1 0 0 1
51 mongoose 1 0 0 1 0 0 1 1 1 1 0 0 4 1 0 1 1
52 moth 1 0 1 0 1 0 0 0 0 1 0 0 6 0 0 0 6
53 newt 0 0 1 0 0 1 1 1 1 1 0 0 4 1 0 0 5
54 octopus 0 0 1 0 0 1 1 0 0 0 0 0 8 0 0 1 7
55 opossum 1 0 0 1 0 0 1 1 1 1 0 0 4 1 0 0 1
56 oryx 1 0 0 1 0 0 0 1 1 1 0 0 4 1 0 1 1
57 ostrich 0 1 1 0 0 0 0 0 1 1 0 0 2 1 0 1 2
58 parakeet 0 1 1 0 1 0 0 0 1 1 0 0 2 1 1 0 2
59 penguin 0 1 1 0 0 1 1 0 1 1 0 0 2 1 0 1 2
60 pheasant 0 1 1 0 1 0 0 0 1 1 0 0 2 1 0 0 2
61 pike 0 0 1 0 0 1 1 1 1 0 0 1 0 1 0 1 4
62 piranha 0 0 1 0 0 1 1 1 1 0 0 1 0 1 0 0 4
63 pitviper 0 0 1 0 0 0 1 1 1 1 1 0 0 1 0 0 3
64 platypus 1 0 1 1 0 1 1 0 1 1 0 0 4 1 0 1 1
65 polecat 1 0 0 1 0 0 1 1 1 1 0 0 4 1 0 1 1
66 pony 1 0 0 1 0 0 0 1 1 1 0 0 4 1 1 1 1
67 porpoise 0 0 0 1 0 1 1 1 1 1 0 1 0 1 0 1 1
68 puma 1 0 0 1 0 0 1 1 1 1 0 0 4 1 0 1 1
69 pussycat 1 0 0 1 0 0 1 1 1 1 0 0 4 1 1 1 1
70 raccoon 1 0 0 1 0 0 1 1 1 1 0 0 4 1 0 1 1
71 reindeer 1 0 0 1 0 0 0 1 1 1 0 0 4 1 1 1 1
72 rhea 0 1 1 0 0 0 1 0 1 1 0 0 2 1 0 1 2
73 scorpion 0 0 0 0 0 0 1 0 0 1 1 0 8 1 0 0 7
74 seahorse 0 0 1 0 0 1 0 1 1 0 0 1 0 1 0 0 4
75 seal 1 0 0 1 0 1 1 1 1 1 0 1 0 0 0 1 1
76 sealion 1 0 0 1 0 1 1 1 1 1 0 1 2 1 0 1 1
77 seasnake 0 0 0 0 0 1 1 1 1 0 1 0 0 1 0 0 3
78 seawasp 0 0 1 0 0 1 1 0 0 0 1 0 0 0 0 0 7
79 skimmer 0 1 1 0 1 1 1 0 1 1 0 0 2 1 0 0 2
80 skua 0 1 1 0 1 1 1 0 1 1 0 0 2 1 0 0 2
81 slowworm 0 0 1 0 0 0 1 1 1 1 0 0 0 1 0 0 3
82 slug 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 7
83 sole 0 0 1 0 0 1 0 1 1 0 0 1 0 1 0 0 4
84 sparrow 0 1 1 0 1 0 0 0 1 1 0 0 2 1 0 0 2
85 squirrel 1 0 0 1 0 0 0 1 1 1 0 0 2 1 0 0 1
86 starfish 0 0 1 0 0 1 1 0 0 0 0 0 5 0 0 0 7
87 stingray 0 0 1 0 0 1 1 1 1 0 1 1 0 1 0 1 4
88 swan 0 1 1 0 1 1 0 0 1 1 0 0 2 1 0 1 2
89 termite 0 0 1 0 0 0 0 0 0 1 0 0 6 0 0 0 6
90 toad 0 0 1 0 0 1 0 1 1 1 0 0 4 0 0 0 5
91 tortoise 0 0 1 0 0 0 0 0 1 1 0 0 4 1 0 1 3
92 tuatara 0 0 1 0 0 0 1 1 1 1 0 0 4 1 0 0 3
93 tuna 0 0 1 0 0 1 1 1 1 0 0 1 0 1 0 1 4
94 vampire 1 0 0 1 1 0 0 1 1 1 0 0 2 1 0 0 1
95 vole 1 0 0 1 0 0 0 1 1 1 0 0 4 1 0 0 1
96 vulture 0 1 1 0 1 0 1 0 1 1 0 0 2 1 0 1 2
97 wallaby 1 0 0 1 0 0 0 1 1 1 0 0 2 1 0 1 1
98 wasp 1 0 1 0 1 0 0 0 0 1 1 0 6 0 0 0 6
99 wolf 1 0 0 1 0 0 1 1 1 1 0 0 4 1 0 1 1
100 worm 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 7
101 wren 0 1 1 0 1 0 0 0 1 1 0 0 2 1 0 0 2
doc/src/week44/week44.do.txt
+1524
View File
File diff suppressed because it is too large Load Diff
doc/src/week45/week45.do.txt
+815
View File
@@ -0,0 +1,815 @@
TITLE: Week 45: Random Forests and Boosting
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
!split
===== Random forests =====
Random forests provide an improvement over bagged trees by way of a
small tweak that decorrelates the trees.
As in bagging, we build a
number of decision trees on bootstrapped training samples. But when
building these decision trees, each time a split in a tree is
considered, a random sample of $m$ predictors is chosen as split
candidates from the full set of $p$ predictors. The split is allowed to
use only one of those $m$ predictors.
A fresh sample of $m$ predictors is
taken at each split, and typically we choose
!bt
\[
m\approx \sqrt{p}.
\]
!et
In building a random forest, at
each split in the tree, the algorithm is not even allowed to consider
a majority of the available predictors.
The reason for this is rather clever. Suppose that there is one very
strong predictor in the data set, along with a number of other
moderately strong predictors. Then in the collection of bagged
variable importance random forest trees, most or all of the trees will
use this strong predictor in the top split. Consequently, all of the
bagged trees will look quite similar to each other. Hence the
predictions from the bagged trees will be highly correlated.
Unfortunately, averaging many highly correlated quantities does not
lead to as large of a reduction in variance as averaging many
uncorrelated quantities. In particular, this means that bagging will
not lead to a substantial reduction in variance over a single tree in
this setting.
!split
===== Random Forest Algorithm =====
The algorithm described here can be applied to both classification and regression problems.
We will grow of forest of say $B$ trees.
o For $b=1:B$
* Draw a bootstrap sample of from the training data organized in our $\bm{X}$ matrix.
* We grow then a random forest tree $T_b$ based on the bootstrapped data by repeating the steps outlined till we reach the maximum node size is reached
o we select $m \le p$ variables at random from the $p$ predictors/features
o pick the best split point among the $m$ features using either the CART algorithm or the ID3 for classification and create a new node
o split the node into daughter nodes
o Output then the ensemble of trees $\{T_b\}_1^{B}$ and make predictions for either a regression type of problem or a classification type of problem.
!split
===== Random Forests Compared with other Methods on the Cancer Data =====
!bc pycod
import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
# Load the data
cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
print(X_train.shape)
print(X_test.shape)
# Logistic Regression
logreg = LogisticRegression(solver='lbfgs')
logreg.fit(X_train, y_train)
print("Test set accuracy with Logistic Regression: {:.2f}".format(logreg.score(X_test,y_test)))
# Support vector machine
svm = SVC(gamma='auto', C=100)
svm.fit(X_train, y_train)
print("Test set accuracy with SVM: {:.2f}".format(svm.score(X_test,y_test)))
# Decision Trees
deep_tree_clf = DecisionTreeClassifier(max_depth=None)
deep_tree_clf.fit(X_train, y_train)
print("Test set accuracy with Decision Trees: {:.2f}".format(deep_tree_clf.score(X_test,y_test)))
#now scale the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Logistic Regression
logreg.fit(X_train_scaled, y_train)
print("Test set accuracy Logistic Regression with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
# Support Vector Machine
svm.fit(X_train_scaled, y_train)
print("Test set accuracy SVM with scaled data: {:.2f}".format(logreg.score(X_test_scaled,y_test)))
# Decision Trees
deep_tree_clf.fit(X_train_scaled, y_train)
print("Test set accuracy with Decision Trees and scaled data: {:.2f}".format(deep_tree_clf.score(X_test_scaled,y_test)))
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import cross_validate
# Data set not specificied
#Instantiate the model with 500 trees and entropy as splitting criteria
Random_Forest_model = RandomForestClassifier(n_estimators=500,criterion="entropy")
Random_Forest_model.fit(X_train_scaled, y_train)
#Cross validation
accuracy = cross_validate(Random_Forest_model,X_test_scaled,y_test,cv=10)['test_score']
print(accuracy)
print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(Random_Forest_model.score(X_test_scaled,y_test)))
import scikitplot as skplt
y_pred = Random_Forest_model.predict(X_test_scaled)
skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
plt.show()
y_probas = Random_Forest_model.predict_proba(X_test_scaled)
skplt.metrics.plot_roc(y_test, y_probas)
plt.show()
skplt.metrics.plot_cumulative_gain(y_test, y_probas)
plt.show()
!ec
!split
===== Compare Bagging on Trees with Random Forests =====
!bc pycod
bag_clf = BaggingClassifier(
DecisionTreeClassifier(splitter="random", max_leaf_nodes=16, random_state=42),
n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1, random_state=42)
!ec
!bc pycod
bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
from sklearn.ensemble import RandomForestClassifier
rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16, n_jobs=-1, random_state=42)
rnd_clf.fit(X_train, y_train)
y_pred_rf = rnd_clf.predict(X_test)
np.sum(y_pred == y_pred_rf) / len(y_pred)
!ec
!split
===== Boosting, a Bird's Eye View =====
The basic idea is to combine weak classifiers in order to create a good
classifier. With a weak classifier we often intend a classifier which
produces results which are only slightly better than we would get by
random guesses.
This is done by applying in an iterative way a weak (or a standard
classifier like decision trees) to modify the data. In each iteration
we emphasize those observations which are misclassified by weighting
them with a factor.
!split
===== What is boosting? Additive Modelling/Iterative Fitting =====
Boosting is a way of fitting an additive expansion in a set of
elementary basis functions like for example some simple polynomials.
Assume for example that we have a function
!bt
\[
f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m),
\]
!et
where $\beta_m$ are the expansion parameters to be determined in a
minimization process and $b(x;\gamma_m)$ are some simple functions of
the multivariable parameter $x$ which is characterized by the
parameters $\gamma_m$.
As an example, consider the Sigmoid function we used in logistic
regression. In that case, we can translate the function
$b(x;\gamma_m)$ into the Sigmoid function
!bt
\[
\sigma(t) = \frac{1}{1+\exp{(-t)}},
\]
!et
where $t=\gamma_0+\gamma_1 x$ and the parameters $\gamma_0$ and
$\gamma_1$ were determined by the Logistic Regression fitting
algorithm.
As another example, consider the cost function we defined for linear regression
!bt
\[
C(\bm{y},\bm{f}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
\]
!et
In this case the function $f(x)$ was replaced by the design matrix
$\bm{X}$ and the unknown linear regression parameters $\bm{\beta}$,
that is $\bm{f}=\bm{X}\bm{\beta}$. In linear regression we can
simply invert a matrix and obtain the parameters $\beta$ by
!bt
\[
\bm{\beta}=\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
In iterative fitting or additive modeling, we minimize the cost function with respect to the parameters $\beta_m$ and $\gamma_m$.
!split
===== Iterative Fitting, Regression and Squared-error Cost Function =====
The way we proceed is as follows (here we specialize to the squared-error cost function)
o Establish a cost function, here ${\cal C}(\bm{y},\bm{f}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-f_M(x_i))^2$ with $f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m)$.
o Initialize with a guess $f_0(x)$. It could be one or even zero or some random numbers.
o For $m=1:M$
o minimize $\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2$ wrt $\gamma$ and $\beta$
o This gives the optimal values $\beta_m$ and $\gamma_m$
o Determine then the new values $f_m(x)=f_{m-1}(x) +\beta_m b(x;\gamma_m)$
We could use any of the algorithms we have discussed till now. If we
use trees, $\gamma$ parameterizes the split variables and split points
at the internal nodes, and the predictions at the terminal nodes.
!split
===== Squared-Error Example and Iterative Fitting =====
To better understand what happens, let us develop the steps for the iterative fitting using the above squared error function.
For simplicity we assume also that our functions $b(x;\gamma)=1+\gamma x$.
This means that for every iteration $m$, we need to optimize
!bt
\[
(\beta_m,\gamma_m) = \mathrm{argmin}_{\beta,\lambda}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta b(x;\gamma))^2=\sum_{i=0}^{n-1}(y_i-f_{m-1}(x_i)-\beta(1+\gamma x_i))^2.
\]
!et
We start our iteration by simply setting $f_0(x)=0$.
Taking the derivatives with respect to $\beta$ and $\gamma$ we obtain
!bt
\[
\frac{\partial {\cal C}}{\partial \beta} = -2\sum_{i}(1+\gamma x_i)(y_i-\beta(1+\gamma x_i))=0,
\]
!et
and
!bt
\[
\frac{\partial {\cal C}}{\partial \gamma} =-2\sum_{i}\beta x_i(y_i-\beta(1+\gamma x_i))=0.
\]
!et
We can then rewrite these equations as (defining $\bm{w}=\bm{e}+\gamma \bm{x})$ with $\bm{e}$ being the unit vector)
!bt
\[
\gamma \bm{w}^T(\bm{y}-\beta\gamma \bm{w})=0,
\]
!et
which gives us $\beta = \bm{w}^T\bm{y}/(\bm{w}^T\bm{w})$. Similarly we have
!bt
\[
\beta\gamma \bm{x}^T(\bm{y}-\beta(1+\gamma \bm{x}))=0,
\]
!et
which leads to $\gamma =(\bm{x}^T\bm{y}-\beta\bm{x}^T\bm{e})/(\beta\bm{x}^T\bm{x})$. Inserting
for $\beta$ gives us an equation for $\gamma$. This is a non-linear equation in the unknown $\gamma$ and has to be solved numerically.
The solution to these two equations gives us in turn $\beta_1$ and $\gamma_1$ leading to the new expression for $f_1(x)$ as
$f_1(x) = \beta_1(1+\gamma_1x)$. Doing this $M$ times results in our final estimate for the function $f$.
!split
===== Iterative Fitting, Classification and AdaBoost =====
Let us consider a binary classification problem with two outcomes $y_i \in \{-1,1\}$ and $i=0,1,2,\dots,n-1$ as our set of
observations. We define a classification function $G(x)$ which produces a prediction taking one or the other of the two values
$\{-1,1\}$.
The error rate of the training sample is then
!bt
\[
\mathrm{\overline{err}}=\frac{1}{n} \sum_{i=0}^{n-1} I(y_i\ne G(x_i)).
\]
!et
The iterative procedure starts with defining a weak classifier whose
error rate is barely better than random guessing. The iterative
procedure in boosting is to sequentially apply a weak
classification algorithm to repeatedly modified versions of the data
producing a sequence of weak classifiers $G_m(x)$.
Here we will express our function $f(x)$ in terms of $G(x)$. That is
!bt
\[
f_M(x) = \sum_{i=1}^M \beta_m b(x;\gamma_m),
\]
!et
will be a function of
!bt
\[
G_M(x) = \mathrm{sign} \sum_{i=1}^M \alpha_m G_m(x).
\]
!et
!split
===== Adaptive Boosting, AdaBoost =====
In our iterative procedure we define thus
!bt
\[
f_m(x) = f_{m-1}(x)+\beta_mG_m(x).
\]
!et
The simplest possible cost function which leads (also simple from a computational point of view) to the AdaBoost algorithm is the
exponential cost/loss function defined as
!bt
\[
C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}\exp{(-y_i(f_{m-1}(x_i)+\beta G(x_i))}.
\]
!et
We optimize $\beta$ and $G$ for each value of $m=1:M$ as we did in the regression case.
This is normally done in two steps. Let us however first rewrite the cost function as
!bt
\[
C(\bm{y},\bm{f}) = \sum_{i=0}^{n-1}w_i^{m}\exp{(-y_i\beta G(x_i))},
\]
!et
where we have defined $w_i^m= \exp{(-y_if_{m-1}(x_i))}$.
!split
===== Building up AdaBoost =====
First, for any $\beta > 0$, we optimize $G$ by setting
!bt
\[
G_m(x) = \mathrm{sign} \sum_{i=0}^{n-1} w_i^m I(y_i \ne G_(x_i)),
\]
!et
which is the classifier that minimizes the weighted error rate in predicting $y$.
We can do this by rewriting
!bt
\[
\exp{-(\beta)}\sum_{y_i=G(x_i)}w_i^m+\exp{(\beta)}\sum_{y_i\ne G(x_i)}w_i^m,
\]
!et
which can be rewritten as
!bt
\[
(\exp{(\beta)}-\exp{-(\beta)})\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i))+\exp{(-\beta)}\sum_{i=0}^{n-1}w_i^m=0,
\]
!et
which leads to
!bt
\[
\beta_m = \frac{1}{2}\log{\frac{1-\mathrm{\overline{err}}}{\mathrm{\overline{err}}}},
\]
!et
where we have redefined the error as
!bt
\[
\mathrm{\overline{err}}_m=\frac{1}{n}\frac{\sum_{i=0}^{n-1}w_i^mI(y_i\ne G(x_i)}{\sum_{i=0}^{n-1}w_i^m},
\]
!et
which leads to an update of
!bt
\[
f_m(x) = f_{m-1}(x) +\beta_m G_m(x).
\]
!et
This leads to the new weights
!bt
\[
w_i^{m+1} = w_i^m \exp{(-y_i\beta_m G_m(x_i))}
\]
!et
!split
===== Adaptive boosting: AdaBoost, Basic Algorithm =====
The algorithm here is rather straightforward. Assume that our weak
classifier is a decision tree and we consider a binary set of outputs
with $y_i \in \{-1,1\}$ and $i=0,1,2,\dots,n-1$ as our set of
observations. Our design matrix is given in terms of the
feature/predictor vectors
$\bm{X}=[\bm{x}_0\bm{x}_1\dots\bm{x}_{p-1}]$. Finally, we define also a
classifier determined by our data via a function $G(x)$. This function tells us how well we are able to classify our outputs/targets $\bm{y}$.
We have already defined the misclassification error $\mathrm{err}$ as
!bt
\[
\mathrm{err}=\frac{1}{n}\sum_{i=0}^{n-1}I(y_i\ne G(x_i)),
\]
!et
where the function $I()$ is one if we misclassify and zero if we classify correctly.
!split
===== Basic Steps of AdaBoost =====
With the above definitions we are now ready to set up the algorithm for AdaBoost.
The basic idea is to set up weights which will be used to scale the correctly classified and the misclassified cases.
o We start by initializing all weights to $w_i = 1/n$, with $i=0,1,2,\dots n-1$. It is easy to see that we must have $\sum_{i=0}^{n-1}w_i = 1$.
o We rewrite the misclassification error as
!bt
\[
\mathrm{\overline{err}}_m=\frac{\sum_{i=0}^{n-1}w_i^m I(y_i\ne G(x_i))}{\sum_{i=0}^{n-1}w_i},
\]
!et
o Then we start looping over all attempts at classifying, namely we start an iterative process for $m=1:M$, where $M$ is the final number of classifications. Our given classifier could for example be a plain decision tree.
o Fit then a given classifier to the training set using the weights $w_i$.
o Compute then $\mathrm{err}$ and figure out which events are classified properly and which are classified wrongly.
o Define a quantity $\alpha_{m} = \log{(1-\mathrm{\overline{err}}_m)/\mathrm{\overline{err}}_m}$
o Set the new weights to $w_i = w_i\times \exp{(\alpha_m I(y_i\ne G(x_i)}$.
o Compute the new classifier $G(x)= \sum_{i=0}^{n-1}\alpha_m I(y_i\ne G(x_i)$.
For the iterations with $m \le 2$ the weights are modified
individually at each steps. The observations which were misclassified
at iteration $m-1$ have a weight which is larger than those which were
classified properly. As this proceeds, the observations which were
difficult to classifiy correctly are given a larger influence. Each
new classification step $m$ is then forced to concentrate on those
observations that are missed in the previous iterations.
!split
===== AdaBoost Examples =====
Using _Scikit-Learn_ it is easy to apply the adaptive boosting algorithm, as done here.
!bc pycod
from sklearn.ensemble import AdaBoostClassifier
ada_clf = AdaBoostClassifier(
DecisionTreeClassifier(max_depth=1), n_estimators=200,
algorithm="SAMME.R", learning_rate=0.5, random_state=42)
ada_clf.fit(X_train, y_train)
from sklearn.ensemble import AdaBoostClassifier
ada_clf = AdaBoostClassifier(
DecisionTreeClassifier(max_depth=1), n_estimators=200,
algorithm="SAMME.R", learning_rate=0.5, random_state=42)
ada_clf.fit(X_train_scaled, y_train)
y_pred = ada_clf.predict(X_test_scaled)
skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
plt.show()
y_probas = ada_clf.predict_proba(X_test_scaled)
skplt.metrics.plot_roc(y_test, y_probas)
plt.show()
skplt.metrics.plot_cumulative_gain(y_test, y_probas)
plt.show()
!ec
!split
===== AdaBoost for Regression =====
Here we present "Drucker's AdaBoost":"https://pdfs.semanticscholar.org/8d49/e2dedb817f2c3330e74b63c5fc86d2399ce3.pdf" tailored for regression.
In bagging, each training example is equally likely to be
picked. In boosting, the probability of a particular
example being in the training set of a particular machine
depends on the performance of the prior machines on
that example. The following is a modification of
Adaboost by Drucker.
Start by selecting a set of training data $n$ and assign to each entry a weight $w_i=1$ for $i=1,2,\dots,n$. As we have done earlier, we could pick say $80\%$ of the data set for training. The algorithm runs as follows:
o We define the probability that the training sample $i$ is in the set by $p_i = w_i/\sum_iw_i$. We pick $n$ samples (with replacement) to form our training set. We pick a number uniformly in the range $[0,\sum_iw_i]$.
o We choose then a regression machine (for example plain linear regression or a simple decision tree). A given regression machine makes then a hypothesis.
o Using every member of the training set with the chosen regression machine we obtain then a prediction $\tilde{y}_i$.
o We calculate then the loss function $L_i$ for each training sample. We can use various types of loss function as long as we have a value
$L_i\in [0,1]$.
!split
===== Gradient boosting: Basics with Steepest Descent =====
Gradient boosting is again a similar technique to Adaptive boosting,
it combines so-called weak classifiers or regressors into a strong
method via a series of iterations.
In order to understand the method, let us illustrate its basics by
bringing back the essential steps in linear regression, where our cost
function was the least squares function.
!split
===== The Squared-Error again! Steepest Descent =====
We start again with our cost function ${\cal C}(\bm{y}m\bm{f})=\sum_{i=0}^{n-1}{\cal L}(y_i, f(x_i))$ where we want to minimize
This means that for every iteration, we need to optimize
!bt
\[
(\hat{\bm{f}}) = \mathrm{argmin}_{\bm{f}}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i-f(x_i))^2.
\]
!et
We define a real function $h_m(x)$ that defines our final function $f_M(x)$ as
!bt
\[
f_M(x) = \sum_{m=0}^M h_m(x).
\]
!et
In the steepest decent approach we approximate $h_m(x) = -\rho_m g_m(x)$, where $\rho_m$ is a scalar and $g_m(x)$ the gradient defined as
!bt
\[
g_m(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{m-1}(x_i)}.
\]
!et
With the new gradient we can update $f_m(x) = f_{m-1}(x) -\rho_m g_m(x)$. Using the above squared-error function we see that
the gradient is $g_m(x_i) = -2(y_i-f(x_i))$.
Choosing $f_0(x)=0$ we obtain $g_m(x) = -2y_i$ and inserting this into the minimization problem for the cost function we have
!bt
\[
(\rho_1) = \mathrm{argmin}_{\rho}\hspace{0.1cm} \sum_{i=0}^{n-1}(y_i+2\rho y_i)^2.
\]
!et
!split
===== Steepest Descent Example =====
Optimizing with respect to $\rho$ we obtain (taking the derivative) that $\rho_1 = -1/2$. We have then that
!bt
\[
f_1(x) = f_{0}(x) -\rho_1 g_1(x)=-y_i.
\]
!et
We can then proceed and compute
!bt
\[
g_2(x_i) = \left[ \frac{\partial {\cal L}(y_i, f(x_i))}{\partial f(x_i)}\right]_{f(x_i)=f_{1}(x_i)=y_i}=-4y_i,
\]
!et
and find a new value for $\rho_2=-1/2$ and continue till we have reached $m=M$. We can modify the steepest descent method, or steepest boosting, by introducing what is called _gradient boosting_.
!split
===== Gradient Boosting, algorithm =====
Suppose we have a cost function $C(f)=\sum_{i=0}^{n-1}L(y_i, f(x_i))$ where $y_i$ is our target and $f(x_i)$ the function which is meant to model $y_i$. The above cost function could be our standard squared-error function
!bt
\[
C(\bm{y},\bm{f})=\sum_{i=0}^{n-1}(y_i-f(x_i))^2.
\]
!et
The way we proceed in an iterative fashion is to
o Initialize our estimate $f_0(x)$.
o For $m=1:M$, we
o compute the negative gradient vector $\bm{u}_m = -\partial C(\bm{y},\bm{f})/\partial \bm{f}(x)$ at $f(x) = f_{m-1}(x)$;
o fit the so-called base-learner to the negative gradient $h_m(u_m,x)$;
o update the estimate $f_m(x) = f_{m-1}(x)+\nu h_m(u_m,x)$;
o The final estimate is then $f_M(x) = \sum_{m=1}^M\nu h_m(u_m,x)$.
!split
===== Gradient Boosting Example, Regression =====
We discuss here the difference between the steepest descent approach and gradient boosting by repeating our simple regression example above.
!split
===== Gradient Boosting, Examples of Regression =====
!bc pycod
import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.preprocessing import StandardScaler
import scikitplot as skplt
from sklearn.metrics import mean_squared_error
n = 100
maxdegree = 6
# Make data set.
x = np.linspace(-3, 3, n).reshape(-1, 1)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
error = np.zeros(maxdegree)
bias = np.zeros(maxdegree)
variance = np.zeros(maxdegree)
polydegree = np.zeros(maxdegree)
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
for degree in range(1,maxdegree):
model = GradientBoostingRegressor(max_depth=degree, n_estimators=100, learning_rate=1.0)
model.fit(X_train_scaled,y_train)
y_pred = model.predict(X_test_scaled)
polydegree[degree] = degree
error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
variance[degree] = np.mean( np.var(y_pred) )
print('Max depth:', degree)
print('Error:', error[degree])
print('Bias^2:', bias[degree])
print('Var:', variance[degree])
print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
plt.xlim(1,maxdegree-1)
plt.plot(polydegree, error, label='Error')
plt.plot(polydegree, bias, label='bias')
plt.plot(polydegree, variance, label='Variance')
plt.legend()
save_fig("gdregression")
plt.show()
!ec
!split
===== Gradient Boosting, Classification Example =====
!bc pycod
import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
import scikitplot as skplt
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_validate
# Load the data
cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
print(X_train.shape)
print(X_test.shape)
#now scale the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
gd_clf = GradientBoostingClassifier(max_depth=3, n_estimators=100, learning_rate=1.0)
gd_clf.fit(X_train_scaled, y_train)
#Cross validation
accuracy = cross_validate(gd_clf,X_test_scaled,y_test,cv=10)['test_score']
print(accuracy)
print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(gd_clf.score(X_test_scaled,y_test)))
import scikitplot as skplt
y_pred = gd_clf.predict(X_test_scaled)
skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
save_fig("gdclassiffierconfusion")
plt.show()
y_probas = gd_clf.predict_proba(X_test_scaled)
skplt.metrics.plot_roc(y_test, y_probas)
save_fig("gdclassiffierroc")
plt.show()
skplt.metrics.plot_cumulative_gain(y_test, y_probas)
save_fig("gdclassiffiercgain")
plt.show()
!ec
!split
===== XGBoost: Extreme Gradient Boosting =====
"XGBoost":"https://github.com/dmlc/xgboost" or Extreme Gradient
Boosting, is an optimized distributed gradient boosting library
designed to be highly efficient, flexible and portable. It implements
machine learning algorithms under the Gradient Boosting
framework. XGBoost provides a parallel tree boosting that solve many
data science problems in a fast and accurate way. See the "article by Chen and Guestrin":"https://arxiv.org/abs/1603.02754".
The authors design and build a highly scalable end-to-end tree
boosting system. It has a theoretically justified weighted quantile
sketch for efficient proposal calculation. It introduces a novel sparsity-aware algorithm for parallel tree learning and an effective cache-aware block structure for out-of-core tree learning.
It is now the algorithm which wins essentially all ML competitions!!!
!split
===== Regression Case =====
!bc pycod
import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import train_test_split
import xgboost as xgb
from sklearn.preprocessing import StandardScaler
import scikitplot as skplt
from sklearn.metrics import mean_squared_error
n = 100
maxdegree = 6
# Make data set.
x = np.linspace(-3, 3, n).reshape(-1, 1)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
error = np.zeros(maxdegree)
bias = np.zeros(maxdegree)
variance = np.zeros(maxdegree)
polydegree = np.zeros(maxdegree)
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
for degree in range(maxdegree):
model = xgb.XGBRegressor(objective ='reg:squarederror', colsaobjective ='reg:squarederror', colsample_bytree = 0.3, learning_rate = 0.1,max_depth = degree, alpha = 10, n_estimators = 200)
model.fit(X_train_scaled,y_train)
y_pred = model.predict(X_test_scaled)
polydegree[degree] = degree
error[degree] = np.mean( np.mean((y_test - y_pred)**2) )
bias[degree] = np.mean( (y_test - np.mean(y_pred))**2 )
variance[degree] = np.mean( np.var(y_pred) )
print('Max depth:', degree)
print('Error:', error[degree])
print('Bias^2:', bias[degree])
print('Var:', variance[degree])
print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
plt.xlim(1,maxdegree-1)
plt.plot(polydegree, error, label='Error')
plt.plot(polydegree, bias, label='bias')
plt.plot(polydegree, variance, label='Variance')
plt.legend()
plt.show()
!ec
!split
===== Xgboost on the Cancer Data =====
As you will see from the confusion matrix below, XGBoots does an excellent job on the Wisconsin cancer data and outperforms essentially all agorithms we have discussed till now.
!bc pycod
import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import cross_validate
import scikitplot as skplt
import xgboost as xgb
# Load the data
cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)
print(X_train.shape)
print(X_test.shape)
#now scale the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
xg_clf = xgb.XGBClassifier()
xg_clf.fit(X_train_scaled,y_train)
y_test = xg_clf.predict(X_test_scaled)
print("Test set accuracy with Random Forests and scaled data: {:.2f}".format(xg_clf.score(X_test_scaled,y_test)))
import scikitplot as skplt
y_pred = xg_clf.predict(X_test_scaled)
skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)
save_fig("xdclassiffierconfusion")
plt.show()
y_probas = xg_clf.predict_proba(X_test_scaled)
skplt.metrics.plot_roc(y_test, y_probas)
save_fig("xdclassiffierroc")
plt.show()
skplt.metrics.plot_cumulative_gain(y_test, y_probas)
save_fig("gdclassiffiercgain")
plt.show()
xgb.plot_tree(xg_clf,num_trees=0)
plt.rcParams['figure.figsize'] = [50, 10]
save_fig("xgtree")
plt.show()
xgb.plot_importance(xg_clf)
plt.rcParams['figure.figsize'] = [5, 5]
save_fig("xgparams")
plt.show()
!ec
doc/src/week46/week46.do.txt
+1143
View File
File diff suppressed because it is too large Load Diff
doc/src/week47/week47.do.txt
+1143
View File
File diff suppressed because it is too large Load Diff