From 77454c4f6922aaa8015ddcb93f46e76713d96dd4 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Mon, 25 Aug 2025 07:41:04 +0200 Subject: [PATCH] update --- .../_build/.doctrees/environment.pickle | Bin 225069 -> 224640 bytes .../_build/.doctrees/week35.doctree | Bin 325202 -> 334628 bytes .../_build/html/_sources/week35.ipynb | 1422 +++++++++-------- doc/LectureNotes/_build/html/searchindex.js | 2 +- doc/LectureNotes/_build/html/week35.html | 350 ++-- .../_build/jupyter_execute/week35.ipynb | 1422 +++++++++-------- doc/LectureNotes/week35.ipynb | 1422 +++++++++-------- 7 files changed, 2519 insertions(+), 2099 deletions(-) diff --git a/doc/LectureNotes/_build/.doctrees/environment.pickle b/doc/LectureNotes/_build/.doctrees/environment.pickle index a983dcf31d25cab5b2a7340374d0242b9beb7f92..e4d56df2c50d5045fb73a239bd191c4ede8d960c 100644 GIT binary patch delta 9041 zcmY*fc|cV~7kB0&k4=#Wh$ss9Z~+%wkVOP>M^H%1veZ--fyb&mat|{XcwuTojw^1N zTQ2pqn_6mCYKDnqRxX&f*=H`fWSPr%&dgjA{+Ril^E)$l=FFLM=gnnDU6VQIo9xbd zzC~nZg-d$Ebfl$gp2=c$>QxjiE)E4L)L;V#y{X+8=ryG#F>BUk<`$k3@7RR2ph%&S_PivzM?FF`bv@bpG#IDD*(X1z& z*oau)-qu>QbrGKJnx9c=J&Ur3Tv?-KHR{}r^$}*@II&aFzP{G)1!eeBP#4+67rjHz ziGF0<82x`WF@#vAB-~y2AQ57U%%$KUkc{;sF#TC286H93Q^) zhzcjzO%EN*pM=Ia3rE;ZN6jmQgwC*=*4T0&eIgA$?sc#B-OVh~BcBN?kV#5B;;H6ZCW7tirq^KWVUo4)-m0KJqL9n)GX77Wuu(o z!^)tJ0R0xIeLy}A^&XI~rhAZM>|S~xtHjnB9_((0Hye`Cgn46Yl+&O!)X##v1?n@{ zI>Uohvu`s3SwT^w7S;H!eEUaQdAp2%g0&8iru!QA- zY*f|=wsew(y>~EzJ<3XAJ**aH&9<|O91D;3B%N6QoFrBa#=CM}XS)h4eECq#gXbrb z05;p2#O`KW_`?w-h`Hq^vEY0Q_bed6Y&i(eXf(T7lmPBp zU)+|T=tGCHRg;rgt6~e^I*bOekSPglL7s&zoKneR3N0+YIE`I_=f@UMKX$G-f!zaT zoNY0Cl5b%*ZJT7Wc^CXGlg%MSBH2_U63ON~B9Ux36wHp` z=_vxR+VZxJjWha6VbB{v4xd@I(&X>iJ0kx@2Qq1y6^3JrRr303T(14`F@@%swhU2;O9nzs%~#H!{saqO&I zp!9>6y+QbZ$?(90ccu(IbMVPj`BX1u}R!qwEz_O?G zAAkHBVXYSw!j7KB4wowz^gGm>fMj+oI-Dzk+YwkL)NKE(2NmGv7iCgYehk(YsgN)2 z5*$7;U4rE5djR1v=F&l$y^kZIfWm<&6^b`e~B{fBgKG1WSG`l_2$mLB0NrAt;(1b8bUC3Dn~5Y=t)8o1;+7eTYKA`u?*@ zA~xr?o1^-HCzbl^4>t+DkK^6PuM$2piJ13Bk^sr-8I+8c&GfG|GF`zJ8)45hN!V}J6x(y~^>u3&g;a#TT6AgNSvM^Jsg32n*SO8c9Iv{apW zw?~5`|D&yWvr5vE-x@+=;9U~K<2sT=e(Nu_NA;?X1U4DSX8lqqnWbQUC$d$+j$Oz; z1@CkvK2+NJcSm=MX0P`ku(t$^i6GfzS&-(V@9jkhPVsmBi_ zxr#b4h|Ew zbo%o&5=~Gx>s@D%F%q&RzdGydXW}LlOy}9ek4iYIlt2;|+P~(Ya|B#47b9H2F7q%1 z1UxaHyrp2lLb8&u)4UK4J6-vva+1gv>s~s%grW!_vp(bvVw31(1%Y&ly76z4krI8j z81{Lbq!T*)CDuS`sB& z>|GC=Td+wR$@_$*?(@+-H<9L&joNICWz`l!DPJ@KdV?Cc~-s zGCOhBHykI=5){37Zj=_yyIm))`j8U@UdBTCTb$NhKmHZTk+Rr5y80Ny@!bNgl!JjW- z7~UTahnw1h*8Ik0NMUV88QC-S3W+5soAuQ{kqQNS|ANUyXutgx9U@@vRrI!iPktv^ z3QoI*5ihj<*NGREu+L3=6$tI=TV$-#_Ps+EDQ(?dOwht$#b5RCxxaBGgx2W+X+v4u z!9qCIr}7^k!j=d>E8n?#zekYsK{V@gACnS+_PWv+1mG_ZpeB7f1R)g`i!{1G+O}&< z-q+hDC;`p<`dhS}ezpTmm9iccN?{Gb z%FN-tyO8ZMV%v&_jW4lubDr~EN+yh2W&gw5k&@`y|C%x!3 z5VuCs{wV$tg(CkxnqCL((HPnhjc@h_$B}BjKA#le53r@obGlkZTO92~K$P#EFXJh! zBuM%*aCQ-h!>iL>cj`|Qg}513mI{laLAa)#(|vTeAry|2*lJTE&62215^YVnZ9I)! zW~MQG+c28Whfkm(y!&u^l)IF}F>c|MB)&U^Ci0Y(Mt1T_rI6`CHtS=ap|H~gyfK2l zqu@KwQpj0C+ietu(~W@D&r=9#0dq&wEef`KiEfZkrmxFmDC|;Ul{F5ci?@9Uqlky! zq>U%QD2g5v@k2_VJb?zmk^C_XV&g;#J4nFjbO*okX+6oUTN>Ic(H+de%0Yq-5Bwo%9>qF)0YsQ9EF#NO)@}eOJLP zyN&5_*i_zn#s~CyMO8cM-9C6PouITIeMpB>*$w>6J*~74_S5$jgkQR)5^hd82qyvY z@Ck=090>(%d<1f@ga<#X*QS3#Us2j?N9j}rZO18``$V&DC+JrSHmRm375w8Q{Y*j* z;qIq@e~J!MROD&gqoQqXEnTkQtTS|_YLI#l;v?s*F*6RE#81EdogwQPKhS(-n&_%U z=`GJ2!OO3^X>s}&7wB!tvmKj#^}0H$soCE7k-93_^rw0l`b#|=_**?3|2s`5eDN_z z2RCic@%h(ao5j_r?fB<48bDAq>w(wlUn~gh{TK zP&&x}K2AbV3;)I!kAS`pXpVxgD`4B9mS22A-E^-<_!{|7p7r=?d8o_C{6^xBy>_U@ z-ocu`>Si+0S`yyONoz+E_z06WM0T;*N$bRWYT8A9>U$c?dpT)hwr1gczTXLmAd}XO z*ETkYZ6+;}4{!$J6Bn&Jf5v1GxF-BZ6A)wF0Nrpg2wWUq>Z)C2e%pI<=4R+ud1whd z%pDqh)CBtM=V1_uO|?G!yax~yJ+&VEMiYa$?gsO#{H&ds-R< zToMiE-CF_iLu+jS9~EE_@RM~Ye;5G7>Od`k2edH=bCA}SA8G@HSFq;KKMFL6dBIvB zj|c){PFt-j4+}O3I8|79CyVCAHwFXiAEI^SJKGxUst_>lV*%pbcHpmmAqElB9{lxX z2oP{L+>xJXXAmuuphZf1AlfCvQl)e-h`q^LEI-`=i0ggek>@%Z1e_FF@WUNo1eu}0 z?(C>_<`cpUHX;=KUl#^M|1hl$KiA11_J_fMUhE9S?06DL{^S$Gj8Oe6p7`*@&d}r4 zF2F~2(L(uyu13RsU0}HuhXbMaBd}843<7@4hw)d#p_x;6VDq|xYrT6I?3Hfd+D$!x zxZE8=?TelUk>3MCZBhgfxdX@mp4-bH;6`8wztandm_ei?H%A!+Tn)75Tb`GaS*(IeGTGj90c#izCc`y2X}v-U=Wl0YCZJqews=o`lbFFys$(Lfm;ke z;Z^T?pc7R0q}|B30jLAJsmzGA3WC>S@3B@L8a zY=vF)fF#WoZjOe*)5jzmPk(kec=lE@3@a@Ky#3EGgQ!b^dCeaV1Voi5zn!Xe;PX-p z7DCID&wU2Ea!%DMc<&L~S#jH+>?*GMlikEUf3mN*?v;blbeYv{bXlxpAXIPGC$c{+~fm?U%tQ+ukfM5`+MjbFYlA7xVeW0)!`_*ic9-s zB5v$Ki`Vs_#JhSh#f$pnL~%=>+*dz1QVaBeGkD$0+S*dQ|6gJ)>e8suV)&3Fr6~=h zc@3lm4J6h;db@$d8%Qe}NUID9KM5|et}#R*ZEGOC*Ff6WKswMsI^IA!(LkzhAbnjg z4Z_FQ)Qc!xY#?1~AYE=C{bWe|@B}R!e#YFIpk)x!g^xX9PwNNEks-;c7#LAAX?1uRc)WCm*OV zfS|$vf(io&D*VhduCk=0f?DMlo_KyfT?-`f{NHrVUu=YOu@B0`u*<}#%fz6|#F)#( zkjun~%fx`o#CXfZaLdGK%S7;(iP$d_pW|<%$up%HV`3_A}STVIB$46>v z=DVy_Z3@t?~vn=zGR+gpy&Y8KF#6M=f?|f%wXU@!=doSbMOU}ElIPcCI zQxQ>9-KNZvWhp4L6l9kH$u2A}v*eZ*6y?=aJ7(u+l>mw8RZ~q!PArx+x?bz8k4-fN6A!&OWiaJkGPVFLbKFsGGmX0^@DIh6Tph`~w!XSX{>fLu>vKi?dk#gat;_d;<%NqWNbm z>aq9*3yhxmCKebm^RHN7l+3rVID^G)y9y{TA-sdd1uX7j(S*fsSYVjU_prbKneStP zVKM)X#bqr1zyjl8{u7HYv3P(5hQj<878n5Y-&mZ-;vp2}~|h}Do3uL_1iu^N=Lj|IcPSPgjx_6vr=u^K$;KM@QA zWHrRxJs=nc$!a)QbV#t%6@q@^a9B``menvQ@`zvPa`9Tge|&T5!3 z^O#^5JgecuR-X%o0kj%?wjCD?gJ?B$dmRI()$mt_WEfqm;mi!lFv3=YlX6;kVwA0h&sx=w3uNskq>?)J{DfPi&X$rn zmXI%O>#|}9Il;b~=*KKM?#vIoXAzSH^-0df!=O_H7du= zG5sy`i>+I=N+)b&j;G^SPLj1}zvXmjGyGjDslt}$c!q2slF{-qv7zSFn8!lXIW^tP zw1(uhUSKC_(EQPscHRxN%j^x&}_wr1vdZ8cta#9joSZHEN zEB)B$ypb%v*u-`ojAD=S(jX=q$^w}=-^xyyO+3zpbY}w!(%A3$CiX$Wd^QB?(dJSX zFvZ0G9ZCFIo5^YH`^hHmQbYpSN+9o)nfS2jB$o9pOk+8vCbq3`IvWKZPCLj{ws%Sz ztFxH+;}c{c%PmS{k!BNXD9U4<%1rE~;#I7p#l*ZyX0zp!OuRUV#<4pkNvx*C#B2M~ zp={06H1-63dfPDCnFW<5u@8z&Y*A?qyFS&#`jt&&e-xW|pCanXz9~y$ZZO|O%TiWd zU}ASHn`Pp855guB&mlx2@zfy_iRTg`k$8SaBofa9L?ZFDwg)5BJrdn5{c&p`{u-xhenZj$`OggGar#iJj)S@#IqieNIW|ciNx~>B9VAbAQFk^ zD?}pke2+*Zo;!#{;`tX)d9|IT*a05#Vl&uQzY;|F`hL`PMKE=Zb(h{-sLkWIgRx3gMn}-Bo>lQfM-mh6Gt%FxA zCfuQl6vLLz<0DoRSN^z)IM@!a>`ssxZI4#X)KJwLvqb#PCK_rBUz@K9+t!6D!kK&S zr%~*R+SV4dF8nDp%Mdl@IHhJ z5%+JeB-(!2ZPjRKC#A3L=ttu;zn*&_e4@qKy@xfIf3iti+fH?pq%ifiw!-}|W8vob z>58Z92M)|5Jn&zL#Lk1WdC6lv63K@W2vVbM$>GTwYI78RDw9xm+pc51s8sKKUa6^7 zCt&EJMbIf2t3ZcqXJ{zBZilGypXsmx`DwvnQLizmR0wz>CU%OUXkeU1cM z{+KI4`~%pdD0%$7MgTVVUK?B8Ll-KnxBlHM<#!)LgX#swy)mS-kklMX#tKb6qmUUA zz0jJpm+CA7fpY{Mt4@w2N!ng+OH^s=>P+C|Eg4CQFdfGL%Z|V{bBFv$9SEHi7UB_aVnMXz(Mjqg+v0XM2wLkR2u z0jG2)+celMjO^Fo{hq{~N(b)q9L^4yNKl_oz60sz*L88F}$Zmo7*=Qw3?Kg-_mcpkg zX&zv80aOVvAGl17e5kIZ4@1Bn7gkdGz0nSraib~~&pJZ1BF=UsYu<~&Ov zrvW#zN2|QlhUdu3lBxqnlR27#KlSYx(p6Ku$C4)+3>`;a)!-*D!qh}h>frH&Y4GOD z&2U+!-l6I@k<@Aqkc=*C@aJ6eodz%Gk&6;?fNj-t1q2Q%4pUt+nLx5^ng(-$mxKb| zd8R&>Ka#Txv)6qj3Amsy(Oi?2XfdEFH1!!xJSXYq}r zg0C*3ZY!+BUj1P@i6cl|)v%f5B?;Nm9}Vh;S#ZFl(tQr`q7sg(Adr%TKb&UB90I=n z8iCX*V2=f)T7#!v#{dcIltp9>;SYw>VtC2v$v0QQM)n;IKMD_@Sc1@~4p~er8a!Q1 zAT0{(gg3~u8a%cX&xNp#Sq^7j!s(3UXx7^`6$fjb+bdew+ZBQ z(b)%U$qo$`t|zftv%MSeNDGIwP2@ujI&a2M2y5(?=8>)0N+{*;Oo4@icenlgRg1oG zS9Xwo1i4Y|xeIrZfS1J*a?u{AJ>xI_)9jHEGsf(nU(DO~v&T z4TkI|@Zb~B`%^d?5`J@ljMRJ+50Mkn%1<99?rP`}0*?{V0Qfo$&iM>;p0K(d$I%@9 zg%;y`=VE^MPpTdMjVH--1i6uSj#pxNuUo`X9a2liX~vcP6%X~~7oL|{Zs@Pt z&R|9qFryy#rGOn9NM9;p{&})fvvz63wGa;1F5o5*u>5N>QiB)2A!UTmja1y#370X7 z!l9{&{Hej~SMe*O04X#t=7O!721{U&Z4VfDU6T&aY8@8E6`)^m5sOPaO+eX>NeUilr!( zCZy7-TA}bBApC<`Q#hLfeqm3yYH)!A-JrpkHWU_3IP7jq+i5+Vb*3qrHQ0s1(@!{T zYES>x;4UMDts<-m?#=Lo2c4=}hk8;t{=(sw7lmaPQ0+)z#sW_2OjER9JK#%U(FRV1 zeYe%0&g3Pp>-(-x0ENs4+^ALr(s_jWEG)*)ow@9AmFvD>8bSbIhIdiZy3o;r@mp`w zV70L;%@98{!qCOF{SfT@wbS)uQ)p1dD zBGmkU(ex(JZ83BJ(qChd@{4iw7FdtR({AW|CxI4{6+tj1_#j)(?*wUUX-TB;Tw33i z%IDIlezaUt^_e8fCFJllZL8W1psA8?d3hi#Ew)ihQs_d>-(d({tijEx6gH@E@J*vW zlrKF&VpbSwJl{TyX7j{)B0eg3#Bh3?Py7xJ$D-0SzIy~s<%jO-hr>04!ovW#QGM|l zh_!^bM^bo65Y~5}qmaS`>@}Lgp%$=i424}T;N-Em!vqW-N8!PYP-e65UZjxTg_Jj* zI#b?f7>&Vqn*`RsEP%JpgrPVOr$Oq}OzKY%x~dzqV0|Qv&Bi^3Rt`JQNv+SJ@LUkw z>s8uW&6q^pG}p2``lBWtK1;pS*9vHY#_PwB*8J)P>ZV>d(@~m#S|NpV9`6rNNO<3R z7Qc!o@F{`vn@>GhL?;oXMm45{Iumw|7sI!51}CL7l`U08T-7P1Flb;#b$uBvktm>? zmP)kw6O#W##mBAr%T(zx9EI{mU6mYqNG)jZA6P3t7*MNwQtj-60KWD zznAFEcjz98#%`qWo`AmwUzxH$wr!oKx@I$lS46aXZGrtIkh+z^OF!5+oXuc0Z#zws z6b@*2_4W=t|JY*mE(&j@8T;MUd+)>M6}GLrwYhLOL2hc!M|6x7>TG&k2k)hs()RH_ zI-H7T@P&I$+8%yF;Ts#fhfiGi&H%J!#6c?EG7m$t6JqP5bc;j>kLkAT&*>}DcJnx$ zCXwYN-7ZnDTKa`V&UN&(M8BS<#{}Y&3#g}h@hlxCg_v`2?!*r@G|-h2%|1_8OXrL~ zV8sg>wQ2A(W@vlwf&1 zqq;tVM}h1>@Y#=_=0hEx6txon*t|ntdj~7tS~pXP(vf&}$gsc-jDLAq?xgZU|I*w- zxCJUMD&w1|GFtR$^fVQtOPF*C0~ek=T=C-3iqgavQze)Ww^Kw!jJ%Dabms@{KsbAm z_TgW&)&*QE{-QkyYg#M4c!5C|g6)+U{?Guz*l%bAZ|A5Bu!ymIvm*!(E&)2wMi*dB z6Zx37N)z+kk-!U_G&|d|Bb;@e@5-MzL7U!A&~3YRdYfZTN?-m?I}k2AE0O%Ti!S81 zQ{s40dl2ThC`0M2Bz~k98H<0L8Nu(nf;rt)3F7}4b^lYYN+@6G2130NraIqU7e>1& z5nS;AA=_Q?r$1;NHMuKY___{Y{?|AG!mG>`TIVV0*7#>2gJfqE&u`A%;T zjNXb5ukWY}Ro;q;PwNCiRYzqA=bd!{{&F;&C-{IcEuIYIy?u27K4jy#<0W{geBcYJ zr?1kPZ}!vGCB8}^_w@&%+E4N3ApyGZ#82_#Y5)lT_$xj6fk0h=_X`t$FFe|& z62h%vx>^tp3*guj1T~pNbC+;kfUoyZ-ZKK4tqcdXb0mamdoNvmGZKESe*_52d&Aa= zi_`_5DA+oCB0-ou5W;?}w=Tfn0EY0g5BRv|Y3$+QbRdV7%xZDS}aY=$Ml=g)Vo+N;kclAf1Sd0kGn&02uK~09^4tAU#!F z2Basct|JvcxRl}%6BZD=DaOKS70;QBdJc6jXQ(1r_cmsPF;` zD%?*{;eLV&_Y+jOpP<6EhpMrUtnsc1C|)!{h5HFA+)q&9eu4@wm&VtWmseA>yjRNN zJtr!DB#V!jsCbKcREc9zDP~bACQ&KoP${NRDP~Y9CQvEHUnz!PDMnu@245-0UMYrN zDMnr?23{$~T`7iLCGN;7#Tlp)L#{M`849DT5+kk>1FjU~tt`(^PH$D7KOOJEvS113 z)Rgcz(8$-F7#)z#8V~nLpx<*-C)%02uNIn4h*g<|x7B z2oKLu!h7uk*^EX0fQ2>YeaKXuVNajSF#g}TQJSEru|KLZ0uK)l5 diff --git a/doc/LectureNotes/_build/.doctrees/week35.doctree b/doc/LectureNotes/_build/.doctrees/week35.doctree index 9f4ea2c83cfbe46e7aaa339baf074970aeed29cf..28f58c1ae71eadf3725eaeba59d478b6535298fa 100644 GIT binary patch delta 55946 zcmeFacX$;=`#0{)E&);?98w8MP70(~N`NFlLJtH&=$(WdNJ1JZRH;fw+5#hpAfO0X zC;@^f2nY%&RY5_ei9RSOMIOb-}m~x?>~>17biP=-=DeL+*5XT_Ts(9 zukQB{XVqCP>=xgtkJ6>5_1k!u!y_a6^y(KG8L=wA?ub?4`Hhyeqc<9u!&l8+HG9=` zDDuj`D6LrH-BbWVMg>IdZ=K*OX*3bV1ZTFX$`8+vpd9&8`9f7LP@)I}(9Pi3>Iu$1 z2>l^l$c%BxY^QQOA$`k{hZG;;A^qSIg+nq)qfp$$RADY1Yvw9e%00AlO^8C1bfaN{ zONs(m*bqs5n@37*+@jnfdz4g`m*!QCsVpchA=l4{0iLQN*0-cnt}ZBoVoC80>f`1z zjHSUj+!GOW5XHGx)oh2?fS9>rxggyiA1n1!@*99PIiMp^lw>O`S?$X=R z@UZ&m4r3pw5;BtN_rsD9!ctCRjARZFoky#8pHgCJL_~cBYH`GMFp@$ht&gmayri7y zDCb<2L9X#>QZAW!Dr*cNsUjbYPD4Yq zdSYVgqk*&@m$52w7itvVUCmUXbG2#^Dr^tf)WU5PUcI4ehSlrqlxb{9-T zX)VUKIWxH@&Sla!mZi_wJWt~tD!HWe#6v~tLhArDRGO5Mqg>FRhpih+ZmB`GKs%-; z;O7~rhYe6@CmVezLeQ*u>91558W$`D`nMm9@Ortn_0pOSiKg||0$=S=jNdc}yj2%? zy|kcXq5?sGsEqeJhA1T-Ivn)#kv*Sb*{N&V8~C(ASRPOo7NDrG)6+F!rI*v6;#;z; zx1>Xz;!U@#WqsNy6F)M@`i3s+E$M~M2?~UV&df;$c@x*GLnBKyBhN09SuY7qC;*7& z&E0uNX-zCPu6214nWdb5$!du0c}sJz+8(cT9fnNtb*y3|p(p{jVfCxAaqKgVIbJ%@ zy(NyX!(t;|a_iB@7DUIaI%Na?y(mrzUp6QT^ zSM*Gw(Z%x6t)5?DQil$L;-v?@T=1$oa>H=XJcM7XBh9Xem!4+2xUN#{K&@Bf*BG5d zsG7ImtzfJ+Q57LAYli6qYx7;mO2r==Bz)9Ejs3NMpA+$MvW|w7d3q3 z#3BnJ>B8s)spmi!U*0AVJgRC1e4bB8oTZJFF(4Pho>uK#n$9W)VmqRFfc z$jQY`P(4^7U8wSqS`Ujs&(qt|@?q|-6^fWN00?;EUGjTlq0&lH^o?=v>`vMR!K-f zcCC^f1$}Cj#1?wnlO&f)Vph6KURf<1VLloyEzN7{B&#YtDD*`4Xso+( zwXRjGhlDP5DAt z<$ROk^T$K-un}&8Qpv+IF+=)#*dpnt5@%#xdbZRRzgd^27`550Jg>!*tZ{1Dco=2Q zsz{Kombn<4z{9d%5I${S-xDAjQ$R*VD|8juYqg$}6$cT%V_>~lWMDmHdSY$oS0+ZF zXQZHM(H_buHRG0~aXpcoC&l3>%t+GsNk(C%f@#~yj}gdOYBhN~%%AzR`skpBQQ;&{ zD$}NfBGeLUiXrBaNOXhIfT~wQJ-K_UnOXqLB|+0tTodZa0YMX6b-}dZ2xl4aP4(;4 zW)7e3i$();f_E^C50sH9k0ULbej1k;U=jy$l!^>9rkiFcwd?ajvSOq#0?jknDSd{< zJVUxYGa1d0%(GH_`6~GUDxv|@l+4~B_9nAl6rB-YaO6$V6>l)emi?Lr5B8yx5!9sm zz-)hkbPUJ!rFwHao8GpTGi$pLAnlynge=%D_`)Bd(l_(8MwI)B2Klz+F|WA|0IAnJ zBE2-mdFKRkPrB%Gsw1bK{|fB9vv6hVs={C0;b3h z*J}CJz&mwenZiqd=r}EKQKnu(Hr>P?(&j}yxuV%i1P>`>ah6`h__i(X0VVR}*>#Dp z^zS99=xu4tl1N&*q9Y!Qx`GtZWchrdj;le27B6&Gusn1h>7!*inub#q)G-auUXM-X z-iByOkPa_(!AkS+sf>*5SDaH+WS&rEsh}%tk&*4pgYqrXz-387w0V-WVi}Ax>r;XK zq;HqSNds5JFyR7A zSUI3Bij>gPRO!LmSXjvmT(=yqcdhHo{tl4LFU7)jpO>mx6P!S$V&rpedUbJj;A zD7cX2yXJEtRJ!-F3=$x<-4G7txf>K9G9^Ovl3R1*D zrL215!LO;l{SUncL_tRaB%j00;5zNsRB75_N<_;}9u9>vD!%!BwXFF4ZcwI*|41nx z3}sZ*nj7xOMc(6eKUI1Af7$}P zdHx&@mEKefi3>(@%+mLHY{z4^M@Wn!~NAEvFr2H$MQpT0ZYV>A%NoltmS6{m_5TU5* zR$op8=T*P?)pGbD69njEO0U(ChFzmg^{s0iH34Xwf9qO6_2lc{Qg2g97vH3n9lS}U zak+Jd{T(3vdMg^PL%-RrTtEIM2Srs6yd6&Knfa}!YW<7-TQpuP?*MFi^4es}Ijp90bY3KbEY5V>8K$P&{1BEpBAtmkm za3oysdPv)5--iKG=%aiTRsF&vALOj$ibw0~OQXN@mp=JAXp9DxB|Jo8ou6*(WyPO70&c8WPTQ>TAlq@)HS7-j1 z1*-S{a~Irr`X@YUpe>dn6hBpb$Wrp%r)DY3(f~z}9}x;%osQJVS-lofsfwQlto{zY z3ZgDXC`X;;=DzLx%<4=Gk1C#dX7vT+Ny8X|o8|=aELIrXl3q@z9?j&6H%XMBVv@-M z$_54{mPtKPyE8yUec=wo6ZuRpUEfM{Be8W*e~{$4I*6s{rvc>GI_O1(_>hwZvG^sS zp|t)fiviNJE`lH+xpmQIHR8)JNe_PyAW`+u3KUgys2-XDb|WdyXb=2=OP&LBRJfpO zxEbb(zN0rAG(dCUXGH_F5N`T4L{s7BnucgF+^o6pj-oJJ!vc*6h9+iDbebubO2WOU zRAi7BS_=d)fq#!OyzPU=!Y`v5KUGPvdj!&rWr}V(l`j+QdzQ}C@f8b z(P9vGTQFLu#bS9^HyN)W6iorfZ$i=Upvbjm;CG-kg`o}9OuNGoL~%`r2(+1A-ikyg z$m2xRiQI`oeQOr9KnD;k=%lqmf$)>l3XRdPHl-mC@>?s^*5u@2HCLn$Z34)JWTa1G zZVzgp&HQDo#W1wl=qnbqEpHD}R>WwHXFNKAkVshiQ=-umoXUJGyB@a3r#=(!7o6$GqV0(0z9C#18;)@<^dO%ip zMqSVqs#FYdpNB-(NZp7RiBA`F9Y`q(g#GYD6i9k>MVaKoL>Nr}GNly#DRa_!F)3() zT<~umY&J^fc0(gg&Yo5g5065SiKKN$+8n5*UqdzEoylytft@{Rg(Rj8a&i4laXwoq z{qcc!5)Q(m{7aQw%XBTiJ>Ldcz9nWT2FFfpnok013!KL(nE!=+WI!9oNm#8sY{r zp&M#kb21C%Ak$7OaY%ADC|twjQ73t?>fAd$$ty5Mvc*WpfasJ1#d3!#3vMakd=cWZ$`s_v^ZKFYaE-ZA%v0#gHfz) z-UQs9v8kTm*}TZ_d$7L^K5}sgYJs!t*VlEZ&HC8b8t2;QyE_yO#i3qoeZyK+I%M@E zeIDZgJjnsZFl4qNGaWSGh4yu2&oiNFni+ccf=si)Q;vV95onA6;VsR6H;RxEXc&Ie zi#>pHySPa+@piA8gptqznhsh!h&};4W~lfxo$;U-@g5DGw+%ij@MjMAZjMIbw)i00 z@plgRVjH4Be2{7C#!0f>i#*9eVfd*R3Ccw==zY>B7X{$T{F;(n$l>q@dDemkHmJ;p z-dEY)GuRH1yG$hSjDo#G@Fd@Y+Il#7L-f+^S&c3I%Veue%rNJQlV0_?wze zxx@Bw6+R>1eRC{IM#bdLSk%!55y(v+Jd-}-&;}guEq6Qh@QI6TLjg|DlrJ;)|kJCjVP zLLN2YMKCNiEeBNN1_M%DjGEyWz4`p&p>r=MpBAGbc)4Qx>+eAyyThB@T!-Ra4P<*t zP)i`A%^NKzbxTnzBhztPOvg%5OS>{%wl7mj84873n4>utj~#nD z*up&h(4nUpk!yq z!iO|+!I6P>0z18$soR{~tVBLaT>I6EYepk$8zR%$4eJULRb^jU4u~l`7H9a73ypCT z@^ckxg0p?(Sj)buX`MlOPDDXKM)P?pcjG^I|Lb&R2bVcEM=mnac3$kr9E?UbMb=GYc<`Y>B;c4P}iBTKcBw}p+2H;EjYWNDec0IVx(}KCHu54LOh+BagIO>L&~H54 zi9JZdY#3(&72SwQSf2Y;jy<eX{m93J4ynq4i`gXW)!he+(6dIbaL*5MX_Czk(s=_5_#Gi z1#rA`ePw%ozY>)Sn_rggIZr|(@JU~CbR|wmxvSap#D8qhn(S35T*O~FR`;T*G0lxv z9I4A_WOXmsQumE%pl<0}3HFtw#xNx7vaz zay4)};3XK>{zV$ifw3Z7&V30SuX26<60D2`+T!Z49>pnK@RKkHmfyClN9}==1v6L0 z3B~v=UPj}DV|QiapL`is2{S0L{J5Wdy#TqOZZ#`5z-S280fJtGMZUr0U$4L-pL~S} zH(^a?<|dR;tLX1dup(rnn!Op;@3bPhJ=fjBmocy=obgPKCL`dD*^Zj)@Jyk@+hJu~ ztC1CT*#R>$W6_BnFkaZp#CGE#yET!UNx~aw6&Wm{Sn~55C=JykjrJ*Sa>ENW0qXe4 zeaE~v(G>*UMWY>HB`-4TZLpGdm9PT>!)qr9Wh{DmC-}lx6ub+j_Qs+uyHGpq<;NmQ zw5~aBHyVynFS+Jkg89mAew zeNClM_Trd!YF@vpc)jmIRDtnbrl>p!K#!QBZXP-@-0pb@qVKNkqPsLWyvf-^(1Nuh z=BLcVC|Ci@z#dKI9@Pl+`(YGns1G-{N!M|U;05Rm_Wb+EP5t}P`w$(*TC0vg#2Jfz zIRbHMESh%Iit5j!FskNt$D?FfVhu8@9W(5ts0XXyX6R(i+`6#}aP1h(6erkJCgTI- zUrU_zAE0)oyVj9;=K~ai7dPd{Mh0P|BiEYR;+u9H_+GZpM{XQPO{uaMeuYM&`xXL$ zMNaP%Xb4&NG9-&jCs12-WX(b7VS+Df0duISoF+zph`vDREGeT6qu)sw9$i*c4%Laz zHYM+#MBzY|b_BX3xU6#u=AH5-{Nx;gbn?b2=oE}aolc|SwTeDF4aw0+756d9t5tOJ zW0U~Zux7!@p$gse6Z9*>0se9yI^rxis=2>B&5G8g4j?Dbq8>oTrt|?M@f^xgt|%`X z{GB=ngFl&qmgU?6Mf#kF8Kz#w=srGgprgByNMCp*ilbw^7ccO7458x(dnE@-CZ}{kHw91f;3n#$ zLyW!!OS?b;B9Yh=$R#l*K=a2xToSB_{02Q=2Tu!N!+*1#rIYft;0Uk6$pMbm1R0S$ zQms-5?Jl0oBdfo+KSSn_z<6}2wC4Wzs6M=;6Tn_bay_EO1>9%m&$x?5kk?Pp%yt(R z`u8#0$GffkmqxiqK^l4wwLuv6CQ-kB5J2|+h;q40Hfiql6Y@8Gpcs>jQ5G^^8bW53 zUIT#L{s|ViZ7>Ghhin8}P`Gz$jEpbbha{^B#ZtuG`!Mi>BJi#^N4!V(aTMwO0G`*m z1@difa|2Jc%_%CADG$*SlYgMq0Ll9in5tPIFV&t_8n=HSYjQUD4pP1(#-~2J;rU4H z&nV4Seaw74?DKv9Gir}Cuo*wOKtQ>QvC#y(i;ZLex##r2_(_K!4k0z#`51FDIIda_~mafwAvC#qau}q zEKp?l=PL^}CBMVlKKRQ(hb{l&b6{Vu!Y=|HwR}b-xBN>Aq1~3hgk1X5e#_^ORCmVx zg+`eEv^MdEzhEpRn)%?pwL=SrKLd&IH{{AcnWbF0I|XF<-{>b(Q?pfbSpF0cLm8qP z*0*)UrsVNcG@%}QxxcWfnQbje(Qp6f4%qh%4s&j}T6l(Su6kzrDzI}pad*K{_R{d9 z3(OfE%#NJFh-7D^D1>&Mv6TGcg6%HYb4Ueb^kWzVWx3)N!m;6=tQj_R!!vP-Sss?) z-!#*YH_Iveh8u2#@v~;SyzeGj-6x&gakXiYl^Y~@ISa2ec)>|$UPLA~!p-m^uz!J& z;HBRfuK+;47a!BO60-cT|C@J#jI<#|-8js>{KvR#V`G zM+hajqIKex8HFsMZAViVCo3l54<)LQ3AFhi{U4v>jJi6Bt^`1hJuSfrC`b$!pgy#_0I>%VNkCb9))gFngs7v{B1o9YIZflOA)?rh=a}W_)1y7 zRG~J7;oGLC)?q;Rl>QlnW7Ulvd$wwATf!_g2g{Z89p-8dv51|LK46fFiPtNSo7K2|F06pKfirYh82J^?3_QxRBu z9p{#xk7~@RBq0u)@zh{8KICrjmA<|%4ju!qW0k^gwP|6vIr%FdB76r`CYoeL;wBjK zrIdhQK?fL_x)IW}g8UGQ1IU_0T!Bt9a=LR22Ol7VBB6X(3*3rYi`nmR3p@aVgJIIW zRc?v!KuGcWwP1PrV`+(>LU07LargaTGOsm`Gu^k+S$?19DKoYj2rYFqU>uu-M{_uA zCjMs<&eF@+@OgF{oL#G^UR&I!R?(8SU;z+{&g!^o6rn`{`eyQ=9gZNu?eGU)=A4o| z*s*CYh1H>Qvt@Elaaob2f_&Q!cfd&@WYcL_fA5lvo2wTngo*M>GVX0GLQ6)OIA@ABXEKnClu(!#s;N%xSx>j3Im=h)mfIP5h_Cr<`JYL@mB3UMmNkC_4Chb%9`X47LUEm%K{ z#pZkr?eT7+q5)%0F^p)f86|priIoa7j7-l zr2D!cKqa zC~o&e(c2!BB|k!ewQ?YJlORW!UZoDcltJ2mh{;pF53aUfn4 zP8agY^^Mrsb&3MQ;%MAL5#HC_VG!=(WsP5{y=LBGu`a>~!&%Nb7*2LB5j&d>T1mqe z0#6%+x%HO@27F6j5S!~TEF8XnLCmUERI*fTXaiktVE639uURJc(xFIB5^Rv1vrG)4 zWpp&p$zm|Vy=7u|gk2*HnK9H~4JXA2gsc#YO-& zh^aOh*@`%Ez*j}YU_714;x#XYq{u5rfd+0kfqDTx}^UGaXxB99&XW-ZFa+eTH#;QexukBnSvr~ z$G<#cj)3>~^@9_0TWjn=k!<7(KM=ug+zn8H4b1a5iOCqJ!Moz{U^*?5oY(@)=IccsY!bcPxE@x6XF137H8ESSkrl0fO-!>@D;v>O7?d;AN=9!L z19dF2X{(rR1BYeC7aYJWeO;VOVW^R~3KTzkSHl@O?7ivyoQW25UO~l}s)`Cr9trA! zgBsFfuj#(6JWEBvxDscQ&|UOSg2Qwy=1B!`)DA4%RaTXkn9D5XV@u16%{k`c(sGNr zykJ~ud1+OJxi|-y;h%yrIYrK7mPPQ0D!0H3+!YmYK2I!re~g~>-Nk_jent^E!@nl` z^))zlU^Naky=LWCHcQ#ZEHg$KH5KY^Rq2^qTmOI4$(A(K=dfr=mG$A3ovWT>UI>Cy zUvRQ0zL+_uKiGOqX`aOltD*2Fdj(juqJS=mf(MG>D8I^bFlH$%myR{(mrgQQmYUh! zV)~MMc0t}au)Stqa8YHxxwNX>Tso;lMx~3bV@iw5asa$CAK)2D#h8Ne1(h*HmYF%_ zidVoX{|{vG2YEGNm@K-{F0o9MS)x0TOUD0jYYxBlzZ70$CKsLvVWe~+8bGFhhg`j= zF-yxU%|!(zmYniY#o&k{XJ?W#2@Rj=9AZv0S6D20=EACqN@{saNktX2BfRHN+ekTB zx3t(?QdOL5DF@H7dlqv=PO-%dt;1}Ar=3&G-Ek#q9R&bhv zrF@K~tP%pJsH(VxoD;BTlo6qwvl;#yJgi-`TW>WKpn0uF2TH47cT*$tnQCHhP?Duo0Oog0j7n7G10LRq^l-;g?F##vD3d6%718D+A zC^bz{?tv&uNs(oor6iAhUk7`N|LxS2e(%4Xni|*6|LxS+@a4aq+J8GWvsvkXX(H!3 zOFgF>d*A-2v2Rlr|U>&67Pat?Cs|*9|5aFUWMO9I@-_6h~!7O z-zbE3_w#&>;qwJa<5FvDgkk9K{fRoY!MWGV0kK~@PF&P=JuVZFWEq4be)T{J+u<`M5_H>v6!@d51vT9F$H%( zU1U$OeU2;N&^(nM#g=j5&DzwdxD<`8*)bJ=hD`kw2dcJZhoq*GvT7WwuHLTFK~!qC z%m=5BPRGF>W3A3*>?fP2Ulb{sjw7tAxJ1U&7Ob0ZG@-V_FbE_8yu6O}V@V_Tpbm*baU zv$K7aiL-EkEk=+goSdBno56ND;2tp>cCy&eW!Y@}E;?qPk*&*XHf5{eSLVReC@6x6 z@a7av*IFZNnjP>Q-imKU@ej*3U*yAqo$)pDc|09rf;H~*%S0!y-#8MrZ39~9T>J`t zCYnb(W)4)JJwR|Vw6J;j1(SPp?MDxE?HIv{G4Jc{x!6r(c8?}q=EH_UYd(_p2kZ`v zw9g3NgNyX$pbQ2mU4VDs_6Ct}m1wo9B0swjFE;hI5}Eief-g1>G*ta)wZ`2$n#3*@ z`~s~RxrcR2T9KSr;7FzO3$Y*RvKaS3aP|pj0|OrOa_?x?$hjL-%pc+TXz~)gAI+ld z@(G!bby*Bb`~L4bAYB{T>(^4FaOdD#eewj>K1}|r9rFBKrE&LYx z;3ctS)>0hnyNVu+rW_w(x~&0jh$ivNVb|7%XqpzS5hDp}S%(bD8~LAI0hs{!X^P|S zfc7lK{kj785FnYcU9Tl8abI+UnS!t^7wlqIb^AJ6epuB=!tY`H6B7u2q2G>6-W;4UVWPRmc1Nj|}2u}=)^CVXPZ1A-&r zk{!e0#4Os5n76|@z6CGhAT{$DYG!+7spRbG_$NSST@5=TA0?4SXY_Vu1i3d11=`}H z?XiagzEJpfm98aAIdj(GFdI;0!XDfWjd6h0`6V1_gOMKZ4_$zL*;&d457I3_!Ru{T z&ho=z`ylKMQ7iYr>Djv>ue@xZkIa9WcQcdW&db=(h91Oo18xO}ui95+$p+XyXhR5k zTs$1UYL}5^1U-adt9lgm3C;POm%mnmnr;&z|K5nV;qPPkES;FhHx8Pm*KC44e!tOr zqe5&`Rx-rS1MlV@+> z7G&q^7|w*hfg_!{L#fT%AeUHJWf96#O^YsMUqvgH0P3^nOZ1y4WWo*X7glKaxX0i0 z_*ppE62_*efE&V2GALreCOx;~Bq|Yohli}%j>BLG!<)-U{nvKL>@#oTR;2e1P>1qK zaNdC%N2O-JPj}#(*tE^cC#)adWvHW2*LGVh>GC#ihPE-=+F;jDi$xzg0BN)ncd*0V z0)63tk$n_&8wt7vn~lX?Fx0ZvKpqkN#6Z&2$I!eAau_m)iP7-Dmve(5Su;JBcnE|^Yi`hV+nFN~{g8I&bk&#y_bq@}<1wwNu4tD^;P7StEdqg!b zwsXM9*M7YkH?>u}+g?y1%K^xey|}#+74XB{^K>sxGFuD84SovW@gy@tV5d@{1LU{& zLD0+brXoAA$%f^R2;(UrRkOoT%@o~Bz!>u(1h0m}L|*YB_OgY=ta01{7M%*& zVq^h(*#YD2k8o#Q2>B$VktcCiy^INV{jN{GLjJ8sn*#Qg-Eo&Iof#C%EjBHqBgO541lYKtIT8|Az zwsUoWeMS~=P6;fOLH!43vDrh1;l@fVMV8tJA}h|}7+WT%Um3u&9WYW`*{+jD5NX{& zaF_ZN2dEva(O>t@L%Pug?wudm8-}m&#VFdTzUrV>YB1Zi(vdmdZ=Y2)8J(KYbKx+U zkN*AkMeJ@%l=ByH7)drc(xUCB@EVuq1sUyATnOKdd1w+6+&GmX!dW&i{0w&xAaS#7 z{`qH+`Tt0e-vZkaAh;wxQ9>VV@een2)l9#H*E`{ci4IR(S=ls*J``|Gbo5Yw5y=k) z>MMkH9}0Xy_I)MTeJa2qJxP#L;LxwZv8MRM+D)5{ZrhNfUP82X;Nj*@AgwG&d?HD_ zfirM?A`5l6+;9WlN!Bi@&EflvPNdCEz}JfDM;vQWkyf{Gi(qR3_!Kd~>6vGw_#qzf zO1%StZrs8#dR5Hp>EFOR%X;m*;nIe#@@wde9bxah4HF~{h+zxg!t1c{B*z~jin9zD zmPOR+Z*dwql`6y%@jH0ucRxi}PF&bisM)bPQJz_C`VK#9I&S3!68A40f<7@ywpV9A zo+vd4_O-?MK4rY-fRQ}h2j^bdsIuQ3;C$eS^S~Ww(Z>_z2O3V_!&|DD9BnSh4>a5i z;*q@X;ot%glkRf{maX5z!0)(hG>#C=z&hj>Kfo)#Nx>~@Mj1A$zeZkMiNP7 z&{nK#bRz|~SaS~=bZ`qML~si-mw6xSv#2IQG;_!e$;|vHFn&R9DzxNEk@Rr zHFdz~jD_a57~lI7AHuN?7}EqHz!oEXsWAw69o;xvs0}o*j_{}HqO~R2 zcI}@trm_P?b}1o zuts^W*AvEKkei*oYZUZ6C79({!9Inb6(ZI-ZCs#6Bshj3ba?eBXBOT`A zx}>AG(1^_Ngn!;@DEN?)hQesP(O}i|(^kqPxC^h4*Lw+UdGs51fv&Rif@D9`Sa!Ce z()l25ea0%FmN6FAYHo*=pPIiV5|3Vqt<+&%^AyMu+?lAFJT{9-5*9}=S9(pq-55H zsqZaZ#&HIi&3mn2(tU(3rVg#G;&rVL*sU9tQyY}*`BLNV(3-gW3ZWQ&*?Cn*`qr*} zy07p%9%ZQg7!KtO!IZe#L^y3KwXS`?AJjg{Q2WIQjk~lpiD@eI#HFp-mQkH3&iHs-<5)-OO90WZ=rUt~>3POe#66|c9JThZwN z;gIQowSe^45_H5M;BB4zKx@**EOf^3OF!%BrQdm7fno!S6J{YA;V%qEdNhfbAuU)~ zZMtJ!Jza)@Q7-QtyisGk)0(i)klkV0dvkaA>dCnf;Q_9f#K)QGLk&c=KyWkRdZ15I z?EvA%v`e~@oBhRb(!IGrKc10R#7(+_f7Aedk_ZkHCYh|U2<@g245ICFvVO`ZNluPB zq$*=kxNy{zVI`wHGs-sNhU)AYN#t6DkbyH8FexpD*34U|Wk{{NLQ6D=>7uTY8NueY6XI*xk8q=YFpZ`Jn@-^d7l6MS#O2^*e7UP%LlX2ow(@^cC4OL(q;A+uW?V znifUv5f}`Ul_6Y26Af0lbySmHwZhzS5L->jxt=t&t-x;NL8cHyZfwC~&8|#gID)NMyoa!R zn2n&@ZrX1x8+&kb*dUuJN#ow%Rt`0LjOQIN(on&$Yn&u3S%-)s7CeZ2)uYYVdtT&cX(GGG(~X#}@b1;X)i*;efH{2qD211B*;>=>Bz0uWT;Hc)pTRK$8apFCl{u=4L_thYOpC-i_vKNk*z zzv-9S-Ex1|{0$QsHjT=+2!9|CN|WH0XMKC$SmCZIIJtIA$em#%Wz(OMjMdqLlS$uv zAp{2}%S+G6WKF&>O21^PL>9oZ1yInO#_qQm9ZipvYiZjV%M3u0WUQ+zM26E=)0TYFz_!~xT`o`Ls-6ugyx^Hx5jgA70 z^h-@P82?2X5m|9sPpbw#`8(w{7}*!&8#`e9ak7wVqejR@z9jIH5DKZ)ey#6LfxrT( zX<%{kyvGMt#8mJIPD$Yd>)37H(9Cj8rwd8&-7=_4-Y|X9B|?oo1Kj%Qf{5@?1Kh1d zj+VfD9QLEl6kdd9c`6*aaRtU#JP`SIrf>{THsIjdJPs4fdA2YHw)s&Ua^6F5^4i9c z=rLWhe>P08@EQYL@HviF4yWShVcCf8)xH8tRGsI;`0p(lo4%;ymiUXtur-Bjo-e>d zk`$|qK#wMau;smCWK3Tmw6SKKOP0Xg30;sGl~W>(jDIYEu5N1z%OoJcf`zaYu3fSg z7qCcZVhtTha!x~h{_H@ecNYoa){K|QzZVHTaGg~3yD5c>h4$+HYNKrLEf)NBVdSjc za0z5>O+Z?fr(ZI8o4f!^P&&o%&hUjk(CqaWgzh*gl^!US;A${B-5R6%CvT|`hI`ne zI=xg#*I`*rY0F?1V8dI~GI&fXKMOD@#wJxDAx$XU4ah(8q-r^IGmw{9H{-oR7=-7g z((|K}@pGwiNF7-rOhR~7D)ZW^R5C<@{I6Xy*?yEjU1L$96fOp9n^?(83_ zw!Y_5xiBBRTF6KESSsI`vpaLNvP>aF=!;Aj6;5uG@w^v>W%#PW8oA>&_KT@x&{~k| zqO1&Do?0smwt+*l)0)p=ZS%1MD7u%DLeZTb*VhT|ws2sL0z0?oonsV-E+tb8Y7_2` z!6x6lBz%W(aC_b+7myV#IeFgdp6my?mnCoOd+h#P%#6;D^Qo zcLPvZJnY&iV{#C7Ej4Pq#{DT*EhZxVjP_DR>uqqFw~&jouLX6 z8;Za)++~Qoa!hcpS@;Wn4v~f)qAT9d46GhaySZ+Z8lbYmHEp*Eb&%;pE0O5CXI1#D zL8d*i8uy3oN!=YnYqK>Yw{)Xzj=kXkX5S8>1O8C95TJU!0qNTsm9X7azcC#39TtN) zYHNd?=|1G>Ho=EKZ%u<|QSJSvFc4~F%llrsN(N)@e^Y?(llgR@(Wx5m*Z@s^C0hEv z4H+f813OUm;btqriZw&;vB2piXg?GhqT{nbv& zE^U&vTWD|U+M#yTkXLpKEpYD+d^>z}S_A9aL4Laie&hDQAh2r(7KQLboAyV0gvPco zncIsTU~Z{~HBcMEuqVkg95AxPqSQWXuMlesgiVI8>=nZCDhEhxpSTSX%J)GiZMDxx zAL9jc7UDh5otIh@e^9t9kd9l0x-|(0g`Wj{(BLrqNVBmL2T}D&!cqb966?8AKZ0)l=MM6Ix0FE91wZH@ zcNJfJB$OizPhHs}t>u&u1QJZ}7nMcYmg6)`pgD00BtspU`{6R|Gz=V$Hswh|&cXuw3EP8*cU-lwr2>qy@COo+h8SeV%0*QYi&9AM171miZqVqtmilF*F4b*qJ? zvA!0UA$!X8Q5JZ~(38A&8MZlC3$Kk_T+{e$XT%%(xe%q}F)x4kx!|dZ^ULR;U-LA1 z^k8Hi`GwG_7VG6Ngh-9m?+Rqvjy73exB_kO7(q?Ngj5ccbKO7PnCgF3fK#+bra9c0 zn)?;F%&Tx%nxl=Wj7Z)OI#?mJw=p%d85#A15E>fm{me593*+G51y%I@*T213a^jF6 zz}Q%-;nzYxG9wWd%2#|Xw84wgl$R@n7&Yb1Pm`0B-!&Lh;}_Fd;(0NRtiJ)>kbVi@ zQq9vZnZFL+6tZg-rQH(tnO;n@%2urNJ7NfjN?in~ZTdGtFNy_Yrd00Ed<6>H)IW7h zo7+OiI{2Dn$IOWY{k;mmbm$mHBs=Dd|H(07zA;@*MpFIFtdX)I!<2Mm-iU#Oq9sb)}r$H;&VU3EV~(*&_V4|*67s!=_KYS zSlZI5>A)Ob(X~TO2j`5Ky~)P`f#fWtGVcqWgmNKI^`$;-SzIXjvR^cLt@)A@v``LCZ(NEsU3@J8YnPHnKQa zobITBG9oz+7AS;v8)zh1jzqf+ltTjJFl#3LDhw0AXv~`P>WNd(LC%`TKGdvPqgb`(aM>|#YcoMZar^pnm}Li1YW{43{W&of2y&o4AA5vNqiId7{@wB z+SC!LR#ol%;ECj$1}_+@B3Jz2TLjNj(WY9VrvZWLvts017KORYZ@RvCqV0B&F)M9#OM7^r0;h)!KHbe8kqOUg(0xYFgy?r z8UIUmMSt$DA`=_#u_y7^r87SR&CRTh^jDC0$mG}ADn{vS428fK!SuFUUa2wrbtY*c zVu0D25$g?BoxQ`?D(u629wH9Iew|r-auwX;HJKJFzKw7%gC=uc(u7fLvZT3q*fiW) zli^|FAe3hiVdXK6d3a~??=X1kZi5j_(9!|hTj64gEjG5Hc7X%Nk_gd^hs!E+k{DF} zJVIQG@hb+EH-E25t*Cr5TBM&31?$2pIX#sIUhY(0`D|zT>9{>cc&^Qbkdcu+9Y4zq5zO7;Md$PX=mc>*8JZwY zg$Lb1rDC$Wta59vCe1yvsQ{WsQli*R$Zj4-W+#ey*sqJcSoi2njn0Q8JO^~@mW^ws zh%QzwfbG^u=)(69k9CBIF7j>-8;oqXMkfc1Y#IpwZ?u9h@7Mx>T^po+fj$8=*1Nv7 z7>*)H+GH^hesU&*BEPj3+v12WtQB#@K8jW}AJB*F7h!WVZ#n!KXOV{$qDRfHHexGm zT4-er+LhxK2AxcQXv_<{kcr7+GqjKih2ip0GJN&G8iqYq#cw#kc_T%PR&d~lDd(Ch z`q{*yX9}EffOQ~Mj8m{YbXmB)W$XicUphcI*k0`72!Wa9Ap=ks6%A%ldq(Jx+mHus z#Wt>7uPj^3wqz{$>UQOW&j3sgbQJs4vQtW$7-qvxOvWJlP^?#4$YRHpt3z%gH6znS zGs11Vk_#CSZe7VMoy4thNh8g*i5>w;gL<*E_zv#VmCe`sbR`vCpqqkAAolj>i21Pm zwTrkPTe|YSRB&*Dn#Ozx3rqKI;!|X*>RP*zGTo*dbW7}^F;{`OJ;V?jeDZ4WG6#&w z8KN()k~IOGi5c+0B>nPXhSWPt#_}IrHqRAv^h?&<-^+!s^x1%B z`h_`w&Km=b%LXHpsV!qnP^Li!nYb=Uq~B@a!c-SD=N9rHYx2adHfmr=t)G3#Sn~xa zh$AL9~+Ex+o{xB6SZ z)P`r@wZ!+jOYo&EvSYRA;<`^UDVzJr-?RMP0D`-;P#dNqBm15uawd)T(D?)GvLQ#- zi_Qil`E!;&J$Pw?B_4>i82NLS;XP>Jo5FioITt=>iCWTnw1uF=gBnP95BYPJGCPJJ z`h10b&e9H4ZwFB9bC%Y8>}z{yG-FghXK9c@bps65YJJYqC>Q;lC6$q;O%A|de))5j zvkX>Czo{v#hKT$*%jL{!czF-`bC%1ktj9iQiC!`2oL#S_>P+B=&smzS`S_{0@;yAX z$3BYuaZ61a<^=k2OOT8y%T;MmmVMk3Uoa?}8>fk<#v1*&CAvwgS6=Q5+*>QwZ0{GF0o$*AN89* z=4+u*kG>}tYQ)A~aDG~a{tYv8PHyQ$OSG9x*dm72?A#>2hQc!C?``Ox$+FLqh@&!C z_YsvLe>yHIqjqbO_ogOO?PAnf$DT);bSeWIo>+f6PJ?F-W}l8T7O_vqsUZ4duH1k| zuv*o(XAHHK{GbrMj z>wDq=bkV+$EDQjOju^SR!F$YtuOEU%7t;?`79?zC^^pNLTsPqlJ!>YsFS;T8gN)4G z0%SI_JAXtRCP2KfFdA|cR@gmz$pg5=D9sh7UgTjj7!H4bR6I^LeI+J(x8!fC_JI|( zUyg}qiSsovMcs*$p3S4M%^6|GMGF~l143(AMX-1F6plbgs=0Jrj77LlFFv+%U*Txw zxE}i<%-u)zqJuu~@{hRN9MtI##TVhr;s&TWKXJ73@btt<(F>WT_mVmFFeN{p5?kQ8 zz33b#QJGJy^h{FYRApeDOC9LO3=@j!y;$_ZPr_<1vi~%EzgRD#gSw=TMNe8pJd@O#C-(vc?t1SpU8ZRK$NurlneN^@v*(;|c4y|^d;IKU<^gucPZZb> z4p}p9$1&WAMsjJ!p%V6BKj%0KSbGsalwzi)6($_pcf$ooj^hDXdc-}L-gCladBY=a zz{&2LZ^rAo`cisCzl;K}^xbfdUImnYV@`&=DUo1SwUh>T+3>CD%_xf8ngm+jbzWq^ zb?)XwyP}9c{*>9|73tpn*`4P0VZr?d@7XuBnR@^AHdOUpGT<;y>`!#p|Hyy4Hg~YA+?1w< z;Ys@EUmg_7?3Z&!G|<%bTz9 zN5hdFIH5PYKnJ=U@L6_tnADJWW`J9!?MUn%9_ESFl!ua%{G#)w%fEgOqi^Q!4Gnri z3%@>HykNmI!78pR%j&98T%~LUO4J4`^`f%Ms*0sGrKm`{Y}E@(m$L+Hd0BO=Vrf-j)0(SrpZjbl60s4Z z!|VoPVF@swI2b4Q6Bi!=E)YW>12TxoxY1}Q5iPifWA_n-`1NCF5rc6?w`s(6qL&!L zJh%&}sZUvnQoq%!E6b`H*EUYDOcT)*5ls)#)JPZ23(=eq+Y6|+Nx{YDsM=y%oz&)q;Cq!?x(7?8b~p^HoO)|a1}5DGRB^K@1(?ds zA5(!hxj8Nk_!*o2msia6xT>-z6aU*b6zHW$Ri8V#srr16a#elCDOc6!*OaU3^A(Cz z_4yes)#u|Bsp>PCB2|61(^7qYM7gRy*JZ&85GXL2P<^)0QhhET4NLX8f^t=T9vcHo z^*L)SEY;^3TB^^`y|7fDO_ZzZ^LNTs^~vhlPM<8sqWX+mR%USe?4w>)pInvI>GJ^P zs`})DV@{u35z6V4YqB_faz44!XP6>YeRAHR(Bcz-@IUd33lh4zgKKWR|>64fLPM^G3bo$JoNU2YYFO&;c%~kK&sDHZ0)cy@6 C1)3ND delta 54346 zcmeEvcYGB^`>!**0n&t=gjAB86w*jfAOr|0lt4o02uM8%CnO<_Bs4)O7P>$d8Jbd5 zKt(ALV@D7Rpa_T}C|!yo-G(UKXJ*gt&gK-~_ul)t_n)7i_vK_~=K0Rk=9#*)FTLe; z^cPQYmeX?&?GWE_MjfROojW=&FKIq%d5Om&_T;ww>E*MRKfXMw#Jl9EG&?qar>ieQ zKheKjI!!8xDv5@SOVR+hL2ZW=nCi?mrKWOYaaE16uEcCCHy71Ssxcd@DvbqImE%m) zIvJ%SZboUntKWZ7`ft|&T=tM;@=S8+qSW{g)EFfdySHo6&4Mg#a1U@YnkvT`$@(kU zr*&yvX;q~nik(8zszt4k4@H(@A9hAw(pSFiB-6uRp>!#!Whc}@dcxo*J>BvVRKUu( zw{k{#w2VvlwX)=8x$qb!Rkd>SPDT$sw5VBijk&mV>hh9iWvTF7)?QlH$_sEdwOWr$ z5eXTCJ4&-WnnB83kGJqtIi-0)&$T7qWwf^TXujm^*{3IyqZYIWGSnJV;*;Xw|A|zO z=5lj|xw5X-SXFGStuxh?*4C94)f%G;O+^#N*Hlfa92cz!&7gY!pFyd{FIx;M=!6z9 z@p~!a&y|jNUPUi4?OgYAMjKdm9-kf7(7!dp=sjtmPc8Zc^cpSQ@M-7qF{q~`rl5~f z^gXGaZwY=6Nv6<57d7>5z)Qt*k(7Nc)RC5Z`vu_+NV=8i-7wS78>+)3sR$*w6%72? zVyOFwWV<2HhKv525NatcYBL)9VQE=jtcMR~oD5bTz!!Wru6(zV93{yz@ONWi^S+2ii z3`voe8#}|(k`RAO+P;u((x8yeLH#LP?Mj46h0qj5qq(j~Yl&t3fxAw)K(1YdrnT=d zJZ~W^iRxPt(|1x~nJ+o_2+GATGjR35 z@D}%KOuQu~Aw!WM+fFsXK50RGKQuuKNsK{Xlk2y|7^MM}C^Z#HSqW*lNRZH&AXFr+ zOo+AtLq5DC7NBP|oHkOccxP|MlBGp)fO8Y$C?M5=nhK75EAbY3l}X(y*&S~d$c7!z zFp`ruA^f&n{oB&rlqAF3O{MUaX0`W|)+QUJNvT=rbAX67lv(^~+|QPV_O|qEs?h>O zez+rnv)7KZ+|i!<>sW;m3Qm1&+75;P*6 zN^?e=rORpKvA-xC`K*(duSnxSUX2HO$RY};hQDOG%y$`#Np0`d zMQQdBe|%VGtalGry+4fYu@XHjnfi5BTxsX)s!5erHlpVUWC(I<>JQ{dKjcNmnlTMv z%lgN8%B?DI?=cl2Nb1xp6$d#Hv>ydaOM3OQ0V53<9f`8+0u1XNtNEKwGtfIcpSOFf zlslv&{d?gZj87?lfUBFXR*%+b7D7k(TJc1cme%?`I;Ze1bsg+#k!kecvG|4)*^z*Q zk&|?GaE#%G6<<=DAz|nNWy{NAS+A8U^bIHGR)Ar8A0fN*O7Y4vVruCA~N-Oo|#7ulS&jr1|i+ zIIvWuZT8OJQO2e8vFK+M=E0HDh7;ebt?NDXVox*&W&$9>Hll?SREPdqGH zaxPrpQe&>IhS9#+7$vPM4sQ6Ouo`)&X_HMbu_`P#lXDfgjp3xVsME&fLSM>Rdd@3q zH76xx&Os;Z#{{F3(l~RPEqWR`{_!8uSN4Npq`~sKxhcTDj7;7HXs=Qg(TU z4N_@oc|m=-+8T8fK^5I_Npt3aTsrT7gIe< zsXwebjNE8jR!@L&A}Wt1Ssx3{q}Qum{CH^Sd!yx*0jEV*CDW8-XYOq5C=bA<>pd$y-E@3AtR*g_gmkmw_vS80M<$nHR6TtnySBN6IK(Uo9saCTDE zBct4%;g9>uD4~g39)}{0eEa|!$%7;h)e6e2Jn@8~+(M#_JBi0;A%HCXT+G3BE$Hl5 zam1V_RQhu1t(l#1c?+^^gqRerrSoF&wl8*(W_S0LOtT)v>Yszi11BeEw1{VM;LBFM>m7qjCe_r!$`Pz5LDm$r zd4ja_slJA_)}kFbf`g>Qr-$N~bdnW*p;o(A`uyn#8>IEm5NYnyY7^5@eD_QT8x)dp zZh}Ofu}!jcZWmi5r{_lEllEEbIWGsTl|mK-+oFAUUJAZxpEiDewM+~DW-suPzMfye zJ&)!7_L9C`FhKV_a$#@0wuL-_{%D~e3Y9!o#*(EHjAdLGxk6E&MKNlDk&6nX6ALpS z_41>#v)TCEN_~|8SAQy)iqVnHCKT1 zR>J=#o26%-O%lU1jYFk(pH0Js&a!gdmL#E4O}SM|TxS+KE0Fq2UVdLsp1PP|!{G_keg-dR0Wso3g_}VB)pR?8iL|T-Jt)&((w31qmX(c_o#(-fq z+WGknG7NF^65Jv84v1eL?zceRq~S z2v(!UZrF%?8-$IIVGG=QFHyL~F9%62UZ&!ty*z|HKZ5p2KfYXszLxrJ%Ecce>Dib4 zrQWdMd2Ew6@@u%dsY7#T%QR?XiFCaoup#%2AS!Ow;S_1rHVXQ}jv(pMwrEIGR8X*8 zNngEv11sBF8nB}eE4vDwseF@mDrNTWEP}L%29u1{u&Lo!TEUuKl*3!QzF~hyO4r|_ z1h;ktNu3)HTR?ihZAoib@%AYyYTbJ;L!s;sgQSk{w}ofZZyC~>_h})TcIEwWNTZx* zeW0W*{vel?lZ|KiP);8PX_U+3mggWTe0MGAb>;5$px5+|T2c;s@0kt22ltF&02xxr zUP^p$Pmr{0ZwE;GW$y_2S+?!Ty|U%$?~5PjLjf4~O5Xbvq9XVU{O(4i^o}JhfLWsL zCtwMw%O^iWZse!8;BUXrZUYx?pQj1%Q1DZj^xl^S>4o7Lp z@kM*Z4W2p#wbdUElIo7cNSTMI<5#T;mmyTG@R)TN%tHx3 zI?P)-@`#F;f_zDQ9txMfu%tcL4!KJaM?m8ULJ z6H_i(br)JI8ozy!Qiois zhtf*>xOGWR3zE8Cj$`Q=(kqvhwBIk!0v(wMhS&v5PU*TW(X*QS+TFwW#5$T=a`ymjkHjw+T?{jo*W$tG}hfv+?)E?0Gs+#r)BX zYBTB2F*4`0IbFLu3uLeSYdd5F--F#2+E6LPnfpP~YyX5YNVze)-Z%O_H{c=I?1BGa zS1}m=A2N!qSIfnR-B+D<`+dsnnfrTX)q6>AJt&Yt!HuC5jRw!HMRFlpQ2;3yQ41Qv zEmkG@NXaHE5gKS=1RR<~VKrwGMgt%R{EGalrOr5v-AP{o4TjoZ6cAN2h(mltw9-=e zl&C;Ri-xq6-v5Vr0-4~1z!k_GC$v#<{F-khkGnx6zZqJF{2EU+Lr;M6$jBCGC;Wh6 zoRwZrfX3u7W^n`C*S z1Mou^yGSoD$`zU7g;qfcSVDQ9QXKL|#T1qmsqtk+Hu$0$kVH3wh}lm`3SxducKe~% zmCQH&tutL3eFK0s5OiZ3bXIY>FF-lFh3McgW82K^cTo?KMXuVj}GgF zQ@WG52sE0E4MWYzmA2>#uyiB>f%i9#i9{QzLU1(tq=R=q7~ItuC)JvZCzTs(%;m6; z3QI4_#;Kib@*oR>aq#qV@)3=eV+ul7Bp~l#oq5N&j4ZA(6-^&mZG!P790Q1wmYr=w)`uW}X~~J^ z?Wkn6(<IEtAURffn~l3JU0=x5&7ro-(q$YW(z(lg%}CJfGQ*vm|NH3Ev@d zg*s`oA0ItEF6r^;(SzevSNiXf+x;7|Dp7Zxg+hO279#IdW49z00OI6$5~sxzQ%L~y zcY&jpFY;~@BFN!v!H@iuf#S($(Wo09<3;w25|Z2kS}JQnBfZ$ZY~#on)QE6}7uy%E zAgQsa$nr#gU{jNP5sT6d6<$_rj%?TeX)k`6Cso6*AnW2#v09BAkTyl`B8xKNud??Jq z2Xz>QOL+{sPmP`@hf~ll7YNx9Q0bx&nV*X82`-ae$hV!)RdTL98i2nCb^4)TGN=O@ ztUP$C8sgR|zVCnr;Tudx%0hKbqgvQsZARnm<;`Zw4Kz5{}4({B%+#-#oie~9x4}H?ybTnF@GXljRibYRUkSm>050pdMmY0Owc&5pOF6aytWwU%PW26_%^1aCO8K@7*$VIKB zH{J=7KA%>F%rX$&B66Q|TGeu4Yqk+hzUzuc8){oO^<%bIKAovfS*?6ZBd#UKGEp3^ zh1o6mCznxZ0b>4{mxW*$E0F=&NL|&fcWR}yHHCwQY}6LLD04~IyuD+&BFa(?Y*dBh zzy{?G9oo)8Tv1t`KHX3l+QZQ3LN`et3?8pz>+NpH3!$%cKu0%N0VVW6%M53%6eI6e zU)E{n{O2n58S-dPl!MPO&A4+Jg{|o-47uJD^)qN@D9NBl6w26&%yjJH&DOTvZ{JYy zntRIusZB3578!iKo7#yy+Z%;pqc@E=$}Z=x8o94Gxzig(+ajb>R-DWTl}4x|R5TBb zFoI0%g(CPxD{DsRJOOuez{i={aI(Y~cBjS!!4FxRkFvE#){G5(&=6akPU{0AnjJu& z?gJxrfTqsK$H zptJ`=SCxD*>DC{`+o6+>7~su10eE6>5-$pYwy4_eg`?anbV7pdk+k^J$x(-B- z;>FAswyK3Gcx!TIAPT}~9bgPXMq9*m@`G7)3>G6odo6)Kl_lPi zW^~44E2*0F8jH5#-AvAy;8~R3TF8MXnUL>OF?`We1eK$pfca3CD#Ic!Dhky|$L}mf}mO1q2 zg=jnsmo$eiLChHPMva&L=tc(6d)~V#Fa{ID! zKo(g~m%}n&14*ht?cwkcbFz)kt0Fg$mnu*QZU9|?;kk_e-te1KD-l8`9O`1{KBkLr z9qEFVk#+GkuM1G~+4;ywqN`DBd#2`SnfMd%U$T|=UR6c@?G|>e8-$BntwxC3dxG61!L-S+*9If_9yH2Jx=7AG|oED{}Cvu0<0Ch*2zfwX8!G zxD#*;=P%sPrvqzpt!aMp81+yc5(V7LkH!}GA-_&RW3@*%#G5}Ax`bZDC?S7BLlhqC zM~~_!y8o!b0cHF&)LElshbGCoX)wi9!5LcH$55{(NgE$S{)S$DR>AYoW6-@W@T0*q z+p?Xit5PPg9Q%jeWp0*c18v-NKwHbultH%+jeI#BWi;U|bA}cFpUgn9iZqYLbAg?__t;Qx;x^dS+6eZ#wjwKKMmPwxDP;ypCmORr^&~AX3 zP4+FdFFA)4?=R1{gO;HmaIwEU>S$dLO}f%wZqmvBsNzHZ?0VD#%CXo0k5}qZnuRO} z&yy+qbQFrhVE*Y#mM%lS{NjuX#Nt4+<)|ygrmoLn#QQVXU$GpGCoUg=aoRnH7UBkG zoJ9AWa4#i*ZTFW0Sc?_VZQ(tPNmxV3TM4Uv@)P}`D^2HDLL*a?h(}k!ixjRx1sc)e zRcM@6f)SOjhQ_KAv7{TTQAe6YIqt4wgiS-ItwC^t^?}UD1M<>ZWU@TbAIeCl=V8hV ze^Y=yIsm#}*Vy$jkh810g)+(lZxhWctxtjhY= z(C{Wn#;qt2l4idHz1^^_ux$BF0Bagf|2jd}|vEV>$($mQDVYScngNi(^R zf){nmpi0^@KME7vDF%xTk2FGT(DU+LBdjE=N%983qi=&2)HDWj`fa`#OB(YIgeM($ zpWBq8`Mc2Qm2%ndLP$Pov)t?NqQEA#K6n>(G7N6hBu@B}{P$o9?pK|&=rkgo{h^I5 zy3>@dSs-d=+uu#6^eKef7T!2vC@dK2O zqXT8Xsad7^O+=s^%0K=9eTQ&bAax#p(zqK%;mknUdFr&%kRS9BWTgc%=YhwPkI+Qz zk*WRRJ3uNZCen$#b7V2O&SvAL@qfK7&q2EnwUa%V;}4 zLp__6iu+tGr4TlJ4vxnYj%Y%&eLwP{NmK@or-(piBEJKun*au3t$V=%G!*}AvC^E6 zR4ZM!SSjTo9L~UhTI6MG77r2Aw;*`e1l8^ScsX;{aZ8$t)kdT?mOG7Vyp;~+t$u+(MW_hGi0=xBzy;9WOI-f zrqNrJ)MCRoHTuou=yx!Dk_qV#_0s_-zysAJ60#qL65pKwM};)7SO!PG!lL;9e0tKj z_#}GD3I7=6aQd)!B#7u;b}q=#^nn$U2hpc31?>)^7m`2E+8;!7NT5+R-WzxhIV0pm z2KlyAxpX9X8+STjl(I^3o!3@;#0;~}RvEt-#mC(%XRke6PunhUZ1y~oyvBAvz zvBAXWA}Z7#nd;xT2-~Ky!B*bPT&p`pS6`g_QL6y3axcO1w++JCmtfKH5eI}fFTt2Y z<%u!T_cHRQNnq<0f5KK>fQ270vLD%T8GVi?2lLJH_K~Kdk{_<1MTTdr<#_BT(9*Ly zcJN)5`k7$XC~eRM{tR^M8J)7%!_$#vKcj40^)bEev`^RPD)d=UEe)I;U_dZ4@bRlC zLpcAcY~Y+<&|LHlg_2+Hb2k{hm1uHQwlE2}hPLAi%)saZ`Q;kwX}Hi-3HwO*>nI)F zp-5~<|H^eVN*U67X_yy+KNXK98w|@>vF=hR)Xq3Thtzz^W44*1=no6!z9l>lQ#lOM8 z(abdE&gCe$&3{KH4NqI+-uWGH7c*SN!&5cfr^(Pi&_s{tf-NUV**!F6Ps{e;%KdNH zBjZmL>C*i1<)H4@92+I)IcAhC4voSJ$wqnk-;5%W;R3e1Z^I!SmD^*3fR_o->{!69 z5%ElX95|*+p99}lO>oR8N84^r*aPE>prV3A_rX^AWK(n8V7Oy#2>PxtzNa(9BO3J` za-s!pi|;Va`*0b@EJ!R!<~rlQuxAMWR$<`UNWy@SMo%prFXYp(u(#DxwQF-lZqz6ty z%VeaTo49Na>01v#TBk$W`z2RY9uoO^;^x?}%}R0fZ35KD^rqf9ar9S}d>h%`8mHT$ zGyWbxx_V=SwEe5*B*7cEL;D?|tntRZ@HUw0z-(X}5q)q^8<=FI4=#Y7f*R2_Ge7&_ zel`f%w)Srh==S^KTpM)qIp{PaR^QmuiN4+sr`e#QDkHHWc8JFNNu)d!C*$)0=cC~h!~82G?aukZ4T_LOLZvU^TBy|r5Tlkl4(Xf&IX;VkU1L5no`E`*0aN1DpSefE%WSFn%zY)67rNx}pNO9iRYvhOyUy z2gY&PT$}FqEe;^@fjEI=MdL?shz_%GFh?Y-^=>q-#YtggVmHwGbS|4Y-G?!_yeiRhgDmIhLH`)I2aELBR|FA1jDefrXI$jC!FNY)c{P>=?S_<#w6ii zh$$Io!nl~S4nKR6afIawuMd-#y8=@n!pJ2cjn;-Hll3Whk~@DQ5@=~f3pG(J6*bOJ z#c*QqQ@M&y$@cbmjP{h+0S|{KM!Qr;%a**q?|^$E{9_pF=OlP%7XK1P7krXjwX#2w z&pN`p2tUdd{75{~agpT-D4BL%Ne2<+0!mW5PPkW-q%EDWFC;NumTUNml?|P75XRx* zZ1NQzP6lS+5!@raS%F;0!2O!0m2}moCE(WNdRN?^FPDJ5NL?l#prtXs|H{OjX%Y>L zt9g!?kIv1)WAL1f#A71#`>nH~x2Omw9tYtXh5f3FmXQs4*sF0{HXe-ebjCZHPQtt6 zc*FE?D+guAycUL2-&8`%at&)bIo}=UTTti^o$43%z{Bi9z09DLxX43=6Ly)@^l*6} zcv??z@ag1WPdH}w9&qU%MKZ_W)`EB0bdr~g)9?W`(V4uNi@lWUPvzp#O*pFVg@@B5 zP?`@{Wtl}>8uuCvy>S-8zl8JStBp4`tu=zy_9MgJc2p#`FsR@j#qxi;%Hc;()N%AP6jd2`rih zW5Ag|2zRlCqrT;i=QseHJQ)81D6B7ibpm#E0}c4t|wM?#{7Q7S198EL;yFEhpl5GG`>t#wdbyRc_}Osr=t3u!X(E&@#fx^sHtd zxGPILYy6`|*D`{HjE1UPMo^`@aXI(NhSB&5oB##J;Pe)^8=x-lGM4H^CXd10aaTFZ z^NuCUqI1>&Ta3lS5H5(|8{luWoILSk?0PotHQ}xpkBT76+T-AqCcMKQvH(|ia+?dI znv{V18a2PF(20n@SL*`!JL6!iYlB+7YX;ABK+Q&Y7AE1+G1|!8igBtfz=6fM3a?^V zsg|u19VfDN*ol?qM)TCtx>~-E)z|UJc-+ZW9n7E&4)_fzfmxj%>%9^jVGB!Bz?TjH zi%W4V0J3IzdNOuJ%zR$t$5OluT_lsspliCp=smAd6UF4U(A0mCJSfA{@I^-M!)4s( z{Dth9fN$gGk-WPYaDdKTyoh%NzHIP`Z0h3lomMjAd+Z{W9fDUNS7CP?9Lf88Xtavt z6G@(}geC;PaNM;c**X~-72UUR*B9zlg-6+_iu6AqCg5%k(4(qx9vsWvFcao6C#Er7 zZLfyO%fQ7@(cuF)6BkFyFEI62nJS88<4o}ey!=Z{U*b+=-9en(cXh6gF`SfSrJJ*zJj6G z*QE1TxV^FhsjG@S^%V}c27s4IkKiaWvkp6v{$F4h(&8u%=N>yB#jf}jof`G9Z1ezZ z(#H0}VeDnAiUa5H8ob*9$OgFQ#ug#)HVGebKsfIxHrgV5_b7f3UvNOkLWQQ?TVKQQ zwvIW24<}!d#iVSCHtQJPgoP;G3~37Q8gw*I=6{19ho6Z{a4c!{li1Ar1+It^RbcTQ zW=ZRDhz_mr6RxN{HyU^xFE%7aHEl@Du~0g#McqWG`9-BpiXzqD!D!H$khQ>8t%Put z&xn=5B~MI34!~^w;jx0R50|V{#p)CIRXj$={2b3VN)Uin!c?*OBwlE!vSz;f_jo#* zqGP_klS*9`Mc~X+fGxuKAMipn-w`3bU$rWV#y@usQent^3OB-$d1jlXqj58DtyZ0Q z1M-A6i<;NC)V{{Xhg zvFZZMZcwy5hu*tQ<>G#nJcsV`BYq!axYC;LY_^OhGq*$BXc=wO1#xgRAHMf#Xh8VY zWtgL=I4nT)y8?q(E3AD)x`K;wx?E*?Gzs_#$IwSpXcRptHSs6hT~V993Ko2z3L1QV z{|Sd$W3zi%@Gu8#W)4EZHt6KoUFLwW-&N4LUcT3_;z(Oqu)yoT;ulzqe#!w})h{^4 z1~I)@2Cr~H$19=WnoS1OW{tv#H?D;Y{Gok1vhzBOI`oS0ya6rP23FP$ zm`Hu+0ITr^jMr=s(tAtr&khJ#K-V*|`zG9y3`qlaCy~xev5VV%OABS;+;|Ib!7ed; zI6wL=-x^p9Yj+zj!~QYs_1@hpV4}lgG&2o$?ij-?VuNnu??5-uKApVV2=ufGcZ?x}{=mtoBSBkWU5KkdVej}K z_%N=KOV`AZ{eR-VmPfA|m1*t;wx1{iF)_%;d9uqUlmv3 zSq=kms}VOVbJ2v|A494iz$VgurafEq+S`7Y?SnkmMaZ;6oPzJ#r<2!ipmG`j6Oa&O z14L{4E!)|b;xrNhY!H(jt8fYqaX>g93o*6`S=2~zK*++Dh4qPo5NZoVjhnq40I}CQ zZCGxUDCCmgzSOn@^mgOx_jD54*s4F<2}0jQ2X%ksBy_bHpZ-uYq%;%KjMnV%&L8Sc z9xf8S@j?g4=bH(UO^~}b7yNB;*{`M04!`68>v(gaTNA8)ErfnelD=*sgxi95b_V!Q z9l%d^7CJY^@I!Dgfx7q9a}%)a_-XsuNU!j7ak=W55VMajl0kuhsDaoYKOC^(>nC*A*pQF%)%yuOxiq%(@vuL% zY*lo|ZoR({1aA^DZE?IP+QI?CE65NTXValk0-1PSn`Ig`)kMaTm_Q-XXiaD%)}d?% zFwD`cVN`XZf%b{n7|EJe^&&LRJ}qHw$_6p*^Fvq{1LU6Ko_ya#7Mij#!`Tj^Jlg+A8l zZlo+lNM>wA=KD(vZ+{0N zTL3R(-q$fr$VbOj?{gp6GDw+19*dW!kl&;UGYl6kuE<@&RMZh}54^^kZLmgsF6SoJ zGhjg2ErD?XkBc*eLhcb>F*0e7ETU}g3Vo|ez>?Z!3Z68Hvd*C^c3YY$T*7nYD&{1R zm03`Q`p5thvH<{+D8MLKl)$5MhaA;y@w8 z1~I#07r_NLb_w4ufRVg)t>k8b@BzKx#xCKFzA(eLK}atiCLQ|;QOM+gI(UfC(H8YD z{e<>#;*(aZjJ0_|S9l42(2{rf3rX%=y5d$}_7{G2e$L5qn3?PuAOss0T4|I`&fu~# zUbdGsZ7#HE(*_|MioNWBkZIEfA!yT&oEju_Bl`ylCO9T)ziy^i8xWda_dB3wdbL4a zIt29kJtI`~%8ShOx=_|Dm#*kF?-Ai|!yRiys$;Z!I+iwR%Dj_E&J7jX+ag|2A|@es zvMf;ZX+7bZVM2;6LanU_JD?siTu8JdFN4K zCGMTXH|U*r)z%0~+b}{HYZ#u?v?&wMkuY#H>6qSY3|6|=;Yno1NTDqrp2XrWmjjSV zuZUjc*O5ZG0I`+@;b-AKOSmTl^k;c-`)rNX&Mm`uVq`;vVUiZf| z>iQ&-JVr>cMW|YQt9@b?W;wXomipOBcg9%Zef*J5cjqg4zFFrn$|S5K?ZSmtaB7x3 zTnN7Vovyyy@Wz7WP{`pVI_pvkH_*aA+b>LFii9p|X;qxR?NiVztQY~km)wV|Str-H zal#esmCXAD{y=3?vBx&E5RAi<*^yPlk6bp_U4VuH$RQ8Zj4UY@I+F+2gy!V2VqrK= z)}e>G8Wj~N=pDujyKs(-o}A2ObLgoh!ne4;4k>3iS5)?!ic+D6VQjKY$BppzmBRW? zNix5zHE@SYQE;*>O2KRc$E~{=&0#EEH~5K@?$}%7p`lrPeH*D+d-< z>R6~bq*51~R&H}V0 z*2sER38V1AWclROoZr-XJ|}z9;6jMss)P>=C#;!N$L#o=j?j!pF>7VUEE$<)J1QiK$};?2*FTY-2YSz&VAm zo9diXsLjYL9Ryc5o`L0}KN(dahLToe#cVQVl2D7IQ>d3K%vu`J9KOcu@<(yf>w zU8e}I<7^!xdzY!i=_w>`sxZYCA?>{IAP0nOE}zaMaN{5?{E%tF9>WA{el#Q1>ah1| zOif53&eMfFJb_{-d21QGiW=8W7g`~_P^XroZ>y47*l78yB%@QQTLF>xdO5fMI%5l6(d@zNuyF3rd zmA~GMA!3QR$iFo{h(z6oz1YFNMu*?xm3YXJEn(Q{(#lu9PUOR&tp zb*#w^kDZ;u@%-u$a1HP3_>pJk3xm)A3LsBG+u9-LZ<&xK_0m}4mh#w0qS%3e2Nwt- zHl!i<7YGy4R0o7R77B??G~>NUNVb8cMi;nR&A!o2ED~ay;B{II+jN>(>XP5&#li%% z)d3c3=b#okJ>?Ymc`^&!i=P$X_MbE%o`fzDX2P|}nX6(g(H z3+>4d5(EeL_Ey}m<mz0Pwwqm)GrtZ4V&kk zpcjQulDIkfbhsFp|dSQHMGO)1HA6^)v{ezsQ$oaVFd2SRI#GCsU|A7$eIWvAFUA< zW1>^XoMKfBMIGDM3Jr$M)|vp8Mv=cwLaP;HgdpWOef?sUesg32JA)$!~F=HzVsk3y)N{#MeXzk?78oBKsfmgX!17Le&-D#4()S57_|+Cx;6;4 zCjYGi>Ji(8Ra9u&gK&~4n(w(ofHy<0Go`=}`F$ts6swPH#4*1?C}@)8`lc|TNz$@6 zg|Re=*7Lh&uP;(rPt|RgFqimNK~MGPE}=J$Op^~l;&rNF!_(wmN_tDUhuf#Im(4}% z?6Zx+RYSKltA?T~1Lvpl1OH|kM7K0D;~gOkcT1C(oYTmjcZ4z8BNKMWyRaSsB|r=x zjyBvKOG zakhp$>oZ{wahWH`J6ZG(y5dp$IkcWu#ACh|riN;?Vn?>(p%gakhr*%u3$rWdSsfIb zZRMsqDD8Vd$WThNm^tf!5KiVV&=$;etdL6wARIFZV0k2z%R9w6iKv~nYh3?@&>7KX zjb`*uaGg;ddDkhN&+C*aDfbX;h*nUZWHuxuhL;{Y#X5!m9g3%g`JRBG}d*9q`5oO0>1fsX*|$*6 z>P)OBtT+LS?^cA@N#hBKy|-mT%PTp0!ibZ?1mKPZ3}B$)By38lkE|v2{vM8nSwZ`f zN#Dbgi&r`yi}cW!z89jb3HK4>4-g zC-pcjxKo%NDIBrg*$y1+m!dwBO5erc{=Y^qobGqE8oZq9m%uDIAlU+S8OhI^8 zIy2+0bQtu>kC(0giS{^jTQ611mg%uqJ6}Vd~KZjKY2-j zYuzuT^9}v>6)VpAUJ(W$!<}?1m)`M{ums=JsRy@;R2t)*bTa2>p+CMOiv*9q{wxf$ zfkWdN{eUgpkhkY$Ew<{R*hmhb=)n+|UxZe+aOmys|NrDSBzcDXw2CynlU#M&ZQ(9nWT`s)j*yL(b5*Bm1W<-G z38m~GcVG>ElUBB^uB_q~B_8}@$D&Tm9RcFSUxggQqE1#mzy`FtDXMxZM2E|i4lq>d zT*h<@5>q;C@*}>v(`JdHtCaV&AIU5bTM~F`jc(}J z8u+z}c)1fvz9)1rS_AUu1(Y+ms52iR^)P4e37Pn^tR+Cr{u_e7H7e`hvA+YH(|J`{Vpuo-xEYnTbPdQix)en>$oUJ+X7^#)z>*7q)aMYz3C*z+X7+hC+^L} zD7?!75<7KnLriG6gWCHeSC&Po&wa${Lh{p*EJ{uE1z4{xa!+>D zHCnN-M;F;k;M=}62#0o&qvI-nF$BkSk$dc8T4~6?>n~=}G4g5H6N1MP0b-OPw2PHr zu~AC*E_@@@<1}j{hS?&dLoWP?1HzZuh-nrE;3w%cIAW_nF;7d{VPTD}7VioKxG6Rz z2?!E<+TZ}bEuMnsJ79o0o2BN~!D4_743ZNJ&1#(k5cYbSwcgk?0U5<`yu&^*-4%A{ zv5n$bKg)?s1$vk-9(!e4jcgU|V7D|gr$CT>NB3)XgJ@#3nE zx7)*2@(W$anYLmyy1@Ly2E95BLWpGltaU$njCUg7C2zoD78D}Iw(#aHlvWmPboGso z6gyk$gCFJ&$0NmLJ9V=m3*6Oix9ZhVV!I}Ae?^I{DK{R`Vh+l+%Se5+*sTerONP{p&sT;63f zG8`RHWrgGes?#h5AG%#2^-!_Qi|oZ~%^} zg$^^ zg?`0s^tVGBu|l$q?&@r0{r#;14{{|=ggces(<3&%zr`Ve%4sZVyxv(1McCMt4JD0T zNpgl*q&>2+&VdZE4}1k^*>6G;sW>OVkwlr6lyntm;jUffnYWlJ2BDth)?Z5V?eM#5 zUnpbB6g$(htf{j>`7pa>>7X2SWfi(}C|fd^SIy26y9wYt%&Tr^fjhv41+A5O6*+hw z7@XQw4wTPji@WfAzS3n}rQHK#l-?1_e3B!k!prGQ%JNIlE~j=?dtw^~wvklbSkg^A zkMXvy4%_9x?oh3F+19R(+9fL_+x!(vL3;#?aGY3C|4Xoy4(`tQ3v9mEe3@cWv;FGRM?V>J*+P#9e%Vm z`bqPWku~*dMVJ0-Geg@_KkIW*c<7-pW4Ee`3Q$#PMRiq89a%REMbC6`fe)x#l-bjw z%c!Zb)1o`MIKyXxE0W*?*opCJ?PDsd~Kozi0XynJQU)cD#GQ?)tT*wvT>sm@b@1<+NEX|lOIYARIZJdID87VYBV+!s<9 zE$H9593Dz5>!PA6sbb*Y`9nLz82b!Qv~>IpLzm?o4~jyG1AYfHx0l#Zk9 zjx*Jns;fY#TISzHRTb5f>XfDsS*tc$7tnu(G1Z0YBPlV_#iF}f@Xjcv!^&}074efQ zOF^)TsKi)fQmnBrNNUmOzm|IpaOF6`xwa8ZK()t=8p~NVqcYkE|GBsfs~!jW)#Kv3 zgB(C#VU8LW4U&>G38J41ZP>NuI^(2jV%jNqms*NbqBtY`8((2IRYpa-6je<&)s&hl zi_BvxOm#J-R5+Rq>5yApLv3zVCR+J4=>Ig>|7ox^DEv=@g^>GywZRtsUv98;gz(%O zzdgkMW0#`;<9J{-=N&e%`peqMq#%wl*SB6K+BfBVg`7ZNc#E3KhiyYkOE+~_r{U|s!Q$H7jT6+Fx z!Ui@K)*2jT(s|10tp8Ij=#cgNTDF04hfKQCr5tFiKdTNSgDe8d2O2?{+5k5wlWi|> zg)RQ#dSY-1I0Wt2YRQ1Zf@h}1EoxzWu8d5kl!6x&nq{}4eI!kADKbrhLe(`@g;1{6 zIHkl~33?*0!w`99ywL~-EHYJs8&y=7PjeYpQ&mk}f`_F@MOBrxbv2WU>cB0Es%mP? z|a&hU6;Q<0nW!Ox(buV!m>ER^8 zmkYARW^T>fDRbNDnet@#Y%lSeVNnz1f?v~3BGSH@sJ`9vl195IQ{EzC9_+(qOjq3< zmZ|sTi3S9(+TIe?ZMmKCR`qx2BW^OhZ(YB(i_o_~bR`i5Vg}yFnwfIsZ|@Kl?EOsk zstlK+%0tA~mh8G2<|#lAO&iJ##9Ou``LFn5+pt`NZL zm@nBqTwI1nv!ZPr6_paV2imG?|D)n4T+Qee`-k#Xk&+Rx(%%c@g>8K15gasmjgvEY^lTohas#74c#M=~5$h#AmWCdzaVLfTDE-L*#G+!LWnAGOEHctCL7JSa&&c5$*?F&crI~=f*rDjm zsbYIoh}*)xg-6 z!kQVdooa(IW`X$*uyP-V&Q4~`OQYxB^3hmY=K!Yn6JigCFibTFU|T&V~c!Bd5_G#9#_d0Ko6M|b1B5*Mj%rFiAK zXT%4{klxLvSHeA010EIC(@KT~DtUT0k}w~R64)SQ2XqH9LS<&ABUD&~@fS8maRab) zSv-JtS|AR`>Las*v_OotRU5Sf+%;{V&Hf9aN6}-wwNPwp3rqD@`XxrYaxYp09$`an z#tFP?V3%;#2cmxyPNc=qf7-)(*8$d$XW>MRUaFnXI>0*S0E>AlU{O!yEcHV+>kv*a z5p#s2*W%c;rtea*zu^ySX93NPLJxFxgziwC<&SPKO^6^%chLLhwd7yn>>$15d(|>A z*WlUR$^$7V>0PJ~-QgO&XLqu!9!3M6-B~-}a(=aN$l|o}-Vq`FmWvOQ#IZ1cJ0yvh za7uT+Z6trE@~bfB^PD)<(9N1Lc(Wu0=Xd9Orkn>Vb+_*9J*aNL-3YO>8_!GQ%oSoi z#zne{2Sy|)T?JHZS|xVDwcW{VBTDiv;Bu5^1M`PVtHdV|o~=WvspHDZdi`vTIK{Bc zihXt%guV}oSLsBUvqnW&)}4H{R_ux4_nbdY^44;6(u{td^07n5NBv$E1*DNXzIEa> zWY}ZH2zw_9@7FQ0@h6phPj^zf0Zwx4>CU1aJnq^cnzTn27V{sq@I<;cphHt+@OVjl)CR-y&A2{2c%RaN%TpiF z1#P8Vj>T6Si(iIwFmS!#vYGH!WI_*eaFdvBNa$hJs93+1sjFwRrgI>CZZn+UP;r=& z+iVfzY+=#y0r_c?Hd)gnvAB}ZgK^B|0LO3_$ddv@0v1y;CgR?BMh|(MxwBQG(io=o zU~b&F_*HQaf)~Ma8#?UBF;{-sFob_JkI4YpZw&;q^WE2EI3~DKx2=j5G0e zfGS8NBdbMMw`mrIv3cOD+r$qM{!j;U{%I?agF9e~eTaeh+}_G%Yy6O#)9_mSgANHF z^wg|_jy7%O56dIb; z)ck+X#;p+=OwK(8jyb4OTO=PYX)4%&4{2Cjp%W~pT4lxJ*JQ?A_@>n+0DJ7=qCKsa?dj^J?EU9rcO<9=EibChAv;g+`W zSxcJW0RQLbaGxev6IS3*NMcidZ;ct98opftp8>-Qb!ynVMP=M#8!~$(T%o_lN^I6^ z>c1%TqK?O-U#U=Qa>>k9@VOXk6#16(T@DB%R^u$ZMm8u1$xA}Et;S{eOPwI+e^+Z) z1evi0XLLMeB?!NiHkG=G&D+x2!AC;jU$!D|S3sZRP(N#)-H)flx2 zw?u~Z*5a$vKD%!gHqsVH;-Qy;k}{ux@fI{%#&gUEWz^Dd`hFf z*qeNK65a>8D4UbZIc9-k^ZlORi%(#uJl>l1FYzp}_8R$vSb_|#^P0LP`(Acn9v{dz z9#pAY=Mm#+cvYb_IY(O%JXV|*6KugKmoDKn2Uu}u;4KOn%S&U0TR&ZUMx1~P93b>Q zD<)bH;0GOqpzy|7SRU5U)XRzfIV%pq;~iizBe%|D4VjY}jK(x}|D4zZ8D?5(jIceM zxjNu2n#yM8HGY0UbcYutWMnSuSZ_9}jJzn05Wq|5wp}dQaS;|kUd^*i2XCvY-<(H0 z!r@!QS(n6p_-(*~9^p+sTec)~fr888A^fS%1@`*2SDGOpv2~{>u83y*Ekh!HM`0Ad z*&-VoQO5ox#=|$7bs*=*TY-G=vlxJXl|g2H*Ayh=s<;wioXLV zn>r)uejQet{PTHlG@Me&4f*7q>tY*h$Y;F{{0xx1iTe$BX+le)i`dWJfOlzVHeKP< z=oLl$c0>G$%swH;OIi7%ySG(=#>87<5+ZRYMGv%3S`gol%s2@PGw?C7>$k-i1tARZ)-J%DXc&px~x*L=p2GJQVR=#)*w%VM(6oa>G2CD=2;rR-UTYMLda&saJ&i*ld(f+mN!A*td~*~ zf6DDiCXG9X0~{(ncDVqJq46yjr%?hv+o!Sh!%k;~|5envy+jp7VLV4q7rMbrV>vpb zWf!9oO3^H&#KMZ^p_d@CvM3vNpixa2T@LiNK*GwLmnbwX3W}irAV>(xFruiJ>_$Q$ zT@VB*`o3>tz5G~bb#Jv)4)1tgo$|+O|=k1ApNZHpisW2)-~^5e-TsCF$Ju zXpmfyO#knZ&PY}D)Tu20omlBlycG==^TfeuP+d@r6LY(wK|Hc1or}kUl^Y*nY1{*1 z68gId;urCq=+6gwi7)`p5&NeDdx4&&JdFH|1>3?s3-Hix;y@f|Cgx*rh?_%vAwCk- zMZj9(8!<|(N&pqaec~Q*tQg1;n@fOtqC5$d5ub?<#MKngl?uT{r9hl`K|CXd%K%Jw zo#twm09C|I;ySUV9B3epE(Hz~lgoe!qGbiJkC?=*_S^)KTn!|MLfpQ<#fV`bbVDo$ zFc9AL5f?DC*qtSwtOFhsPchuj4HAoMffVtMctiBo0XK+_4M3Kdhpnh?HZf2S^buKX z(Qzk;Ps9k(5pDvr#C=>5?ICbXuk(1r$yDJ>~#?&+B5^NY#vzQvi)FkGb(z)O9pi=!| z>J3w0n0mt052ju)^?|7eO!=F#H|1{1+?2N|YE#Unh)waDqBX^8iWE-|Nt$a35hD%| zp(#UCemEJanPM|VW{S%cl_@4uM5cI5(U@YvSxCbahbapD6anB5z~Y-$-?Z|ERd3@% zTyp$MtTa{%t87|k(-K=YQ^ZB3hnadwn7QH&tf7W7@DD2)cqbnf4RjX+V9`wAAP=w2 z0x;HmVXz~Vj^i;rARU`&XI>$2h;~ZM89bDj(S@ia<|r$PSxY-5<`^r9SxGx3=0jEz zvyGlg%tm@DF_$MplwB#{D@B!\n", "\n", "\n", - "2. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)\n", + "2. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra\n", "\n", "3. Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4.\n", "\n", @@ -69,7 +69,7 @@ }, { "cell_type": "markdown", - "id": "f5f74962", + "id": "d3703d2f", "metadata": { "editable": true }, @@ -102,7 +102,7 @@ }, { "cell_type": "markdown", - "id": "d61252cd", + "id": "f22b90af", "metadata": { "editable": true }, @@ -118,7 +118,7 @@ }, { "cell_type": "markdown", - "id": "3dd45959", + "id": "3fe305bb", "metadata": { "editable": true }, @@ -130,7 +130,7 @@ }, { "cell_type": "markdown", - "id": "0ab40f7c", + "id": "f2f38355", "metadata": { "editable": true }, @@ -140,7 +140,7 @@ }, { "cell_type": "markdown", - "id": "d219bba3", + "id": "c3d9a4b6", "metadata": { "editable": true }, @@ -152,7 +152,7 @@ }, { "cell_type": "markdown", - "id": "806e921f", + "id": "ff3ba043", "metadata": { "editable": true }, @@ -173,7 +173,7 @@ }, { "cell_type": "markdown", - "id": "5a10e185", + "id": "5de294d5", "metadata": { "editable": true }, @@ -185,7 +185,7 @@ }, { "cell_type": "markdown", - "id": "87b9af07", + "id": "5721942a", "metadata": { "editable": true }, @@ -198,7 +198,7 @@ }, { "cell_type": "markdown", - "id": "b5e5e170", + "id": "2b2aa843", "metadata": { "editable": true }, @@ -210,7 +210,7 @@ }, { "cell_type": "markdown", - "id": "bacf306d", + "id": "2b12c06f", "metadata": { "editable": true }, @@ -222,7 +222,7 @@ }, { "cell_type": "markdown", - "id": "971edd52", + "id": "276d4eba", "metadata": { "editable": true }, @@ -232,7 +232,7 @@ }, { "cell_type": "markdown", - "id": "1e764eaf", + "id": "4751c9b4", "metadata": { "editable": true }, @@ -244,7 +244,7 @@ }, { "cell_type": "markdown", - "id": "f2f1e2f4", + "id": "dd00bf4b", "metadata": { "editable": true }, @@ -257,7 +257,7 @@ }, { "cell_type": "markdown", - "id": "f00007b0", + "id": "bc9028f0", "metadata": { "editable": true }, @@ -269,7 +269,7 @@ }, { "cell_type": "markdown", - "id": "ad127894", + "id": "d6a27e54", "metadata": { "editable": true }, @@ -279,7 +279,7 @@ }, { "cell_type": "markdown", - "id": "87bea682", + "id": "82f1494a", "metadata": { "editable": true }, @@ -291,7 +291,7 @@ }, { "cell_type": "markdown", - "id": "592cb7cc", + "id": "d6427a35", "metadata": { "editable": true }, @@ -303,7 +303,7 @@ }, { "cell_type": "markdown", - "id": "8ad2c594", + "id": "38c78421", "metadata": { "editable": true }, @@ -314,7 +314,7 @@ }, { "cell_type": "markdown", - "id": "0d53b5e7", + "id": "f23ac12e", "metadata": { "editable": true }, @@ -326,39 +326,39 @@ }, { "cell_type": "markdown", - "id": "a3cbbda7", + "id": "88fd9c03", "metadata": { "editable": true }, "source": [ "where $\\langle y_i \\rangle$ is the mean value. Keep in mind also that\n", "till now we have treated $y_i$ as the exact value. Normally, the\n", - "response (dependent or outcome) variable $y_i$ is the outcome of a\n", + "output (response, target, dependent or outcome) variable $y_i$ is the outcome of a\n", "numerical experiment or another type of experiment and could thus be treated itself as an\n", "approximation to the true value. It is then always accompanied by an\n", "error estimate, often limited to a statistical error estimate given by\n", "the standard deviation discussed earlier. In the discussion here we\n", - "will treat $y_i$ as our exact value for the response variable.\n", + "will treat $y_i$ as our exact value for the output variable.\n", "\n", "In order to find the parameters $\\theta_i$ we will then minimize the spread of $C(\\boldsymbol{\\theta})$, that is we are going to solve the problem" ] }, { "cell_type": "markdown", - "id": "169c08ab", + "id": "a953b1a3", "metadata": { "editable": true }, "source": [ "$$\n", - "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "\\hat{\\boldsymbol{\\theta}}={\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "710a21eb", + "id": "19eb0838", "metadata": { "editable": true }, @@ -368,7 +368,7 @@ }, { "cell_type": "markdown", - "id": "f18b4513", + "id": "a4f910a4", "metadata": { "editable": true }, @@ -380,7 +380,7 @@ }, { "cell_type": "markdown", - "id": "20b56c08", + "id": "5060e089", "metadata": { "editable": true }, @@ -390,7 +390,7 @@ }, { "cell_type": "markdown", - "id": "7f6b7e59", + "id": "8542c5a3", "metadata": { "editable": true }, @@ -402,7 +402,7 @@ }, { "cell_type": "markdown", - "id": "36cdc402", + "id": "475e1436", "metadata": { "editable": true }, @@ -412,7 +412,7 @@ }, { "cell_type": "markdown", - "id": "cb8648c4", + "id": "88abf4ad", "metadata": { "editable": true }, @@ -424,7 +424,7 @@ }, { "cell_type": "markdown", - "id": "9d60b674", + "id": "62f558d9", "metadata": { "editable": true }, @@ -435,7 +435,7 @@ }, { "cell_type": "markdown", - "id": "475a553b", + "id": "99cc5104", "metadata": { "editable": true }, @@ -447,7 +447,7 @@ }, { "cell_type": "markdown", - "id": "69814fd7", + "id": "29043759", "metadata": { "editable": true }, @@ -457,7 +457,7 @@ }, { "cell_type": "markdown", - "id": "75c465dc", + "id": "e9d0f067", "metadata": { "editable": true }, @@ -469,7 +469,7 @@ }, { "cell_type": "markdown", - "id": "fb3f5818", + "id": "c34bfdd2", "metadata": { "editable": true }, @@ -479,19 +479,19 @@ }, { "cell_type": "markdown", - "id": "01c31fce", + "id": "3533f92e", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\theta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "\\hat{\\boldsymbol{\\theta}} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "869d8825", + "id": "135631d4", "metadata": { "editable": true }, @@ -504,14 +504,14 @@ "supervised learning algorithms like classification with logistic\n", "regression or support vector machines, exhibit dimensionalities which\n", "allow for the usage of direct linear algebra methods such as **LU** decomposition or **Singular Value Decomposition** (SVD) for finding the inverse of the matrix\n", - "$\\boldsymbol{X}^T\\boldsymbol{X}$. This is discussed on Thursday this week.\n", + "$\\boldsymbol{X}^T\\boldsymbol{X}$.\n", "\n", - "**Small question**: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$? What kind of problems can we expect?" + "**Small question**: When inverting the matrix $\\boldsymbol{X}^T\\boldsymbol{X}, what kind of problems can we expect?" ] }, { "cell_type": "markdown", - "id": "86056e8c", + "id": "cf25f4a7", "metadata": { "editable": true }, @@ -538,7 +538,7 @@ }, { "cell_type": "markdown", - "id": "d36297d2", + "id": "5d82400a", "metadata": { "editable": true }, @@ -550,7 +550,7 @@ }, { "cell_type": "markdown", - "id": "42a61ab6", + "id": "00d8d123", "metadata": { "editable": true }, @@ -562,7 +562,7 @@ }, { "cell_type": "markdown", - "id": "6216fdfa", + "id": "cce17e14", "metadata": { "editable": true }, @@ -578,7 +578,7 @@ }, { "cell_type": "markdown", - "id": "1d8f89cb", + "id": "65efb775", "metadata": { "editable": true }, @@ -596,7 +596,7 @@ }, { "cell_type": "markdown", - "id": "4c3d1ea5", + "id": "2df9efd6", "metadata": { "editable": true }, @@ -608,7 +608,7 @@ }, { "cell_type": "markdown", - "id": "41d96182", + "id": "37b621d8", "metadata": { "editable": true }, @@ -620,7 +620,7 @@ }, { "cell_type": "markdown", - "id": "187e7d47", + "id": "297be9c9", "metadata": { "editable": true }, @@ -631,7 +631,7 @@ }, { "cell_type": "markdown", - "id": "fe831c6b", + "id": "fe288c99", "metadata": { "editable": true }, @@ -643,7 +643,7 @@ }, { "cell_type": "markdown", - "id": "5e9942fa", + "id": "d4333779", "metadata": { "editable": true }, @@ -653,7 +653,7 @@ }, { "cell_type": "markdown", - "id": "a1589dc4", + "id": "f7e72c3d", "metadata": { "editable": true }, @@ -665,7 +665,7 @@ }, { "cell_type": "markdown", - "id": "f4980fc7", + "id": "3423b702", "metadata": { "editable": true }, @@ -679,7 +679,7 @@ }, { "cell_type": "markdown", - "id": "5e5803ca", + "id": "fe66f713", "metadata": { "editable": true }, @@ -691,7 +691,7 @@ }, { "cell_type": "markdown", - "id": "8ff2d8a0", + "id": "43eed51b", "metadata": { "editable": true }, @@ -703,7 +703,7 @@ }, { "cell_type": "markdown", - "id": "a14f7d1e", + "id": "6f943ccb", "metadata": { "editable": true }, @@ -715,7 +715,7 @@ }, { "cell_type": "markdown", - "id": "8d0c272b", + "id": "5df09c55", "metadata": { "editable": true }, @@ -725,7 +725,7 @@ }, { "cell_type": "markdown", - "id": "9bfc54a6", + "id": "b92d9f2a", "metadata": { "editable": true }, @@ -737,7 +737,7 @@ }, { "cell_type": "markdown", - "id": "f68602df", + "id": "c6cf49ab", "metadata": { "editable": true }, @@ -749,7 +749,7 @@ }, { "cell_type": "markdown", - "id": "f3a9b46b", + "id": "3e000448", "metadata": { "editable": true }, @@ -761,7 +761,7 @@ }, { "cell_type": "markdown", - "id": "8b9c9796", + "id": "fbaa984b", "metadata": { "editable": true }, @@ -775,7 +775,7 @@ }, { "cell_type": "markdown", - "id": "e55c0131", + "id": "f4e444ea", "metadata": { "editable": true }, @@ -787,7 +787,7 @@ }, { "cell_type": "markdown", - "id": "0213690c", + "id": "454ba5ba", "metadata": { "editable": true }, @@ -799,7 +799,7 @@ }, { "cell_type": "markdown", - "id": "4bebb2ab", + "id": "c0434cd9", "metadata": { "editable": true }, @@ -811,7 +811,7 @@ }, { "cell_type": "markdown", - "id": "f90da960", + "id": "f2bcda9d", "metadata": { "editable": true }, @@ -821,7 +821,7 @@ }, { "cell_type": "markdown", - "id": "8178ffe4", + "id": "aea0ba73", "metadata": { "editable": true }, @@ -833,7 +833,7 @@ }, { "cell_type": "markdown", - "id": "d937141e", + "id": "5a58d6e8", "metadata": { "editable": true }, @@ -843,7 +843,7 @@ }, { "cell_type": "markdown", - "id": "a5a8213f", + "id": "927428aa", "metadata": { "editable": true }, @@ -855,7 +855,7 @@ }, { "cell_type": "markdown", - "id": "3bdb5bee", + "id": "372ad443", "metadata": { "editable": true }, @@ -865,7 +865,7 @@ }, { "cell_type": "markdown", - "id": "4a9ff5dc", + "id": "37a99489", "metadata": { "editable": true }, @@ -877,7 +877,7 @@ }, { "cell_type": "markdown", - "id": "97098140", + "id": "4872a1cd", "metadata": { "editable": true }, @@ -889,7 +889,7 @@ }, { "cell_type": "markdown", - "id": "76008e78", + "id": "b21e9cb2", "metadata": { "editable": true }, @@ -901,7 +901,7 @@ }, { "cell_type": "markdown", - "id": "7e07b9bf", + "id": "442375d1", "metadata": { "editable": true }, @@ -916,7 +916,7 @@ }, { "cell_type": "markdown", - "id": "a691ed11", + "id": "aabab237", "metadata": { "editable": true }, @@ -928,7 +928,7 @@ }, { "cell_type": "markdown", - "id": "73cfbba0", + "id": "106f09df", "metadata": { "editable": true }, @@ -938,7 +938,7 @@ }, { "cell_type": "markdown", - "id": "d78a9df5", + "id": "73d8f824", "metadata": { "editable": true }, @@ -950,7 +950,7 @@ }, { "cell_type": "markdown", - "id": "9e1e3a87", + "id": "f2626be7", "metadata": { "editable": true }, @@ -960,7 +960,7 @@ }, { "cell_type": "markdown", - "id": "aea5a5e4", + "id": "81d89d86", "metadata": { "editable": true }, @@ -972,7 +972,7 @@ }, { "cell_type": "markdown", - "id": "89e6fde8", + "id": "59c2576a", "metadata": { "editable": true }, @@ -982,7 +982,7 @@ }, { "cell_type": "markdown", - "id": "6c0f4449", + "id": "4b7bf3f9", "metadata": { "editable": true }, @@ -994,7 +994,7 @@ }, { "cell_type": "markdown", - "id": "a390ac6e", + "id": "ea1e8943", "metadata": { "editable": true }, @@ -1006,7 +1006,7 @@ }, { "cell_type": "markdown", - "id": "59d6ab1c", + "id": "d0425d72", "metadata": { "editable": true }, @@ -1018,7 +1018,7 @@ }, { "cell_type": "markdown", - "id": "8c99ee21", + "id": "38386834", "metadata": { "editable": true }, @@ -1028,7 +1028,7 @@ }, { "cell_type": "markdown", - "id": "7ea454e1", + "id": "bfbc4bd3", "metadata": { "editable": true }, @@ -1040,7 +1040,7 @@ }, { "cell_type": "markdown", - "id": "5a555af4", + "id": "3ced3cec", "metadata": { "editable": true }, @@ -1053,7 +1053,7 @@ }, { "cell_type": "markdown", - "id": "a39c02c1", + "id": "fc6f1dec", "metadata": { "editable": true }, @@ -1065,7 +1065,7 @@ }, { "cell_type": "markdown", - "id": "a9804433", + "id": "699662d2", "metadata": { "editable": true }, @@ -1075,7 +1075,7 @@ }, { "cell_type": "markdown", - "id": "8be8de93", + "id": "0991592c", "metadata": { "editable": true }, @@ -1087,7 +1087,7 @@ }, { "cell_type": "markdown", - "id": "4139dd13", + "id": "450dd961", "metadata": { "editable": true }, @@ -1097,7 +1097,7 @@ }, { "cell_type": "markdown", - "id": "6f1c778a", + "id": "a13de0fa", "metadata": { "editable": true }, @@ -1109,7 +1109,7 @@ }, { "cell_type": "markdown", - "id": "03ae847f", + "id": "c64bc916", "metadata": { "editable": true }, @@ -1119,7 +1119,7 @@ }, { "cell_type": "markdown", - "id": "054f8ba7", + "id": "ab1e18bc", "metadata": { "editable": true }, @@ -1131,7 +1131,7 @@ }, { "cell_type": "markdown", - "id": "61c62efa", + "id": "f1e74f34", "metadata": { "editable": true }, @@ -1141,7 +1141,7 @@ }, { "cell_type": "markdown", - "id": "ac84f4fd", + "id": "1ad3ebd9", "metadata": { "editable": true }, @@ -1153,7 +1153,7 @@ }, { "cell_type": "markdown", - "id": "b2f0a3d0", + "id": "37d1b4ca", "metadata": { "editable": true }, @@ -1163,7 +1163,7 @@ }, { "cell_type": "markdown", - "id": "9c6460f2", + "id": "3d5fc67a", "metadata": { "editable": true }, @@ -1175,67 +1175,7 @@ }, { "cell_type": "markdown", - "id": "0c4aebc4", - "metadata": { - "editable": true - }, - "source": [ - "## Other useful relations\n", - "\n", - "We list here some other useful relations we may encounter (recall that vectors are defined by boldfaced low-key letters)" - ] - }, - { - "cell_type": "markdown", - "id": "67aef36e", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{\\partial (\\boldsymbol{x}^T\\boldsymbol{a})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "30c88be7", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "fb84fcda", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{\\partial tr(\\boldsymbol{B}\\boldsymbol{A})}{\\partial \\boldsymbol{A}} = \\boldsymbol{B}^T,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "ebbeeb3b", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{\\partial \\log{\\vert\\boldsymbol{A}\\vert}}{\\partial \\boldsymbol{A}} = (\\boldsymbol{A}^{-1})^T.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "fb88afbf", + "id": "d98472d9", "metadata": { "editable": true }, @@ -1251,7 +1191,7 @@ }, { "cell_type": "markdown", - "id": "89512329", + "id": "5d859f20", "metadata": { "editable": true }, @@ -1263,7 +1203,7 @@ }, { "cell_type": "markdown", - "id": "a651d788", + "id": "e7bb91ec", "metadata": { "editable": true }, @@ -1273,7 +1213,7 @@ }, { "cell_type": "markdown", - "id": "dca9d68f", + "id": "1553864b", "metadata": { "editable": true }, @@ -1285,25 +1225,25 @@ }, { "cell_type": "markdown", - "id": "2865ced8", + "id": "f8cc9c71", "metadata": { "editable": true }, "source": [ "For ordinary least squares, it is inversely proportional (derivation\n", "next week) with the variance of the optimal parameters\n", - "$\\hat{\\boldsymbol{\\theta}}$. Furthermore, we will see later this week that it is\n", + "$\\hat{\\boldsymbol{\\theta}}$. Furthermore, we will see next week that it is\n", "(aside the factor $1/n$) equal to the covariance matrix. It plays also a very\n", "important role in optmization algorithms and Principal Component\n", "Analysis as a way to reduce the dimensionality of a machine learning/data analysis\n", - "problem.\n", - "v\n", + "problem. We will discuss this in greater detail next week when we introduce gradient methods.\n", + "\n", "**Linear algebra question:** Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!)." ] }, { "cell_type": "markdown", - "id": "1a7a4045", + "id": "f1f01477", "metadata": { "editable": true }, @@ -1315,7 +1255,7 @@ }, { "cell_type": "markdown", - "id": "a483362e", + "id": "f2dea7e1", "metadata": { "editable": true }, @@ -1327,7 +1267,7 @@ }, { "cell_type": "markdown", - "id": "dcf896a3", + "id": "76b0f99e", "metadata": { "editable": true }, @@ -1337,7 +1277,7 @@ }, { "cell_type": "markdown", - "id": "13c80a7c", + "id": "cfa3ca25", "metadata": { "editable": true }, @@ -1349,7 +1289,7 @@ }, { "cell_type": "markdown", - "id": "582225c6", + "id": "9a9163d8", "metadata": { "editable": true }, @@ -1359,7 +1299,7 @@ }, { "cell_type": "markdown", - "id": "90476b18", + "id": "c6b54a1e", "metadata": { "editable": true }, @@ -1371,7 +1311,7 @@ }, { "cell_type": "markdown", - "id": "267ae1ff", + "id": "659c98d0", "metadata": { "editable": true }, @@ -1381,7 +1321,7 @@ }, { "cell_type": "markdown", - "id": "8c9cb8c5", + "id": "209b5b37", "metadata": { "editable": true }, @@ -1395,7 +1335,7 @@ }, { "cell_type": "markdown", - "id": "acbe4114", + "id": "b325d0c3", "metadata": { "editable": true }, @@ -1407,7 +1347,7 @@ }, { "cell_type": "markdown", - "id": "ba945f21", + "id": "c49a2fcd", "metadata": { "editable": true }, @@ -1418,7 +1358,7 @@ }, { "cell_type": "markdown", - "id": "dc442b90", + "id": "6978ef70", "metadata": { "editable": true }, @@ -1431,14 +1371,14 @@ { "cell_type": "code", "execution_count": 1, - "id": "883026f0", + "id": "01530c39", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ - "# matrix inversion to find beta\n", + "# matrix inversion to find theta\n", "# First we set up the data\n", "import numpy as np\n", "x = np.random.rand(100)\n", @@ -1451,14 +1391,14 @@ "X[:,2] = x**2\n", "X[:,3] = x**3\n", "X[:,4] = x**4\n", - "beta = (np.linalg.inv(X.T @ X) @ X.T ) @ y\n", + "theta = (np.linalg.inv(X.T @ X) @ X.T ) @ y\n", "# and then make the prediction\n", - "ytilde = X @ beta" + "ytilde = X @ theta" ] }, { "cell_type": "markdown", - "id": "826466d3", + "id": "90ff28d4", "metadata": { "editable": true }, @@ -1469,7 +1409,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "decea3b4", + "id": "1d052cbe", "metadata": { "collapsed": false, "editable": true @@ -1482,7 +1422,7 @@ }, { "cell_type": "markdown", - "id": "b52ed870", + "id": "3362efd4", "metadata": { "editable": true }, @@ -1496,7 +1436,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "ee4d5722", + "id": "3e9e681f", "metadata": { "collapsed": false, "editable": true @@ -1509,7 +1449,7 @@ }, { "cell_type": "markdown", - "id": "01892aaa", + "id": "78e4427f", "metadata": { "editable": true }, @@ -1520,7 +1460,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "d6ce54e2", + "id": "20aab659", "metadata": { "collapsed": false, "editable": true @@ -1532,7 +1472,7 @@ }, { "cell_type": "markdown", - "id": "91ed5956", + "id": "326f641d", "metadata": { "editable": true }, @@ -1543,7 +1483,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "7ffd63cb", + "id": "c8e80cf1", "metadata": { "collapsed": false, "editable": true @@ -1559,7 +1499,7 @@ }, { "cell_type": "markdown", - "id": "737370ed", + "id": "aba7b289", "metadata": { "editable": true }, @@ -1570,7 +1510,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "f1bbe22d", + "id": "7235bf1b", "metadata": { "collapsed": false, "editable": true @@ -1584,7 +1524,7 @@ }, { "cell_type": "markdown", - "id": "8e3a976b", + "id": "3eaf5f8d", "metadata": { "editable": true }, @@ -1605,7 +1545,7 @@ }, { "cell_type": "markdown", - "id": "8c2dee8c", + "id": "6fc5dc26", "metadata": { "editable": true }, @@ -1616,7 +1556,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "89d0750d", + "id": "98ca5627", "metadata": { "collapsed": false, "editable": true @@ -1651,16 +1591,16 @@ "X[:,4] = x**4\n", "# We split the data in test and training data\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n", - "# matrix inversion to find beta\n", - "beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", - "print(beta)\n", + "# matrix inversion to find theta\n", + "theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", + "print(theta)\n", "# and then make the prediction\n", - "ytilde = X_train @ beta\n", + "ytilde = X_train @ theta\n", "print(\"Training R2\")\n", "print(R2(y_train,ytilde))\n", "print(\"Training MSE\")\n", "print(MSE(y_train,ytilde))\n", - "ypredict = X_test @ beta\n", + "ypredict = X_test @ theta\n", "print(\"Test R2\")\n", "print(R2(y_test,ypredict))\n", "print(\"Test MSE\")\n", @@ -1669,7 +1609,7 @@ }, { "cell_type": "markdown", - "id": "12c44026", + "id": "09f0c1c6", "metadata": { "editable": true }, @@ -1680,7 +1620,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "6daa8ad9", + "id": "02220d97", "metadata": { "collapsed": false, "editable": true @@ -1705,7 +1645,7 @@ }, { "cell_type": "markdown", - "id": "9b4cbdbc", + "id": "5494af4d", "metadata": { "editable": true }, @@ -1717,7 +1657,7 @@ }, { "cell_type": "markdown", - "id": "7f685bde", + "id": "048560ee", "metadata": { "editable": true }, @@ -1746,7 +1686,7 @@ }, { "cell_type": "markdown", - "id": "a8e92a0b", + "id": "85b56c79", "metadata": { "editable": true }, @@ -1771,7 +1711,7 @@ }, { "cell_type": "markdown", - "id": "b0ac1e21", + "id": "60937483", "metadata": { "editable": true }, @@ -1791,7 +1731,7 @@ }, { "cell_type": "markdown", - "id": "3c8d9613", + "id": "5d0ea6a3", "metadata": { "editable": true }, @@ -1818,7 +1758,7 @@ }, { "cell_type": "markdown", - "id": "1d817e20", + "id": "8f6fc4e6", "metadata": { "editable": true }, @@ -1831,7 +1771,7 @@ }, { "cell_type": "markdown", - "id": "20673f07", + "id": "4bd392ed", "metadata": { "editable": true }, @@ -1843,7 +1783,7 @@ }, { "cell_type": "markdown", - "id": "4a36afc7", + "id": "4e33d3e3", "metadata": { "editable": true }, @@ -1854,7 +1794,7 @@ }, { "cell_type": "markdown", - "id": "4db40306", + "id": "7e1c73ac", "metadata": { "editable": true }, @@ -1870,7 +1810,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "ac782b4e", + "id": "2c315558", "metadata": { "collapsed": false, "editable": true @@ -1904,7 +1844,7 @@ }, { "cell_type": "markdown", - "id": "d60792bf", + "id": "aaa58775", "metadata": { "editable": true }, @@ -1914,7 +1854,7 @@ }, { "cell_type": "markdown", - "id": "6dbeeee3", + "id": "02b198ed", "metadata": { "editable": true }, @@ -1929,7 +1869,7 @@ }, { "cell_type": "markdown", - "id": "f1bd9e06", + "id": "4cef197d", "metadata": { "editable": true }, @@ -1941,7 +1881,7 @@ }, { "cell_type": "markdown", - "id": "c4e9c9cc", + "id": "23b34528", "metadata": { "editable": true }, @@ -1951,7 +1891,7 @@ }, { "cell_type": "markdown", - "id": "c682dfb2", + "id": "b2847956", "metadata": { "editable": true }, @@ -1967,7 +1907,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "f95e4477", + "id": "b8dacc33", "metadata": { "collapsed": false, "editable": true @@ -1984,7 +1924,7 @@ }, { "cell_type": "markdown", - "id": "3eaa6d2e", + "id": "cd89705f", "metadata": { "editable": true }, @@ -1997,7 +1937,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "fe340fe4", + "id": "d35541fd", "metadata": { "collapsed": false, "editable": true @@ -2044,7 +1984,7 @@ }, { "cell_type": "markdown", - "id": "e2f14a01", + "id": "36014ec9", "metadata": { "editable": true }, @@ -2058,7 +1998,7 @@ }, { "cell_type": "markdown", - "id": "ad741d5a", + "id": "16fb5f8f", "metadata": { "editable": true }, @@ -2070,7 +2010,7 @@ }, { "cell_type": "markdown", - "id": "05b080a8", + "id": "f1ec698e", "metadata": { "editable": true }, @@ -2082,7 +2022,7 @@ }, { "cell_type": "markdown", - "id": "5c64cd00", + "id": "787f58f5", "metadata": { "editable": true }, @@ -2094,7 +2034,7 @@ }, { "cell_type": "markdown", - "id": "66d8648a", + "id": "e23435fd", "metadata": { "editable": true }, @@ -2104,7 +2044,7 @@ }, { "cell_type": "markdown", - "id": "47178854", + "id": "ccaad283", "metadata": { "editable": true }, @@ -2116,7 +2056,7 @@ }, { "cell_type": "markdown", - "id": "9d89dcc7", + "id": "be8092da", "metadata": { "editable": true }, @@ -2126,7 +2066,7 @@ }, { "cell_type": "markdown", - "id": "d037dda7", + "id": "6e055e08", "metadata": { "editable": true }, @@ -2138,7 +2078,7 @@ }, { "cell_type": "markdown", - "id": "4d559b4d", + "id": "ff86c9bc", "metadata": { "editable": true }, @@ -2149,7 +2089,7 @@ }, { "cell_type": "markdown", - "id": "f8eec117", + "id": "b4f2e747", "metadata": { "editable": true }, @@ -2161,7 +2101,7 @@ }, { "cell_type": "markdown", - "id": "ecc2fed3", + "id": "5a9a9f87", "metadata": { "editable": true }, @@ -2173,7 +2113,7 @@ }, { "cell_type": "markdown", - "id": "8d3ae448", + "id": "1232e11a", "metadata": { "editable": true }, @@ -2183,7 +2123,7 @@ }, { "cell_type": "markdown", - "id": "0d7267fa", + "id": "a807825b", "metadata": { "editable": true }, @@ -2195,7 +2135,7 @@ }, { "cell_type": "markdown", - "id": "06e5716f", + "id": "4fa9855c", "metadata": { "editable": true }, @@ -2207,7 +2147,7 @@ }, { "cell_type": "markdown", - "id": "f8f001a6", + "id": "dcffefcf", "metadata": { "editable": true }, @@ -2217,7 +2157,7 @@ }, { "cell_type": "markdown", - "id": "761f6c51", + "id": "3c08ff3a", "metadata": { "editable": true }, @@ -2229,7 +2169,7 @@ }, { "cell_type": "markdown", - "id": "468bc321", + "id": "c2483a13", "metadata": { "editable": true }, @@ -2239,7 +2179,7 @@ }, { "cell_type": "markdown", - "id": "eb61e7ef", + "id": "9a82a8c5", "metadata": { "editable": true }, @@ -2251,7 +2191,7 @@ }, { "cell_type": "markdown", - "id": "be3586ce", + "id": "743f7309", "metadata": { "editable": true }, @@ -2261,7 +2201,7 @@ }, { "cell_type": "markdown", - "id": "f79b7c51", + "id": "e6d7b31a", "metadata": { "editable": true }, @@ -2278,7 +2218,7 @@ "As we will also see in the first project, \n", "this may\n", "however not the be case in general and a standard matrix inversion\n", - "algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.\n", + "algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below and in other examples.\n", "\n", "There is however a way to circumvent this problem and also\n", "gain some insights about the ordinary least squares approach, and\n", @@ -2301,7 +2241,7 @@ }, { "cell_type": "markdown", - "id": "ea84dc35", + "id": "518d6bb4", "metadata": { "editable": true }, @@ -2318,7 +2258,7 @@ }, { "cell_type": "markdown", - "id": "0ad4b3f9", + "id": "056673c8", "metadata": { "editable": true }, @@ -2341,7 +2281,7 @@ }, { "cell_type": "markdown", - "id": "671100e8", + "id": "63daba59", "metadata": { "editable": true }, @@ -2358,7 +2298,7 @@ }, { "cell_type": "markdown", - "id": "9d07ce4a", + "id": "89b74667", "metadata": { "editable": true }, @@ -2377,7 +2317,7 @@ }, { "cell_type": "markdown", - "id": "a26b13de", + "id": "588dd662", "metadata": { "editable": true }, @@ -2388,7 +2328,7 @@ }, { "cell_type": "markdown", - "id": "2352aaa2", + "id": "eb8348c6", "metadata": { "editable": true }, @@ -2400,7 +2340,7 @@ }, { "cell_type": "markdown", - "id": "5c201fe7", + "id": "62027bc1", "metadata": { "editable": true }, @@ -2418,7 +2358,7 @@ }, { "cell_type": "markdown", - "id": "a711f9aa", + "id": "0c73c4b7", "metadata": { "editable": true }, @@ -2434,7 +2374,7 @@ }, { "cell_type": "markdown", - "id": "816ee812", + "id": "a3a2d330", "metadata": { "editable": true }, @@ -2446,7 +2386,7 @@ }, { "cell_type": "markdown", - "id": "ca4f2d5d", + "id": "374b88cc", "metadata": { "editable": true }, @@ -2456,14 +2396,412 @@ }, { "cell_type": "markdown", - "id": "11b18875", + "id": "d5aa42e6", + "metadata": { + "editable": true + }, + "source": [ + "## Ridge and LASSO Regression\n", + "\n", + "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", + "our optimization problem is" + ] + }, + { + "cell_type": "markdown", + "id": "1c01a816", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f556eddc", + "metadata": { + "editable": true + }, + "source": [ + "or we can state it as" + ] + }, + { + "cell_type": "markdown", + "id": "dcf14a16", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "08c37095", + "metadata": { + "editable": true + }, + "source": [ + "where we have used the definition of a norm-2 vector, that is" + ] + }, + { + "cell_type": "markdown", + "id": "6815fa1b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "daf50946", + "metadata": { + "editable": true + }, + "source": [ + "By minimizing the above equation with respect to the parameters\n", + "$\\boldsymbol{\\theta}$ we could then obtain an analytical expression for the\n", + "parameters $\\boldsymbol{\\theta}$. We can add a regularization parameter $\\lambda$ by\n", + "defining a new cost function to be optimized, that is" + ] + }, + { + "cell_type": "markdown", + "id": "a7a60cb3", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "38105bfd", + "metadata": { + "editable": true + }, + "source": [ + "which leads to the Ridge regression minimization problem where we\n", + "require that $\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\\le t$, where $t$ is\n", + "a finite number larger than zero. By defining" + ] + }, + { + "cell_type": "markdown", + "id": "3d697464", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c298d7ba", + "metadata": { + "editable": true + }, + "source": [ + "we have a new optimization equation" + ] + }, + { + "cell_type": "markdown", + "id": "45265b59", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ab6ce408", + "metadata": { + "editable": true + }, + "source": [ + "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", + "\n", + "Here we have defined the norm-1 as" + ] + }, + { + "cell_type": "markdown", + "id": "ca0350a7", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e8146e6f", + "metadata": { + "editable": true + }, + "source": [ + "## Deriving the Ridge Regression Equations\n", + "\n", + "Using the matrix-vector expression for Ridge regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have" + ] + }, + { + "cell_type": "markdown", + "id": "05d97a86", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\boldsymbol{\\theta}^T\\boldsymbol{\\theta},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f84e7118", + "metadata": { + "editable": true + }, + "source": [ + "and \n", + "taking the derivatives with respect to $\\boldsymbol{\\theta}$ we obtain then\n", + "a slightly modified matrix inversion problem which for finite values\n", + "of $\\lambda$ does not suffer from singularity problems. We obtain\n", + "the optimal parameters" + ] + }, + { + "cell_type": "markdown", + "id": "28be9ec4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b9f5a5d8", + "metadata": { + "editable": true + }, + "source": [ + "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" + ] + }, + { + "cell_type": "markdown", + "id": "86b43b21", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\sum_{i=0}^{p-1} \\theta_i^2 \\leq t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "58744172", + "metadata": { + "editable": true + }, + "source": [ + "with $t$ a finite positive number. \n", + "\n", + "If we keep the $1/n$ factor, the equation for the optimal $\\theta$ changes to" + ] + }, + { + "cell_type": "markdown", + "id": "dd515e6f", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e8c98bfe", + "metadata": { + "editable": true + }, + "source": [ + "In many textbooks the $1/n$ term is often omitted. Note that a library like **Scikit-Learn** does not include the $1/n$ factor in the setup of the cost function.\n", + "\n", + "When we compare this with the ordinary least squares result we have" + ] + }, + { + "cell_type": "markdown", + "id": "08cc9626", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "725df93e", + "metadata": { + "editable": true + }, + "source": [ + "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", + "\n", + "We see that Ridge regression is nothing but the standard OLS with a\n", + "modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n", + "particular for our discussion of the bias-variance tradeoff are rather\n", + "interesting. We will see that for specific values of $\\lambda$, we may\n", + "even reduce the variance of the optimal parameters $\\boldsymbol{\\theta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n", + "\n", + "When we have discussed the singular value decomposition of the design\n", + "matrix $\\boldsymbol{X}$, we will in turn perform a more rigorous mathematical\n", + "discussion of Ridge regression.\n", + "\n", + "The code here is a simple demonstration of how to implement Ridge regression with our own code and compare this with scikit-learn." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "8d0397cd", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd\n", + "import matplotlib.pyplot as plt\n", + "from sklearn.model_selection import train_test_split\n", + "from sklearn import linear_model\n", + "\n", + "def MSE(y_data,y_model):\n", + " n = np.size(y_model)\n", + " return np.sum((y_data-y_model)**2)/n\n", + "\n", + "\n", + "# A seed just to ensure that the random numbers are the same for every run.\n", + "# Useful for eventual debugging.\n", + "np.random.seed(3155)\n", + "\n", + "n = 100\n", + "x = np.random.rand(n)\n", + "y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)\n", + "\n", + "Maxpolydegree = 20\n", + "X = np.zeros((n,Maxpolydegree))\n", + "#We include explicitely the intercept column\n", + "for degree in range(Maxpolydegree):\n", + " X[:,degree] = x**degree\n", + "# We split the data in test and training data\n", + "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n", + "\n", + "p = Maxpolydegree\n", + "I = np.eye(p,p)\n", + "# Decide which values of lambda to use\n", + "nlambdas = 6\n", + "MSEOwnRidgePredict = np.zeros(nlambdas)\n", + "MSERidgePredict = np.zeros(nlambdas)\n", + "lambdas = np.logspace(-4, 2, nlambdas)\n", + "for i in range(nlambdas):\n", + " lmb = lambdas[i]\n", + " OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n", + " # Note: we include the intercept column and no scaling\n", + " RegRidge = linear_model.Ridge(lmb,fit_intercept=False)\n", + " RegRidge.fit(X_train,y_train)\n", + " # and then make the prediction\n", + " ytildeOwnRidge = X_train @ OwnRidgeTheta\n", + " ypredictOwnRidge = X_test @ OwnRidgeTheta\n", + " ytildeRidge = RegRidge.predict(X_train)\n", + " ypredictRidge = RegRidge.predict(X_test)\n", + " MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)\n", + " MSERidgePredict[i] = MSE(y_test,ypredictRidge)\n", + " print(\"Theta values for own Ridge implementation\")\n", + " print(OwnRidgeTheta)\n", + " print(\"Theta values for Scikit-Learn Ridge implementation\")\n", + " print(RegRidge.coef_)\n", + " print(\"MSE values for own Ridge implementation\")\n", + " print(MSEOwnRidgePredict[i])\n", + " print(\"MSE values for Scikit-Learn Ridge implementation\")\n", + " print(MSERidgePredict[i])\n", + "\n", + "# Now plot the results\n", + "plt.figure()\n", + "plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'r', label = 'MSE own Ridge Test')\n", + "plt.plot(np.log10(lambdas), MSERidgePredict, 'g', label = 'MSE Ridge Test')\n", + "\n", + "plt.xlabel('log10(lambda)')\n", + "plt.ylabel('MSE')\n", + "plt.legend()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "dbc56c2b", + "metadata": { + "editable": true + }, + "source": [ + "The results here agree when we force **Scikit-Learn**'s Ridge function to include the first column in our design matrix.\n", + "We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.\n", + "What happens if we do not include the intercept in our fit? We will discuss this in more detail next week." + ] + }, + { + "cell_type": "markdown", + "id": "cfab477f", "metadata": { "editable": true }, "source": [ "## Basic math of the SVD\n", "\n", - "From standard linear algebra we know that a square matrix $\\boldsymbol{X}$ can be diagonalized if and only it is \n", + "From standard linear algebra we know that a square matrix $\\boldsymbol{X}$ can be diagonalized if and only if it is \n", "a so-called [normal matrix](https://en.wikipedia.org/wiki/Normal_matrix), that is if $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times n}$\n", "we have $\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{X}^T\\boldsymbol{X}$ or if $\\boldsymbol{X}\\in {\\mathbb{C}}^{n\\times n}$ we have $\\boldsymbol{X}\\boldsymbol{X}^{\\dagger}=\\boldsymbol{X}^{\\dagger}\\boldsymbol{X}$.\n", "The matrix has then a set of eigenpairs" @@ -2471,7 +2809,7 @@ }, { "cell_type": "markdown", - "id": "a6cebe96", + "id": "bc176ed4", "metadata": { "editable": true }, @@ -2483,7 +2821,7 @@ }, { "cell_type": "markdown", - "id": "4056c51d", + "id": "a9eef8e3", "metadata": { "editable": true }, @@ -2493,7 +2831,7 @@ }, { "cell_type": "markdown", - "id": "8c6fc251", + "id": "c33552ec", "metadata": { "editable": true }, @@ -2505,7 +2843,7 @@ }, { "cell_type": "markdown", - "id": "85d151ae", + "id": "9465d4f6", "metadata": { "editable": true }, @@ -2515,7 +2853,7 @@ }, { "cell_type": "markdown", - "id": "a7ed48ac", + "id": "3cd89a35", "metadata": { "editable": true }, @@ -2527,7 +2865,7 @@ }, { "cell_type": "markdown", - "id": "d61100de", + "id": "13a9f453", "metadata": { "editable": true }, @@ -2539,7 +2877,7 @@ }, { "cell_type": "markdown", - "id": "ae931201", + "id": "6c909f72", "metadata": { "editable": true }, @@ -2554,7 +2892,7 @@ }, { "cell_type": "markdown", - "id": "a1a55970", + "id": "b32ee709", "metadata": { "editable": true }, @@ -2565,7 +2903,7 @@ }, { "cell_type": "markdown", - "id": "7aff14ea", + "id": "e5d3f72c", "metadata": { "editable": true }, @@ -2585,7 +2923,7 @@ }, { "cell_type": "markdown", - "id": "c5bcd441", + "id": "18f488f0", "metadata": { "editable": true }, @@ -2597,7 +2935,7 @@ }, { "cell_type": "markdown", - "id": "ed33684a", + "id": "93618783", "metadata": { "editable": true }, @@ -2607,7 +2945,7 @@ }, { "cell_type": "markdown", - "id": "1d6ca687", + "id": "02f57528", "metadata": { "editable": true }, @@ -2619,7 +2957,7 @@ }, { "cell_type": "markdown", - "id": "c0535cd8", + "id": "762833f4", "metadata": { "editable": true }, @@ -2648,7 +2986,7 @@ }, { "cell_type": "markdown", - "id": "c6c08bf2", + "id": "9ab8ec4f", "metadata": { "editable": true }, @@ -2675,7 +3013,7 @@ }, { "cell_type": "markdown", - "id": "487a7ae7", + "id": "3e526d1f", "metadata": { "editable": true }, @@ -2685,8 +3023,8 @@ }, { "cell_type": "code", - "execution_count": 12, - "id": "3ff87df9", + "execution_count": 13, + "id": "f5d258ce", "metadata": { "collapsed": false, "editable": true @@ -2726,7 +3064,7 @@ }, { "cell_type": "markdown", - "id": "600f74f5", + "id": "4e4bf198", "metadata": { "editable": true }, @@ -2743,7 +3081,7 @@ }, { "cell_type": "markdown", - "id": "da923c99", + "id": "33a42ac4", "metadata": { "editable": true }, @@ -2766,7 +3104,7 @@ }, { "cell_type": "markdown", - "id": "989e9de3", + "id": "7b5d1bbb", "metadata": { "editable": true }, @@ -2780,7 +3118,7 @@ }, { "cell_type": "markdown", - "id": "b010904b", + "id": "1fee5f54", "metadata": { "editable": true }, @@ -2799,7 +3137,7 @@ }, { "cell_type": "markdown", - "id": "1385bdc7", + "id": "235f2975", "metadata": { "editable": true }, @@ -2809,7 +3147,7 @@ }, { "cell_type": "markdown", - "id": "1824f1de", + "id": "fd7415b7", "metadata": { "editable": true }, @@ -2821,7 +3159,7 @@ }, { "cell_type": "markdown", - "id": "50fb0869", + "id": "7c265b5d", "metadata": { "editable": true }, @@ -2835,7 +3173,7 @@ }, { "cell_type": "markdown", - "id": "242e0186", + "id": "976c31a2", "metadata": { "editable": true }, @@ -2847,7 +3185,7 @@ }, { "cell_type": "markdown", - "id": "b3d8ff35", + "id": "8dc8acd7", "metadata": { "editable": true }, @@ -2857,7 +3195,7 @@ }, { "cell_type": "markdown", - "id": "a8799cf8", + "id": "a1dc8e8e", "metadata": { "editable": true }, @@ -2869,7 +3207,7 @@ }, { "cell_type": "markdown", - "id": "69ecbf25", + "id": "eb688af0", "metadata": { "editable": true }, @@ -2886,7 +3224,7 @@ }, { "cell_type": "markdown", - "id": "accaad01", + "id": "eeadd85e", "metadata": { "editable": true }, @@ -2896,7 +3234,7 @@ }, { "cell_type": "markdown", - "id": "7c6d6458", + "id": "e9375ad7", "metadata": { "editable": true }, @@ -2912,7 +3250,7 @@ }, { "cell_type": "markdown", - "id": "92943a13", + "id": "025f1ca1", "metadata": { "editable": true }, @@ -2922,7 +3260,7 @@ }, { "cell_type": "markdown", - "id": "27a56b68", + "id": "d44951a1", "metadata": { "editable": true }, @@ -2938,7 +3276,7 @@ }, { "cell_type": "markdown", - "id": "7fab099a", + "id": "ecc9216b", "metadata": { "editable": true }, @@ -2948,7 +3286,7 @@ }, { "cell_type": "markdown", - "id": "86ef772d", + "id": "28e2ff44", "metadata": { "editable": true }, @@ -2964,7 +3302,7 @@ }, { "cell_type": "markdown", - "id": "a9fb58a3", + "id": "935324af", "metadata": { "editable": true }, @@ -2974,7 +3312,7 @@ }, { "cell_type": "markdown", - "id": "218d36d5", + "id": "a15e7526", "metadata": { "editable": true }, @@ -2991,7 +3329,7 @@ }, { "cell_type": "markdown", - "id": "3c358960", + "id": "5d3cef55", "metadata": { "editable": true }, @@ -3003,7 +3341,7 @@ }, { "cell_type": "markdown", - "id": "384a8da4", + "id": "0fab1783", "metadata": { "editable": true }, @@ -3015,7 +3353,7 @@ }, { "cell_type": "markdown", - "id": "506802df", + "id": "f9a8450e", "metadata": { "editable": true }, @@ -3027,7 +3365,7 @@ }, { "cell_type": "markdown", - "id": "d160546b", + "id": "485d1023", "metadata": { "editable": true }, @@ -3037,7 +3375,7 @@ }, { "cell_type": "markdown", - "id": "22fa787b", + "id": "a2da5dbe", "metadata": { "editable": true }, @@ -3049,7 +3387,7 @@ }, { "cell_type": "markdown", - "id": "4f9cc0fa", + "id": "13959099", "metadata": { "editable": true }, @@ -3061,7 +3399,7 @@ }, { "cell_type": "markdown", - "id": "5228fab9", + "id": "23f0b903", "metadata": { "editable": true }, @@ -3073,7 +3411,7 @@ }, { "cell_type": "markdown", - "id": "803c129e", + "id": "1c344e77", "metadata": { "editable": true }, @@ -3083,7 +3421,7 @@ }, { "cell_type": "markdown", - "id": "a0229d06", + "id": "d8064053", "metadata": { "editable": true }, @@ -3095,7 +3433,7 @@ }, { "cell_type": "markdown", - "id": "927c0677", + "id": "714ab01f", "metadata": { "editable": true }, @@ -3105,7 +3443,7 @@ }, { "cell_type": "markdown", - "id": "9b1faff4", + "id": "a3ff6988", "metadata": { "editable": true }, @@ -3117,7 +3455,7 @@ }, { "cell_type": "markdown", - "id": "2a9ba785", + "id": "5c204f79", "metadata": { "editable": true }, @@ -3133,7 +3471,7 @@ }, { "cell_type": "markdown", - "id": "13dd8452", + "id": "78c51f10", "metadata": { "editable": true }, @@ -3145,7 +3483,7 @@ }, { "cell_type": "markdown", - "id": "db18ae22", + "id": "2c432ed4", "metadata": { "editable": true }, @@ -3157,7 +3495,7 @@ }, { "cell_type": "markdown", - "id": "cc002ef4", + "id": "68a7b505", "metadata": { "editable": true }, @@ -3167,7 +3505,7 @@ }, { "cell_type": "markdown", - "id": "7808b2cd", + "id": "14268cd5", "metadata": { "editable": true }, @@ -3179,7 +3517,7 @@ }, { "cell_type": "markdown", - "id": "f6a8e671", + "id": "f3207844", "metadata": { "editable": true }, @@ -3190,7 +3528,7 @@ }, { "cell_type": "markdown", - "id": "e6298500", + "id": "4c7f5556", "metadata": { "editable": true }, @@ -3202,7 +3540,7 @@ }, { "cell_type": "markdown", - "id": "1a70e57a", + "id": "01bd8826", "metadata": { "editable": true }, @@ -3212,7 +3550,7 @@ }, { "cell_type": "markdown", - "id": "f2091227", + "id": "cb083248", "metadata": { "editable": true }, @@ -3224,7 +3562,7 @@ }, { "cell_type": "markdown", - "id": "029a7fce", + "id": "66bd1ec9", "metadata": { "editable": true }, @@ -3234,7 +3572,7 @@ }, { "cell_type": "markdown", - "id": "62dd4376", + "id": "b92a06bb", "metadata": { "editable": true }, @@ -3246,7 +3584,7 @@ }, { "cell_type": "markdown", - "id": "00865f17", + "id": "dad0b409", "metadata": { "editable": true }, @@ -3257,7 +3595,7 @@ }, { "cell_type": "markdown", - "id": "64bd2f73", + "id": "75d85fa8", "metadata": { "editable": true }, @@ -3269,7 +3607,7 @@ }, { "cell_type": "markdown", - "id": "ae49067e", + "id": "d0918773", "metadata": { "editable": true }, @@ -3287,7 +3625,7 @@ }, { "cell_type": "markdown", - "id": "9ea73c78", + "id": "65592b3b", "metadata": { "editable": true }, @@ -3303,7 +3641,7 @@ }, { "cell_type": "markdown", - "id": "6dcf8fff", + "id": "442404b8", "metadata": { "editable": true }, @@ -3315,7 +3653,7 @@ }, { "cell_type": "markdown", - "id": "8f12f6ca", + "id": "50eca816", "metadata": { "editable": true }, @@ -3327,7 +3665,7 @@ }, { "cell_type": "markdown", - "id": "03362376", + "id": "a27e487e", "metadata": { "editable": true }, @@ -3339,7 +3677,7 @@ }, { "cell_type": "markdown", - "id": "76d4b169", + "id": "49c74d48", "metadata": { "editable": true }, @@ -3352,7 +3690,7 @@ }, { "cell_type": "markdown", - "id": "6775daac", + "id": "214feab7", "metadata": { "editable": true }, @@ -3368,7 +3706,7 @@ }, { "cell_type": "markdown", - "id": "dc1392b1", + "id": "e61c0669", "metadata": { "editable": true }, @@ -3382,7 +3720,7 @@ }, { "cell_type": "markdown", - "id": "7faf1d79", + "id": "9f3db006", "metadata": { "editable": true }, @@ -3392,7 +3730,7 @@ }, { "cell_type": "markdown", - "id": "4c148d48", + "id": "6dfeb8ca", "metadata": { "editable": true }, @@ -3404,7 +3742,7 @@ }, { "cell_type": "markdown", - "id": "d2549195", + "id": "da47505d", "metadata": { "editable": true }, @@ -3414,7 +3752,7 @@ }, { "cell_type": "markdown", - "id": "4d7f14be", + "id": "87c1a756", "metadata": { "editable": true }, @@ -3426,7 +3764,7 @@ }, { "cell_type": "markdown", - "id": "3d795b82", + "id": "5028dae1", "metadata": { "editable": true }, @@ -3436,7 +3774,7 @@ }, { "cell_type": "markdown", - "id": "033e1b21", + "id": "98da161b", "metadata": { "editable": true }, @@ -3450,7 +3788,7 @@ }, { "cell_type": "markdown", - "id": "5d172bfd", + "id": "69065ef4", "metadata": { "editable": true }, @@ -3467,7 +3805,7 @@ }, { "cell_type": "markdown", - "id": "672de375", + "id": "3455143e", "metadata": { "editable": true }, @@ -3483,7 +3821,7 @@ }, { "cell_type": "markdown", - "id": "d0ed234f", + "id": "e322fdc8", "metadata": { "editable": true }, @@ -3495,7 +3833,7 @@ }, { "cell_type": "markdown", - "id": "7a2bde58", + "id": "f50b583d", "metadata": { "editable": true }, @@ -3508,7 +3846,7 @@ }, { "cell_type": "markdown", - "id": "88344390", + "id": "b12b62dc", "metadata": { "editable": true }, @@ -3522,7 +3860,7 @@ }, { "cell_type": "markdown", - "id": "4019cf50", + "id": "40355c87", "metadata": { "editable": true }, @@ -3532,7 +3870,7 @@ }, { "cell_type": "markdown", - "id": "95d6e396", + "id": "ee70666b", "metadata": { "editable": true }, @@ -3545,7 +3883,7 @@ }, { "cell_type": "markdown", - "id": "d5baa57d", + "id": "dc6d0cc7", "metadata": { "editable": true }, @@ -3564,7 +3902,7 @@ }, { "cell_type": "markdown", - "id": "f310ebda", + "id": "d6b6f792", "metadata": { "editable": true }, @@ -3576,7 +3914,7 @@ }, { "cell_type": "markdown", - "id": "a8366c66", + "id": "39c76bf5", "metadata": { "editable": true }, @@ -3588,7 +3926,7 @@ }, { "cell_type": "markdown", - "id": "9983b7fb", + "id": "f2d754e5", "metadata": { "editable": true }, @@ -3598,7 +3936,7 @@ }, { "cell_type": "markdown", - "id": "196e9dff", + "id": "77a984ab", "metadata": { "editable": true }, @@ -3610,7 +3948,7 @@ }, { "cell_type": "markdown", - "id": "fd64e6a2", + "id": "33060c1a", "metadata": { "editable": true }, @@ -3623,7 +3961,7 @@ }, { "cell_type": "markdown", - "id": "46a1e1da", + "id": "ed1e8279", "metadata": { "editable": true }, @@ -3642,7 +3980,7 @@ }, { "cell_type": "markdown", - "id": "6ea1ae21", + "id": "22b40263", "metadata": { "editable": true }, @@ -3652,7 +3990,7 @@ }, { "cell_type": "markdown", - "id": "fccc043c", + "id": "1b02469c", "metadata": { "editable": true }, @@ -3671,7 +4009,7 @@ }, { "cell_type": "markdown", - "id": "5d65cf6e", + "id": "0660e456", "metadata": { "editable": true }, @@ -3689,7 +4027,7 @@ }, { "cell_type": "markdown", - "id": "f6c00582", + "id": "0a40e5c0", "metadata": { "editable": true }, @@ -3703,7 +4041,7 @@ }, { "cell_type": "markdown", - "id": "fd88cfa1", + "id": "82d7e99d", "metadata": { "editable": true }, @@ -3717,8 +4055,8 @@ }, { "cell_type": "code", - "execution_count": 13, - "id": "c6da7058", + "execution_count": 14, + "id": "6f4b429b", "metadata": { "collapsed": false, "editable": true @@ -3739,7 +4077,7 @@ }, { "cell_type": "markdown", - "id": "bd31ca44", + "id": "a38f8430", "metadata": { "editable": true }, @@ -3755,8 +4093,8 @@ }, { "cell_type": "code", - "execution_count": 14, - "id": "d809cece", + "execution_count": 15, + "id": "87d44fdf", "metadata": { "collapsed": false, "editable": true @@ -3788,7 +4126,7 @@ }, { "cell_type": "markdown", - "id": "8253074c", + "id": "5f08f4dc", "metadata": { "editable": true }, @@ -3802,7 +4140,7 @@ }, { "cell_type": "markdown", - "id": "7a74656a", + "id": "a25b356f", "metadata": { "editable": true }, @@ -3814,8 +4152,8 @@ }, { "cell_type": "code", - "execution_count": 15, - "id": "804a8ee1", + "execution_count": 16, + "id": "928ae649", "metadata": { "collapsed": false, "editable": true @@ -3840,99 +4178,7 @@ }, { "cell_type": "markdown", - "id": "3871673a", - "metadata": { - "editable": true - }, - "source": [ - "We expand this model to the Franke function discussed above." - ] - }, - { - "cell_type": "markdown", - "id": "5e75414e", - "metadata": { - "editable": true - }, - "source": [ - "## Correlation Matrix with Pandas and the Franke function" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "b3efcdb9", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], - "source": [ - "# Common imports\n", - "import numpy as np\n", - "import pandas as pd\n", - "\n", - "\n", - "def FrankeFunction(x,y):\n", - "\tterm1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))\n", - "\tterm2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))\n", - "\tterm3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))\n", - "\tterm4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)\n", - "\treturn term1 + term2 + term3 + term4\n", - "\n", - "\n", - "def create_X(x, y, n ):\n", - "\tif len(x.shape) > 1:\n", - "\t\tx = np.ravel(x)\n", - "\t\ty = np.ravel(y)\n", - "\n", - "\tN = len(x)\n", - "\tl = int((n+1)*(n+2)/2)\t\t# Number of elements in beta\n", - "\tX = np.ones((N,l))\n", - "\n", - "\tfor i in range(1,n+1):\n", - "\t\tq = int((i)*(i+1)/2)\n", - "\t\tfor k in range(i+1):\n", - "\t\t\tX[:,q+k] = (x**(i-k))*(y**k)\n", - "\n", - "\treturn X\n", - "\n", - "\n", - "# Making meshgrid of datapoints and compute Franke's function\n", - "n = 4\n", - "N = 100\n", - "x = np.sort(np.random.uniform(0, 1, N))\n", - "y = np.sort(np.random.uniform(0, 1, N))\n", - "z = FrankeFunction(x, y)\n", - "X = create_X(x, y, n=n) \n", - "\n", - "Xpd = pd.DataFrame(X)\n", - "# subtract the mean values and set up the covariance matrix\n", - "Xpd = Xpd - Xpd.mean()\n", - "covariance_matrix = Xpd.cov()\n", - "print(covariance_matrix)" - ] - }, - { - "cell_type": "markdown", - "id": "03523168", - "metadata": { - "editable": true - }, - "source": [ - "We note here that the covariance is zero for the first rows and\n", - "columns since all matrix elements in the design matrix were set to one\n", - "(we are fitting the function in terms of a polynomial of degree $n$).\n", - "\n", - "This means that the variance for these elements will be zero and will\n", - "cause problems when we set up the correlation matrix. We can simply\n", - "drop these elements and construct a correlation\n", - "matrix without these elements." - ] - }, - { - "cell_type": "markdown", - "id": "ab0fcb46", + "id": "cab70377", "metadata": { "editable": true }, @@ -3944,7 +4190,7 @@ }, { "cell_type": "markdown", - "id": "9e11454d", + "id": "22ac4a6d", "metadata": { "editable": true }, @@ -3956,7 +4202,7 @@ }, { "cell_type": "markdown", - "id": "35eaf4ef", + "id": "cd0697a9", "metadata": { "editable": true }, @@ -3966,7 +4212,7 @@ }, { "cell_type": "markdown", - "id": "a426a48c", + "id": "b5043405", "metadata": { "editable": true }, @@ -3983,7 +4229,7 @@ }, { "cell_type": "markdown", - "id": "f0ac2bfa", + "id": "6a4edab6", "metadata": { "editable": true }, @@ -3993,7 +4239,7 @@ }, { "cell_type": "markdown", - "id": "c6d8c9b4", + "id": "e1b67496", "metadata": { "editable": true }, @@ -4008,7 +4254,7 @@ }, { "cell_type": "markdown", - "id": "a87b1d66", + "id": "47a041cb", "metadata": { "editable": true }, @@ -4018,7 +4264,7 @@ }, { "cell_type": "markdown", - "id": "cb950b85", + "id": "575264b7", "metadata": { "editable": true }, @@ -4032,7 +4278,7 @@ }, { "cell_type": "markdown", - "id": "c79e96c6", + "id": "7b049c58", "metadata": { "editable": true }, @@ -4044,7 +4290,7 @@ }, { "cell_type": "markdown", - "id": "1fceefdc", + "id": "9780789f", "metadata": { "editable": true }, @@ -4056,7 +4302,7 @@ }, { "cell_type": "markdown", - "id": "a8db1029", + "id": "da396c4a", "metadata": { "editable": true }, @@ -4068,7 +4314,7 @@ }, { "cell_type": "markdown", - "id": "03d9d476", + "id": "87a74417", "metadata": { "editable": true }, @@ -4078,7 +4324,7 @@ }, { "cell_type": "markdown", - "id": "9ccc8461", + "id": "c5ca565a", "metadata": { "editable": true }, @@ -4090,7 +4336,7 @@ }, { "cell_type": "markdown", - "id": "c6183bd4", + "id": "10944fc5", "metadata": { "editable": true }, @@ -4100,7 +4346,7 @@ }, { "cell_type": "markdown", - "id": "5cc84134", + "id": "7b985955", "metadata": { "editable": true }, @@ -4117,7 +4363,7 @@ }, { "cell_type": "markdown", - "id": "788d9c68", + "id": "1b748051", "metadata": { "editable": true }, @@ -4127,7 +4373,7 @@ }, { "cell_type": "markdown", - "id": "bd0e5aee", + "id": "40c1c1eb", "metadata": { "editable": true }, @@ -4139,7 +4385,7 @@ }, { "cell_type": "markdown", - "id": "dc52b90c", + "id": "e44c9e87", "metadata": { "editable": true }, @@ -4149,7 +4395,7 @@ }, { "cell_type": "markdown", - "id": "b593db6e", + "id": "28abe46c", "metadata": { "editable": true }, @@ -4161,7 +4407,7 @@ }, { "cell_type": "markdown", - "id": "8dcd6cc8", + "id": "178bc776", "metadata": { "editable": true }, @@ -4175,7 +4421,7 @@ }, { "cell_type": "markdown", - "id": "29312718", + "id": "1f0ead95", "metadata": { "editable": true }, @@ -4187,7 +4433,7 @@ }, { "cell_type": "markdown", - "id": "9818fcb1", + "id": "fcce3bb5", "metadata": { "editable": true }, @@ -4209,7 +4455,7 @@ }, { "cell_type": "markdown", - "id": "56fbca85", + "id": "5f9a7281", "metadata": { "editable": true }, @@ -4221,7 +4467,7 @@ }, { "cell_type": "markdown", - "id": "d9d152fe", + "id": "d3ea695c", "metadata": { "editable": true }, @@ -4236,7 +4482,7 @@ }, { "cell_type": "markdown", - "id": "b82dd67d", + "id": "edfff9e0", "metadata": { "editable": true }, @@ -4248,7 +4494,7 @@ }, { "cell_type": "markdown", - "id": "7dab117e", + "id": "c3c83244", "metadata": { "editable": true }, @@ -4260,7 +4506,7 @@ }, { "cell_type": "markdown", - "id": "3649c008", + "id": "00df5ee5", "metadata": { "editable": true }, @@ -4270,7 +4516,7 @@ }, { "cell_type": "markdown", - "id": "665e42ab", + "id": "4661cbd7", "metadata": { "editable": true }, @@ -4282,7 +4528,7 @@ }, { "cell_type": "markdown", - "id": "254d4fd4", + "id": "07b95265", "metadata": { "editable": true }, @@ -4292,7 +4538,7 @@ }, { "cell_type": "markdown", - "id": "4b84bb18", + "id": "8c9aeba4", "metadata": { "editable": true }, @@ -4304,7 +4550,7 @@ }, { "cell_type": "markdown", - "id": "d84538dd", + "id": "03690a72", "metadata": { "editable": true }, @@ -4314,7 +4560,7 @@ }, { "cell_type": "markdown", - "id": "f9a8f033", + "id": "478e2a9d", "metadata": { "editable": true }, @@ -4326,7 +4572,7 @@ }, { "cell_type": "markdown", - "id": "0c900c57", + "id": "e997196a", "metadata": { "editable": true }, @@ -4343,12 +4589,12 @@ }, { "cell_type": "markdown", - "id": "ef8aafcf", + "id": "770e587e", "metadata": { "editable": true }, "source": [ - "## Ridge and LASSO Regression\n", + "## Back to Ridge and LASSO Regression\n", "\n", "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", "our optimization problem is" @@ -4356,7 +4602,7 @@ }, { "cell_type": "markdown", - "id": "efc29850", + "id": "5cb701c2", "metadata": { "editable": true }, @@ -4368,7 +4614,7 @@ }, { "cell_type": "markdown", - "id": "586af0fa", + "id": "dee9d948", "metadata": { "editable": true }, @@ -4378,7 +4624,7 @@ }, { "cell_type": "markdown", - "id": "116d6e8b", + "id": "121cd014", "metadata": { "editable": true }, @@ -4391,7 +4637,7 @@ }, { "cell_type": "markdown", - "id": "90567753", + "id": "2ee745b4", "metadata": { "editable": true }, @@ -4401,7 +4647,7 @@ }, { "cell_type": "markdown", - "id": "aa3053f1", + "id": "94a15fb9", "metadata": { "editable": true }, @@ -4413,7 +4659,7 @@ }, { "cell_type": "markdown", - "id": "b2382074", + "id": "6341fe06", "metadata": { "editable": true }, @@ -4426,7 +4672,7 @@ }, { "cell_type": "markdown", - "id": "716a3c2a", + "id": "5bef6925", "metadata": { "editable": true }, @@ -4439,7 +4685,7 @@ }, { "cell_type": "markdown", - "id": "dfd4ebf1", + "id": "4539e82b", "metadata": { "editable": true }, @@ -4451,7 +4697,7 @@ }, { "cell_type": "markdown", - "id": "d782ab23", + "id": "1f496d16", "metadata": { "editable": true }, @@ -4463,7 +4709,7 @@ }, { "cell_type": "markdown", - "id": "f16d590d", + "id": "ee78adfa", "metadata": { "editable": true }, @@ -4473,7 +4719,7 @@ }, { "cell_type": "markdown", - "id": "a373fb74", + "id": "47a0c860", "metadata": { "editable": true }, @@ -4486,7 +4732,7 @@ }, { "cell_type": "markdown", - "id": "c4c2a6db", + "id": "6b452b5d", "metadata": { "editable": true }, @@ -4498,7 +4744,7 @@ }, { "cell_type": "markdown", - "id": "f14edde4", + "id": "a3331a79", "metadata": { "editable": true }, @@ -4510,134 +4756,12 @@ }, { "cell_type": "markdown", - "id": "899bc3ae", + "id": "9befd0e3", "metadata": { "editable": true }, "source": [ - "## Deriving the Ridge Regression Equations\n", - "\n", - "Using the matrix-vector expression for Ridge regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have" - ] - }, - { - "cell_type": "markdown", - "id": "0968804d", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\boldsymbol{\\theta}^T\\boldsymbol{\\theta},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "87643992", - "metadata": { - "editable": true - }, - "source": [ - "and \n", - "taking the derivatives with respect to $\\boldsymbol{\\theta}$ we obtain then\n", - "a slightly modified matrix inversion problem which for finite values\n", - "of $\\lambda$ does not suffer from singularity problems. We obtain\n", - "the optimal parameters" - ] - }, - { - "cell_type": "markdown", - "id": "900e9e55", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "6d3a5c46", - "metadata": { - "editable": true - }, - "source": [ - "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" - ] - }, - { - "cell_type": "markdown", - "id": "db3c68ed", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\sum_{i=0}^{p-1} \\theta_i^2 \\leq t,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "ab6d0410", - "metadata": { - "editable": true - }, - "source": [ - "with $t$ a finite positive number. \n", - "\n", - "If we keep the $1/n$ factor, the equation for the optimal $\\theta$ changes to" - ] - }, - { - "cell_type": "markdown", - "id": "ae454f20", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "2a508398", - "metadata": { - "editable": true - }, - "source": [ - "In many textbooks the $1/n$ term is often omitted. Note that a library like **Scikit-Learn** does not include the $1/n$ factor in the setup of the cost function.\n", - "\n", - "When we compare this with the ordinary least squares result we have" - ] - }, - { - "cell_type": "markdown", - "id": "6df6cf96", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "4ab38d70", - "metadata": { - "editable": true - }, - "source": [ - "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", - "\n", - "We see that Ridge regression is nothing but the standard OLS with a\n", + "Ridge regression, as discussed above, is nothing but the standard OLS with a\n", "modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n", "particular for our discussion of the bias-variance tradeoff are rather\n", "interesting. We will see that for specific values of $\\lambda$, we may\n", @@ -4649,7 +4773,7 @@ }, { "cell_type": "markdown", - "id": "cfa1a762", + "id": "9d8f2ba3", "metadata": { "editable": true }, @@ -4661,7 +4785,7 @@ }, { "cell_type": "markdown", - "id": "7394c1c2", + "id": "da712e9d", "metadata": { "editable": true }, @@ -4671,7 +4795,7 @@ }, { "cell_type": "markdown", - "id": "c67e8228", + "id": "d91cd5f7", "metadata": { "editable": true }, @@ -4683,7 +4807,7 @@ }, { "cell_type": "markdown", - "id": "95c698ae", + "id": "18940f72", "metadata": { "editable": true }, @@ -4693,7 +4817,7 @@ }, { "cell_type": "markdown", - "id": "372c4fae", + "id": "51fe3527", "metadata": { "editable": true }, @@ -4705,7 +4829,7 @@ }, { "cell_type": "markdown", - "id": "826e010b", + "id": "2e85d1eb", "metadata": { "editable": true }, @@ -4717,7 +4841,7 @@ }, { "cell_type": "markdown", - "id": "b3c2a178", + "id": "96c47485", "metadata": { "editable": true }, @@ -4733,7 +4857,7 @@ }, { "cell_type": "markdown", - "id": "2ec82017", + "id": "c1adcbbc", "metadata": { "editable": true }, @@ -4745,7 +4869,7 @@ }, { "cell_type": "markdown", - "id": "dacb08f8", + "id": "9693074c", "metadata": { "editable": true }, @@ -4757,7 +4881,7 @@ }, { "cell_type": "markdown", - "id": "af53eaa0", + "id": "46b05531", "metadata": { "editable": true }, @@ -4767,7 +4891,7 @@ }, { "cell_type": "markdown", - "id": "128f95fc", + "id": "2e4ce02b", "metadata": { "editable": true }, @@ -4779,7 +4903,7 @@ }, { "cell_type": "markdown", - "id": "bd5b4b5d", + "id": "dbd3f781", "metadata": { "editable": true }, @@ -4789,7 +4913,7 @@ }, { "cell_type": "markdown", - "id": "d316a1ae", + "id": "779e1ac5", "metadata": { "editable": true }, @@ -4801,7 +4925,7 @@ }, { "cell_type": "markdown", - "id": "1198b684", + "id": "ef53f2a5", "metadata": { "editable": true }, @@ -4818,7 +4942,7 @@ }, { "cell_type": "markdown", - "id": "bb24826b", + "id": "57adfaff", "metadata": { "editable": true }, @@ -4830,7 +4954,7 @@ }, { "cell_type": "markdown", - "id": "b9e921e5", + "id": "a8f94cba", "metadata": { "editable": true }, @@ -4842,7 +4966,7 @@ }, { "cell_type": "markdown", - "id": "06ad064c", + "id": "d60431ff", "metadata": { "editable": true }, @@ -4852,7 +4976,7 @@ }, { "cell_type": "markdown", - "id": "ab63b0d3", + "id": "f78d1dfd", "metadata": { "editable": true }, @@ -4864,7 +4988,7 @@ }, { "cell_type": "markdown", - "id": "49285a6a", + "id": "445800de", "metadata": { "editable": true }, @@ -4874,7 +4998,7 @@ }, { "cell_type": "markdown", - "id": "46be0f62", + "id": "3c54c49c", "metadata": { "editable": true }, @@ -4886,7 +5010,7 @@ }, { "cell_type": "markdown", - "id": "1ad7dd7c", + "id": "11366dc1", "metadata": { "editable": true }, @@ -4896,7 +5020,7 @@ }, { "cell_type": "markdown", - "id": "3d0b7ab5", + "id": "d68d3ac1", "metadata": { "editable": true }, @@ -4908,7 +5032,7 @@ }, { "cell_type": "markdown", - "id": "b360d642", + "id": "63fedbd6", "metadata": { "editable": true }, @@ -4918,7 +5042,7 @@ }, { "cell_type": "markdown", - "id": "8677a588", + "id": "470b20a8", "metadata": { "editable": true }, @@ -4930,12 +5054,12 @@ }, { "cell_type": "markdown", - "id": "d4f0eccb", + "id": "f3277c65", "metadata": { "editable": true }, "source": [ - "This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later." + "This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss how to code LASSO regression next week, when we have introduced gradient methods." ] } ], diff --git a/doc/LectureNotes/_build/html/searchindex.js b/doc/LectureNotes/_build/html/searchindex.js index 0b0bb1af9..27046ee33 100644 --- a/doc/LectureNotes/_build/html/searchindex.js +++ b/doc/LectureNotes/_build/html/searchindex.js @@ -1 +1 @@ -Search.setIndex({"alltitles": {"A Classification Tree": [[11, "a-classification-tree"]], "A Frequentist approach to data analysis": [[2, "a-frequentist-approach-to-data-analysis"], [23, "a-frequentist-approach-to-data-analysis"]], "A better approach": [[10, "a-better-approach"]], "A first summary": [[23, "a-first-summary"]], "A quick Reminder on Lagrangian Multipliers": [[10, "a-quick-reminder-on-lagrangian-multipliers"]], "A simple example": [[6, "a-simple-example"]], "A soft classifier": [[10, "a-soft-classifier"]], "A top-down perspective on Neural networks": [[3, "a-top-down-perspective-on-neural-networks"]], "ADAM optimizer": [[15, "adam-optimizer"]], "Activation functions": [[14, "activation-functions"]], "Adaptive boosting: AdaBoost, Basic Algorithm": [[12, "adaptive-boosting-adaboost-basic-algorithm"]], "Adding error analysis and training set up": [[23, "adding-error-analysis-and-training-set-up"], [24, "adding-error-analysis-and-training-set-up"]], "Adjust hyperparameters": [[3, "adjust-hyperparameters"]], "Algorithms for Setting up Decision Trees": [[11, "algorithms-for-setting-up-decision-trees"]], "An Overview of Ensemble Methods": [[12, "an-overview-of-ensemble-methods"]], "An extrapolation example": [[6, "an-extrapolation-example"]], "An optimization/minimization problem": [[23, "an-optimization-minimization-problem"]], "And finally \\boldsymbol{X}\\boldsymbol{X}^T": [[24, "and-finally-boldsymbol-x-boldsymbol-x-t"]], "And what about using neural networks?": [[23, "and-what-about-using-neural-networks"]], "Another example, the moons again": [[11, "another-example-the-moons-again"]], "Applied Data Analysis and Machine Learning": [[17, null]], "Autocorrelation function": [[20, "autocorrelation-function"]], "Automatic differentiation": [[15, "automatic-differentiation"]], "Back to the Cancer Data": [[13, "back-to-the-cancer-data"]], "Bagging": [[12, "bagging"]], "Bagging Examples": [[12, "bagging-examples"]], "Basic Matrix Features": [[18, "basic-matrix-features"]], "Basic ideas of the Principal Component Analysis (PCA)": [[13, null]], "Basic math of the SVD": [[7, "basic-math-of-the-svd"], [24, "basic-math-of-the-svd"]], "Basics": [[9, "basics"]], "Basics of a tree": [[11, "basics-of-a-tree"]], "Batch Normalization": [[3, "batch-normalization"]], "Bayes\u2019 Theorem and Ridge and Lasso Regression": [[7, "bayes-theorem-and-ridge-and-lasso-regression"]], "Boosting, a Bird\u2019s Eye View": [[12, "boosting-a-bird-s-eye-view"]], "Bootstrap": [[8, "bootstrap"]], "Bringing it together, first back propagation equation": [[14, "bringing-it-together-first-back-propagation-equation"]], "Building a Feed Forward Neural Network": [[3, null]], "Building a tree, regression": [[11, "building-a-tree-regression"]], "Building neural networks in Tensorflow and Keras": [[3, "building-neural-networks-in-tensorflow-and-keras"]], "CNNs in more detail, building convolutional neural networks in Tensorflow and Keras": [[5, "cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras"]], "Cancer Data again now with Decision Trees and other Methods": [[11, "cancer-data-again-now-with-decision-trees-and-other-methods"]], "Choose cost function and optimizer": [[3, "choose-cost-function-and-optimizer"]], "Classical PCA Theorem": [[13, "classical-pca-theorem"]], "Clustering and Unsupervised Learning": [[16, null]], "Code for SVD and Inversion of Matrices": [[7, "code-for-svd-and-inversion-of-matrices"]], "Codes and Approaches": [[16, "codes-and-approaches"]], "Codes for the SVD": [[7, "codes-for-the-svd"], [24, "codes-for-the-svd"]], "Coding Setup and Linear Regression": [[0, "coding-setup-and-linear-regression"]], "Collect and pre-process data": [[3, "collect-and-pre-process-data"]], "Communication channels": [[23, "communication-channels"]], "Compare Bagging on Trees with Random Forests": [[12, "compare-bagging-on-trees-with-random-forests"]], "Comparing with a numerical scheme": [[4, "comparing-with-a-numerical-scheme"]], "Computing the Gini index": [[11, "computing-the-gini-index"]], "Conjugate gradient method": [[15, "conjugate-gradient-method"]], "Convex functions": [[15, "convex-functions"]], "Convolution Examples: Polynomial multiplication": [[5, "convolution-examples-polynomial-multiplication"]], "Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)": [[5, "convolution-examples-principle-of-superposition-and-periodic-forces-fourier-transforms"]], "Convolutional Neural Network": [[14, "convolutional-neural-network"]], "Convolutional Neural Networks": [[5, null]], "Correlation Function and Design/Feature Matrix": [[24, "correlation-function-and-design-feature-matrix"]], "Correlation Matrix": [[13, "correlation-matrix"], [24, "correlation-matrix"]], "Correlation Matrix with Pandas": [[24, "correlation-matrix-with-pandas"]], "Correlation Matrix with Pandas and the Franke function": [[24, "correlation-matrix-with-pandas-and-the-franke-function"]], "Course Format": [[23, "course-format"]], "Course setting": [[19, null]], "Covariance Matrix Examples": [[24, "covariance-matrix-examples"]], "Covariance and Correlation Matrix": [[24, "covariance-and-correlation-matrix"]], "Cross-validation": [[8, "cross-validation"]], "Deadlines for projects (tentative)": [[23, "deadlines-for-projects-tentative"]], "Decision trees, overarching aims": [[11, null]], "Deep learning methods": [[23, "deep-learning-methods"]], "Define model and architecture": [[3, "define-model-and-architecture"]], "Defining the cost function": [[3, "defining-the-cost-function"]], "Deliverables": [[0, "deliverables"], [1, "deliverables"]], "Derivatives and the chain rule": [[14, "derivatives-and-the-chain-rule"]], "Derivatives, example 1": [[24, "derivatives-example-1"]], "Deriving OLS from a probability distribution": [[7, "deriving-ols-from-a-probability-distribution"]], "Deriving and Implementing Ordinary Least Squares": [[1, "deriving-and-implementing-ordinary-least-squares"]], "Deriving the Lasso Regression Equations": [[24, "deriving-the-lasso-regression-equations"]], "Deriving the Ridge Regression Equations": [[24, "deriving-the-ridge-regression-equations"]], "Deriving the back propagation code for a multilayer perceptron model": [[14, "deriving-the-back-propagation-code-for-a-multilayer-perceptron-model"]], "Developing a code for doing neural networks with back propagation": [[3, "developing-a-code-for-doing-neural-networks-with-back-propagation"]], "Diagonalize the sample covariance matrix to obtain the principal components": [[13, "diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components"]], "Different kernels and Mercer\u2019s theorem": [[10, "different-kernels-and-mercer-s-theorem"]], "Disadvantages": [[11, "disadvantages"]], "Discriminative Modeling": [[23, "discriminative-modeling"]], "Domains and probabilities": [[20, "domains-and-probabilities"]], "Dropout": [[3, "dropout"]], "Economy-size SVD": [[24, "economy-size-svd"]], "Elements of Probability Theory and Statistical Data Analysis": [[20, null]], "Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods": [[12, null]], "Entropy and the ID3 algorithm": [[11, "entropy-and-the-id3-algorithm"]], "Essential elements of ML": [[23, "essential-elements-of-ml"]], "Evaluate model performance on test data": [[3, "evaluate-model-performance-on-test-data"]], "Example 2": [[24, "example-2"]], "Example 3": [[24, "example-3"]], "Example 4": [[24, "example-4"]], "Example Matrix": [[24, "example-matrix"]], "Example of discriminative modeling, taken from Generative Deep Learning by David Foster": [[23, "example-of-discriminative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example of generative modeling, taken from Generative Deep Learning by David Foster": [[23, "example-of-generative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example of own Standard scaling": [[24, "example-of-own-standard-scaling"]], "Example relevant for the exercises": [[24, "example-relevant-for-the-exercises"]], "Example: Exponential decay": [[4, "example-exponential-decay"]], "Example: Population growth": [[4, "example-population-growth"]], "Example: The diffusion equation": [[4, "example-the-diffusion-equation"]], "Example: binary classification problem": [[3, "example-binary-classification-problem"]], "Examples": [[23, "examples"]], "Examples of likelihood functions used in logistic regression and neural networks": [[9, "examples-of-likelihood-functions-used-in-logistic-regression-and-neural-networks"]], "Exercise 1 - Finding the derivative of Matrix-Vector expressions": [[1, "exercise-1-finding-the-derivative-of-matrix-vector-expressions"]], "Exercise 1 - Github Setup": [[0, "exercise-1-github-setup"]], "Exercise 1: Setting up various Python environments": [[2, "exercise-1-setting-up-various-python-environments"]], "Exercise 2 - Deriving the expression for OLS": [[1, "exercise-2-deriving-the-expression-for-ols"]], "Exercise 2 - Setting up a Github repository": [[0, "exercise-2-setting-up-a-github-repository"]], "Exercise 2: making your own data and exploring scikit-learn": [[2, "exercise-2-making-your-own-data-and-exploring-scikit-learn"]], "Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression": [[1, "exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression"]], "Exercise 3 - Fitting an OLS model to data": [[0, "exercise-3-fitting-an-ols-model-to-data"]], "Exercise 3 - Setting up a Python virtual environment": [[0, "exercise-3-setting-up-a-python-virtual-environment"]], "Exercise 3: Normalizing our data": [[2, "exercise-3-normalizing-our-data"]], "Exercise 4 - Fitting a polynomial": [[1, "exercise-4-fitting-a-polynomial"]], "Exercise 4 - The train-test split": [[0, "exercise-4-the-train-test-split"]], "Exercise 4: Adding Ridge Regression": [[2, "exercise-4-adding-ridge-regression"]], "Exercise 5 - Comparing your code with sklearn": [[1, "exercise-5-comparing-your-code-with-sklearn"]], "Exercise 5: Analytical exercises": [[2, "exercise-5-analytical-exercises"]], "Exercise: Cross-validation as resampling techniques, adding more complexity": [[8, "exercise-cross-validation-as-resampling-techniques-adding-more-complexity"]], "Exercise: Analysis of real data": [[8, "exercise-analysis-of-real-data"]], "Exercise: Bias-variance trade-off and resampling techniques": [[8, "exercise-bias-variance-trade-off-and-resampling-techniques"]], "Exercise: Lasso Regression on the Franke function with resampling": [[8, "exercise-lasso-regression-on-the-franke-function-with-resampling"]], "Exercise: Ordinary Least Square (OLS) on the Franke function": [[8, "exercise-ordinary-least-square-ols-on-the-franke-function"]], "Exercise: Ridge Regression on the Franke function with resampling": [[8, "exercise-ridge-regression-on-the-franke-function-with-resampling"]], "Exercises": [[2, "exercises"]], "Exercises and Projects": [[8, "exercises-and-projects"]], "Exercises week 34": [[0, null]], "Exercises week 35": [[1, null]], "Expectation values": [[20, "expectation-values"]], "Extremely useful tools, strongly recommended": [[23, "extremely-useful-tools-strongly-recommended"]], "Feed-forward neural networks": [[14, "feed-forward-neural-networks"]], "Feed-forward pass": [[3, "feed-forward-pass"]], "Final back propagating equation": [[14, "final-back-propagating-equation"]], "Fine-tuning neural network hyperparameters": [[3, "fine-tuning-neural-network-hyperparameters"]], "Fitting an Equation of State for Dense Nuclear Matter": [[2, "fitting-an-equation-of-state-for-dense-nuclear-matter"]], "Fixing the singularity": [[24, "fixing-the-singularity"]], "Frequently used scaling functions": [[24, "frequently-used-scaling-functions"]], "From one to many layers, the universal approximation theorem": [[14, "from-one-to-many-layers-the-universal-approximation-theorem"]], "Functionality in Scikit-Learn": [[24, "functionality-in-scikit-learn"]], "Further Dimensionality Remarks": [[5, "further-dimensionality-remarks"]], "Further properties (important for our analyses later)": [[7, "further-properties-important-for-our-analyses-later"], [24, "further-properties-important-for-our-analyses-later"]], "Gaussian Elimination": [[18, "gaussian-elimination"]], "General Features": [[11, "general-features"]], "General linear models and linear algebra": [[23, "general-linear-models-and-linear-algebra"]], "Generalizing the fitting procedure as a linear algebra problem": [[23, "generalizing-the-fitting-procedure-as-a-linear-algebra-problem"], [23, "id1"]], "Generative Adversarial Networks": [[6, "generative-adversarial-networks"]], "Generative Models": [[6, "generative-models"]], "Generative Versus Discriminative Modeling": [[23, "generative-versus-discriminative-modeling"]], "Geometric Interpretation and link with Singular Value Decomposition": [[13, "geometric-interpretation-and-link-with-singular-value-decomposition"]], "Gradient Boosting, Classification Example": [[12, "gradient-boosting-classification-example"]], "Gradient Boosting, Examples of Regression": [[12, "gradient-boosting-examples-of-regression"]], "Gradient Clipping": [[3, "gradient-clipping"]], "Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent": [[12, "gradient-boosting-basics-with-steepest-descent-functional-gradient-descent"]], "Gradient descent": [[4, "gradient-descent"]], "Grading": [[21, "grading"], [21, "id2"], [23, "grading"]], "Housing data, the code": [[2, "housing-data-the-code"]], "How to take derivatives of Matrix-Vector expressions": [[1, "how-to-take-derivatives-of-matrix-vector-expressions"]], "Hyperplanes and all that": [[10, "hyperplanes-and-all-that"]], "Important Matrix and vector handling packages": [[18, "important-matrix-and-vector-handling-packages"]], "Improving performance": [[3, "improving-performance"]], "In summary": [[21, "in-summary"]], "Including Stochastic Gradient Descent with Autograd": [[15, "including-stochastic-gradient-descent-with-autograd"]], "Incremental PCA": [[13, "incremental-pca"]], "Installing R, C++, cython or Julia": [[23, "installing-r-c-cython-or-julia"]], "Installing R, C++, cython, Numba etc": [[23, "installing-r-c-cython-numba-etc"]], "Instructor information": [[21, "instructor-information"]], "Interpretations and optimizing our parameters": [[23, "interpretations-and-optimizing-our-parameters"], [23, "id2"], [23, "id3"], [24, "interpretations-and-optimizing-our-parameters"], [24, "id1"], [24, "id2"]], "Interpreting the Ridge results": [[24, "interpreting-the-ridge-results"]], "Introducing JAX": [[15, "introducing-jax"]], "Introducing the Covariance and Correlation functions": [[13, "introducing-the-covariance-and-correlation-functions"], [24, "introducing-the-covariance-and-correlation-functions"]], "Introduction": [[2, "introduction"], [8, "introduction"], [17, "introduction"], [18, "introduction"]], "Iterative Fitting, Classification and AdaBoost": [[12, "iterative-fitting-classification-and-adaboost"]], "Iterative Fitting, Regression and Squared-error Cost Function": [[12, "iterative-fitting-regression-and-squared-error-cost-function"]], "Kernel PCA": [[13, "kernel-pca"]], "Kernels and non-linearity": [[10, "kernels-and-non-linearity"]], "LU Decomposition, the inverse of a matrix": [[18, "lu-decomposition-the-inverse-of-a-matrix"]], "Layers": [[3, "layers"]], "Layers used to build CNNs": [[5, "layers-used-to-build-cnns"]], "Learning goals": [[0, "learning-goals"], [1, "learning-goals"]], "Learning outcomes": [[17, "learning-outcomes"], [23, "learning-outcomes"]], "Lectures and ComputerLab": [[23, "lectures-and-computerlab"]], "Limitations of supervised learning with deep networks": [[3, "limitations-of-supervised-learning-with-deep-networks"]], "Linear Algebra, Handling of Arrays and more Python Features": [[18, null]], "Linear Regression": [[2, null]], "Linear Regression Problems": [[24, "linear-regression-problems"]], "Linear Regression, basic elements": [[2, "linear-regression-basic-elements"]], "Linking Bayes\u2019 Theorem with Ridge and Lasso Regression": [[7, "linking-bayes-theorem-with-ridge-and-lasso-regression"]], "Linking the regression analysis with a statistical interpretation": [[7, "linking-the-regression-analysis-with-a-statistical-interpretation"]], "Linking with the SVD": [[7, "linking-with-the-svd"], [24, "linking-with-the-svd"]], "Links to relevant courses at the University of Oslo": [[22, "links-to-relevant-courses-at-the-university-of-oslo"]], "Logistic Regression": [[9, null], [9, "id1"]], "MNIST and GANs": [[6, "mnist-and-gans"]], "Machine Learning": [[23, "machine-learning"]], "Machine learning": [[17, "machine-learning"]], "Main textbooks": [[23, "main-textbooks"]], "Making a tree": [[11, "making-a-tree"]], "Making your own Bootstrap: Changing the Level of the Decision Tree": [[12, "making-your-own-bootstrap-changing-the-level-of-the-decision-tree"]], "Making your own test-train splitting": [[24, "making-your-own-test-train-splitting"]], "Mathematical Interpretation of Ordinary Least Squares": [[7, "mathematical-interpretation-of-ordinary-least-squares"], [24, "mathematical-interpretation-of-ordinary-least-squares"]], "Mathematical optimization of convex functions": [[10, "mathematical-optimization-of-convex-functions"]], "Mathematics of CNNs": [[5, "mathematics-of-cnns"]], "Mathematics of the SVD and implications": [[7, "mathematics-of-the-svd-and-implications"], [24, "mathematics-of-the-svd-and-implications"]], "Matrices in Python": [[23, "matrices-in-python"]], "Matrix multiplication": [[3, "matrix-multiplication"]], "Matrix-vector notation and activation": [[14, "matrix-vector-notation-and-activation"]], "Meet the covariance!": [[20, "meet-the-covariance"]], "Meet the Covariance Matrix": [[7, "meet-the-covariance-matrix"], [24, "meet-the-covariance-matrix"]], "Meet the Hessian Matrix": [[24, "meet-the-hessian-matrix"]], "Meet the Pandas": [[23, "meet-the-pandas"]], "Min-Max Scaling": [[24, "min-max-scaling"]], "Momentum based GD": [[15, "momentum-based-gd"]], "More complicated Example: The Ising model": [[8, "more-complicated-example-the-ising-model"]], "More interpretations": [[24, "more-interpretations"]], "More on Dimensionalities": [[5, "more-on-dimensionalities"]], "More on Rescaling data": [[8, "more-on-rescaling-data"]], "More preprocessing": [[24, "more-preprocessing"]], "Multilayer perceptrons": [[14, "multilayer-perceptrons"]], "Network requirements": [[4, "network-requirements"]], "Neural Networks vs CNNs": [[5, "neural-networks-vs-cnns"]], "Neural networks": [[14, null]], "Note about SVD Calculations": [[24, "note-about-svd-calculations"]], "Numerical experiments and the covariance, central limit theorem": [[20, "numerical-experiments-and-the-covariance-central-limit-theorem"]], "Numpy and arrays": [[18, "numpy-and-arrays"], [23, "numpy-and-arrays"]], "Numpy examples and Important Matrix and vector handling packages": [[23, "numpy-examples-and-important-matrix-and-vector-handling-packages"]], "Optimization, the central part of any Machine Learning algortithm": [[15, null]], "Optimizing our parameters": [[23, "optimizing-our-parameters"]], "Optimizing our parameters, more details": [[23, "optimizing-our-parameters-more-details"]], "Optimizing the cost function": [[3, "optimizing-the-cost-function"]], "Organizing our data": [[2, "organizing-our-data"], [23, "organizing-our-data"]], "Other Matrix and Vector Operations": [[18, "other-matrix-and-vector-operations"]], "Other Types of Recurrent Neural Networks": [[6, "other-types-of-recurrent-neural-networks"]], "Other courses on Data science and Machine Learning at UiO": [[23, "other-courses-on-data-science-and-machine-learning-at-uio"]], "Other courses on Data science and Machine Learning at UiO, contn": [[23, "other-courses-on-data-science-and-machine-learning-at-uio-contn"]], "Other popular texts": [[23, "other-popular-texts"]], "Other techniques": [[13, "other-techniques"]], "Other types of networks": [[14, "other-types-of-networks"]], "Other useful relations": [[24, "other-useful-relations"]], "Other ways of visualizing the trees": [[11, "other-ways-of-visualizing-the-trees"]], "Our model for the nuclear binding energies": [[23, "our-model-for-the-nuclear-binding-energies"]], "Overview of first week": [[23, "overview-of-first-week"]], "Own code for Ordinary Least Squares": [[23, "own-code-for-ordinary-least-squares"], [24, "own-code-for-ordinary-least-squares"]], "PCA and scikit-learn": [[13, "pca-and-scikit-learn"]], "Pandas AI": [[23, "pandas-ai"]], "Partial Differential Equations": [[4, "partial-differential-equations"]], "Plans for week 35": [[24, "plans-for-week-35"]], "Practical tips": [[15, "practical-tips"]], "Practicalities": [[21, "practicalities"], [21, "id1"]], "Predicting New Points With A Trained Recurrent Neural Network": [[6, "predicting-new-points-with-a-trained-recurrent-neural-network"]], "Preprocessing our data": [[24, "preprocessing-our-data"]], "Prerequisites": [[23, "prerequisites"]], "Prerequisites and background": [[17, "prerequisites-and-background"]], "Prerequisites: Collect and pre-process data": [[5, "prerequisites-collect-and-pre-process-data"]], "Probability Distribution Functions": [[20, "probability-distribution-functions"]], "Program for stochastic gradient": [[15, "program-for-stochastic-gradient"]], "Properties of PDFs": [[20, "properties-of-pdfs"]], "Pros and cons of trees, pros": [[11, "pros-and-cons-of-trees-pros"]], "Python installers": [[17, "python-installers"], [23, "python-installers"]], "RMS prop": [[15, "rms-prop"]], "Random Numbers": [[20, "random-numbers"]], "Random forests": [[12, "random-forests"]], "Randomized PCA": [[13, "randomized-pca"]], "Reading material": [[23, "reading-material"]], "Reading recommendations:": [[24, "reading-recommendations"]], "Reading suggestions week 34": [[23, "reading-suggestions-week-34"]], "Recurrent neural networks": [[14, "recurrent-neural-networks"]], "Recurrent neural networks: Overarching view": [[6, null]], "Reducing the number of degrees of freedom, overarching view": [[2, "reducing-the-number-of-degrees-of-freedom-overarching-view"], [24, "reducing-the-number-of-degrees-of-freedom-overarching-view"]], "Reformulating the problem": [[4, "reformulating-the-problem"]], "Regression Case": [[12, "regression-case"]], "Regression analysis, overarching aims": [[23, "regression-analysis-overarching-aims"]], "Regression analysis, overarching aims II": [[23, "regression-analysis-overarching-aims-ii"]], "Regularization": [[3, "regularization"]], "Reminder from last week": [[24, "reminder-from-last-week"]], "Reminder on Statistics": [[8, "reminder-on-statistics"]], "Replace or not": [[15, "replace-or-not"]], "Required Technologies": [[17, "required-technologies"]], "Resampling Methods": [[8, null]], "Resampling methods": [[8, "id1"]], "Residual Error": [[24, "residual-error"]], "Resources on differential equations and deep learning": [[4, "resources-on-differential-equations-and-deep-learning"]], "Revisiting our Linear Regression Solvers": [[15, "revisiting-our-linear-regression-solvers"]], "Rewriting the Covariance and/or Correlation Matrix": [[24, "rewriting-the-covariance-and-or-correlation-matrix"]], "Rewriting the fitting procedure as a linear algebra problem": [[23, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem"]], "Rewriting the fitting procedure as a linear algebra problem, more details": [[23, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem-more-details"]], "Ridge and LASSO Regression": [[24, "ridge-and-lasso-regression"]], "Ridge and Lasso Regression": [[7, null], [7, "id1"]], "Same code but now with momentum gradient descent": [[15, "same-code-but-now-with-momentum-gradient-descent"]], "Schedule first week": [[23, "schedule-first-week"]], "Schematic Regression Procedure": [[11, "schematic-regression-procedure"]], "Setting up the Back propagation algorithm": [[14, "setting-up-the-back-propagation-algorithm"]], "Setting up the Matrix to be inverted": [[24, "setting-up-the-matrix-to-be-inverted"]], "Setting up the network using Autograd; The full program": [[4, "setting-up-the-network-using-autograd-the-full-program"]], "Similar (second order function now) problem but now with AdaGrad": [[15, "similar-second-order-function-now-problem-but-now-with-adagrad"]], "Simple Python Code to read in Data and perform Classification": [[11, "simple-python-code-to-read-in-data-and-perform-classification"]], "Simple case": [[24, "simple-case"]], "Simple linear regression model using scikit-learn": [[2, "simple-linear-regression-model-using-scikit-learn"], [23, "simple-linear-regression-model-using-scikit-learn"]], "Software and needed installations": [[23, "software-and-needed-installations"]], "Solving Differential Equations with Deep Learning": [[4, null]], "Solving the one dimensional Poisson equation": [[4, "solving-the-one-dimensional-poisson-equation"]], "Solving the wave equation with Neural Networks": [[4, "solving-the-wave-equation-with-neural-networks"]], "Some famous Matrices": [[18, "some-famous-matrices"]], "Some simple problems": [[15, "some-simple-problems"]], "Some useful matrix and vector expressions": [[24, "some-useful-matrix-and-vector-expressions"]], "Splitting our Data in Training and Test data": [[2, "splitting-our-data-in-training-and-test-data"], [24, "splitting-our-data-in-training-and-test-data"]], "Standard steepest descent": [[15, "standard-steepest-descent"]], "Statistical analysis and optimization of data": [[17, "statistical-analysis-and-optimization-of-data"], [23, "statistical-analysis-and-optimization-of-data"]], "Steepest descent": [[15, "steepest-descent"]], "Stochastic Gradient Descent (SGD)": [[15, "stochastic-gradient-descent-sgd"]], "Stochastic variables and the main concepts, the discrete case": [[20, "stochastic-variables-and-the-main-concepts-the-discrete-case"]], "Support Vector Machines, overarching aims": [[10, null]], "Systematic reduction": [[5, "systematic-reduction"]], "Teachers": [[23, "teachers"]], "Teachers and Grading": [[21, null]], "Teaching Assistants Fall semester 2023": [[21, "teaching-assistants-fall-semester-2023"]], "Tentative deadllines for projects": [[21, "tentative-deadllines-for-projects"]], "Testing the Means Squared Error as function of Complexity": [[2, "testing-the-means-squared-error-as-function-of-complexity"], [24, "testing-the-means-squared-error-as-function-of-complexity"]], "Textbooks": [[22, null]], "The Algorithm before theorem": [[13, "the-algorithm-before-theorem"]], "The Boston housing data example": [[2, "the-boston-housing-data-example"]], "The Breast Cancer Data, now with Keras": [[3, "the-breast-cancer-data-now-with-keras"]], "The CART algorithm for Classification": [[11, "the-cart-algorithm-for-classification"]], "The CART algorithm for Regression": [[11, "the-cart-algorithm-for-regression"]], "The CIFAR01 data set": [[5, "the-cifar01-data-set"]], "The Jacobian": [[24, "the-jacobian"]], "The MNIST dataset again": [[5, "the-mnist-dataset-again"]], "The RELU function family": [[3, "the-relu-function-family"]], "The SVD, a Fantastic Algorithm": [[24, "the-svd-a-fantastic-algorithm"]], "The Softmax function": [[3, "the-softmax-function"]], "The \\chi^2 function": [[2, "the-chi-2-function"], [23, "the-chi-2-function"], [23, "id4"], [23, "id5"], [23, "id6"], [23, "id7"], [23, "id8"]], "The bias-variance tradeoff": [[8, "the-bias-variance-tradeoff"]], "The code for solving the ODE": [[4, "the-code-for-solving-the-ode"]], "The complete code with a simple data set": [[24, "the-complete-code-with-a-simple-data-set"]], "The cost/loss function": [[24, "the-cost-loss-function"]], "The course has two central parts": [[17, "the-course-has-two-central-parts"]], "The equations for ordinary least squares": [[24, "the-equations-for-ordinary-least-squares"]], "The logistic function": [[9, "the-logistic-function"]], "The mean squared error and its derivative": [[24, "the-mean-squared-error-and-its-derivative"]], "The moons example": [[10, "the-moons-example"]], "The multilayer perceptron (MLP)": [[14, "the-multilayer-perceptron-mlp"]], "The network with one input layer, specified number of hidden layers, and one output layer": [[4, "the-network-with-one-input-layer-specified-number-of-hidden-layers-and-one-output-layer"]], "The plethora of machine learning algorithms/methods": [[23, "the-plethora-of-machine-learning-algorithms-methods"]], "The singular value decomposition": [[7, "the-singular-value-decomposition"], [24, "the-singular-value-decomposition"]], "The two-dimensional case": [[10, "the-two-dimensional-case"]], "To our real data: nuclear binding energies. Brief reminder on masses and binding energies": [[23, "to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies"]], "Topics covered in this course: Statistical analysis and optimization of data": [[23, "topics-covered-in-this-course-statistical-analysis-and-optimization-of-data"]], "Towards the PCA theorem": [[13, "towards-the-pca-theorem"]], "Train and test datasets": [[3, "train-and-test-datasets"]], "Two-dimensional Objects": [[5, "two-dimensional-objects"]], "Type of problem": [[4, "type-of-problem"]], "Types of Machine Learning": [[23, "types-of-machine-learning"]], "Useful Python libraries": [[17, "useful-python-libraries"], [23, "useful-python-libraries"]], "Using Autograd": [[15, "using-autograd"]], "Using forward Euler to solve the ODE": [[4, "using-forward-euler-to-solve-the-ode"]], "Using gradient descent methods, limitations": [[15, "using-gradient-descent-methods-limitations"]], "Visualization": [[3, "visualization"], [3, "id1"]], "Visualizing the Tree, Classification": [[11, "visualizing-the-tree-classification"]], "Week 34: Introduction to the course, Logistics and Practicalities": [[23, null]], "Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression": [[24, null]], "What Is Generative Modeling?": [[23, "what-is-generative-modeling"]], "What does it mean?": [[24, "what-does-it-mean"]], "What is Machine Learning?": [[2, "what-is-machine-learning"]], "What is a good model?": [[2, "what-is-a-good-model"], [23, "what-is-a-good-model"]], "What is a good model? Can we define it?": [[23, "what-is-a-good-model-can-we-define-it"]], "Which activation function should I use?": [[3, "which-activation-function-should-i-use"]], "Why Linear Regression (aka Ordinary Least Squares and family)": [[23, "why-linear-regression-aka-ordinary-least-squares-and-family"]], "Wisconsin Cancer Data": [[9, "wisconsin-cancer-data"]], "Writing Our First Generative Adversarial Network": [[6, "writing-our-first-generative-adversarial-network"]], "Writing our own PCA code": [[13, "writing-our-own-pca-code"]], "XGBoost: Extreme Gradient Boosting": [[12, "xgboost-extreme-gradient-boosting"]], "scikit-learn implementation": [[3, "scikit-learn-implementation"]]}, "docnames": ["E1", "E2", "chapter1", "chapter10", "chapter11", "chapter12", "chapter13", "chapter2", "chapter3", "chapter4", "chapter5", "chapter6", "chapter7", "chapter8", "chapter9", "chapteroptimization", "clustering", "intro", "linalg", "schedule", "statistics", "teachers", "textbooks", "week34", "week35"], "envversion": {"sphinx": 62, "sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1}, "filenames": ["E1.ipynb", "E2.ipynb", "chapter1.ipynb", "chapter10.ipynb", "chapter11.ipynb", "chapter12.ipynb", "chapter13.ipynb", "chapter2.ipynb", "chapter3.ipynb", "chapter4.ipynb", "chapter5.ipynb", "chapter6.ipynb", "chapter7.ipynb", "chapter8.ipynb", "chapter9.ipynb", "chapteroptimization.ipynb", "clustering.ipynb", "intro.md", "linalg.ipynb", "schedule.md", "statistics.ipynb", "teachers.md", "textbooks.md", "week34.ipynb", "week35.ipynb"], "indexentries": {}, "objects": {}, "objnames": {}, "objtypes": {}, "terms": {"": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 15, 17, 18, 20, 21, 23, 24], "0": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23, 24], "00": [2, 3, 7, 13, 23, 24], "000": [3, 5], "00000000e": [], "001": [4, 10, 15], "004": 7, "00727646693": [2, 23], "0086649156": [2, 23], "01": [2, 3, 4, 7, 11, 13, 15, 22, 23, 24], "0110": 20, "01719003e": [], "02": [2, 6, 9, 14, 23], "02334824": [], "02857": 6, "02f": 8, "03077640549": 6, "03097597e": [], "031": 7, "04": 13, "0458": 11, "05": [6, 8], "062292565": 6, "062435": [], "06730814": [], "07": [], "0713": [2, 23], "07285": 5, "08": 20, "08078025e": [], "08336233266": 6, "0917": 11, "0n": [2, 23], "1": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23], "10": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 19, 20, 21, 23, 24], "100": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 18, 20, 21, 23, 24], "1000": [2, 3, 4, 6, 7, 10, 13, 15, 16, 17, 20, 23], "10000": [4, 7, 8, 12, 13, 15, 20], "100000": 10, "10001": 12, "1001": 20, "1002": 20, "1003": 20, "1005": 20, "1009": 20, "101": 1, "1011": 20, "1013": 20, "1013904243": 20, "1015": 20, "102": 1, "1023": 20, "1024": 5, "1026": 20, "1027": 20, "103": 3, "1030": 20, "1037": 20, "1038": 20, "1040": 20, "1047": 20, "107": 1, "108": [], "10th": 11, "10x": [2, 23], "11": [1, 2, 4, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 22, 23, 24], "110": [], "1100": 20, "1101": 20, "111": [3, 9, 14], "112": 1, "11340253": [], "11590451": [], "116": 1, "117": 1, "118": 1, "12": [2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 18, 20, 22, 23, 24], "120": 5, "121": [1, 10, 11, 12], "1215pm": [21, 23], "122": [10, 11, 12], "124": [2, 23], "125": 1, "127": [1, 6], "128": [5, 6, 15], "129": 1, "1298": 11, "12pm": [21, 23], "13": [2, 4, 11, 14, 18, 20, 23], "131": 1, "133": 9, "135": 1, "136": 1, "14": [2, 4, 6, 8, 10, 11, 12, 14, 18, 20, 22, 24], "141": 1, "143": 1, "1446729567": 6, "149": 1, "14g": 8, "15": [2, 4, 6, 8, 9, 10, 11, 14, 15, 20, 23], "150": [6, 10], "152": 1, "153760": [], "156": 1, "157": [], "158": [], "159": 1, "15g": 8, "15pm": 23, "16": [3, 4, 5, 6, 7, 10, 11, 12, 20, 23], "160": 1, "1603": 5, "161": 1, "162": 1, "16231451": 6, "163": 1, "16384": 5, "164": 1, "167": 1, "17": [3, 4, 10, 20], "172": 1, "173": 1, "176": 1, "178": 1, "179": 1, "1797": 3, "18": [4, 8, 9, 10, 11, 12, 20, 23], "1807": 6, "18392847": [], "19": [4, 20, 23], "1940": 2, "1943": 14, "1970": [18, 23], "1973": 11, "1979": 8, "1_1": 14, "1_2": 14, "1_3": 14, "1cm": [2, 10, 12, 20, 23], "1d": [3, 4, 5], "1e": [4, 6, 15, 16], "1e10": 16, "1e4": 8, "1f": 3, "1k": 18, "1n": [2, 23], "1x": [2, 23], "2": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22], "20": [1, 2, 3, 4, 8, 9, 10, 20, 21, 23], "200": [2, 4, 5, 6, 10, 11, 12], "2000": [2, 24], "2004": 15, "2006": 22, "20072279": [], "2008": 23, "2010": 3, "2011": 3, "2014": 6, "2015": 3, "2016": [2, 23], "2018": [2, 8, 24], "2021": [8, 16], "2022": 23, "2025": [23, 24], "21": [2, 3, 7, 9, 11, 14, 18, 23, 24], "2116753732": 6, "215pm": [21, 23], "2167072": [], "22": [2, 3, 7, 14, 15, 18, 23, 24], "221": 10, "225": 6, "22948497": [], "23": [3, 14, 18], "24": [2, 3, 18, 23], "25": [4, 5, 6, 7, 8, 10, 11, 13, 24], "250": [4, 6, 9, 11], "25000": 2, "250154": [], "253775": [], "255": 5, "256": 6, "26": [], "26303845": [], "264": [], "265": [], "265109911": 6, "266": [], "269": [], "27": [2, 3], "270": [], "27n_": 20, "28": [3, 5, 6], "2830637392": 6, "2861": 20, "2873": 11, "2882": 20, "2886": 20, "2890": [2, 23], "2892": 20, "29": 24, "2915": 20, "2931": 23, "29364655": [], "294399745619595": [], "296247": [], "2968": 23, "2980": 23, "298273": [], "298375": [], "2990": 23, "2_": 14, "2_1": 14, "2_2": 14, "2_3": 14, "2_i": 14, "2_m": [8, 20], "2_t": 15, "2_x": 20, "2b": 20, "2cm": 10, "2d": [3, 5, 13, 14, 17, 23], "2e": 8, "2f": [2, 9, 11, 12, 13, 14, 23], "2g": 4, "2g_i": 4, "2k": 5, "2m": 8, "2n": [2, 4, 5, 23, 24], "2nd": 11, "2p": 20, "2pt": 6, "2x": [2, 5, 10, 15, 23], "2x_ix_jy_iy_j": 10, "2x_j": 10, "2y_i": 12, "2y_j": 10, "3": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23], "30": [2, 3, 6, 8, 9, 12, 15, 21], "30000": [2, 23], "3072": 5, "31": [14, 18, 20], "315": 8, "3155": [2, 7, 8], "32": [5, 6, 8, 14, 15, 18, 20], "3200": 3, "3250": 3, "3297": [], "33": [14, 18, 21], "3303": [], "3310": [], "332331": [], "333": 9, "3331": [], "3337": [], "34": 18, "3436": [2, 23], "3437": [2, 23], "35": [2, 8, 23], "3581341341": 6, "359": 7, "36": [2, 7, 8, 20], "370782966": 6, "38": 20, "39": [2, 21, 23], "3d": [1, 4, 5, 6, 8, 15], "3f": [3, 5, 11], "3n": 18, "3x": [4, 10], "3x_i": 4, "3y": 10, "4": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "40": [3, 8, 21, 23], "400": 6, "4000": 23, "4050": [22, 23], "41": 18, "4155": [0, 4], "41589548": [], "42": [3, 6, 10, 11, 12, 18], "43": [2, 9, 18], "4310": 23, "436462435": 6, "44": [2, 18], "45": [21, 23], "46": [21, 23], "462": 9, "47": [21, 23], "479465113": 6, "47958494": [], "48": [], "48257387": [21, 23], "49": [7, 8, 13, 24], "49152": 5, "4940954": [2, 23], "4990": 20, "4992": 20, "4997": 20, "4c4c7f": [11, 12], "4d": 5, "4f": 8, "4pm": [21, 23], "4y": 10, "4y_i": 12, "5": [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "50": [3, 4, 5, 6, 8, 9, 10, 12, 15, 23, 24], "500": [3, 5, 6, 8, 11, 12, 15], "5018": 20, "506": 2, "507d50": [11, 12], "50j": 15, "50x10": 3, "51": 12, "510": 3, "512132": [], "5177783846": 6, "53": 11, "54": [8, 20], "5411205": [], "54894451": [], "55": 3, "56": 3, "56536": [2, 23], "569": 3, "57": [2, 10, 21, 23], "571": 7, "58": [12, 21, 23], "591317992": 6, "5cm": 20, "5f": 10, "5x": 10, "5y": 10, "6": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 21, 23, 24], "60": [3, 5], "60000": 6, "6019067271": 6, "606439": [], "625": 9, "63": [2, 3], "64": [3, 5, 6, 15, 18, 23], "64x50": 3, "65": [3, 10, 11], "6887363571": 6, "69": [1, 20], "69069n_": 20, "691": [], "6n_": 20, "7": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 22, 23, 24], "70": [3, 9], "70653767": 6, "71": 3, "724": 5, "73": [], "7304881": [], "75": [7, 8, 10, 13, 24], "76": [21, 23], "765": 9, "77": [21, 23], "7718": 11, "7782028952": 6, "77893972": [], "78": [], "7d7d58": [11, 12], "8": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 21, 23], "80": [2, 3, 7, 10, 24], "800": [6, 9], "81": 3, "815am": [21, 23], "85": 3, "8702784034": 6, "88": 23, "8f": 8, "8g": 8, "8n": 18, "8x8": 3, "9": [2, 3, 4, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 23, 24], "90": 3, "9040": 11, "91": [21, 23], "92": [21, 23], "93": 1, "931": [2, 23], "933": 7, "937": 20, "938": 20, "939": [2, 20, 23], "94": 20, "95": [3, 13], "954": 20, "955820c21e8b": 6, "96": 8, "960": 20, "961": 20, "962": 20, "9649652536": 6, "96611194e": [], "9780387310732": 22, "9780387848570": 22, "9781098134174": 23, "9781492032632": 22, "9781801819312": 23, "98": [1, 2, 3], "985": 20, "986": 20, "989": 20, "9898ff": [11, 12], "99": [1, 15], "991": 20, "992": 20, "993": 20, "996": 7, "999": [11, 20], "9x": 8, "9y": 8, "A": [0, 1, 4, 5, 7, 8, 9, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 24], "AND": 4, "And": [2, 5, 6, 7, 8, 11, 15, 17, 20], "As": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 15, 18, 20, 23, 24], "At": [2, 6, 8, 15, 23], "BE": [2, 23], "Be": [4, 17, 23], "Being": 15, "But": [1, 2, 3, 4, 5, 7, 8, 11, 12, 20, 24], "By": [2, 5, 7, 8, 14, 15, 18, 23, 24], "For": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23, 24], "IF": 8, "IN": 22, "If": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "In": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23, 24], "Ising": [7, 14, 24], "It": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "Its": [3, 4, 6, 13], "No": [8, 11, 23], "Not": [2, 3, 7, 8, 24], "OR": 20, "Of": 20, "On": [2, 5, 20, 21, 22, 23], "One": [2, 3, 5, 6, 7, 8, 9, 10, 13, 14, 15, 20, 24], "Or": [2, 3, 8, 23], "Such": [1, 2, 8, 14, 20], "That": [2, 7, 9, 12, 13, 14, 16, 20, 23], "The": [1, 6, 12, 15, 16, 18, 19, 20, 21, 22], "Then": [0, 1, 2, 3, 8, 10, 11, 12, 13, 14, 15, 16, 18, 23], "There": [0, 2, 5, 6, 7, 8, 10, 11, 13, 14, 16, 18, 20, 21, 23, 24], "These": [2, 5, 6, 7, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23, 24], "To": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 24], "With": [1, 2, 7, 8, 10, 11, 12, 13, 14, 16, 18, 20, 23, 24], "_": [1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 23, 24], "_0": [7, 10, 12, 13, 15, 24], "_1": [4, 7, 8, 10, 12, 13, 14, 15, 16, 18, 24], "_2": [4, 7, 10, 13, 14, 15, 18, 24], "_3": 18, "_4": 18, "_9": 15, "__class__": 12, "__doc__": 8, "__future__": [10, 11], "__init__": 3, "__main__": 4, "__name__": [4, 12], "_auto1": [4, 5, 6, 7, 8, 9, 14, 15, 18, 20, 24], "_auto10": [8, 14], "_auto11": 8, "_auto12": 8, "_auto2": [4, 5, 6, 7, 8, 14, 15, 18, 20], "_auto3": [5, 6, 7, 8, 14, 15, 18], "_auto4": [6, 8, 14, 15, 18], "_auto5": [6, 8, 14, 15, 18], "_auto6": [6, 8, 14, 18], "_auto7": [6, 8, 14, 18], "_auto8": [8, 14], "_auto9": [8, 14], "_build": [2, 17, 22, 23], "_c": 3, "_compon": 13, "_depth": 11, "_export": [0, 1], "_fraction": 11, "_i": [2, 3, 4, 7, 8, 9, 10, 13, 14, 15, 23, 24], "_j": [2, 3, 4, 5, 7, 8, 10, 15, 24], "_k": 15, "_l": 14, "_lambda": 8, "_leaf": 11, "_m": 12, "_multilayer_perceptron": [], "_n": [4, 7, 10, 13, 15, 24], "_node": 11, "_p": [7, 10, 24], "_ratio": 13, "_sampl": 11, "_split": [8, 11], "_t": 15, "_test": 8, "_varianc": 13, "_weight": 11, "a0": 5, "a0faa0": [11, 12], "a1": [2, 23], "a2": [2, 23], "a3": [2, 23], "a4": [2, 23], "a_": [1, 2, 3, 18, 23, 24], "a_0": [2, 23], "a_1a": [2, 23], "a_2a": [2, 23], "a_3": [2, 23], "a_3a": [2, 23], "a_4": [2, 23], "a_4a": [2, 23], "a_h": 3, "a_i": [2, 3, 4, 14, 23], "a_j": [3, 14], "a_k": [2, 3, 14], "aaron": 22, "ab": [2, 4, 7, 15, 16, 23, 24], "ab_channel": 17, "abandon": 3, "abid": 20, "abil": [2, 12], "abl": [1, 2, 3, 6, 7, 8, 9, 12, 14, 15, 24], "about": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 21], "abov": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 22, 23, 24], "abovement": [8, 23], "abscissa": 15, "absolut": [2, 4, 7, 8, 15, 23, 24], "absorb": 24, "abstract": 3, "acceler": 15, "accept": [2, 5, 8, 11, 24], "access": [2, 5, 13, 20, 23], "accid": [6, 8], "accompani": [2, 23, 24], "accomplish": [10, 11, 15], "accord": [2, 3, 4, 7, 8, 11, 14, 15, 16, 20, 23], "accordingli": 13, "account": [0, 1, 2, 5, 7, 15, 20, 23], "accumul": [14, 15, 20], "accur": [2, 5, 6, 8, 12, 15], "accuraci": [2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 23, 24], "accuracy_scor": [2, 3, 12, 23], "accuracy_score_numpi": 3, "achiev": [2, 3, 7, 8, 10, 14, 18, 23], "aco": 20, "acquaint": 17, "acquir": [3, 17, 23], "acr": 2, "across": [3, 5, 8, 11, 17, 23], "act": [3, 5, 18], "action": 20, "activ": [0, 2, 4, 5, 6, 11, 19, 21, 23], "actual": [0, 1, 2, 3, 6, 7, 8, 10, 13, 18, 20, 23, 24], "ad": [0, 1, 3, 5, 6, 7, 10, 15, 18], "ada_clf": 12, "adaboostclassifi": 12, "adadelta": 15, "adam": [3, 5, 6, 23], "adapt": [6, 8, 15, 22], "add": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 20, 21, 23, 24], "add_subplot": [3, 9, 14, 16], "addendum": 7, "addit": [0, 2, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18, 20, 21, 22, 23, 24], "addition": [14, 15], "address": [3, 11, 13, 15, 23], "adjac": [5, 14], "adjoint": [7, 24], "adjust": [2, 7, 14, 15], "admir": [2, 23], "advanc": [6, 8, 14, 22, 23], "advantag": [3, 5, 7, 8, 12, 15, 18], "adversari": 23, "afecionado": 23, "affect": [0, 5], "affin": [2, 5, 10, 13, 24], "afford": 5, "aficionado": 23, "aforement": 16, "african": 2, "after": [0, 1, 2, 3, 4, 6, 7, 8, 11, 13, 14, 15, 17, 18, 20, 23, 24], "afterward": [2, 23], "ag": [2, 9, 23, 24], "ag_0": 4, "again": [2, 3, 6, 7, 8, 9, 10, 12, 13, 14, 15, 20, 23, 24], "against": [3, 6, 9, 12], "agegroup": 9, "agegroupmean": 9, "aggreg": [11, 12], "agorithm": 12, "agre": [7, 8, 20], "agreement": 15, "ahead": 11, "ai": [2, 22], "aid": 13, "aim": [1, 2, 3, 6, 8, 9, 13, 16, 17, 18, 24], "ainv": 7, "airplan": 5, "aka": 7, "al": [1, 2, 4, 6, 22, 23, 24], "alarm": [7, 9], "aldo": 24, "algebra": [2, 5, 7, 15, 17, 24], "algorithm": [1, 2, 3, 4, 6, 7, 8, 9, 10, 15, 16, 17, 18, 20, 22], "align": [2, 4, 7, 8, 9, 10, 15, 20, 23, 24], "all": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24], "allevi": [3, 15], "alloc": [5, 18], "allow": [0, 2, 3, 4, 5, 7, 8, 10, 12, 15, 17, 18, 23, 24], "almost": [2, 3, 8, 10, 13, 15, 20], "alon": [4, 11], "along": [0, 4, 5, 6, 7, 8, 11, 12, 13, 17, 18, 23, 24], "alpha": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 15, 16, 20, 23, 24], "alpha_": 12, "alpha_0": 5, "alpha_1": 5, "alpha_2": 5, "alpha_i": [5, 15], "alpha_k": 15, "alpha_m": 12, "alpha_n": 5, "alpha_opt": 15, "alreadi": [0, 4, 5, 6, 7, 8, 12, 14, 17, 18, 20, 23, 24], "also": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "alter": 3, "altern": [0, 2, 3, 6, 7, 8, 10, 11, 13, 15, 18, 23, 24], "although": [1, 2, 3, 7, 8, 10, 12, 15, 23], "alwai": [1, 2, 5, 7, 8, 14, 15, 20, 23, 24], "am": 6, "ame2016": [2, 23], "american": 2, "among": [2, 5, 7, 11, 12, 14, 18, 23, 24], "amongst": 7, "amount": [2, 3, 5, 6, 8, 10, 12, 16, 17], "an": [1, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24], "an_": 20, "anaconda": [2, 3, 17, 23], "analogi": 15, "analys": 8, "analysi": [3, 5, 6, 9, 16, 18, 22], "analyt": [4, 5, 7, 8, 9, 14, 15, 17, 23, 24], "analyz": [1, 2, 3, 5, 6, 7, 8, 20, 24], "andrew": 3, "angl": [2, 5, 11, 24], "anharmon": 5, "ani": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 20, 23, 24], "anim": [6, 14], "ann": 14, "annot": [2, 3, 5, 9, 10, 23], "announc": 23, "anoth": [0, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 23, 24], "ansatz": [2, 23], "answer": [2, 3, 5, 7, 8, 18, 21, 23], "antialias": [4, 8], "anticip": 6, "anymor": [3, 10], "anyon": [0, 6, 10], "anyth": [0, 1, 3, 20], "anytim": [21, 23], "apach": 3, "apart": [13, 15], "api": [3, 17, 23], "appar": 4, "appear": [2, 3, 5, 15, 18, 20], "append": [3, 5, 6, 10, 11, 15, 23], "appli": [2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 20, 22, 23, 24], "applic": [1, 2, 3, 5, 6, 7, 8, 9, 11, 14, 15, 18, 20, 22, 23, 24], "apply_gradi": 6, "approach": [0, 1, 3, 4, 6, 7, 8, 11, 12, 13, 14, 15, 17, 20, 22, 24], "appropri": [4, 8, 11, 14, 15, 17, 20], "approv": 23, "approx": [2, 4, 5, 8, 12, 13, 15, 20, 23], "approxim": [2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 15, 20, 23, 24], "apt": [2, 17, 23], "aq": 20, "ar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24], "aragorn": 23, "arang": [3, 5, 6, 8, 9, 11, 12, 14, 15, 23], "arbitrari": [3, 6, 8, 10, 14, 15, 20], "arbitrarili": [2, 3, 13, 23], "arc": 8, "architectur": [5, 6, 14], "area": [2, 5, 8, 22, 23], "argmax": [3, 13], "argmin": [6, 12, 16], "argsort": 13, "argu": [3, 15], "argument": [2, 4, 5, 7, 13, 14, 15, 23, 24], "aris": [2, 8, 14, 15, 20, 23], "arithmet": [2, 15, 18, 23], "arm": 8, "armadillo": 18, "around": [2, 3, 6, 7, 8, 13, 20, 23], "arrai": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 15, 16, 17, 20, 24], "arrang": [5, 23], "arraybox": 15, "arriv": [2, 8, 11, 13, 18, 20, 23], "arrow": 14, "arrowprop": 10, "art": [2, 3, 17], "articl": [2, 5, 6, 8, 12, 23, 24], "artifici": [2, 4, 9, 14, 22, 23], "artificialneuron": 14, "arug": 15, "arxiv": [5, 6], "asarrai": [2, 8, 11], "asid": 24, "ask": [0, 7, 8, 13, 14], "aspect": [2, 8, 17, 23], "assembl": 5, "assembli": [2, 23], "assert": 6, "assess": [2, 8, 23], "assici": 6, "assign": [0, 2, 9, 10, 11, 14, 15, 16, 19, 21, 22, 23], "associ": [2, 8, 11, 14, 16, 20, 23], "assum": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "assumpt": [2, 5, 7, 8, 11, 13, 20, 23], "ast": [2, 7, 8, 23], "astyp": [6, 11, 12], "asymmetri": [2, 23], "asymptot": [6, 8], "atom": [2, 23], "attempt": [2, 6, 8, 9, 10, 12, 23], "attend": 23, "attent": [2, 18, 23], "attract": [2, 12, 23], "attribut": [2, 11, 23], "audi": [2, 23], "audio": [5, 6], "august": [23, 24], "aurelien": [2, 22, 23], "austfjel": 8, "auth": 0, "authent": 0, "author": [2, 3, 12, 20], "authour": 23, "auto": [11, 12, 20], "autocor": 20, "autocorrelation_tim": 20, "autocorrelform": 20, "autocovari": 20, "autoencod": [6, 17, 23], "autoencond": 17, "autograd": [17, 23], "autom": [2, 17, 22, 23], "automac": 18, "automag": 23, "automat": [1, 2, 3, 4, 5, 6, 13, 17, 18, 23], "automobil": 5, "autonom": 6, "avail": [2, 3, 6, 8, 12, 13, 17, 18, 19, 21, 22, 23], "averag": [2, 3, 5, 8, 11, 12, 15, 16, 20, 21, 23], "avoid": [2, 6, 7, 8, 11, 13, 15, 18, 24], "awai": [4, 5, 8, 24], "awar": [4, 12], "award": [21, 23], "ax": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 23], "axes3d": [4, 8, 15], "axes_grid1": 8, "axhlin": 10, "axi": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "axiom": 7, "axvlin": [6, 10], "axvspan": 6, "b": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 15, 16, 20, 21, 23, 24], "b1": 10, "b2": 10, "b3": 10, "b_": [2, 3, 18], "b_0": 2, "b_1": [2, 4, 14, 15], "b_2": [2, 15], "b_5": 15, "b_group": 11, "b_i": [2, 3, 4, 14, 23], "b_ia_": [2, 23], "b_ia_i": 2, "b_index": 11, "b_j": [3, 14], "b_k": [2, 3, 14, 15], "b_m": 14, "b_score": 11, "b_valu": 11, "babcock": 23, "bachelor": [19, 21], "back": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 18, 20, 23, 24], "backbon": 18, "backend": [3, 6], "background": [22, 23, 24], "backpropag": 3, "backtrack": 11, "backup": 18, "backward": [3, 4, 6, 14, 18], "bad": 8, "badli": 20, "bag": [11, 17, 23], "bag_clf": 12, "baggin": 23, "baggingboot": 12, "baggingclassifi": 12, "baggingtre": 12, "balanc": 8, "band": 18, "bandwidth": 18, "bar": [2, 8, 13, 23], "barber": 22, "bare": [6, 12], "base": [0, 1, 2, 3, 5, 6, 7, 9, 10, 11, 12, 16, 17, 20, 21, 22, 23, 24], "basi": [7, 9, 10, 12, 13, 14, 15, 18, 24], "basic": [0, 8, 10, 14, 15, 16, 17, 20, 23], "batch": [5, 6, 13, 14, 15], "batch_shap": 6, "batch_siz": [3, 5, 6], "batchnorm": 6, "bay": 9, "bayesian": [7, 17, 22, 23], "becaus": [2, 3, 4, 5, 6, 7, 8, 10, 11, 14, 15, 16, 23], "becom": [2, 3, 4, 7, 8, 9, 11, 14, 15, 20, 23, 24], "been": [2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 17, 18, 23], "befor": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 18, 20, 23, 24], "beforehand": [2, 20, 23], "begin": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 20, 21, 23, 24], "behav": [3, 8, 15], "behavior": [2, 3, 15, 23], "behaviour": 14, "behind": [2, 3, 8, 10, 15, 23], "being": [2, 3, 4, 5, 6, 7, 9, 10, 12, 13, 14, 15, 20, 23, 24], "believ": [11, 18], "belong": [9, 10, 11, 15, 16], "below": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "benchmark": 12, "benefici": [3, 15], "benefit": [2, 3, 6, 13, 15, 17, 23], "bengio": [3, 22, 23, 24], "benign": [3, 9], "besid": [6, 7], "bessel": [7, 24], "best": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 21, 23, 24], "beta": [1, 2, 3, 5, 12, 13, 15, 23, 24], "beta_": [5, 15], "beta_0": [3, 5, 15], "beta_1": [3, 5, 12, 15], "beta_1x_i": 15, "beta_2": [5, 15], "beta_3": 5, "beta_i": 5, "beta_j": 15, "beta_k": 15, "beta_linreg": 15, "beta_m": 12, "beta_mg_m": 12, "beta_n": 5, "better": [2, 3, 4, 5, 6, 8, 11, 12, 13, 14, 15, 23, 24], "between": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 20, 23, 24], "beyond": [2, 3, 7, 8, 10, 15, 23, 24], "bf": [15, 16, 18, 20], "bg": 23, "bgd": 15, "bia": [2, 3, 4, 5, 7, 10, 11, 12, 14, 15, 23, 24], "bias": [3, 4, 5, 7, 8, 11, 14], "big": [2, 3, 4, 7, 8, 16], "bigger": [3, 8], "bigr": 14, "bike": 11, "bilbo": 23, "billion": [5, 14, 17], "bin": [2, 9, 20], "binari": [2, 5, 7, 9, 11, 12, 14, 23], "binarycrossentropi": 6, "bind": [2, 24], "binomi": [17, 20, 23], "binsboot": 8, "bioinformat": 2, "biolog": [3, 14], "bios1100": [17, 23], "bird": [2, 5], "birth": 23, "bishop": [22, 23], "bit": [3, 6, 18, 20, 23], "bitwis": 20, "bivari": 4, "bk": [2, 15], "bla": [18, 23], "black": [10, 11, 16], "block": [8, 12, 17, 18, 20, 23], "blog": 23, "blogpost": 6, "blue": [2, 5], "bmatrix": [2, 3, 5, 7, 9, 10, 13, 15, 18, 23, 24], "bmi": 3, "bodi": [2, 3, 6, 14], "bold": 3, "boldfac": [1, 2, 7, 24], "boldsymbol": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 15, 16, 23], "boltzmann": [14, 17, 23], "book": [22, 23, 24], "book1": 22, "boolean": 6, "boost": [3, 11, 17, 23], "boostrap": 12, "bootstrap": [3, 15, 17, 23], "borrow": 23, "boston_dataset": 2, "bot": 10, "both": [0, 1, 2, 3, 6, 7, 8, 10, 11, 12, 15, 16, 17, 18, 20, 21, 23, 24], "bottl": 9, "bound": [2, 10, 14], "boundari": [4, 6, 10, 13, 14], "box": [6, 11], "boyd": [10, 15], "bracket": [6, 20], "brain": [3, 9, 14], "branch": [11, 23], "break": [2, 6, 8, 13, 16, 23], "breast": [7, 9, 13], "breviti": 15, "brew": [2, 17, 23], "brg": 10, "brief": 24, "briefli": [1, 2, 23], "bring": [2, 7, 8, 12], "britt": [21, 23], "broad": 2, "broadli": 23, "brought": [15, 17, 23], "brownle": 6, "browser": [0, 23], "brute": [5, 7, 13, 24], "buffer_s": 6, "bui": 6, "build": [1, 2, 6, 7, 8, 12, 18, 20, 23], "built": [2, 3, 5, 6, 8], "bunch": 13, "busi": 2, "byte": [18, 23], "c": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24], "c1": [10, 13], "c2": [10, 13], "c_": [2, 10, 11, 12, 15, 20], "c_0": 20, "c_1": 14, "c_2": 14, "c_3": 14, "c_4": 14, "c_i": [14, 15], "c_k": 20, "ca": [3, 23], "cach": 12, "cal": [2, 10, 12, 14, 15], "calcul": [1, 2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "call": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23, 24], "calor": [2, 24], "cambridg": [15, 22], "can": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24], "cancel": [2, 15, 23, 24], "cancer": [7, 12], "cancerpd": 9, "candid": [10, 11, 12], "cannot": [2, 3, 6, 7, 8, 9, 10, 11, 20, 24], "canopi": [2, 17, 23], "canva": [0, 1, 23], "cap": 7, "capabl": [2, 3, 10, 15, 17, 23], "capac": [4, 21], "capita": 2, "captur": [6, 13, 14, 23], "car": [5, 6], "card": [2, 9, 23], "cardin": 3, "care": [0, 13], "carefulli": 15, "carlo": [2, 8, 17, 20, 22, 23], "carri": [4, 8, 9], "cart": 12, "case": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15, 16, 17, 18, 23], "casella": 22, "cast": 3, "cat": [5, 6], "catch": 2, "categor": [2, 3, 5, 11, 13, 23], "categori": [2, 3, 5, 9, 12, 14, 16, 23], "categorical_crossentropi": [3, 5], "caus": [2, 7, 8, 20, 23, 24], "causal": 2, "causat": [2, 23], "cax": 3, "cb": [8, 23], "cbar": 3, "cc": [2, 3, 7, 15, 23, 24], "ccc": [7, 14], "cdf": 20, "cdot": [2, 4, 8, 14, 15, 16, 18, 20, 23], "celebr": 15, "cell": 6, "center": [2, 3, 8, 9, 10, 11, 13, 16, 20, 23], "central": [1, 2, 5, 7, 8, 10, 18, 23, 24], "centroid": [16, 20], "centroid_differ": 16, "centuri": 5, "certain": [2, 5, 8, 9, 11, 20, 23, 24], "cg": 15, "cha": 2, "chain": [2, 3, 15, 17, 20, 23], "challeng": 0, "chanc": [3, 7, 15, 20], "chang": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 15, 16, 18, 20, 23, 24], "channel": 5, "chapter": [1, 2, 8, 12, 13, 18, 22, 23, 24], "chapter3": 2, "charact": [2, 5, 7, 23, 24], "character": [10, 11, 12, 14, 20], "characterist": [2, 3, 5, 12, 15, 23], "charg": [2, 23], "charl": 2, "chase": 6, "chatgpt": 0, "chd": 9, "chddata": 9, "cheap": [7, 24], "cheaper": [3, 15], "check": [0, 1, 2, 3, 5, 6, 7, 13, 15, 18, 23], "checkmark": 5, "checkpoint": 6, "checkpoint_dir": 6, "checkpoint_prefix": 6, "chen": 12, "cheng": 24, "chiaramont": 4, "childcar": 1, "children": 1, "choic": [2, 3, 4, 5, 6, 8, 11, 14, 15, 16, 18, 23, 24], "choleski": [7, 18, 24], "choos": [0, 4, 5, 8, 11, 12, 13, 15, 16], "chosen": [1, 2, 3, 4, 8, 10, 11, 12, 15, 20, 23], "chosen_datapoint": 3, "christian": 22, "christoph": [22, 23], "cifar": 5, "cifar10": 5, "circ": [3, 14], "circl": [2, 10, 14, 24], "circuit": 5, "circumfer": 11, "circumv": [3, 7, 15, 24], "ckpt": 6, "clariti": 20, "class": [2, 3, 5, 6, 8, 9, 10, 11, 13, 14, 15, 20, 23], "class_nam": [5, 11], "class_val": 11, "class_valu": 11, "classic": [9, 11, 15], "classif": [2, 5, 7, 8, 9, 10, 13, 14, 17, 22, 23, 24], "classifi": [2, 3, 6, 9, 11, 12, 13, 23], "classificaton": 3, "classifii": 12, "clean": 3, "clear": [3, 7, 12, 14, 15], "clearli": [2, 5, 7, 8, 9, 10, 20], "clever": [3, 12], "clf": [2, 8, 10, 11, 12, 23, 24], "clf3": 2, "clf_lasso": 8, "clf_ridg": 8, "cli": 0, "clip": [5, 20], "clone": [0, 21], "close": [2, 3, 4, 6, 8, 10, 11, 13, 14, 15, 16, 20, 22, 23], "closer": [5, 7, 15, 24], "closest": [10, 13, 15, 16], "closur": [17, 23], "cloud": [17, 23], "cluster": [2, 3, 6, 8, 13, 17, 23], "cluster_label": 16, "cm": [3, 4, 5, 8, 10, 15], "cmap": [2, 3, 4, 5, 6, 8, 10, 11, 12, 23], "cmap_arg": 8, "cmd": [0, 11], "cn_": 20, "cnn": 14, "cnn_kera": 5, "cntk": [17, 23], "co": [2, 4, 5, 8, 11, 15, 23], "code": [5, 6, 8, 9, 10, 17, 18, 20, 22], "coef": [2, 23], "coef0": 10, "coef_": [1, 2, 7, 8, 10, 11, 15, 23], "coeff": 7, "coeffici": [2, 5, 7, 8, 9, 10, 11, 15, 18, 23], "coerc": [2, 8, 23], "coin": [12, 20], "coin_toss": 12, "col": [2, 13, 23, 24], "colab": [17, 23], "cold": 11, "colinear": 2, "collaps": 10, "collect": [4, 8, 12, 13, 17, 20, 22, 23], "collinear": [7, 24], "color": [2, 5, 6, 8, 10, 11, 12, 20], "color_channel": 5, "color_cod": 8, "colorbar": [3, 8], "colsample_bytre": 12, "colsaobject": 12, "column": [1, 2, 3, 4, 7, 8, 9, 10, 11, 13, 14, 18, 23, 24], "columntransform": 11, "com": [0, 1, 6, 8, 17, 22, 23], "combin": [0, 3, 4, 7, 8, 9, 12, 20], "come": [0, 2, 3, 5, 6, 7, 14, 15, 16, 23, 24], "command": [0, 2, 3], "comment": [2, 6, 7, 8], "commerci": [2, 17, 23], "commit": 0, "commod": [2, 23], "common": [1, 2, 3, 5, 7, 8, 9, 11, 13, 15, 16, 20, 23, 24], "commonli": [2, 3, 6, 8, 9, 11, 15, 16, 24], "commun": [0, 2, 14], "commut": 5, "commutatitav": 5, "compact": [2, 3, 5, 7, 8, 9, 11, 13, 14, 15, 16, 23, 24], "compair": 2, "compar": [2, 5, 6, 7, 8, 13, 15, 18, 23, 24], "comparison": [4, 6, 15], "compat": 9, "compet": 2, "competit": 12, "compil": [2, 3, 5, 6, 15, 17, 18, 23], "complet": [0, 1, 2, 4, 5, 6, 11, 14, 23], "completenn": 14, "complex": [1, 3, 7, 10, 11, 13, 14, 15, 23], "complic": [2, 3, 11, 15, 23], "compoment": 24, "compon": [1, 2, 3, 5, 6, 7, 8, 9, 11, 16, 17, 23, 24], "components_": 13, "compos": [11, 14, 15, 16, 17, 23], "compphys": [1, 2, 8, 17, 19, 21, 22, 23], "compress": [2, 23, 24], "compris": 8, "compromis": [7, 24], "compulsori": [17, 23], "comput": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 18, 19, 20, 22, 23, 24], "computation": [2, 5, 8, 11, 15, 20, 23], "computationalscienceuio": 23, "concaten": [4, 6, 8, 16], "concav": [3, 15, 24], "concentr": [2, 12], "concept": [2, 4, 17, 23, 24], "conceptu": [14, 15], "concern": [2, 3, 6, 9, 23], "concic": 23, "conclud": [2, 7, 15], "conclus": 3, "cond": 4, "conda": [2, 3, 17, 23], "condis": 24, "condit": [2, 4, 6, 7, 8, 10, 11, 13, 15, 20, 23, 24], "conduct": 17, "condwav": 4, "confid": [2, 7, 8, 9, 10, 23, 24], "configur": 5, "confirm": [7, 14], "confus": [7, 8, 9, 12, 18], "confusion_matrix": 11, "congruenti": 20, "conjug": [6, 10], "conjugaci": 15, "conjunct": 5, "connect": [2, 3, 5, 6, 11, 13, 14, 15, 18, 23, 24], "consequ": [7, 8, 10, 12, 14, 15, 24], "conserv": [7, 16, 24], "consid": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 23, 24], "consider": [2, 3, 7, 15, 23, 24], "consist": [2, 3, 4, 5, 6, 8, 14, 15, 20, 24], "constant": [1, 2, 4, 6, 7, 8, 10, 14, 15, 20, 23, 24], "constitu": [2, 23], "constitut": [4, 8], "constrain": [3, 5, 7, 9, 13], "constraint": [7, 8, 10, 15, 24], "construct": [2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 18, 20, 23, 24], "contact": [2, 23], "contain": [0, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 22, 23, 24], "contemporari": 23, "content": [0, 3, 17, 18, 23], "context": [8, 12, 15], "contigu": 18, "continu": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 23, 24], "contour": [11, 12, 15], "contourf": [10, 11, 12], "contrast": [3, 6, 11, 12, 14, 23], "contribut": [2, 5, 7, 15, 20, 23, 24], "contributor": 2, "control": [0, 2, 3, 5, 11, 15, 17, 23], "conv": [5, 6], "conv2d": [5, 6], "conv2dtranspos": 6, "convei": 23, "conveni": [7, 8, 14, 15, 18, 23], "convent": [14, 24], "converg": [3, 4, 6, 7, 10, 15, 16, 24], "convergencewarn": [], "convert": [2, 3, 6, 7, 11, 13, 15, 18, 23, 24], "converttomatrix": 6, "convex": [6, 7, 9, 24], "convinc": 15, "convolut": [3, 6, 17, 23], "cool": [6, 11], "coolwarm": 8, "coordin": [7, 14, 16, 24], "coorel": 2, "copi": [0, 2, 3, 16, 24], "core": 12, "corel": 23, "coronari": 9, "corr": [2, 7, 9, 13, 24], "correalt": [13, 17], "correct": [0, 2, 3, 4, 5, 6, 7, 9, 15, 18, 20, 23, 24], "correctli": [3, 4, 8, 9, 12], "correl": [2, 3, 5, 7, 8, 9, 12, 14, 15, 17, 20, 23], "correlation_matrix": [2, 7, 9, 13, 24], "correspond": [2, 5, 7, 8, 10, 11, 13, 14, 17, 18, 20, 23, 24], "cortex": 14, "cosin": [5, 8], "cost": [1, 2, 4, 5, 7, 8, 9, 10, 11, 14, 15, 23], "cost_deep_grad": 4, "cost_funct": 4, "cost_function_deep": 4, "cost_function_deep_grad": 4, "cost_function_grad": 4, "cost_grad": 4, "cost_sum": 4, "costol": 15, "could": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "coulomb": [2, 23], "count": [0, 2, 11, 19, 20, 21, 23], "counterpart": 23, "countor": 15, "coupl": [6, 7, 8], "cours": [0, 1, 2, 3, 5, 7, 13, 21, 24], "coursework": 0, "courvil": [22, 23, 24], "cov": [7, 8, 13, 18, 20, 23, 24], "cov_xi": [7, 13, 24], "cov_xx": [7, 13, 24], "cov_yi": [7, 13, 24], "covari": [2, 9, 17, 18, 23], "covariance_matrix": [7, 13, 16, 24], "cover": [2, 7, 17, 21, 22, 24], "covert": [2, 23], "covxi": 20, "covxx": 20, "covxz": 20, "covyi": 20, "covyz": 20, "covzz": 20, "cpu": 3, "craft": 5, "creat": [0, 3, 5, 6, 7, 11, 12, 13, 14, 17, 23], "create_biases_and_weight": 3, "create_convolutional_neural_network_kera": 5, "create_neural_network_kera": 3, "create_x": [7, 13, 24], "credit": [2, 9, 21, 23], "crim": 2, "crime": 2, "criteria": [2, 6, 11, 12, 16, 20, 23], "criterion": [11, 12, 15], "critic": 8, "cross": [0, 2, 3, 5, 9, 11, 12, 15, 17, 20, 23, 24], "cross_entropi": 6, "cross_val_scor": 8, "cross_valid": [9, 12], "crossvalid": 8, "crucial": [3, 20], "cs231": 5, "csr_matrix": [18, 23], "csv": [2, 6, 8, 9, 11], "ctnk": 3, "cubic": 2, "cumbersom": 7, "cumsum": [12, 13, 23], "cumul": [9, 12, 20], "cumulative_heads_ratio": 12, "cup": 7, "current": [0, 1, 3, 4, 5, 6, 15, 16, 22], "curs": [2, 24], "curv": [8, 9, 12, 14], "curvatur": 15, "custom": [8, 16], "custom_cmap": [11, 12], "custom_cmap2": [11, 12], "cutpoint": 11, "cv": [8, 9, 12], "cvxbook": 15, "cvxopt": [7, 10, 24], "cycl": [3, 14], "d": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 23, 24], "d2_g_t": 4, "d_f": 15, "d_g_t": 4, "d_net_out": 4, "da": 5, "dagger": [7, 18, 24], "dai": [3, 11, 17], "damp": 5, "darget": 11, "darkr": 20, "dat": [2, 23], "dat_id": [2, 8, 9, 11, 23], "data": [1, 4, 6, 7, 10, 12, 14, 15, 16, 18, 22], "data1": 16, "data2": 16, "data3": 16, "data4": 16, "data_id": [2, 8, 9, 11, 23], "data_indic": 3, "data_panda": 23, "data_path": [2, 8, 9, 11, 23], "databas": 3, "datafil": [2, 8, 9, 11, 23], "datafram": [2, 6, 7, 9, 11, 13, 23, 24], "datapoint": [1, 3, 7, 8, 9, 13, 15, 24], "datasci": [0, 1], "dataset": [1, 2, 6, 8, 9, 10, 11, 12, 13, 15, 16, 23], "datatyp": 6, "date": [0, 23, 24], "daughter": 12, "david": 22, "dbh": 3, "dbo": 3, "dcomposit": 18, "ddot": 4, "dead": 3, "deadlin": 0, "deal": [2, 3, 5, 7, 8, 10, 13, 15, 16, 18, 20, 23, 24], "dealt": 2, "debt": 9, "debug": [2, 7, 8], "decad": [2, 5], "decai": [2, 15, 20, 23], "decemb": [21, 23], "decent": 12, "decid": [2, 4, 5, 7, 8, 11], "decim": [2, 23], "decis": [2, 3, 10, 13, 17, 22, 23], "decision_funct": 10, "decision_tre": 11, "decisiontreeclassifi": [11, 12], "decisiontreeregressor": [2, 11, 12], "declar": [2, 6, 18, 23], "decompos": [7, 8, 18, 24], "decomposit": [2, 8, 14, 23], "decompost": [7, 24], "deconvolut": 5, "decorrel": [12, 15], "decreas": [3, 4, 6, 7, 8, 12, 13, 15], "deduc": [2, 23], "deep": [5, 9, 14, 15, 17, 22, 24], "deep_neural_network": 4, "deep_param": 4, "deep_tree_clf": [11, 12], "deep_tree_clf1": 11, "deep_tree_clf2": 11, "deepen": [7, 17, 23], "deeper": [2, 5, 6, 23], "deeplearningbook": [22, 23], "deer": 5, "def": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 20, 23, 24], "def_covari": 20, "default": [2, 3, 4, 6, 8, 9, 18, 23], "default_tim": 6, "defect": [7, 24], "defici": [7, 24], "defin": [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 24], "definit": [3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 24], "defint": 20, "degre": [0, 1, 5, 7, 8, 10, 11, 12, 13, 20, 23], "deisenroth": 24, "del": 3, "delet": [0, 8], "delimit": 6, "deliv": [0, 19, 23], "delta": [2, 4, 5, 8, 10, 14, 15, 16, 23], "delta_": [3, 18], "delta_0": 5, "delta_1": 5, "delta_2": 5, "delta_3": 5, "delta_4": 5, "delta_5": 5, "delta_h": [2, 3, 23], "delta_j": [5, 14], "delta_k": 14, "delta_l": [3, 5], "delta_momentum": 15, "delta_n": [2, 5, 23], "delug": 17, "delv": 2, "demand": 15, "demonstr": [2, 5, 7, 8, 9, 13, 14, 17, 23, 24], "den": 6, "denomin": [3, 7], "denot": [3, 4, 8, 9, 15, 20], "dens": [3, 5, 6], "densiti": [2, 4, 8, 20], "depart": [21, 23, 24], "depend": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 20, 23, 24], "depict": 20, "deploy": [2, 17, 23], "depth": [2, 5, 11, 12, 18], "deriv": [2, 3, 4, 8, 9, 10, 12, 13, 15, 17, 23], "derivati": 15, "derivative_fn": 15, "descend": [7, 11, 13, 24], "descent": [2, 3, 5, 9, 10, 14, 23, 24], "describ": [2, 4, 6, 7, 8, 10, 12, 13, 14, 15, 18, 23], "descript": [2, 10, 11, 23], "design": [2, 3, 5, 6, 7, 8, 9, 12, 13, 14, 15, 23], "designmatrix": [2, 23], "desir": [2, 4, 6, 7, 15, 16, 23, 24], "desktop": 0, "despit": [3, 14], "destroi": 18, "det": [7, 18, 24], "detail": [2, 8, 13, 15, 16, 18, 24], "detect": [5, 10, 14], "determin": [2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "determinist": [9, 15, 20], "dev": 3, "develop": [2, 5, 7, 10, 12, 13, 14, 17, 18, 23, 24], "deviat": [2, 3, 4, 6, 7, 8, 20, 23, 24], "devis": 14, "df": [6, 10, 13, 15, 23], "df1": 23, "di": 2, "diag": [7, 10, 24], "diagnost": [3, 12], "diagon": [2, 7, 9, 15, 18, 20, 23, 24], "diagonaliz": [7, 24], "diagram": 12, "diagsvd": 8, "dice": [8, 20], "dict": [8, 10], "dictionari": 2, "did": [1, 2, 3, 7, 8, 9, 12, 13, 16, 23], "die": 3, "diff": 4, "diff1": 4, "diff2": 4, "diff_ag": 4, "diffeent": 10, "differ": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23, 24], "differenti": [1, 2, 5, 17, 18, 23, 24], "difficult": [2, 3, 8, 12, 15, 20, 23], "difficulti": [2, 3, 15, 23], "diffonedim": 4, "digit": [2, 3, 5, 6, 8, 21, 23], "dilemma": 15, "dilut": 3, "dim": [6, 13, 16, 18], "dimens": [1, 2, 3, 4, 5, 6, 7, 10, 13, 16, 18, 23, 24], "dimension": [2, 6, 7, 8, 11, 13, 15, 16, 17, 18, 23, 24], "dimensionless": [2, 5, 23], "diment": 18, "dimnsion": 6, "diod": 5, "direct": [2, 3, 4, 6, 13, 14, 15, 16, 23, 24], "directli": [3, 6, 7, 8, 20, 24], "disadvantag": [2, 23], "disappear": [5, 8], "disc_loss": 6, "disc_tap": 6, "discard": [8, 13], "disciplin": [2, 5, 14], "disclaim": 20, "discord": 23, "discourag": [0, 15], "discov": [2, 23], "discover": 7, "discret": [3, 5, 7, 9, 15], "discrimin": [6, 9, 12, 13], "discriminator_loss": 6, "discriminator_loss_list": 6, "discriminator_model": 6, "discriminator_optim": 6, "discuss": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23, 24], "diseas": 9, "disguis": 8, "disord": [3, 9], "displai": [2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 20, 23, 24], "displaystyl": [2, 7, 23, 24], "disregard": [2, 23], "dissimilar": [13, 16], "dist": 16, "distanc": [2, 10, 11, 13, 16, 20], "distance_list": 11, "distinct": [5, 9, 10, 11, 12, 16], "distinctli": 10, "distinguish": [2, 6, 9, 10, 20, 23], "distplot": 2, "distribut": [2, 3, 6, 8, 9, 12, 13, 15, 16, 17, 18, 23, 24], "distrubut": [2, 17, 23], "dive": [2, 10, 18, 23], "diverg": [3, 15], "divid": [2, 3, 5, 7, 8, 9, 10, 11, 13, 14, 20, 23, 24], "divis": [8, 10, 11, 15, 18, 20], "dna": 9, "dnn": [2, 3, 4, 6, 14, 23], "dnn1": 6, "dnn2_gru2": 6, "dnn_kera": 3, "dnn_model": 3, "dnn_numpi": 3, "dnn_scikit": [2, 3, 23], "do": [0, 1, 2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 18, 23, 24], "doc": [0, 1, 2, 17, 19, 21, 22, 23], "document": [0, 6, 15], "doe": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 18, 20, 23], "doesn": [5, 11, 14, 23], "dog": [3, 5, 6], "domain": [7, 10, 15], "domin": [2, 23], "don": [0, 1, 2, 3, 5, 7, 8, 10, 13, 15, 17, 23, 24], "done": [1, 2, 4, 5, 6, 7, 8, 11, 12, 13, 15, 18, 23, 24], "dot": [2, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "doubl": [1, 5, 6, 18, 23], "doubli": 3, "down": [2, 5, 8, 11, 13, 14, 15], "download": [0, 2, 3, 5, 7, 8, 18, 22, 23], "downsampl": 5, "dozen": 3, "dq": 8, "drag": 15, "dramat": 13, "drastic": 6, "draw": [6, 8, 12, 15], "drawback": [2, 3, 5, 15, 24], "drawn": [3, 6, 8, 9, 13, 20, 23], "drive": [5, 6], "driven": 5, "drop": [2, 3, 7, 8, 13, 15, 20, 23, 24], "dropna": [2, 8, 23], "dropout": 6, "dt": [4, 5, 15, 20], "dtype": [2, 3, 5, 6, 16, 18, 23], "dub": [2, 23], "due": [3, 4, 7, 8, 10, 12, 14, 15, 21, 23, 24], "dummi": 2, "dure": [2, 3, 5, 6, 10, 11, 13, 17, 23], "dwell": 2, "dwh": 3, "dwo": 3, "dx": [4, 5, 10, 20], "dx_1": 20, "dx_1p": 8, "dx_2p": 8, "dx_mp": 8, "dx_n": 20, "dxp": 8, "dy": [3, 10, 20], "dynam": 6, "dz": 10, "e": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 21, 23, 24], "e_": [2, 4, 23], "each": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23, 24], "eapprox": [2, 23], "earli": [3, 15], "earlier": [2, 7, 9, 10, 11, 13, 14, 15, 23, 24], "earthexplor": 8, "eas": [8, 11, 16], "easi": [0, 2, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 23, 24], "easier": [0, 7, 8, 10, 11, 15, 20, 23, 24], "easiest": 15, "easili": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 23, 24], "eastern": [21, 23], "ebind": [2, 23], "eblock": 11, "econometr": 23, "economi": 7, "ecosystem": [17, 23], "ect": 19, "edg": 5, "edgecolor": 8, "editor": 0, "edu": 15, "educ": [2, 23], "eff": 20, "effect": [1, 3, 6, 12, 15, 20], "effic": 3, "effici": [2, 5, 12, 15, 17, 18, 20, 23], "efron": 8, "egrad": 15, "eig": [7, 13, 15, 18, 20, 23, 24], "eigen": 20, "eigenpair": [7, 13, 24], "eigenvalu": [2, 7, 10, 13, 15, 18, 23, 24], "eigenvector": [7, 13, 15, 24], "eight": [18, 23], "eigval": [18, 20, 23], "eigvalu": [13, 15], "eigvec": [18, 20, 23], "eigvector": [13, 15], "eir": [21, 23], "eispack": [18, 23], "either": [3, 7, 8, 9, 10, 11, 12, 13, 15, 20, 23, 24], "eivind": 21, "eivinsto": 21, "ekstr\u00f8m": 6, "elabor": 20, "elarn": 5, "electr": [2, 5, 14, 23], "electron": 23, "eleg": 13, "element": [3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 22, 24], "elementari": [12, 15, 18], "elementwis": [5, 15], "elementwise_grad": [4, 15], "elessar": 23, "elif": 16, "elim": 18, "elimin": [5, 10], "elin": [21, 23], "ellipsi": 1, "els": [1, 3, 5, 6, 9, 11, 14, 15, 18], "elu": 3, "elus": [2, 23], "email": [19, 21, 23], "embed": [2, 13, 24], "embodi": 8, "emit": 20, "emner": 22, "emphas": [2, 12, 17, 23], "emphasi": [2, 17, 22, 23], "empir": [3, 13, 20], "emploi": [2, 3, 7, 8, 13, 15, 20, 23, 24], "employ": 2, "empti": [0, 8, 12], "emul": 14, "en": [17, 22], "enabl": 13, "enbodi": 8, "encod": [2, 5, 7, 11, 13, 16, 23, 24], "encompass": [2, 20], "encount": [0, 2, 3, 7, 9, 15, 20, 23, 24], "encourag": 0, "end": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23, 24], "endpoint": [5, 8], "energi": [2, 6, 8, 24], "enforc": 14, "eng": 22, "engin": [2, 3, 5, 6, 17, 23], "enorm": 5, "enough": [2, 8, 15, 23], "ensembl": [3, 11, 23], "ensur": [2, 3, 4, 5, 7, 8, 13, 15, 20, 24], "entail": 23, "enter": [7, 8, 24], "enthought": [2, 17, 23], "entir": [3, 5, 9, 11, 17, 20, 23], "entiti": [11, 14, 18, 23], "entri": [2, 7, 10, 13, 14, 18, 23, 24], "entropi": [3, 5, 9, 12, 15, 23], "enumer": [2, 3, 4, 5, 6, 8, 10, 23], "env": 20, "environ": [4, 17, 23], "environemnt": 0, "eo": [2, 8], "eol": 2, "eosfit": 2, "epoch": [2, 3, 5, 6, 14, 15, 23], "epsilon": [2, 7, 8, 9, 15, 23, 24], "epsilon_": [2, 23], "epsilon_0": [2, 23], "epsilon_1": [2, 23], "epsilon_2": [2, 23], "epsilon_i": [2, 23, 24], "eq": [5, 15, 16, 18, 20], "eqnarrai": [5, 7, 8], "equal": [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 15, 16, 18, 20, 23, 24], "equat": [3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23], "equilibrium": [4, 14], "equiv": [5, 15, 18, 20], "equival": [2, 3, 7, 9, 10, 13, 15, 17, 18, 23, 24], "erf": 20, "eriador": 23, "err": [2, 12], "err_": 8, "err_sqr": 4, "errat": 15, "erron": 4, "error": [0, 1, 3, 4, 6, 7, 8, 9, 11, 13, 14, 15, 17, 18, 20], "error_estimate_corr_tim": 20, "error_hidden": 3, "error_output": 3, "escap": 15, "especi": [0, 3, 5, 11, 14, 15], "essenti": [0, 2, 7, 8, 11, 12, 14, 16, 20, 24], "establish": [1, 2, 8, 12, 13], "estim": [2, 3, 7, 8, 9, 12, 13, 15, 17, 20, 23, 24], "estimated_mse_fold": 8, "estimated_mse_kfold": 8, "estimated_mse_sklearn": 8, "et": [1, 2, 4, 6, 22, 23, 24], "eta": [2, 3, 5, 10, 14, 15, 23], "eta0": [10, 15], "eta_": 15, "eta_t": 15, "eta_v": [2, 3, 5, 23], "etc": [2, 3, 5, 7, 9, 10, 11, 13, 14, 15, 16, 17, 18, 20, 24], "ethic": 17, "euclidean": [2, 16, 24], "evalu": [0, 1, 2, 4, 5, 6, 7, 8, 11, 15, 20, 23, 24], "evalut": 15, "even": [2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "evenli": 6, "event": [7, 9, 12, 20], "eventu": [2, 7, 8, 13, 14, 15, 21, 24], "everi": [0, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 17, 20, 21, 23, 24], "everyth": [1, 6, 14], "everywher": [6, 15], "evolv": 2, "exact": [2, 7, 13, 14, 15, 18, 20, 23, 24], "exactli": [2, 5, 6, 8, 14, 17, 24], "exam": 23, "examin": 8, "exampl": [0, 1, 7, 13, 14, 15, 17, 18, 20, 22], "exce": [3, 14, 15], "excel": [2, 3, 6, 7, 12, 23, 24], "except": [5, 6, 8, 10, 11, 18], "excess": [2, 23], "excit": 2, "exclud": [3, 8, 14], "exclus": [2, 3, 5, 8, 20, 23], "execut": [0, 4, 7, 15, 24], "exemplifi": 15, "exercic": [21, 23], "exercis": [7, 17, 19, 21, 23], "exhaust": 8, "exhibit": [2, 7, 8, 10, 23, 24], "exist": [2, 3, 4, 5, 7, 8, 9, 10, 11, 15, 18, 23], "exit": [7, 18, 24], "exp": [1, 2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 20, 24], "exp_term": 3, "expand": [7, 9, 13, 15, 24], "expans": [2, 5, 7, 10, 12, 14, 15, 23, 24], "expect": [0, 2, 3, 7, 8, 9, 13, 14, 15, 17, 23, 24], "expectation_value_of_h_wrt_p": 20, "expens": [1, 8, 12, 15], "experi": [0, 2, 3, 8, 10, 15, 17, 23, 24], "experiment": [2, 6, 8, 11, 20, 23], "expert": [3, 11], "explain": [1, 2, 8, 11, 12, 13, 15, 23], "explained_variance_ratio_": 13, "explanatori": [2, 23], "explicit": [2, 5, 8, 15, 18, 23, 24], "explicitli": [2, 6], "explod": 3, "exploit": [2, 5, 14, 15, 23], "explor": [3, 6, 8, 10, 15, 17, 23], "expon": 3, "exponenti": [2, 3, 7, 8, 12, 15, 20, 23], "export": [0, 1, 11], "export_graphviz": 11, "export_text": 11, "exporttext": 11, "expos": 17, "express": [2, 4, 5, 7, 8, 9, 12, 14, 15, 18, 20, 23], "exptmean": 20, "exptvari": 20, "extend": [2, 4, 9, 13, 15, 17, 23], "extens": [0, 2, 14, 17, 23], "extent": [2, 3, 8, 22], "extern": [5, 8, 11], "extra": [0, 3, 5, 7, 21, 23, 24], "extract": [1, 2, 5, 7, 8, 9, 10, 13, 15, 18, 23, 24], "extrapol": [2, 23], "extrem": [0, 1, 2, 3, 6, 7, 8, 9, 10, 11, 15, 18, 24], "extremum": 15, "extrins": 13, "ey": [2, 7, 8, 15, 16, 18, 23], "f": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 15, 16, 18, 20, 21, 23, 24], "f1": 15, "f11": [2, 23], "f12": [2, 23], "f13": [2, 23], "f1_grad": 15, "f1d": 15, "f2": 15, "f2_grad_x1": 15, "f2_grad_x1_analyt": 15, "f2_grad_x2": 15, "f2_grad_x2_analyt": 15, "f3": 15, "f3_grad": 15, "f3_grad_analyt": 15, "f4": 15, "f4_grad": 15, "f4_grad_analyt": 15, "f5": 15, "f5_grad": 15, "f6": 15, "f6_for": 15, "f6_for_grad": 15, "f6_grad_analyt": 15, "f6_while": 15, "f6_while_grad": 15, "f7": 15, "f7_grad": 15, "f7_grad_analyt": 15, "f8": 15, "f8_grad": 15, "f9": [2, 15, 23], "f9_altern": 15, "f9_alternative_grad": 15, "f9_grad": 15, "f_": 12, "f_0": [5, 12], "f_1": [12, 15], "f_2": [14, 15], "f_3": 14, "f_d": 20, "f_grad": 15, "f_grad_analyt": 15, "f_i": [1, 2, 8, 14], "f_m": [5, 12], "f_n": 5, "f_vec": 4, "face": [15, 23], "facecolor": [8, 10, 20], "facil": [2, 17], "facilit": 14, "fact": [2, 3, 5, 7, 11, 13, 14, 15, 23, 24], "factor": [2, 3, 5, 7, 8, 11, 12, 13, 15, 18, 20, 23, 24], "factori": 15, "fade": 8, "fafab0": [11, 12], "fail": [2, 8, 15, 21, 23], "failur": 9, "fairli": [3, 4, 20], "faisal": [1, 24], "fake": 6, "fake_loss": 6, "fake_output": 6, "fall": [10, 11, 19], "fals": [1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 16, 18, 23, 24], "famili": [2, 9, 10, 20, 24], "familiar": [0, 2, 5, 7, 8, 10, 17, 18, 20, 23], "famou": [8, 14], "far": [1, 2, 5, 6, 7, 8, 10, 13, 14, 15, 16, 23, 24], "fashion": [2, 11, 12, 23], "fast": [3, 5, 8, 12, 14, 15, 17, 20, 23], "faster": [3, 13, 15], "fastest": [15, 18], "favor": 9, "favorit": 20, "fc": 5, "featur": [0, 2, 3, 5, 7, 8, 9, 10, 12, 13, 14, 15, 17, 20, 23], "feature_nam": [2, 3, 9, 11], "feautur": 11, "fed": 3, "feed": [2, 4, 5, 13, 17, 23], "feed_forward": 3, "feed_forward_out": 3, "feed_forward_train": 3, "feedback": [6, 23], "feeddorward": 6, "feedforward": [3, 6, 14], "feel": [0, 1, 2, 7, 8, 13, 15, 17, 21, 23], "feet": 2, "fetch": [0, 8], "few": [3, 5, 6, 7, 11, 20, 23], "fewer": [2, 11, 13, 23], "ffnn": [3, 14], "field": [2, 5, 8, 14, 17], "fifth": [2, 8, 23], "fig": [2, 3, 4, 5, 6, 8, 9, 14, 15, 16, 23], "fig_id": [2, 8, 9, 11, 23], "figaxi": 20, "figsiz": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 23], "figur": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 23, 24], "figure_id": [2, 8, 9, 11, 23], "figurefil": [2, 8, 9, 11, 23], "file": [0, 2, 6, 7, 8, 9, 11, 23], "file_prefix": 6, "filenam": 23, "fill": [7, 11, 24], "filter": [5, 6], "final": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 19, 20, 21, 23], "financ": 2, "find": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23, 24], "fine": [2, 16], "finish": 4, "finit": [5, 7, 8, 14, 15, 20, 24], "finnicki": 0, "first": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 22, 24], "firsteigvector": 13, "fit": [3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 20, 24], "fit_intercept": [1, 2, 7, 8, 24], "fit_mod": 11, "fit_theta": 8, "fit_transform": [0, 2, 8, 10, 11, 13], "fiti": [2, 23], "five": [2, 11, 23, 24], "fix": [2, 5, 6, 8, 12, 13, 14, 15, 23], "flag": 6, "flat": [14, 15], "flatten": [3, 5, 6, 7, 18], "flexibl": [3, 8, 10, 12, 14, 23], "flip": [21, 23], "float": [2, 5, 6, 7, 11, 13, 15, 16, 18, 23, 24], "float32": [6, 11], "float64": [6, 18, 23], "flop": [7, 18, 24], "flow": [3, 6, 14], "fluctuat": 7, "fly": 13, "fm": 2, "fmax": 5, "fmesh": 15, "fn": 9, "focu": [0, 2, 5, 6, 7, 8, 17, 22, 23], "focus": [3, 8, 9, 18], "fold": [8, 11], "folder": [0, 2, 6, 8, 23], "follow": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 23, 24], "font": [9, 20, 23], "fontdict": 20, "fontsiz": [3, 8, 10, 11, 12, 20], "fontweight": 3, "footprint": 5, "foral": [10, 24], "forc": [2, 7, 8, 12, 13, 24], "forcast": 6, "forecast": [6, 14], "forest": [2, 3, 11, 17, 23], "forget": 13, "form": [0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23, 24], "formal": [5, 6, 16, 20], "format": [2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 17, 20, 22], "format_data": 6, "formatstrformatt": [8, 15], "formul": [6, 8, 13, 16], "formula": [5, 15, 20], "forth": [6, 14], "fortran": [2, 17, 18, 23], "fortran2003": [17, 23], "fortran90": 20, "fortun": [2, 13, 24], "forward": [2, 5, 8, 17, 18, 23], "found": [3, 4, 6, 7, 8, 14, 15, 23, 24], "foundat": [17, 23], "four": [6, 7, 8, 10, 14, 18, 19, 21, 23], "fourier": [2, 23], "fourierdef1": 5, "fourierdef2": 5, "fourierseriessign": 5, "fourth": [14, 23, 24], "fp": 9, "frac": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "fraction": 11, "frame": 9, "framework": [3, 10, 12, 20], "frank": [7, 13], "frankefunct": [7, 8, 13, 24], "fredli": [21, 23], "free": [0, 1, 2, 8, 13, 15, 17, 18, 20, 21, 22, 23], "freecodecamp": 17, "freedom": 7, "freeli": 2, "freez": 0, "frequenc": [5, 8, 9, 20], "frequent": [2, 10, 11, 15], "frequentist": 17, "fresh": 12, "fridai": [0, 21, 23], "friedman": [8, 22, 23], "friendli": 6, "frodo": 23, "frog": 5, "from": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 15, 16, 17, 18, 20, 21, 22], "from_cod": 11, "from_logit": [5, 6], "from_tensor_slic": 6, "front": [2, 6, 7, 23, 24], "frustrat": 0, "fulfil": [4, 7, 14, 24], "full": [2, 3, 5, 7, 9, 11, 12, 15, 20, 23, 24], "full_matric": [7, 24], "fulli": [5, 8, 14, 20], "fun": [17, 23], "func": 4, "function": [0, 1, 4, 5, 6, 7, 11, 16, 17, 18], "functionali": 13, "fundament": [2, 8, 17, 23], "funtion": 4, "further": [4, 9, 11, 23], "furthermor": [2, 5, 7, 8, 9, 13, 14, 15, 17, 23, 24], "futur": [2, 6, 10, 11, 23], "fy": [0, 19, 21, 22, 23], "fys5419": [22, 23], "fys5429": [22, 23], "f\u00f8470": [21, 23], "g": [0, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 20, 23], "g0": 4, "g_": [4, 11, 12], "g_0": 4, "g_1": [4, 12], "g_2": [4, 12], "g_analyt": 4, "g_dnn_ag": 4, "g_euler": 4, "g_i": 4, "g_m": [5, 12], "g_n": 5, "g_re": 4, "g_t": 4, "g_t_d2t": 4, "g_t_d2x": 4, "g_t_dt": 4, "g_t_hessian": 4, "g_t_hessian_func": 4, "g_t_jacobian": 4, "g_t_jacobian_func": 4, "g_trial": 4, "g_trial_deep": 4, "g_vec": 4, "gain": [3, 7, 9, 11, 12, 15, 24], "galleri": [2, 23], "game": 6, "gamge": 23, "gamma": [2, 4, 10, 11, 12, 13, 15, 23], "gamma1": 10, "gamma2": 10, "gamma_": [2, 23], "gamma_0": 12, "gamma_1": 12, "gamma_1x": 12, "gamma_i": [2, 10, 20, 23], "gamma_j": 15, "gamma_k": 15, "gamma_m": 12, "gamma_x": [2, 23], "gap": 10, "gate": [6, 14], "gather": [2, 3, 14, 24], "gaug": 14, "gaussbacksub": 18, "gaussian": [6, 7, 8, 10, 16, 20, 23], "gaussian_point": 16, "gaussian_rbf": 10, "gave": 15, "gavra": 23, "gbc": 23, "gca": [4, 8, 10, 15], "gd": 3, "gd_clf": 12, "gdclassiffiercgain": 12, "gdclassiffierconfus": 12, "gdclassiffierroc": 12, "gdm": 15, "gdregress": 12, "ge": [3, 7, 9, 20, 24], "gen_loss": 6, "gen_tap": 6, "gender": [2, 23], "genener": 6, "gener": [0, 1, 2, 3, 4, 5, 7, 8, 10, 12, 13, 14, 15, 16, 18, 20, 22, 24], "generaliz": 1, "generallay": 14, "generate_and_save_imag": 6, "generate_imag": 6, "generate_latent_point": 6, "generate_simple_clustering_dataset": 16, "generated_imag": 6, "generator_loss": 6, "generator_loss_list": 6, "generator_model": 6, "generator_optim": 6, "genom": 17, "geodes": 13, "geometr": [2, 15, 23], "geometri": 7, "georg": 22, "geotif": 8, "geq": [4, 7, 10, 11, 15, 24], "geron": [2, 22, 23], "get": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 17, 18, 20, 21, 23, 24], "get_dummi": 11, "get_paramet": 4, "get_split": 11, "get_yaxi": 10, "get_yticklabel": 8, "gh": 0, "gibb": [17, 23], "gif": 6, "gini": 12, "gini_index": 11, "ginvers": 15, "git": [0, 2, 17, 23], "giter": 15, "github": [2, 17, 19, 21, 22, 23, 24], "gitignor": 0, "gitlab": [0, 2, 17, 23], "give": [2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 20, 23, 24], "given": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "global": [8, 9, 15], "glorot": 3, "gnew": 15, "go": [0, 1, 2, 3, 5, 7, 8, 10, 11, 13, 14, 15, 23, 24], "goal": [2, 9, 11, 23], "goe": [0, 2, 3, 4, 7, 8, 15, 16, 18, 23, 24], "golden": 15, "gone": [7, 24], "gong": 3, "good": [0, 3, 5, 6, 7, 8, 11, 12, 13, 15, 17, 20, 22, 24], "goodfellow": [6, 22, 23, 24], "googl": [3, 6, 17, 23], "got": [3, 8], "gotten": 23, "gov": 8, "govern": 23, "gp": 22, "gpu": [3, 15, 17, 23], "grad": [4, 15], "grad_analyt": 15, "grade": 19, "gradient": [2, 5, 6, 9, 10, 11, 14, 17, 23, 24], "gradientboostingclassifi": 12, "gradientboostingregressor": 12, "gradients_of_discrimin": 6, "gradients_of_gener": 6, "gradienttap": 6, "gradual": [3, 16], "grai": [6, 8], "graph": [1, 3, 11, 13, 14, 15], "graph_from_dot_data": 11, "graphic": [0, 2, 3, 11, 23], "grasp": 2, "gray_r": [3, 5], "grayscal": 5, "great": [0, 7, 15], "greater": [3, 9, 20], "greatli": 15, "greedi": 11, "green": [2, 5, 11, 20], "grei": 6, "grid": [3, 5, 8, 9, 10, 14, 20], "grossli": 15, "ground": [2, 23], "group": [0, 2, 8, 9, 11, 16, 17, 19, 21, 23], "groupbi": [2, 23], "grow": [3, 5, 11, 12], "growth": [2, 23], "gru": 6, "guarante": [2, 6, 15, 20, 23, 24], "guess": [3, 6, 12, 15, 16], "guestrin": 12, "gui": 0, "guid": 3, "h": [0, 2, 3, 7, 8, 10, 15, 20, 21, 22, 23, 24], "h1": 4, "h_": [2, 15, 23], "h_1": [4, 15], "h_2": [4, 15], "h_m": 12, "ha": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "haanen": [21, 23], "habit": [2, 24], "had": [2, 3, 8, 9, 15, 23], "hadamard": [3, 14, 15], "half": [3, 10, 11], "halv": 12, "hand": [2, 3, 4, 5, 7, 13, 14, 15, 17, 18, 20, 21, 22, 23, 24], "handi": 5, "handl": [0, 2, 3, 4, 7, 11, 13, 17, 24], "handle_unknown": 11, "handsid": 14, "handwrit": 14, "handwritten": [3, 7], "happen": [3, 4, 5, 6, 7, 8, 12, 15, 20, 24], "hard": [3, 9, 10, 12, 15], "hardcopi": [17, 23], "harder": [2, 3, 24], "harmon": 5, "hasn": [], "hassl": [2, 17, 23], "hast": [17, 23], "hasti": [1, 2, 8, 22, 23, 24], "hat": [1, 2, 3, 7, 8, 9, 11, 12, 13, 14, 15, 18, 24], "have": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23, 24], "haven": 3, "he": 9, "head": [2, 6, 12, 20], "header": [2, 23], "heads_proba": 12, "health": [2, 24], "hear": [2, 15, 23], "heart": [2, 9, 23], "heatmap": [2, 3, 5, 9, 23], "heavili": 2, "heavisid": 3, "height": [3, 5, 8], "held": 15, "help": [0, 1, 2, 3, 6, 14, 15, 23], "helper": [6, 16], "henc": [2, 7, 8, 10, 11, 12, 14, 15, 23, 24], "henrik": [21, 23], "her": 9, "here": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "hereaft": [2, 10, 14, 23], "hermitian": 18, "hessenberg": 18, "hessian": [2, 4, 7, 15], "heterogen": [11, 12], "hi": 9, "hidden": [3, 5, 6, 14], "hidden_bia": 3, "hidden_bias_gradi": 3, "hidden_layer_s": [2, 3, 23], "hidden_neuron": 6, "hidden_weight": 3, "hidden_weights_gradi": 3, "hierarch": [7, 24], "high": [2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 15, 16, 17, 18, 23, 24], "higher": [2, 3, 5, 7, 8, 10, 15, 23, 24], "highest": [3, 4], "highli": [2, 5, 6, 12, 17, 18, 22, 23, 24], "highwai": 2, "hing": 10, "hint": [0, 1, 15, 24], "hip": 17, "hire": 2, "hist": [6, 8, 9, 20], "histogram": [2, 8, 9, 20], "histor": [9, 13], "histori": [0, 5, 6, 14], "hitherto": 7, "hjorth": [21, 23, 24], "hobbi": 20, "hoc": [7, 24], "hoff": 22, "hold": [3, 5, 8, 15, 16], "holder": [2, 23], "home": 2, "homepag": 23, "homework": [8, 15], "homogen": [3, 5, 11, 12, 15], "honchar": 4, "hopefulli": [0, 2, 13, 20, 23], "horizont": 13, "horlyk": [21, 23], "hors": [5, 9, 23], "hot": [3, 11], "hour": [3, 17, 19, 20, 21, 23], "how": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "howev": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "hspace": [2, 6, 10, 12, 20, 23], "hstack": 3, "htf": 23, "html": [1, 2, 17, 19, 21, 22, 23], "http": [0, 1, 2, 5, 6, 8, 15, 17, 18, 19, 21, 22, 23, 24], "huang": [2, 23], "huber": [2, 23], "huge": [3, 5, 6, 17], "human": [2, 3, 5, 8, 11, 14], "humid": 11, "hundr": 3, "hungri": 3, "hybrid": 19, "hydrogen": [2, 23], "hyperbol": [3, 6, 14], "hyperparam": 10, "hyperparamet": [5, 6, 7, 8, 11, 15, 24], "hyperplan": 13, "h\u00f8rlyk": [21, 23], "i": [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24], "i0": [2, 23], "i1": [2, 8, 10, 14, 23], "i2": [2, 10, 14, 23], "i3": [2, 14, 23], "i5": [2, 23], "i_": 15, "i_1": [7, 8], "i_2": [7, 8], "ian": 22, "ic": 3, "id": [9, 15], "ida": [21, 23], "idea": [2, 3, 4, 5, 6, 8, 11, 12, 14, 15, 18], "ideal": [2, 4, 8, 10, 15, 20, 23], "idem": 8, "ident": [7, 8, 14, 15, 18, 24], "identifi": [2, 3, 9, 11, 13, 14, 15, 16, 23, 24], "ieor": 20, "ifi": 22, "ifs": [17, 23], "ignor": [0, 2, 3, 5, 11, 24], "ii": [18, 20], "iii": [18, 23], "ij": [1, 2, 3, 5, 8, 10, 14, 16, 18, 20, 23, 24], "ik": [2, 18, 23, 24], "illustr": [7, 9, 12, 14, 15, 16, 17, 23], "im": 8, "imag": [3, 5, 6, 8, 11, 13, 14, 16, 22, 23], "image_at_epoch_": 6, "image_batch": 6, "image_height": 5, "image_path": [2, 8, 9, 11, 23], "image_width": 5, "imageio": 8, "images_from_seed_imag": 6, "imagin": 3, "immedi": [2, 5, 6, 8, 17, 23], "implement": [2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 20, 23, 24], "impli": [5, 7, 8, 9, 15, 18, 24], "implicit": 5, "implicitli": [13, 20], "import": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20], "importantli": 5, "impos": [2, 8, 13, 14, 23], "imposs": [2, 7, 23, 24], "impress": [2, 14, 23], "improv": [0, 2, 6, 7, 11, 12, 13, 15, 24], "impur": 11, "imread": 8, "imshow": [3, 5, 6, 8], "in3050": [22, 23], "in3310": 23, "in4080": [22, 23], "in4300": [22, 23], "in4310": 22, "in5400": 5, "in5550": 22, "in_out_neuron": 6, "inaccur": 15, "inact": 14, "inadequ": [2, 23], "inch": 8, "includ": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 17, 20, 21, 22, 23, 24], "include_bia": [8, 11], "incom": [1, 14], "incorrect": 3, "incoveni": 10, "increas": [2, 3, 5, 6, 7, 8, 11, 14, 15, 20, 23], "increasingli": 20, "ind": 8, "inde": [2, 4, 6, 7, 8, 15, 23, 24], "indefinit": 6, "independ": [2, 7, 8, 9, 10, 14, 15, 20, 23, 24], "index": [2, 3, 5, 6, 12, 16, 17, 18, 20, 22, 23], "index_col": [2, 23], "indic": [1, 2, 3, 5, 6, 7, 8, 11, 12, 13, 15, 23, 24], "indispens": 8, "individu": [3, 8, 9, 12, 14, 20, 23, 24], "indu": 2, "indx": 18, "indx1": 4, "indx2": 4, "indx3": 4, "ineffici": [5, 15], "inequ": [10, 15], "inertia": 15, "inf1000": [17, 23], "inf1100": [17, 23], "inf1100l": [17, 23], "inf1110": [17, 23], "inf3000": 23, "infeas": 11, "infer": [2, 3, 6, 8, 22, 23], "inferenc": 3, "infil": [2, 8, 9, 11, 23], "infin": [7, 8, 9, 13, 24], "infinit": 5, "infinitesim": 20, "influenc": [8, 12], "influenti": 3, "info": 23, "inform": [2, 3, 5, 6, 8, 11, 13, 14, 15, 16, 18, 22, 23], "inforom": 0, "infti": [5, 8, 15, 20], "ingeni": 15, "ingredi": [2, 11, 23], "inher": 8, "inherit": [18, 23], "initi": [2, 3, 4, 8, 12, 15, 16, 18, 20, 23], "inject": 16, "inlin": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "inner": [2, 15, 24], "inp": 6, "inplac": 15, "input": [1, 2, 3, 5, 6, 7, 8, 9, 10, 14, 15, 16, 20, 23, 24], "input_dim": 3, "input_shap": [5, 6], "inputs": 3, "inputs_shuffl": [2, 3, 24], "insert": [5, 7, 8, 10, 12, 20, 24], "insid": [2, 6, 9], "insight": [2, 3, 7, 17, 23, 24], "insist": [8, 15], "inspir": [2, 3, 14, 23], "instabl": 4, "instal": [0, 2, 3, 7, 8, 11], "instanc": [1, 2, 3, 4, 6, 8, 11, 13, 15, 23, 24], "instanti": 12, "instead": [2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 15, 16, 18, 20, 23, 24], "institut": 3, "instruct": [0, 2, 3], "int": [2, 3, 4, 5, 6, 7, 8, 13, 15, 16, 18, 20, 24], "int32": 12, "int_": [5, 8, 20], "int_0": 20, "int_a": 20, "intak": [2, 24], "integ": [3, 4, 15, 16, 18, 20, 23], "integer_vector": 3, "integr": [5, 8, 20, 23], "intellig": [2, 16, 22, 23], "intend": 12, "intens": 3, "intention": 16, "interact": [2, 8, 11, 14, 17, 23], "intercept": [1, 2, 8, 10, 13, 15, 23, 24], "intercept_": [2, 8, 10, 11, 15, 23], "interchang": [7, 14, 18], "interconnect": 3, "interest": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 17, 20, 23, 24], "interfac": [0, 2, 3, 18, 24], "interior": [2, 11, 23], "intermedi": [18, 24], "intern": [3, 12, 14], "interpol": [3, 5, 6, 8, 14], "interpr": [7, 24], "interpret": [0, 1, 2, 3, 8, 11, 12, 14, 15, 18, 20], "interv": [2, 5, 7, 8, 9, 15, 20, 23, 24], "intial": 15, "intract": [2, 6, 24], "intrins": [5, 13, 18, 20, 23], "intro": [17, 22, 23], "introduc": [2, 3, 7, 8, 10, 12, 14, 18, 20, 23], "introduct": [3, 4, 6, 15, 22, 24], "introductori": [2, 6, 18, 22, 23, 24], "intuit": [2, 7, 8, 10, 14, 15, 23], "inv": [2, 7, 15, 23, 24], "invalu": [2, 15, 17, 23], "invari": 3, "invd": 7, "inver": 10, "invers": [2, 5, 8, 15, 23, 24], "inverse_transform": 10, "invert": [1, 2, 7, 9, 12, 15, 23], "invh": 15, "invok": [2, 10], "involv": [2, 4, 8, 9, 13, 14, 23, 24], "io": [2, 17, 19, 21, 22, 23, 24], "ip": [2, 10, 20, 23], "ipca": 13, "ipynb": [17, 23], "ipython": [2, 7, 9, 11, 13, 16, 17, 23, 24], "iq": 8, "iri": [10, 11], "irreduc": 8, "irrelev": [7, 24], "irrespect": [2, 23], "isn": 7, "isnul": 2, "isomap": 13, "issu": [0, 3, 11, 18], "it_arrai": 15, "item": [2, 15, 23], "items": [18, 23], "iter": [3, 4, 6, 8, 10, 15, 16, 20], "its": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "itself": [7, 8, 14, 20, 23, 24], "j": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 15, 16, 18, 20, 22, 23, 24], "j1": 18, "j_": 8, "j_lasso_sk": 8, "j_ridge_sk": 8, "j_sk": 8, "jackknif": [8, 17, 23], "jacobian": [4, 15], "jason": 6, "jax": [17, 23], "jensen": [21, 23, 24], "jerom": 22, "ji": [14, 18], "jit": 15, "jj": [2, 7, 8, 23], "jk": [2, 3, 8, 14, 18, 23], "jl": [2, 23], "jm": 18, "jnp": 15, "job": [0, 4, 10, 12], "join": [2, 6, 8, 9, 11, 23], "joint": [6, 7], "judg": 15, "judgement": 8, "julia": [17, 18], "jump": 20, "junk": 6, "jupit": 23, "jupyt": [0, 1, 2, 17, 22, 23], "just": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23, 24], "justif": 2, "justifi": [5, 12], "k": [2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23, 24], "k0": 9, "k1": 9, "kaggl": 8, "kappa_d": 20, "karl": [21, 23], "karush": 10, "katrin": [21, 23], "keep": [0, 2, 3, 6, 7, 8, 13, 15, 16, 18, 23, 24], "keepdim": [3, 8, 12, 18], "kei": [2, 3, 5, 8, 14, 24], "kept": [6, 8, 16], "kera": [2, 6, 17, 23], "kernel": [2, 3, 5, 17, 23, 24], "kernel_regular": [3, 5], "kernel_s": 6, "kernelpca": 13, "kev": [2, 23], "kevin": [22, 23], "keyword": [18, 23], "kfold": 8, "kg": 3, "ki": 18, "kick": [3, 15], "kiener": 4, "kilomet": 8, "kind": [2, 4, 5, 6, 10, 14, 15, 16, 23, 24], "kj": [8, 14, 18, 24], "kjm": [17, 23], "kkt": 10, "kl": 20, "km": [14, 23], "kmean": 16, "kmeanspoint": 16, "kn_k": 16, "know": [0, 1, 2, 3, 4, 7, 8, 10, 15, 17, 23, 24], "knowledg": [2, 17, 23], "known": [3, 5, 6, 7, 8, 9, 10, 11, 14, 18, 20, 22], "kondev": [2, 23], "kp": 20, "kpca": 13, "kroneck": 16, "kuhn": 10, "kvalsund": [21, 23], "kwown": [2, 23], "l": [2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 23, 24], "l0": 9, "l1": [2, 3, 5, 9, 23], "l1_l2": [3, 5], "l1regl": 7, "l2": [3, 5], "l_": 18, "l_1": 9, "l_2": [9, 15], "l_j": 14, "la": 15, "la_i": 14, "la_k": 14, "lab": [17, 23], "label": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 20, 23, 24], "labelencod": [9, 12], "labels": [8, 10, 11], "labels_shuffl": [2, 3, 24], "laboratori": 19, "lack": [2, 23], "lagari": 4, "lagrang": [10, 13], "lambda": [2, 3, 4, 5, 7, 8, 9, 10, 12, 14, 15, 20, 23, 24], "lambda_": 13, "lambda_0": 13, "lambda_1": [7, 10, 13, 24], "lambda_2": [10, 13], "lambda_i": [10, 13], "lambda_iy_i": 10, "lambda_jy_iy_j": 10, "lambda_k": 10, "lambda_n": [7, 10, 24], "lamda": 3, "land": [2, 10], "landmark": 10, "landscap": 15, "langl": [2, 8, 13, 20, 23, 24], "languag": [2, 3, 6, 10, 17, 18, 22, 23], "lapack": [18, 23], "laplac": 7, "laptop": [0, 17], "larg": [2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 20, 22, 23, 24], "larger": [2, 5, 7, 8, 10, 12, 13, 15, 20, 23, 24], "largest": [6, 10, 13], "lasso": [2, 9, 17, 23], "lasso_sk": 8, "last": [1, 2, 3, 5, 6, 7, 8, 9, 10, 14, 18, 20, 21, 23], "latent": 6, "latent_dim": 6, "latent_point": 6, "latent_space_value_rang": 6, "later": [0, 2, 3, 6, 9, 10, 14, 15, 16, 17, 23], "latest": [0, 6, 17], "latest_checkpoint": 6, "latex": 23, "latter": [2, 5, 8, 9, 10, 13, 15, 18, 20, 23, 24], "lattic": 14, "law": 2, "layer": [2, 6, 15, 23], "lbfg": [9, 11, 12], "lcc": [7, 8], "lda": 13, "ldot": [2, 8, 13, 23], "le": [7, 9, 12, 15, 20, 24], "lead": [1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "leaf": 11, "leaki": 3, "leakyrelu": 6, "lear": 15, "learn": [5, 6, 7, 8, 9, 10, 11, 12, 14, 18, 21, 22], "learnabl": 5, "learner": 12, "learnig": 23, "learning_r": [10, 12], "learning_rate_init": [2, 3, 23], "learning_schedul": 15, "least": [2, 9, 10, 12, 13, 17, 18, 20], "leat": 15, "leav": [2, 3, 5, 7, 8, 11, 13, 23], "lectur": [2, 3, 7, 12, 13, 14, 15, 17, 18, 19, 21, 22, 24], "lecturenot": [2, 17, 22, 23], "left": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "leftarrow": [10, 14], "legend": [0, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 23, 24], "leinonen": 23, "len": [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 18, 23, 24], "length": [1, 2, 3, 5, 6, 10, 11, 15, 17, 23, 24], "length_of_sequ": 6, "leq": [2, 7, 9, 10, 15, 16, 20, 23, 24], "less": [2, 3, 5, 6, 7, 8, 10, 11, 15, 17, 20, 23, 24], "lessen": 3, "let": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "letter": [1, 2, 18, 20, 23, 24], "level": [2, 3, 7, 8, 11, 17, 18, 19, 21, 23], "li": [10, 13], "lib": [], "liblinear": 12, "librari": [2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 18, 20, 22, 24], "licens": [2, 3, 17, 23], "lie": [2, 8, 13, 20, 23, 24], "life": [2, 3, 10, 14, 23], "lifetim": 15, "like": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "likelihood": [2, 3, 7, 11, 23, 24], "lim_": 20, "limit": [2, 7, 8, 10, 14, 18, 23, 24], "lin_clf": 10, "lin_model": 2, "lin_reg": 11, "linalg": [2, 4, 7, 8, 10, 13, 15, 18, 20, 23, 24], "line": [0, 1, 2, 5, 8, 10, 13, 15, 23], "line1": 10, "line2": 10, "line3": 10, "line_model": 0, "line_ms": 0, "line_predict": 0, "linear": [1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 17, 20], "linear_model": [0, 1, 2, 7, 8, 9, 10, 11, 12, 13, 15, 23, 24], "linear_regress": 8, "linearli": [7, 24], "linearloc": [8, 15], "linearregress": [0, 1, 2, 8, 9, 11, 23, 24], "linearsvc": 10, "liner": [3, 5], "linerar": 12, "linewidth": [2, 4, 6, 8, 10, 11, 12], "link": [0, 2, 6, 11, 14, 17, 19, 21, 23], "linlag": 7, "linpack": [18, 23], "linreg": [2, 23], "linspac": [1, 2, 4, 5, 6, 8, 10, 11, 12, 15, 18, 20, 23, 24], "linu": 6, "linux": [2, 3, 17, 23], "liquid": [2, 23], "list": [0, 2, 3, 4, 5, 6, 11, 17, 23, 24], "listedcolormap": [11, 12], "literatur": [3, 9, 16, 22], "littl": [3, 5, 11, 14], "live": [1, 10], "ll": [2, 20, 23, 24], "lle": [2, 24], "lloyd": [6, 16], "lmb": [2, 4, 7, 8], "lmbd": [2, 3, 5, 23], "lmbd_val": [2, 3, 5, 23], "lmbda": 15, "ln": [3, 15], "load": [2, 3, 6, 8, 9, 11, 12], "load_boston": 2, "load_breast_canc": [3, 9, 11, 12, 13], "load_data": [5, 6], "load_digit": [3, 5], "load_iri": [10, 11], "loc": [5, 8, 9, 10, 11, 12, 23], "local": [0, 2, 3, 5, 9, 14, 15, 24], "locat": [0, 4, 5, 10], "log": [0, 2, 3, 4, 6, 7, 8, 9, 11, 12, 13, 15, 18, 23, 24], "log10": [2, 7, 8], "log_": [2, 23], "log_clf": 12, "logarithm": [2, 7, 9, 18, 23], "logic": [2, 3, 11, 23], "login": 0, "logist": [2, 3, 4, 10, 11, 12, 13, 14, 15, 17, 24], "logisticregress": [9, 11, 12, 13], "logit": 9, "logreg": [9, 11, 12, 13], "logspac": [2, 3, 5, 7, 8, 23], "long": [2, 3, 5, 6, 14, 15, 23], "longer": [4, 5, 10, 12, 16, 18, 20, 23], "loocv": 8, "look": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 23, 24], "loop": [1, 3, 6, 8, 12, 14, 16, 17, 18, 23], "lose": 3, "loss": [2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15, 18, 23], "loss_fil": 6, "lossfil": 6, "lost": 6, "lot": [1, 2, 3, 6, 8], "low": [2, 8, 11, 12, 13, 23, 24], "lower": [1, 2, 3, 5, 8, 11, 12, 18, 24], "lowercas": [18, 23], "lowest": [11, 15, 20], "lr": [3, 5, 6, 12], "lstat": 2, "lstm": 6, "lstm_2layer": 6, "lstsq": [2, 23, 24], "lt": 8, "lu": [2, 7, 23, 24], "lubksb": 18, "luckili": 4, "ludcmp": 18, "lux": 18, "lvert": 3, "lw": [2, 23], "m": [0, 2, 3, 4, 5, 7, 8, 10, 11, 12, 13, 14, 15, 18, 20, 21, 22, 23, 24], "m_": [11, 14], "m_1": 16, "m_h": [2, 23], "m_k": 16, "m_l": 14, "m_n": [2, 23], "m_p": [2, 23], "m_t": 15, "ma": 13, "machin": [0, 1, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 18, 22, 24], "machinelearn": [1, 2, 8, 17, 19, 21, 22, 23], "mackai": 22, "made": [2, 3, 5, 6, 7, 8, 9, 11, 13, 14, 23, 24], "mae": [2, 23], "magic": 6, "magnitud": [3, 8, 9, 15], "mai": [2, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23, 24], "mail": [19, 21], "main": [2, 3, 5, 6, 7, 8, 9, 11, 18, 22, 24], "mainli": [2, 7, 8, 9, 11, 23, 24], "maintain": 8, "major": [3, 8, 11, 12, 15, 18, 23], "make": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 20, 22, 23], "make_axes_locat": 8, "make_moon": [10, 11, 12], "make_pipelin": [2, 8, 12, 24], "makedir": [2, 8, 9, 11, 23], "malcondit": 18, "malign": [3, 9, 11], "mammographi": 7, "manag": [0, 2, 4, 5, 17, 23], "mandatori": [21, 23], "mani": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 17, 18, 20, 22, 23, 24], "manifold": 13, "manner": 5, "manual": 8, "map": [2, 3, 4, 8, 9, 10, 13, 14, 16, 20, 23], "marc": 24, "margin": [2, 7, 10], "marit": [2, 23], "mark": 23, "marker": [2, 9, 18, 23], "markov": [17, 23], "marsaglia": 20, "mass": [2, 3, 7, 15, 24], "massag": [2, 23], "masses2016": [2, 23], "masses2016ol": [2, 23], "masses2016tre": 2, "masseval2016": [2, 23], "master": [19, 21], "mat": [17, 23], "mat1100": [17, 23], "mat1110": [17, 23], "mat1120": [17, 23], "match": [0, 3, 6, 7, 15, 16, 24], "materi": [0, 6, 7, 9, 15, 18, 19, 21, 24], "math": [5, 9, 14, 15, 18, 20, 22, 23], "mathbb": [2, 6, 7, 8, 9, 10, 13, 14, 15, 16, 18, 20, 23, 24], "mathbf": [2, 7, 8, 9, 10, 15, 18, 23, 24], "mathcal": [3, 7, 8, 9, 15], "matheemat": 5, "mathemat": [2, 8, 13, 14, 15, 17, 18, 20, 22, 23], "mathemati": 23, "mathrm": [2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 23, 24], "matmul": [3, 4, 7], "matnat": 22, "matplotlib": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "matric": [1, 2, 3, 5, 6, 8, 9, 10, 13, 15, 17, 24], "matrix": [2, 4, 5, 6, 8, 9, 10, 12, 15, 20], "matshow": 3, "matter": [4, 5, 15, 24], "max": [2, 3, 4, 5, 6, 11, 12, 14, 15, 21, 23], "max_depth": [2, 11, 12], "max_diff": 4, "max_diff1": 4, "max_diff2": 4, "max_it": [2, 3, 10, 15, 23], "max_iter": 16, "max_leaf_nod": 12, "max_sampl": 12, "maxdegre": [2, 8, 12, 24], "maxdepth": 12, "maxim": [3, 6, 7, 9, 10, 13], "maximum": [2, 4, 5, 7, 9, 10, 11, 12, 15, 16, 23, 24], "maxpolydegre": [7, 8], "maxpooling2d": 5, "mbox": [7, 8, 24], "mcculloch": 14, "md": 13, "mdoel": 6, "mean": [0, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "mean_absolute_error": [2, 23], "mean_divisor": 16, "mean_i": 20, "mean_matrix": 16, "mean_squared_error": [0, 2, 6, 8, 9, 12, 23, 24], "mean_squared_log_error": [2, 23], "mean_vector": 16, "mean_x": 20, "meaning": [2, 6, 9, 23], "meansquarederror": [2, 23], "meant": [5, 9, 12, 15], "measur": [1, 2, 3, 4, 7, 8, 11, 13, 14, 16, 20, 23, 24], "mechan": [2, 6, 20, 23], "median": [2, 23, 24], "medicin": 14, "medium": [6, 10, 15], "medv": 2, "meet": [2, 21], "mehta": [2, 23, 24], "memori": [5, 6, 13, 14, 15, 18], "mention": [2, 14, 15, 20, 23], "mere": 2, "meshgrid": [4, 7, 8, 10, 11, 12, 13, 24], "mess": 0, "messag": [7, 15], "messi": 4, "met": [2, 5, 10, 24], "meteorolog": 11, "meter": 8, "method": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 13, 14, 16, 17, 18, 20, 22, 24], "metion": 8, "metric": [0, 2, 3, 5, 8, 9, 11, 12, 16, 23, 24], "metropoli": [17, 23], "mev": [2, 20, 23], "mgd": 15, "mglearn": [17, 23], "mgrid": 15, "mhjensen": [], "mi": 12, "mia": [21, 23], "microsoft": 22, "mid": 3, "midel": 6, "midnight": 0, "midpoint": 11, "might": [0, 2, 3, 4, 6, 8, 11, 15, 24], "mild": 11, "millimet": 8, "million": [2, 23, 24], "mimic": 14, "min": [2, 4, 7, 10, 11], "min_": [2, 4, 7, 16, 23, 24], "min_samples_leaf": 11, "mind": [0, 2, 8, 15, 23, 24], "mindboard": 6, "mine": [17, 23], "mini": [3, 13, 14, 15], "minibatch": [3, 13, 15], "minibathc": 15, "miniforge3": [], "minim": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 24], "minima": [2, 3, 9, 15, 23], "minimum": [2, 3, 4, 8, 10, 11, 13, 15, 24], "minmaxscal": [2, 24], "minor": 20, "minst": 3, "minu": 9, "mirjalili": 23, "mirror": 11, "misc": 8, "misclassif": [10, 11, 12], "misclassifi": [10, 12], "miser": 2, "mismatch": 3, "miss": [2, 9, 12], "mistak": 6, "mit": 22, "mix": [3, 4, 23], "mixtur": 15, "mk": [11, 18], "mkdir": [2, 8, 9, 11, 23], "ml": [2, 3, 12, 15, 18, 24], "mlab": 20, "mle": [7, 9], "mlp": 3, "mlpclassifi": 3, "mlpregressor": [2, 23], "mm": 18, "mml": 24, "mn": [14, 20], "mnist": [3, 13], "mod": 20, "mode": [19, 21, 23], "model": [1, 4, 5, 7, 9, 10, 11, 12, 13, 15, 16, 17, 20, 22, 24], "model_select": [0, 1, 2, 3, 5, 7, 8, 9, 11, 12, 13, 23, 24], "moder": 12, "modern": [2, 8, 9, 17, 23], "modif": [4, 14, 15], "modifi": [2, 3, 5, 7, 9, 10, 12, 14, 15, 23, 24], "modul": [1, 2, 18, 23], "modular": 20, "modulo": 20, "moe": [13, 24], "moment": [7, 8, 15, 20], "mondai": [21, 23, 24], "monitor": 15, "monoton": [7, 14, 20], "mont": [2, 8, 17, 20, 22, 23], "montli": 1, "moor": [7, 8], "more": [1, 2, 3, 4, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20], "moreov": [2, 5], "morten": [21, 23, 24], "mortenhj": 23, "most": [0, 1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23, 24], "mostli": [3, 13], "motion": [2, 15], "motiv": [3, 6], "move": [0, 1, 2, 6, 7, 8, 9, 11, 14, 15, 16, 20, 24], "mpl": [9, 23], "mpl_toolkit": [4, 8, 15], "mplot3d": [4, 8, 15], "mplregressor": 3, "mse": [0, 1, 2, 6, 7, 8, 11, 12, 23, 24], "mse_simpletre": 12, "mselassopredict": 7, "mselassotrain": 7, "mseownridgepredict": 8, "msepredict": 7, "mseridgepredict": [2, 7, 8], "msetrain": 7, "msle": [2, 23], "mt": [9, 14], "mu": [2, 8, 13, 15, 20, 23], "mu0": 20, "mu1": 20, "mu2": 20, "mu_": [8, 20], "mu_i": 8, "mu_n": 13, "mu_x": 20, "much": [0, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "multi": [2, 3, 5, 9, 17, 23], "multiclass": [3, 9], "multidimension": [13, 14, 23], "multilay": 3, "multinomi": 9, "multipl": [0, 4, 6, 7, 8, 9, 14, 15, 20, 24], "multipli": [5, 7, 8, 13, 15, 18, 20, 24], "multiplum": 10, "multivari": [2, 4, 12, 13, 17, 20, 23], "multivariate_norm": [13, 16], "multpli": 1, "murphi": [13, 22, 23], "must": [0, 3, 4, 7, 8, 10, 12, 14, 15, 16, 20, 24], "mutat": 9, "mutual": [3, 5, 8, 15], "mx_": 20, "my": 23, "myenv": [], "myriad": [2, 17, 23], "mz1": 20, "mz2": 20, "m\u00f8svatn": 8, "n": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 23, 24], "n1": 18, "n2": 18, "n_": [3, 4, 5, 10, 14, 20], "n_0": [14, 20], "n_boostrap": [8, 12], "n_bootstrap": 8, "n_categori": [3, 5], "n_cluster": 16, "n_compon": 13, "n_epoch": 15, "n_estim": 12, "n_examples_to_gener": 6, "n_featur": 3, "n_filter": 5, "n_hidden": 4, "n_hidden_neuron": [2, 3, 23], "n_i": 20, "n_input": [2, 3, 5, 24], "n_instanc": 11, "n_job": 12, "n_k": 16, "n_l": [14, 20], "n_layer": 3, "n_m": 11, "n_neuron": 3, "n_neurons_connect": 5, "n_neurons_layer1": 3, "n_neurons_layer2": 3, "n_point": 16, "n_sampl": [8, 10, 11, 12, 16], "n_split": 8, "n_step": 6, "n_t": 4, "n_x": 4, "nabla": [3, 15], "nabla_": [4, 15], "nabla_w": 15, "nag": 15, "naimi": [2, 23], "naiv": 9, "naive_kmean": 16, "name": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 20, 21, 23, 24], "narrow": 15, "nation": [3, 7], "nativ": [17, 23], "natur": [2, 3, 6, 10, 11, 14, 15, 20, 22, 23], "navier": 14, "navig": 0, "nb": 20, "nb_": 18, "nbconvert": 23, "nd": 16, "ndarrai": 8, "ne": [11, 12, 18, 20, 24], "nearest": [3, 5, 8, 13], "nearli": 15, "neat": 23, "neccesari": 8, "necess": 4, "necessari": [2, 3, 5, 6, 10, 16, 23], "necessarili": [2, 6, 13, 20, 23], "necesserali": 7, "neck": 9, "need": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 24], "neg": [2, 3, 5, 7, 8, 9, 12, 15, 18, 20, 23], "neg_mean_squared_error": 8, "neglect": 20, "neglig": 20, "neighbor": [5, 8, 13], "neither": [6, 15], "neq": [15, 16, 20], "nervou": 14, "nest": [11, 14], "nesterov": 15, "net": [4, 6, 14], "netlib": [18, 23], "network": [2, 11, 15, 17, 22, 24], "neural": [2, 15, 17, 22, 24], "neural_network": [2, 3, 4, 23], "neuralnetwork": 3, "neuron": [3, 4, 5, 6, 14], "neutral": [2, 23], "neutron": [2, 23], "never": [3, 6, 8, 11, 20], "new": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 23, 24], "new_chang": 15, "new_hobbit": 23, "newaxi": [2, 5, 8, 11], "newli": [2, 23], "newton": [3, 9, 10, 15, 20], "next": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 15, 16, 23, 24], "next_guess": 15, "next_input": 6, "ng": 3, "ni": 16, "nice": [2, 3, 7, 13, 23, 24], "niter": 15, "nitric": 2, "nlambda": [2, 7, 8], "nlp": 22, "nm": 20, "nm_n": [2, 23], "nmse": 8, "nn": [4, 7, 8, 14, 18, 23], "nn_model": 3, "nnmin": 4, "node": [3, 5, 11, 12, 14], "nois": [2, 6, 7, 8, 10, 11, 12, 15, 23, 24], "noise_dimens": 6, "noisi": [3, 8], "non": [2, 3, 5, 7, 8, 9, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "none": [2, 3, 4, 6, 7, 11, 12, 15, 20, 23, 24], "nonlinear": [5, 8, 10, 11, 13, 14], "nonneg": [8, 11, 15], "nonparametr": 8, "nonsens": 20, "nonsingular": 18, "nonumb": [5, 9, 10, 15, 18], "nor": [3, 6, 15], "norm": [2, 3, 7, 8, 10, 13, 15, 23, 24], "normal": [1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "normali": [18, 23], "norwai": [8, 23], "notat": [2, 4, 7, 8, 15, 16, 20, 23, 24], "note": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 16, 17, 18, 20, 22, 23], "notebook": [0, 1, 2, 3, 5, 11, 17, 23], "noth": [3, 4, 7, 10, 14, 16, 20, 24], "notic": [6, 7, 14, 15, 18, 20, 23], "notion": 5, "novel": [5, 8, 12, 23], "novemb": [3, 21, 23], "now": [0, 1, 2, 4, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20, 23, 24], "nowadai": [2, 3, 5, 11, 17, 23], "nox": 2, "np": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23, 24], "npr": 4, "nsampl": 8, "nt": 4, "nu": 20, "nuclear": [7, 24], "nuclei": [2, 20, 23], "nucleon": [2, 23], "nucleu": [2, 23], "num": 6, "num_coordin": 4, "num_hidden_neuron": 4, "num_it": 4, "num_neuron": 4, "num_neurons_hidden": 4, "num_point": 4, "num_tre": 12, "num_valu": 4, "number": [1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 23], "numberid": 9, "numberparamet": 5, "numer": [2, 7, 8, 11, 12, 13, 14, 15, 17, 18, 22, 23, 24], "numpi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 24], "nunmpi": [7, 24], "nx": 4, "ny": 20, "o": [2, 3, 6, 7, 8, 9, 10, 11, 13, 18, 21, 22, 23, 24], "obei": [8, 13, 15], "object": [0, 2, 3, 6, 10, 12, 18, 23], "obliqu": [7, 24], "observ": [2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 23], "obtain": [2, 3, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 23, 24], "obviou": [7, 8, 13, 20, 24], "obviouli": 23, "obvious": [2, 6, 7, 8, 18, 23], "oc": 24, "occupi": 2, "occur": [2, 8, 10, 11, 18, 20, 23], "octob": [21, 23], "od": 2, "odd": [2, 5, 9, 23, 24], "odenum": 4, "odesi": 4, "oen": 2, "off": [3, 5, 6, 7, 11, 15, 20], "offer": [8, 13, 17, 18, 19, 21, 23], "offic": [21, 23], "offici": [19, 23], "often": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "ofter": [18, 23], "ol": [2, 15, 24], "old": [0, 3, 7, 12, 15], "ols_paramet": 1, "ols_sk": 8, "ols_svd": 8, "olstheta": [2, 7], "omega": [4, 5, 8], "omega_0": 5, "omit": [2, 7, 23, 24], "onc": [3, 8, 11, 13, 15], "one": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 20, 21, 23, 24], "onehot": 3, "onehot_vector": 3, "onehotencod": 11, "ones": [1, 2, 4, 7, 8, 10, 11, 12, 13, 15, 18, 23, 24], "ones_lik": 6, "ong": 24, "onl": 5, "onli": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "onlin": [0, 13, 19], "onto": [7, 13, 24], "open": [0, 2, 3, 6, 8, 9, 11, 17, 19, 21, 23], "oper": [0, 1, 2, 3, 5, 7, 8, 12, 13, 14, 15, 17, 20, 23, 24], "operation": 20, "oplu": 20, "opmiz": 15, "opportun": 2, "oppos": [8, 15], "opposit": [3, 7, 10, 24], "opt": [3, 7, 23], "optim": [1, 2, 4, 5, 6, 7, 8, 9, 11, 12, 13, 16], "optimis": [3, 5], "option": [0, 2, 3, 5, 7, 8, 10, 13, 18, 24], "optmiz": [3, 10, 15, 24], "oral": 23, "orang": 2, "order": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 18, 20, 23, 24], "ordinari": [2, 4, 5, 9, 13, 15, 17], "oreilli": [22, 23], "org": [1, 2, 5, 6, 17, 18, 22, 23], "organ": [8, 9, 12, 18], "orient": [3, 7, 20, 24], "origin": [0, 2, 5, 7, 8, 10, 13, 14, 15, 18, 23, 24], "orthogn": [7, 24], "orthogon": [2, 7, 8, 10, 13, 15, 18, 23, 24], "orthonorm": [7, 24], "os": [21, 23], "oscar": 3, "oscil": [5, 15], "oskar": 23, "oskarlei": 23, "oslo": [2, 17, 19, 21, 23, 24], "osx": [2, 17, 23], "other": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 15, 16, 17, 19, 20, 21, 22], "otherwis": [2, 3, 6, 9, 15, 18, 23], "ouput": [7, 9, 14], "our": [0, 1, 3, 4, 5, 8, 9, 10, 11, 12, 14, 16, 17, 18, 20], "ourmodel": 2, "ourselv": [2, 7, 8, 10, 13, 15, 23, 24], "out": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "out_fil": 11, "outcom": [2, 9, 11, 12, 14, 20, 24], "outdoor": 11, "outer": [8, 14, 15], "outfil": 6, "outlier": [2, 10, 23, 24], "outlin": [8, 12, 13], "outlook": 11, "outperform": 12, "output": [2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 23, 24], "output_bia": 3, "output_bias_gradi": 3, "output_shap": 6, "output_weight": 3, "output_weights_gradi": 3, "outputlayer1": 14, "outputlayer2": 14, "outsid": 6, "over": [0, 1, 2, 3, 5, 6, 7, 8, 11, 12, 14, 15, 18, 23, 24], "over1": 15, "overal": [3, 12], "overcast": 11, "overcom": [14, 15], "overdetermin": [2, 23], "overfit": [2, 3, 5, 8, 11, 12, 15], "overflow": 7, "overhead": 14, "overlap": [5, 9, 10, 11], "overlin": [2, 7, 8, 11, 12, 13, 16, 18, 23, 24], "overst": 2, "overtrain": 6, "overview": 5, "own": [1, 6, 7, 8, 10, 14, 15, 17, 18], "owner": 2, "ownmsepredict": 2, "ownmsetrain": 2, "ownridgetheta": [2, 8], "ownypredictridg": 2, "ownytilderidg": 2, "oxid": 2, "p": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23, 24], "p0": 4, "p1": 4, "p_": [4, 6, 10, 11], "p_hidden": 4, "p_i": [7, 20], "p_j": 20, "p_n": 20, "p_output": 4, "p_x": 20, "pack": [2, 23], "packag": [0, 2, 3, 5, 6, 7, 10, 13, 15, 17, 20, 24], "packtpub": 23, "packtpublish": 23, "pad": [5, 6], "page": [2, 17, 23], "pai": [0, 2, 3, 11, 15], "pair": [2, 4, 5, 11, 17, 20, 23], "paltform": 0, "panda": [2, 6, 7, 8, 9, 11, 13, 17], "panel": 23, "paper": 3, "paradigm": [2, 23], "parallel": [12, 15, 17, 18, 23], "param": 4, "paramat": 4, "paramet": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 20], "parameter": [2, 8, 12, 23, 24], "parametr": [2, 8, 23, 24], "paramt": [5, 7], "part": [2, 3, 5, 7, 8, 12, 18, 19, 20, 21, 23, 24], "partial": [1, 2, 3, 7, 8, 9, 10, 12, 13, 14, 15, 20, 23, 24], "particip": [0, 17, 19, 21, 23], "particl": [2, 6, 15, 20, 23], "particular": [1, 2, 3, 4, 5, 7, 8, 11, 12, 13, 14, 15, 20, 22, 23, 24], "particularli": [7, 8, 10, 13, 15, 20, 24], "partit": [3, 6, 11], "partli": [8, 23], "partner": 0, "pass": [4, 5, 14, 16], "past": [12, 20], "patch": [8, 20], "path": [2, 6, 8, 9, 11, 17, 23], "patient": 9, "patter": 6, "pattern": [2, 5, 6, 14, 22, 23], "pauli": [2, 23], "pc": [0, 13, 17], "pca": [2, 9, 17, 23, 24], "pd": [2, 6, 7, 8, 9, 11, 13, 23, 24], "pde": 4, "pdf": [0, 1, 2, 5, 6, 7, 8, 11, 22, 23], "pedagog": [2, 23, 24], "penal": 8, "penalti": [8, 15], "penros": [7, 8], "pentagon": 15, "peopl": [2, 3, 11, 15, 17], "per": [2, 3, 8, 19, 21, 23], "percentag": [2, 12, 13, 21], "perceptron": [2, 3, 9, 23], "peregrin": 23, "perfect": [2, 3, 15, 23], "perfectli": [6, 8], "perform": [1, 2, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "performac": 6, "perhap": [2, 7, 15, 23, 24], "perimet": 3, "period": [3, 6, 20], "permiss": 0, "permut": 13, "persist": 15, "person": [1, 7, 8, 9, 19, 21, 23], "perspect": 22, "pertin": [14, 23], "petal": [10, 11], "peter": [22, 24], "phantom": 20, "phase": [8, 14], "phenomena": 20, "phi": 10, "phi_k": 10, "philosophi": 15, "phone": [21, 23], "photo": [6, 23], "phrase": [2, 23], "physic": [2, 3, 6, 9, 14, 15, 20, 21, 22, 23, 24], "pi": [4, 5, 7, 8, 9, 11, 14, 15, 20], "pick": [3, 11, 12, 13, 15, 16], "pickl": 3, "pictur": [2, 23], "pie": [17, 23], "piec": [13, 16], "pillow": [2, 17, 23], "pinv": [7, 8, 15], "pip": [0, 2, 3, 17, 23], "pip3": [2, 3, 23], "pipelin": [2, 8, 10, 12, 24], "pippin": 23, "pit": 6, "pitfal": 8, "pitt": 14, "pixel": [3, 5, 6, 23], "pixel_height": [3, 5], "pixel_width": [3, 5], "place": [0, 2, 6, 8, 10, 15, 18, 23], "plai": [2, 5, 6, 7, 8, 10, 13, 17, 23, 24], "plain": [10, 12, 14, 15, 16], "plan": [8, 11, 21, 22, 23], "plane": [10, 11], "plateau": 7, "platform": [17, 23], "plausibl": 14, "pleas": [15, 21, 23], "plenti": 3, "plethora": [5, 14], "plot": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "plot_confusion_matrix": [9, 12], "plot_count": 8, "plot_cumulative_gain": [9, 12], "plot_data": 3, "plot_dataset": 10, "plot_decision_boundari": [11, 12], "plot_import": 12, "plot_max": 6, "plot_min": 6, "plot_model": 6, "plot_numb": 6, "plot_predict": 10, "plot_regression_predict": 11, "plot_result": 6, "plot_roc": [9, 12], "plot_surfac": [4, 8, 15], "plot_train": 11, "plot_tre": [11, 12], "plt": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "plu": [2, 5, 7, 9, 23, 24], "pm": 10, "pmatrix": 4, "pml": 22, "pn": 5, "png": [2, 6, 8, 9, 11, 23], "point": [2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 23, 24], "point_1": 6, "point_2": 6, "poisson": [17, 20, 23], "poli": [8, 10], "poly100_kernel_svm_clf": 10, "poly3": 2, "poly3_plot": 2, "poly_featur": [0, 10, 11], "poly_features10": 11, "poly_fit": 11, "poly_fit10": 11, "poly_kernel_svm_clf": 10, "poly_model": 0, "poly_ms": 0, "poly_predict": 0, "polydegre": [2, 7, 8, 12, 24], "polygon": 15, "polym": 14, "polynomi": [0, 2, 7, 8, 9, 10, 11, 12, 13, 23, 24], "polynomial_featur": [0, 1, 8], "polynomial_svm_clf": 10, "polynomialfeatur": [0, 1, 2, 8, 10, 11, 24], "polytrop": [2, 8], "pool": 5, "pool_siz": 5, "poor": [3, 15], "poorli": [2, 24], "popul": [2, 7, 23, 24], "popular": [0, 2, 3, 5, 8, 9, 10, 11, 13, 14, 17, 18, 20, 24], "popularli": [2, 23], "portabl": 12, "portion": [13, 15], "pose": [2, 6, 7, 8, 13, 20, 23], "posit": [2, 3, 4, 5, 7, 9, 10, 12, 13, 15, 16, 18, 20, 23, 24], "possibl": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 21, 23, 24], "possibli": [8, 10, 15], "posterior": 7, "postpon": [2, 24], "postul": 7, "potenti": [2, 5, 7, 8, 14, 15], "pott": 14, "power": [2, 3, 7, 8, 10, 11, 14, 15, 23, 24], "pp": [7, 8], "practic": [1, 2, 7, 8, 9, 10, 20, 24], "practition": [2, 3, 5, 23], "pre": 23, "preced": [3, 13, 14, 20], "preceed": 6, "preceq": 10, "precis": [2, 4, 7, 13, 15, 18, 20, 23, 24], "pred": 8, "predicit": 2, "predict": [0, 1, 2, 3, 7, 8, 9, 10, 11, 12, 17, 22, 23, 24], "predict_prob": 3, "predict_proba": [9, 12], "predictor": [2, 7, 8, 9, 11, 12, 13, 23, 24], "prefer": [0, 2, 3, 8, 10, 11, 13, 15, 17, 23], "prepar": [2, 8, 18, 23, 24], "preprocess": [0, 1, 2, 6, 8, 9, 10, 11, 12, 13], "prerequisit": 2, "presenc": 15, "present": [2, 7, 8, 9, 11, 14, 15, 18, 20, 23, 24], "preserv": [5, 13, 18], "press": [0, 15, 22], "pretrain": [3, 6], "pretti": [2, 6, 10, 11, 17, 23], "prev_centroid": 16, "prevent": [15, 20], "previou": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 18, 20, 24], "previous": [4, 5, 11, 12, 20], "price": [2, 6, 11, 15], "primal": 10, "primari": [2, 9, 23], "prime": 20, "princip": [2, 7, 9, 17, 23, 24], "principl": [2, 8, 9, 10, 16, 23], "print": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23, 24], "print_funct": [10, 11], "printout": [2, 23], "prior": [2, 7, 8, 23], "privat": 2, "prob": [3, 20], "probabilist": [2, 22, 23, 24], "probabl": [2, 3, 5, 6, 8, 9, 12, 15, 17, 23, 24], "problem": [2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 20], "probml": 22, "proce": [2, 7, 8, 9, 10, 11, 12, 13, 15, 18, 23, 24], "procedur": [4, 6, 7, 8, 10, 12, 13, 15, 24], "proceed": 18, "process": [2, 4, 6, 8, 11, 12, 14, 15, 17, 18, 20, 22, 23], "prod": 22, "prod_": [3, 7, 9], "produc": [2, 5, 6, 7, 8, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "product": [1, 2, 3, 5, 7, 8, 9, 10, 14, 15, 17, 18, 23, 24], "profess": [2, 23], "program": [0, 2, 3, 6, 7, 8, 10, 14, 16, 17, 18, 19, 20, 21, 23, 24], "programm": 18, "progress": [3, 6, 16], "prohibit": 8, "project": [0, 2, 3, 4, 5, 7, 13, 15, 17, 19, 24], "project_root_dir": [2, 8, 9, 11, 23], "promin": 14, "promis": 10, "promot": [21, 23], "prone": [0, 11], "pronounc": [15, 17, 23], "proof": [2, 13, 14, 15, 23], "propag": [4, 5, 15], "proper": [2, 4, 8, 9], "properli": [3, 8, 10, 12, 15], "properti": [1, 2, 3, 5, 14, 15, 18, 23], "proport": [2, 3, 7, 11, 13, 15, 20, 23, 24], "propos": [3, 6, 8, 12, 23], "propto": [7, 15], "proton": [2, 23], "prove": [5, 15], "provid": [2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 15, 17, 18, 20, 23, 24], "proxi": [3, 15], "prune": 11, "pseudo": [18, 20], "pseudoinv": 7, "pseudoinvers": [7, 8], "pseudorandom": [8, 20], "psychologi": [2, 23], "pt": 15, "public": [0, 2, 17, 23], "pull": 0, "punish": [2, 3, 23], "pure": [5, 11, 20], "purest": 11, "puriti": 11, "purpos": [2, 5, 12, 14, 16, 23], "push": 0, "put": 3, "py": 7, "pycod": 23, "pydata": 17, "pydot": 11, "pyhton2": 23, "pylab": [9, 23], "pypi": 17, "pyplot": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "pythagora": 7, "python": [3, 4, 5, 7, 8, 10, 13, 14, 15, 16, 20, 24], "python2": 2, "python3": [2, 17, 23], "pytorch": [2, 17, 23], "q": [7, 8, 10, 13, 20, 24], "qp": 10, "qquad": [4, 13, 15, 18], "qr": [7, 8, 18, 24], "quad": [3, 15, 18], "quadrat": [2, 10, 11, 15, 23], "qualit": [6, 11, 20], "qualiti": [2, 11, 17, 23, 24], "quantifi": 3, "quantil": 12, "quantit": [2, 8, 11, 23], "quantiti": [1, 2, 4, 7, 8, 9, 11, 12, 13, 14, 16, 18, 20, 23, 24], "quantum": [6, 14, 22, 23], "quartil": [2, 24], "quench": 7, "queri": 11, "question": [2, 7, 8, 11, 13, 14, 15, 21, 23, 24], "qugan": 6, "quick": [6, 20], "quickli": [3, 5, 11, 13, 15], "quit": [0, 3, 7, 8, 11, 12, 14, 24], "quot": 6, "r": [0, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 24], "r2": [2, 7, 8, 23, 24], "r2_score": [2, 23], "r2score": [2, 23], "r_1": 11, "r_2": 11, "r_j": 11, "r_m": 11, "rad": 2, "radial": [2, 10, 14], "radioact": 20, "radiu": [2, 3, 24], "rain": 11, "ramp": 3, "ran0": 20, "ran1": 20, "ran2": 20, "ran3": 20, "rand": [0, 2, 6, 7, 8, 11, 12, 15, 18, 23, 24], "randint": [8, 11, 15], "randn": [0, 2, 3, 4, 7, 8, 11, 13, 15, 23, 24], "random": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 15, 16, 17, 18, 23, 24], "random_forest_model": 12, "random_index": 15, "random_indic": [3, 5], "random_st": [2, 9, 10, 11, 12, 13], "randomforestclassifi": 12, "randomli": [3, 8, 11, 15, 16], "rang": [2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "rangl": [2, 8, 13, 20, 23, 24], "rangle_x": 20, "rank": [7, 24], "rankdir": 6, "raphson": [3, 10, 15], "rapidli": 2, "rare": [3, 15], "raschka": [23, 24], "rasckha": 23, "rate": [2, 3, 4, 5, 6, 10, 11, 12, 14, 15], "rather": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "ratio": [6, 9, 11, 12, 13], "rational": [2, 23], "ravel": [7, 8, 9, 10, 11, 12, 13, 15, 18, 24], "raw": 5, "rbf": [10, 13, 14], "rbf_kernel_svm_clf": 10, "rbf_pca": 13, "rc": [2, 20], "rcond": [2, 23, 24], "rcparam": [3, 5, 9, 10, 11, 12, 20, 23], "re": [0, 4, 6, 15], "reach": [3, 6, 7, 8, 11, 12, 14, 15, 16], "read": [1, 2, 4, 5, 6, 7, 8, 9, 10, 13, 14, 18, 20, 22], "read_csv": [2, 8, 9, 11], "read_fwf": [2, 23], "reader": [2, 8, 18, 20, 23], "readi": [2, 3, 7, 8, 10, 12, 13, 14, 18, 23], "readili": 3, "readm": 0, "readthedoc": 17, "real": [1, 2, 3, 6, 9, 12, 13, 14, 18, 24], "real_loss": 6, "real_output": 6, "realist": [10, 23], "realiti": 20, "realiz": [3, 14], "realli": [2, 3, 23], "rearrang": 15, "reason": [2, 3, 5, 6, 12, 15, 22, 23], "reassign": 3, "recal": [7, 8, 11, 12, 13, 14, 18, 20, 23, 24], "recast": 5, "receiv": [3, 5, 12, 14, 20], "recent": [2, 8, 15, 22], "recept": [5, 14], "receptive_field": 5, "recip": [2, 8, 9, 18, 23, 24], "reciproc": 7, "recogn": [2, 6, 7, 12, 23], "recognit": [2, 3, 5, 14, 22, 23], "recommen": 23, "recommend": [0, 2, 4, 5, 6, 7, 8, 10, 15, 17, 18, 22], "reconsid": 11, "reconstruct": 13, "record": [12, 19, 21, 23], "recreat": 0, "rectangl": [11, 15], "rectangular": [7, 24], "rectifi": [3, 5, 14], "recur": [2, 17, 23], "recurr": [2, 3, 17, 23], "recurs": [11, 17, 18, 23], "red": [2, 5, 6, 8, 10, 11], "redefin": [2, 12, 23, 24], "reduc": [3, 5, 7, 8, 11, 12, 13, 15, 23], "reduct": [2, 12, 13, 17, 20, 23, 24], "refer": [2, 3, 4, 5, 7, 8, 13, 14, 15, 16, 18, 22, 23, 24], "referenc": 4, "refin": 14, "refit": 8, "reflect": [2, 3, 6, 7, 20, 23], "refresh": [17, 23], "refreshprogrammingskil": 23, "reg": [12, 13], "regard": [3, 11, 15], "regardless": [1, 14], "region": [5, 6, 8, 11, 14], "regist": [8, 20], "reglasso": 7, "regr_1": [2, 11], "regr_2": [2, 11], "regr_3": [2, 11], "regress": [1, 3, 10, 13, 14, 17, 18], "regressor": [2, 9, 12], "regridg": [2, 7, 8], "regular": [2, 5, 6, 7, 8, 9, 11, 15, 21, 23, 24], "regularli": 0, "reilli": [2, 22, 23], "reinforc": [2, 10, 17, 23], "reiter": 3, "reject": 9, "rel": [2, 6, 8, 9, 11, 14, 15, 20, 23, 24], "relat": [2, 3, 5, 6, 7, 13, 15, 16, 18, 20, 23], "relationship": [2, 6, 11, 23], "relativeerror": [2, 23, 24], "releas": [3, 17, 23], "relev": [2, 3, 7, 9, 13, 17, 20, 23], "reli": [2, 8, 10], "reliabl": [9, 20], "relu": [5, 6, 23], "remain": [3, 4, 6, 8, 14, 18, 20], "remaind": 20, "reman": 4, "remark": 3, "rememb": [2, 10, 15, 18, 23], "remind": [2, 7, 13, 15, 18, 20], "remot": 0, "remov": [2, 6, 7, 8, 24], "renam": 0, "render": [2, 23, 24], "reorder": [7, 9, 24], "reorgan": [2, 23], "repeat": [2, 3, 5, 6, 7, 8, 11, 12, 13, 15, 16, 18, 20, 23, 24], "repeated": 23, "repeatedli": [2, 8, 12, 15], "repet": 5, "repetit": [8, 23, 24], "rephras": 15, "replac": [2, 3, 5, 6, 7, 8, 12, 14, 16, 17, 23, 24], "replica": 8, "repo": 0, "report": 23, "repositori": [2, 6, 23], "repres": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 20, 23, 24], "represent": [2, 3, 5, 8, 20, 23], "representd": 5, "reproduc": [0, 1, 2, 7, 8, 11, 14, 17, 20, 23, 24], "repuls": [2, 23], "request": [2, 15], "requir": [0, 2, 3, 5, 6, 7, 8, 10, 11, 13, 14, 15, 18, 23, 24], "res1": 4, "res2": 4, "res3": 4, "res_analyt": 4, "res_analytical1": 4, "res_analytical2": 4, "res_analytical3": 4, "resaml": 8, "resampl": [2, 9, 12, 17, 23, 24], "rescal": [2, 13, 14, 24], "rescu": 7, "reseach": 8, "research": [2, 6, 15, 17, 22, 23], "resembl": [8, 20], "reserv": [3, 7, 8, 20], "reshap": [2, 3, 4, 5, 6, 8, 10, 11, 12, 18, 23, 24], "residenti": 2, "residu": [2, 7, 15, 23], "resiz": [7, 24], "resourc": 23, "respect": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 16, 20, 23, 24], "respond": 14, "respons": [2, 9, 11, 14, 23, 24], "rest": [2, 7, 24], "restat": [2, 14, 23], "restor": 6, "restored_discrimin": 6, "restored_gener": 6, "restrict": [2, 5, 11, 14, 23], "result": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "retail": 2, "retain": [7, 8, 24], "return": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 18, 20, 23, 24], "return_data": 16, "return_sequ": 6, "return_x_i": 11, "reus": [3, 5, 8], "reveal": [2, 14, 23], "revers": [3, 18], "review": [17, 18], "revisit": 16, "revolut": 23, "reward": [2, 6, 23], "rewrit": [1, 2, 5, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20], "rewritten": [4, 8, 10, 12, 20], "rewrot": 15, "rf": 12, "rgb": 5, "rgoj5yh7evk": 17, "rh": 8, "rho": [2, 12, 15], "rho_1": 12, "rho_2": 12, "rho_m": 12, "rich": [2, 23], "ride": 11, "rideclass": 11, "ridedata": 11, "ridg": [9, 13, 15, 17, 23], "ridge_sk": 8, "ridgetheta": 7, "right": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 23, 24], "right_sid": 4, "rightarrow": [2, 3, 7, 8, 10, 13, 14, 15, 20, 23, 24], "rigor": [2, 23], "ring": 8, "rise": [2, 23], "risk": [2, 15, 23], "rival": 6, "river": 2, "rlm": 23, "rm": [2, 20], "rmse": 2, "rmsporp": 15, "rmsprop": [3, 5, 6, 15], "rnd_clf": 12, "rng": 20, "rnn": [6, 14], "rnn1": 6, "rnn2": 6, "rnn_2layer": 6, "rnn_input": 6, "rnn_output": 6, "rnn_train": 6, "rntrick1": 20, "rntrick2": 20, "rntrick3": 20, "rntrick4": 20, "ro": [2, 15, 23], "robert": 22, "robust": [2, 23], "robustscal": [2, 24], "roc": [9, 12], "role": [2, 4, 7, 8, 10, 17, 23, 24], "roll": 8, "room": [2, 21, 23], "root": [0, 2, 7, 11, 15, 20, 24], "rot": 23, "rotat": [3, 10, 11, 12], "rotation_matrix": 11, "roughli": [3, 5], "round": [2, 9, 11, 15], "routin": [15, 18, 23], "row": [1, 2, 3, 4, 7, 8, 11, 13, 18, 23, 24], "rr": [7, 24], "rrr": [7, 24], "rug": 15, "rule": [2, 3, 7, 8, 15, 23, 24], "run": [0, 2, 3, 4, 6, 7, 8, 10, 11, 13, 15, 17, 23, 24], "runtim": [0, 3, 8, 16], "rust": [2, 17, 18, 23], "rvert": 3, "rvert_2": 3, "s_": [5, 8], "s_1": 8, "s_i": [8, 9], "s_j": 8, "s_k": 8, "saddl": 15, "sai": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 20, 23, 24], "said": [8, 11, 15], "sake": [2, 7, 9, 13, 23, 24], "sale": [2, 23], "sam": 23, "same": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 16, 18, 20, 23, 24], "samm": 12, "sampl": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 16, 17, 18, 20, 23, 24], "sample_vari": 16, "sampleexptvari": 20, "samwis": 23, "sastri": 13, "satisfactori": [2, 23], "satisfi": [3, 4, 5, 8, 10, 15, 18, 20], "satur": [3, 8], "save": [2, 6, 8, 9, 11, 15, 23], "save_fig": [2, 8, 9, 11, 12, 23], "savefig": [2, 6, 8, 9, 11, 20, 23], "savetxt": 6, "saw": [7, 24], "scalabl": 12, "scalar": [4, 7, 8, 12, 24], "scale": [2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 21, 23], "scale_mean": 6, "scale_std": 6, "scaler": [2, 9, 10, 11, 12, 13, 24], "scan": [7, 9], "scari": 7, "scatter": [0, 2, 3, 8, 9, 10, 11, 16, 23], "scenario": [8, 15], "schedul": 15, "scheme": [3, 15], "schrage": 20, "sch\u00f8yen": 8, "scienc": [2, 3, 12, 14, 15, 17, 19, 20, 21, 22], "scientif": [2, 17, 23], "scientist": [2, 23], "scikit": [0, 1, 5, 7, 8, 10, 11, 12, 15, 17, 18, 22], "scikit_learn": 2, "scikitlearn": 23, "scikitplot": [9, 12], "scipi": [2, 5, 7, 8, 15, 17, 18, 23, 24], "scl": 8, "scm": 0, "score": [0, 1, 2, 3, 5, 8, 9, 11, 12, 13, 21, 23, 24], "scores_kfold": 8, "scratch": [1, 3, 15], "sdg": 15, "seaborn": [2, 3, 5, 8, 9, 23], "seamless": [2, 17, 23], "search": [0, 2, 3, 5, 7, 11, 15, 23], "sebastian": 23, "sebastianraschka": 23, "sec": 8, "second": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 16, 17, 18, 20, 21, 23, 24], "secondeigvector": 13, "secondli": 14, "section": [1, 6, 13, 18, 20, 24], "sector": 2, "see": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 18, 20, 23, 24], "seed": [2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 15, 16, 20, 23, 24], "seed_imag": 6, "seek": [3, 4, 10], "seem": [3, 5, 6], "seemingli": [2, 23], "seen": [2, 3, 5, 7, 12, 14, 20], "segment": 15, "seismic": 8, "seldomli": [2, 23], "select": [0, 3, 7, 8, 10, 11, 12, 13, 19, 20, 21, 22, 23, 24], "selevet": 0, "self": [3, 7, 24], "sell": 6, "semest": [9, 19], "semi": [10, 15], "semilogx": 8, "send": [7, 14, 15, 21, 23], "senior": [19, 21], "sens": [2, 6, 8, 10, 23], "sensibl": 5, "sensit": [2, 7, 8, 11, 15, 23, 24], "sent": 4, "sentenc": [6, 14], "separ": [2, 3, 4, 6, 8, 10, 11, 14, 16, 17, 20, 23], "septemb": 23, "sequenc": [5, 6, 9, 11, 12, 14, 15, 17, 18, 20, 23], "sequenti": [3, 5, 6, 12, 14, 20], "seri": [2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 18, 23, 24], "serif": [9, 20, 23], "serv": [2, 3, 4, 5, 7, 9, 15, 22, 23, 24], "session": [0, 3, 19, 21, 23], "set": [1, 3, 6, 7, 8, 9, 10, 12, 13, 15, 16, 17, 18, 20, 21], "set_major_formatt": 8, "set_major_loc": 8, "set_tick": [3, 10], "set_ticklabel": 3, "set_titl": [2, 3, 4, 5, 9, 14, 16, 23], "set_xlabel": [2, 3, 4, 5, 9, 14, 23], "set_xlim": [9, 14], "set_xticklabel": 3, "set_ylabel": [2, 3, 4, 5, 9, 23], "set_ylim": [9, 14], "set_ytick": 9, "set_yticklabel": [3, 8], "set_zlim": 8, "seth": 6, "setminu": 8, "setosa": [10, 11], "setosa_or_versicolor": 10, "setp": 8, "setup": [3, 6, 8, 10, 17, 23, 24], "sever": [1, 2, 5, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23, 24], "sgd": [3, 5], "sgd_clf": 10, "sgdclassifi": 10, "sgdreg": 15, "sgdregressor": 15, "sgn": [7, 24], "shallow": 15, "shape": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 23, 24], "share": [0, 3, 5, 23], "shareabl": 0, "she": 9, "shift": [0, 3, 8, 14, 20], "ship": 5, "shire": 23, "short": [6, 7], "shortcom": 15, "shorten": 6, "shorter": 20, "shorthand": 23, "shortli": [18, 23], "should": [0, 2, 4, 5, 7, 8, 10, 11, 13, 14, 18, 20, 23, 24], "show": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "show_shap": 6, "shown": [2, 6, 7, 10, 14, 15, 18, 24], "shrink": [5, 7, 8, 10, 13, 24], "shrinkag": [7, 8, 24], "shrunk": 13, "shuffl": [2, 3, 6, 8, 15, 24], "side": [2, 4, 7, 10, 14, 15, 18, 23], "sigh": [17, 23], "sigma": [2, 3, 7, 8, 9, 12, 13, 14, 15, 18, 20, 23, 24], "sigma0": 20, "sigma1": 20, "sigma2": 20, "sigma_": [7, 18, 23, 24], "sigma_0": [7, 24], "sigma_1": [7, 24], "sigma_2": [7, 24], "sigma_fn": [9, 14], "sigma_i": [2, 7, 23, 24], "sigma_j": [7, 24], "sigma_m": [8, 20], "sigma_n": [13, 20], "sigma_t": 15, "sigma_x": 20, "sigmoid": [3, 4, 6, 9, 10, 12, 14], "sigmundson": 8, "sign": [3, 4, 9, 10, 12, 20, 21], "signal": [3, 5, 12, 14], "signifi": 6, "signific": 3, "significantli": [3, 15, 20], "sim": [6, 7, 8, 15, 20], "similar": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 23], "similarli": [2, 3, 5, 7, 10, 12, 15, 20, 23, 24], "simpl": [1, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20], "simplepredict": 12, "simpler": [1, 2, 3, 7, 8, 9, 15, 17, 23], "simplernn": 6, "simplest": [2, 3, 5, 6, 11, 12, 14, 16, 23], "simpletre": 12, "simpli": [2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 17, 18, 20, 23, 24], "simplic": [4, 7, 8, 9, 10, 11, 12, 13, 14, 16, 24], "simplicti": [7, 24], "simplifi": [2, 8, 11, 17, 23], "simplist": [5, 8, 20], "simul": 8, "simultan": 8, "sin": [2, 3, 4, 5, 6, 11, 14, 15, 18, 23], "sinc": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 22, 23, 24], "sine": [5, 14], "singl": [2, 3, 4, 5, 7, 8, 9, 10, 11, 14, 15, 18, 20, 23, 24], "singular": [2, 8, 15, 18, 23], "sinusoid": 5, "site": [2, 19], "situat": [2, 6, 7, 9, 15, 20, 23, 24], "six": [5, 20], "size": [2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 18, 20, 23], "sketch": 12, "ski": 11, "skill": 2, "skip": 13, "skl": [2, 8, 23, 24], "sklearn": [0, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 23, 24], "skplt": [9, 12], "sl": 8, "slack": 10, "slice": [4, 18, 23], "slide": [1, 2, 5, 20, 23, 24], "slight": [8, 15], "slightli": [3, 4, 5, 7, 8, 9, 12, 20, 24], "slope": [10, 13, 14], "slow": [2, 4, 10, 15, 24], "slower": [7, 18, 23, 24], "slowest": 18, "slowli": 14, "slp": 3, "small": [2, 3, 4, 5, 7, 8, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "smaller": [2, 3, 4, 7, 8, 10, 11, 13, 15, 20, 23], "smallest": [2, 6, 16, 23], "smallest_row_index": 16, "smooth": [2, 5, 8, 15, 23], "sn": [2, 3, 5, 8, 9, 23], "sne": 13, "so": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23, 24], "soar": 8, "social": 2, "soft": [3, 9, 12, 14], "soften": 10, "softmax": [5, 9], "softwar": [2, 10, 17, 18], "sol": 10, "sole": [2, 8, 23], "solid": [2, 9], "solut": [2, 3, 4, 5, 7, 8, 10, 12, 13, 15, 18, 20, 23, 24], "soluton": 4, "solv": [1, 2, 3, 5, 7, 8, 10, 12, 13, 14, 15, 18, 23, 24], "solve_expdec": 4, "solve_ode_deep_neural_network": 4, "solve_ode_neural_network": 4, "solve_pde_deep_neural_network": 4, "solveod": 4, "solveode_popul": 4, "solver": [4, 9, 10, 11, 12, 18, 23], "some": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 20, 23], "some_model": 8, "somehow": 6, "someon": 1, "someth": [0, 2, 3, 5, 6, 9, 11, 13, 20, 23, 24], "sometim": [2, 3, 13, 14, 15, 16, 24], "soon": [18, 21, 24], "sophist": [2, 23], "sopt": 15, "sort": [7, 8, 11, 13, 20, 24], "sound": [5, 7], "sourc": [2, 3, 5, 8, 17, 18, 20, 23], "space": [2, 3, 6, 7, 10, 11, 13, 14, 15, 16, 20, 24], "span": [2, 5, 7, 11, 13, 18, 23, 24], "spare": 3, "spars": [5, 8, 18, 23], "sparse_mtx": [18, 23], "sparsecategoricalcrossentropi": 5, "sparsiti": 12, "spatial": [3, 4, 5, 14], "speak": 20, "special": [8, 9, 12, 14, 15, 18, 20, 23], "specif": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 17, 18, 20, 22, 23, 24], "specifi": [2, 5, 7, 8, 9, 11, 13, 15, 16, 20, 23], "specifici": [2, 12, 23], "spectacular": 5, "spectral": 3, "speech": [2, 3, 5, 6, 14], "speed": [3, 4, 6, 15], "spend": [1, 20], "sphere": [2, 24], "spin": 8, "spite": 2, "spline": 10, "split": [1, 3, 5, 6, 7, 8, 10, 11, 12, 13, 16, 20, 23], "splite": 2, "splitter": [3, 12], "spontan": 20, "spot": 5, "spread": [2, 13, 20, 23, 24], "springer": [22, 23], "spuriou": 15, "sqrsignal": 5, "sqrt": [2, 5, 6, 7, 8, 10, 12, 13, 15, 20, 24], "squar": [0, 3, 4, 5, 6, 9, 10, 11, 13, 15, 16, 17, 18, 20], "squarederror": 12, "squaredeuclidean": 16, "squash": 14, "srtm": 8, "srtm_data_norway_1": 8, "stabil": 7, "stabl": [1, 2, 6, 7, 8, 11, 17, 23, 24], "stack": [5, 6], "stage": [0, 7, 15], "stai": [2, 4, 6, 7, 13, 23, 24], "stand": [2, 7, 11, 14, 23, 24], "standard": [2, 3, 6, 7, 8, 9, 10, 12, 14, 18, 20, 23], "standardscal": [2, 8, 9, 10, 11, 12, 13, 24], "stanford": 15, "start": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23, 24], "start_tim": 16, "stat": 8, "state": [3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 20, 23, 24], "statement": [2, 9, 18, 23], "statist": [2, 3, 5, 6, 9, 11, 12, 13, 14, 15, 16, 18, 22, 24], "statu": [0, 2, 9, 23], "stavang": 8, "std": [2, 6, 8, 23, 24], "steep": 15, "step": [0, 2, 3, 4, 6, 8, 9, 11, 12, 13, 14, 15, 16, 18, 23], "step_fn": [9, 14], "step_length": 15, "steps_list": 11, "stereo": 5, "still": [2, 4, 5, 7, 8, 13, 15, 20, 24], "stimuli": 14, "stk": [22, 23], "stk2100": [22, 23], "stk3155": [0, 19, 21], "stk4021": [22, 23], "stk4051": [22, 23], "stk4155": [19, 21], "stk5000": 22, "stochast": [2, 3, 7, 8, 10, 13, 14], "stock": 6, "stoke": 14, "stone": [2, 9], "stop": [3, 6, 11, 15, 16], "storag": [7, 24], "store": [2, 3, 4, 5, 8, 13, 15, 20, 23], "storehaug": [21, 23], "str": [3, 5, 6], "straight": [2, 8, 10, 15, 23], "straightforward": [2, 4, 5, 7, 8, 10, 11, 12, 15, 18, 23, 24], "strategi": [2, 3, 11, 23], "stratifi": 8, "strength": [2, 7, 16, 24], "stretch": 13, "strict": [10, 15], "strictli": [10, 15], "stride": [6, 18], "strike": 8, "string": 3, "stroke": 9, "strong": [5, 8, 11, 12, 14, 18, 20], "strongli": [0, 2, 10, 17, 18], "stronli": 2, "structur": [2, 3, 4, 5, 8, 11, 12, 14, 17, 23], "stuck": [3, 15], "student": [0, 2, 19, 21, 22, 23], "studi": [2, 5, 6, 7, 8, 9, 10, 13, 14, 15, 17, 22, 23, 24], "studier": 22, "style": [9, 11, 18, 23], "st\u00f8land": 21, "sub": [11, 14], "subdivid": [2, 18, 23], "subfield": 2, "subject": [8, 10, 20], "submit": 23, "subplot": [2, 3, 5, 6, 8, 9, 10, 11, 12, 16, 23], "subplots_adjust": [10, 20], "subprogram": [18, 23], "subract": [2, 24], "subroutin": [2, 23], "subscript": 3, "subsequ": [3, 6, 7, 8, 14, 18, 20, 24], "subset": [3, 8, 11, 14, 15, 17, 23], "subspac": [2, 10, 13, 24], "substanti": [11, 12], "substep": 13, "substitut": [1, 5, 8, 14, 18], "subsubset": 11, "subtask": 8, "subtl": 3, "subtract": [2, 6, 7, 8, 13, 15, 18, 20, 24], "subtre": 11, "succeed": [2, 6, 23], "success": [5, 9, 11, 15, 20], "successfulli": [6, 11], "sudo": [2, 17, 23], "suffer": [2, 3, 4, 7, 12, 23, 24], "suffici": [3, 8, 10, 13, 15], "suggest": [3, 15, 22], "suit": [10, 14], "suitabl": [0, 2, 20, 24], "sum": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "sum_": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "sum_i": [2, 4, 7, 8, 10, 15, 24], "sum_j": 8, "sum_ja_": 2, "sum_k": [8, 10, 14, 18], "sum_logist": 15, "sum_m": 5, "sum_n": 5, "sum_nx_": 5, "summar": [7, 8, 11], "summari": [3, 5, 6, 12, 19], "summat": [1, 2, 5, 24], "sunni": 11, "super": [7, 24], "superfici": 5, "superscript": [3, 14], "supervis": [2, 7, 8, 9, 11, 14, 17, 23, 24], "supplement": 9, "support": [2, 3, 11, 12, 13, 15, 17, 23, 24], "suppos": [2, 7, 8, 9, 10, 12, 13, 14, 15, 18, 23, 24], "suppress": [7, 15], "sure": [1, 2, 3, 6, 8], "surf": 8, "surfac": [2, 8, 23], "surpass": 8, "surpris": [2, 23], "surround": [5, 17], "survei": [2, 7, 8, 23], "svc": [10, 11, 12], "svd": [2, 8, 13, 23], "svdinv": 7, "svm": [10, 11, 12, 13], "svm_clf": [10, 12], "swath": [7, 24], "switch": 2, "sy": 15, "symbol": [3, 7, 13, 15, 17, 20, 23, 24], "symmeteri": 3, "symmetr": [2, 7, 10, 13, 14, 15, 18, 23, 24], "symmetri": 8, "sympi": [2, 17, 23], "synonim": 20, "syntax": 15, "system": [0, 2, 3, 5, 6, 8, 9, 11, 12, 14, 15, 17, 18, 23], "systemat": [6, 8], "t": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "t0": [5, 8, 15], "t1": [4, 15], "t2": 4, "t3": 4, "t_": 4, "t_0": [4, 11, 15], "t_1": 15, "t_b": 12, "t_i": [3, 4, 7, 14, 24], "t_j": 14, "t_k": 11, "tabl": [11, 20, 21, 23], "tabul": [2, 23], "tabular": 23, "tackl": 6, "tag": [4, 5, 6, 7, 8, 9, 14, 15, 16, 18, 20, 24], "taht": [2, 23], "tail": 20, "tailor": [4, 10, 13, 23], "taiwan": [2, 23], "take": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "taken": [2, 3, 5, 8, 12, 15, 18], "tan": 5, "tangent": [3, 6, 14, 15], "tanh": [3, 6, 9, 10, 14], "target": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 23, 24], "target_nam": 11, "task": [2, 3, 5, 8, 11, 13, 14, 16, 23], "tau": [5, 7, 20], "taught": 23, "tax": 2, "taylor": [4, 15], "taylornr": 15, "tc": 10, "teach": [0, 19, 23], "team": 3, "teaser": 2, "technic": [2, 7, 8, 15], "techniqu": [2, 3, 10, 12, 15, 17, 20, 22, 23, 24], "technologi": [2, 3], "tell": [1, 2, 6, 8, 12, 13, 15, 20], "temp": 3, "temp1": 3, "temp2": 3, "temperatur": [2, 11, 23], "temporarili": 3, "ten": [5, 23], "tend": [5, 7, 8, 10, 11, 12, 14, 15, 16], "tendenc": [2, 23], "tension": 8, "tensor": 5, "tensorflow": [2, 4, 6, 10, 16, 17, 18, 22, 23, 24], "term": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 23, 24], "term1": [7, 8, 13, 24], "term2": [7, 8, 13, 24], "term3": [7, 8, 13, 24], "term4": [7, 8, 13, 24], "termin": [0, 2, 6, 7, 11, 12, 15, 24], "terminarl": 0, "terrain": 8, "terrain1": 8, "test": [1, 5, 6, 7, 8, 9, 10, 11, 12, 15, 18, 20, 23], "test_acc": 5, "test_accuraci": [3, 5], "test_error": 8, "test_imag": [5, 6], "test_ind": 8, "test_input": 6, "test_label": [5, 6], "test_loss": 5, "test_pr": 3, "test_predict": 3, "test_rnn": 6, "test_scor": [9, 12], "test_siz": [0, 2, 3, 5, 7, 8, 12, 24], "test_split": 11, "testerror": [2, 8, 24], "testi": 6, "testpredict": 6, "testx": 6, "text": [0, 2, 3, 4, 6, 7, 10, 11, 13, 15, 18, 20, 22, 24], "textbook": [1, 24], "textual": 11, "textur": 3, "tf": [3, 5, 6, 15, 16], "th": [2, 3, 4, 7, 8, 9, 11, 14, 15, 16, 18, 20, 23, 24], "than": [2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 17, 20, 23, 24], "thank": [6, 8], "theano": [3, 17, 23], "thei": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 23, 24], "them": [2, 3, 5, 6, 8, 10, 11, 12, 13, 14, 15, 18, 23, 24], "theme": [0, 2, 23], "themselv": [2, 20, 23], "thenc": 8, "theorem": [4, 8, 9, 24], "theoret": [2, 6, 12], "theori": [2, 3, 5, 10, 11, 14, 15, 17, 22, 23], "thereaft": [2, 7, 8, 13, 14, 18, 23], "therebi": [2, 7, 9, 13, 23, 24], "therefor": [2, 3, 4, 5, 6, 8, 9, 10, 13, 15, 20, 23, 24], "therein": 13, "thereof": [2, 8, 15, 23], "theta": [1, 2, 3, 6, 7, 8, 9, 15, 20, 23, 24], "theta_": [2, 3, 8, 9, 15, 23, 24], "theta_0": [1, 2, 7, 8, 9, 23, 24], "theta_0x_": [2, 23, 24], "theta_1": [2, 7, 8, 9, 23, 24], "theta_1x_": [2, 23, 24], "theta_1x_0": [2, 23], "theta_1x_1": [2, 9, 23], "theta_1x_2": [2, 23], "theta_1x_i": [9, 24], "theta_2": [2, 23, 24], "theta_2x_": [2, 23, 24], "theta_2x_0": [2, 23], "theta_2x_1": [2, 23], "theta_2x_2": [2, 9, 23], "theta_2x_i": 24, "theta_3x_i": 24, "theta_4x_i": 24, "theta_i": [2, 3, 7, 23, 24], "theta_j": [2, 7, 8, 23, 24], "theta_linreg": 15, "theta_p": 9, "theta_px_p": 9, "theta_t": 15, "thetavalu": 7, "thi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24], "thing": [0, 1, 2, 3, 4, 6, 7, 9, 11, 20, 23], "think": [2, 3, 5, 6, 8, 11, 14, 15, 16, 20, 23, 24], "third": [2, 5, 8, 15, 21, 23], "thirti": 9, "thorughout": 23, "those": [2, 5, 7, 8, 10, 11, 12, 13, 18, 23, 24], "though": [1, 3, 4, 5, 6, 15, 18, 20], "thought": [8, 16, 20], "thousand": [2, 3, 24], "three": [2, 3, 5, 7, 8, 10, 11, 14, 18, 19, 20, 21, 23, 24], "threshold": [3, 5, 11, 12, 13, 14, 15], "through": [0, 2, 3, 4, 5, 6, 7, 8, 10, 13, 14, 15, 16, 17, 18, 20, 23, 24], "throughout": [0, 2, 6, 7, 16, 17, 18, 20, 23], "throw": [5, 8, 20], "thu": [2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 21, 23, 24], "thumb": [2, 8, 24], "thursdai": 24, "tibshirani": [8, 22, 23], "tick_param": 8, "ticker": [8, 15, 20], "tif": 8, "tight_layout": [3, 9], "tightli": 13, "tild": [2, 7, 8, 9, 13, 20, 23, 24], "till": [2, 6, 9, 10, 11, 12, 14, 18, 23, 24], "time": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "timeit": 6, "timer": 6, "tini": 3, "tip": 5, "titl": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 15, 20, 23], "tmp": 15, "tn": [4, 5, 9], "to_categor": [3, 5, 6], "to_categorical_numpi": 3, "to_numer": [2, 8, 23], "todai": 5, "togeth": [2, 5, 8, 10, 13, 15, 17, 23], "toi": 16, "told": 15, "toler": [4, 16], "tolist": 6, "tomographi": 14, "too": [2, 4, 6, 7, 8, 11, 13, 15, 20, 22, 24], "took": [10, 23], "tool": [0, 2, 3, 5, 8, 15, 17, 24], "toolbox": 10, "top": [2, 5, 7, 8, 11, 12, 17, 23], "topic": [2, 7, 8, 9, 10, 17, 24], "topolog": [5, 14], "topologi": [3, 14], "torkjellsdatt": [21, 23], "toss": [12, 20], "total": [2, 3, 4, 5, 6, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 21, 23, 24], "total_loss": 6, "totalclustervari": 16, "totalscatt": 16, "toward": [0, 3, 4, 9, 14, 15], "town": 2, "tp": [6, 9], "tpng": 11, "tpu": [15, 17, 23], "tqdm": 8, "tr": 24, "track": [0, 5, 15, 16, 18, 24], "tract": 2, "tractabl": [2, 23, 24], "trade": [7, 11], "tradeoff": [2, 7, 23, 24], "tradit": [2, 3, 6, 8, 23], "train": [1, 4, 5, 7, 8, 10, 11, 12, 13, 14, 15], "train_accuraci": [2, 3, 5, 23], "train_dataset": 6, "train_end": [2, 3, 24], "train_error": 8, "train_imag": [5, 6], "train_ind": 8, "train_label": [5, 6], "train_pr": 3, "train_siz": [2, 3, 5, 24], "train_step": 6, "train_test_split": [0, 1, 2, 3, 5, 7, 8, 9, 11, 12, 13, 23, 24], "train_test_split_numpi": [2, 3, 24], "trainable_vari": 6, "trained_model": 8, "trainerror": [2, 24], "traini": 6, "training_checkpoint": 6, "training_dataset": 6, "training_gradi": 15, "trainingerror": 8, "trainpredict": 6, "trainscor": 6, "trainx": 6, "trait": [2, 23], "trajectori": 6, "transfer": [11, 23], "transform": [2, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 23, 24], "transit": [8, 14], "translat": [3, 6, 8, 12, 23], "transpos": [3, 7, 13, 18, 24], "travers": [2, 7], "treat": [2, 3, 5, 8, 14, 15, 20, 23, 24], "tree": [2, 3, 17, 23], "tree_clf": [11, 12], "tree_clf_": 11, "tree_clf_sr": 11, "tree_reg": 11, "tree_reg1": 11, "tree_reg2": 11, "trend": 20, "treue": 9, "trevor": 22, "tri": [1, 4, 5, 6, 11, 15], "triain": 2, "trial": [2, 4, 6, 8, 15, 20, 23], "triangl": 15, "triangular": 18, "trick": [5, 6, 10, 13, 15, 20], "trickier": 20, "tridiagon": 18, "trillion": 17, "trivial": [2, 3, 7, 13, 20, 23], "troubl": [0, 2, 10, 14, 24], "truck": 5, "true": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 23, 24], "true_fun": 8, "true_theta": 8, "truli": 23, "try": [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 20, 23], "tucker": 10, "tuesdai": [21, 23], "tumor": [9, 11], "tumour": 9, "tunabl": 3, "tune": [6, 11, 15, 18, 23], "turn": [2, 3, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "tutori": [3, 6], "tv": 4, "tveito": 4, "tweak": [3, 6, 12, 20], "twice": 15, "twist": 13, "two": [0, 2, 3, 4, 6, 7, 8, 9, 11, 12, 13, 14, 15, 18, 19, 20, 22, 23, 24], "tx": 15, "tx_1": 15, "txt": [0, 6], "ty": 15, "type": [2, 3, 5, 8, 10, 12, 15, 18, 20, 24], "typic": [0, 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 14, 15, 20, 23, 24], "u": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 22, 23, 24], "u_": 18, "u_i": 14, "u_m": 12, "ua": [2, 23], "ubuntu": [2, 17, 23], "uci": 2, "uio": [0, 21, 22], "un": 16, "unabl": 0, "unari": [18, 23], "unbalanc": [8, 11], "unbias": [2, 7, 8, 23], "uncent": 8, "uncertainti": [2, 7, 23], "uncertitud": 20, "unchang": [3, 5], "uncorrel": [12, 20], "undefin": [7, 24], "under": [2, 3, 7, 8, 12, 15, 17, 23], "underdetermin": [2, 23], "underfit": [3, 8], "underflowproblem": 7, "undergo": 7, "undergradu": [19, 21], "underli": [2, 3, 11, 15, 20, 23], "underset": [6, 16], "understand": [0, 2, 3, 5, 7, 8, 12, 15, 16, 17, 23, 24], "understood": [10, 15], "undesir": 10, "undetermin": [7, 10], "undo": 6, "unexpect": 8, "unexpected": 20, "unfair": 8, "unfortun": [3, 10, 11, 12], "unicode_liter": [10, 11], "uniform": [2, 3, 7, 8, 13, 15, 20, 23, 24], "uniformli": [15, 20], "unifrompdf": 20, "unimport": 15, "union": [7, 8], "uniqu": [2, 4, 8, 15, 16, 18, 23], "unique_cluster_label": 16, "unit": [2, 3, 5, 6, 7, 12, 14, 20, 23, 24], "unitari": [7, 8, 18, 24], "unitarili": [18, 23], "uniti": 20, "univari": 20, "univers": [2, 3, 4, 15, 17, 19, 21, 23, 24], "unix": 3, "unknow": [2, 18, 23], "unknown": [2, 3, 5, 6, 7, 8, 10, 12, 15, 18, 23, 24], "unknowwn": 14, "unlabel": 3, "unless": [2, 5, 8, 13, 15, 23], "unlik": [3, 5, 10, 15], "unnecessarili": 11, "unord": 5, "unravel": 3, "unrol": [5, 13], "unseen": [0, 2, 9, 11], "unstabl": 3, "unsupervis": [2, 3, 6, 14, 17, 23], "unsymmetr": [18, 23], "until": [3, 4, 6, 11, 14, 15, 16], "untouch": 2, "unusu": 14, "up": [1, 3, 5, 6, 7, 8, 10, 12, 13, 15, 16, 17, 18, 20, 21], "updat": [0, 3, 4, 12, 14, 15, 16], "uploa": 23, "upload": [0, 17, 22], "upon": [2, 3, 8, 9, 13, 18], "upper": [1, 2, 10, 11, 18, 24], "uppercas": [18, 23], "upsampl": 6, "upscal": 6, "url": [23, 24], "us": [0, 6, 7, 8, 10, 11, 12, 13, 14, 16, 18, 20, 22], "usag": [2, 10, 17, 23, 24], "usd": 2, "usd10000": 2, "use_bia": 6, "usecol": [2, 23], "useless": 3, "user": [0, 2, 3, 4, 6, 8, 9, 17, 18, 23], "usernam": 0, "usetex": 20, "usg": 8, "usr": 20, "usual": [2, 5, 6, 9, 14, 15, 16, 23], "ut": 7, "util": [3, 5, 6, 8, 9, 12, 16, 23], "ux": 18, "v": [0, 2, 4, 6, 7, 8, 13, 15, 17, 24], "v0": 20, "v1": 20, "v2": 20, "v_0": 13, "va": 3, "vahid": 23, "val": 15, "val_accuraci": 5, "val_loss": 6, "vale": 4, "valid": [2, 3, 6, 9, 11, 12, 15, 17, 20, 23, 24], "validation_data": 5, "validation_split": 6, "valu": [1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 23], "valuat": 11, "valy": 6, "van": [2, 23, 24], "vandenbergh": [10, 15], "vandermond": [2, 23], "vanilla": [2, 8, 13, 16, 24], "vanish": [3, 6, 15, 20], "var": [7, 8, 12, 13, 20, 24], "var_x": 20, "varabl": 10, "varepsilon": [7, 8], "varepsilon_": [7, 8], "varepsilon_i": [7, 8], "vari": [2, 3, 5, 7, 8, 12, 23], "variabl": [2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 23, 24], "varianc": [2, 3, 7, 9, 11, 12, 13, 15, 16, 17, 18, 20, 23, 24], "variance_i": [7, 13, 24], "variance_x": [7, 13, 24], "variant": [2, 3, 8, 10, 14, 15, 23, 24], "variat": [5, 6, 13, 23], "varieti": [2, 5, 14, 17, 23], "variou": [1, 3, 5, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23, 24], "varydimens": 6, "vastli": 5, "vaue": 3, "vault": 2, "vdot": [4, 15], "vec": 8, "vector": [2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 17], "vector_mean": 16, "ventur": [2, 10, 17, 23], "venv": 0, "verbos": [3, 5, 6], "veri": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 22, 23, 24], "verifi": [5, 13, 18, 23], "versatil": [10, 23], "versicolor": [10, 11], "version": [0, 2, 5, 12, 15, 16, 17, 18, 20, 23], "versu": 3, "vert": [1, 2, 3, 7, 8, 9, 10, 11, 13, 15, 23, 24], "vert_1": [7, 8, 24], "vert_2": [7, 8, 13, 24], "via": [2, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 20, 21, 23, 24], "vidal": 13, "video": [2, 3, 14, 17, 19, 21, 23], "view": [3, 5, 7, 8, 14, 15, 20, 22, 23], "violat": 10, "virginica": 11, "viridi": [2, 3, 4, 5, 23], "virtual": 3, "viscos": 15, "viscou": 15, "visibl": 0, "vision": [2, 5], "visual": [2, 5, 13, 14, 17, 23, 24], "visualis": 3, "visualstudio": [0, 1], "viz": [8, 10, 20], "vmap": 15, "vmax": [3, 8], "vmin": [3, 8], "voic": 5, "volum": [2, 5, 23], "vote": [12, 23], "voting_clf": 12, "votingclassifi": 12, "votingsimpl": 12, "vstack": [7, 13, 18, 20, 23, 24], "vt": [7, 24], "w": [2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 23, 24], "w1": 10, "w2": [10, 13], "w3": 10, "w_": [3, 14], "w_1": [10, 18], "w_1x_": 10, "w_1x_1": 10, "w_2": [10, 18], "w_2x_": 10, "w_2x_2": 10, "w_3": 18, "w_4": 18, "w_hidden": 4, "w_i": [3, 4, 12], "w_ix_i": 14, "w_j": 18, "w_m": 18, "w_output": 4, "w_px_": 10, "w_px_p": 10, "wa": [2, 3, 5, 6, 7, 8, 9, 12, 13, 14, 16, 18, 23, 24], "wai": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 23, 24], "walk": 11, "walker": 20, "wang": [2, 23], "want": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23, 24], "warn": 6, "warrant": 8, "wast": 5, "watch": 17, "wave": 5, "wavelet": 10, "we": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24], "weak": [11, 12, 16], "weather": [3, 14], "web": [17, 19, 21, 23], "webpag": 23, "websit": [8, 18, 19, 23], "wedg": [10, 20], "wednesdai": [21, 23], "wee": 13, "week": [2, 7, 8, 9, 19, 21], "weekli": [0, 1, 17, 19, 21, 22, 23], "weight": [2, 3, 4, 5, 8, 9, 11, 12, 14, 15, 20], "weigth": 4, "welcom": [0, 10, 17], "well": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18, 20, 22, 23, 24], "went": 10, "were": [2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 20, 23, 24], "wessel": [2, 23, 24], "what": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20], "whatev": 5, "when": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "whenev": [0, 15, 20], "where": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23, 24], "wherea": [8, 20], "wherein": [3, 14], "whether": [2, 5, 7, 9, 11, 20, 23], "which": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23, 24], "whichev": [3, 5], "while": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 20, 23, 24], "white": 11, "who": [0, 2], "whole": [3, 5, 6, 7, 11, 13, 15], "whose": [2, 8, 12, 20, 24], "whow": [13, 24], "why": [0, 1, 2, 3, 5, 8, 15], "wide": [2, 3, 5, 8, 9, 14, 17, 18, 23], "widehat": 8, "width": [2, 5, 10, 11, 23], "wieringen": [2, 23, 24], "win": 12, "wind": 11, "wing": [21, 23], "winther": 4, "wiothout": 8, "wiscons": 9, "wisconsin": 12, "wisdom": 8, "wise": [2, 3, 7, 14, 15, 24], "wish": [2, 4, 7, 9, 10, 13, 15, 16, 18, 23, 24], "with_std": [2, 24], "wither": 8, "within": [2, 4, 5, 6, 9, 11, 14, 15, 16, 20, 22, 23], "withinclust": 16, "without": [0, 2, 3, 7, 8, 10, 11, 13, 14, 15, 23, 24], "won": [0, 2, 23], "wonder": 10, "word": [2, 3, 5, 6, 7, 8, 9, 16, 20, 23, 24], "work": [0, 1, 2, 3, 6, 8, 9, 10, 11, 15, 17, 19, 20, 21, 23, 24], "workshop": 23, "world": [1, 2, 10, 24], "worldwid": [2, 23], "worri": 0, "wors": [2, 3, 5, 6, 8, 23], "worth": 11, "would": [1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "wrap": [8, 18, 23], "write": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 14, 15, 18, 23, 24], "written": [1, 2, 4, 5, 7, 13, 14, 15, 17, 18, 20, 23, 24], "wrong": [0, 3, 10], "wrongli": 12, "wrote": [7, 13, 24], "wrt": [12, 15], "wth": [12, 15], "www": [17, 18, 22, 23], "wx_1": 10, "x": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "x0": 10, "x1": [6, 10, 11, 12, 15], "x1_exampl": 10, "x1d": 10, "x2": [10, 11, 12, 15], "x2d": [10, 13], "x2d_train": 13, "x2dsl": 13, "x3": 10, "x_": [2, 4, 5, 7, 8, 10, 12, 13, 15, 16, 18, 20, 23, 24], "x_0": [2, 7, 13, 18, 23, 24], "x_1": [2, 4, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 23, 24], "x_2": [2, 4, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 23, 24], "x_3": [10, 18, 20], "x_4": 18, "x_center": 13, "x_data": 3, "x_data_ful": 3, "x_hidden": 4, "x_i": [2, 3, 4, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "x_input": 4, "x_ix_": [2, 23], "x_iy_i": 10, "x_j": [1, 2, 4, 10, 11, 14, 20, 24], "x_jy_j": 10, "x_k": [14, 16, 18, 20, 24], "x_l": 20, "x_m": [8, 14, 18, 20], "x_n": [2, 4, 5, 8, 10, 13, 14, 15, 18, 20, 23], "x_new": [11, 12], "x_offset": 8, "x_output": 4, "x_p": [5, 9, 11], "x_poli": 11, "x_poly10": 11, "x_pred": 6, "x_prev": 4, "x_reduc": 13, "x_scale": 10, "x_small": 15, "x_test": [0, 1, 2, 3, 5, 7, 8, 9, 11, 12, 13, 24], "x_test_own": 8, "x_test_scal": [2, 8, 9, 11, 12, 13, 24], "x_tot": 6, "x_train": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 23, 24], "x_train_mean": 8, "x_train_own": 8, "x_train_scal": [2, 8, 9, 11, 12, 13, 24], "x_val": 3, "xarrai": [17, 23], "xavier": 3, "xbnew": 15, "xcode": [2, 17, 23], "xdclassiffierconfus": 12, "xdclassiffierroc": 12, "xg_clf": 12, "xgb": 12, "xgbclassifi": 12, "xgboost": 11, "xgboot": 12, "xgbregressor": 12, "xgparam": 12, "xgtree": 12, "xi": [10, 15], "xi_": 10, "xi_1": 10, "xi_i": 10, "xk": 10, "xla": [15, 17, 23], "xlabel": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 20, 23], "xlim": [8, 12], "xm": 11, "xmesh": 15, "xnew": [2, 15, 23], "xp": 20, "xpanda": [2, 24], "xpd": [7, 13, 24], "xplot": 2, "xscale": [2, 24], "xsr": 11, "xt_x": 15, "xtest": 8, "xtick": [5, 8, 10, 11], "xtrain": 8, "xu": [2, 23], "xx": [2, 18, 23], "xy": [2, 8, 10, 18, 23], "xytext": 10, "xz": [18, 23], "y": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "y1": 6, "y2": 6, "y3": 6, "y_": [2, 3, 7, 8, 12, 13, 18, 23, 24], "y_0": [2, 7, 13, 18, 23, 24], "y_1": [2, 7, 10, 11, 13, 15, 18, 23, 24], "y_1y_1": 10, "y_1y_1k": 10, "y_1y_2": 10, "y_1y_2k": 10, "y_1y_n": 10, "y_1y_nk": 10, "y_2": [2, 7, 10, 11, 13, 18, 23, 24], "y_2y_1": 10, "y_2y_1k": 10, "y_2y_2": 10, "y_2y_2k": 10, "y_3": [2, 11, 18], "y_4": 18, "y_data": [2, 3, 7, 8, 23, 24], "y_data_ful": 3, "y_decis": 10, "y_fit": [2, 24], "y_i": [2, 3, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 23, 24], "y_if_": 12, "y_ix_": [2, 23], "y_ix_i": [9, 10, 15, 24], "y_iy_jk": 10, "y_j": [8, 10, 14], "y_k": 14, "y_m": 18, "y_model": [2, 6, 7, 8, 23, 24], "y_n": [10, 15], "y_ny_1": 10, "y_ny_1k": 10, "y_ny_2": 10, "y_ny_2k": 10, "y_ny_n": 10, "y_ny_nk": 10, "y_offset": 8, "y_plot": 11, "y_pred": [2, 3, 6, 8, 9, 10, 11, 12, 24], "y_pred1": 11, "y_pred2": 11, "y_pred_rf": 12, "y_pred_tre": 12, "y_proba": [9, 12], "y_scaler": 8, "y_test": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 24], "y_test_onehot": 3, "y_test_predict": 2, "y_tot": 6, "y_train": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 23, 24], "y_train_mean": 8, "y_train_onehot": 3, "y_train_predict": 2, "y_train_scal": 8, "y_val": 3, "ye": [5, 8, 9], "year": [2, 17, 23], "yet": [2, 3, 8, 10, 13, 15, 23], "yi": 15, "yield": [2, 4, 7, 8, 10, 12, 14, 15, 16, 18, 20, 23], "yk": 10, "ylabel": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 20, 23], "ylim": [5, 8], "ym": 11, "ymesh": 15, "yn": 2, "yo": [10, 11, 12], "yoshua": [3, 22], "you": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 20, 21, 22, 23, 24], "young": 2, "your": [0, 3, 4, 6, 7, 8, 10, 13, 15, 17, 18, 23], "your_model_object": 1, "yourself": [13, 15, 23], "youtub": 17, "ypred": 8, "ypredict": [2, 15, 23, 24], "ypredict2": 15, "ypredictlasso": 7, "ypredictol": [2, 7], "ypredictown": 8, "ypredictownridg": 8, "ypredictridg": [2, 7, 8], "ypredictskl": 8, "ytest": 8, "ytick": [5, 8, 10, 11], "ytild": [2, 8, 23, 24], "ytildelasso": 7, "ytildenp": [2, 23, 24], "ytildeol": [2, 7], "ytildeownridg": 8, "ytilderidg": [7, 8], "ytrain": 8, "yuxi": 23, "yx": [18, 23], "yy": [18, 23], "yz": [18, 23], "z": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 23, 24], "z_": [3, 4, 14, 18, 23], "z_0": [18, 23], "z_1": [18, 23], "z_2": [18, 23], "z_c": 3, "z_h": 3, "z_hidden": 4, "z_i": [3, 14], "z_j": [3, 14], "z_k": [14, 24], "z_m": 3, "z_mod": 11, "z_o": 3, "z_output": 4, "zaman": 20, "zaxi": 8, "zero": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "zeros_lik": 6, "zeroth": 24, "zfill": 6, "zip": [6, 8], "zm_h": [2, 23], "zn": 2, "zone": 2, "zoom": 23, "zx": [18, 23], "zy": [18, 23], "zz": [18, 23], "\u00f8yvind": 8}, "titles": ["Exercises week 34", "Exercises week 35", "3. Linear Regression", "14. Building a Feed Forward Neural Network", "15. Solving Differential Equations with Deep Learning", "16. Convolutional Neural Networks", "17. Recurrent neural networks: Overarching view", "4. Ridge and Lasso Regression", "5. Resampling Methods", "6. Logistic Regression", "8. Support Vector Machines, overarching aims", "9. Decision trees, overarching aims", "10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods", "11. Basic ideas of the Principal Component Analysis (PCA)", "13. Neural networks", "7. Optimization, the central part of any Machine Learning algortithm", "12. Clustering and Unsupervised Learning", "Applied Data Analysis and Machine Learning", "2. Linear Algebra, Handling of Arrays and more Python Features", "Course setting", "1. Elements of Probability Theory and Statistical Data Analysis", "Teachers and Grading", "Textbooks", "Week 34: Introduction to the course, Logistics and Practicalities", "Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression"], "titleterms": {"": [10, 12], "1": [0, 1, 2, 24], "2": [0, 1, 2, 23, 24], "2023": 21, "3": [0, 1, 2, 24], "34": [0, 23], "35": [1, 24], "4": [0, 1, 2, 24], "5": [1, 2], "A": [2, 3, 6, 10, 11, 23], "And": [23, 24], "In": 21, "Ising": 8, "The": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 17, 23, 24], "To": 23, "With": 6, "about": [23, 24], "activ": [3, 14], "ad": [2, 8, 23, 24], "adaboost": 12, "adagrad": 15, "adam": 15, "adapt": 12, "adjust": 3, "adversari": 6, "again": [5, 11], "ai": 23, "aim": [10, 11, 23], "aka": 23, "algebra": [18, 23], "algorithm": [11, 12, 13, 14, 23, 24], "algortithm": 15, "all": 10, "an": [0, 2, 6, 12, 23], "analys": [7, 24], "analysi": [2, 7, 8, 13, 17, 20, 23, 24], "analyt": [1, 2], "ani": 15, "anoth": 11, "appli": 17, "approach": [2, 10, 16, 23], "approxim": 14, "architectur": 3, "arrai": [18, 23], "assist": 21, "autocorrel": 20, "autograd": [4, 15], "automat": 15, "back": [3, 13, 14], "background": 17, "bag": 12, "base": 15, "basic": [2, 7, 9, 11, 12, 13, 18, 24], "batch": 3, "bay": 7, "befor": 13, "better": 10, "bia": 8, "binari": 3, "bind": 23, "bird": 12, "boldsymbol": 24, "boost": 12, "bootstrap": [8, 12], "boston": 2, "breast": 3, "brief": 23, "bring": 14, "build": [3, 5, 11], "c": 23, "calcul": 24, "can": 23, "cancer": [3, 9, 11, 13], "cart": 11, "case": [10, 12, 20, 24], "central": [15, 17, 20], "chain": 14, "chang": 12, "channel": 23, "chi": [2, 23], "choos": 3, "cifar01": 5, "classic": 13, "classif": [3, 11, 12], "classifi": 10, "clip": 3, "cluster": 16, "cnn": 5, "code": [0, 1, 2, 3, 4, 7, 11, 13, 14, 15, 16, 23, 24], "collect": [3, 5], "commun": 23, "compar": [1, 4, 12], "complet": 24, "complex": [2, 8, 24], "complic": 8, "compon": 13, "comput": 11, "computerlab": 23, "con": 11, "concept": 20, "conjug": 15, "contn": 23, "convex": [10, 15], "convolut": [5, 14], "correl": [13, 24], "cost": [3, 12, 24], "cours": [17, 19, 22, 23], "covari": [7, 13, 20, 24], "cover": 23, "creat": 1, "cross": 8, "cython": 23, "data": [0, 2, 3, 5, 8, 9, 11, 13, 17, 20, 23, 24], "dataset": [3, 5], "david": 23, "deadlin": 23, "deadllin": 21, "decai": 4, "decis": [11, 12], "decomposit": [7, 13, 18, 24], "deeep": [], "deep": [3, 4, 23], "defin": [3, 23], "degre": [2, 24], "deliver": [0, 1], "dens": 2, "deriv": [1, 7, 14, 24], "descent": [4, 12, 15], "design": 24, "detail": [5, 23], "develop": 3, "diagon": 13, "differ": 10, "differenti": [4, 15], "diffus": 4, "dimension": [4, 5, 10], "disadvantag": 11, "discret": 20, "discrimin": 23, "distribut": [7, 20], "do": 3, "doe": 24, "domain": 20, "down": 3, "dropout": 3, "economi": 24, "element": [2, 20, 23], "elimin": 18, "energi": 23, "ensembl": 12, "entropi": 11, "environ": [0, 2], "equat": [2, 4, 14, 24], "error": [2, 12, 23, 24], "essenti": 23, "etc": 23, "euler": 4, "evalu": 3, "exampl": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 23, 24], "exercis": [0, 1, 2, 8, 24], "expect": 20, "experi": 20, "explor": 2, "exponenti": 4, "express": [1, 24], "extrapol": 6, "extrem": [12, 23], "ey": 12, "fall": 21, "famili": [3, 23], "famou": 18, "fantast": 24, "featur": [1, 11, 18, 24], "feed": [3, 14], "final": [14, 24], "find": 1, "fine": 3, "first": [6, 14, 23], "fit": [0, 1, 2, 12, 23], "fix": 24, "forc": 5, "forest": 12, "format": 23, "forward": [3, 4, 14], "foster": 23, "fourier": 5, "frank": [8, 24], "freedom": [2, 24], "frequent": 24, "frequentist": [2, 23], "from": [7, 12, 14, 23, 24], "full": 4, "function": [2, 3, 8, 9, 10, 12, 13, 14, 15, 20, 23, 24], "further": [5, 7, 24], "gan": 6, "gaussian": 18, "gd": 15, "gener": [6, 11, 23], "geometr": 13, "gini": 11, "github": 0, "goal": [0, 1], "good": [2, 23], "grade": [21, 23], "gradient": [3, 4, 12, 15], "growth": 4, "ha": 17, "handl": [18, 23], "hessian": 24, "hidden": 4, "hous": 2, "how": 1, "hyperparamet": 3, "hyperplan": 10, "i": [2, 3, 23], "id3": 11, "idea": 13, "ii": 23, "implement": [1, 3], "implic": [7, 24], "import": [7, 18, 23, 24], "improv": 3, "includ": 15, "increment": 13, "index": 11, "inform": 21, "input": 4, "instal": [17, 23], "instructor": 21, "interpret": [7, 13, 23, 24], "introduc": [13, 15, 24], "introduct": [2, 8, 17, 18, 23], "invers": [7, 18], "invert": 24, "iter": 12, "its": 24, "jacobian": 24, "jax": 15, "julia": 23, "jungl": 12, "kera": [3, 5], "kernel": [10, 13], "lagrangian": 10, "lasso": [7, 8, 24], "last": 24, "later": [7, 24], "layer": [3, 4, 5, 14], "learn": [0, 1, 2, 3, 4, 13, 15, 16, 17, 23, 24], "least": [1, 7, 8, 23, 24], "lectur": 23, "level": 12, "librari": [17, 23], "likelihood": 9, "limit": [3, 15, 20], "linear": [0, 2, 10, 15, 18, 23, 24], "link": [7, 13, 22, 24], "logist": [9, 23], "loss": 24, "lu": 18, "machin": [2, 10, 15, 17, 23], "main": [20, 23], "make": [2, 11, 12, 24], "mani": [12, 14], "mass": 23, "materi": 23, "math": [7, 24], "mathemat": [5, 7, 10, 24], "matric": [7, 18, 23], "matrix": [1, 3, 7, 13, 14, 18, 23, 24], "matter": 2, "max": 24, "mean": [2, 24], "meet": [7, 12, 20, 23, 24], "mercer": 10, "method": [8, 11, 12, 15, 23], "min": 24, "minim": 23, "ml": 23, "mlp": 14, "mnist": [5, 6], "model": [0, 2, 3, 6, 8, 14, 23], "momentum": 15, "moon": [10, 11], "more": [5, 8, 18, 23, 24], "multilay": 14, "multipl": [3, 5], "multipli": 10, "need": 23, "network": [3, 4, 5, 6, 9, 14, 23], "neural": [3, 4, 5, 6, 9, 14, 23], "new": 6, "non": 10, "normal": [2, 3], "notat": 14, "note": 24, "now": [3, 11, 15], "nuclear": [2, 23], "numba": 23, "number": [2, 4, 20, 24], "numer": [4, 20], "numpi": [18, 23], "object": 5, "obtain": 13, "od": 4, "off": 8, "ol": [0, 1, 7, 8], "one": [4, 14], "oper": 18, "optim": [3, 10, 15, 17, 23, 24], "order": 15, "ordinari": [1, 7, 8, 23, 24], "organ": [2, 23], "oslo": 22, "other": [6, 11, 13, 14, 18, 23, 24], "our": [2, 6, 7, 13, 15, 23, 24], "outcom": [17, 23], "output": 4, "overarch": [2, 6, 10, 11, 23, 24], "overview": [12, 23], "own": [2, 12, 13, 23, 24], "packag": [18, 23], "panda": [23, 24], "paramet": [23, 24], "part": [15, 17], "partial": 4, "pass": 3, "pca": 13, "pdf": 20, "perceptron": 14, "perform": [3, 11], "period": 5, "perspect": 3, "plan": 24, "plethora": 23, "point": 6, "poisson": 4, "polynomi": [1, 5], "popul": 4, "popular": 23, "practic": [15, 21, 23], "pre": [3, 5], "predict": 6, "preprocess": 24, "prerequisit": [5, 17, 23], "princip": 13, "principl": 5, "pro": 11, "probabl": [7, 20], "problem": [3, 4, 15, 23, 24], "procedur": [11, 23], "process": [3, 5], "program": [4, 15], "project": [8, 21, 23], "prop": 15, "propag": [3, 14], "properti": [7, 20, 24], "python": [0, 2, 11, 17, 18, 23], "quick": 10, "r": 23, "random": [12, 13, 20], "read": [11, 23, 24], "real": [8, 23], "recommend": [23, 24], "recurr": [6, 14], "reduc": [2, 24], "reduct": 5, "reformul": 4, "regress": [0, 2, 7, 8, 9, 11, 12, 15, 23, 24], "regular": 3, "relat": 24, "relev": [22, 24], "relu": 3, "remark": 5, "remind": [8, 10, 23, 24], "replac": 15, "repositori": 0, "requir": [4, 17], "resampl": 8, "rescal": 8, "residu": 24, "resourc": 4, "result": 24, "revisit": 15, "rewrit": [23, 24], "ridg": [2, 7, 8, 24], "rm": 15, "rule": 14, "same": 15, "sampl": 13, "scale": 24, "schedul": 23, "schemat": 11, "scheme": 4, "scienc": 23, "scikit": [2, 3, 13, 23, 24], "second": 15, "semest": 21, "set": [0, 2, 4, 5, 11, 14, 19, 23, 24], "setup": 0, "sgd": 15, "should": 3, "similar": 15, "simpl": [2, 6, 11, 15, 23, 24], "singl": 12, "singular": [7, 13, 24], "size": 24, "sklearn": 1, "soft": 10, "softmax": 3, "softwar": 23, "solv": 4, "solver": 15, "some": [15, 18, 24], "specifi": 4, "split": [0, 2, 24], "squar": [1, 2, 7, 8, 12, 23, 24], "standard": [15, 24], "state": 2, "statist": [7, 8, 17, 20, 23], "steepest": [12, 15], "stochast": [15, 20], "strongli": 23, "suggest": 23, "summari": [21, 23], "superposit": 5, "supervis": 3, "support": 10, "svd": [7, 24], "systemat": 5, "t": 24, "take": 1, "taken": 23, "teach": 21, "teacher": [21, 23], "techniqu": [8, 13], "technologi": 17, "tensorflow": [3, 5], "tent": [21, 23], "test": [0, 2, 3, 24], "text": 23, "textbook": [22, 23], "theorem": [7, 10, 13, 14, 20], "theori": 20, "thi": 23, "tip": 15, "togeth": 14, "tool": 23, "top": 3, "topic": 23, "toward": 13, "trade": 8, "tradeoff": 8, "train": [0, 2, 3, 6, 23, 24], "transform": 5, "tree": [11, 12], "tune": 3, "two": [5, 10, 17], "type": [4, 6, 14, 23], "uio": 23, "univers": [14, 22], "unsupervis": 16, "up": [0, 2, 4, 11, 14, 23, 24], "us": [1, 2, 3, 4, 5, 9, 15, 17, 23, 24], "v": 5, "valid": 8, "valu": [7, 13, 20, 24], "variabl": 20, "varianc": 8, "variou": 2, "vector": [1, 10, 14, 18, 23, 24], "versu": 23, "view": [2, 6, 12, 24], "virtual": 0, "visual": [3, 11], "wai": 11, "wave": 4, "we": 23, "week": [0, 1, 23, 24], "what": [2, 23, 24], "which": 3, "why": 23, "wisconsin": 9, "write": [6, 13], "x": 24, "xgboost": 12, "your": [1, 2, 12, 24]}}) \ No newline at end of file +Search.setIndex({"alltitles": {"A Classification Tree": [[11, "a-classification-tree"]], "A Frequentist approach to data analysis": [[2, "a-frequentist-approach-to-data-analysis"], [23, "a-frequentist-approach-to-data-analysis"]], "A better approach": [[10, "a-better-approach"]], "A first summary": [[23, "a-first-summary"]], "A quick Reminder on Lagrangian Multipliers": [[10, "a-quick-reminder-on-lagrangian-multipliers"]], "A simple example": [[6, "a-simple-example"]], "A soft classifier": [[10, "a-soft-classifier"]], "A top-down perspective on Neural networks": [[3, "a-top-down-perspective-on-neural-networks"]], "ADAM optimizer": [[15, "adam-optimizer"]], "Activation functions": [[14, "activation-functions"]], "Adaptive boosting: AdaBoost, Basic Algorithm": [[12, "adaptive-boosting-adaboost-basic-algorithm"]], "Adding error analysis and training set up": [[23, "adding-error-analysis-and-training-set-up"], [24, "adding-error-analysis-and-training-set-up"]], "Adjust hyperparameters": [[3, "adjust-hyperparameters"]], "Algorithms for Setting up Decision Trees": [[11, "algorithms-for-setting-up-decision-trees"]], "An Overview of Ensemble Methods": [[12, "an-overview-of-ensemble-methods"]], "An extrapolation example": [[6, "an-extrapolation-example"]], "An optimization/minimization problem": [[23, "an-optimization-minimization-problem"]], "And finally \\boldsymbol{X}\\boldsymbol{X}^T": [[24, "and-finally-boldsymbol-x-boldsymbol-x-t"]], "And what about using neural networks?": [[23, "and-what-about-using-neural-networks"]], "Another example, the moons again": [[11, "another-example-the-moons-again"]], "Applied Data Analysis and Machine Learning": [[17, null]], "Autocorrelation function": [[20, "autocorrelation-function"]], "Automatic differentiation": [[15, "automatic-differentiation"]], "Back to Ridge and LASSO Regression": [[24, "back-to-ridge-and-lasso-regression"]], "Back to the Cancer Data": [[13, "back-to-the-cancer-data"]], "Bagging": [[12, "bagging"]], "Bagging Examples": [[12, "bagging-examples"]], "Basic Matrix Features": [[18, "basic-matrix-features"]], "Basic ideas of the Principal Component Analysis (PCA)": [[13, null]], "Basic math of the SVD": [[7, "basic-math-of-the-svd"], [24, "basic-math-of-the-svd"]], "Basics": [[9, "basics"]], "Basics of a tree": [[11, "basics-of-a-tree"]], "Batch Normalization": [[3, "batch-normalization"]], "Bayes\u2019 Theorem and Ridge and Lasso Regression": [[7, "bayes-theorem-and-ridge-and-lasso-regression"]], "Boosting, a Bird\u2019s Eye View": [[12, "boosting-a-bird-s-eye-view"]], "Bootstrap": [[8, "bootstrap"]], "Bringing it together, first back propagation equation": [[14, "bringing-it-together-first-back-propagation-equation"]], "Building a Feed Forward Neural Network": [[3, null]], "Building a tree, regression": [[11, "building-a-tree-regression"]], "Building neural networks in Tensorflow and Keras": [[3, "building-neural-networks-in-tensorflow-and-keras"]], "CNNs in more detail, building convolutional neural networks in Tensorflow and Keras": [[5, "cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras"]], "Cancer Data again now with Decision Trees and other Methods": [[11, "cancer-data-again-now-with-decision-trees-and-other-methods"]], "Choose cost function and optimizer": [[3, "choose-cost-function-and-optimizer"]], "Classical PCA Theorem": [[13, "classical-pca-theorem"]], "Clustering and Unsupervised Learning": [[16, null]], "Code for SVD and Inversion of Matrices": [[7, "code-for-svd-and-inversion-of-matrices"]], "Codes and Approaches": [[16, "codes-and-approaches"]], "Codes for the SVD": [[7, "codes-for-the-svd"], [24, "codes-for-the-svd"]], "Coding Setup and Linear Regression": [[0, "coding-setup-and-linear-regression"]], "Collect and pre-process data": [[3, "collect-and-pre-process-data"]], "Communication channels": [[23, "communication-channels"]], "Compare Bagging on Trees with Random Forests": [[12, "compare-bagging-on-trees-with-random-forests"]], "Comparing with a numerical scheme": [[4, "comparing-with-a-numerical-scheme"]], "Computing the Gini index": [[11, "computing-the-gini-index"]], "Conjugate gradient method": [[15, "conjugate-gradient-method"]], "Convex functions": [[15, "convex-functions"]], "Convolution Examples: Polynomial multiplication": [[5, "convolution-examples-polynomial-multiplication"]], "Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)": [[5, "convolution-examples-principle-of-superposition-and-periodic-forces-fourier-transforms"]], "Convolutional Neural Network": [[14, "convolutional-neural-network"]], "Convolutional Neural Networks": [[5, null]], "Correlation Function and Design/Feature Matrix": [[24, "correlation-function-and-design-feature-matrix"]], "Correlation Matrix": [[13, "correlation-matrix"], [24, "correlation-matrix"]], "Correlation Matrix with Pandas": [[24, "correlation-matrix-with-pandas"]], "Course Format": [[23, "course-format"]], "Course setting": [[19, null]], "Covariance Matrix Examples": [[24, "covariance-matrix-examples"]], "Covariance and Correlation Matrix": [[24, "covariance-and-correlation-matrix"]], "Cross-validation": [[8, "cross-validation"]], "Deadlines for projects (tentative)": [[23, "deadlines-for-projects-tentative"]], "Decision trees, overarching aims": [[11, null]], "Deep learning methods": [[23, "deep-learning-methods"]], "Define model and architecture": [[3, "define-model-and-architecture"]], "Defining the cost function": [[3, "defining-the-cost-function"]], "Deliverables": [[0, "deliverables"], [1, "deliverables"]], "Derivatives and the chain rule": [[14, "derivatives-and-the-chain-rule"]], "Derivatives, example 1": [[24, "derivatives-example-1"]], "Deriving OLS from a probability distribution": [[7, "deriving-ols-from-a-probability-distribution"]], "Deriving and Implementing Ordinary Least Squares": [[1, "deriving-and-implementing-ordinary-least-squares"]], "Deriving the Lasso Regression Equations": [[24, "deriving-the-lasso-regression-equations"]], "Deriving the Ridge Regression Equations": [[24, "deriving-the-ridge-regression-equations"]], "Deriving the back propagation code for a multilayer perceptron model": [[14, "deriving-the-back-propagation-code-for-a-multilayer-perceptron-model"]], "Developing a code for doing neural networks with back propagation": [[3, "developing-a-code-for-doing-neural-networks-with-back-propagation"]], "Diagonalize the sample covariance matrix to obtain the principal components": [[13, "diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components"]], "Different kernels and Mercer\u2019s theorem": [[10, "different-kernels-and-mercer-s-theorem"]], "Disadvantages": [[11, "disadvantages"]], "Discriminative Modeling": [[23, "discriminative-modeling"]], "Domains and probabilities": [[20, "domains-and-probabilities"]], "Dropout": [[3, "dropout"]], "Economy-size SVD": [[24, "economy-size-svd"]], "Elements of Probability Theory and Statistical Data Analysis": [[20, null]], "Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods": [[12, null]], "Entropy and the ID3 algorithm": [[11, "entropy-and-the-id3-algorithm"]], "Essential elements of ML": [[23, "essential-elements-of-ml"]], "Evaluate model performance on test data": [[3, "evaluate-model-performance-on-test-data"]], "Example 2": [[24, "example-2"]], "Example 3": [[24, "example-3"]], "Example 4": [[24, "example-4"]], "Example Matrix": [[24, "example-matrix"]], "Example of discriminative modeling, taken from Generative Deep Learning by David Foster": [[23, "example-of-discriminative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example of generative modeling, taken from Generative Deep Learning by David Foster": [[23, "example-of-generative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example of own Standard scaling": [[24, "example-of-own-standard-scaling"]], "Example relevant for the exercises": [[24, "example-relevant-for-the-exercises"]], "Example: Exponential decay": [[4, "example-exponential-decay"]], "Example: Population growth": [[4, "example-population-growth"]], "Example: The diffusion equation": [[4, "example-the-diffusion-equation"]], "Example: binary classification problem": [[3, "example-binary-classification-problem"]], "Examples": [[23, "examples"]], "Examples of likelihood functions used in logistic regression and neural networks": [[9, "examples-of-likelihood-functions-used-in-logistic-regression-and-neural-networks"]], "Exercise 1 - Finding the derivative of Matrix-Vector expressions": [[1, "exercise-1-finding-the-derivative-of-matrix-vector-expressions"]], "Exercise 1 - Github Setup": [[0, "exercise-1-github-setup"]], "Exercise 1: Setting up various Python environments": [[2, "exercise-1-setting-up-various-python-environments"]], "Exercise 2 - Deriving the expression for OLS": [[1, "exercise-2-deriving-the-expression-for-ols"]], "Exercise 2 - Setting up a Github repository": [[0, "exercise-2-setting-up-a-github-repository"]], "Exercise 2: making your own data and exploring scikit-learn": [[2, "exercise-2-making-your-own-data-and-exploring-scikit-learn"]], "Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression": [[1, "exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression"]], "Exercise 3 - Fitting an OLS model to data": [[0, "exercise-3-fitting-an-ols-model-to-data"]], "Exercise 3 - Setting up a Python virtual environment": [[0, "exercise-3-setting-up-a-python-virtual-environment"]], "Exercise 3: Normalizing our data": [[2, "exercise-3-normalizing-our-data"]], "Exercise 4 - Fitting a polynomial": [[1, "exercise-4-fitting-a-polynomial"]], "Exercise 4 - The train-test split": [[0, "exercise-4-the-train-test-split"]], "Exercise 4: Adding Ridge Regression": [[2, "exercise-4-adding-ridge-regression"]], "Exercise 5 - Comparing your code with sklearn": [[1, "exercise-5-comparing-your-code-with-sklearn"]], "Exercise 5: Analytical exercises": [[2, "exercise-5-analytical-exercises"]], "Exercise: Cross-validation as resampling techniques, adding more complexity": [[8, "exercise-cross-validation-as-resampling-techniques-adding-more-complexity"]], "Exercise: Analysis of real data": [[8, "exercise-analysis-of-real-data"]], "Exercise: Bias-variance trade-off and resampling techniques": [[8, "exercise-bias-variance-trade-off-and-resampling-techniques"]], "Exercise: Lasso Regression on the Franke function with resampling": [[8, "exercise-lasso-regression-on-the-franke-function-with-resampling"]], "Exercise: Ordinary Least Square (OLS) on the Franke function": [[8, "exercise-ordinary-least-square-ols-on-the-franke-function"]], "Exercise: Ridge Regression on the Franke function with resampling": [[8, "exercise-ridge-regression-on-the-franke-function-with-resampling"]], "Exercises": [[2, "exercises"]], "Exercises and Projects": [[8, "exercises-and-projects"]], "Exercises week 34": [[0, null]], "Exercises week 35": [[1, null]], "Expectation values": [[20, "expectation-values"]], "Extremely useful tools, strongly recommended": [[23, "extremely-useful-tools-strongly-recommended"]], "Feed-forward neural networks": [[14, "feed-forward-neural-networks"]], "Feed-forward pass": [[3, "feed-forward-pass"]], "Final back propagating equation": [[14, "final-back-propagating-equation"]], "Fine-tuning neural network hyperparameters": [[3, "fine-tuning-neural-network-hyperparameters"]], "Fitting an Equation of State for Dense Nuclear Matter": [[2, "fitting-an-equation-of-state-for-dense-nuclear-matter"]], "Fixing the singularity": [[24, "fixing-the-singularity"]], "Frequently used scaling functions": [[24, "frequently-used-scaling-functions"]], "From one to many layers, the universal approximation theorem": [[14, "from-one-to-many-layers-the-universal-approximation-theorem"]], "Functionality in Scikit-Learn": [[24, "functionality-in-scikit-learn"]], "Further Dimensionality Remarks": [[5, "further-dimensionality-remarks"]], "Further properties (important for our analyses later)": [[7, "further-properties-important-for-our-analyses-later"], [24, "further-properties-important-for-our-analyses-later"]], "Gaussian Elimination": [[18, "gaussian-elimination"]], "General Features": [[11, "general-features"]], "General linear models and linear algebra": [[23, "general-linear-models-and-linear-algebra"]], "Generalizing the fitting procedure as a linear algebra problem": [[23, "generalizing-the-fitting-procedure-as-a-linear-algebra-problem"], [23, "id1"]], "Generative Adversarial Networks": [[6, "generative-adversarial-networks"]], "Generative Models": [[6, "generative-models"]], "Generative Versus Discriminative Modeling": [[23, "generative-versus-discriminative-modeling"]], "Geometric Interpretation and link with Singular Value Decomposition": [[13, "geometric-interpretation-and-link-with-singular-value-decomposition"]], "Gradient Boosting, Classification Example": [[12, "gradient-boosting-classification-example"]], "Gradient Boosting, Examples of Regression": [[12, "gradient-boosting-examples-of-regression"]], "Gradient Clipping": [[3, "gradient-clipping"]], "Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent": [[12, "gradient-boosting-basics-with-steepest-descent-functional-gradient-descent"]], "Gradient descent": [[4, "gradient-descent"]], "Grading": [[21, "grading"], [21, "id2"], [23, "grading"]], "Housing data, the code": [[2, "housing-data-the-code"]], "How to take derivatives of Matrix-Vector expressions": [[1, "how-to-take-derivatives-of-matrix-vector-expressions"]], "Hyperplanes and all that": [[10, "hyperplanes-and-all-that"]], "Important Matrix and vector handling packages": [[18, "important-matrix-and-vector-handling-packages"]], "Improving performance": [[3, "improving-performance"]], "In summary": [[21, "in-summary"]], "Including Stochastic Gradient Descent with Autograd": [[15, "including-stochastic-gradient-descent-with-autograd"]], "Incremental PCA": [[13, "incremental-pca"]], "Installing R, C++, cython or Julia": [[23, "installing-r-c-cython-or-julia"]], "Installing R, C++, cython, Numba etc": [[23, "installing-r-c-cython-numba-etc"]], "Instructor information": [[21, "instructor-information"]], "Interpretations and optimizing our parameters": [[23, "interpretations-and-optimizing-our-parameters"], [23, "id2"], [23, "id3"], [24, "interpretations-and-optimizing-our-parameters"], [24, "id1"], [24, "id2"]], "Interpreting the Ridge results": [[24, "interpreting-the-ridge-results"]], "Introducing JAX": [[15, "introducing-jax"]], "Introducing the Covariance and Correlation functions": [[13, "introducing-the-covariance-and-correlation-functions"], [24, "introducing-the-covariance-and-correlation-functions"]], "Introduction": [[2, "introduction"], [8, "introduction"], [17, "introduction"], [18, "introduction"]], "Iterative Fitting, Classification and AdaBoost": [[12, "iterative-fitting-classification-and-adaboost"]], "Iterative Fitting, Regression and Squared-error Cost Function": [[12, "iterative-fitting-regression-and-squared-error-cost-function"]], "Kernel PCA": [[13, "kernel-pca"]], "Kernels and non-linearity": [[10, "kernels-and-non-linearity"]], "LU Decomposition, the inverse of a matrix": [[18, "lu-decomposition-the-inverse-of-a-matrix"]], "Layers": [[3, "layers"]], "Layers used to build CNNs": [[5, "layers-used-to-build-cnns"]], "Learning goals": [[0, "learning-goals"], [1, "learning-goals"]], "Learning outcomes": [[17, "learning-outcomes"], [23, "learning-outcomes"]], "Lectures and ComputerLab": [[23, "lectures-and-computerlab"]], "Limitations of supervised learning with deep networks": [[3, "limitations-of-supervised-learning-with-deep-networks"]], "Linear Algebra, Handling of Arrays and more Python Features": [[18, null]], "Linear Regression": [[2, null]], "Linear Regression Problems": [[24, "linear-regression-problems"]], "Linear Regression, basic elements": [[2, "linear-regression-basic-elements"]], "Linking Bayes\u2019 Theorem with Ridge and Lasso Regression": [[7, "linking-bayes-theorem-with-ridge-and-lasso-regression"]], "Linking the regression analysis with a statistical interpretation": [[7, "linking-the-regression-analysis-with-a-statistical-interpretation"]], "Linking with the SVD": [[7, "linking-with-the-svd"], [24, "linking-with-the-svd"]], "Links to relevant courses at the University of Oslo": [[22, "links-to-relevant-courses-at-the-university-of-oslo"]], "Logistic Regression": [[9, null], [9, "id1"]], "MNIST and GANs": [[6, "mnist-and-gans"]], "Machine Learning": [[23, "machine-learning"]], "Machine learning": [[17, "machine-learning"]], "Main textbooks": [[23, "main-textbooks"]], "Making a tree": [[11, "making-a-tree"]], "Making your own Bootstrap: Changing the Level of the Decision Tree": [[12, "making-your-own-bootstrap-changing-the-level-of-the-decision-tree"]], "Making your own test-train splitting": [[24, "making-your-own-test-train-splitting"]], "Mathematical Interpretation of Ordinary Least Squares": [[7, "mathematical-interpretation-of-ordinary-least-squares"], [24, "mathematical-interpretation-of-ordinary-least-squares"]], "Mathematical optimization of convex functions": [[10, "mathematical-optimization-of-convex-functions"]], "Mathematics of CNNs": [[5, "mathematics-of-cnns"]], "Mathematics of the SVD and implications": [[7, "mathematics-of-the-svd-and-implications"], [24, "mathematics-of-the-svd-and-implications"]], "Matrices in Python": [[23, "matrices-in-python"]], "Matrix multiplication": [[3, "matrix-multiplication"]], "Matrix-vector notation and activation": [[14, "matrix-vector-notation-and-activation"]], "Meet the covariance!": [[20, "meet-the-covariance"]], "Meet the Covariance Matrix": [[7, "meet-the-covariance-matrix"], [24, "meet-the-covariance-matrix"]], "Meet the Hessian Matrix": [[24, "meet-the-hessian-matrix"]], "Meet the Pandas": [[23, "meet-the-pandas"]], "Min-Max Scaling": [[24, "min-max-scaling"]], "Momentum based GD": [[15, "momentum-based-gd"]], "More complicated Example: The Ising model": [[8, "more-complicated-example-the-ising-model"]], "More interpretations": [[24, "more-interpretations"]], "More on Dimensionalities": [[5, "more-on-dimensionalities"]], "More on Rescaling data": [[8, "more-on-rescaling-data"]], "More preprocessing": [[24, "more-preprocessing"]], "Multilayer perceptrons": [[14, "multilayer-perceptrons"]], "Network requirements": [[4, "network-requirements"]], "Neural Networks vs CNNs": [[5, "neural-networks-vs-cnns"]], "Neural networks": [[14, null]], "Note about SVD Calculations": [[24, "note-about-svd-calculations"]], "Numerical experiments and the covariance, central limit theorem": [[20, "numerical-experiments-and-the-covariance-central-limit-theorem"]], "Numpy and arrays": [[18, "numpy-and-arrays"], [23, "numpy-and-arrays"]], "Numpy examples and Important Matrix and vector handling packages": [[23, "numpy-examples-and-important-matrix-and-vector-handling-packages"]], "Optimization, the central part of any Machine Learning algortithm": [[15, null]], "Optimizing our parameters": [[23, "optimizing-our-parameters"]], "Optimizing our parameters, more details": [[23, "optimizing-our-parameters-more-details"]], "Optimizing the cost function": [[3, "optimizing-the-cost-function"]], "Organizing our data": [[2, "organizing-our-data"], [23, "organizing-our-data"]], "Other Matrix and Vector Operations": [[18, "other-matrix-and-vector-operations"]], "Other Types of Recurrent Neural Networks": [[6, "other-types-of-recurrent-neural-networks"]], "Other courses on Data science and Machine Learning at UiO": [[23, "other-courses-on-data-science-and-machine-learning-at-uio"]], "Other courses on Data science and Machine Learning at UiO, contn": [[23, "other-courses-on-data-science-and-machine-learning-at-uio-contn"]], "Other popular texts": [[23, "other-popular-texts"]], "Other techniques": [[13, "other-techniques"]], "Other types of networks": [[14, "other-types-of-networks"]], "Other ways of visualizing the trees": [[11, "other-ways-of-visualizing-the-trees"]], "Our model for the nuclear binding energies": [[23, "our-model-for-the-nuclear-binding-energies"]], "Overview of first week": [[23, "overview-of-first-week"]], "Own code for Ordinary Least Squares": [[23, "own-code-for-ordinary-least-squares"], [24, "own-code-for-ordinary-least-squares"]], "PCA and scikit-learn": [[13, "pca-and-scikit-learn"]], "Pandas AI": [[23, "pandas-ai"]], "Partial Differential Equations": [[4, "partial-differential-equations"]], "Plans for week 35": [[24, "plans-for-week-35"]], "Practical tips": [[15, "practical-tips"]], "Practicalities": [[21, "practicalities"], [21, "id1"]], "Predicting New Points With A Trained Recurrent Neural Network": [[6, "predicting-new-points-with-a-trained-recurrent-neural-network"]], "Preprocessing our data": [[24, "preprocessing-our-data"]], "Prerequisites": [[23, "prerequisites"]], "Prerequisites and background": [[17, "prerequisites-and-background"]], "Prerequisites: Collect and pre-process data": [[5, "prerequisites-collect-and-pre-process-data"]], "Probability Distribution Functions": [[20, "probability-distribution-functions"]], "Program for stochastic gradient": [[15, "program-for-stochastic-gradient"]], "Properties of PDFs": [[20, "properties-of-pdfs"]], "Pros and cons of trees, pros": [[11, "pros-and-cons-of-trees-pros"]], "Python installers": [[17, "python-installers"], [23, "python-installers"]], "RMS prop": [[15, "rms-prop"]], "Random Numbers": [[20, "random-numbers"]], "Random forests": [[12, "random-forests"]], "Randomized PCA": [[13, "randomized-pca"]], "Reading material": [[23, "reading-material"]], "Reading recommendations:": [[24, "reading-recommendations"]], "Reading suggestions week 34": [[23, "reading-suggestions-week-34"]], "Recurrent neural networks": [[14, "recurrent-neural-networks"]], "Recurrent neural networks: Overarching view": [[6, null]], "Reducing the number of degrees of freedom, overarching view": [[2, "reducing-the-number-of-degrees-of-freedom-overarching-view"], [24, "reducing-the-number-of-degrees-of-freedom-overarching-view"]], "Reformulating the problem": [[4, "reformulating-the-problem"]], "Regression Case": [[12, "regression-case"]], "Regression analysis, overarching aims": [[23, "regression-analysis-overarching-aims"]], "Regression analysis, overarching aims II": [[23, "regression-analysis-overarching-aims-ii"]], "Regularization": [[3, "regularization"]], "Reminder from last week": [[24, "reminder-from-last-week"]], "Reminder on Statistics": [[8, "reminder-on-statistics"]], "Replace or not": [[15, "replace-or-not"]], "Required Technologies": [[17, "required-technologies"]], "Resampling Methods": [[8, null]], "Resampling methods": [[8, "id1"]], "Residual Error": [[24, "residual-error"]], "Resources on differential equations and deep learning": [[4, "resources-on-differential-equations-and-deep-learning"]], "Revisiting our Linear Regression Solvers": [[15, "revisiting-our-linear-regression-solvers"]], "Rewriting the Covariance and/or Correlation Matrix": [[24, "rewriting-the-covariance-and-or-correlation-matrix"]], "Rewriting the fitting procedure as a linear algebra problem": [[23, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem"]], "Rewriting the fitting procedure as a linear algebra problem, more details": [[23, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem-more-details"]], "Ridge and LASSO Regression": [[24, "ridge-and-lasso-regression"]], "Ridge and Lasso Regression": [[7, null], [7, "id1"]], "Same code but now with momentum gradient descent": [[15, "same-code-but-now-with-momentum-gradient-descent"]], "Schedule first week": [[23, "schedule-first-week"]], "Schematic Regression Procedure": [[11, "schematic-regression-procedure"]], "Setting up the Back propagation algorithm": [[14, "setting-up-the-back-propagation-algorithm"]], "Setting up the Matrix to be inverted": [[24, "setting-up-the-matrix-to-be-inverted"]], "Setting up the network using Autograd; The full program": [[4, "setting-up-the-network-using-autograd-the-full-program"]], "Similar (second order function now) problem but now with AdaGrad": [[15, "similar-second-order-function-now-problem-but-now-with-adagrad"]], "Simple Python Code to read in Data and perform Classification": [[11, "simple-python-code-to-read-in-data-and-perform-classification"]], "Simple case": [[24, "simple-case"]], "Simple linear regression model using scikit-learn": [[2, "simple-linear-regression-model-using-scikit-learn"], [23, "simple-linear-regression-model-using-scikit-learn"]], "Software and needed installations": [[23, "software-and-needed-installations"]], "Solving Differential Equations with Deep Learning": [[4, null]], "Solving the one dimensional Poisson equation": [[4, "solving-the-one-dimensional-poisson-equation"]], "Solving the wave equation with Neural Networks": [[4, "solving-the-wave-equation-with-neural-networks"]], "Some famous Matrices": [[18, "some-famous-matrices"]], "Some simple problems": [[15, "some-simple-problems"]], "Some useful matrix and vector expressions": [[24, "some-useful-matrix-and-vector-expressions"]], "Splitting our Data in Training and Test data": [[2, "splitting-our-data-in-training-and-test-data"], [24, "splitting-our-data-in-training-and-test-data"]], "Standard steepest descent": [[15, "standard-steepest-descent"]], "Statistical analysis and optimization of data": [[17, "statistical-analysis-and-optimization-of-data"], [23, "statistical-analysis-and-optimization-of-data"]], "Steepest descent": [[15, "steepest-descent"]], "Stochastic Gradient Descent (SGD)": [[15, "stochastic-gradient-descent-sgd"]], "Stochastic variables and the main concepts, the discrete case": [[20, "stochastic-variables-and-the-main-concepts-the-discrete-case"]], "Support Vector Machines, overarching aims": [[10, null]], "Systematic reduction": [[5, "systematic-reduction"]], "Teachers": [[23, "teachers"]], "Teachers and Grading": [[21, null]], "Teaching Assistants Fall semester 2023": [[21, "teaching-assistants-fall-semester-2023"]], "Tentative deadllines for projects": [[21, "tentative-deadllines-for-projects"]], "Testing the Means Squared Error as function of Complexity": [[2, "testing-the-means-squared-error-as-function-of-complexity"], [24, "testing-the-means-squared-error-as-function-of-complexity"]], "Textbooks": [[22, null]], "The Algorithm before theorem": [[13, "the-algorithm-before-theorem"]], "The Boston housing data example": [[2, "the-boston-housing-data-example"]], "The Breast Cancer Data, now with Keras": [[3, "the-breast-cancer-data-now-with-keras"]], "The CART algorithm for Classification": [[11, "the-cart-algorithm-for-classification"]], "The CART algorithm for Regression": [[11, "the-cart-algorithm-for-regression"]], "The CIFAR01 data set": [[5, "the-cifar01-data-set"]], "The Jacobian": [[24, "the-jacobian"]], "The MNIST dataset again": [[5, "the-mnist-dataset-again"]], "The RELU function family": [[3, "the-relu-function-family"]], "The SVD, a Fantastic Algorithm": [[24, "the-svd-a-fantastic-algorithm"]], "The Softmax function": [[3, "the-softmax-function"]], "The \\chi^2 function": [[2, "the-chi-2-function"], [23, "the-chi-2-function"], [23, "id4"], [23, "id5"], [23, "id6"], [23, "id7"], [23, "id8"]], "The bias-variance tradeoff": [[8, "the-bias-variance-tradeoff"]], "The code for solving the ODE": [[4, "the-code-for-solving-the-ode"]], "The complete code with a simple data set": [[24, "the-complete-code-with-a-simple-data-set"]], "The cost/loss function": [[24, "the-cost-loss-function"]], "The course has two central parts": [[17, "the-course-has-two-central-parts"]], "The equations for ordinary least squares": [[24, "the-equations-for-ordinary-least-squares"]], "The logistic function": [[9, "the-logistic-function"]], "The mean squared error and its derivative": [[24, "the-mean-squared-error-and-its-derivative"]], "The moons example": [[10, "the-moons-example"]], "The multilayer perceptron (MLP)": [[14, "the-multilayer-perceptron-mlp"]], "The network with one input layer, specified number of hidden layers, and one output layer": [[4, "the-network-with-one-input-layer-specified-number-of-hidden-layers-and-one-output-layer"]], "The plethora of machine learning algorithms/methods": [[23, "the-plethora-of-machine-learning-algorithms-methods"]], "The singular value decomposition": [[7, "the-singular-value-decomposition"], [24, "the-singular-value-decomposition"]], "The two-dimensional case": [[10, "the-two-dimensional-case"]], "To our real data: nuclear binding energies. Brief reminder on masses and binding energies": [[23, "to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies"]], "Topics covered in this course: Statistical analysis and optimization of data": [[23, "topics-covered-in-this-course-statistical-analysis-and-optimization-of-data"]], "Towards the PCA theorem": [[13, "towards-the-pca-theorem"]], "Train and test datasets": [[3, "train-and-test-datasets"]], "Two-dimensional Objects": [[5, "two-dimensional-objects"]], "Type of problem": [[4, "type-of-problem"]], "Types of Machine Learning": [[23, "types-of-machine-learning"]], "Useful Python libraries": [[17, "useful-python-libraries"], [23, "useful-python-libraries"]], "Using Autograd": [[15, "using-autograd"]], "Using forward Euler to solve the ODE": [[4, "using-forward-euler-to-solve-the-ode"]], "Using gradient descent methods, limitations": [[15, "using-gradient-descent-methods-limitations"]], "Visualization": [[3, "visualization"], [3, "id1"]], "Visualizing the Tree, Classification": [[11, "visualizing-the-tree-classification"]], "Week 34: Introduction to the course, Logistics and Practicalities": [[23, null]], "Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression": [[24, null]], "What Is Generative Modeling?": [[23, "what-is-generative-modeling"]], "What does it mean?": [[24, "what-does-it-mean"]], "What is Machine Learning?": [[2, "what-is-machine-learning"]], "What is a good model?": [[2, "what-is-a-good-model"], [23, "what-is-a-good-model"]], "What is a good model? Can we define it?": [[23, "what-is-a-good-model-can-we-define-it"]], "Which activation function should I use?": [[3, "which-activation-function-should-i-use"]], "Why Linear Regression (aka Ordinary Least Squares and family)": [[23, "why-linear-regression-aka-ordinary-least-squares-and-family"]], "Wisconsin Cancer Data": [[9, "wisconsin-cancer-data"]], "Writing Our First Generative Adversarial Network": [[6, "writing-our-first-generative-adversarial-network"]], "Writing our own PCA code": [[13, "writing-our-own-pca-code"]], "XGBoost: Extreme Gradient Boosting": [[12, "xgboost-extreme-gradient-boosting"]], "scikit-learn implementation": [[3, "scikit-learn-implementation"]]}, "docnames": ["E1", "E2", "chapter1", "chapter10", "chapter11", "chapter12", "chapter13", "chapter2", "chapter3", "chapter4", "chapter5", "chapter6", "chapter7", "chapter8", "chapter9", "chapteroptimization", "clustering", "intro", "linalg", "schedule", "statistics", "teachers", "textbooks", "week34", "week35"], "envversion": {"sphinx": 62, "sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1}, "filenames": ["E1.ipynb", "E2.ipynb", "chapter1.ipynb", "chapter10.ipynb", "chapter11.ipynb", "chapter12.ipynb", "chapter13.ipynb", "chapter2.ipynb", "chapter3.ipynb", "chapter4.ipynb", "chapter5.ipynb", "chapter6.ipynb", "chapter7.ipynb", "chapter8.ipynb", "chapter9.ipynb", "chapteroptimization.ipynb", "clustering.ipynb", "intro.md", "linalg.ipynb", "schedule.md", "statistics.ipynb", "teachers.md", "textbooks.md", "week34.ipynb", "week35.ipynb"], "indexentries": {}, "objects": {}, "objnames": {}, "objtypes": {}, "terms": {"": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 15, 17, 18, 20, 21, 23, 24], "0": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23, 24], "00": [2, 3, 7, 13, 23, 24], "000": [3, 5], "00000000e": [], "001": [4, 10, 15], "004": 7, "00727646693": [2, 23], "0086649156": [2, 23], "01": [2, 3, 4, 7, 11, 13, 15, 22, 23, 24], "0110": 20, "01719003e": [], "02": [2, 6, 9, 14, 23], "02334824": [], "02857": 6, "02f": 8, "03077640549": 6, "03097597e": [], "031": 7, "04": 13, "0458": 11, "05": [6, 8], "062292565": 6, "062435": [], "06730814": [], "07": [], "0713": [2, 23], "07285": 5, "08": 20, "08078025e": [], "08336233266": 6, "0917": 11, "0n": [2, 23], "1": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23], "10": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 19, 20, 21, 23, 24], "100": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 18, 20, 21, 23, 24], "1000": [2, 3, 4, 6, 7, 10, 13, 15, 16, 17, 20, 23], "10000": [4, 7, 8, 12, 13, 15, 20], "100000": 10, "10001": 12, "1001": 20, "1002": 20, "1003": 20, "1005": 20, "1009": 20, "101": 1, "1011": 20, "1013": 20, "1013904243": 20, "1015": 20, "102": 1, "1023": 20, "1024": 5, "1026": 20, "1027": 20, "103": 3, "1030": 20, "1037": 20, "1038": 20, "1040": 20, "1047": 20, "107": 1, "108": [], "10th": 11, "10x": [2, 23], "11": [1, 2, 4, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 22, 23, 24], "110": [], "1100": 20, "1101": 20, "111": [3, 9, 14], "112": 1, "11340253": [], "11590451": [], "116": 1, "117": 1, "118": 1, "12": [2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 18, 20, 22, 23, 24], "120": 5, "121": [1, 10, 11, 12], "1215pm": [21, 23], "122": [10, 11, 12], "124": [2, 23], "125": 1, "127": [1, 6], "128": [5, 6, 15], "129": 1, "1298": 11, "12pm": [21, 23], "13": [2, 4, 11, 14, 18, 20, 23], "131": 1, "133": 9, "135": 1, "136": 1, "14": [2, 4, 6, 8, 10, 11, 12, 14, 18, 20, 22, 24], "141": 1, "143": 1, "1446729567": 6, "149": 1, "14g": 8, "15": [2, 4, 6, 8, 9, 10, 11, 14, 15, 20, 23], "150": [6, 10], "152": 1, "153760": [], "156": 1, "157": [], "158": [], "159": 1, "15g": 8, "15pm": 23, "16": [3, 4, 5, 6, 7, 10, 11, 12, 20, 23], "160": 1, "1603": 5, "161": 1, "162": 1, "16231451": 6, "163": 1, "16384": 5, "164": 1, "167": 1, "17": [3, 4, 10, 20], "172": 1, "173": 1, "176": 1, "178": 1, "179": 1, "1797": 3, "18": [4, 8, 9, 10, 11, 12, 20, 23], "1807": 6, "18392847": [], "19": [4, 20, 23], "1940": 2, "1943": 14, "1970": [18, 23], "1973": 11, "1979": 8, "1_1": 14, "1_2": 14, "1_3": 14, "1cm": [2, 10, 12, 20, 23], "1d": [3, 4, 5], "1e": [4, 6, 15, 16], "1e10": 16, "1e4": 8, "1f": 3, "1k": 18, "1n": [2, 23], "1x": [2, 23], "2": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22], "20": [1, 2, 3, 4, 8, 9, 10, 20, 21, 23, 24], "200": [2, 4, 5, 6, 10, 11, 12], "2000": [2, 24], "2004": 15, "2006": 22, "20072279": [], "2008": 23, "2010": 3, "2011": 3, "2014": 6, "2015": 3, "2016": [2, 23], "2018": [2, 8, 24], "2021": [8, 16], "2022": 23, "2025": [23, 24], "21": [2, 3, 7, 9, 11, 14, 18, 23, 24], "2116753732": 6, "215pm": [21, 23], "2167072": [], "22": [2, 3, 7, 14, 15, 18, 23, 24], "221": 10, "225": 6, "22948497": [], "23": [3, 14, 18], "24": [2, 3, 18, 23], "25": [4, 5, 6, 7, 8, 10, 11, 13, 24], "250": [4, 6, 9, 11], "25000": 2, "250154": [], "253775": [], "255": 5, "256": 6, "26": [], "26303845": [], "264": [], "265": [], "265109911": 6, "266": [], "269": [], "27": [2, 3], "270": [], "27n_": 20, "28": [3, 5, 6], "2830637392": 6, "2861": 20, "2873": 11, "2882": 20, "2886": 20, "2890": [2, 23], "2892": 20, "29": 24, "2915": 20, "2931": 23, "29364655": [], "294399745619595": [], "296247": [], "2968": 23, "2980": 23, "298273": [], "298375": [], "2990": 23, "2_": 14, "2_1": 14, "2_2": 14, "2_3": 14, "2_i": 14, "2_m": [8, 20], "2_t": 15, "2_x": 20, "2b": 20, "2cm": 10, "2d": [3, 5, 13, 14, 17, 23], "2e": 8, "2f": [2, 9, 11, 12, 13, 14, 23], "2g": 4, "2g_i": 4, "2k": 5, "2m": 8, "2n": [2, 4, 5, 23, 24], "2nd": 11, "2p": 20, "2pt": 6, "2x": [2, 5, 10, 15, 23], "2x_ix_jy_iy_j": 10, "2x_j": 10, "2y_i": 12, "2y_j": 10, "3": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23], "30": [2, 3, 6, 8, 9, 12, 15, 21], "30000": [2, 23], "3072": 5, "31": [14, 18, 20], "315": 8, "3155": [2, 7, 8, 24], "32": [5, 6, 8, 14, 15, 18, 20], "3200": 3, "3250": 3, "3297": [], "33": [14, 18, 21], "3303": [], "3310": [], "332331": [], "333": 9, "3331": [], "3337": [], "34": 18, "3436": [2, 23], "3437": [2, 23], "35": [2, 8, 23], "3581341341": 6, "359": 7, "36": [2, 7, 8, 20], "370782966": 6, "38": 20, "39": [2, 21, 23], "3d": [1, 4, 5, 6, 8, 15], "3f": [3, 5, 11], "3n": 18, "3x": [4, 10], "3x_i": 4, "3y": 10, "4": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "40": [3, 8, 21, 23], "400": 6, "4000": 23, "4050": [22, 23], "41": 18, "4155": [0, 4], "41589548": [], "42": [3, 6, 10, 11, 12, 18], "43": [2, 9, 18], "4310": 23, "436462435": 6, "44": [2, 18], "45": [21, 23], "46": [21, 23], "462": 9, "47": [21, 23], "479465113": 6, "47958494": [], "48": [], "48257387": [21, 23], "49": [7, 8, 13], "49152": 5, "4940954": [2, 23], "4990": 20, "4992": 20, "4997": 20, "4c4c7f": [11, 12], "4d": 5, "4f": 8, "4pm": [21, 23], "4y": 10, "4y_i": 12, "5": [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "50": [3, 4, 5, 6, 8, 9, 10, 12, 15, 23, 24], "500": [3, 5, 6, 8, 11, 12, 15], "5018": 20, "506": 2, "507d50": [11, 12], "50j": 15, "50x10": 3, "51": 12, "510": 3, "512132": [], "5177783846": 6, "53": 11, "54": [8, 20], "5411205": [], "54894451": [], "55": 3, "56": 3, "56536": [2, 23], "569": 3, "57": [2, 10, 21, 23], "571": 7, "58": [12, 21, 23], "591317992": 6, "5cm": 20, "5f": 10, "5x": 10, "5y": 10, "6": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 21, 23, 24], "60": [3, 5], "60000": 6, "6019067271": 6, "606439": [], "625": 9, "63": [2, 3], "64": [3, 5, 6, 15, 18, 23], "64x50": 3, "65": [3, 10, 11], "6887363571": 6, "69": [1, 20], "69069n_": 20, "691": [], "6n_": 20, "7": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 22, 23], "70": [3, 9], "70653767": 6, "71": 3, "724": 5, "73": [], "7304881": [], "75": [7, 8, 10, 13], "76": [21, 23], "765": 9, "77": [21, 23], "7718": 11, "7782028952": 6, "77893972": [], "78": [], "7d7d58": [11, 12], "8": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 21, 23], "80": [2, 3, 7, 10, 24], "800": [6, 9], "81": 3, "815am": [21, 23], "85": 3, "8702784034": 6, "88": 23, "8f": 8, "8g": 8, "8n": 18, "8x8": 3, "9": [2, 3, 4, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 23], "90": 3, "9040": 11, "91": [21, 23], "92": [21, 23], "93": 1, "931": [2, 23], "933": 7, "937": 20, "938": 20, "939": [2, 20, 23], "94": 20, "95": [3, 13], "954": 20, "955820c21e8b": 6, "96": 8, "960": 20, "961": 20, "962": 20, "9649652536": 6, "96611194e": [], "9780387310732": 22, "9780387848570": 22, "9781098134174": 23, "9781492032632": 22, "9781801819312": 23, "98": [1, 2, 3], "985": 20, "986": 20, "989": 20, "9898ff": [11, 12], "99": [1, 15], "991": 20, "992": 20, "993": 20, "996": 7, "999": [11, 20], "9x": 8, "9y": 8, "A": [0, 1, 4, 5, 7, 8, 9, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 24], "AND": 4, "And": [2, 5, 6, 7, 8, 11, 15, 17, 20], "As": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 15, 18, 20, 23, 24], "At": [2, 6, 8, 15, 23], "BE": [2, 23], "Be": [4, 17, 23], "Being": 15, "But": [1, 2, 3, 4, 5, 7, 8, 11, 12, 20, 24], "By": [2, 5, 7, 8, 14, 15, 18, 23, 24], "For": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23, 24], "IF": 8, "IN": 22, "If": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "In": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23, 24], "Ising": [7, 14, 24], "It": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "Its": [3, 4, 6, 13], "No": [8, 11, 23], "Not": [2, 3, 7, 8, 24], "OR": 20, "Of": 20, "On": [2, 5, 20, 21, 22, 23], "One": [2, 3, 5, 6, 7, 8, 9, 10, 13, 14, 15, 20, 24], "Or": [2, 3, 8, 23], "Such": [1, 2, 8, 14, 20], "That": [2, 7, 9, 12, 13, 14, 16, 20, 23], "The": [1, 6, 12, 15, 16, 18, 19, 20, 21, 22], "Then": [0, 1, 2, 3, 8, 10, 11, 12, 13, 14, 15, 16, 18, 23], "There": [0, 2, 5, 6, 7, 8, 10, 11, 13, 14, 16, 18, 20, 21, 23, 24], "These": [2, 5, 6, 7, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23, 24], "To": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 24], "With": [1, 2, 7, 8, 10, 11, 12, 13, 14, 16, 18, 20, 23, 24], "_": [1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 23, 24], "_0": [7, 10, 12, 13, 15, 24], "_1": [4, 7, 8, 10, 12, 13, 14, 15, 16, 18, 24], "_2": [4, 7, 10, 13, 14, 15, 18, 24], "_3": 18, "_4": 18, "_9": 15, "__class__": 12, "__doc__": 8, "__future__": [10, 11], "__init__": 3, "__main__": 4, "__name__": [4, 12], "_auto1": [4, 5, 6, 7, 8, 9, 14, 15, 18, 20, 24], "_auto10": [8, 14], "_auto11": 8, "_auto12": 8, "_auto2": [4, 5, 6, 7, 8, 14, 15, 18, 20], "_auto3": [5, 6, 7, 8, 14, 15, 18], "_auto4": [6, 8, 14, 15, 18], "_auto5": [6, 8, 14, 15, 18], "_auto6": [6, 8, 14, 18], "_auto7": [6, 8, 14, 18], "_auto8": [8, 14], "_auto9": [8, 14], "_build": [2, 17, 22, 23], "_c": 3, "_compon": 13, "_depth": 11, "_export": [0, 1], "_fraction": 11, "_i": [2, 3, 4, 7, 8, 9, 10, 13, 14, 15, 23, 24], "_j": [2, 3, 4, 5, 7, 8, 10, 15, 24], "_k": 15, "_l": 14, "_lambda": 8, "_leaf": 11, "_m": 12, "_multilayer_perceptron": [], "_n": [4, 7, 10, 13, 15, 24], "_node": 11, "_p": [7, 10, 24], "_ratio": 13, "_sampl": 11, "_split": [8, 11], "_t": 15, "_test": 8, "_varianc": 13, "_weight": 11, "a0": 5, "a0faa0": [11, 12], "a1": [2, 23], "a2": [2, 23], "a3": [2, 23], "a4": [2, 23], "a_": [1, 2, 3, 18, 23, 24], "a_0": [2, 23], "a_1a": [2, 23], "a_2a": [2, 23], "a_3": [2, 23], "a_3a": [2, 23], "a_4": [2, 23], "a_4a": [2, 23], "a_h": 3, "a_i": [2, 3, 4, 14, 23], "a_j": [3, 14], "a_k": [2, 3, 14], "aaron": 22, "ab": [2, 4, 7, 15, 16, 23, 24], "ab_channel": 17, "abandon": 3, "abid": 20, "abil": [2, 12], "abl": [1, 2, 3, 6, 7, 8, 9, 12, 14, 15, 24], "about": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 21], "abov": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 22, 23, 24], "abovement": [8, 23], "abscissa": 15, "absolut": [2, 4, 7, 8, 15, 23, 24], "absorb": 24, "abstract": 3, "acceler": 15, "accept": [2, 5, 8, 11, 24], "access": [2, 5, 13, 20, 23], "accid": [6, 8], "accompani": [2, 23, 24], "accomplish": [10, 11, 15], "accord": [2, 3, 4, 7, 8, 11, 14, 15, 16, 20, 23], "accordingli": 13, "account": [0, 1, 2, 5, 7, 15, 20, 23], "accumul": [14, 15, 20], "accur": [2, 5, 6, 8, 12, 15], "accuraci": [2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 23, 24], "accuracy_scor": [2, 3, 12, 23], "accuracy_score_numpi": 3, "achiev": [2, 3, 7, 8, 10, 14, 18, 23], "aco": 20, "acquaint": 17, "acquir": [3, 17, 23], "acr": 2, "across": [3, 5, 8, 11, 17, 23], "act": [3, 5, 18], "action": 20, "activ": [0, 2, 4, 5, 6, 11, 19, 21, 23], "actual": [0, 1, 2, 3, 6, 7, 8, 10, 13, 18, 20, 23, 24], "ad": [0, 1, 3, 5, 6, 7, 10, 15, 18], "ada_clf": 12, "adaboostclassifi": 12, "adadelta": 15, "adam": [3, 5, 6, 23], "adapt": [6, 8, 15, 22], "add": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 20, 21, 23, 24], "add_subplot": [3, 9, 14, 16], "addendum": 7, "addit": [0, 2, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18, 20, 21, 22, 23, 24], "addition": [14, 15], "address": [3, 11, 13, 15, 23], "adjac": [5, 14], "adjoint": [7, 24], "adjust": [2, 7, 14, 15], "admir": [2, 23], "advanc": [6, 8, 14, 22, 23], "advantag": [3, 5, 7, 8, 12, 15, 18], "adversari": 23, "afecionado": 23, "affect": [0, 5], "affin": [2, 5, 10, 13, 24], "afford": 5, "aficionado": 23, "aforement": 16, "african": 2, "after": [0, 1, 2, 3, 4, 6, 7, 8, 11, 13, 14, 15, 17, 18, 20, 23, 24], "afterward": [2, 23], "ag": [2, 9, 23, 24], "ag_0": 4, "again": [2, 3, 6, 7, 8, 9, 10, 12, 13, 14, 15, 20, 23, 24], "against": [3, 6, 9, 12], "agegroup": 9, "agegroupmean": 9, "aggreg": [11, 12], "agorithm": 12, "agre": [7, 8, 20, 24], "agreement": 15, "ahead": 11, "ai": [2, 22], "aid": 13, "aim": [1, 2, 3, 6, 8, 9, 13, 16, 17, 18, 24], "ainv": 7, "airplan": 5, "aka": 7, "al": [1, 2, 4, 6, 22, 23, 24], "alarm": [7, 9], "aldo": 24, "algebra": [2, 5, 7, 15, 17, 24], "algorithm": [1, 2, 3, 4, 6, 7, 8, 9, 10, 15, 16, 17, 18, 20, 22], "align": [2, 4, 7, 8, 9, 10, 15, 20, 23, 24], "all": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24], "allevi": [3, 15], "alloc": [5, 18], "allow": [0, 2, 3, 4, 5, 7, 8, 10, 12, 15, 17, 18, 23, 24], "almost": [2, 3, 8, 10, 13, 15, 20], "alon": [4, 11], "along": [0, 4, 5, 6, 7, 8, 11, 12, 13, 17, 18, 23, 24], "alpha": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 15, 16, 20, 23, 24], "alpha_": 12, "alpha_0": 5, "alpha_1": 5, "alpha_2": 5, "alpha_i": [5, 15], "alpha_k": 15, "alpha_m": 12, "alpha_n": 5, "alpha_opt": 15, "alreadi": [0, 4, 5, 6, 7, 8, 12, 14, 17, 18, 20, 23, 24], "also": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "alter": 3, "altern": [0, 2, 3, 6, 7, 8, 10, 11, 13, 15, 18, 23, 24], "although": [1, 2, 3, 7, 8, 10, 12, 15, 23], "alwai": [1, 2, 5, 7, 8, 14, 15, 20, 23, 24], "am": 6, "ame2016": [2, 23], "american": 2, "among": [2, 5, 7, 11, 12, 14, 18, 23, 24], "amongst": 7, "amount": [2, 3, 5, 6, 8, 10, 12, 16, 17], "an": [1, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24], "an_": 20, "anaconda": [2, 3, 17, 23], "analogi": 15, "analys": 8, "analysi": [3, 5, 6, 9, 16, 18, 22], "analyt": [4, 5, 7, 8, 9, 14, 15, 17, 23, 24], "analyz": [1, 2, 3, 5, 6, 7, 8, 20, 24], "andrew": 3, "angl": [2, 5, 11, 24], "anharmon": 5, "ani": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 20, 23, 24], "anim": [6, 14], "ann": 14, "annot": [2, 3, 5, 9, 10, 23], "announc": 23, "anoth": [0, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 23, 24], "ansatz": [2, 23], "answer": [2, 3, 5, 7, 8, 18, 21, 23], "antialias": [4, 8], "anticip": 6, "anymor": [3, 10], "anyon": [0, 6, 10], "anyth": [0, 1, 3, 20], "anytim": [21, 23], "apach": 3, "apart": [13, 15], "api": [3, 17, 23], "appar": 4, "appear": [2, 3, 5, 15, 18, 20], "append": [3, 5, 6, 10, 11, 15, 23], "appli": [2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 20, 22, 23, 24], "applic": [1, 2, 3, 5, 6, 7, 8, 9, 11, 14, 15, 18, 20, 22, 23, 24], "apply_gradi": 6, "approach": [0, 1, 3, 4, 6, 7, 8, 11, 12, 13, 14, 15, 17, 20, 22, 24], "appropri": [4, 8, 11, 14, 15, 17, 20], "approv": 23, "approx": [2, 4, 5, 8, 12, 13, 15, 20, 23], "approxim": [2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 15, 20, 23, 24], "apt": [2, 17, 23], "aq": 20, "ar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24], "aragorn": 23, "arang": [3, 5, 6, 8, 9, 11, 12, 14, 15, 23], "arbitrari": [3, 6, 8, 10, 14, 15, 20], "arbitrarili": [2, 3, 13, 23], "arc": 8, "architectur": [5, 6, 14], "area": [2, 5, 8, 22, 23], "argmax": [3, 13], "argmin": [6, 12, 16], "argsort": 13, "argu": [3, 15], "argument": [2, 4, 5, 7, 13, 14, 15, 23, 24], "aris": [2, 8, 14, 15, 20, 23], "arithmet": [2, 15, 18, 23], "arm": 8, "armadillo": 18, "around": [2, 3, 6, 7, 8, 13, 20, 23], "arrai": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 15, 16, 17, 20, 24], "arrang": [5, 23], "arraybox": 15, "arriv": [2, 8, 11, 13, 18, 20, 23], "arrow": 14, "arrowprop": 10, "art": [2, 3, 17], "articl": [2, 5, 6, 8, 12, 23, 24], "artifici": [2, 4, 9, 14, 22, 23], "artificialneuron": 14, "arug": 15, "arxiv": [5, 6], "asarrai": [2, 8, 11], "asid": 24, "ask": [0, 7, 8, 13, 14], "aspect": [2, 8, 17, 23], "assembl": 5, "assembli": [2, 23], "assert": 6, "assess": [2, 8, 23], "assici": 6, "assign": [0, 2, 9, 10, 11, 14, 15, 16, 19, 21, 22, 23], "associ": [2, 8, 11, 14, 16, 20, 23], "assum": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "assumpt": [2, 5, 7, 8, 11, 13, 20, 23], "ast": [2, 7, 8, 23], "astyp": [6, 11, 12], "asymmetri": [2, 23], "asymptot": [6, 8], "atom": [2, 23], "attempt": [2, 6, 8, 9, 10, 12, 23], "attend": 23, "attent": [2, 18, 23], "attract": [2, 12, 23], "attribut": [2, 11, 23], "audi": [2, 23], "audio": [5, 6], "august": [23, 24], "aurelien": [2, 22, 23], "austfjel": 8, "auth": 0, "authent": 0, "author": [2, 3, 12, 20], "authour": 23, "auto": [11, 12, 20], "autocor": 20, "autocorrelation_tim": 20, "autocorrelform": 20, "autocovari": 20, "autoencod": [6, 17, 23], "autoencond": 17, "autograd": [17, 23], "autom": [2, 17, 22, 23], "automac": 18, "automag": 23, "automat": [1, 2, 3, 4, 5, 6, 13, 17, 18, 23], "automobil": 5, "autonom": 6, "avail": [2, 3, 6, 8, 12, 13, 17, 18, 19, 21, 22, 23], "averag": [2, 3, 5, 8, 11, 12, 15, 16, 20, 21, 23], "avoid": [2, 6, 7, 8, 11, 13, 15, 18, 24], "awai": [4, 5, 8, 24], "awar": [4, 12], "award": [21, 23], "ax": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 23], "axes3d": [4, 8, 15], "axes_grid1": 8, "axhlin": 10, "axi": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "axiom": 7, "axvlin": [6, 10], "axvspan": 6, "b": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 15, 16, 20, 21, 23, 24], "b1": 10, "b2": 10, "b3": 10, "b_": [2, 3, 18], "b_0": 2, "b_1": [2, 4, 14, 15], "b_2": [2, 15], "b_5": 15, "b_group": 11, "b_i": [2, 3, 4, 14, 23], "b_ia_": [2, 23], "b_ia_i": 2, "b_index": 11, "b_j": [3, 14], "b_k": [2, 3, 14, 15], "b_m": 14, "b_score": 11, "b_valu": 11, "babcock": 23, "bachelor": [19, 21], "back": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 18, 20, 23], "backbon": 18, "backend": [3, 6], "background": [22, 23], "backpropag": 3, "backtrack": 11, "backup": 18, "backward": [3, 4, 6, 14, 18], "bad": 8, "badli": 20, "bag": [11, 17, 23], "bag_clf": 12, "baggin": 23, "baggingboot": 12, "baggingclassifi": 12, "baggingtre": 12, "balanc": 8, "band": 18, "bandwidth": 18, "bar": [2, 8, 13, 23], "barber": 22, "bare": [6, 12], "base": [0, 1, 2, 3, 5, 6, 7, 9, 10, 11, 12, 16, 17, 20, 21, 22, 23, 24], "basi": [7, 9, 10, 12, 13, 14, 15, 18, 24], "basic": [0, 8, 10, 14, 15, 16, 17, 20, 23], "batch": [5, 6, 13, 14, 15], "batch_shap": 6, "batch_siz": [3, 5, 6], "batchnorm": 6, "bay": 9, "bayesian": [7, 17, 22, 23], "becaus": [2, 3, 4, 5, 6, 7, 8, 10, 11, 14, 15, 16, 23], "becom": [2, 3, 4, 7, 8, 9, 11, 14, 15, 20, 23, 24], "been": [2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 17, 18, 23], "befor": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 18, 20, 23, 24], "beforehand": [2, 20, 23], "begin": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 20, 21, 23, 24], "behav": [3, 8, 15], "behavior": [2, 3, 15, 23], "behaviour": 14, "behind": [2, 3, 8, 10, 15, 23], "being": [2, 3, 4, 5, 6, 7, 9, 10, 12, 13, 14, 15, 20, 23, 24], "believ": [11, 18], "belong": [9, 10, 11, 15, 16], "below": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "benchmark": 12, "benefici": [3, 15], "benefit": [2, 3, 6, 13, 15, 17, 23], "bengio": [3, 22, 23, 24], "benign": [3, 9], "besid": [6, 7], "bessel": [7, 24], "best": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 21, 23, 24], "beta": [1, 2, 3, 5, 12, 13, 15, 23], "beta_": [5, 15], "beta_0": [3, 5, 15], "beta_1": [3, 5, 12, 15], "beta_1x_i": 15, "beta_2": [5, 15], "beta_3": 5, "beta_i": 5, "beta_j": 15, "beta_k": 15, "beta_linreg": 15, "beta_m": 12, "beta_mg_m": 12, "beta_n": 5, "better": [2, 3, 4, 5, 6, 8, 11, 12, 13, 14, 15, 23, 24], "between": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 20, 23, 24], "beyond": [2, 3, 7, 8, 10, 15, 23, 24], "bf": [15, 16, 18, 20], "bg": 23, "bgd": 15, "bia": [2, 3, 4, 5, 7, 10, 11, 12, 14, 15, 23, 24], "bias": [3, 4, 5, 7, 8, 11, 14], "big": [2, 3, 4, 7, 8, 16], "bigger": [3, 8], "bigr": 14, "bike": 11, "bilbo": 23, "billion": [5, 14, 17], "bin": [2, 9, 20], "binari": [2, 5, 7, 9, 11, 12, 14, 23], "binarycrossentropi": 6, "bind": 2, "binomi": [17, 20, 23], "binsboot": 8, "bioinformat": 2, "biolog": [3, 14], "bios1100": [17, 23], "bird": [2, 5], "birth": 23, "bishop": [22, 23], "bit": [3, 6, 18, 20, 23], "bitwis": 20, "bivari": 4, "bk": [2, 15], "bla": [18, 23], "black": [10, 11, 16], "block": [8, 12, 17, 18, 20, 23], "blog": 23, "blogpost": 6, "blue": [2, 5], "bmatrix": [2, 3, 5, 7, 9, 10, 13, 15, 18, 23, 24], "bmi": 3, "bodi": [2, 3, 6, 14], "bold": 3, "boldfac": [1, 2, 7, 24], "boldsymbol": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 15, 16, 23], "boltzmann": [14, 17, 23], "book": [22, 23, 24], "book1": 22, "boolean": 6, "boost": [3, 11, 17, 23], "boostrap": 12, "bootstrap": [3, 15, 17, 23], "borrow": 23, "boston_dataset": 2, "bot": 10, "both": [0, 1, 2, 3, 6, 7, 8, 10, 11, 12, 15, 16, 17, 18, 20, 21, 23, 24], "bottl": 9, "bound": [2, 10, 14], "boundari": [4, 6, 10, 13, 14], "box": [6, 11], "boyd": [10, 15], "bracket": [6, 20], "brain": [3, 9, 14], "branch": [11, 23], "break": [2, 6, 8, 13, 16, 23], "breast": [7, 9, 13], "breviti": 15, "brew": [2, 17, 23], "brg": 10, "brief": 24, "briefli": [1, 2, 23], "bring": [2, 7, 8, 12], "britt": [21, 23], "broad": 2, "broadli": 23, "brought": [15, 17, 23], "brownle": 6, "browser": [0, 23], "brute": [5, 7, 13, 24], "buffer_s": 6, "bui": 6, "build": [1, 2, 6, 7, 8, 12, 18, 20, 23], "built": [2, 3, 5, 6, 8], "bunch": 13, "busi": 2, "byte": [18, 23], "c": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24], "c1": [10, 13], "c2": [10, 13], "c_": [2, 10, 11, 12, 15, 20], "c_0": 20, "c_1": 14, "c_2": 14, "c_3": 14, "c_4": 14, "c_i": [14, 15], "c_k": 20, "ca": [3, 23], "cach": 12, "cal": [2, 10, 12, 14, 15], "calcul": [1, 2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "call": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23, 24], "calor": [2, 24], "cambridg": [15, 22], "can": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24], "cancel": [2, 15, 23, 24], "cancer": [7, 12], "cancerpd": 9, "candid": [10, 11, 12], "cannot": [2, 3, 6, 7, 8, 9, 10, 11, 20, 24], "canopi": [2, 17, 23], "canva": [0, 1, 23], "cap": 7, "capabl": [2, 3, 10, 15, 17, 23], "capac": [4, 21], "capita": 2, "captur": [6, 13, 14, 23], "car": [5, 6], "card": [2, 9, 23], "cardin": 3, "care": [0, 13], "carefulli": 15, "carlo": [2, 8, 17, 20, 22, 23], "carri": [4, 8, 9], "cart": 12, "case": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15, 16, 17, 18, 23], "casella": 22, "cast": 3, "cat": [5, 6], "catch": 2, "categor": [2, 3, 5, 11, 13, 23], "categori": [2, 3, 5, 9, 12, 14, 16, 23], "categorical_crossentropi": [3, 5], "caus": [2, 7, 8, 20, 23, 24], "causal": 2, "causat": [2, 23], "cax": 3, "cb": [8, 23], "cbar": 3, "cc": [2, 3, 7, 15, 23, 24], "ccc": [7, 14], "cdf": 20, "cdot": [2, 4, 8, 14, 15, 16, 18, 20, 23], "celebr": 15, "cell": 6, "center": [2, 3, 8, 9, 10, 11, 13, 16, 20, 23], "central": [1, 2, 5, 7, 8, 10, 18, 23, 24], "centroid": [16, 20], "centroid_differ": 16, "centuri": 5, "certain": [2, 5, 8, 9, 11, 20, 23, 24], "cg": 15, "cha": 2, "chain": [2, 3, 15, 17, 20, 23], "challeng": 0, "chanc": [3, 7, 15, 20], "chang": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 15, 16, 18, 20, 23, 24], "channel": 5, "chapter": [1, 2, 8, 12, 13, 18, 22, 23, 24], "chapter3": 2, "charact": [2, 5, 7, 23, 24], "character": [10, 11, 12, 14, 20], "characterist": [2, 3, 5, 12, 15, 23], "charg": [2, 23], "charl": 2, "chase": 6, "chatgpt": 0, "chd": 9, "chddata": 9, "cheap": [7, 24], "cheaper": [3, 15], "check": [0, 1, 2, 3, 5, 6, 7, 13, 15, 18, 23], "checkmark": 5, "checkpoint": 6, "checkpoint_dir": 6, "checkpoint_prefix": 6, "chen": 12, "cheng": 24, "chiaramont": 4, "childcar": 1, "children": 1, "choic": [2, 3, 4, 5, 6, 8, 11, 14, 15, 16, 18, 23, 24], "choleski": [7, 18, 24], "choos": [0, 4, 5, 8, 11, 12, 13, 15, 16], "chosen": [1, 2, 3, 4, 8, 10, 11, 12, 15, 20, 23], "chosen_datapoint": 3, "christian": 22, "christoph": [22, 23], "cifar": 5, "cifar10": 5, "circ": [3, 14], "circl": [2, 10, 14, 24], "circuit": 5, "circumfer": 11, "circumv": [3, 7, 15, 24], "ckpt": 6, "clariti": 20, "class": [2, 3, 5, 6, 8, 9, 10, 11, 13, 14, 15, 20, 23], "class_nam": [5, 11], "class_val": 11, "class_valu": 11, "classic": [9, 11, 15], "classif": [2, 5, 7, 8, 9, 10, 13, 14, 17, 22, 23, 24], "classifi": [2, 3, 6, 9, 11, 12, 13, 23], "classificaton": 3, "classifii": 12, "clean": 3, "clear": [3, 7, 12, 14, 15], "clearli": [2, 5, 7, 8, 9, 10, 20], "clever": [3, 12], "clf": [2, 8, 10, 11, 12, 23, 24], "clf3": 2, "clf_lasso": 8, "clf_ridg": 8, "cli": 0, "clip": [5, 20], "clone": [0, 21], "close": [2, 3, 4, 6, 8, 10, 11, 13, 14, 15, 16, 20, 22, 23], "closer": [5, 7, 15, 24], "closest": [10, 13, 15, 16], "closur": [17, 23], "cloud": [17, 23], "cluster": [2, 3, 6, 8, 13, 17, 23], "cluster_label": 16, "cm": [3, 4, 5, 8, 10, 15], "cmap": [2, 3, 4, 5, 6, 8, 10, 11, 12, 23], "cmap_arg": 8, "cmd": [0, 11], "cn_": 20, "cnn": 14, "cnn_kera": 5, "cntk": [17, 23], "co": [2, 4, 5, 8, 11, 15, 23], "code": [5, 6, 8, 9, 10, 17, 18, 20, 22], "coef": [2, 23], "coef0": 10, "coef_": [1, 2, 7, 8, 10, 11, 15, 23, 24], "coeff": 7, "coeffici": [2, 5, 7, 8, 9, 10, 11, 15, 18, 23], "coerc": [2, 8, 23], "coin": [12, 20], "coin_toss": 12, "col": [2, 13, 23, 24], "colab": [17, 23], "cold": 11, "colinear": 2, "collaps": 10, "collect": [4, 8, 12, 13, 17, 20, 22, 23], "collinear": [7, 24], "color": [2, 5, 6, 8, 10, 11, 12, 20], "color_channel": 5, "color_cod": 8, "colorbar": [3, 8], "colsample_bytre": 12, "colsaobject": 12, "column": [1, 2, 3, 4, 7, 8, 9, 10, 11, 13, 14, 18, 23, 24], "columntransform": 11, "com": [0, 1, 6, 8, 17, 22, 23], "combin": [0, 3, 4, 7, 8, 9, 12, 20], "come": [0, 2, 3, 5, 6, 7, 14, 15, 16, 23, 24], "command": [0, 2, 3], "comment": [2, 6, 7, 8], "commerci": [2, 17, 23], "commit": 0, "commod": [2, 23], "common": [1, 2, 3, 5, 7, 8, 9, 11, 13, 15, 16, 20, 23, 24], "commonli": [2, 3, 6, 8, 9, 11, 15, 16, 24], "commun": [0, 2, 14], "commut": 5, "commutatitav": 5, "compact": [2, 3, 5, 7, 8, 9, 11, 13, 14, 15, 16, 23, 24], "compair": 2, "compar": [2, 5, 6, 7, 8, 13, 15, 18, 23, 24], "comparison": [4, 6, 15], "compat": 9, "compet": 2, "competit": 12, "compil": [2, 3, 5, 6, 15, 17, 18, 23], "complet": [0, 1, 2, 4, 5, 6, 11, 14, 23], "completenn": 14, "complex": [1, 3, 7, 10, 11, 13, 14, 15, 23], "complic": [2, 3, 11, 15, 23], "compoment": 24, "compon": [1, 2, 3, 5, 6, 7, 8, 9, 11, 16, 17, 23, 24], "components_": 13, "compos": [11, 14, 15, 16, 17, 23], "compphys": [1, 2, 8, 17, 19, 21, 22, 23], "compress": [2, 23, 24], "compris": 8, "compromis": [7, 24], "compulsori": [17, 23], "comput": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 18, 19, 20, 22, 23, 24], "computation": [2, 5, 8, 11, 15, 20, 23], "computationalscienceuio": 23, "concaten": [4, 6, 8, 16], "concav": [3, 15, 24], "concentr": [2, 12], "concept": [2, 4, 17, 23, 24], "conceptu": [14, 15], "concern": [2, 3, 6, 9, 23], "concic": 23, "conclud": [2, 7, 15], "conclus": 3, "cond": 4, "conda": [2, 3, 17, 23], "condis": 24, "condit": [2, 4, 6, 7, 8, 10, 11, 13, 15, 20, 23, 24], "conduct": 17, "condwav": 4, "confid": [2, 7, 8, 9, 10, 23, 24], "configur": 5, "confirm": [7, 14], "confus": [7, 8, 9, 12, 18], "confusion_matrix": 11, "congruenti": 20, "conjug": [6, 10], "conjugaci": 15, "conjunct": 5, "connect": [2, 3, 5, 6, 11, 13, 14, 15, 18, 23, 24], "consequ": [7, 8, 10, 12, 14, 15, 24], "conserv": [7, 16, 24], "consid": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 23, 24], "consider": [2, 3, 7, 15, 23, 24], "consist": [2, 3, 4, 5, 6, 8, 14, 15, 20, 24], "constant": [1, 2, 4, 6, 7, 8, 10, 14, 15, 20, 23, 24], "constitu": [2, 23], "constitut": [4, 8], "constrain": [3, 5, 7, 9, 13], "constraint": [7, 8, 10, 15, 24], "construct": [2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 18, 20, 23, 24], "contact": [2, 23], "contain": [0, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 22, 23, 24], "contemporari": 23, "content": [0, 3, 17, 18, 23], "context": [8, 12, 15], "contigu": 18, "continu": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 23, 24], "contour": [11, 12, 15], "contourf": [10, 11, 12], "contrast": [3, 6, 11, 12, 14, 23], "contribut": [2, 5, 7, 15, 20, 23, 24], "contributor": 2, "control": [0, 2, 3, 5, 11, 15, 17, 23], "conv": [5, 6], "conv2d": [5, 6], "conv2dtranspos": 6, "convei": 23, "conveni": [7, 8, 14, 15, 18, 23], "convent": [14, 24], "converg": [3, 4, 6, 7, 10, 15, 16, 24], "convergencewarn": [], "convert": [2, 3, 6, 7, 11, 13, 15, 18, 23, 24], "converttomatrix": 6, "convex": [6, 7, 9, 24], "convinc": 15, "convolut": [3, 6, 17, 23], "cool": [6, 11], "coolwarm": 8, "coordin": [7, 14, 16, 24], "coorel": 2, "copi": [0, 2, 3, 16, 24], "core": 12, "corel": 23, "coronari": 9, "corr": [2, 7, 9, 13, 24], "correalt": [13, 17], "correct": [0, 2, 3, 4, 5, 6, 7, 9, 15, 18, 20, 23, 24], "correctli": [3, 4, 8, 9, 12], "correl": [2, 3, 5, 7, 8, 9, 12, 14, 15, 17, 20, 23], "correlation_matrix": [2, 7, 9, 13, 24], "correspond": [2, 5, 7, 8, 10, 11, 13, 14, 17, 18, 20, 23, 24], "cortex": 14, "cosin": [5, 8], "cost": [1, 2, 4, 5, 7, 8, 9, 10, 11, 14, 15, 23], "cost_deep_grad": 4, "cost_funct": 4, "cost_function_deep": 4, "cost_function_deep_grad": 4, "cost_function_grad": 4, "cost_grad": 4, "cost_sum": 4, "costol": 15, "could": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "coulomb": [2, 23], "count": [0, 2, 11, 19, 20, 21, 23], "counterpart": 23, "countor": 15, "coupl": [6, 7, 8], "cours": [0, 1, 2, 3, 5, 7, 13, 21, 24], "coursework": 0, "courvil": [22, 23, 24], "cov": [7, 8, 13, 18, 20, 23, 24], "cov_xi": [7, 13, 24], "cov_xx": [7, 13, 24], "cov_yi": [7, 13, 24], "covari": [2, 9, 17, 18, 23], "covariance_matrix": [7, 13, 16], "cover": [2, 7, 17, 21, 22, 24], "covert": [2, 23], "covxi": 20, "covxx": 20, "covxz": 20, "covyi": 20, "covyz": 20, "covzz": 20, "cpu": 3, "craft": 5, "creat": [0, 3, 5, 6, 7, 11, 12, 13, 14, 17, 23], "create_biases_and_weight": 3, "create_convolutional_neural_network_kera": 5, "create_neural_network_kera": 3, "create_x": [7, 13], "credit": [2, 9, 21, 23], "crim": 2, "crime": 2, "criteria": [2, 6, 11, 12, 16, 20, 23], "criterion": [11, 12, 15], "critic": 8, "cross": [0, 2, 3, 5, 9, 11, 12, 15, 17, 20, 23, 24], "cross_entropi": 6, "cross_val_scor": 8, "cross_valid": [9, 12], "crossvalid": 8, "crucial": [3, 20], "cs231": 5, "csr_matrix": [18, 23], "csv": [2, 6, 8, 9, 11], "ctnk": 3, "cubic": 2, "cumbersom": 7, "cumsum": [12, 13, 23], "cumul": [9, 12, 20], "cumulative_heads_ratio": 12, "cup": 7, "current": [0, 1, 3, 4, 5, 6, 15, 16, 22], "curs": [2, 24], "curv": [8, 9, 12, 14], "curvatur": 15, "custom": [8, 16], "custom_cmap": [11, 12], "custom_cmap2": [11, 12], "cutpoint": 11, "cv": [8, 9, 12], "cvxbook": 15, "cvxopt": [7, 10, 24], "cycl": [3, 14], "d": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 23, 24], "d2_g_t": 4, "d_f": 15, "d_g_t": 4, "d_net_out": 4, "da": 5, "dagger": [7, 18, 24], "dai": [3, 11, 17], "damp": 5, "darget": 11, "darkr": 20, "dat": [2, 23], "dat_id": [2, 8, 9, 11, 23], "data": [1, 4, 6, 7, 10, 12, 14, 15, 16, 18, 22], "data1": 16, "data2": 16, "data3": 16, "data4": 16, "data_id": [2, 8, 9, 11, 23], "data_indic": 3, "data_panda": 23, "data_path": [2, 8, 9, 11, 23], "databas": 3, "datafil": [2, 8, 9, 11, 23], "datafram": [2, 6, 7, 9, 11, 13, 23, 24], "datapoint": [1, 3, 7, 8, 9, 13, 15], "datasci": [0, 1], "dataset": [1, 2, 6, 8, 9, 10, 11, 12, 13, 15, 16, 23], "datatyp": 6, "date": [0, 23, 24], "daughter": 12, "david": 22, "dbh": 3, "dbo": 3, "dcomposit": 18, "ddot": 4, "dead": 3, "deadlin": 0, "deal": [2, 3, 5, 7, 8, 10, 13, 15, 16, 18, 20, 23, 24], "dealt": 2, "debt": 9, "debug": [2, 7, 8, 24], "decad": [2, 5], "decai": [2, 15, 20, 23], "decemb": [21, 23], "decent": 12, "decid": [2, 4, 5, 7, 8, 11, 24], "decim": [2, 23], "decis": [2, 3, 10, 13, 17, 22, 23], "decision_funct": 10, "decision_tre": 11, "decisiontreeclassifi": [11, 12], "decisiontreeregressor": [2, 11, 12], "declar": [2, 6, 18, 23], "decompos": [7, 8, 18, 24], "decomposit": [2, 8, 14, 23], "decompost": [7, 24], "deconvolut": 5, "decorrel": [12, 15], "decreas": [3, 4, 6, 7, 8, 12, 13, 15], "deduc": [2, 23], "deep": [5, 9, 14, 15, 17, 22, 24], "deep_neural_network": 4, "deep_param": 4, "deep_tree_clf": [11, 12], "deep_tree_clf1": 11, "deep_tree_clf2": 11, "deepen": [7, 17, 23], "deeper": [2, 5, 6, 23], "deeplearningbook": [22, 23], "deer": 5, "def": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 20, 23, 24], "def_covari": 20, "default": [2, 3, 4, 6, 8, 9, 18, 23], "default_tim": 6, "defect": [7, 24], "defici": [7, 24], "defin": [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 24], "definit": [3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 24], "defint": 20, "degre": [0, 1, 5, 7, 8, 10, 11, 12, 13, 20, 23], "deisenroth": 24, "del": 3, "delet": [0, 8], "delimit": 6, "deliv": [0, 19, 23], "delta": [2, 4, 5, 8, 10, 14, 15, 16, 23], "delta_": [3, 18], "delta_0": 5, "delta_1": 5, "delta_2": 5, "delta_3": 5, "delta_4": 5, "delta_5": 5, "delta_h": [2, 3, 23], "delta_j": [5, 14], "delta_k": 14, "delta_l": [3, 5], "delta_momentum": 15, "delta_n": [2, 5, 23], "delug": 17, "delv": 2, "demand": 15, "demonstr": [2, 5, 7, 8, 9, 13, 14, 17, 23, 24], "den": 6, "denomin": [3, 7], "denot": [3, 4, 8, 9, 15, 20], "dens": [3, 5, 6], "densiti": [2, 4, 8, 20], "depart": [21, 23, 24], "depend": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 20, 23, 24], "depict": 20, "deploy": [2, 17, 23], "depth": [2, 5, 11, 12, 18], "deriv": [2, 3, 4, 8, 9, 10, 12, 13, 15, 17, 23], "derivati": 15, "derivative_fn": 15, "descend": [7, 11, 13, 24], "descent": [2, 3, 5, 9, 10, 14, 23, 24], "describ": [2, 4, 6, 7, 8, 10, 12, 13, 14, 15, 18, 23], "descript": [2, 10, 11, 23], "design": [2, 3, 5, 6, 7, 8, 9, 12, 13, 14, 15, 23], "designmatrix": [2, 23], "desir": [2, 4, 6, 7, 15, 16, 23, 24], "desktop": 0, "despit": [3, 14], "destroi": 18, "det": [7, 18, 24], "detail": [2, 8, 13, 15, 16, 18, 24], "detect": [5, 10, 14], "determin": [2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "determinist": [9, 15, 20], "dev": 3, "develop": [2, 5, 7, 10, 12, 13, 14, 17, 18, 23, 24], "deviat": [2, 3, 4, 6, 7, 8, 20, 23, 24], "devis": 14, "df": [6, 10, 13, 15, 23], "df1": 23, "di": 2, "diag": [7, 10, 24], "diagnost": [3, 12], "diagon": [2, 7, 9, 15, 18, 20, 23, 24], "diagonaliz": [7, 24], "diagram": 12, "diagsvd": 8, "dice": [8, 20], "dict": [8, 10], "dictionari": 2, "did": [1, 2, 3, 7, 8, 9, 12, 13, 16, 23], "die": 3, "diff": 4, "diff1": 4, "diff2": 4, "diff_ag": 4, "diffeent": 10, "differ": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23, 24], "differenti": [1, 2, 5, 17, 18, 23, 24], "difficult": [2, 3, 8, 12, 15, 20, 23], "difficulti": [2, 3, 15, 23], "diffonedim": 4, "digit": [2, 3, 5, 6, 8, 21, 23], "dilemma": 15, "dilut": 3, "dim": [6, 13, 16, 18], "dimens": [1, 2, 3, 4, 5, 6, 7, 10, 13, 16, 18, 23, 24], "dimension": [2, 6, 7, 8, 11, 13, 15, 16, 17, 18, 23, 24], "dimensionless": [2, 5, 23], "diment": 18, "dimnsion": 6, "diod": 5, "direct": [2, 3, 4, 6, 13, 14, 15, 16, 23, 24], "directli": [3, 6, 7, 8, 20, 24], "disadvantag": [2, 23], "disappear": [5, 8], "disc_loss": 6, "disc_tap": 6, "discard": [8, 13], "disciplin": [2, 5, 14], "disclaim": 20, "discord": 23, "discourag": [0, 15], "discov": [2, 23], "discover": 7, "discret": [3, 5, 7, 9, 15], "discrimin": [6, 9, 12, 13], "discriminator_loss": 6, "discriminator_loss_list": 6, "discriminator_model": 6, "discriminator_optim": 6, "discuss": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23, 24], "diseas": 9, "disguis": 8, "disord": [3, 9], "displai": [2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 20, 23, 24], "displaystyl": [2, 7, 23, 24], "disregard": [2, 23], "dissimilar": [13, 16], "dist": 16, "distanc": [2, 10, 11, 13, 16, 20], "distance_list": 11, "distinct": [5, 9, 10, 11, 12, 16], "distinctli": 10, "distinguish": [2, 6, 9, 10, 20, 23], "distplot": 2, "distribut": [2, 3, 6, 8, 9, 12, 13, 15, 16, 17, 18, 23, 24], "distrubut": [2, 17, 23], "dive": [2, 10, 18, 23], "diverg": [3, 15], "divid": [2, 3, 5, 7, 8, 9, 10, 11, 13, 14, 20, 23, 24], "divis": [8, 10, 11, 15, 18, 20], "dna": 9, "dnn": [2, 3, 4, 6, 14, 23], "dnn1": 6, "dnn2_gru2": 6, "dnn_kera": 3, "dnn_model": 3, "dnn_numpi": 3, "dnn_scikit": [2, 3, 23], "do": [0, 1, 2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 18, 23, 24], "doc": [0, 1, 2, 17, 19, 21, 22, 23], "document": [0, 6, 15], "doe": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 18, 20, 23], "doesn": [5, 11, 14, 23], "dog": [3, 5, 6], "domain": [7, 10, 15], "domin": [2, 23], "don": [0, 1, 2, 3, 5, 7, 8, 10, 13, 15, 17, 23, 24], "done": [1, 2, 4, 5, 6, 7, 8, 11, 12, 13, 15, 18, 23, 24], "dot": [2, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "doubl": [1, 5, 6, 18, 23], "doubli": 3, "down": [2, 5, 8, 11, 13, 14, 15], "download": [0, 2, 3, 5, 7, 8, 18, 22, 23], "downsampl": 5, "dozen": 3, "dq": 8, "drag": 15, "dramat": 13, "drastic": 6, "draw": [6, 8, 12, 15], "drawback": [2, 3, 5, 15, 24], "drawn": [3, 6, 8, 9, 13, 20, 23], "drive": [5, 6], "driven": 5, "drop": [2, 3, 7, 8, 13, 15, 20, 23, 24], "dropna": [2, 8, 23], "dropout": 6, "dt": [4, 5, 15, 20], "dtype": [2, 3, 5, 6, 16, 18, 23], "dub": [2, 23], "due": [3, 4, 7, 8, 10, 12, 14, 15, 21, 23, 24], "dummi": 2, "dure": [2, 3, 5, 6, 10, 11, 13, 17, 23], "dwell": 2, "dwh": 3, "dwo": 3, "dx": [4, 5, 10, 20], "dx_1": 20, "dx_1p": 8, "dx_2p": 8, "dx_mp": 8, "dx_n": 20, "dxp": 8, "dy": [3, 10, 20], "dynam": 6, "dz": 10, "e": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 21, 23, 24], "e_": [2, 4, 23], "each": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23, 24], "eapprox": [2, 23], "earli": [3, 15], "earlier": [2, 7, 9, 10, 11, 13, 14, 15, 23, 24], "earthexplor": 8, "eas": [8, 11, 16], "easi": [0, 2, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 23, 24], "easier": [0, 7, 8, 10, 11, 15, 20, 23, 24], "easiest": 15, "easili": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 23, 24], "eastern": [21, 23], "ebind": [2, 23], "eblock": 11, "econometr": 23, "economi": 7, "ecosystem": [17, 23], "ect": 19, "edg": 5, "edgecolor": 8, "editor": 0, "edu": 15, "educ": [2, 23], "eff": 20, "effect": [1, 3, 6, 12, 15, 20], "effic": 3, "effici": [2, 5, 12, 15, 17, 18, 20, 23], "efron": 8, "egrad": 15, "eig": [7, 13, 15, 18, 20, 23, 24], "eigen": 20, "eigenpair": [7, 13, 24], "eigenvalu": [2, 7, 10, 13, 15, 18, 23, 24], "eigenvector": [7, 13, 15, 24], "eight": [18, 23], "eigval": [18, 20, 23], "eigvalu": [13, 15], "eigvec": [18, 20, 23], "eigvector": [13, 15], "eir": [21, 23], "eispack": [18, 23], "either": [3, 7, 8, 9, 10, 11, 12, 13, 15, 20, 23, 24], "eivind": 21, "eivinsto": 21, "ekstr\u00f8m": 6, "elabor": 20, "elarn": 5, "electr": [2, 5, 14, 23], "electron": 23, "eleg": 13, "element": [3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 22, 24], "elementari": [12, 15, 18], "elementwis": [5, 15], "elementwise_grad": [4, 15], "elessar": 23, "elif": 16, "elim": 18, "elimin": [5, 10], "elin": [21, 23], "ellipsi": 1, "els": [1, 3, 5, 6, 9, 11, 14, 15, 18], "elu": 3, "elus": [2, 23], "email": [19, 21, 23], "embed": [2, 13, 24], "embodi": 8, "emit": 20, "emner": 22, "emphas": [2, 12, 17, 23], "emphasi": [2, 17, 22, 23], "empir": [3, 13, 20], "emploi": [2, 3, 7, 8, 13, 15, 20, 23, 24], "employ": 2, "empti": [0, 8, 12], "emul": 14, "en": [17, 22], "enabl": 13, "enbodi": 8, "encod": [2, 5, 7, 11, 13, 16, 23, 24], "encompass": [2, 20], "encount": [0, 2, 3, 7, 9, 15, 20, 23, 24], "encourag": 0, "end": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23, 24], "endpoint": [5, 8], "energi": [2, 6, 8], "enforc": 14, "eng": 22, "engin": [2, 3, 5, 6, 17, 23], "enorm": 5, "enough": [2, 8, 15, 23], "ensembl": [3, 11, 23], "ensur": [2, 3, 4, 5, 7, 8, 13, 15, 20, 24], "entail": 23, "enter": [7, 8, 24], "enthought": [2, 17, 23], "entir": [3, 5, 9, 11, 17, 20, 23], "entiti": [11, 14, 18, 23], "entri": [2, 7, 10, 13, 14, 18, 23, 24], "entropi": [3, 5, 9, 12, 15, 23], "enumer": [2, 3, 4, 5, 6, 8, 10, 23], "env": 20, "environ": [4, 17, 23], "environemnt": 0, "eo": [2, 8], "eol": 2, "eosfit": 2, "epoch": [2, 3, 5, 6, 14, 15, 23], "epsilon": [2, 7, 8, 9, 15, 23, 24], "epsilon_": [2, 23], "epsilon_0": [2, 23], "epsilon_1": [2, 23], "epsilon_2": [2, 23], "epsilon_i": [2, 23, 24], "eq": [5, 15, 16, 18, 20], "eqnarrai": [5, 7, 8], "equal": [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 15, 16, 18, 20, 23, 24], "equat": [3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23], "equilibrium": [4, 14], "equiv": [5, 15, 18, 20], "equival": [2, 3, 7, 9, 10, 13, 15, 17, 18, 23, 24], "erf": 20, "eriador": 23, "err": [2, 12], "err_": 8, "err_sqr": 4, "errat": 15, "erron": 4, "error": [0, 1, 3, 4, 6, 7, 8, 9, 11, 13, 14, 15, 17, 18, 20], "error_estimate_corr_tim": 20, "error_hidden": 3, "error_output": 3, "escap": 15, "especi": [0, 3, 5, 11, 14, 15], "essenti": [0, 2, 7, 8, 11, 12, 14, 16, 20, 24], "establish": [1, 2, 8, 12, 13], "estim": [2, 3, 7, 8, 9, 12, 13, 15, 17, 20, 23, 24], "estimated_mse_fold": 8, "estimated_mse_kfold": 8, "estimated_mse_sklearn": 8, "et": [1, 2, 4, 6, 22, 23, 24], "eta": [2, 3, 5, 10, 14, 15, 23], "eta0": [10, 15], "eta_": 15, "eta_t": 15, "eta_v": [2, 3, 5, 23], "etc": [2, 3, 5, 7, 9, 10, 11, 13, 14, 15, 16, 17, 18, 20, 24], "ethic": 17, "euclidean": [2, 16, 24], "evalu": [0, 1, 2, 4, 5, 6, 7, 8, 11, 15, 20, 23, 24], "evalut": 15, "even": [2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "evenli": 6, "event": [7, 9, 12, 20], "eventu": [2, 7, 8, 13, 14, 15, 21, 24], "everi": [0, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 17, 20, 21, 23, 24], "everyth": [1, 6, 14], "everywher": [6, 15], "evolv": 2, "exact": [2, 7, 13, 14, 15, 18, 20, 23, 24], "exactli": [2, 5, 6, 8, 14, 17, 24], "exam": 23, "examin": 8, "exampl": [0, 1, 7, 13, 14, 15, 17, 18, 20, 22], "exce": [3, 14, 15], "excel": [2, 3, 6, 7, 12, 23, 24], "except": [5, 6, 8, 10, 11, 18], "excess": [2, 23], "excit": 2, "exclud": [3, 8, 14], "exclus": [2, 3, 5, 8, 20, 23], "execut": [0, 4, 7, 15, 24], "exemplifi": 15, "exercic": [21, 23], "exercis": [7, 17, 19, 21, 23], "exhaust": 8, "exhibit": [2, 7, 8, 10, 23, 24], "exist": [2, 3, 4, 5, 7, 8, 9, 10, 11, 15, 18, 23], "exit": [7, 18, 24], "exp": [1, 2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 20, 24], "exp_term": 3, "expand": [7, 9, 13, 15], "expans": [2, 5, 7, 10, 12, 14, 15, 23, 24], "expect": [0, 2, 3, 7, 8, 9, 13, 14, 15, 17, 23, 24], "expectation_value_of_h_wrt_p": 20, "expens": [1, 8, 12, 15], "experi": [0, 2, 3, 8, 10, 15, 17, 23, 24], "experiment": [2, 6, 8, 11, 20, 23], "expert": [3, 11], "explain": [1, 2, 8, 11, 12, 13, 15, 23], "explained_variance_ratio_": 13, "explanatori": [2, 23], "explicit": [2, 5, 8, 15, 18, 23, 24], "explicitli": [2, 6], "explod": 3, "exploit": [2, 5, 14, 15, 23], "explor": [3, 6, 8, 10, 15, 17, 23], "expon": 3, "exponenti": [2, 3, 7, 8, 12, 15, 20, 23], "export": [0, 1, 11], "export_graphviz": 11, "export_text": 11, "exporttext": 11, "expos": 17, "express": [2, 4, 5, 7, 8, 9, 12, 14, 15, 18, 20, 23], "exptmean": 20, "exptvari": 20, "extend": [2, 4, 9, 13, 15, 17, 23], "extens": [0, 2, 14, 17, 23], "extent": [2, 3, 8, 22], "extern": [5, 8, 11], "extra": [0, 3, 5, 7, 21, 23, 24], "extract": [1, 2, 5, 7, 8, 9, 10, 13, 15, 18, 23, 24], "extrapol": [2, 23], "extrem": [0, 1, 2, 3, 6, 7, 8, 9, 10, 11, 15, 18, 24], "extremum": 15, "extrins": 13, "ey": [2, 7, 8, 15, 16, 18, 23, 24], "f": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 15, 16, 18, 20, 21, 23, 24], "f1": 15, "f11": [2, 23], "f12": [2, 23], "f13": [2, 23], "f1_grad": 15, "f1d": 15, "f2": 15, "f2_grad_x1": 15, "f2_grad_x1_analyt": 15, "f2_grad_x2": 15, "f2_grad_x2_analyt": 15, "f3": 15, "f3_grad": 15, "f3_grad_analyt": 15, "f4": 15, "f4_grad": 15, "f4_grad_analyt": 15, "f5": 15, "f5_grad": 15, "f6": 15, "f6_for": 15, "f6_for_grad": 15, "f6_grad_analyt": 15, "f6_while": 15, "f6_while_grad": 15, "f7": 15, "f7_grad": 15, "f7_grad_analyt": 15, "f8": 15, "f8_grad": 15, "f9": [2, 15, 23], "f9_altern": 15, "f9_alternative_grad": 15, "f9_grad": 15, "f_": 12, "f_0": [5, 12], "f_1": [12, 15], "f_2": [14, 15], "f_3": 14, "f_d": 20, "f_grad": 15, "f_grad_analyt": 15, "f_i": [1, 2, 8, 14], "f_m": [5, 12], "f_n": 5, "f_vec": 4, "face": [15, 23], "facecolor": [8, 10, 20], "facil": [2, 17], "facilit": 14, "fact": [2, 3, 5, 7, 11, 13, 14, 15, 23, 24], "factor": [2, 3, 5, 7, 8, 11, 12, 13, 15, 18, 20, 23, 24], "factori": 15, "fade": 8, "fafab0": [11, 12], "fail": [2, 8, 15, 21, 23], "failur": 9, "fairli": [3, 4, 20], "faisal": [1, 24], "fake": 6, "fake_loss": 6, "fake_output": 6, "fall": [10, 11, 19], "fals": [1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 16, 18, 23, 24], "famili": [2, 9, 10, 20, 24], "familiar": [0, 2, 5, 7, 8, 10, 17, 18, 20, 23], "famou": [8, 14], "far": [1, 2, 5, 6, 7, 8, 10, 13, 14, 15, 16, 23, 24], "fashion": [2, 11, 12, 23], "fast": [3, 5, 8, 12, 14, 15, 17, 20, 23], "faster": [3, 13, 15], "fastest": [15, 18], "favor": 9, "favorit": 20, "fc": 5, "featur": [0, 2, 3, 5, 7, 8, 9, 10, 12, 13, 14, 15, 17, 20, 23], "feature_nam": [2, 3, 9, 11], "feautur": 11, "fed": 3, "feed": [2, 4, 5, 13, 17, 23], "feed_forward": 3, "feed_forward_out": 3, "feed_forward_train": 3, "feedback": [6, 23], "feeddorward": 6, "feedforward": [3, 6, 14], "feel": [0, 1, 2, 7, 8, 13, 15, 17, 21, 23], "feet": 2, "fetch": [0, 8], "few": [3, 5, 6, 7, 11, 20, 23], "fewer": [2, 11, 13, 23], "ffnn": [3, 14], "field": [2, 5, 8, 14, 17], "fifth": [2, 8, 23], "fig": [2, 3, 4, 5, 6, 8, 9, 14, 15, 16, 23], "fig_id": [2, 8, 9, 11, 23], "figaxi": 20, "figsiz": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 23], "figur": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 23, 24], "figure_id": [2, 8, 9, 11, 23], "figurefil": [2, 8, 9, 11, 23], "file": [0, 2, 6, 7, 8, 9, 11, 23], "file_prefix": 6, "filenam": 23, "fill": [7, 11, 24], "filter": [5, 6], "final": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 19, 20, 21, 23], "financ": 2, "find": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23, 24], "fine": [2, 16], "finish": 4, "finit": [5, 7, 8, 14, 15, 20, 24], "finnicki": 0, "first": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 22, 24], "firsteigvector": 13, "fit": [3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 20, 24], "fit_intercept": [1, 2, 7, 8, 24], "fit_mod": 11, "fit_theta": 8, "fit_transform": [0, 2, 8, 10, 11, 13], "fiti": [2, 23], "five": [2, 11, 23, 24], "fix": [2, 5, 6, 8, 12, 13, 14, 15, 23], "flag": 6, "flat": [14, 15], "flatten": [3, 5, 6, 7, 18], "flexibl": [3, 8, 10, 12, 14, 23], "flip": [21, 23], "float": [2, 5, 6, 7, 11, 13, 15, 16, 18, 23, 24], "float32": [6, 11], "float64": [6, 18, 23], "flop": [7, 18, 24], "flow": [3, 6, 14], "fluctuat": 7, "fly": 13, "fm": 2, "fmax": 5, "fmesh": 15, "fn": 9, "focu": [0, 2, 5, 6, 7, 8, 17, 22, 23], "focus": [3, 8, 9, 18], "fold": [8, 11], "folder": [0, 2, 6, 8, 23], "follow": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 23, 24], "font": [9, 20, 23], "fontdict": 20, "fontsiz": [3, 8, 10, 11, 12, 20], "fontweight": 3, "footprint": 5, "foral": [10, 24], "forc": [2, 7, 8, 12, 13, 24], "forcast": 6, "forecast": [6, 14], "forest": [2, 3, 11, 17, 23], "forget": 13, "form": [0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23, 24], "formal": [5, 6, 16, 20], "format": [2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 17, 20, 22], "format_data": 6, "formatstrformatt": [8, 15], "formul": [6, 8, 13, 16], "formula": [5, 15, 20], "forth": [6, 14], "fortran": [2, 17, 18, 23], "fortran2003": [17, 23], "fortran90": 20, "fortun": [2, 13, 24], "forward": [2, 5, 8, 17, 18, 23], "found": [3, 4, 6, 7, 8, 14, 15, 23, 24], "foundat": [17, 23], "four": [6, 7, 8, 10, 14, 18, 19, 21, 23], "fourier": [2, 23], "fourierdef1": 5, "fourierdef2": 5, "fourierseriessign": 5, "fourth": [14, 23, 24], "fp": 9, "frac": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "fraction": 11, "frame": 9, "framework": [3, 10, 12, 20], "frank": [7, 13], "frankefunct": [7, 8, 13], "fredli": [21, 23], "free": [0, 1, 2, 8, 13, 15, 17, 18, 20, 21, 22, 23], "freecodecamp": 17, "freedom": 7, "freeli": 2, "freez": 0, "frequenc": [5, 8, 9, 20], "frequent": [2, 10, 11, 15], "frequentist": 17, "fresh": 12, "fridai": [0, 21, 23], "friedman": [8, 22, 23], "friendli": 6, "frodo": 23, "frog": 5, "from": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 15, 16, 17, 18, 20, 21, 22], "from_cod": 11, "from_logit": [5, 6], "from_tensor_slic": 6, "front": [2, 6, 7, 23, 24], "frustrat": 0, "fulfil": [4, 7, 14, 24], "full": [2, 3, 5, 7, 9, 11, 12, 15, 20, 23, 24], "full_matric": [7, 24], "fulli": [5, 8, 14, 20], "fun": [17, 23], "func": 4, "function": [0, 1, 4, 5, 6, 7, 11, 16, 17, 18], "functionali": 13, "fundament": [2, 8, 17, 23], "funtion": 4, "further": [4, 9, 11, 23], "furthermor": [2, 5, 7, 8, 9, 13, 14, 15, 17, 23, 24], "futur": [2, 6, 10, 11, 23], "fy": [0, 19, 21, 22, 23], "fys5419": [22, 23], "fys5429": [22, 23], "f\u00f8470": [21, 23], "g": [0, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 20, 23, 24], "g0": 4, "g_": [4, 11, 12], "g_0": 4, "g_1": [4, 12], "g_2": [4, 12], "g_analyt": 4, "g_dnn_ag": 4, "g_euler": 4, "g_i": 4, "g_m": [5, 12], "g_n": 5, "g_re": 4, "g_t": 4, "g_t_d2t": 4, "g_t_d2x": 4, "g_t_dt": 4, "g_t_hessian": 4, "g_t_hessian_func": 4, "g_t_jacobian": 4, "g_t_jacobian_func": 4, "g_trial": 4, "g_trial_deep": 4, "g_vec": 4, "gain": [3, 7, 9, 11, 12, 15, 24], "galleri": [2, 23], "game": 6, "gamge": 23, "gamma": [2, 4, 10, 11, 12, 13, 15, 23], "gamma1": 10, "gamma2": 10, "gamma_": [2, 23], "gamma_0": 12, "gamma_1": 12, "gamma_1x": 12, "gamma_i": [2, 10, 20, 23], "gamma_j": 15, "gamma_k": 15, "gamma_m": 12, "gamma_x": [2, 23], "gap": 10, "gate": [6, 14], "gather": [2, 3, 14, 24], "gaug": 14, "gaussbacksub": 18, "gaussian": [6, 7, 8, 10, 16, 20, 23], "gaussian_point": 16, "gaussian_rbf": 10, "gave": 15, "gavra": 23, "gbc": 23, "gca": [4, 8, 10, 15], "gd": 3, "gd_clf": 12, "gdclassiffiercgain": 12, "gdclassiffierconfus": 12, "gdclassiffierroc": 12, "gdm": 15, "gdregress": 12, "ge": [3, 7, 9, 20, 24], "gen_loss": 6, "gen_tap": 6, "gender": [2, 23], "genener": 6, "gener": [0, 1, 2, 3, 4, 5, 7, 8, 10, 12, 13, 14, 15, 16, 18, 20, 22, 24], "generaliz": 1, "generallay": 14, "generate_and_save_imag": 6, "generate_imag": 6, "generate_latent_point": 6, "generate_simple_clustering_dataset": 16, "generated_imag": 6, "generator_loss": 6, "generator_loss_list": 6, "generator_model": 6, "generator_optim": 6, "genom": 17, "geodes": 13, "geometr": [2, 15, 23], "geometri": 7, "georg": 22, "geotif": 8, "geq": [4, 7, 10, 11, 15, 24], "geron": [2, 22, 23], "get": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 17, 18, 20, 21, 23, 24], "get_dummi": 11, "get_paramet": 4, "get_split": 11, "get_yaxi": 10, "get_yticklabel": 8, "gh": 0, "gibb": [17, 23], "gif": 6, "gini": 12, "gini_index": 11, "ginvers": 15, "git": [0, 2, 17, 23], "giter": 15, "github": [2, 17, 19, 21, 22, 23, 24], "gitignor": 0, "gitlab": [0, 2, 17, 23], "give": [2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 20, 23, 24], "given": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "global": [8, 9, 15], "glorot": 3, "gnew": 15, "go": [0, 1, 2, 3, 5, 7, 8, 10, 11, 13, 14, 15, 23, 24], "goal": [2, 9, 11, 23], "goe": [0, 2, 3, 4, 7, 8, 15, 16, 18, 23, 24], "golden": 15, "gone": [7, 24], "gong": 3, "good": [0, 3, 5, 6, 7, 8, 11, 12, 13, 15, 17, 20, 22, 24], "goodfellow": [6, 22, 23, 24], "googl": [3, 6, 17, 23], "got": [3, 8], "gotten": 23, "gov": 8, "govern": 23, "gp": 22, "gpu": [3, 15, 17, 23], "grad": [4, 15], "grad_analyt": 15, "grade": 19, "gradient": [2, 5, 6, 9, 10, 11, 14, 17, 23, 24], "gradientboostingclassifi": 12, "gradientboostingregressor": 12, "gradients_of_discrimin": 6, "gradients_of_gener": 6, "gradienttap": 6, "gradual": [3, 16], "grai": [6, 8], "graph": [1, 3, 11, 13, 14, 15], "graph_from_dot_data": 11, "graphic": [0, 2, 3, 11, 23], "grasp": 2, "gray_r": [3, 5], "grayscal": 5, "great": [0, 7, 15], "greater": [3, 9, 20, 24], "greatli": 15, "greedi": 11, "green": [2, 5, 11, 20], "grei": 6, "grid": [3, 5, 8, 9, 10, 14, 20], "grossli": 15, "ground": [2, 23], "group": [0, 2, 8, 9, 11, 16, 17, 19, 21, 23], "groupbi": [2, 23], "grow": [3, 5, 11, 12], "growth": [2, 23], "gru": 6, "guarante": [2, 6, 15, 20, 23, 24], "guess": [3, 6, 12, 15, 16], "guestrin": 12, "gui": 0, "guid": 3, "h": [0, 2, 3, 7, 8, 10, 15, 20, 21, 22, 23, 24], "h1": 4, "h_": [2, 15, 23], "h_1": [4, 15], "h_2": [4, 15], "h_m": 12, "ha": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "haanen": [21, 23], "habit": [2, 24], "had": [2, 3, 8, 9, 15, 23], "hadamard": [3, 14, 15], "half": [3, 10, 11], "halv": 12, "hand": [2, 3, 4, 5, 7, 13, 14, 15, 17, 18, 20, 21, 22, 23, 24], "handi": 5, "handl": [0, 2, 3, 4, 7, 11, 13, 17, 24], "handle_unknown": 11, "handsid": 14, "handwrit": 14, "handwritten": [3, 7], "happen": [3, 4, 5, 6, 7, 8, 12, 15, 20, 24], "hard": [3, 9, 10, 12, 15], "hardcopi": [17, 23], "harder": [2, 3, 24], "harmon": 5, "hasn": [], "hassl": [2, 17, 23], "hast": [17, 23], "hasti": [1, 2, 8, 22, 23, 24], "hat": [1, 2, 3, 7, 8, 9, 11, 12, 13, 14, 15, 18, 24], "have": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23, 24], "haven": 3, "he": 9, "head": [2, 6, 12, 20], "header": [2, 23], "heads_proba": 12, "health": [2, 24], "hear": [2, 15, 23], "heart": [2, 9, 23], "heatmap": [2, 3, 5, 9, 23], "heavili": 2, "heavisid": 3, "height": [3, 5, 8], "held": 15, "help": [0, 1, 2, 3, 6, 14, 15, 23], "helper": [6, 16], "henc": [2, 7, 8, 10, 11, 12, 14, 15, 23, 24], "henrik": [21, 23], "her": 9, "here": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "hereaft": [2, 10, 14, 23], "hermitian": 18, "hessenberg": 18, "hessian": [2, 4, 7, 15], "heterogen": [11, 12], "hi": 9, "hidden": [3, 5, 6, 14], "hidden_bia": 3, "hidden_bias_gradi": 3, "hidden_layer_s": [2, 3, 23], "hidden_neuron": 6, "hidden_weight": 3, "hidden_weights_gradi": 3, "hierarch": [7, 24], "high": [2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 15, 16, 17, 18, 23, 24], "higher": [2, 3, 5, 7, 8, 10, 15, 23, 24], "highest": [3, 4], "highli": [2, 5, 6, 12, 17, 18, 22, 23, 24], "highwai": 2, "hing": 10, "hint": [0, 1, 15, 24], "hip": 17, "hire": 2, "hist": [6, 8, 9, 20], "histogram": [2, 8, 9, 20], "histor": [9, 13], "histori": [0, 5, 6, 14], "hitherto": 7, "hjorth": [21, 23, 24], "hobbi": 20, "hoc": [7, 24], "hoff": 22, "hold": [3, 5, 8, 15, 16], "holder": [2, 23], "home": 2, "homepag": 23, "homework": [8, 15], "homogen": [3, 5, 11, 12, 15], "honchar": 4, "hopefulli": [0, 2, 13, 20, 23], "horizont": 13, "horlyk": [21, 23], "hors": [5, 9, 23], "hot": [3, 11], "hour": [3, 17, 19, 20, 21, 23], "how": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "howev": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "hspace": [2, 6, 10, 12, 20, 23], "hstack": 3, "htf": 23, "html": [1, 2, 17, 19, 21, 22, 23], "http": [0, 1, 2, 5, 6, 8, 15, 17, 18, 19, 21, 22, 23, 24], "huang": [2, 23], "huber": [2, 23], "huge": [3, 5, 6, 17], "human": [2, 3, 5, 8, 11, 14], "humid": 11, "hundr": 3, "hungri": 3, "hybrid": 19, "hydrogen": [2, 23], "hyperbol": [3, 6, 14], "hyperparam": 10, "hyperparamet": [5, 6, 7, 8, 11, 15, 24], "hyperplan": 13, "h\u00f8rlyk": [21, 23], "i": [0, 1, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24], "i0": [2, 23], "i1": [2, 8, 10, 14, 23], "i2": [2, 10, 14, 23], "i3": [2, 14, 23], "i5": [2, 23], "i_": 15, "i_1": [7, 8], "i_2": [7, 8], "ian": 22, "ic": 3, "id": [9, 15], "ida": [21, 23], "idea": [2, 3, 4, 5, 6, 8, 11, 12, 14, 15, 18], "ideal": [2, 4, 8, 10, 15, 20, 23], "idem": 8, "ident": [7, 8, 14, 15, 18, 24], "identifi": [2, 3, 9, 11, 13, 14, 15, 16, 23, 24], "ieor": 20, "ifi": 22, "ifs": [17, 23], "ignor": [0, 2, 3, 5, 11, 24], "ii": [18, 20], "iii": [18, 23], "ij": [1, 2, 3, 5, 8, 10, 14, 16, 18, 20, 23, 24], "ik": [2, 18, 23, 24], "illustr": [7, 9, 12, 14, 15, 16, 17, 23], "im": 8, "imag": [3, 5, 6, 8, 11, 13, 14, 16, 22, 23], "image_at_epoch_": 6, "image_batch": 6, "image_height": 5, "image_path": [2, 8, 9, 11, 23], "image_width": 5, "imageio": 8, "images_from_seed_imag": 6, "imagin": 3, "immedi": [2, 5, 6, 8, 17, 23], "implement": [2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 20, 23, 24], "impli": [5, 7, 8, 9, 15, 18, 24], "implicit": 5, "implicitli": [13, 20], "import": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20], "importantli": 5, "impos": [2, 8, 13, 14, 23], "imposs": [2, 7, 23, 24], "impress": [2, 14, 23], "improv": [0, 2, 6, 7, 11, 12, 13, 15, 24], "impur": 11, "imread": 8, "imshow": [3, 5, 6, 8], "in3050": [22, 23], "in3310": 23, "in4080": [22, 23], "in4300": [22, 23], "in4310": 22, "in5400": 5, "in5550": 22, "in_out_neuron": 6, "inaccur": 15, "inact": 14, "inadequ": [2, 23], "inch": 8, "includ": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 17, 20, 21, 22, 23, 24], "include_bia": [8, 11], "incom": [1, 14], "incorrect": 3, "incoveni": 10, "increas": [2, 3, 5, 6, 7, 8, 11, 14, 15, 20, 23], "increasingli": 20, "ind": 8, "inde": [2, 4, 6, 7, 8, 15, 23, 24], "indefinit": 6, "independ": [2, 7, 8, 9, 10, 14, 15, 20, 23, 24], "index": [2, 3, 5, 6, 12, 16, 17, 18, 20, 22, 23], "index_col": [2, 23], "indic": [1, 2, 3, 5, 6, 7, 8, 11, 12, 13, 15, 23, 24], "indispens": 8, "individu": [3, 8, 9, 12, 14, 20, 23, 24], "indu": 2, "indx": 18, "indx1": 4, "indx2": 4, "indx3": 4, "ineffici": [5, 15], "inequ": [10, 15], "inertia": 15, "inf1000": [17, 23], "inf1100": [17, 23], "inf1100l": [17, 23], "inf1110": [17, 23], "inf3000": 23, "infeas": 11, "infer": [2, 3, 6, 8, 22, 23], "inferenc": 3, "infil": [2, 8, 9, 11, 23], "infin": [7, 8, 9, 13, 24], "infinit": 5, "infinitesim": 20, "influenc": [8, 12], "influenti": 3, "info": 23, "inform": [2, 3, 5, 6, 8, 11, 13, 14, 15, 16, 18, 22, 23], "inforom": 0, "infti": [5, 8, 15, 20], "ingeni": 15, "ingredi": [2, 11, 23], "inher": 8, "inherit": [18, 23], "initi": [2, 3, 4, 8, 12, 15, 16, 18, 20, 23], "inject": 16, "inlin": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "inner": [2, 15, 24], "inp": 6, "inplac": 15, "input": [1, 2, 3, 5, 6, 7, 8, 9, 10, 14, 15, 16, 20, 23, 24], "input_dim": 3, "input_shap": [5, 6], "inputs": 3, "inputs_shuffl": [2, 3, 24], "insert": [5, 7, 8, 10, 12, 20, 24], "insid": [2, 6, 9], "insight": [2, 3, 7, 17, 23, 24], "insist": [8, 15], "inspir": [2, 3, 14, 23], "instabl": 4, "instal": [0, 2, 3, 7, 8, 11], "instanc": [1, 2, 3, 4, 6, 8, 11, 13, 15, 23, 24], "instanti": 12, "instead": [2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 15, 16, 18, 20, 23, 24], "institut": 3, "instruct": [0, 2, 3], "int": [2, 3, 4, 5, 6, 7, 8, 13, 15, 16, 18, 20, 24], "int32": 12, "int_": [5, 8, 20], "int_0": 20, "int_a": 20, "intak": [2, 24], "integ": [3, 4, 15, 16, 18, 20, 23], "integer_vector": 3, "integr": [5, 8, 20, 23], "intellig": [2, 16, 22, 23], "intend": 12, "intens": 3, "intention": 16, "interact": [2, 8, 11, 14, 17, 23], "intercept": [1, 2, 8, 10, 13, 15, 23, 24], "intercept_": [2, 8, 10, 11, 15, 23], "interchang": [7, 14, 18], "interconnect": 3, "interest": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 17, 20, 23, 24], "interfac": [0, 2, 3, 18, 24], "interior": [2, 11, 23], "intermedi": [18, 24], "intern": [3, 12, 14], "interpol": [3, 5, 6, 8, 14], "interpr": [7, 24], "interpret": [0, 1, 2, 3, 8, 11, 12, 14, 15, 18, 20], "interv": [2, 5, 7, 8, 9, 15, 20, 23, 24], "intial": 15, "intract": [2, 6, 24], "intrins": [5, 13, 18, 20, 23], "intro": [17, 22, 23], "introduc": [2, 3, 7, 8, 10, 12, 14, 18, 20, 23], "introduct": [3, 4, 6, 15, 22, 24], "introductori": [2, 6, 18, 22, 23, 24], "intuit": [2, 7, 8, 10, 14, 15, 23], "inv": [2, 7, 15, 23, 24], "invalu": [2, 15, 17, 23], "invari": 3, "invd": 7, "inver": 10, "invers": [2, 5, 8, 15, 23, 24], "inverse_transform": 10, "invert": [1, 2, 7, 9, 12, 15, 23], "invh": 15, "invok": [2, 10], "involv": [2, 4, 8, 9, 13, 14, 23, 24], "io": [2, 17, 19, 21, 22, 23, 24], "ip": [2, 10, 20, 23], "ipca": 13, "ipynb": [17, 23], "ipython": [2, 7, 9, 11, 13, 16, 17, 23, 24], "iq": 8, "iri": [10, 11], "irreduc": 8, "irrelev": [7, 24], "irrespect": [2, 23], "isn": 7, "isnul": 2, "isomap": 13, "issu": [0, 3, 11, 18], "it_arrai": 15, "item": [2, 15, 23], "items": [18, 23], "iter": [3, 4, 6, 8, 10, 15, 16, 20], "its": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "itself": [7, 8, 14, 20, 23, 24], "j": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 15, 16, 18, 20, 22, 23, 24], "j1": 18, "j_": 8, "j_lasso_sk": 8, "j_ridge_sk": 8, "j_sk": 8, "jackknif": [8, 17, 23], "jacobian": [4, 15], "jason": 6, "jax": [17, 23], "jensen": [21, 23, 24], "jerom": 22, "ji": [14, 18], "jit": 15, "jj": [2, 7, 8, 23], "jk": [2, 3, 8, 14, 18, 23], "jl": [2, 23], "jm": 18, "jnp": 15, "job": [0, 4, 10, 12], "join": [2, 6, 8, 9, 11, 23], "joint": [6, 7], "judg": 15, "judgement": 8, "julia": [17, 18], "jump": 20, "junk": 6, "jupit": 23, "jupyt": [0, 1, 2, 17, 22, 23], "just": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23, 24], "justif": 2, "justifi": [5, 12], "k": [2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23, 24], "k0": 9, "k1": 9, "kaggl": 8, "kappa_d": 20, "karl": [21, 23], "karush": 10, "katrin": [21, 23], "keep": [0, 2, 3, 6, 7, 8, 13, 15, 16, 18, 23, 24], "keepdim": [3, 8, 12, 18], "kei": [2, 3, 5, 8, 14], "kept": [6, 8, 16], "kera": [2, 6, 17, 23], "kernel": [2, 3, 5, 17, 23, 24], "kernel_regular": [3, 5], "kernel_s": 6, "kernelpca": 13, "kev": [2, 23], "kevin": [22, 23], "keyword": [18, 23], "kfold": 8, "kg": 3, "ki": 18, "kick": [3, 15], "kiener": 4, "kilomet": 8, "kind": [2, 4, 5, 6, 10, 14, 15, 16, 23, 24], "kj": [8, 14, 18, 24], "kjm": [17, 23], "kkt": 10, "kl": 20, "km": [14, 23], "kmean": 16, "kmeanspoint": 16, "kn_k": 16, "know": [0, 1, 2, 3, 4, 7, 8, 10, 15, 17, 23, 24], "knowledg": [2, 17, 23], "known": [3, 5, 6, 7, 8, 9, 10, 11, 14, 18, 20, 22], "kondev": [2, 23], "kp": 20, "kpca": 13, "kroneck": 16, "kuhn": 10, "kvalsund": [21, 23], "kwown": [2, 23], "l": [2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 23], "l0": 9, "l1": [2, 3, 5, 9, 23], "l1_l2": [3, 5], "l1regl": 7, "l2": [3, 5], "l_": 18, "l_1": 9, "l_2": [9, 15], "l_j": 14, "la": 15, "la_i": 14, "la_k": 14, "lab": [17, 23], "label": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 20, 23, 24], "labelencod": [9, 12], "labels": [8, 10, 11], "labels_shuffl": [2, 3, 24], "laboratori": 19, "lack": [2, 23], "lagari": 4, "lagrang": [10, 13], "lambda": [2, 3, 4, 5, 7, 8, 9, 10, 12, 14, 15, 20, 23, 24], "lambda_": 13, "lambda_0": 13, "lambda_1": [7, 10, 13, 24], "lambda_2": [10, 13], "lambda_i": [10, 13], "lambda_iy_i": 10, "lambda_jy_iy_j": 10, "lambda_k": 10, "lambda_n": [7, 10, 24], "lamda": 3, "land": [2, 10], "landmark": 10, "landscap": 15, "langl": [2, 8, 13, 20, 23, 24], "languag": [2, 3, 6, 10, 17, 18, 22, 23], "lapack": [18, 23], "laplac": 7, "laptop": [0, 17], "larg": [2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 20, 22, 23, 24], "larger": [2, 5, 7, 8, 10, 12, 13, 15, 20, 23, 24], "largest": [6, 10, 13], "lasso": [2, 9, 17, 23], "lasso_sk": 8, "last": [1, 2, 3, 5, 6, 7, 8, 9, 10, 14, 18, 20, 21, 23], "latent": 6, "latent_dim": 6, "latent_point": 6, "latent_space_value_rang": 6, "later": [0, 2, 3, 6, 9, 10, 14, 15, 16, 17, 23], "latest": [0, 6, 17], "latest_checkpoint": 6, "latex": 23, "latter": [2, 5, 8, 9, 10, 13, 15, 18, 20, 23, 24], "lattic": 14, "law": 2, "layer": [2, 6, 15, 23], "lbfg": [9, 11, 12], "lcc": [7, 8], "lda": 13, "ldot": [2, 8, 13, 23], "le": [7, 9, 12, 15, 20, 24], "lead": [1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "leaf": 11, "leaki": 3, "leakyrelu": 6, "lear": 15, "learn": [5, 6, 7, 8, 9, 10, 11, 12, 14, 18, 21, 22], "learnabl": 5, "learner": 12, "learnig": 23, "learning_r": [10, 12], "learning_rate_init": [2, 3, 23], "learning_schedul": 15, "least": [2, 9, 10, 12, 13, 17, 18, 20], "leat": 15, "leav": [2, 3, 5, 7, 8, 11, 13, 23], "lectur": [2, 3, 7, 12, 13, 14, 15, 17, 18, 19, 21, 22, 24], "lecturenot": [2, 17, 22, 23], "left": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "leftarrow": [10, 14], "legend": [0, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 23, 24], "leinonen": 23, "len": [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 18, 23, 24], "length": [1, 2, 3, 5, 6, 10, 11, 15, 17, 23, 24], "length_of_sequ": 6, "leq": [2, 7, 9, 10, 15, 16, 20, 23, 24], "less": [2, 3, 5, 6, 7, 8, 10, 11, 15, 17, 20, 23, 24], "lessen": 3, "let": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "letter": [1, 2, 18, 20, 23, 24], "level": [2, 3, 7, 8, 11, 17, 18, 19, 21, 23], "li": [10, 13], "lib": [], "liblinear": 12, "librari": [2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 18, 20, 22, 24], "licens": [2, 3, 17, 23], "lie": [2, 8, 13, 20, 23, 24], "life": [2, 3, 10, 14, 23], "lifetim": 15, "like": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "likelihood": [2, 3, 7, 11, 23, 24], "lim_": 20, "limit": [2, 7, 8, 10, 14, 18, 23, 24], "lin_clf": 10, "lin_model": 2, "lin_reg": 11, "linalg": [2, 4, 7, 8, 10, 13, 15, 18, 20, 23, 24], "line": [0, 1, 2, 5, 8, 10, 13, 15, 23], "line1": 10, "line2": 10, "line3": 10, "line_model": 0, "line_ms": 0, "line_predict": 0, "linear": [1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 17, 20], "linear_model": [0, 1, 2, 7, 8, 9, 10, 11, 12, 13, 15, 23, 24], "linear_regress": 8, "linearli": [7, 24], "linearloc": [8, 15], "linearregress": [0, 1, 2, 8, 9, 11, 23, 24], "linearsvc": 10, "liner": [3, 5], "linerar": 12, "linewidth": [2, 4, 6, 8, 10, 11, 12], "link": [0, 2, 6, 11, 14, 17, 19, 21, 23], "linlag": 7, "linpack": [18, 23], "linreg": [2, 23], "linspac": [1, 2, 4, 5, 6, 8, 10, 11, 12, 15, 18, 20, 23, 24], "linu": 6, "linux": [2, 3, 17, 23], "liquid": [2, 23], "list": [0, 2, 3, 4, 5, 6, 11, 17, 23], "listedcolormap": [11, 12], "literatur": [3, 9, 16, 22], "littl": [3, 5, 11, 14], "live": [1, 10], "ll": [2, 20, 23, 24], "lle": [2, 24], "lloyd": [6, 16], "lmb": [2, 4, 7, 8, 24], "lmbd": [2, 3, 5, 23], "lmbd_val": [2, 3, 5, 23], "lmbda": 15, "ln": [3, 15], "load": [2, 3, 6, 8, 9, 11, 12], "load_boston": 2, "load_breast_canc": [3, 9, 11, 12, 13], "load_data": [5, 6], "load_digit": [3, 5], "load_iri": [10, 11], "loc": [5, 8, 9, 10, 11, 12, 23], "local": [0, 2, 3, 5, 9, 14, 15, 24], "locat": [0, 4, 5, 10], "log": [0, 2, 3, 4, 6, 7, 8, 9, 11, 12, 13, 15, 18, 23], "log10": [2, 7, 8, 24], "log_": [2, 23], "log_clf": 12, "logarithm": [2, 7, 9, 18, 23], "logic": [2, 3, 11, 23], "login": 0, "logist": [2, 3, 4, 10, 11, 12, 13, 14, 15, 17, 24], "logisticregress": [9, 11, 12, 13], "logit": 9, "logreg": [9, 11, 12, 13], "logspac": [2, 3, 5, 7, 8, 23, 24], "long": [2, 3, 5, 6, 14, 15, 23], "longer": [4, 5, 10, 12, 16, 18, 20, 23], "loocv": 8, "look": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 23, 24], "loop": [1, 3, 6, 8, 12, 14, 16, 17, 18, 23], "lose": 3, "loss": [2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15, 18, 23], "loss_fil": 6, "lossfil": 6, "lost": 6, "lot": [1, 2, 3, 6, 8], "low": [2, 8, 11, 12, 13, 23, 24], "lower": [1, 2, 3, 5, 8, 11, 12, 18, 24], "lowercas": [18, 23], "lowest": [11, 15, 20], "lr": [3, 5, 6, 12], "lstat": 2, "lstm": 6, "lstm_2layer": 6, "lstsq": [2, 23, 24], "lt": 8, "lu": [2, 7, 23, 24], "lubksb": 18, "luckili": 4, "ludcmp": 18, "lux": 18, "lvert": 3, "lw": [2, 23], "m": [0, 2, 3, 4, 5, 7, 8, 10, 11, 12, 13, 14, 15, 18, 20, 21, 22, 23, 24], "m_": [11, 14], "m_1": 16, "m_h": [2, 23], "m_k": 16, "m_l": 14, "m_n": [2, 23], "m_p": [2, 23], "m_t": 15, "ma": 13, "machin": [0, 1, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 18, 22, 24], "machinelearn": [1, 2, 8, 17, 19, 21, 22, 23], "mackai": 22, "made": [2, 3, 5, 6, 7, 8, 9, 11, 13, 14, 23, 24], "mae": [2, 23], "magic": 6, "magnitud": [3, 8, 9, 15], "mai": [2, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23, 24], "mail": [19, 21], "main": [2, 3, 5, 6, 7, 8, 9, 11, 18, 22, 24], "mainli": [2, 7, 8, 9, 11, 23, 24], "maintain": 8, "major": [3, 8, 11, 12, 15, 18, 23], "make": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 17, 18, 20, 22, 23], "make_axes_locat": 8, "make_moon": [10, 11, 12], "make_pipelin": [2, 8, 12, 24], "makedir": [2, 8, 9, 11, 23], "malcondit": 18, "malign": [3, 9, 11], "mammographi": 7, "manag": [0, 2, 4, 5, 17, 23], "mandatori": [21, 23], "mani": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 17, 18, 20, 22, 23, 24], "manifold": 13, "manner": 5, "manual": 8, "map": [2, 3, 4, 8, 9, 10, 13, 14, 16, 20, 23], "marc": 24, "margin": [2, 7, 10], "marit": [2, 23], "mark": 23, "marker": [2, 9, 18, 23], "markov": [17, 23], "marsaglia": 20, "mass": [2, 3, 7, 15, 24], "massag": [2, 23], "masses2016": [2, 23], "masses2016ol": [2, 23], "masses2016tre": 2, "masseval2016": [2, 23], "master": [19, 21], "mat": [17, 23], "mat1100": [17, 23], "mat1110": [17, 23], "mat1120": [17, 23], "match": [0, 3, 6, 7, 15, 16, 24], "materi": [0, 6, 7, 9, 15, 18, 19, 21, 24], "math": [5, 9, 14, 15, 18, 20, 22, 23], "mathbb": [2, 6, 7, 8, 9, 10, 13, 14, 15, 16, 18, 20, 23, 24], "mathbf": [2, 7, 8, 9, 10, 15, 18, 23, 24], "mathcal": [3, 7, 8, 9, 15], "matheemat": 5, "mathemat": [2, 8, 13, 14, 15, 17, 18, 20, 22, 23], "mathemati": 23, "mathrm": [2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 23, 24], "matmul": [3, 4, 7], "matnat": 22, "matplotlib": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "matric": [1, 2, 3, 5, 6, 8, 9, 10, 13, 15, 17, 24], "matrix": [2, 4, 5, 6, 8, 9, 10, 12, 15, 20], "matshow": 3, "matter": [4, 5, 15, 24], "max": [2, 3, 4, 5, 6, 11, 12, 14, 15, 21, 23], "max_depth": [2, 11, 12], "max_diff": 4, "max_diff1": 4, "max_diff2": 4, "max_it": [2, 3, 10, 15, 23], "max_iter": 16, "max_leaf_nod": 12, "max_sampl": 12, "maxdegre": [2, 8, 12, 24], "maxdepth": 12, "maxim": [3, 6, 7, 9, 10, 13], "maximum": [2, 4, 5, 7, 9, 10, 11, 12, 15, 16, 23, 24], "maxpolydegre": [7, 8, 24], "maxpooling2d": 5, "mbox": [7, 8, 24], "mcculloch": 14, "md": 13, "mdoel": 6, "mean": [0, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "mean_absolute_error": [2, 23], "mean_divisor": 16, "mean_i": 20, "mean_matrix": 16, "mean_squared_error": [0, 2, 6, 8, 9, 12, 23, 24], "mean_squared_log_error": [2, 23], "mean_vector": 16, "mean_x": 20, "meaning": [2, 6, 9, 23], "meansquarederror": [2, 23], "meant": [5, 9, 12, 15], "measur": [1, 2, 3, 4, 7, 8, 11, 13, 14, 16, 20, 23, 24], "mechan": [2, 6, 20, 23], "median": [2, 23, 24], "medicin": 14, "medium": [6, 10, 15], "medv": 2, "meet": [2, 21], "mehta": [2, 23, 24], "memori": [5, 6, 13, 14, 15, 18], "mention": [2, 14, 15, 20, 23], "mere": 2, "meshgrid": [4, 7, 8, 10, 11, 12, 13], "mess": 0, "messag": [7, 15], "messi": 4, "met": [2, 5, 10, 24], "meteorolog": 11, "meter": 8, "method": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 13, 14, 16, 17, 18, 20, 22, 24], "metion": 8, "metric": [0, 2, 3, 5, 8, 9, 11, 12, 16, 23, 24], "metropoli": [17, 23], "mev": [2, 20, 23], "mgd": 15, "mglearn": [17, 23], "mgrid": 15, "mhjensen": [], "mi": 12, "mia": [21, 23], "microsoft": 22, "mid": 3, "midel": 6, "midnight": 0, "midpoint": 11, "might": [0, 2, 3, 4, 6, 8, 11, 15, 24], "mild": 11, "millimet": 8, "million": [2, 23, 24], "mimic": 14, "min": [2, 4, 7, 10, 11], "min_": [2, 4, 7, 16, 23, 24], "min_samples_leaf": 11, "mind": [0, 2, 8, 15, 23, 24], "mindboard": 6, "mine": [17, 23], "mini": [3, 13, 14, 15], "minibatch": [3, 13, 15], "minibathc": 15, "miniforge3": [], "minim": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 24], "minima": [2, 3, 9, 15, 23], "minimum": [2, 3, 4, 8, 10, 11, 13, 15, 24], "minmaxscal": [2, 24], "minor": 20, "minst": 3, "minu": 9, "mirjalili": 23, "mirror": 11, "misc": 8, "misclassif": [10, 11, 12], "misclassifi": [10, 12], "miser": 2, "mismatch": 3, "miss": [2, 9, 12], "mistak": 6, "mit": 22, "mix": [3, 4, 23], "mixtur": 15, "mk": [11, 18], "mkdir": [2, 8, 9, 11, 23], "ml": [2, 3, 12, 15, 18, 24], "mlab": 20, "mle": [7, 9], "mlp": 3, "mlpclassifi": 3, "mlpregressor": [2, 23], "mm": 18, "mml": 24, "mn": [14, 20], "mnist": [3, 13], "mod": 20, "mode": [19, 21, 23], "model": [1, 4, 5, 7, 9, 10, 11, 12, 13, 15, 16, 17, 20, 22, 24], "model_select": [0, 1, 2, 3, 5, 7, 8, 9, 11, 12, 13, 23, 24], "moder": 12, "modern": [2, 8, 9, 17, 23], "modif": [4, 14, 15], "modifi": [2, 3, 5, 7, 9, 10, 12, 14, 15, 23, 24], "modul": [1, 2, 18, 23], "modular": 20, "modulo": 20, "moe": [13, 24], "moment": [7, 8, 15, 20], "mondai": [21, 23], "monitor": 15, "monoton": [7, 14, 20], "mont": [2, 8, 17, 20, 22, 23], "montli": 1, "moor": [7, 8], "more": [1, 2, 3, 4, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20], "moreov": [2, 5], "morten": [21, 23, 24], "mortenhj": 23, "most": [0, 1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23, 24], "mostli": [3, 13], "motion": [2, 15], "motiv": [3, 6], "move": [0, 1, 2, 6, 7, 8, 9, 11, 14, 15, 16, 20, 24], "mpl": [9, 23], "mpl_toolkit": [4, 8, 15], "mplot3d": [4, 8, 15], "mplregressor": 3, "mse": [0, 1, 2, 6, 7, 8, 11, 12, 23, 24], "mse_simpletre": 12, "mselassopredict": 7, "mselassotrain": 7, "mseownridgepredict": [8, 24], "msepredict": 7, "mseridgepredict": [2, 7, 8, 24], "msetrain": 7, "msle": [2, 23], "mt": [9, 14], "mu": [2, 8, 13, 15, 20, 23], "mu0": 20, "mu1": 20, "mu2": 20, "mu_": [8, 20], "mu_i": 8, "mu_n": 13, "mu_x": 20, "much": [0, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "multi": [2, 3, 5, 9, 17, 23], "multiclass": [3, 9], "multidimension": [13, 14, 23], "multilay": 3, "multinomi": 9, "multipl": [0, 4, 6, 7, 8, 9, 14, 15, 20, 24], "multipli": [5, 7, 8, 13, 15, 18, 20, 24], "multiplum": 10, "multivari": [2, 4, 12, 13, 17, 20, 23], "multivariate_norm": [13, 16], "multpli": 1, "murphi": [13, 22, 23], "must": [0, 3, 4, 7, 8, 10, 12, 14, 15, 16, 20, 24], "mutat": 9, "mutual": [3, 5, 8, 15], "mx_": 20, "my": 23, "myenv": [], "myriad": [2, 17, 23], "mz1": 20, "mz2": 20, "m\u00f8svatn": 8, "n": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 23, 24], "n1": 18, "n2": 18, "n_": [3, 4, 5, 10, 14, 20], "n_0": [14, 20], "n_boostrap": [8, 12], "n_bootstrap": 8, "n_categori": [3, 5], "n_cluster": 16, "n_compon": 13, "n_epoch": 15, "n_estim": 12, "n_examples_to_gener": 6, "n_featur": 3, "n_filter": 5, "n_hidden": 4, "n_hidden_neuron": [2, 3, 23], "n_i": 20, "n_input": [2, 3, 5, 24], "n_instanc": 11, "n_job": 12, "n_k": 16, "n_l": [14, 20], "n_layer": 3, "n_m": 11, "n_neuron": 3, "n_neurons_connect": 5, "n_neurons_layer1": 3, "n_neurons_layer2": 3, "n_point": 16, "n_sampl": [8, 10, 11, 12, 16], "n_split": 8, "n_step": 6, "n_t": 4, "n_x": 4, "nabla": [3, 15], "nabla_": [4, 15], "nabla_w": 15, "nag": 15, "naimi": [2, 23], "naiv": 9, "naive_kmean": 16, "name": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 20, 21, 23, 24], "narrow": 15, "nation": [3, 7], "nativ": [17, 23], "natur": [2, 3, 6, 10, 11, 14, 15, 20, 22, 23], "navier": 14, "navig": 0, "nb": 20, "nb_": 18, "nbconvert": 23, "nd": 16, "ndarrai": 8, "ne": [11, 12, 18, 20, 24], "nearest": [3, 5, 8, 13], "nearli": 15, "neat": 23, "neccesari": 8, "necess": 4, "necessari": [2, 3, 5, 6, 10, 16, 23], "necessarili": [2, 6, 13, 20, 23], "necesserali": 7, "neck": 9, "need": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 24], "neg": [2, 3, 5, 7, 8, 9, 12, 15, 18, 20, 23], "neg_mean_squared_error": 8, "neglect": 20, "neglig": 20, "neighbor": [5, 8, 13], "neither": [6, 15], "neq": [15, 16, 20], "nervou": 14, "nest": [11, 14], "nesterov": 15, "net": [4, 6, 14], "netlib": [18, 23], "network": [2, 11, 15, 17, 22, 24], "neural": [2, 15, 17, 22, 24], "neural_network": [2, 3, 4, 23], "neuralnetwork": 3, "neuron": [3, 4, 5, 6, 14], "neutral": [2, 23], "neutron": [2, 23], "never": [3, 6, 8, 11, 20], "new": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 23, 24], "new_chang": 15, "new_hobbit": 23, "newaxi": [2, 5, 8, 11], "newli": [2, 23], "newton": [3, 9, 10, 15, 20], "next": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 15, 16, 23, 24], "next_guess": 15, "next_input": 6, "ng": 3, "ni": 16, "nice": [2, 3, 7, 13, 23, 24], "niter": 15, "nitric": 2, "nlambda": [2, 7, 8, 24], "nlp": 22, "nm": 20, "nm_n": [2, 23], "nmse": 8, "nn": [4, 7, 8, 14, 18, 23], "nn_model": 3, "nnmin": 4, "node": [3, 5, 11, 12, 14], "nois": [2, 6, 7, 8, 10, 11, 12, 15, 23, 24], "noise_dimens": 6, "noisi": [3, 8], "non": [2, 3, 5, 7, 8, 9, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "none": [2, 3, 4, 6, 7, 11, 12, 15, 20, 23, 24], "nonlinear": [5, 8, 10, 11, 13, 14], "nonneg": [8, 11, 15], "nonparametr": 8, "nonsens": 20, "nonsingular": 18, "nonumb": [5, 9, 10, 15, 18], "nor": [3, 6, 15], "norm": [2, 3, 7, 8, 10, 13, 15, 23, 24], "normal": [1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "normali": [18, 23], "norwai": [8, 23], "notat": [2, 4, 7, 8, 15, 16, 20, 23, 24], "note": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 16, 17, 18, 20, 22, 23], "notebook": [0, 1, 2, 3, 5, 11, 17, 23], "noth": [3, 4, 7, 10, 14, 16, 20, 24], "notic": [6, 7, 14, 15, 18, 20, 23], "notion": 5, "novel": [5, 8, 12, 23], "novemb": [3, 21, 23], "now": [0, 1, 2, 4, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20, 23, 24], "nowadai": [2, 3, 5, 11, 17, 23], "nox": 2, "np": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23, 24], "npr": 4, "nsampl": 8, "nt": 4, "nu": 20, "nuclear": [7, 24], "nuclei": [2, 20, 23], "nucleon": [2, 23], "nucleu": [2, 23], "num": 6, "num_coordin": 4, "num_hidden_neuron": 4, "num_it": 4, "num_neuron": 4, "num_neurons_hidden": 4, "num_point": 4, "num_tre": 12, "num_valu": 4, "number": [1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 23], "numberid": 9, "numberparamet": 5, "numer": [2, 7, 8, 11, 12, 13, 14, 15, 17, 18, 22, 23, 24], "numpi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 24], "nunmpi": [7, 24], "nx": 4, "ny": 20, "o": [2, 3, 6, 7, 8, 9, 10, 11, 13, 18, 21, 22, 23, 24], "obei": [8, 13, 15], "object": [0, 2, 3, 6, 10, 12, 18, 23], "obliqu": [7, 24], "observ": [2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 23], "obtain": [2, 3, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 23, 24], "obviou": [7, 8, 13, 20, 24], "obviouli": 23, "obvious": [2, 6, 7, 8, 18, 23], "oc": 24, "occupi": 2, "occur": [2, 8, 10, 11, 18, 20, 23], "octob": [21, 23], "od": 2, "odd": [2, 5, 9, 23, 24], "odenum": 4, "odesi": 4, "oen": 2, "off": [3, 5, 6, 7, 11, 15, 20], "offer": [8, 13, 17, 18, 19, 21, 23], "offic": [21, 23], "offici": [19, 23], "often": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "ofter": [18, 23], "ol": [2, 15, 24], "old": [0, 3, 7, 12, 15], "ols_paramet": 1, "ols_sk": 8, "ols_svd": 8, "olstheta": [2, 7], "omega": [4, 5, 8], "omega_0": 5, "omit": [2, 7, 23, 24], "onc": [3, 8, 11, 13, 15], "one": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 20, 21, 23, 24], "onehot": 3, "onehot_vector": 3, "onehotencod": 11, "ones": [1, 2, 4, 7, 8, 10, 11, 12, 13, 15, 18, 23, 24], "ones_lik": 6, "ong": 24, "onl": 5, "onli": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "onlin": [0, 13, 19], "onto": [7, 13, 24], "open": [0, 2, 3, 6, 8, 9, 11, 17, 19, 21, 23], "oper": [0, 1, 2, 3, 5, 7, 8, 12, 13, 14, 15, 17, 20, 23, 24], "operation": 20, "oplu": 20, "opmiz": 15, "opportun": 2, "oppos": [8, 15], "opposit": [3, 7, 10, 24], "opt": [3, 7, 23], "optim": [1, 2, 4, 5, 6, 7, 8, 9, 11, 12, 13, 16], "optimis": [3, 5], "option": [0, 2, 3, 5, 7, 8, 10, 13, 18, 24], "optmiz": [3, 10, 15, 24], "oral": 23, "orang": 2, "order": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 18, 20, 23, 24], "ordinari": [2, 4, 5, 9, 13, 15, 17], "oreilli": [22, 23], "org": [1, 2, 5, 6, 17, 18, 22, 23], "organ": [8, 9, 12, 18], "orient": [3, 7, 20, 24], "origin": [0, 2, 5, 7, 8, 10, 13, 14, 15, 18, 23, 24], "orthogn": [7, 24], "orthogon": [2, 7, 8, 10, 13, 15, 18, 23, 24], "orthonorm": [7, 24], "os": [21, 23], "oscar": 3, "oscil": [5, 15], "oskar": 23, "oskarlei": 23, "oslo": [2, 17, 19, 21, 23, 24], "osx": [2, 17, 23], "other": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 15, 16, 17, 19, 20, 21, 22, 24], "otherwis": [2, 3, 6, 9, 15, 18, 23], "ouput": [7, 9, 14], "our": [0, 1, 3, 4, 5, 8, 9, 10, 11, 12, 14, 16, 17, 18, 20], "ourmodel": 2, "ourselv": [2, 7, 8, 10, 13, 15, 23, 24], "out": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "out_fil": 11, "outcom": [2, 9, 11, 12, 14, 20, 24], "outdoor": 11, "outer": [8, 14, 15], "outfil": 6, "outlier": [2, 10, 23, 24], "outlin": [8, 12, 13], "outlook": 11, "outperform": 12, "output": [2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18, 20, 23, 24], "output_bia": 3, "output_bias_gradi": 3, "output_shap": 6, "output_weight": 3, "output_weights_gradi": 3, "outputlayer1": 14, "outputlayer2": 14, "outsid": 6, "over": [0, 1, 2, 3, 5, 6, 7, 8, 11, 12, 14, 15, 18, 23, 24], "over1": 15, "overal": [3, 12], "overcast": 11, "overcom": [14, 15], "overdetermin": [2, 23], "overfit": [2, 3, 5, 8, 11, 12, 15], "overflow": 7, "overhead": 14, "overlap": [5, 9, 10, 11], "overlin": [2, 7, 8, 11, 12, 13, 16, 18, 23, 24], "overst": 2, "overtrain": 6, "overview": 5, "own": [1, 6, 7, 8, 10, 14, 15, 17, 18], "owner": 2, "ownmsepredict": 2, "ownmsetrain": 2, "ownridgetheta": [2, 8, 24], "ownypredictridg": 2, "ownytilderidg": 2, "oxid": 2, "p": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23, 24], "p0": 4, "p1": 4, "p_": [4, 6, 10, 11], "p_hidden": 4, "p_i": [7, 20], "p_j": 20, "p_n": 20, "p_output": 4, "p_x": 20, "pack": [2, 23], "packag": [0, 2, 3, 5, 6, 7, 10, 13, 15, 17, 20, 24], "packtpub": 23, "packtpublish": 23, "pad": [5, 6], "page": [2, 17, 23], "pai": [0, 2, 3, 11, 15], "pair": [2, 4, 5, 11, 17, 20, 23], "paltform": 0, "panda": [2, 6, 7, 8, 9, 11, 13, 17], "panel": 23, "paper": 3, "paradigm": [2, 23], "parallel": [12, 15, 17, 18, 23], "param": 4, "paramat": 4, "paramet": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 20], "parameter": [2, 8, 12, 23, 24], "parametr": [2, 8, 23, 24], "paramt": [5, 7], "part": [2, 3, 5, 7, 8, 12, 18, 19, 20, 21, 23, 24], "partial": [1, 2, 3, 7, 8, 9, 10, 12, 13, 14, 15, 20, 23, 24], "particip": [0, 17, 19, 21, 23], "particl": [2, 6, 15, 20, 23], "particular": [1, 2, 3, 4, 5, 7, 8, 11, 12, 13, 14, 15, 20, 22, 23, 24], "particularli": [7, 8, 10, 13, 15, 20, 24], "partit": [3, 6, 11], "partli": [8, 23], "partner": 0, "pass": [4, 5, 14, 16], "past": [12, 20], "patch": [8, 20], "path": [2, 6, 8, 9, 11, 17, 23], "patient": 9, "patter": 6, "pattern": [2, 5, 6, 14, 22, 23], "pauli": [2, 23], "pc": [0, 13, 17], "pca": [2, 9, 17, 23, 24], "pd": [2, 6, 7, 8, 9, 11, 13, 23, 24], "pde": 4, "pdf": [0, 1, 2, 5, 6, 7, 8, 11, 22, 23], "pedagog": [2, 23, 24], "penal": 8, "penalti": [8, 15], "penros": [7, 8], "pentagon": 15, "peopl": [2, 3, 11, 15, 17], "per": [2, 3, 8, 19, 21, 23], "percentag": [2, 12, 13, 21], "perceptron": [2, 3, 9, 23], "peregrin": 23, "perfect": [2, 3, 15, 23], "perfectli": [6, 8], "perform": [1, 2, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "performac": 6, "perhap": [2, 7, 15, 23, 24], "perimet": 3, "period": [3, 6, 20], "permiss": 0, "permut": 13, "persist": 15, "person": [1, 7, 8, 9, 19, 21, 23], "perspect": 22, "pertin": [14, 23], "petal": [10, 11], "peter": [22, 24], "phantom": 20, "phase": [8, 14], "phenomena": 20, "phi": 10, "phi_k": 10, "philosophi": 15, "phone": [21, 23], "photo": [6, 23], "phrase": [2, 23], "physic": [2, 3, 6, 9, 14, 15, 20, 21, 22, 23, 24], "pi": [4, 5, 7, 8, 9, 11, 14, 15, 20], "pick": [3, 11, 12, 13, 15, 16], "pickl": 3, "pictur": [2, 23], "pie": [17, 23], "piec": [13, 16], "pillow": [2, 17, 23], "pinv": [7, 8, 15, 24], "pip": [0, 2, 3, 17, 23], "pip3": [2, 3, 23], "pipelin": [2, 8, 10, 12, 24], "pippin": 23, "pit": 6, "pitfal": 8, "pitt": 14, "pixel": [3, 5, 6, 23], "pixel_height": [3, 5], "pixel_width": [3, 5], "place": [0, 2, 6, 8, 10, 15, 18, 23], "plai": [2, 5, 6, 7, 8, 10, 13, 17, 23, 24], "plain": [10, 12, 14, 15, 16], "plan": [8, 11, 21, 22, 23], "plane": [10, 11], "plateau": 7, "platform": [17, 23], "plausibl": 14, "pleas": [15, 21, 23], "plenti": 3, "plethora": [5, 14], "plot": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "plot_confusion_matrix": [9, 12], "plot_count": 8, "plot_cumulative_gain": [9, 12], "plot_data": 3, "plot_dataset": 10, "plot_decision_boundari": [11, 12], "plot_import": 12, "plot_max": 6, "plot_min": 6, "plot_model": 6, "plot_numb": 6, "plot_predict": 10, "plot_regression_predict": 11, "plot_result": 6, "plot_roc": [9, 12], "plot_surfac": [4, 8, 15], "plot_train": 11, "plot_tre": [11, 12], "plt": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "plu": [2, 5, 7, 9, 23, 24], "pm": 10, "pmatrix": 4, "pml": 22, "pn": 5, "png": [2, 6, 8, 9, 11, 23], "point": [2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 23, 24], "point_1": 6, "point_2": 6, "poisson": [17, 20, 23], "poli": [8, 10], "poly100_kernel_svm_clf": 10, "poly3": 2, "poly3_plot": 2, "poly_featur": [0, 10, 11], "poly_features10": 11, "poly_fit": 11, "poly_fit10": 11, "poly_kernel_svm_clf": 10, "poly_model": 0, "poly_ms": 0, "poly_predict": 0, "polydegre": [2, 7, 8, 12, 24], "polygon": 15, "polym": 14, "polynomi": [0, 2, 7, 8, 9, 10, 11, 12, 13, 23, 24], "polynomial_featur": [0, 1, 8], "polynomial_svm_clf": 10, "polynomialfeatur": [0, 1, 2, 8, 10, 11, 24], "polytrop": [2, 8], "pool": 5, "pool_siz": 5, "poor": [3, 15], "poorli": [2, 24], "popul": [2, 7, 23, 24], "popular": [0, 2, 3, 5, 8, 9, 10, 11, 13, 14, 17, 18, 20, 24], "popularli": [2, 23], "portabl": 12, "portion": [13, 15], "pose": [2, 6, 7, 8, 13, 20, 23], "posit": [2, 3, 4, 5, 7, 9, 10, 12, 13, 15, 16, 18, 20, 23, 24], "possibl": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 21, 23, 24], "possibli": [8, 10, 15], "posterior": 7, "postpon": [2, 24], "postul": 7, "potenti": [2, 5, 7, 8, 14, 15], "pott": 14, "power": [2, 3, 7, 8, 10, 11, 14, 15, 23, 24], "pp": [7, 8], "practic": [1, 2, 7, 8, 9, 10, 20, 24], "practition": [2, 3, 5, 23], "pre": 23, "preced": [3, 13, 14, 20], "preceed": 6, "preceq": 10, "precis": [2, 4, 7, 13, 15, 18, 20, 23, 24], "pred": 8, "predicit": 2, "predict": [0, 1, 2, 3, 7, 8, 9, 10, 11, 12, 17, 22, 23, 24], "predict_prob": 3, "predict_proba": [9, 12], "predictor": [2, 7, 8, 9, 11, 12, 13, 23, 24], "prefer": [0, 2, 3, 8, 10, 11, 13, 15, 17, 23], "prepar": [2, 8, 18, 23, 24], "preprocess": [0, 1, 2, 6, 8, 9, 10, 11, 12, 13], "prerequisit": 2, "presenc": 15, "present": [2, 7, 8, 9, 11, 14, 15, 18, 20, 23, 24], "preserv": [5, 13, 18], "press": [0, 15, 22], "pretrain": [3, 6], "pretti": [2, 6, 10, 11, 17, 23], "prev_centroid": 16, "prevent": [15, 20], "previou": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 18, 20, 24], "previous": [4, 5, 11, 12, 20], "price": [2, 6, 11, 15], "primal": 10, "primari": [2, 9, 23], "prime": 20, "princip": [2, 7, 9, 17, 23, 24], "principl": [2, 8, 9, 10, 16, 23], "print": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 23, 24], "print_funct": [10, 11], "printout": [2, 23], "prior": [2, 7, 8, 23], "privat": 2, "prob": [3, 20], "probabilist": [2, 22, 23, 24], "probabl": [2, 3, 5, 6, 8, 9, 12, 15, 17, 23, 24], "problem": [2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 20], "probml": 22, "proce": [2, 7, 8, 9, 10, 11, 12, 13, 15, 18, 23, 24], "procedur": [4, 6, 7, 8, 10, 12, 13, 15, 24], "proceed": 18, "process": [2, 4, 6, 8, 11, 12, 14, 15, 17, 18, 20, 22, 23], "prod": 22, "prod_": [3, 7, 9], "produc": [2, 5, 6, 7, 8, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "product": [1, 2, 3, 5, 7, 8, 9, 10, 14, 15, 17, 18, 23, 24], "profess": [2, 23], "program": [0, 2, 3, 6, 7, 8, 10, 14, 16, 17, 18, 19, 20, 21, 23, 24], "programm": 18, "progress": [3, 6, 16], "prohibit": 8, "project": [0, 2, 3, 4, 5, 7, 13, 15, 17, 19, 24], "project_root_dir": [2, 8, 9, 11, 23], "promin": 14, "promis": 10, "promot": [21, 23], "prone": [0, 11], "pronounc": [15, 17, 23], "proof": [2, 13, 14, 15, 23], "propag": [4, 5, 15], "proper": [2, 4, 8, 9], "properli": [3, 8, 10, 12, 15], "properti": [1, 2, 3, 5, 14, 15, 18, 23], "proport": [2, 3, 7, 11, 13, 15, 20, 23, 24], "propos": [3, 6, 8, 12, 23], "propto": [7, 15], "proton": [2, 23], "prove": [5, 15], "provid": [2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 15, 17, 18, 20, 23, 24], "proxi": [3, 15], "prune": 11, "pseudo": [18, 20], "pseudoinv": 7, "pseudoinvers": [7, 8], "pseudorandom": [8, 20], "psychologi": [2, 23], "pt": 15, "public": [0, 2, 17, 23], "pull": 0, "punish": [2, 3, 23], "pure": [5, 11, 20], "purest": 11, "puriti": 11, "purpos": [2, 5, 12, 14, 16, 23], "push": 0, "put": 3, "py": 7, "pycod": 23, "pydata": 17, "pydot": 11, "pyhton2": 23, "pylab": [9, 23], "pypi": 17, "pyplot": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "pythagora": 7, "python": [3, 4, 5, 7, 8, 10, 13, 14, 15, 16, 20, 24], "python2": 2, "python3": [2, 17, 23], "pytorch": [2, 17, 23], "q": [7, 8, 10, 13, 20], "qp": 10, "qquad": [4, 13, 15, 18], "qr": [7, 8, 18, 24], "quad": [3, 15, 18], "quadrat": [2, 10, 11, 15, 23], "qualit": [6, 11, 20], "qualiti": [2, 11, 17, 23, 24], "quantifi": 3, "quantil": 12, "quantit": [2, 8, 11, 23], "quantiti": [1, 2, 4, 7, 8, 9, 11, 12, 13, 14, 16, 18, 20, 23, 24], "quantum": [6, 14, 22, 23], "quartil": [2, 24], "quench": 7, "queri": 11, "question": [2, 7, 8, 11, 13, 14, 15, 21, 23, 24], "qugan": 6, "quick": [6, 20], "quickli": [3, 5, 11, 13, 15], "quit": [0, 3, 7, 8, 11, 12, 14, 24], "quot": 6, "r": [0, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 24], "r2": [2, 7, 8, 23, 24], "r2_score": [2, 23], "r2score": [2, 23], "r_1": 11, "r_2": 11, "r_j": 11, "r_m": 11, "rad": 2, "radial": [2, 10, 14], "radioact": 20, "radiu": [2, 3, 24], "rain": 11, "ramp": 3, "ran0": 20, "ran1": 20, "ran2": 20, "ran3": 20, "rand": [0, 2, 6, 7, 8, 11, 12, 15, 18, 23, 24], "randint": [8, 11, 15], "randn": [0, 2, 3, 4, 7, 8, 11, 13, 15, 23, 24], "random": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 15, 16, 17, 18, 23, 24], "random_forest_model": 12, "random_index": 15, "random_indic": [3, 5], "random_st": [2, 9, 10, 11, 12, 13], "randomforestclassifi": 12, "randomli": [3, 8, 11, 15, 16], "rang": [2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "rangl": [2, 8, 13, 20, 23, 24], "rangle_x": 20, "rank": [7, 24], "rankdir": 6, "raphson": [3, 10, 15], "rapidli": 2, "rare": [3, 15], "raschka": [23, 24], "rasckha": 23, "rate": [2, 3, 4, 5, 6, 10, 11, 12, 14, 15], "rather": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "ratio": [6, 9, 11, 12, 13], "rational": [2, 23], "ravel": [7, 8, 9, 10, 11, 12, 13, 15, 18], "raw": 5, "rbf": [10, 13, 14], "rbf_kernel_svm_clf": 10, "rbf_pca": 13, "rc": [2, 20], "rcond": [2, 23, 24], "rcparam": [3, 5, 9, 10, 11, 12, 20, 23], "re": [0, 4, 6, 15], "reach": [3, 6, 7, 8, 11, 12, 14, 15, 16], "read": [1, 2, 4, 5, 6, 7, 8, 9, 10, 13, 14, 18, 20, 22], "read_csv": [2, 8, 9, 11], "read_fwf": [2, 23], "reader": [2, 8, 18, 20, 23], "readi": [2, 3, 7, 8, 10, 12, 13, 14, 18, 23], "readili": 3, "readm": 0, "readthedoc": 17, "real": [1, 2, 3, 6, 9, 12, 13, 14, 18, 24], "real_loss": 6, "real_output": 6, "realist": [10, 23], "realiti": 20, "realiz": [3, 14], "realli": [2, 3, 23], "rearrang": 15, "reason": [2, 3, 5, 6, 12, 15, 22, 23], "reassign": 3, "recal": [7, 8, 11, 12, 13, 14, 18, 20, 23, 24], "recast": 5, "receiv": [3, 5, 12, 14, 20], "recent": [2, 8, 15, 22], "recept": [5, 14], "receptive_field": 5, "recip": [2, 8, 9, 18, 23, 24], "reciproc": 7, "recogn": [2, 6, 7, 12, 23], "recognit": [2, 3, 5, 14, 22, 23], "recommen": 23, "recommend": [0, 2, 4, 5, 6, 7, 8, 10, 15, 17, 18, 22], "reconsid": 11, "reconstruct": 13, "record": [12, 19, 21, 23], "recreat": 0, "rectangl": [11, 15], "rectangular": [7, 24], "rectifi": [3, 5, 14], "recur": [2, 17, 23], "recurr": [2, 3, 17, 23], "recurs": [11, 17, 18, 23], "red": [2, 5, 6, 8, 10, 11], "redefin": [2, 12, 23, 24], "reduc": [3, 5, 7, 8, 11, 12, 13, 15, 23], "reduct": [2, 12, 13, 17, 20, 23, 24], "refer": [2, 3, 4, 5, 7, 8, 13, 14, 15, 16, 18, 22, 23, 24], "referenc": 4, "refin": 14, "refit": 8, "reflect": [2, 3, 6, 7, 20, 23], "refresh": [17, 23], "refreshprogrammingskil": 23, "reg": [12, 13], "regard": [3, 11, 15], "regardless": [1, 14], "region": [5, 6, 8, 11, 14], "regist": [8, 20], "reglasso": 7, "regr_1": [2, 11], "regr_2": [2, 11], "regr_3": [2, 11], "regress": [1, 3, 10, 13, 14, 17, 18], "regressor": [2, 9, 12], "regridg": [2, 7, 8, 24], "regular": [2, 5, 6, 7, 8, 9, 11, 15, 21, 23, 24], "regularli": 0, "reilli": [2, 22, 23], "reinforc": [2, 10, 17, 23], "reiter": 3, "reject": 9, "rel": [2, 6, 8, 9, 11, 14, 15, 20, 23, 24], "relat": [2, 3, 5, 6, 7, 13, 15, 16, 18, 20, 23, 24], "relationship": [2, 6, 11, 23], "relativeerror": [2, 23, 24], "releas": [3, 17, 23], "relev": [2, 3, 7, 9, 13, 17, 20, 23], "reli": [2, 8, 10], "reliabl": [9, 20], "relu": [5, 6, 23], "remain": [3, 4, 6, 8, 14, 18, 20], "remaind": 20, "reman": 4, "remark": 3, "rememb": [2, 10, 15, 18, 23], "remind": [2, 7, 13, 15, 18, 20], "remot": 0, "remov": [2, 6, 7, 8, 24], "renam": 0, "render": [2, 23, 24], "reorder": [7, 9, 24], "reorgan": [2, 23], "repeat": [2, 3, 5, 6, 7, 8, 11, 12, 13, 15, 16, 18, 20, 23, 24], "repeated": 23, "repeatedli": [2, 8, 12, 15], "repet": 5, "repetit": [8, 23, 24], "rephras": 15, "replac": [2, 3, 5, 6, 7, 8, 12, 14, 16, 17, 23, 24], "replica": 8, "repo": 0, "report": 23, "repositori": [2, 6, 23], "repres": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 20, 23, 24], "represent": [2, 3, 5, 8, 20, 23], "representd": 5, "reproduc": [0, 1, 2, 7, 8, 11, 14, 17, 20, 23, 24], "repuls": [2, 23], "request": [2, 15], "requir": [0, 2, 3, 5, 6, 7, 8, 10, 11, 13, 14, 15, 18, 23, 24], "res1": 4, "res2": 4, "res3": 4, "res_analyt": 4, "res_analytical1": 4, "res_analytical2": 4, "res_analytical3": 4, "resaml": 8, "resampl": [2, 9, 12, 17, 23, 24], "rescal": [2, 13, 14, 24], "rescu": 7, "reseach": 8, "research": [2, 6, 15, 17, 22, 23], "resembl": [8, 20], "reserv": [3, 7, 8, 20], "reshap": [2, 3, 4, 5, 6, 8, 10, 11, 12, 18, 23, 24], "residenti": 2, "residu": [2, 7, 15, 23], "resiz": [7, 24], "resourc": 23, "respect": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 16, 20, 23, 24], "respond": 14, "respons": [2, 9, 11, 14, 23, 24], "rest": [2, 7, 24], "restat": [2, 14, 23], "restor": 6, "restored_discrimin": 6, "restored_gener": 6, "restrict": [2, 5, 11, 14, 23], "result": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23], "retail": 2, "retain": [7, 8, 24], "return": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 18, 20, 23, 24], "return_data": 16, "return_sequ": 6, "return_x_i": 11, "reus": [3, 5, 8], "reveal": [2, 14, 23], "revers": [3, 18], "review": [17, 18], "revisit": 16, "revolut": 23, "reward": [2, 6, 23], "rewrit": [1, 2, 5, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20], "rewritten": [4, 8, 10, 12, 20], "rewrot": 15, "rf": 12, "rgb": 5, "rgoj5yh7evk": 17, "rh": 8, "rho": [2, 12, 15], "rho_1": 12, "rho_2": 12, "rho_m": 12, "rich": [2, 23], "ride": 11, "rideclass": 11, "ridedata": 11, "ridg": [9, 13, 15, 17, 23], "ridge_sk": 8, "ridgetheta": 7, "right": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 23, 24], "right_sid": 4, "rightarrow": [2, 3, 7, 8, 10, 13, 14, 15, 20, 23, 24], "rigor": [2, 23, 24], "ring": 8, "rise": [2, 23], "risk": [2, 15, 23], "rival": 6, "river": 2, "rlm": 23, "rm": [2, 20], "rmse": 2, "rmsporp": 15, "rmsprop": [3, 5, 6, 15], "rnd_clf": 12, "rng": 20, "rnn": [6, 14], "rnn1": 6, "rnn2": 6, "rnn_2layer": 6, "rnn_input": 6, "rnn_output": 6, "rnn_train": 6, "rntrick1": 20, "rntrick2": 20, "rntrick3": 20, "rntrick4": 20, "ro": [2, 15, 23], "robert": 22, "robust": [2, 23], "robustscal": [2, 24], "roc": [9, 12], "role": [2, 4, 7, 8, 10, 17, 23, 24], "roll": 8, "room": [2, 21, 23], "root": [0, 2, 7, 11, 15, 20, 24], "rot": 23, "rotat": [3, 10, 11, 12], "rotation_matrix": 11, "roughli": [3, 5], "round": [2, 9, 11, 15], "routin": [15, 18, 23], "row": [1, 2, 3, 4, 7, 8, 11, 13, 18, 23, 24], "rr": [7, 24], "rrr": [7, 24], "rug": 15, "rule": [2, 3, 7, 8, 15, 23, 24], "run": [0, 2, 3, 4, 6, 7, 8, 10, 11, 13, 15, 17, 23, 24], "runtim": [0, 3, 8, 16], "rust": [2, 17, 18, 23], "rvert": 3, "rvert_2": 3, "s_": [5, 8], "s_1": 8, "s_i": [8, 9], "s_j": 8, "s_k": 8, "saddl": 15, "sai": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 20, 23, 24], "said": [8, 11, 15], "sake": [2, 7, 9, 13, 23, 24], "sale": [2, 23], "sam": 23, "same": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 16, 18, 20, 23, 24], "samm": 12, "sampl": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 16, 17, 18, 20, 23, 24], "sample_vari": 16, "sampleexptvari": 20, "samwis": 23, "sastri": 13, "satisfactori": [2, 23], "satisfi": [3, 4, 5, 8, 10, 15, 18, 20], "satur": [3, 8], "save": [2, 6, 8, 9, 11, 15, 23], "save_fig": [2, 8, 9, 11, 12, 23], "savefig": [2, 6, 8, 9, 11, 20, 23], "savetxt": 6, "saw": [7, 24], "scalabl": 12, "scalar": [4, 7, 8, 12, 24], "scale": [2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 21, 23], "scale_mean": 6, "scale_std": 6, "scaler": [2, 9, 10, 11, 12, 13, 24], "scan": [7, 9], "scari": 7, "scatter": [0, 2, 3, 8, 9, 10, 11, 16, 23], "scenario": [8, 15], "schedul": 15, "scheme": [3, 15], "schrage": 20, "sch\u00f8yen": 8, "scienc": [2, 3, 12, 14, 15, 17, 19, 20, 21, 22], "scientif": [2, 17, 23], "scientist": [2, 23], "scikit": [0, 1, 5, 7, 8, 10, 11, 12, 15, 17, 18, 22], "scikit_learn": 2, "scikitlearn": 23, "scikitplot": [9, 12], "scipi": [2, 5, 7, 8, 15, 17, 18, 23, 24], "scl": 8, "scm": 0, "score": [0, 1, 2, 3, 5, 8, 9, 11, 12, 13, 21, 23, 24], "scores_kfold": 8, "scratch": [1, 3, 15], "sdg": 15, "seaborn": [2, 3, 5, 8, 9, 23], "seamless": [2, 17, 23], "search": [0, 2, 3, 5, 7, 11, 15, 23], "sebastian": 23, "sebastianraschka": 23, "sec": 8, "second": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 16, 17, 18, 20, 21, 23, 24], "secondeigvector": 13, "secondli": 14, "section": [1, 6, 13, 18, 20, 24], "sector": 2, "see": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 18, 20, 23, 24], "seed": [2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 15, 16, 20, 23, 24], "seed_imag": 6, "seek": [3, 4, 10], "seem": [3, 5, 6], "seemingli": [2, 23], "seen": [2, 3, 5, 7, 12, 14, 20], "segment": 15, "seismic": 8, "seldomli": [2, 23], "select": [0, 3, 7, 8, 10, 11, 12, 13, 19, 20, 21, 22, 23, 24], "selevet": 0, "self": [3, 7, 24], "sell": 6, "semest": [9, 19], "semi": [10, 15], "semilogx": 8, "send": [7, 14, 15, 21, 23], "senior": [19, 21], "sens": [2, 6, 8, 10, 23], "sensibl": 5, "sensit": [2, 7, 8, 11, 15, 23, 24], "sent": 4, "sentenc": [6, 14], "separ": [2, 3, 4, 6, 8, 10, 11, 14, 16, 17, 20, 23], "septemb": 23, "sequenc": [5, 6, 9, 11, 12, 14, 15, 17, 18, 20, 23], "sequenti": [3, 5, 6, 12, 14, 20], "seri": [2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 18, 23, 24], "serif": [9, 20, 23], "serv": [2, 3, 4, 5, 7, 9, 15, 22, 23, 24], "session": [0, 3, 19, 21, 23], "set": [1, 3, 6, 7, 8, 9, 10, 12, 13, 15, 16, 17, 18, 20, 21], "set_major_formatt": 8, "set_major_loc": 8, "set_tick": [3, 10], "set_ticklabel": 3, "set_titl": [2, 3, 4, 5, 9, 14, 16, 23], "set_xlabel": [2, 3, 4, 5, 9, 14, 23], "set_xlim": [9, 14], "set_xticklabel": 3, "set_ylabel": [2, 3, 4, 5, 9, 23], "set_ylim": [9, 14], "set_ytick": 9, "set_yticklabel": [3, 8], "set_zlim": 8, "seth": 6, "setminu": 8, "setosa": [10, 11], "setosa_or_versicolor": 10, "setp": 8, "setup": [3, 6, 8, 10, 17, 23, 24], "sever": [1, 2, 5, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23, 24], "sgd": [3, 5], "sgd_clf": 10, "sgdclassifi": 10, "sgdreg": 15, "sgdregressor": 15, "sgn": [7, 24], "shallow": 15, "shape": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 23, 24], "share": [0, 3, 5, 23], "shareabl": 0, "she": 9, "shift": [0, 3, 8, 14, 20], "ship": 5, "shire": 23, "short": [6, 7], "shortcom": 15, "shorten": 6, "shorter": 20, "shorthand": 23, "shortli": [18, 23], "should": [0, 2, 4, 5, 7, 8, 10, 11, 13, 14, 18, 20, 23, 24], "show": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "show_shap": 6, "shown": [2, 6, 7, 10, 14, 15, 18, 24], "shrink": [5, 7, 8, 10, 13, 24], "shrinkag": [7, 8, 24], "shrunk": 13, "shuffl": [2, 3, 6, 8, 15, 24], "side": [2, 4, 7, 10, 14, 15, 18, 23], "sigh": [17, 23], "sigma": [2, 3, 7, 8, 9, 12, 13, 14, 15, 18, 20, 23, 24], "sigma0": 20, "sigma1": 20, "sigma2": 20, "sigma_": [7, 18, 23, 24], "sigma_0": [7, 24], "sigma_1": [7, 24], "sigma_2": [7, 24], "sigma_fn": [9, 14], "sigma_i": [2, 7, 23, 24], "sigma_j": [7, 24], "sigma_m": [8, 20], "sigma_n": [13, 20], "sigma_t": 15, "sigma_x": 20, "sigmoid": [3, 4, 6, 9, 10, 12, 14], "sigmundson": 8, "sign": [3, 4, 9, 10, 12, 20, 21], "signal": [3, 5, 12, 14], "signifi": 6, "signific": 3, "significantli": [3, 15, 20], "sim": [6, 7, 8, 15, 20], "similar": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 23], "similarli": [2, 3, 5, 7, 10, 12, 15, 20, 23, 24], "simpl": [1, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20], "simplepredict": 12, "simpler": [1, 2, 3, 7, 8, 9, 15, 17, 23], "simplernn": 6, "simplest": [2, 3, 5, 6, 11, 12, 14, 16, 23], "simpletre": 12, "simpli": [2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 17, 18, 20, 23, 24], "simplic": [4, 7, 8, 9, 10, 11, 12, 13, 14, 16, 24], "simplicti": [7, 24], "simplifi": [2, 8, 11, 17, 23], "simplist": [5, 8, 20], "simul": 8, "simultan": 8, "sin": [2, 3, 4, 5, 6, 11, 14, 15, 18, 23], "sinc": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 22, 23, 24], "sine": [5, 14], "singl": [2, 3, 4, 5, 7, 8, 9, 10, 11, 14, 15, 18, 20, 23, 24], "singular": [2, 8, 15, 18, 23], "sinusoid": 5, "site": [2, 19], "situat": [2, 6, 7, 9, 15, 20, 23, 24], "six": [5, 20], "size": [2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 18, 20, 23], "sketch": 12, "ski": 11, "skill": 2, "skip": 13, "skl": [2, 8, 23, 24], "sklearn": [0, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 15, 16, 23, 24], "skplt": [9, 12], "sl": 8, "slack": 10, "slice": [4, 18, 23], "slide": [1, 2, 5, 20, 23, 24], "slight": [8, 15], "slightli": [3, 4, 5, 7, 8, 9, 12, 20, 24], "slope": [10, 13, 14], "slow": [2, 4, 10, 15, 24], "slower": [7, 18, 23, 24], "slowest": 18, "slowli": 14, "slp": 3, "small": [2, 3, 4, 5, 7, 8, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "smaller": [2, 3, 4, 7, 8, 10, 11, 13, 15, 20, 23], "smallest": [2, 6, 16, 23], "smallest_row_index": 16, "smooth": [2, 5, 8, 15, 23], "sn": [2, 3, 5, 8, 9, 23], "sne": 13, "so": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23, 24], "soar": 8, "social": 2, "soft": [3, 9, 12, 14], "soften": 10, "softmax": [5, 9], "softwar": [2, 10, 17, 18], "sol": 10, "sole": [2, 8, 23], "solid": [2, 9], "solut": [2, 3, 4, 5, 7, 8, 10, 12, 13, 15, 18, 20, 23, 24], "soluton": 4, "solv": [1, 2, 3, 5, 7, 8, 10, 12, 13, 14, 15, 18, 23, 24], "solve_expdec": 4, "solve_ode_deep_neural_network": 4, "solve_ode_neural_network": 4, "solve_pde_deep_neural_network": 4, "solveod": 4, "solveode_popul": 4, "solver": [4, 9, 10, 11, 12, 18, 23], "some": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 20, 23], "some_model": 8, "somehow": 6, "someon": 1, "someth": [0, 2, 3, 5, 6, 9, 11, 13, 20, 23, 24], "sometim": [2, 3, 13, 14, 15, 16, 24], "soon": [18, 21, 24], "sophist": [2, 23], "sopt": 15, "sort": [7, 8, 11, 13, 20], "sound": [5, 7], "sourc": [2, 3, 5, 8, 17, 18, 20, 23], "space": [2, 3, 6, 7, 10, 11, 13, 14, 15, 16, 20, 24], "span": [2, 5, 7, 11, 13, 18, 23, 24], "spare": 3, "spars": [5, 8, 18, 23], "sparse_mtx": [18, 23], "sparsecategoricalcrossentropi": 5, "sparsiti": 12, "spatial": [3, 4, 5, 14], "speak": 20, "special": [8, 9, 12, 14, 15, 18, 20, 23], "specif": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 17, 18, 20, 22, 23, 24], "specifi": [2, 5, 7, 8, 9, 11, 13, 15, 16, 20, 23], "specifici": [2, 12, 23], "spectacular": 5, "spectral": 3, "speech": [2, 3, 5, 6, 14], "speed": [3, 4, 6, 15], "spend": [1, 20], "sphere": [2, 24], "spin": 8, "spite": 2, "spline": 10, "split": [1, 3, 5, 6, 7, 8, 10, 11, 12, 13, 16, 20, 23], "splite": 2, "splitter": [3, 12], "spontan": 20, "spot": 5, "spread": [2, 13, 20, 23, 24], "springer": [22, 23], "spuriou": 15, "sqrsignal": 5, "sqrt": [2, 5, 6, 7, 8, 10, 12, 13, 15, 20, 24], "squar": [0, 3, 4, 5, 6, 9, 10, 11, 13, 15, 16, 17, 18, 20], "squarederror": 12, "squaredeuclidean": 16, "squash": 14, "srtm": 8, "srtm_data_norway_1": 8, "stabil": 7, "stabl": [1, 2, 6, 7, 8, 11, 17, 23, 24], "stack": [5, 6], "stage": [0, 7, 15], "stai": [2, 4, 6, 7, 13, 23, 24], "stand": [2, 7, 11, 14, 23, 24], "standard": [2, 3, 6, 7, 8, 9, 10, 12, 14, 18, 20, 23], "standardscal": [2, 8, 9, 10, 11, 12, 13, 24], "stanford": 15, "start": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23, 24], "start_tim": 16, "stat": 8, "state": [3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 15, 17, 20, 23, 24], "statement": [2, 9, 18, 23], "statist": [2, 3, 5, 6, 9, 11, 12, 13, 14, 15, 16, 18, 22, 24], "statu": [0, 2, 9, 23], "stavang": 8, "std": [2, 6, 8, 23, 24], "steep": 15, "step": [0, 2, 3, 4, 6, 8, 9, 11, 12, 13, 14, 15, 16, 18, 23], "step_fn": [9, 14], "step_length": 15, "steps_list": 11, "stereo": 5, "still": [2, 4, 5, 7, 8, 13, 15, 20, 24], "stimuli": 14, "stk": [22, 23], "stk2100": [22, 23], "stk3155": [0, 19, 21], "stk4021": [22, 23], "stk4051": [22, 23], "stk4155": [19, 21], "stk5000": 22, "stochast": [2, 3, 7, 8, 10, 13, 14], "stock": 6, "stoke": 14, "stone": [2, 9], "stop": [3, 6, 11, 15, 16], "storag": [7, 24], "store": [2, 3, 4, 5, 8, 13, 15, 20, 23], "storehaug": [21, 23], "str": [3, 5, 6], "straight": [2, 8, 10, 15, 23], "straightforward": [2, 4, 5, 7, 8, 10, 11, 12, 15, 18, 23, 24], "strategi": [2, 3, 11, 23], "stratifi": 8, "strength": [2, 7, 16, 24], "stretch": 13, "strict": [10, 15], "strictli": [10, 15], "stride": [6, 18], "strike": 8, "string": 3, "stroke": 9, "strong": [5, 8, 11, 12, 14, 18, 20], "strongli": [0, 2, 10, 17, 18], "stronli": 2, "structur": [2, 3, 4, 5, 8, 11, 12, 14, 17, 23], "stuck": [3, 15], "student": [0, 2, 19, 21, 22, 23], "studi": [2, 5, 6, 7, 8, 9, 10, 13, 14, 15, 17, 22, 23, 24], "studier": 22, "style": [9, 11, 18, 23], "st\u00f8land": 21, "sub": [11, 14], "subdivid": [2, 18, 23], "subfield": 2, "subject": [8, 10, 20], "submit": 23, "subplot": [2, 3, 5, 6, 8, 9, 10, 11, 12, 16, 23], "subplots_adjust": [10, 20], "subprogram": [18, 23], "subract": [2, 24], "subroutin": [2, 23], "subscript": 3, "subsequ": [3, 6, 7, 8, 14, 18, 20, 24], "subset": [3, 8, 11, 14, 15, 17, 23], "subspac": [2, 10, 13, 24], "substanti": [11, 12], "substep": 13, "substitut": [1, 5, 8, 14, 18], "subsubset": 11, "subtask": 8, "subtl": 3, "subtract": [2, 6, 7, 8, 13, 15, 18, 20, 24], "subtre": 11, "succeed": [2, 6, 23], "success": [5, 9, 11, 15, 20], "successfulli": [6, 11], "sudo": [2, 17, 23], "suffer": [2, 3, 4, 7, 12, 23, 24], "suffici": [3, 8, 10, 13, 15], "suggest": [3, 15, 22], "suit": [10, 14], "suitabl": [0, 2, 20, 24], "sum": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "sum_": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "sum_i": [2, 4, 7, 8, 10, 15, 24], "sum_j": 8, "sum_ja_": 2, "sum_k": [8, 10, 14, 18], "sum_logist": 15, "sum_m": 5, "sum_n": 5, "sum_nx_": 5, "summar": [7, 8, 11], "summari": [3, 5, 6, 12, 19], "summat": [1, 2, 5, 24], "sunni": 11, "super": [7, 24], "superfici": 5, "superscript": [3, 14], "supervis": [2, 7, 8, 9, 11, 14, 17, 23, 24], "supplement": 9, "support": [2, 3, 11, 12, 13, 15, 17, 23, 24], "suppos": [2, 7, 8, 9, 10, 12, 13, 14, 15, 18, 23, 24], "suppress": [7, 15], "sure": [1, 2, 3, 6, 8], "surf": 8, "surfac": [2, 8, 23], "surpass": 8, "surpris": [2, 23], "surround": [5, 17], "survei": [2, 7, 8, 23], "svc": [10, 11, 12], "svd": [2, 8, 13, 23], "svdinv": 7, "svm": [10, 11, 12, 13], "svm_clf": [10, 12], "swath": [7, 24], "switch": 2, "sy": 15, "symbol": [3, 7, 13, 15, 17, 20, 23, 24], "symmeteri": 3, "symmetr": [2, 7, 10, 13, 14, 15, 18, 23, 24], "symmetri": 8, "sympi": [2, 17, 23], "synonim": 20, "syntax": 15, "system": [0, 2, 3, 5, 6, 8, 9, 11, 12, 14, 15, 17, 18, 23], "systemat": [6, 8], "t": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23], "t0": [5, 8, 15], "t1": [4, 15], "t2": 4, "t3": 4, "t_": 4, "t_0": [4, 11, 15], "t_1": 15, "t_b": 12, "t_i": [3, 4, 7, 14, 24], "t_j": 14, "t_k": 11, "tabl": [11, 20, 21, 23], "tabul": [2, 23], "tabular": 23, "tackl": 6, "tag": [4, 5, 6, 7, 8, 9, 14, 15, 16, 18, 20, 24], "taht": [2, 23], "tail": 20, "tailor": [4, 10, 13, 23], "taiwan": [2, 23], "take": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 23, 24], "taken": [2, 3, 5, 8, 12, 15, 18], "tan": 5, "tangent": [3, 6, 14, 15], "tanh": [3, 6, 9, 10, 14], "target": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 23, 24], "target_nam": 11, "task": [2, 3, 5, 8, 11, 13, 14, 16, 23], "tau": [5, 7, 20], "taught": 23, "tax": 2, "taylor": [4, 15], "taylornr": 15, "tc": 10, "teach": [0, 19, 23], "team": 3, "teaser": 2, "technic": [2, 7, 8, 15], "techniqu": [2, 3, 10, 12, 15, 17, 20, 22, 23, 24], "technologi": [2, 3], "tell": [1, 2, 6, 8, 12, 13, 15, 20], "temp": 3, "temp1": 3, "temp2": 3, "temperatur": [2, 11, 23], "temporarili": 3, "ten": [5, 23], "tend": [5, 7, 8, 10, 11, 12, 14, 15, 16], "tendenc": [2, 23], "tension": 8, "tensor": 5, "tensorflow": [2, 4, 6, 10, 16, 17, 18, 22, 23, 24], "term": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 23, 24], "term1": [7, 8, 13], "term2": [7, 8, 13], "term3": [7, 8, 13], "term4": [7, 8, 13], "termin": [0, 2, 6, 7, 11, 12, 15, 24], "terminarl": 0, "terrain": 8, "terrain1": 8, "test": [1, 5, 6, 7, 8, 9, 10, 11, 12, 15, 18, 20, 23], "test_acc": 5, "test_accuraci": [3, 5], "test_error": 8, "test_imag": [5, 6], "test_ind": 8, "test_input": 6, "test_label": [5, 6], "test_loss": 5, "test_pr": 3, "test_predict": 3, "test_rnn": 6, "test_scor": [9, 12], "test_siz": [0, 2, 3, 5, 7, 8, 12, 24], "test_split": 11, "testerror": [2, 8, 24], "testi": 6, "testpredict": 6, "testx": 6, "text": [0, 2, 3, 4, 6, 7, 10, 11, 13, 15, 18, 20, 22, 24], "textbook": [1, 24], "textual": 11, "textur": 3, "tf": [3, 5, 6, 15, 16], "th": [2, 3, 4, 7, 8, 9, 11, 14, 15, 16, 18, 20, 23, 24], "than": [2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 17, 20, 23, 24], "thank": [6, 8], "theano": [3, 17, 23], "thei": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 23, 24], "them": [2, 3, 5, 6, 8, 10, 11, 12, 13, 14, 15, 18, 23, 24], "theme": [0, 2, 23], "themselv": [2, 20, 23], "thenc": 8, "theorem": [4, 8, 9, 24], "theoret": [2, 6, 12], "theori": [2, 3, 5, 10, 11, 14, 15, 17, 22, 23], "thereaft": [2, 7, 8, 13, 14, 18, 23], "therebi": [2, 7, 9, 13, 23, 24], "therefor": [2, 3, 4, 5, 6, 8, 9, 10, 13, 15, 20, 23, 24], "therein": 13, "thereof": [2, 8, 15, 23], "theta": [1, 2, 3, 6, 7, 8, 9, 15, 20, 23, 24], "theta_": [2, 3, 8, 9, 15, 23, 24], "theta_0": [1, 2, 7, 8, 9, 23, 24], "theta_0x_": [2, 23, 24], "theta_1": [2, 7, 8, 9, 23, 24], "theta_1x_": [2, 23, 24], "theta_1x_0": [2, 23], "theta_1x_1": [2, 9, 23], "theta_1x_2": [2, 23], "theta_1x_i": [9, 24], "theta_2": [2, 23, 24], "theta_2x_": [2, 23, 24], "theta_2x_0": [2, 23], "theta_2x_1": [2, 23], "theta_2x_2": [2, 9, 23], "theta_2x_i": 24, "theta_3x_i": 24, "theta_4x_i": 24, "theta_i": [2, 3, 7, 23, 24], "theta_j": [2, 7, 8, 23, 24], "theta_linreg": 15, "theta_p": 9, "theta_px_p": 9, "theta_t": 15, "thetavalu": 7, "thi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24], "thing": [0, 1, 2, 3, 4, 6, 7, 9, 11, 20, 23], "think": [2, 3, 5, 6, 8, 11, 14, 15, 16, 20, 23, 24], "third": [2, 5, 8, 15, 21, 23], "thirti": 9, "thorughout": 23, "those": [2, 5, 7, 8, 10, 11, 12, 13, 18, 23, 24], "though": [1, 3, 4, 5, 6, 15, 18, 20], "thought": [8, 16, 20], "thousand": [2, 3, 24], "three": [2, 3, 5, 7, 8, 10, 11, 14, 18, 19, 20, 21, 23, 24], "threshold": [3, 5, 11, 12, 13, 14, 15], "through": [0, 2, 3, 4, 5, 6, 7, 8, 10, 13, 14, 15, 16, 17, 18, 20, 23, 24], "throughout": [0, 2, 6, 7, 16, 17, 18, 20, 23], "throw": [5, 8, 20], "thu": [2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 21, 23, 24], "thumb": [2, 8, 24], "thursdai": [], "tibshirani": [8, 22, 23], "tick_param": 8, "ticker": [8, 15, 20], "tif": 8, "tight_layout": [3, 9], "tightli": 13, "tild": [2, 7, 8, 9, 13, 20, 23, 24], "till": [2, 6, 9, 10, 11, 12, 14, 18, 23, 24], "time": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 23, 24], "timeit": 6, "timer": 6, "tini": 3, "tip": 5, "titl": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 15, 20, 23], "tmp": 15, "tn": [4, 5, 9], "to_categor": [3, 5, 6], "to_categorical_numpi": 3, "to_numer": [2, 8, 23], "todai": 5, "togeth": [2, 5, 8, 10, 13, 15, 17, 23], "toi": 16, "told": 15, "toler": [4, 16], "tolist": 6, "tomographi": 14, "too": [2, 4, 6, 7, 8, 11, 13, 15, 20, 22, 24], "took": [10, 23], "tool": [0, 2, 3, 5, 8, 15, 17, 24], "toolbox": 10, "top": [2, 5, 7, 8, 11, 12, 17, 23], "topic": [2, 7, 8, 9, 10, 17, 24], "topolog": [5, 14], "topologi": [3, 14], "torkjellsdatt": [21, 23], "toss": [12, 20], "total": [2, 3, 4, 5, 6, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 21, 23, 24], "total_loss": 6, "totalclustervari": 16, "totalscatt": 16, "toward": [0, 3, 4, 9, 14, 15], "town": 2, "tp": [6, 9], "tpng": 11, "tpu": [15, 17, 23], "tqdm": 8, "tr": [], "track": [0, 5, 15, 16, 18, 24], "tract": 2, "tractabl": [2, 23, 24], "trade": [7, 11], "tradeoff": [2, 7, 23, 24], "tradit": [2, 3, 6, 8, 23], "train": [1, 4, 5, 7, 8, 10, 11, 12, 13, 14, 15], "train_accuraci": [2, 3, 5, 23], "train_dataset": 6, "train_end": [2, 3, 24], "train_error": 8, "train_imag": [5, 6], "train_ind": 8, "train_label": [5, 6], "train_pr": 3, "train_siz": [2, 3, 5, 24], "train_step": 6, "train_test_split": [0, 1, 2, 3, 5, 7, 8, 9, 11, 12, 13, 23, 24], "train_test_split_numpi": [2, 3, 24], "trainable_vari": 6, "trained_model": 8, "trainerror": [2, 24], "traini": 6, "training_checkpoint": 6, "training_dataset": 6, "training_gradi": 15, "trainingerror": 8, "trainpredict": 6, "trainscor": 6, "trainx": 6, "trait": [2, 23], "trajectori": 6, "transfer": [11, 23], "transform": [2, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 23, 24], "transit": [8, 14], "translat": [3, 6, 8, 12, 23], "transpos": [3, 7, 13, 18, 24], "travers": [2, 7], "treat": [2, 3, 5, 8, 14, 15, 20, 23, 24], "tree": [2, 3, 17, 23], "tree_clf": [11, 12], "tree_clf_": 11, "tree_clf_sr": 11, "tree_reg": 11, "tree_reg1": 11, "tree_reg2": 11, "trend": 20, "treue": 9, "trevor": 22, "tri": [1, 4, 5, 6, 11, 15], "triain": 2, "trial": [2, 4, 6, 8, 15, 20, 23], "triangl": 15, "triangular": 18, "trick": [5, 6, 10, 13, 15, 20], "trickier": 20, "tridiagon": 18, "trillion": 17, "trivial": [2, 3, 7, 13, 20, 23], "troubl": [0, 2, 10, 14, 24], "truck": 5, "true": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 20, 23, 24], "true_fun": 8, "true_theta": 8, "truli": 23, "try": [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 20, 23], "tucker": 10, "tuesdai": [21, 23], "tumor": [9, 11], "tumour": 9, "tunabl": 3, "tune": [6, 11, 15, 18, 23], "turn": [2, 3, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "tutori": [3, 6], "tv": 4, "tveito": 4, "tweak": [3, 6, 12, 20], "twice": 15, "twist": 13, "two": [0, 2, 3, 4, 6, 7, 8, 9, 11, 12, 13, 14, 15, 18, 19, 20, 22, 23, 24], "tx": 15, "tx_1": 15, "txt": [0, 6], "ty": 15, "type": [2, 3, 5, 8, 10, 12, 15, 18, 20, 24], "typic": [0, 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 14, 15, 20, 23, 24], "u": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 22, 23, 24], "u_": 18, "u_i": 14, "u_m": 12, "ua": [2, 23], "ubuntu": [2, 17, 23], "uci": 2, "uio": [0, 21, 22], "un": 16, "unabl": 0, "unari": [18, 23], "unbalanc": [8, 11], "unbias": [2, 7, 8, 23], "uncent": 8, "uncertainti": [2, 7, 23], "uncertitud": 20, "unchang": [3, 5], "uncorrel": [12, 20], "undefin": [7, 24], "under": [2, 3, 7, 8, 12, 15, 17, 23], "underdetermin": [2, 23], "underfit": [3, 8], "underflowproblem": 7, "undergo": 7, "undergradu": [19, 21], "underli": [2, 3, 11, 15, 20, 23], "underset": [6, 16], "understand": [0, 2, 3, 5, 7, 8, 12, 15, 16, 17, 23, 24], "understood": [10, 15], "undesir": 10, "undetermin": [7, 10], "undo": 6, "unexpect": 8, "unexpected": 20, "unfair": 8, "unfortun": [3, 10, 11, 12], "unicode_liter": [10, 11], "uniform": [2, 3, 7, 8, 13, 15, 20, 23], "uniformli": [15, 20], "unifrompdf": 20, "unimport": 15, "union": [7, 8], "uniqu": [2, 4, 8, 15, 16, 18, 23], "unique_cluster_label": 16, "unit": [2, 3, 5, 6, 7, 12, 14, 20, 23, 24], "unitari": [7, 8, 18, 24], "unitarili": [18, 23], "uniti": 20, "univari": 20, "univers": [2, 3, 4, 15, 17, 19, 21, 23, 24], "unix": 3, "unknow": [2, 18, 23], "unknown": [2, 3, 5, 6, 7, 8, 10, 12, 15, 18, 23, 24], "unknowwn": 14, "unlabel": 3, "unless": [2, 5, 8, 13, 15, 23], "unlik": [3, 5, 10, 15], "unnecessarili": 11, "unord": 5, "unravel": 3, "unrol": [5, 13], "unseen": [0, 2, 9, 11], "unstabl": 3, "unsupervis": [2, 3, 6, 14, 17, 23], "unsymmetr": [18, 23], "until": [3, 4, 6, 11, 14, 15, 16], "untouch": 2, "unusu": 14, "up": [1, 3, 5, 6, 7, 8, 10, 12, 13, 15, 16, 17, 18, 20, 21], "updat": [0, 3, 4, 12, 14, 15, 16], "uploa": 23, "upload": [0, 17, 22], "upon": [2, 3, 8, 9, 13, 18], "upper": [1, 2, 10, 11, 18, 24], "uppercas": [18, 23], "upsampl": 6, "upscal": 6, "url": [23, 24], "us": [0, 6, 7, 8, 10, 11, 12, 13, 14, 16, 18, 20, 22], "usag": [2, 10, 17, 23, 24], "usd": 2, "usd10000": 2, "use_bia": 6, "usecol": [2, 23], "useless": 3, "user": [0, 2, 3, 4, 6, 8, 9, 17, 18, 23], "usernam": 0, "usetex": 20, "usg": 8, "usr": 20, "usual": [2, 5, 6, 9, 14, 15, 16, 23], "ut": 7, "util": [3, 5, 6, 8, 9, 12, 16, 23], "ux": 18, "v": [0, 2, 4, 6, 7, 8, 13, 15, 17, 24], "v0": 20, "v1": 20, "v2": 20, "v_0": 13, "va": 3, "vahid": 23, "val": 15, "val_accuraci": 5, "val_loss": 6, "vale": 4, "valid": [2, 3, 6, 9, 11, 12, 15, 17, 20, 23, 24], "validation_data": 5, "validation_split": 6, "valu": [1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 23], "valuat": 11, "valy": 6, "van": [2, 23, 24], "vandenbergh": [10, 15], "vandermond": [2, 23], "vanilla": [2, 8, 13, 16, 24], "vanish": [3, 6, 15, 20], "var": [7, 8, 12, 13, 20, 24], "var_x": 20, "varabl": 10, "varepsilon": [7, 8], "varepsilon_": [7, 8], "varepsilon_i": [7, 8], "vari": [2, 3, 5, 7, 8, 12, 23], "variabl": [2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 23, 24], "varianc": [2, 3, 7, 9, 11, 12, 13, 15, 16, 17, 18, 20, 23, 24], "variance_i": [7, 13, 24], "variance_x": [7, 13, 24], "variant": [2, 3, 8, 10, 14, 15, 23, 24], "variat": [5, 6, 13, 23], "varieti": [2, 5, 14, 17, 23], "variou": [1, 3, 5, 7, 8, 9, 10, 11, 13, 14, 15, 17, 18, 20, 23, 24], "varydimens": 6, "vastli": 5, "vaue": 3, "vault": 2, "vdot": [4, 15], "vec": 8, "vector": [2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 17], "vector_mean": 16, "ventur": [2, 10, 17, 23], "venv": 0, "verbos": [3, 5, 6], "veri": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 22, 23, 24], "verifi": [5, 13, 18, 23], "versatil": [10, 23], "versicolor": [10, 11], "version": [0, 2, 5, 12, 15, 16, 17, 18, 20, 23], "versu": 3, "vert": [1, 2, 3, 7, 8, 9, 10, 11, 13, 15, 23, 24], "vert_1": [7, 8, 24], "vert_2": [7, 8, 13, 24], "via": [2, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 20, 21, 23, 24], "vidal": 13, "video": [2, 3, 14, 17, 19, 21, 23], "view": [3, 5, 7, 8, 14, 15, 20, 22, 23], "violat": 10, "virginica": 11, "viridi": [2, 3, 4, 5, 23], "virtual": 3, "viscos": 15, "viscou": 15, "visibl": 0, "vision": [2, 5], "visual": [2, 5, 13, 14, 17, 23, 24], "visualis": 3, "visualstudio": [0, 1], "viz": [8, 10, 20], "vmap": 15, "vmax": [3, 8], "vmin": [3, 8], "voic": 5, "volum": [2, 5, 23], "vote": [12, 23], "voting_clf": 12, "votingclassifi": 12, "votingsimpl": 12, "vstack": [7, 13, 18, 20, 23, 24], "vt": [7, 24], "w": [2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 23, 24], "w1": 10, "w2": [10, 13], "w3": 10, "w_": [3, 14], "w_1": [10, 18], "w_1x_": 10, "w_1x_1": 10, "w_2": [10, 18], "w_2x_": 10, "w_2x_2": 10, "w_3": 18, "w_4": 18, "w_hidden": 4, "w_i": [3, 4, 12], "w_ix_i": 14, "w_j": 18, "w_m": 18, "w_output": 4, "w_px_": 10, "w_px_p": 10, "wa": [2, 3, 5, 6, 7, 8, 9, 12, 13, 14, 16, 18, 23, 24], "wai": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 18, 20, 23, 24], "walk": 11, "walker": 20, "wang": [2, 23], "want": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 20, 23, 24], "warn": 6, "warrant": 8, "wast": 5, "watch": 17, "wave": 5, "wavelet": 10, "we": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24], "weak": [11, 12, 16], "weather": [3, 14], "web": [17, 19, 21, 23], "webpag": 23, "websit": [8, 18, 19, 23], "wedg": [10, 20], "wednesdai": [21, 23], "wee": 13, "week": [2, 7, 8, 9, 19, 21], "weekli": [0, 1, 17, 19, 21, 22, 23], "weight": [2, 3, 4, 5, 8, 9, 11, 12, 14, 15, 20], "weigth": 4, "welcom": [0, 10, 17], "well": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18, 20, 22, 23, 24], "went": 10, "were": [2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 20, 23], "wessel": [2, 23, 24], "what": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20], "whatev": 5, "when": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "whenev": [0, 15, 20], "where": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23, 24], "wherea": [8, 20], "wherein": [3, 14], "whether": [2, 5, 7, 9, 11, 20, 23], "which": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23, 24], "whichev": [3, 5], "while": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 20, 23, 24], "white": 11, "who": [0, 2], "whole": [3, 5, 6, 7, 11, 13, 15], "whose": [2, 8, 12, 20, 24], "whow": [13, 24], "why": [0, 1, 2, 3, 5, 8, 15], "wide": [2, 3, 5, 8, 9, 14, 17, 18, 23], "widehat": 8, "width": [2, 5, 10, 11, 23], "wieringen": [2, 23, 24], "win": 12, "wind": 11, "wing": [21, 23], "winther": 4, "wiothout": 8, "wiscons": 9, "wisconsin": 12, "wisdom": 8, "wise": [2, 3, 7, 14, 15, 24], "wish": [2, 4, 7, 9, 10, 13, 15, 16, 18, 23, 24], "with_std": [2, 24], "wither": 8, "within": [2, 4, 5, 6, 9, 11, 14, 15, 16, 20, 22, 23], "withinclust": 16, "without": [0, 2, 3, 7, 8, 10, 11, 13, 14, 15, 23, 24], "won": [0, 2, 23], "wonder": 10, "word": [2, 3, 5, 6, 7, 8, 9, 16, 20, 23, 24], "work": [0, 1, 2, 3, 6, 8, 9, 10, 11, 15, 17, 19, 20, 21, 23, 24], "workshop": 23, "world": [1, 2, 10, 24], "worldwid": [2, 23], "worri": 0, "wors": [2, 3, 5, 6, 8, 23], "worth": 11, "would": [1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 20, 23, 24], "wrap": [8, 18, 23], "write": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 14, 15, 18, 23, 24], "written": [1, 2, 4, 5, 7, 13, 14, 15, 17, 18, 20, 23, 24], "wrong": [0, 3, 10], "wrongli": 12, "wrote": [7, 13, 24], "wrt": [12, 15], "wth": [12, 15], "www": [17, 18, 22, 23], "wx_1": 10, "x": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23], "x0": 10, "x1": [6, 10, 11, 12, 15], "x1_exampl": 10, "x1d": 10, "x2": [10, 11, 12, 15], "x2d": [10, 13], "x2d_train": 13, "x2dsl": 13, "x3": 10, "x_": [2, 4, 5, 7, 8, 10, 12, 13, 15, 16, 18, 20, 23, 24], "x_0": [2, 7, 13, 18, 23, 24], "x_1": [2, 4, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 23, 24], "x_2": [2, 4, 7, 8, 9, 10, 11, 12, 13, 15, 18, 20, 23, 24], "x_3": [10, 18, 20], "x_4": 18, "x_center": 13, "x_data": 3, "x_data_ful": 3, "x_hidden": 4, "x_i": [2, 3, 4, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "x_input": 4, "x_ix_": [2, 23], "x_iy_i": 10, "x_j": [1, 2, 4, 10, 11, 14, 20, 24], "x_jy_j": 10, "x_k": [14, 16, 18, 20, 24], "x_l": 20, "x_m": [8, 14, 18, 20], "x_n": [2, 4, 5, 8, 10, 13, 14, 15, 18, 20, 23], "x_new": [11, 12], "x_offset": 8, "x_output": 4, "x_p": [5, 9, 11], "x_poli": 11, "x_poly10": 11, "x_pred": 6, "x_prev": 4, "x_reduc": 13, "x_scale": 10, "x_small": 15, "x_test": [0, 1, 2, 3, 5, 7, 8, 9, 11, 12, 13, 24], "x_test_own": 8, "x_test_scal": [2, 8, 9, 11, 12, 13, 24], "x_tot": 6, "x_train": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 23, 24], "x_train_mean": 8, "x_train_own": 8, "x_train_scal": [2, 8, 9, 11, 12, 13, 24], "x_val": 3, "xarrai": [17, 23], "xavier": 3, "xbnew": 15, "xcode": [2, 17, 23], "xdclassiffierconfus": 12, "xdclassiffierroc": 12, "xg_clf": 12, "xgb": 12, "xgbclassifi": 12, "xgboost": 11, "xgboot": 12, "xgbregressor": 12, "xgparam": 12, "xgtree": 12, "xi": [10, 15], "xi_": 10, "xi_1": 10, "xi_i": 10, "xk": 10, "xla": [15, 17, 23], "xlabel": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 20, 23, 24], "xlim": [8, 12], "xm": 11, "xmesh": 15, "xnew": [2, 15, 23], "xp": 20, "xpanda": [2, 24], "xpd": [7, 13, 24], "xplot": 2, "xscale": [2, 24], "xsr": 11, "xt_x": 15, "xtest": 8, "xtick": [5, 8, 10, 11], "xtrain": 8, "xu": [2, 23], "xx": [2, 18, 23], "xy": [2, 8, 10, 18, 23], "xytext": 10, "xz": [18, 23], "y": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "y1": 6, "y2": 6, "y3": 6, "y_": [2, 3, 7, 8, 12, 13, 18, 23, 24], "y_0": [2, 7, 13, 18, 23, 24], "y_1": [2, 7, 10, 11, 13, 15, 18, 23, 24], "y_1y_1": 10, "y_1y_1k": 10, "y_1y_2": 10, "y_1y_2k": 10, "y_1y_n": 10, "y_1y_nk": 10, "y_2": [2, 7, 10, 11, 13, 18, 23, 24], "y_2y_1": 10, "y_2y_1k": 10, "y_2y_2": 10, "y_2y_2k": 10, "y_3": [2, 11, 18], "y_4": 18, "y_data": [2, 3, 7, 8, 23, 24], "y_data_ful": 3, "y_decis": 10, "y_fit": [2, 24], "y_i": [2, 3, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 23, 24], "y_if_": 12, "y_ix_": [2, 23], "y_ix_i": [9, 10, 15, 24], "y_iy_jk": 10, "y_j": [8, 10, 14], "y_k": 14, "y_m": 18, "y_model": [2, 6, 7, 8, 23, 24], "y_n": [10, 15], "y_ny_1": 10, "y_ny_1k": 10, "y_ny_2": 10, "y_ny_2k": 10, "y_ny_n": 10, "y_ny_nk": 10, "y_offset": 8, "y_plot": 11, "y_pred": [2, 3, 6, 8, 9, 10, 11, 12, 24], "y_pred1": 11, "y_pred2": 11, "y_pred_rf": 12, "y_pred_tre": 12, "y_proba": [9, 12], "y_scaler": 8, "y_test": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 24], "y_test_onehot": 3, "y_test_predict": 2, "y_tot": 6, "y_train": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 23, 24], "y_train_mean": 8, "y_train_onehot": 3, "y_train_predict": 2, "y_train_scal": 8, "y_val": 3, "ye": [5, 8, 9], "year": [2, 17, 23], "yet": [2, 3, 8, 10, 13, 15, 23], "yi": 15, "yield": [2, 4, 7, 8, 10, 12, 14, 15, 16, 18, 20, 23], "yk": 10, "ylabel": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 20, 23, 24], "ylim": [5, 8], "ym": 11, "ymesh": 15, "yn": 2, "yo": [10, 11, 12], "yoshua": [3, 22], "you": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 20, 21, 22, 23, 24], "young": 2, "your": [0, 3, 4, 6, 7, 8, 10, 13, 15, 17, 18, 23], "your_model_object": 1, "yourself": [13, 15, 23], "youtub": 17, "ypred": 8, "ypredict": [2, 15, 23, 24], "ypredict2": 15, "ypredictlasso": 7, "ypredictol": [2, 7], "ypredictown": 8, "ypredictownridg": [8, 24], "ypredictridg": [2, 7, 8, 24], "ypredictskl": 8, "ytest": 8, "ytick": [5, 8, 10, 11], "ytild": [2, 8, 23, 24], "ytildelasso": 7, "ytildenp": [2, 23, 24], "ytildeol": [2, 7], "ytildeownridg": [8, 24], "ytilderidg": [7, 8, 24], "ytrain": 8, "yuxi": 23, "yx": [18, 23], "yy": [18, 23], "yz": [18, 23], "z": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 23, 24], "z_": [3, 4, 14, 18, 23], "z_0": [18, 23], "z_1": [18, 23], "z_2": [18, 23], "z_c": 3, "z_h": 3, "z_hidden": 4, "z_i": [3, 14], "z_j": [3, 14], "z_k": [14, 24], "z_m": 3, "z_mod": 11, "z_o": 3, "z_output": 4, "zaman": 20, "zaxi": 8, "zero": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 23, 24], "zeros_lik": 6, "zeroth": 24, "zfill": 6, "zip": [6, 8], "zm_h": [2, 23], "zn": 2, "zone": 2, "zoom": 23, "zx": [18, 23], "zy": [18, 23], "zz": [18, 23], "\u00f8yvind": 8}, "titles": ["Exercises week 34", "Exercises week 35", "3. Linear Regression", "14. Building a Feed Forward Neural Network", "15. Solving Differential Equations with Deep Learning", "16. Convolutional Neural Networks", "17. Recurrent neural networks: Overarching view", "4. Ridge and Lasso Regression", "5. Resampling Methods", "6. Logistic Regression", "8. Support Vector Machines, overarching aims", "9. Decision trees, overarching aims", "10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods", "11. Basic ideas of the Principal Component Analysis (PCA)", "13. Neural networks", "7. Optimization, the central part of any Machine Learning algortithm", "12. Clustering and Unsupervised Learning", "Applied Data Analysis and Machine Learning", "2. Linear Algebra, Handling of Arrays and more Python Features", "Course setting", "1. Elements of Probability Theory and Statistical Data Analysis", "Teachers and Grading", "Textbooks", "Week 34: Introduction to the course, Logistics and Practicalities", "Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression"], "titleterms": {"": [10, 12], "1": [0, 1, 2, 24], "2": [0, 1, 2, 23, 24], "2023": 21, "3": [0, 1, 2, 24], "34": [0, 23], "35": [1, 24], "4": [0, 1, 2, 24], "5": [1, 2], "A": [2, 3, 6, 10, 11, 23], "And": [23, 24], "In": 21, "Ising": 8, "The": [0, 2, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 17, 23, 24], "To": 23, "With": 6, "about": [23, 24], "activ": [3, 14], "ad": [2, 8, 23, 24], "adaboost": 12, "adagrad": 15, "adam": 15, "adapt": 12, "adjust": 3, "adversari": 6, "again": [5, 11], "ai": 23, "aim": [10, 11, 23], "aka": 23, "algebra": [18, 23], "algorithm": [11, 12, 13, 14, 23, 24], "algortithm": 15, "all": 10, "an": [0, 2, 6, 12, 23], "analys": [7, 24], "analysi": [2, 7, 8, 13, 17, 20, 23, 24], "analyt": [1, 2], "ani": 15, "anoth": 11, "appli": 17, "approach": [2, 10, 16, 23], "approxim": 14, "architectur": 3, "arrai": [18, 23], "assist": 21, "autocorrel": 20, "autograd": [4, 15], "automat": 15, "back": [3, 13, 14, 24], "background": 17, "bag": 12, "base": 15, "basic": [2, 7, 9, 11, 12, 13, 18, 24], "batch": 3, "bay": 7, "befor": 13, "better": 10, "bia": 8, "binari": 3, "bind": 23, "bird": 12, "boldsymbol": 24, "boost": 12, "bootstrap": [8, 12], "boston": 2, "breast": 3, "brief": 23, "bring": 14, "build": [3, 5, 11], "c": 23, "calcul": 24, "can": 23, "cancer": [3, 9, 11, 13], "cart": 11, "case": [10, 12, 20, 24], "central": [15, 17, 20], "chain": 14, "chang": 12, "channel": 23, "chi": [2, 23], "choos": 3, "cifar01": 5, "classic": 13, "classif": [3, 11, 12], "classifi": 10, "clip": 3, "cluster": 16, "cnn": 5, "code": [0, 1, 2, 3, 4, 7, 11, 13, 14, 15, 16, 23, 24], "collect": [3, 5], "commun": 23, "compar": [1, 4, 12], "complet": 24, "complex": [2, 8, 24], "complic": 8, "compon": 13, "comput": 11, "computerlab": 23, "con": 11, "concept": 20, "conjug": 15, "contn": 23, "convex": [10, 15], "convolut": [5, 14], "correl": [13, 24], "cost": [3, 12, 24], "cours": [17, 19, 22, 23], "covari": [7, 13, 20, 24], "cover": 23, "creat": 1, "cross": 8, "cython": 23, "data": [0, 2, 3, 5, 8, 9, 11, 13, 17, 20, 23, 24], "dataset": [3, 5], "david": 23, "deadlin": 23, "deadllin": 21, "decai": 4, "decis": [11, 12], "decomposit": [7, 13, 18, 24], "deeep": [], "deep": [3, 4, 23], "defin": [3, 23], "degre": [2, 24], "deliver": [0, 1], "dens": 2, "deriv": [1, 7, 14, 24], "descent": [4, 12, 15], "design": 24, "detail": [5, 23], "develop": 3, "diagon": 13, "differ": 10, "differenti": [4, 15], "diffus": 4, "dimension": [4, 5, 10], "disadvantag": 11, "discret": 20, "discrimin": 23, "distribut": [7, 20], "do": 3, "doe": 24, "domain": 20, "down": 3, "dropout": 3, "economi": 24, "element": [2, 20, 23], "elimin": 18, "energi": 23, "ensembl": 12, "entropi": 11, "environ": [0, 2], "equat": [2, 4, 14, 24], "error": [2, 12, 23, 24], "essenti": 23, "etc": 23, "euler": 4, "evalu": 3, "exampl": [2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 23, 24], "exercis": [0, 1, 2, 8, 24], "expect": 20, "experi": 20, "explor": 2, "exponenti": 4, "express": [1, 24], "extrapol": 6, "extrem": [12, 23], "ey": 12, "fall": 21, "famili": [3, 23], "famou": 18, "fantast": 24, "featur": [1, 11, 18, 24], "feed": [3, 14], "final": [14, 24], "find": 1, "fine": 3, "first": [6, 14, 23], "fit": [0, 1, 2, 12, 23], "fix": 24, "forc": 5, "forest": 12, "format": 23, "forward": [3, 4, 14], "foster": 23, "fourier": 5, "frank": 8, "freedom": [2, 24], "frequent": 24, "frequentist": [2, 23], "from": [7, 12, 14, 23, 24], "full": 4, "function": [2, 3, 8, 9, 10, 12, 13, 14, 15, 20, 23, 24], "further": [5, 7, 24], "gan": 6, "gaussian": 18, "gd": 15, "gener": [6, 11, 23], "geometr": 13, "gini": 11, "github": 0, "goal": [0, 1], "good": [2, 23], "grade": [21, 23], "gradient": [3, 4, 12, 15], "growth": 4, "ha": 17, "handl": [18, 23], "hessian": 24, "hidden": 4, "hous": 2, "how": 1, "hyperparamet": 3, "hyperplan": 10, "i": [2, 3, 23], "id3": 11, "idea": 13, "ii": 23, "implement": [1, 3], "implic": [7, 24], "import": [7, 18, 23, 24], "improv": 3, "includ": 15, "increment": 13, "index": 11, "inform": 21, "input": 4, "instal": [17, 23], "instructor": 21, "interpret": [7, 13, 23, 24], "introduc": [13, 15, 24], "introduct": [2, 8, 17, 18, 23], "invers": [7, 18], "invert": 24, "iter": 12, "its": 24, "jacobian": 24, "jax": 15, "julia": 23, "jungl": 12, "kera": [3, 5], "kernel": [10, 13], "lagrangian": 10, "lasso": [7, 8, 24], "last": 24, "later": [7, 24], "layer": [3, 4, 5, 14], "learn": [0, 1, 2, 3, 4, 13, 15, 16, 17, 23, 24], "least": [1, 7, 8, 23, 24], "lectur": 23, "level": 12, "librari": [17, 23], "likelihood": 9, "limit": [3, 15, 20], "linear": [0, 2, 10, 15, 18, 23, 24], "link": [7, 13, 22, 24], "logist": [9, 23], "loss": 24, "lu": 18, "machin": [2, 10, 15, 17, 23], "main": [20, 23], "make": [2, 11, 12, 24], "mani": [12, 14], "mass": 23, "materi": 23, "math": [7, 24], "mathemat": [5, 7, 10, 24], "matric": [7, 18, 23], "matrix": [1, 3, 7, 13, 14, 18, 23, 24], "matter": 2, "max": 24, "mean": [2, 24], "meet": [7, 12, 20, 23, 24], "mercer": 10, "method": [8, 11, 12, 15, 23], "min": 24, "minim": 23, "ml": 23, "mlp": 14, "mnist": [5, 6], "model": [0, 2, 3, 6, 8, 14, 23], "momentum": 15, "moon": [10, 11], "more": [5, 8, 18, 23, 24], "multilay": 14, "multipl": [3, 5], "multipli": 10, "need": 23, "network": [3, 4, 5, 6, 9, 14, 23], "neural": [3, 4, 5, 6, 9, 14, 23], "new": 6, "non": 10, "normal": [2, 3], "notat": 14, "note": 24, "now": [3, 11, 15], "nuclear": [2, 23], "numba": 23, "number": [2, 4, 20, 24], "numer": [4, 20], "numpi": [18, 23], "object": 5, "obtain": 13, "od": 4, "off": 8, "ol": [0, 1, 7, 8], "one": [4, 14], "oper": 18, "optim": [3, 10, 15, 17, 23, 24], "order": 15, "ordinari": [1, 7, 8, 23, 24], "organ": [2, 23], "oslo": 22, "other": [6, 11, 13, 14, 18, 23], "our": [2, 6, 7, 13, 15, 23, 24], "outcom": [17, 23], "output": 4, "overarch": [2, 6, 10, 11, 23, 24], "overview": [12, 23], "own": [2, 12, 13, 23, 24], "packag": [18, 23], "panda": [23, 24], "paramet": [23, 24], "part": [15, 17], "partial": 4, "pass": 3, "pca": 13, "pdf": 20, "perceptron": 14, "perform": [3, 11], "period": 5, "perspect": 3, "plan": 24, "plethora": 23, "point": 6, "poisson": 4, "polynomi": [1, 5], "popul": 4, "popular": 23, "practic": [15, 21, 23], "pre": [3, 5], "predict": 6, "preprocess": 24, "prerequisit": [5, 17, 23], "princip": 13, "principl": 5, "pro": 11, "probabl": [7, 20], "problem": [3, 4, 15, 23, 24], "procedur": [11, 23], "process": [3, 5], "program": [4, 15], "project": [8, 21, 23], "prop": 15, "propag": [3, 14], "properti": [7, 20, 24], "python": [0, 2, 11, 17, 18, 23], "quick": 10, "r": 23, "random": [12, 13, 20], "read": [11, 23, 24], "real": [8, 23], "recommend": [23, 24], "recurr": [6, 14], "reduc": [2, 24], "reduct": 5, "reformul": 4, "regress": [0, 2, 7, 8, 9, 11, 12, 15, 23, 24], "regular": 3, "relat": [], "relev": [22, 24], "relu": 3, "remark": 5, "remind": [8, 10, 23, 24], "replac": 15, "repositori": 0, "requir": [4, 17], "resampl": 8, "rescal": 8, "residu": 24, "resourc": 4, "result": 24, "revisit": 15, "rewrit": [23, 24], "ridg": [2, 7, 8, 24], "rm": 15, "rule": 14, "same": 15, "sampl": 13, "scale": 24, "schedul": 23, "schemat": 11, "scheme": 4, "scienc": 23, "scikit": [2, 3, 13, 23, 24], "second": 15, "semest": 21, "set": [0, 2, 4, 5, 11, 14, 19, 23, 24], "setup": 0, "sgd": 15, "should": 3, "similar": 15, "simpl": [2, 6, 11, 15, 23, 24], "singl": 12, "singular": [7, 13, 24], "size": 24, "sklearn": 1, "soft": 10, "softmax": 3, "softwar": 23, "solv": 4, "solver": 15, "some": [15, 18, 24], "specifi": 4, "split": [0, 2, 24], "squar": [1, 2, 7, 8, 12, 23, 24], "standard": [15, 24], "state": 2, "statist": [7, 8, 17, 20, 23], "steepest": [12, 15], "stochast": [15, 20], "strongli": 23, "suggest": 23, "summari": [21, 23], "superposit": 5, "supervis": 3, "support": 10, "svd": [7, 24], "systemat": 5, "t": 24, "take": 1, "taken": 23, "teach": 21, "teacher": [21, 23], "techniqu": [8, 13], "technologi": 17, "tensorflow": [3, 5], "tent": [21, 23], "test": [0, 2, 3, 24], "text": 23, "textbook": [22, 23], "theorem": [7, 10, 13, 14, 20], "theori": 20, "thi": 23, "tip": 15, "togeth": 14, "tool": 23, "top": 3, "topic": 23, "toward": 13, "trade": 8, "tradeoff": 8, "train": [0, 2, 3, 6, 23, 24], "transform": 5, "tree": [11, 12], "tune": 3, "two": [5, 10, 17], "type": [4, 6, 14, 23], "uio": 23, "univers": [14, 22], "unsupervis": 16, "up": [0, 2, 4, 11, 14, 23, 24], "us": [1, 2, 3, 4, 5, 9, 15, 17, 23, 24], "v": 5, "valid": 8, "valu": [7, 13, 20, 24], "variabl": 20, "varianc": 8, "variou": 2, "vector": [1, 10, 14, 18, 23, 24], "versu": 23, "view": [2, 6, 12, 24], "virtual": 0, "visual": [3, 11], "wai": 11, "wave": 4, "we": 23, "week": [0, 1, 23, 24], "what": [2, 23, 24], "which": 3, "why": 23, "wisconsin": 9, "write": [6, 13], "x": 24, "xgboost": 12, "your": [1, 2, 12, 24]}}) \ No newline at end of file diff --git a/doc/LectureNotes/_build/html/week35.html b/doc/LectureNotes/_build/html/week35.html index 1299e2bfb..15c73ed19 100644 --- a/doc/LectureNotes/_build/html/week35.html +++ b/doc/LectureNotes/_build/html/week35.html @@ -390,7 +390,6 @@ document.write(`
  • Example 3
  • Example 4
  • The mean squared error and its derivative
    • -
  • Other useful relations
  • Meet the Hessian Matrix
  • Interpretations and optimizing our parameters
  • Example relevant for the exercises
    • @@ -413,6 +412,8 @@ document.write(`
  • The singular value decomposition
  • Linear Regression Problems
  • Fixing the singularity
    • +
  • Ridge and LASSO Regression
    • +
  • Deriving the Ridge Regression Equations
  • Basic math of the SVD
  • The SVD, a Fantastic Algorithm
  • Economy-size SVD
    • @@ -429,13 +430,11 @@ document.write(`
  • Covariance Matrix Examples
  • Correlation Matrix
  • Correlation Matrix with Pandas
    • -
  • Correlation Matrix with Pandas and the Franke function
  • Rewriting the Covariance and/or Correlation Matrix
  • Linking with the SVD
  • What does it mean?
  • And finally \(\boldsymbol{X}\boldsymbol{X}^T\)
    • -
  • Ridge and LASSO Regression
    • -
  • Deriving the Ridge Regression Equations
    • +
  • Back to Ridge and LASSO Regression
  • Interpreting the Ridge results
  • More interpretations
  • Deriving the Lasso Regression Equations
    • @@ -461,10 +460,10 @@ doconce format html week35.do.txt --no_mako -->

    The main topics are:

    1. Brief repetition from last week

      2. -

    2. Discussions of the equations for ordinary least squares

      3. +

    3. Discussions of the equations for ordinary least squares (OLS)

    4. Discussion on how to prepare data and examples of applications of linear regression

      5. -

    5. Material for the lecture on Monday: Mathematical interpretations of linear regression

      6. -

    6. Monday: Ridge and Lasso regression and Singular Value Decomposition

      7. +

    7. Mathematical interpretations of OLS

      8. +

    8. Introduction of Ridge and Lasso regression

    Reading recommendations:#

    @@ -474,7 +473,7 @@ doconce format html week35.do.txt --no_mako -->
      -

    1. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)

      2. +

    2. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra

    3. Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4.

    4. For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Marc Peter Deisenroth on the Mathematics of Machine Learning, may be very relevant. In particular chapter 5 at URL”https://mml-book.github.io/” (section 5.5 on derivatives) is very useful for exercise 1 this coming week.

    @@ -568,16 +567,16 @@ y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dot \]

    where \(\langle y_i \rangle\) is the mean value. Keep in mind also that till now we have treated \(y_i\) as the exact value. Normally, the -response (dependent or outcome) variable \(y_i\) is the outcome of a +output (response, target, dependent or outcome) variable \(y_i\) is the outcome of a numerical experiment or another type of experiment and could thus be treated itself as an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we -will treat \(y_i\) as our exact value for the response variable.

    +will treat \(y_i\) as our exact value for the output variable.

    In order to find the parameters \(\theta_i\) we will then minimize the spread of \(C(\boldsymbol{\theta})\), that is we are going to solve the problem

    \[ -{\displaystyle \min_{\boldsymbol{\theta}\in +\hat{\boldsymbol{\theta}}={\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}. \]

    In practical terms it means we will require

    @@ -611,7 +610,7 @@ will treat \(y_i\) as our exac

    and if the matrix \(\boldsymbol{X}^T\boldsymbol{X}\) is invertible we have the solution

    \[ -\boldsymbol{\theta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\hat{\boldsymbol{\theta}} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. \]

    We note also that since our design matrix is defined as \(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\), the product \(\boldsymbol{X}^T\boldsymbol{X} \in @@ -621,8 +620,8 @@ matrices to invert. The methods discussed here and for many other supervised learning algorithms like classification with logistic regression or support vector machines, exhibit dimensionalities which allow for the usage of direct linear algebra methods such as LU decomposition or Singular Value Decomposition (SVD) for finding the inverse of the matrix -\(\boldsymbol{X}^T\boldsymbol{X}\). This is discussed on Thursday this week.

    -

    Small question: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix \(\boldsymbol{X}^T\boldsymbol{X}\)? What kind of problems can we expect?

    +\(\boldsymbol{X}^T\boldsymbol{X}\).

    +

    Small question: When inverting the matrix $\boldsymbol{X}^T\boldsymbol{X}, what kind of problems can we expect?

    Some useful matrix and vector expressions#

    @@ -821,26 +820,6 @@ C(\boldsymbol{\theta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w}, \frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right). \]
    -
    -

    Other useful relations#

    -

    We list here some other useful relations we may encounter (recall that vectors are defined by boldfaced low-key letters)

    -
    -\[ -\frac{\partial (\boldsymbol{x}^T\boldsymbol{a})}{\partial \boldsymbol{x}} = \boldsymbol{a}^T, -\]
    -
    -\[ -\frac{\partial (\boldsymbol{a}^T\boldsymbol{x})}{\partial \boldsymbol{x}} = \boldsymbol{a}^T, -\]
    -
    -\[ -\frac{\partial tr(\boldsymbol{B}\boldsymbol{A})}{\partial \boldsymbol{A}} = \boldsymbol{B}^T, -\]
    -
    -\[ -\frac{\partial \log{\vert\boldsymbol{A}\vert}}{\partial \boldsymbol{A}} = (\boldsymbol{A}^{-1})^T. -\]
    -

    Meet the Hessian Matrix#

    A very important matrix we will meet again and again in machine @@ -859,13 +838,12 @@ second derivative of the mean squared error as cost function is,

    \]

    For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters -\(\hat{\boldsymbol{\theta}}\). Furthermore, we will see later this week that it is +\(\hat{\boldsymbol{\theta}}\). Furthermore, we will see next week that it is (aside the factor \(1/n\)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning/data analysis -problem. -v -Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).

    +problem. We will discuss this in greater detail next week when we introduce gradient methods.

    +

    Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).

    Interpretations and optimizing our parameters#

    @@ -903,7 +881,7 @@ We assume our data can represented by a fourth-order polynomial. For the It is rather straightforward to implement the matrix inversion and obtain the parameters \(\boldsymbol{\theta}\). After having defined the matrix \(\boldsymbol{X}\) and the outputs \(\boldsymbol{y}\) we have

    -
    # matrix inversion to find beta
+
    # matrix inversion to find theta
 # First we set up the data
 import numpy as np
 x = np.random.rand(100)
@@ -916,9 +894,9 @@ X[:,1] = x
 X[:,2] = x**2
 X[:,3] = x**3
 X[:,4] = x**4
-beta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
+theta = (np.linalg.inv(X.T @ X) @ X.T ) @ y
 # and then make the prediction
-ytilde = X @ beta
+ytilde = X @ theta
    @@ -1021,16 +999,16 @@ X[:,3] = x**3 X[:,4] = x**4 # We split the data in test and training data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) -# matrix inversion to find beta -beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train -print(beta) +# matrix inversion to find theta +theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train +print(theta) # and then make the prediction -ytilde = X_train @ beta +ytilde = X_train @ theta print("Training R2") print(R2(y_train,ytilde)) print("Training MSE") print(MSE(y_train,ytilde)) -ypredict = X_test @ beta +ypredict = X_test @ theta print("Test R2") print(R2(y_test,ypredict)) print("Test MSE") @@ -1319,7 +1297,7 @@ decomposition.

    As we will also see in the first project, this may however not the be case in general and a standard matrix inversion -algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.

    +algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below and in other examples.

    There is however a way to circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions.

    @@ -1404,9 +1382,175 @@ the regression parameters \(\theta_i\ \]

    where \(\boldsymbol{I}\) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \(\lambda\) is called a hyperparameter. More about this later.

    +
    +

    Ridge and LASSO Regression#

    +

    Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is

    +
    +\[ +{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}. +\]
    +

    or we can state it as

    +
    +\[ +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2, +\]
    +

    where we have used the definition of a norm-2 vector, that is

    +
    +\[ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +\]
    +

    By minimizing the above equation with respect to the parameters +\(\boldsymbol{\theta}\) we could then obtain an analytical expression for the +parameters \(\boldsymbol{\theta}\). We can add a regularization parameter \(\lambda\) by +defining a new cost function to be optimized, that is

    +
    +\[ +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2 +\]
    +

    which leads to the Ridge regression minimization problem where we +require that \(\vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t\), where \(t\) is +a finite number larger than zero. By defining

    +
    +\[ +C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1, +\]
    +

    we have a new optimization equation

    +
    +\[ +{\displaystyle \min_{\boldsymbol{\theta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1 +\]
    +

    which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.

    +

    Here we have defined the norm-1 as

    +
    +\[ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +\]
    +
    +
    +

    Deriving the Ridge Regression Equations#

    +

    Using the matrix-vector expression for Ridge regression and dropping the parameter \(1/n\) in front of the standard means squared error equation, we have

    +
    +\[ +C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta}, +\]
    +

    and +taking the derivatives with respect to \(\boldsymbol{\theta}\) we obtain then +a slightly modified matrix inversion problem which for finite values +of \(\lambda\) does not suffer from singularity problems. We obtain +the optimal parameters

    +
    +\[ +\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +\]
    +

    with \(\boldsymbol{I}\) being a \(p\times p\) identity matrix with the constraint that

    +
    +\[ +\sum_{i=0}^{p-1} \theta_i^2 \leq t, +\]
    +

    with \(t\) a finite positive number.

    +

    If we keep the \(1/n\) factor, the equation for the optimal \(\theta\) changes to

    +
    +\[ +\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\]
    +

    In many textbooks the \(1/n\) term is often omitted. Note that a library like Scikit-Learn does not include the \(1/n\) factor in the setup of the cost function.

    +

    When we compare this with the ordinary least squares result we have

    +
    +\[ +\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +\]
    +

    which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \(\boldsymbol{X}^T\boldsymbol{X}\).

    +

    We see that Ridge regression is nothing but the standard OLS with a +modified diagonal term added to \(\boldsymbol{X}^T\boldsymbol{X}\). The consequences, in +particular for our discussion of the bias-variance tradeoff are rather +interesting. We will see that for specific values of \(\lambda\), we may +even reduce the variance of the optimal parameters \(\boldsymbol{\theta}\). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.

    +

    When we have discussed the singular value decomposition of the design +matrix \(\boldsymbol{X}\), we will in turn perform a more rigorous mathematical +discussion of Ridge regression.

    +

    The code here is a simple demonstration of how to implement Ridge regression with our own code and compare this with scikit-learn.

    +
    +
    +
    import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+from sklearn import linear_model
+
+def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
+
+
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
+np.random.seed(3155)
+
+n = 100
+x = np.random.rand(n)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)
+
+Maxpolydegree = 20
+X = np.zeros((n,Maxpolydegree))
+#We include explicitely the intercept column
+for degree in range(Maxpolydegree):
+    X[:,degree] = x**degree
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
+
+p = Maxpolydegree
+I = np.eye(p,p)
+# Decide which values of lambda to use
+nlambdas = 6
+MSEOwnRidgePredict = np.zeros(nlambdas)
+MSERidgePredict = np.zeros(nlambdas)
+lambdas = np.logspace(-4, 2, nlambdas)
+for i in range(nlambdas):
+    lmb = lambdas[i]
+    OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
+    # Note: we include the intercept column and no scaling
+    RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
+    RegRidge.fit(X_train,y_train)
+    # and then make the prediction
+    ytildeOwnRidge = X_train @ OwnRidgeTheta
+    ypredictOwnRidge = X_test @ OwnRidgeTheta
+    ytildeRidge = RegRidge.predict(X_train)
+    ypredictRidge = RegRidge.predict(X_test)
+    MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)
+    MSERidgePredict[i] = MSE(y_test,ypredictRidge)
+    print("Theta values for own Ridge implementation")
+    print(OwnRidgeTheta)
+    print("Theta values for Scikit-Learn Ridge implementation")
+    print(RegRidge.coef_)
+    print("MSE values for own Ridge implementation")
+    print(MSEOwnRidgePredict[i])
+    print("MSE values for Scikit-Learn Ridge implementation")
+    print(MSERidgePredict[i])
+
+# Now plot the results
+plt.figure()
+plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'r', label = 'MSE own Ridge Test')
+plt.plot(np.log10(lambdas), MSERidgePredict, 'g', label = 'MSE Ridge Test')
+
+plt.xlabel('log10(lambda)')
+plt.ylabel('MSE')
+plt.legend()
+plt.show()
+
    +
    +
    +
    +

    The results here agree when we force Scikit-Learn’s Ridge function to include the first column in our design matrix. +We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix. +What happens if we do not include the intercept in our fit? We will discuss this in more detail next week.

    +

    Basic math of the SVD#

    -

    From standard linear algebra we know that a square matrix \(\boldsymbol{X}\) can be diagonalized if and only it is +

    From standard linear algebra we know that a square matrix \(\boldsymbol{X}\) can be diagonalized if and only if it is a so-called normal matrix, that is if \(\boldsymbol{X}\in {\mathbb{R}}^{n\times n}\) we have \(\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X}\) or if \(\boldsymbol{X}\in {\mathbb{C}}^{n\times n}\) we have \(\boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X}\). The matrix has then a set of eigenpairs

    @@ -1946,66 +2090,6 @@ print(correlation_matrix) -

    We expand this model to the Franke function discussed above.

    -
    -
    -

    Correlation Matrix with Pandas and the Franke function#

    -
    -
    -
    # Common imports
-import numpy as np
-import pandas as pd
-
-
-def FrankeFunction(x,y):
-	term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
-	term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
-	term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
-	term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
-	return term1 + term2 + term3 + term4
-
-
-def create_X(x, y, n ):
-	if len(x.shape) > 1:
-		x = np.ravel(x)
-		y = np.ravel(y)
-
-	N = len(x)
-	l = int((n+1)*(n+2)/2)		# Number of elements in beta
-	X = np.ones((N,l))
-
-	for i in range(1,n+1):
-		q = int((i)*(i+1)/2)
-		for k in range(i+1):
-			X[:,q+k] = (x**(i-k))*(y**k)
-
-	return X
-
-
-# Making meshgrid of datapoints and compute Franke's function
-n = 4
-N = 100
-x = np.sort(np.random.uniform(0, 1, N))
-y = np.sort(np.random.uniform(0, 1, N))
-z = FrankeFunction(x, y)
-X = create_X(x, y, n=n)    
-
-Xpd = pd.DataFrame(X)
-# subtract the mean values and set up the covariance matrix
-Xpd = Xpd - Xpd.mean()
-covariance_matrix = Xpd.cov()
-print(covariance_matrix)
-
    -
    -
    -
    -

    We note here that the covariance is zero for the first rows and -columns since all matrix elements in the design matrix were set to one -(we are fitting the function in terms of a polynomial of degree \(n\)).

    -

    This means that the variance for these elements will be zero and will -cause problems when we set up the correlation matrix. We can simply -drop these elements and construct a correlation -matrix without these elements.

    Rewriting the Covariance and/or Correlation Matrix#

    @@ -2136,8 +2220,8 @@ measure how much correlations are contained in the rows of \(\boldsymbol{X}\), the quantity of interest for us are the non-zero singular values and the column vectors of \(\boldsymbol{V}\).

    -
    -

    Ridge and LASSO Regression#

    +
    +

    Back to Ridge and LASSO Regression#

    Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is our optimization problem is

    @@ -2183,42 +2267,7 @@ C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\bold \[ \vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. \]
    -
    -
    -

    Deriving the Ridge Regression Equations#

    -

    Using the matrix-vector expression for Ridge regression and dropping the parameter \(1/n\) in front of the standard means squared error equation, we have

    -
    -\[ -C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta}, -\]
    -

    and -taking the derivatives with respect to \(\boldsymbol{\theta}\) we obtain then -a slightly modified matrix inversion problem which for finite values -of \(\lambda\) does not suffer from singularity problems. We obtain -the optimal parameters

    -
    -\[ -\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -\]
    -

    with \(\boldsymbol{I}\) being a \(p\times p\) identity matrix with the constraint that

    -
    -\[ -\sum_{i=0}^{p-1} \theta_i^2 \leq t, -\]
    -

    with \(t\) a finite positive number.

    -

    If we keep the \(1/n\) factor, the equation for the optimal \(\theta\) changes to

    -
    -\[ -\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. -\]
    -

    In many textbooks the \(1/n\) term is often omitted. Note that a library like Scikit-Learn does not include the \(1/n\) factor in the setup of the cost function.

    -

    When we compare this with the ordinary least squares result we have

    -
    -\[ -\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -\]
    -

    which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \(\boldsymbol{X}^T\boldsymbol{X}\).

    -

    We see that Ridge regression is nothing but the standard OLS with a +

    Ridge regression, as discussed above, is nothing but the standard OLS with a modified diagonal term added to \(\boldsymbol{X}^T\boldsymbol{X}\). The consequences, in particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \(\lambda\), we may @@ -2301,7 +2350,7 @@ C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsym \[ \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}. \] -

    This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.

    +

    This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss how to code LASSO regression next week, when we have introduced gradient methods.

    @@ -2376,7 +2425,6 @@ C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsym
  • Example 3
  • Example 4
  • The mean squared error and its derivative
    • -
  • Other useful relations
  • Meet the Hessian Matrix
  • Interpretations and optimizing our parameters
  • Example relevant for the exercises
    • @@ -2399,6 +2447,8 @@ C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsym
  • The singular value decomposition
  • Linear Regression Problems
  • Fixing the singularity
    • +
  • Ridge and LASSO Regression
    • +
  • Deriving the Ridge Regression Equations
  • Basic math of the SVD
  • The SVD, a Fantastic Algorithm
  • Economy-size SVD
    • @@ -2415,13 +2465,11 @@ C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsym
  • Covariance Matrix Examples
  • Correlation Matrix
  • Correlation Matrix with Pandas
    • -
  • Correlation Matrix with Pandas and the Franke function
  • Rewriting the Covariance and/or Correlation Matrix
  • Linking with the SVD
  • What does it mean?
  • And finally \(\boldsymbol{X}\boldsymbol{X}^T\)
    • -
  • Ridge and LASSO Regression
    • -
  • Deriving the Ridge Regression Equations
    • +
  • Back to Ridge and LASSO Regression
  • Interpreting the Ridge results
  • More interpretations
  • Deriving the Lasso Regression Equations
    • diff --git a/doc/LectureNotes/_build/jupyter_execute/week35.ipynb b/doc/LectureNotes/_build/jupyter_execute/week35.ipynb index d202acb1e..f83c664a9 100644 --- a/doc/LectureNotes/_build/jupyter_execute/week35.ipynb +++ b/doc/LectureNotes/_build/jupyter_execute/week35.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "3803481e", + "id": "82c2f725", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "90e0b055", + "id": "d2fc786b", "metadata": { "editable": true }, @@ -27,7 +27,7 @@ }, { "cell_type": "markdown", - "id": "0b94b0fb", + "id": "77512f33", "metadata": { "editable": true }, @@ -38,18 +38,18 @@ "\n", "1. Brief repetition from last week\n", "\n", - "2. Discussions of the equations for ordinary least squares\n", + "2. Discussions of the equations for ordinary least squares (**OLS**)\n", "\n", "3. Discussion on how to prepare data and examples of applications of linear regression\n", "\n", - "4. Material for the lecture on Monday: Mathematical interpretations of linear regression\n", + "4. Mathematical interpretations of OLS\n", "\n", - "5. Monday: Ridge and Lasso regression and Singular Value Decomposition" + "5. Introduction of Ridge and Lasso regression" ] }, { "cell_type": "markdown", - "id": "c8ad9a9b", + "id": "41516757", "metadata": { "editable": true }, @@ -60,7 +60,7 @@ "\n", "\n", "\n", - "2. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)\n", + "2. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra\n", "\n", "3. Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4.\n", "\n", @@ -69,7 +69,7 @@ }, { "cell_type": "markdown", - "id": "f5f74962", + "id": "d3703d2f", "metadata": { "editable": true }, @@ -102,7 +102,7 @@ }, { "cell_type": "markdown", - "id": "d61252cd", + "id": "f22b90af", "metadata": { "editable": true }, @@ -118,7 +118,7 @@ }, { "cell_type": "markdown", - "id": "3dd45959", + "id": "3fe305bb", "metadata": { "editable": true }, @@ -130,7 +130,7 @@ }, { "cell_type": "markdown", - "id": "0ab40f7c", + "id": "f2f38355", "metadata": { "editable": true }, @@ -140,7 +140,7 @@ }, { "cell_type": "markdown", - "id": "d219bba3", + "id": "c3d9a4b6", "metadata": { "editable": true }, @@ -152,7 +152,7 @@ }, { "cell_type": "markdown", - "id": "806e921f", + "id": "ff3ba043", "metadata": { "editable": true }, @@ -173,7 +173,7 @@ }, { "cell_type": "markdown", - "id": "5a10e185", + "id": "5de294d5", "metadata": { "editable": true }, @@ -185,7 +185,7 @@ }, { "cell_type": "markdown", - "id": "87b9af07", + "id": "5721942a", "metadata": { "editable": true }, @@ -198,7 +198,7 @@ }, { "cell_type": "markdown", - "id": "b5e5e170", + "id": "2b2aa843", "metadata": { "editable": true }, @@ -210,7 +210,7 @@ }, { "cell_type": "markdown", - "id": "bacf306d", + "id": "2b12c06f", "metadata": { "editable": true }, @@ -222,7 +222,7 @@ }, { "cell_type": "markdown", - "id": "971edd52", + "id": "276d4eba", "metadata": { "editable": true }, @@ -232,7 +232,7 @@ }, { "cell_type": "markdown", - "id": "1e764eaf", + "id": "4751c9b4", "metadata": { "editable": true }, @@ -244,7 +244,7 @@ }, { "cell_type": "markdown", - "id": "f2f1e2f4", + "id": "dd00bf4b", "metadata": { "editable": true }, @@ -257,7 +257,7 @@ }, { "cell_type": "markdown", - "id": "f00007b0", + "id": "bc9028f0", "metadata": { "editable": true }, @@ -269,7 +269,7 @@ }, { "cell_type": "markdown", - "id": "ad127894", + "id": "d6a27e54", "metadata": { "editable": true }, @@ -279,7 +279,7 @@ }, { "cell_type": "markdown", - "id": "87bea682", + "id": "82f1494a", "metadata": { "editable": true }, @@ -291,7 +291,7 @@ }, { "cell_type": "markdown", - "id": "592cb7cc", + "id": "d6427a35", "metadata": { "editable": true }, @@ -303,7 +303,7 @@ }, { "cell_type": "markdown", - "id": "8ad2c594", + "id": "38c78421", "metadata": { "editable": true }, @@ -314,7 +314,7 @@ }, { "cell_type": "markdown", - "id": "0d53b5e7", + "id": "f23ac12e", "metadata": { "editable": true }, @@ -326,39 +326,39 @@ }, { "cell_type": "markdown", - "id": "a3cbbda7", + "id": "88fd9c03", "metadata": { "editable": true }, "source": [ "where $\\langle y_i \\rangle$ is the mean value. Keep in mind also that\n", "till now we have treated $y_i$ as the exact value. Normally, the\n", - "response (dependent or outcome) variable $y_i$ is the outcome of a\n", + "output (response, target, dependent or outcome) variable $y_i$ is the outcome of a\n", "numerical experiment or another type of experiment and could thus be treated itself as an\n", "approximation to the true value. It is then always accompanied by an\n", "error estimate, often limited to a statistical error estimate given by\n", "the standard deviation discussed earlier. In the discussion here we\n", - "will treat $y_i$ as our exact value for the response variable.\n", + "will treat $y_i$ as our exact value for the output variable.\n", "\n", "In order to find the parameters $\\theta_i$ we will then minimize the spread of $C(\\boldsymbol{\\theta})$, that is we are going to solve the problem" ] }, { "cell_type": "markdown", - "id": "169c08ab", + "id": "a953b1a3", "metadata": { "editable": true }, "source": [ "$$\n", - "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "\\hat{\\boldsymbol{\\theta}}={\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "710a21eb", + "id": "19eb0838", "metadata": { "editable": true }, @@ -368,7 +368,7 @@ }, { "cell_type": "markdown", - "id": "f18b4513", + "id": "a4f910a4", "metadata": { "editable": true }, @@ -380,7 +380,7 @@ }, { "cell_type": "markdown", - "id": "20b56c08", + "id": "5060e089", "metadata": { "editable": true }, @@ -390,7 +390,7 @@ }, { "cell_type": "markdown", - "id": "7f6b7e59", + "id": "8542c5a3", "metadata": { "editable": true }, @@ -402,7 +402,7 @@ }, { "cell_type": "markdown", - "id": "36cdc402", + "id": "475e1436", "metadata": { "editable": true }, @@ -412,7 +412,7 @@ }, { "cell_type": "markdown", - "id": "cb8648c4", + "id": "88abf4ad", "metadata": { "editable": true }, @@ -424,7 +424,7 @@ }, { "cell_type": "markdown", - "id": "9d60b674", + "id": "62f558d9", "metadata": { "editable": true }, @@ -435,7 +435,7 @@ }, { "cell_type": "markdown", - "id": "475a553b", + "id": "99cc5104", "metadata": { "editable": true }, @@ -447,7 +447,7 @@ }, { "cell_type": "markdown", - "id": "69814fd7", + "id": "29043759", "metadata": { "editable": true }, @@ -457,7 +457,7 @@ }, { "cell_type": "markdown", - "id": "75c465dc", + "id": "e9d0f067", "metadata": { "editable": true }, @@ -469,7 +469,7 @@ }, { "cell_type": "markdown", - "id": "fb3f5818", + "id": "c34bfdd2", "metadata": { "editable": true }, @@ -479,19 +479,19 @@ }, { "cell_type": "markdown", - "id": "01c31fce", + "id": "3533f92e", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\theta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "\\hat{\\boldsymbol{\\theta}} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "869d8825", + "id": "135631d4", "metadata": { "editable": true }, @@ -504,14 +504,14 @@ "supervised learning algorithms like classification with logistic\n", "regression or support vector machines, exhibit dimensionalities which\n", "allow for the usage of direct linear algebra methods such as **LU** decomposition or **Singular Value Decomposition** (SVD) for finding the inverse of the matrix\n", - "$\\boldsymbol{X}^T\\boldsymbol{X}$. This is discussed on Thursday this week.\n", + "$\\boldsymbol{X}^T\\boldsymbol{X}$.\n", "\n", - "**Small question**: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$? What kind of problems can we expect?" + "**Small question**: When inverting the matrix $\\boldsymbol{X}^T\\boldsymbol{X}, what kind of problems can we expect?" ] }, { "cell_type": "markdown", - "id": "86056e8c", + "id": "cf25f4a7", "metadata": { "editable": true }, @@ -538,7 +538,7 @@ }, { "cell_type": "markdown", - "id": "d36297d2", + "id": "5d82400a", "metadata": { "editable": true }, @@ -550,7 +550,7 @@ }, { "cell_type": "markdown", - "id": "42a61ab6", + "id": "00d8d123", "metadata": { "editable": true }, @@ -562,7 +562,7 @@ }, { "cell_type": "markdown", - "id": "6216fdfa", + "id": "cce17e14", "metadata": { "editable": true }, @@ -578,7 +578,7 @@ }, { "cell_type": "markdown", - "id": "1d8f89cb", + "id": "65efb775", "metadata": { "editable": true }, @@ -596,7 +596,7 @@ }, { "cell_type": "markdown", - "id": "4c3d1ea5", + "id": "2df9efd6", "metadata": { "editable": true }, @@ -608,7 +608,7 @@ }, { "cell_type": "markdown", - "id": "41d96182", + "id": "37b621d8", "metadata": { "editable": true }, @@ -620,7 +620,7 @@ }, { "cell_type": "markdown", - "id": "187e7d47", + "id": "297be9c9", "metadata": { "editable": true }, @@ -631,7 +631,7 @@ }, { "cell_type": "markdown", - "id": "fe831c6b", + "id": "fe288c99", "metadata": { "editable": true }, @@ -643,7 +643,7 @@ }, { "cell_type": "markdown", - "id": "5e9942fa", + "id": "d4333779", "metadata": { "editable": true }, @@ -653,7 +653,7 @@ }, { "cell_type": "markdown", - "id": "a1589dc4", + "id": "f7e72c3d", "metadata": { "editable": true }, @@ -665,7 +665,7 @@ }, { "cell_type": "markdown", - "id": "f4980fc7", + "id": "3423b702", "metadata": { "editable": true }, @@ -679,7 +679,7 @@ }, { "cell_type": "markdown", - "id": "5e5803ca", + "id": "fe66f713", "metadata": { "editable": true }, @@ -691,7 +691,7 @@ }, { "cell_type": "markdown", - "id": "8ff2d8a0", + "id": "43eed51b", "metadata": { "editable": true }, @@ -703,7 +703,7 @@ }, { "cell_type": "markdown", - "id": "a14f7d1e", + "id": "6f943ccb", "metadata": { "editable": true }, @@ -715,7 +715,7 @@ }, { "cell_type": "markdown", - "id": "8d0c272b", + "id": "5df09c55", "metadata": { "editable": true }, @@ -725,7 +725,7 @@ }, { "cell_type": "markdown", - "id": "9bfc54a6", + "id": "b92d9f2a", "metadata": { "editable": true }, @@ -737,7 +737,7 @@ }, { "cell_type": "markdown", - "id": "f68602df", + "id": "c6cf49ab", "metadata": { "editable": true }, @@ -749,7 +749,7 @@ }, { "cell_type": "markdown", - "id": "f3a9b46b", + "id": "3e000448", "metadata": { "editable": true }, @@ -761,7 +761,7 @@ }, { "cell_type": "markdown", - "id": "8b9c9796", + "id": "fbaa984b", "metadata": { "editable": true }, @@ -775,7 +775,7 @@ }, { "cell_type": "markdown", - "id": "e55c0131", + "id": "f4e444ea", "metadata": { "editable": true }, @@ -787,7 +787,7 @@ }, { "cell_type": "markdown", - "id": "0213690c", + "id": "454ba5ba", "metadata": { "editable": true }, @@ -799,7 +799,7 @@ }, { "cell_type": "markdown", - "id": "4bebb2ab", + "id": "c0434cd9", "metadata": { "editable": true }, @@ -811,7 +811,7 @@ }, { "cell_type": "markdown", - "id": "f90da960", + "id": "f2bcda9d", "metadata": { "editable": true }, @@ -821,7 +821,7 @@ }, { "cell_type": "markdown", - "id": "8178ffe4", + "id": "aea0ba73", "metadata": { "editable": true }, @@ -833,7 +833,7 @@ }, { "cell_type": "markdown", - "id": "d937141e", + "id": "5a58d6e8", "metadata": { "editable": true }, @@ -843,7 +843,7 @@ }, { "cell_type": "markdown", - "id": "a5a8213f", + "id": "927428aa", "metadata": { "editable": true }, @@ -855,7 +855,7 @@ }, { "cell_type": "markdown", - "id": "3bdb5bee", + "id": "372ad443", "metadata": { "editable": true }, @@ -865,7 +865,7 @@ }, { "cell_type": "markdown", - "id": "4a9ff5dc", + "id": "37a99489", "metadata": { "editable": true }, @@ -877,7 +877,7 @@ }, { "cell_type": "markdown", - "id": "97098140", + "id": "4872a1cd", "metadata": { "editable": true }, @@ -889,7 +889,7 @@ }, { "cell_type": "markdown", - "id": "76008e78", + "id": "b21e9cb2", "metadata": { "editable": true }, @@ -901,7 +901,7 @@ }, { "cell_type": "markdown", - "id": "7e07b9bf", + "id": "442375d1", "metadata": { "editable": true }, @@ -916,7 +916,7 @@ }, { "cell_type": "markdown", - "id": "a691ed11", + "id": "aabab237", "metadata": { "editable": true }, @@ -928,7 +928,7 @@ }, { "cell_type": "markdown", - "id": "73cfbba0", + "id": "106f09df", "metadata": { "editable": true }, @@ -938,7 +938,7 @@ }, { "cell_type": "markdown", - "id": "d78a9df5", + "id": "73d8f824", "metadata": { "editable": true }, @@ -950,7 +950,7 @@ }, { "cell_type": "markdown", - "id": "9e1e3a87", + "id": "f2626be7", "metadata": { "editable": true }, @@ -960,7 +960,7 @@ }, { "cell_type": "markdown", - "id": "aea5a5e4", + "id": "81d89d86", "metadata": { "editable": true }, @@ -972,7 +972,7 @@ }, { "cell_type": "markdown", - "id": "89e6fde8", + "id": "59c2576a", "metadata": { "editable": true }, @@ -982,7 +982,7 @@ }, { "cell_type": "markdown", - "id": "6c0f4449", + "id": "4b7bf3f9", "metadata": { "editable": true }, @@ -994,7 +994,7 @@ }, { "cell_type": "markdown", - "id": "a390ac6e", + "id": "ea1e8943", "metadata": { "editable": true }, @@ -1006,7 +1006,7 @@ }, { "cell_type": "markdown", - "id": "59d6ab1c", + "id": "d0425d72", "metadata": { "editable": true }, @@ -1018,7 +1018,7 @@ }, { "cell_type": "markdown", - "id": "8c99ee21", + "id": "38386834", "metadata": { "editable": true }, @@ -1028,7 +1028,7 @@ }, { "cell_type": "markdown", - "id": "7ea454e1", + "id": "bfbc4bd3", "metadata": { "editable": true }, @@ -1040,7 +1040,7 @@ }, { "cell_type": "markdown", - "id": "5a555af4", + "id": "3ced3cec", "metadata": { "editable": true }, @@ -1053,7 +1053,7 @@ }, { "cell_type": "markdown", - "id": "a39c02c1", + "id": "fc6f1dec", "metadata": { "editable": true }, @@ -1065,7 +1065,7 @@ }, { "cell_type": "markdown", - "id": "a9804433", + "id": "699662d2", "metadata": { "editable": true }, @@ -1075,7 +1075,7 @@ }, { "cell_type": "markdown", - "id": "8be8de93", + "id": "0991592c", "metadata": { "editable": true }, @@ -1087,7 +1087,7 @@ }, { "cell_type": "markdown", - "id": "4139dd13", + "id": "450dd961", "metadata": { "editable": true }, @@ -1097,7 +1097,7 @@ }, { "cell_type": "markdown", - "id": "6f1c778a", + "id": "a13de0fa", "metadata": { "editable": true }, @@ -1109,7 +1109,7 @@ }, { "cell_type": "markdown", - "id": "03ae847f", + "id": "c64bc916", "metadata": { "editable": true }, @@ -1119,7 +1119,7 @@ }, { "cell_type": "markdown", - "id": "054f8ba7", + "id": "ab1e18bc", "metadata": { "editable": true }, @@ -1131,7 +1131,7 @@ }, { "cell_type": "markdown", - "id": "61c62efa", + "id": "f1e74f34", "metadata": { "editable": true }, @@ -1141,7 +1141,7 @@ }, { "cell_type": "markdown", - "id": "ac84f4fd", + "id": "1ad3ebd9", "metadata": { "editable": true }, @@ -1153,7 +1153,7 @@ }, { "cell_type": "markdown", - "id": "b2f0a3d0", + "id": "37d1b4ca", "metadata": { "editable": true }, @@ -1163,7 +1163,7 @@ }, { "cell_type": "markdown", - "id": "9c6460f2", + "id": "3d5fc67a", "metadata": { "editable": true }, @@ -1175,67 +1175,7 @@ }, { "cell_type": "markdown", - "id": "0c4aebc4", - "metadata": { - "editable": true - }, - "source": [ - "## Other useful relations\n", - "\n", - "We list here some other useful relations we may encounter (recall that vectors are defined by boldfaced low-key letters)" - ] - }, - { - "cell_type": "markdown", - "id": "67aef36e", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{\\partial (\\boldsymbol{x}^T\\boldsymbol{a})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "30c88be7", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "fb84fcda", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{\\partial tr(\\boldsymbol{B}\\boldsymbol{A})}{\\partial \\boldsymbol{A}} = \\boldsymbol{B}^T,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "ebbeeb3b", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{\\partial \\log{\\vert\\boldsymbol{A}\\vert}}{\\partial \\boldsymbol{A}} = (\\boldsymbol{A}^{-1})^T.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "fb88afbf", + "id": "d98472d9", "metadata": { "editable": true }, @@ -1251,7 +1191,7 @@ }, { "cell_type": "markdown", - "id": "89512329", + "id": "5d859f20", "metadata": { "editable": true }, @@ -1263,7 +1203,7 @@ }, { "cell_type": "markdown", - "id": "a651d788", + "id": "e7bb91ec", "metadata": { "editable": true }, @@ -1273,7 +1213,7 @@ }, { "cell_type": "markdown", - "id": "dca9d68f", + "id": "1553864b", "metadata": { "editable": true }, @@ -1285,25 +1225,25 @@ }, { "cell_type": "markdown", - "id": "2865ced8", + "id": "f8cc9c71", "metadata": { "editable": true }, "source": [ "For ordinary least squares, it is inversely proportional (derivation\n", "next week) with the variance of the optimal parameters\n", - "$\\hat{\\boldsymbol{\\theta}}$. Furthermore, we will see later this week that it is\n", + "$\\hat{\\boldsymbol{\\theta}}$. Furthermore, we will see next week that it is\n", "(aside the factor $1/n$) equal to the covariance matrix. It plays also a very\n", "important role in optmization algorithms and Principal Component\n", "Analysis as a way to reduce the dimensionality of a machine learning/data analysis\n", - "problem.\n", - "v\n", + "problem. We will discuss this in greater detail next week when we introduce gradient methods.\n", + "\n", "**Linear algebra question:** Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!)." ] }, { "cell_type": "markdown", - "id": "1a7a4045", + "id": "f1f01477", "metadata": { "editable": true }, @@ -1315,7 +1255,7 @@ }, { "cell_type": "markdown", - "id": "a483362e", + "id": "f2dea7e1", "metadata": { "editable": true }, @@ -1327,7 +1267,7 @@ }, { "cell_type": "markdown", - "id": "dcf896a3", + "id": "76b0f99e", "metadata": { "editable": true }, @@ -1337,7 +1277,7 @@ }, { "cell_type": "markdown", - "id": "13c80a7c", + "id": "cfa3ca25", "metadata": { "editable": true }, @@ -1349,7 +1289,7 @@ }, { "cell_type": "markdown", - "id": "582225c6", + "id": "9a9163d8", "metadata": { "editable": true }, @@ -1359,7 +1299,7 @@ }, { "cell_type": "markdown", - "id": "90476b18", + "id": "c6b54a1e", "metadata": { "editable": true }, @@ -1371,7 +1311,7 @@ }, { "cell_type": "markdown", - "id": "267ae1ff", + "id": "659c98d0", "metadata": { "editable": true }, @@ -1381,7 +1321,7 @@ }, { "cell_type": "markdown", - "id": "8c9cb8c5", + "id": "209b5b37", "metadata": { "editable": true }, @@ -1395,7 +1335,7 @@ }, { "cell_type": "markdown", - "id": "acbe4114", + "id": "b325d0c3", "metadata": { "editable": true }, @@ -1407,7 +1347,7 @@ }, { "cell_type": "markdown", - "id": "ba945f21", + "id": "c49a2fcd", "metadata": { "editable": true }, @@ -1418,7 +1358,7 @@ }, { "cell_type": "markdown", - "id": "dc442b90", + "id": "6978ef70", "metadata": { "editable": true }, @@ -1431,14 +1371,14 @@ { "cell_type": "code", "execution_count": 1, - "id": "883026f0", + "id": "01530c39", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ - "# matrix inversion to find beta\n", + "# matrix inversion to find theta\n", "# First we set up the data\n", "import numpy as np\n", "x = np.random.rand(100)\n", @@ -1451,14 +1391,14 @@ "X[:,2] = x**2\n", "X[:,3] = x**3\n", "X[:,4] = x**4\n", - "beta = (np.linalg.inv(X.T @ X) @ X.T ) @ y\n", + "theta = (np.linalg.inv(X.T @ X) @ X.T ) @ y\n", "# and then make the prediction\n", - "ytilde = X @ beta" + "ytilde = X @ theta" ] }, { "cell_type": "markdown", - "id": "826466d3", + "id": "90ff28d4", "metadata": { "editable": true }, @@ -1469,7 +1409,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "decea3b4", + "id": "1d052cbe", "metadata": { "collapsed": false, "editable": true @@ -1482,7 +1422,7 @@ }, { "cell_type": "markdown", - "id": "b52ed870", + "id": "3362efd4", "metadata": { "editable": true }, @@ -1496,7 +1436,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "ee4d5722", + "id": "3e9e681f", "metadata": { "collapsed": false, "editable": true @@ -1509,7 +1449,7 @@ }, { "cell_type": "markdown", - "id": "01892aaa", + "id": "78e4427f", "metadata": { "editable": true }, @@ -1520,7 +1460,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "d6ce54e2", + "id": "20aab659", "metadata": { "collapsed": false, "editable": true @@ -1532,7 +1472,7 @@ }, { "cell_type": "markdown", - "id": "91ed5956", + "id": "326f641d", "metadata": { "editable": true }, @@ -1543,7 +1483,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "7ffd63cb", + "id": "c8e80cf1", "metadata": { "collapsed": false, "editable": true @@ -1559,7 +1499,7 @@ }, { "cell_type": "markdown", - "id": "737370ed", + "id": "aba7b289", "metadata": { "editable": true }, @@ -1570,7 +1510,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "f1bbe22d", + "id": "7235bf1b", "metadata": { "collapsed": false, "editable": true @@ -1584,7 +1524,7 @@ }, { "cell_type": "markdown", - "id": "8e3a976b", + "id": "3eaf5f8d", "metadata": { "editable": true }, @@ -1605,7 +1545,7 @@ }, { "cell_type": "markdown", - "id": "8c2dee8c", + "id": "6fc5dc26", "metadata": { "editable": true }, @@ -1616,7 +1556,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "89d0750d", + "id": "98ca5627", "metadata": { "collapsed": false, "editable": true @@ -1651,16 +1591,16 @@ "X[:,4] = x**4\n", "# We split the data in test and training data\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n", - "# matrix inversion to find beta\n", - "beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", - "print(beta)\n", + "# matrix inversion to find theta\n", + "theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", + "print(theta)\n", "# and then make the prediction\n", - "ytilde = X_train @ beta\n", + "ytilde = X_train @ theta\n", "print(\"Training R2\")\n", "print(R2(y_train,ytilde))\n", "print(\"Training MSE\")\n", "print(MSE(y_train,ytilde))\n", - "ypredict = X_test @ beta\n", + "ypredict = X_test @ theta\n", "print(\"Test R2\")\n", "print(R2(y_test,ypredict))\n", "print(\"Test MSE\")\n", @@ -1669,7 +1609,7 @@ }, { "cell_type": "markdown", - "id": "12c44026", + "id": "09f0c1c6", "metadata": { "editable": true }, @@ -1680,7 +1620,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "6daa8ad9", + "id": "02220d97", "metadata": { "collapsed": false, "editable": true @@ -1705,7 +1645,7 @@ }, { "cell_type": "markdown", - "id": "9b4cbdbc", + "id": "5494af4d", "metadata": { "editable": true }, @@ -1717,7 +1657,7 @@ }, { "cell_type": "markdown", - "id": "7f685bde", + "id": "048560ee", "metadata": { "editable": true }, @@ -1746,7 +1686,7 @@ }, { "cell_type": "markdown", - "id": "a8e92a0b", + "id": "85b56c79", "metadata": { "editable": true }, @@ -1771,7 +1711,7 @@ }, { "cell_type": "markdown", - "id": "b0ac1e21", + "id": "60937483", "metadata": { "editable": true }, @@ -1791,7 +1731,7 @@ }, { "cell_type": "markdown", - "id": "3c8d9613", + "id": "5d0ea6a3", "metadata": { "editable": true }, @@ -1818,7 +1758,7 @@ }, { "cell_type": "markdown", - "id": "1d817e20", + "id": "8f6fc4e6", "metadata": { "editable": true }, @@ -1831,7 +1771,7 @@ }, { "cell_type": "markdown", - "id": "20673f07", + "id": "4bd392ed", "metadata": { "editable": true }, @@ -1843,7 +1783,7 @@ }, { "cell_type": "markdown", - "id": "4a36afc7", + "id": "4e33d3e3", "metadata": { "editable": true }, @@ -1854,7 +1794,7 @@ }, { "cell_type": "markdown", - "id": "4db40306", + "id": "7e1c73ac", "metadata": { "editable": true }, @@ -1870,7 +1810,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "ac782b4e", + "id": "2c315558", "metadata": { "collapsed": false, "editable": true @@ -1904,7 +1844,7 @@ }, { "cell_type": "markdown", - "id": "d60792bf", + "id": "aaa58775", "metadata": { "editable": true }, @@ -1914,7 +1854,7 @@ }, { "cell_type": "markdown", - "id": "6dbeeee3", + "id": "02b198ed", "metadata": { "editable": true }, @@ -1929,7 +1869,7 @@ }, { "cell_type": "markdown", - "id": "f1bd9e06", + "id": "4cef197d", "metadata": { "editable": true }, @@ -1941,7 +1881,7 @@ }, { "cell_type": "markdown", - "id": "c4e9c9cc", + "id": "23b34528", "metadata": { "editable": true }, @@ -1951,7 +1891,7 @@ }, { "cell_type": "markdown", - "id": "c682dfb2", + "id": "b2847956", "metadata": { "editable": true }, @@ -1967,7 +1907,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "f95e4477", + "id": "b8dacc33", "metadata": { "collapsed": false, "editable": true @@ -1984,7 +1924,7 @@ }, { "cell_type": "markdown", - "id": "3eaa6d2e", + "id": "cd89705f", "metadata": { "editable": true }, @@ -1997,7 +1937,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "fe340fe4", + "id": "d35541fd", "metadata": { "collapsed": false, "editable": true @@ -2044,7 +1984,7 @@ }, { "cell_type": "markdown", - "id": "e2f14a01", + "id": "36014ec9", "metadata": { "editable": true }, @@ -2058,7 +1998,7 @@ }, { "cell_type": "markdown", - "id": "ad741d5a", + "id": "16fb5f8f", "metadata": { "editable": true }, @@ -2070,7 +2010,7 @@ }, { "cell_type": "markdown", - "id": "05b080a8", + "id": "f1ec698e", "metadata": { "editable": true }, @@ -2082,7 +2022,7 @@ }, { "cell_type": "markdown", - "id": "5c64cd00", + "id": "787f58f5", "metadata": { "editable": true }, @@ -2094,7 +2034,7 @@ }, { "cell_type": "markdown", - "id": "66d8648a", + "id": "e23435fd", "metadata": { "editable": true }, @@ -2104,7 +2044,7 @@ }, { "cell_type": "markdown", - "id": "47178854", + "id": "ccaad283", "metadata": { "editable": true }, @@ -2116,7 +2056,7 @@ }, { "cell_type": "markdown", - "id": "9d89dcc7", + "id": "be8092da", "metadata": { "editable": true }, @@ -2126,7 +2066,7 @@ }, { "cell_type": "markdown", - "id": "d037dda7", + "id": "6e055e08", "metadata": { "editable": true }, @@ -2138,7 +2078,7 @@ }, { "cell_type": "markdown", - "id": "4d559b4d", + "id": "ff86c9bc", "metadata": { "editable": true }, @@ -2149,7 +2089,7 @@ }, { "cell_type": "markdown", - "id": "f8eec117", + "id": "b4f2e747", "metadata": { "editable": true }, @@ -2161,7 +2101,7 @@ }, { "cell_type": "markdown", - "id": "ecc2fed3", + "id": "5a9a9f87", "metadata": { "editable": true }, @@ -2173,7 +2113,7 @@ }, { "cell_type": "markdown", - "id": "8d3ae448", + "id": "1232e11a", "metadata": { "editable": true }, @@ -2183,7 +2123,7 @@ }, { "cell_type": "markdown", - "id": "0d7267fa", + "id": "a807825b", "metadata": { "editable": true }, @@ -2195,7 +2135,7 @@ }, { "cell_type": "markdown", - "id": "06e5716f", + "id": "4fa9855c", "metadata": { "editable": true }, @@ -2207,7 +2147,7 @@ }, { "cell_type": "markdown", - "id": "f8f001a6", + "id": "dcffefcf", "metadata": { "editable": true }, @@ -2217,7 +2157,7 @@ }, { "cell_type": "markdown", - "id": "761f6c51", + "id": "3c08ff3a", "metadata": { "editable": true }, @@ -2229,7 +2169,7 @@ }, { "cell_type": "markdown", - "id": "468bc321", + "id": "c2483a13", "metadata": { "editable": true }, @@ -2239,7 +2179,7 @@ }, { "cell_type": "markdown", - "id": "eb61e7ef", + "id": "9a82a8c5", "metadata": { "editable": true }, @@ -2251,7 +2191,7 @@ }, { "cell_type": "markdown", - "id": "be3586ce", + "id": "743f7309", "metadata": { "editable": true }, @@ -2261,7 +2201,7 @@ }, { "cell_type": "markdown", - "id": "f79b7c51", + "id": "e6d7b31a", "metadata": { "editable": true }, @@ -2278,7 +2218,7 @@ "As we will also see in the first project, \n", "this may\n", "however not the be case in general and a standard matrix inversion\n", - "algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.\n", + "algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below and in other examples.\n", "\n", "There is however a way to circumvent this problem and also\n", "gain some insights about the ordinary least squares approach, and\n", @@ -2301,7 +2241,7 @@ }, { "cell_type": "markdown", - "id": "ea84dc35", + "id": "518d6bb4", "metadata": { "editable": true }, @@ -2318,7 +2258,7 @@ }, { "cell_type": "markdown", - "id": "0ad4b3f9", + "id": "056673c8", "metadata": { "editable": true }, @@ -2341,7 +2281,7 @@ }, { "cell_type": "markdown", - "id": "671100e8", + "id": "63daba59", "metadata": { "editable": true }, @@ -2358,7 +2298,7 @@ }, { "cell_type": "markdown", - "id": "9d07ce4a", + "id": "89b74667", "metadata": { "editable": true }, @@ -2377,7 +2317,7 @@ }, { "cell_type": "markdown", - "id": "a26b13de", + "id": "588dd662", "metadata": { "editable": true }, @@ -2388,7 +2328,7 @@ }, { "cell_type": "markdown", - "id": "2352aaa2", + "id": "eb8348c6", "metadata": { "editable": true }, @@ -2400,7 +2340,7 @@ }, { "cell_type": "markdown", - "id": "5c201fe7", + "id": "62027bc1", "metadata": { "editable": true }, @@ -2418,7 +2358,7 @@ }, { "cell_type": "markdown", - "id": "a711f9aa", + "id": "0c73c4b7", "metadata": { "editable": true }, @@ -2434,7 +2374,7 @@ }, { "cell_type": "markdown", - "id": "816ee812", + "id": "a3a2d330", "metadata": { "editable": true }, @@ -2446,7 +2386,7 @@ }, { "cell_type": "markdown", - "id": "ca4f2d5d", + "id": "374b88cc", "metadata": { "editable": true }, @@ -2456,14 +2396,412 @@ }, { "cell_type": "markdown", - "id": "11b18875", + "id": "d5aa42e6", + "metadata": { + "editable": true + }, + "source": [ + "## Ridge and LASSO Regression\n", + "\n", + "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", + "our optimization problem is" + ] + }, + { + "cell_type": "markdown", + "id": "1c01a816", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f556eddc", + "metadata": { + "editable": true + }, + "source": [ + "or we can state it as" + ] + }, + { + "cell_type": "markdown", + "id": "dcf14a16", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "08c37095", + "metadata": { + "editable": true + }, + "source": [ + "where we have used the definition of a norm-2 vector, that is" + ] + }, + { + "cell_type": "markdown", + "id": "6815fa1b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "daf50946", + "metadata": { + "editable": true + }, + "source": [ + "By minimizing the above equation with respect to the parameters\n", + "$\\boldsymbol{\\theta}$ we could then obtain an analytical expression for the\n", + "parameters $\\boldsymbol{\\theta}$. We can add a regularization parameter $\\lambda$ by\n", + "defining a new cost function to be optimized, that is" + ] + }, + { + "cell_type": "markdown", + "id": "a7a60cb3", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "38105bfd", + "metadata": { + "editable": true + }, + "source": [ + "which leads to the Ridge regression minimization problem where we\n", + "require that $\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\\le t$, where $t$ is\n", + "a finite number larger than zero. By defining" + ] + }, + { + "cell_type": "markdown", + "id": "3d697464", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c298d7ba", + "metadata": { + "editable": true + }, + "source": [ + "we have a new optimization equation" + ] + }, + { + "cell_type": "markdown", + "id": "45265b59", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ab6ce408", + "metadata": { + "editable": true + }, + "source": [ + "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", + "\n", + "Here we have defined the norm-1 as" + ] + }, + { + "cell_type": "markdown", + "id": "ca0350a7", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e8146e6f", + "metadata": { + "editable": true + }, + "source": [ + "## Deriving the Ridge Regression Equations\n", + "\n", + "Using the matrix-vector expression for Ridge regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have" + ] + }, + { + "cell_type": "markdown", + "id": "05d97a86", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\boldsymbol{\\theta}^T\\boldsymbol{\\theta},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f84e7118", + "metadata": { + "editable": true + }, + "source": [ + "and \n", + "taking the derivatives with respect to $\\boldsymbol{\\theta}$ we obtain then\n", + "a slightly modified matrix inversion problem which for finite values\n", + "of $\\lambda$ does not suffer from singularity problems. We obtain\n", + "the optimal parameters" + ] + }, + { + "cell_type": "markdown", + "id": "28be9ec4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b9f5a5d8", + "metadata": { + "editable": true + }, + "source": [ + "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" + ] + }, + { + "cell_type": "markdown", + "id": "86b43b21", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\sum_{i=0}^{p-1} \\theta_i^2 \\leq t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "58744172", + "metadata": { + "editable": true + }, + "source": [ + "with $t$ a finite positive number. \n", + "\n", + "If we keep the $1/n$ factor, the equation for the optimal $\\theta$ changes to" + ] + }, + { + "cell_type": "markdown", + "id": "dd515e6f", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e8c98bfe", + "metadata": { + "editable": true + }, + "source": [ + "In many textbooks the $1/n$ term is often omitted. Note that a library like **Scikit-Learn** does not include the $1/n$ factor in the setup of the cost function.\n", + "\n", + "When we compare this with the ordinary least squares result we have" + ] + }, + { + "cell_type": "markdown", + "id": "08cc9626", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "725df93e", + "metadata": { + "editable": true + }, + "source": [ + "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", + "\n", + "We see that Ridge regression is nothing but the standard OLS with a\n", + "modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n", + "particular for our discussion of the bias-variance tradeoff are rather\n", + "interesting. We will see that for specific values of $\\lambda$, we may\n", + "even reduce the variance of the optimal parameters $\\boldsymbol{\\theta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n", + "\n", + "When we have discussed the singular value decomposition of the design\n", + "matrix $\\boldsymbol{X}$, we will in turn perform a more rigorous mathematical\n", + "discussion of Ridge regression.\n", + "\n", + "The code here is a simple demonstration of how to implement Ridge regression with our own code and compare this with scikit-learn." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "8d0397cd", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd\n", + "import matplotlib.pyplot as plt\n", + "from sklearn.model_selection import train_test_split\n", + "from sklearn import linear_model\n", + "\n", + "def MSE(y_data,y_model):\n", + " n = np.size(y_model)\n", + " return np.sum((y_data-y_model)**2)/n\n", + "\n", + "\n", + "# A seed just to ensure that the random numbers are the same for every run.\n", + "# Useful for eventual debugging.\n", + "np.random.seed(3155)\n", + "\n", + "n = 100\n", + "x = np.random.rand(n)\n", + "y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)\n", + "\n", + "Maxpolydegree = 20\n", + "X = np.zeros((n,Maxpolydegree))\n", + "#We include explicitely the intercept column\n", + "for degree in range(Maxpolydegree):\n", + " X[:,degree] = x**degree\n", + "# We split the data in test and training data\n", + "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n", + "\n", + "p = Maxpolydegree\n", + "I = np.eye(p,p)\n", + "# Decide which values of lambda to use\n", + "nlambdas = 6\n", + "MSEOwnRidgePredict = np.zeros(nlambdas)\n", + "MSERidgePredict = np.zeros(nlambdas)\n", + "lambdas = np.logspace(-4, 2, nlambdas)\n", + "for i in range(nlambdas):\n", + " lmb = lambdas[i]\n", + " OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n", + " # Note: we include the intercept column and no scaling\n", + " RegRidge = linear_model.Ridge(lmb,fit_intercept=False)\n", + " RegRidge.fit(X_train,y_train)\n", + " # and then make the prediction\n", + " ytildeOwnRidge = X_train @ OwnRidgeTheta\n", + " ypredictOwnRidge = X_test @ OwnRidgeTheta\n", + " ytildeRidge = RegRidge.predict(X_train)\n", + " ypredictRidge = RegRidge.predict(X_test)\n", + " MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)\n", + " MSERidgePredict[i] = MSE(y_test,ypredictRidge)\n", + " print(\"Theta values for own Ridge implementation\")\n", + " print(OwnRidgeTheta)\n", + " print(\"Theta values for Scikit-Learn Ridge implementation\")\n", + " print(RegRidge.coef_)\n", + " print(\"MSE values for own Ridge implementation\")\n", + " print(MSEOwnRidgePredict[i])\n", + " print(\"MSE values for Scikit-Learn Ridge implementation\")\n", + " print(MSERidgePredict[i])\n", + "\n", + "# Now plot the results\n", + "plt.figure()\n", + "plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'r', label = 'MSE own Ridge Test')\n", + "plt.plot(np.log10(lambdas), MSERidgePredict, 'g', label = 'MSE Ridge Test')\n", + "\n", + "plt.xlabel('log10(lambda)')\n", + "plt.ylabel('MSE')\n", + "plt.legend()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "dbc56c2b", + "metadata": { + "editable": true + }, + "source": [ + "The results here agree when we force **Scikit-Learn**'s Ridge function to include the first column in our design matrix.\n", + "We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.\n", + "What happens if we do not include the intercept in our fit? We will discuss this in more detail next week." + ] + }, + { + "cell_type": "markdown", + "id": "cfab477f", "metadata": { "editable": true }, "source": [ "## Basic math of the SVD\n", "\n", - "From standard linear algebra we know that a square matrix $\\boldsymbol{X}$ can be diagonalized if and only it is \n", + "From standard linear algebra we know that a square matrix $\\boldsymbol{X}$ can be diagonalized if and only if it is \n", "a so-called [normal matrix](https://en.wikipedia.org/wiki/Normal_matrix), that is if $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times n}$\n", "we have $\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{X}^T\\boldsymbol{X}$ or if $\\boldsymbol{X}\\in {\\mathbb{C}}^{n\\times n}$ we have $\\boldsymbol{X}\\boldsymbol{X}^{\\dagger}=\\boldsymbol{X}^{\\dagger}\\boldsymbol{X}$.\n", "The matrix has then a set of eigenpairs" @@ -2471,7 +2809,7 @@ }, { "cell_type": "markdown", - "id": "a6cebe96", + "id": "bc176ed4", "metadata": { "editable": true }, @@ -2483,7 +2821,7 @@ }, { "cell_type": "markdown", - "id": "4056c51d", + "id": "a9eef8e3", "metadata": { "editable": true }, @@ -2493,7 +2831,7 @@ }, { "cell_type": "markdown", - "id": "8c6fc251", + "id": "c33552ec", "metadata": { "editable": true }, @@ -2505,7 +2843,7 @@ }, { "cell_type": "markdown", - "id": "85d151ae", + "id": "9465d4f6", "metadata": { "editable": true }, @@ -2515,7 +2853,7 @@ }, { "cell_type": "markdown", - "id": "a7ed48ac", + "id": "3cd89a35", "metadata": { "editable": true }, @@ -2527,7 +2865,7 @@ }, { "cell_type": "markdown", - "id": "d61100de", + "id": "13a9f453", "metadata": { "editable": true }, @@ -2539,7 +2877,7 @@ }, { "cell_type": "markdown", - "id": "ae931201", + "id": "6c909f72", "metadata": { "editable": true }, @@ -2554,7 +2892,7 @@ }, { "cell_type": "markdown", - "id": "a1a55970", + "id": "b32ee709", "metadata": { "editable": true }, @@ -2565,7 +2903,7 @@ }, { "cell_type": "markdown", - "id": "7aff14ea", + "id": "e5d3f72c", "metadata": { "editable": true }, @@ -2585,7 +2923,7 @@ }, { "cell_type": "markdown", - "id": "c5bcd441", + "id": "18f488f0", "metadata": { "editable": true }, @@ -2597,7 +2935,7 @@ }, { "cell_type": "markdown", - "id": "ed33684a", + "id": "93618783", "metadata": { "editable": true }, @@ -2607,7 +2945,7 @@ }, { "cell_type": "markdown", - "id": "1d6ca687", + "id": "02f57528", "metadata": { "editable": true }, @@ -2619,7 +2957,7 @@ }, { "cell_type": "markdown", - "id": "c0535cd8", + "id": "762833f4", "metadata": { "editable": true }, @@ -2648,7 +2986,7 @@ }, { "cell_type": "markdown", - "id": "c6c08bf2", + "id": "9ab8ec4f", "metadata": { "editable": true }, @@ -2675,7 +3013,7 @@ }, { "cell_type": "markdown", - "id": "487a7ae7", + "id": "3e526d1f", "metadata": { "editable": true }, @@ -2685,8 +3023,8 @@ }, { "cell_type": "code", - "execution_count": 12, - "id": "3ff87df9", + "execution_count": 13, + "id": "f5d258ce", "metadata": { "collapsed": false, "editable": true @@ -2726,7 +3064,7 @@ }, { "cell_type": "markdown", - "id": "600f74f5", + "id": "4e4bf198", "metadata": { "editable": true }, @@ -2743,7 +3081,7 @@ }, { "cell_type": "markdown", - "id": "da923c99", + "id": "33a42ac4", "metadata": { "editable": true }, @@ -2766,7 +3104,7 @@ }, { "cell_type": "markdown", - "id": "989e9de3", + "id": "7b5d1bbb", "metadata": { "editable": true }, @@ -2780,7 +3118,7 @@ }, { "cell_type": "markdown", - "id": "b010904b", + "id": "1fee5f54", "metadata": { "editable": true }, @@ -2799,7 +3137,7 @@ }, { "cell_type": "markdown", - "id": "1385bdc7", + "id": "235f2975", "metadata": { "editable": true }, @@ -2809,7 +3147,7 @@ }, { "cell_type": "markdown", - "id": "1824f1de", + "id": "fd7415b7", "metadata": { "editable": true }, @@ -2821,7 +3159,7 @@ }, { "cell_type": "markdown", - "id": "50fb0869", + "id": "7c265b5d", "metadata": { "editable": true }, @@ -2835,7 +3173,7 @@ }, { "cell_type": "markdown", - "id": "242e0186", + "id": "976c31a2", "metadata": { "editable": true }, @@ -2847,7 +3185,7 @@ }, { "cell_type": "markdown", - "id": "b3d8ff35", + "id": "8dc8acd7", "metadata": { "editable": true }, @@ -2857,7 +3195,7 @@ }, { "cell_type": "markdown", - "id": "a8799cf8", + "id": "a1dc8e8e", "metadata": { "editable": true }, @@ -2869,7 +3207,7 @@ }, { "cell_type": "markdown", - "id": "69ecbf25", + "id": "eb688af0", "metadata": { "editable": true }, @@ -2886,7 +3224,7 @@ }, { "cell_type": "markdown", - "id": "accaad01", + "id": "eeadd85e", "metadata": { "editable": true }, @@ -2896,7 +3234,7 @@ }, { "cell_type": "markdown", - "id": "7c6d6458", + "id": "e9375ad7", "metadata": { "editable": true }, @@ -2912,7 +3250,7 @@ }, { "cell_type": "markdown", - "id": "92943a13", + "id": "025f1ca1", "metadata": { "editable": true }, @@ -2922,7 +3260,7 @@ }, { "cell_type": "markdown", - "id": "27a56b68", + "id": "d44951a1", "metadata": { "editable": true }, @@ -2938,7 +3276,7 @@ }, { "cell_type": "markdown", - "id": "7fab099a", + "id": "ecc9216b", "metadata": { "editable": true }, @@ -2948,7 +3286,7 @@ }, { "cell_type": "markdown", - "id": "86ef772d", + "id": "28e2ff44", "metadata": { "editable": true }, @@ -2964,7 +3302,7 @@ }, { "cell_type": "markdown", - "id": "a9fb58a3", + "id": "935324af", "metadata": { "editable": true }, @@ -2974,7 +3312,7 @@ }, { "cell_type": "markdown", - "id": "218d36d5", + "id": "a15e7526", "metadata": { "editable": true }, @@ -2991,7 +3329,7 @@ }, { "cell_type": "markdown", - "id": "3c358960", + "id": "5d3cef55", "metadata": { "editable": true }, @@ -3003,7 +3341,7 @@ }, { "cell_type": "markdown", - "id": "384a8da4", + "id": "0fab1783", "metadata": { "editable": true }, @@ -3015,7 +3353,7 @@ }, { "cell_type": "markdown", - "id": "506802df", + "id": "f9a8450e", "metadata": { "editable": true }, @@ -3027,7 +3365,7 @@ }, { "cell_type": "markdown", - "id": "d160546b", + "id": "485d1023", "metadata": { "editable": true }, @@ -3037,7 +3375,7 @@ }, { "cell_type": "markdown", - "id": "22fa787b", + "id": "a2da5dbe", "metadata": { "editable": true }, @@ -3049,7 +3387,7 @@ }, { "cell_type": "markdown", - "id": "4f9cc0fa", + "id": "13959099", "metadata": { "editable": true }, @@ -3061,7 +3399,7 @@ }, { "cell_type": "markdown", - "id": "5228fab9", + "id": "23f0b903", "metadata": { "editable": true }, @@ -3073,7 +3411,7 @@ }, { "cell_type": "markdown", - "id": "803c129e", + "id": "1c344e77", "metadata": { "editable": true }, @@ -3083,7 +3421,7 @@ }, { "cell_type": "markdown", - "id": "a0229d06", + "id": "d8064053", "metadata": { "editable": true }, @@ -3095,7 +3433,7 @@ }, { "cell_type": "markdown", - "id": "927c0677", + "id": "714ab01f", "metadata": { "editable": true }, @@ -3105,7 +3443,7 @@ }, { "cell_type": "markdown", - "id": "9b1faff4", + "id": "a3ff6988", "metadata": { "editable": true }, @@ -3117,7 +3455,7 @@ }, { "cell_type": "markdown", - "id": "2a9ba785", + "id": "5c204f79", "metadata": { "editable": true }, @@ -3133,7 +3471,7 @@ }, { "cell_type": "markdown", - "id": "13dd8452", + "id": "78c51f10", "metadata": { "editable": true }, @@ -3145,7 +3483,7 @@ }, { "cell_type": "markdown", - "id": "db18ae22", + "id": "2c432ed4", "metadata": { "editable": true }, @@ -3157,7 +3495,7 @@ }, { "cell_type": "markdown", - "id": "cc002ef4", + "id": "68a7b505", "metadata": { "editable": true }, @@ -3167,7 +3505,7 @@ }, { "cell_type": "markdown", - "id": "7808b2cd", + "id": "14268cd5", "metadata": { "editable": true }, @@ -3179,7 +3517,7 @@ }, { "cell_type": "markdown", - "id": "f6a8e671", + "id": "f3207844", "metadata": { "editable": true }, @@ -3190,7 +3528,7 @@ }, { "cell_type": "markdown", - "id": "e6298500", + "id": "4c7f5556", "metadata": { "editable": true }, @@ -3202,7 +3540,7 @@ }, { "cell_type": "markdown", - "id": "1a70e57a", + "id": "01bd8826", "metadata": { "editable": true }, @@ -3212,7 +3550,7 @@ }, { "cell_type": "markdown", - "id": "f2091227", + "id": "cb083248", "metadata": { "editable": true }, @@ -3224,7 +3562,7 @@ }, { "cell_type": "markdown", - "id": "029a7fce", + "id": "66bd1ec9", "metadata": { "editable": true }, @@ -3234,7 +3572,7 @@ }, { "cell_type": "markdown", - "id": "62dd4376", + "id": "b92a06bb", "metadata": { "editable": true }, @@ -3246,7 +3584,7 @@ }, { "cell_type": "markdown", - "id": "00865f17", + "id": "dad0b409", "metadata": { "editable": true }, @@ -3257,7 +3595,7 @@ }, { "cell_type": "markdown", - "id": "64bd2f73", + "id": "75d85fa8", "metadata": { "editable": true }, @@ -3269,7 +3607,7 @@ }, { "cell_type": "markdown", - "id": "ae49067e", + "id": "d0918773", "metadata": { "editable": true }, @@ -3287,7 +3625,7 @@ }, { "cell_type": "markdown", - "id": "9ea73c78", + "id": "65592b3b", "metadata": { "editable": true }, @@ -3303,7 +3641,7 @@ }, { "cell_type": "markdown", - "id": "6dcf8fff", + "id": "442404b8", "metadata": { "editable": true }, @@ -3315,7 +3653,7 @@ }, { "cell_type": "markdown", - "id": "8f12f6ca", + "id": "50eca816", "metadata": { "editable": true }, @@ -3327,7 +3665,7 @@ }, { "cell_type": "markdown", - "id": "03362376", + "id": "a27e487e", "metadata": { "editable": true }, @@ -3339,7 +3677,7 @@ }, { "cell_type": "markdown", - "id": "76d4b169", + "id": "49c74d48", "metadata": { "editable": true }, @@ -3352,7 +3690,7 @@ }, { "cell_type": "markdown", - "id": "6775daac", + "id": "214feab7", "metadata": { "editable": true }, @@ -3368,7 +3706,7 @@ }, { "cell_type": "markdown", - "id": "dc1392b1", + "id": "e61c0669", "metadata": { "editable": true }, @@ -3382,7 +3720,7 @@ }, { "cell_type": "markdown", - "id": "7faf1d79", + "id": "9f3db006", "metadata": { "editable": true }, @@ -3392,7 +3730,7 @@ }, { "cell_type": "markdown", - "id": "4c148d48", + "id": "6dfeb8ca", "metadata": { "editable": true }, @@ -3404,7 +3742,7 @@ }, { "cell_type": "markdown", - "id": "d2549195", + "id": "da47505d", "metadata": { "editable": true }, @@ -3414,7 +3752,7 @@ }, { "cell_type": "markdown", - "id": "4d7f14be", + "id": "87c1a756", "metadata": { "editable": true }, @@ -3426,7 +3764,7 @@ }, { "cell_type": "markdown", - "id": "3d795b82", + "id": "5028dae1", "metadata": { "editable": true }, @@ -3436,7 +3774,7 @@ }, { "cell_type": "markdown", - "id": "033e1b21", + "id": "98da161b", "metadata": { "editable": true }, @@ -3450,7 +3788,7 @@ }, { "cell_type": "markdown", - "id": "5d172bfd", + "id": "69065ef4", "metadata": { "editable": true }, @@ -3467,7 +3805,7 @@ }, { "cell_type": "markdown", - "id": "672de375", + "id": "3455143e", "metadata": { "editable": true }, @@ -3483,7 +3821,7 @@ }, { "cell_type": "markdown", - "id": "d0ed234f", + "id": "e322fdc8", "metadata": { "editable": true }, @@ -3495,7 +3833,7 @@ }, { "cell_type": "markdown", - "id": "7a2bde58", + "id": "f50b583d", "metadata": { "editable": true }, @@ -3508,7 +3846,7 @@ }, { "cell_type": "markdown", - "id": "88344390", + "id": "b12b62dc", "metadata": { "editable": true }, @@ -3522,7 +3860,7 @@ }, { "cell_type": "markdown", - "id": "4019cf50", + "id": "40355c87", "metadata": { "editable": true }, @@ -3532,7 +3870,7 @@ }, { "cell_type": "markdown", - "id": "95d6e396", + "id": "ee70666b", "metadata": { "editable": true }, @@ -3545,7 +3883,7 @@ }, { "cell_type": "markdown", - "id": "d5baa57d", + "id": "dc6d0cc7", "metadata": { "editable": true }, @@ -3564,7 +3902,7 @@ }, { "cell_type": "markdown", - "id": "f310ebda", + "id": "d6b6f792", "metadata": { "editable": true }, @@ -3576,7 +3914,7 @@ }, { "cell_type": "markdown", - "id": "a8366c66", + "id": "39c76bf5", "metadata": { "editable": true }, @@ -3588,7 +3926,7 @@ }, { "cell_type": "markdown", - "id": "9983b7fb", + "id": "f2d754e5", "metadata": { "editable": true }, @@ -3598,7 +3936,7 @@ }, { "cell_type": "markdown", - "id": "196e9dff", + "id": "77a984ab", "metadata": { "editable": true }, @@ -3610,7 +3948,7 @@ }, { "cell_type": "markdown", - "id": "fd64e6a2", + "id": "33060c1a", "metadata": { "editable": true }, @@ -3623,7 +3961,7 @@ }, { "cell_type": "markdown", - "id": "46a1e1da", + "id": "ed1e8279", "metadata": { "editable": true }, @@ -3642,7 +3980,7 @@ }, { "cell_type": "markdown", - "id": "6ea1ae21", + "id": "22b40263", "metadata": { "editable": true }, @@ -3652,7 +3990,7 @@ }, { "cell_type": "markdown", - "id": "fccc043c", + "id": "1b02469c", "metadata": { "editable": true }, @@ -3671,7 +4009,7 @@ }, { "cell_type": "markdown", - "id": "5d65cf6e", + "id": "0660e456", "metadata": { "editable": true }, @@ -3689,7 +4027,7 @@ }, { "cell_type": "markdown", - "id": "f6c00582", + "id": "0a40e5c0", "metadata": { "editable": true }, @@ -3703,7 +4041,7 @@ }, { "cell_type": "markdown", - "id": "fd88cfa1", + "id": "82d7e99d", "metadata": { "editable": true }, @@ -3717,8 +4055,8 @@ }, { "cell_type": "code", - "execution_count": 13, - "id": "c6da7058", + "execution_count": 14, + "id": "6f4b429b", "metadata": { "collapsed": false, "editable": true @@ -3739,7 +4077,7 @@ }, { "cell_type": "markdown", - "id": "bd31ca44", + "id": "a38f8430", "metadata": { "editable": true }, @@ -3755,8 +4093,8 @@ }, { "cell_type": "code", - "execution_count": 14, - "id": "d809cece", + "execution_count": 15, + "id": "87d44fdf", "metadata": { "collapsed": false, "editable": true @@ -3788,7 +4126,7 @@ }, { "cell_type": "markdown", - "id": "8253074c", + "id": "5f08f4dc", "metadata": { "editable": true }, @@ -3802,7 +4140,7 @@ }, { "cell_type": "markdown", - "id": "7a74656a", + "id": "a25b356f", "metadata": { "editable": true }, @@ -3814,8 +4152,8 @@ }, { "cell_type": "code", - "execution_count": 15, - "id": "804a8ee1", + "execution_count": 16, + "id": "928ae649", "metadata": { "collapsed": false, "editable": true @@ -3840,99 +4178,7 @@ }, { "cell_type": "markdown", - "id": "3871673a", - "metadata": { - "editable": true - }, - "source": [ - "We expand this model to the Franke function discussed above." - ] - }, - { - "cell_type": "markdown", - "id": "5e75414e", - "metadata": { - "editable": true - }, - "source": [ - "## Correlation Matrix with Pandas and the Franke function" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "b3efcdb9", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], - "source": [ - "# Common imports\n", - "import numpy as np\n", - "import pandas as pd\n", - "\n", - "\n", - "def FrankeFunction(x,y):\n", - "\tterm1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))\n", - "\tterm2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))\n", - "\tterm3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))\n", - "\tterm4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)\n", - "\treturn term1 + term2 + term3 + term4\n", - "\n", - "\n", - "def create_X(x, y, n ):\n", - "\tif len(x.shape) > 1:\n", - "\t\tx = np.ravel(x)\n", - "\t\ty = np.ravel(y)\n", - "\n", - "\tN = len(x)\n", - "\tl = int((n+1)*(n+2)/2)\t\t# Number of elements in beta\n", - "\tX = np.ones((N,l))\n", - "\n", - "\tfor i in range(1,n+1):\n", - "\t\tq = int((i)*(i+1)/2)\n", - "\t\tfor k in range(i+1):\n", - "\t\t\tX[:,q+k] = (x**(i-k))*(y**k)\n", - "\n", - "\treturn X\n", - "\n", - "\n", - "# Making meshgrid of datapoints and compute Franke's function\n", - "n = 4\n", - "N = 100\n", - "x = np.sort(np.random.uniform(0, 1, N))\n", - "y = np.sort(np.random.uniform(0, 1, N))\n", - "z = FrankeFunction(x, y)\n", - "X = create_X(x, y, n=n) \n", - "\n", - "Xpd = pd.DataFrame(X)\n", - "# subtract the mean values and set up the covariance matrix\n", - "Xpd = Xpd - Xpd.mean()\n", - "covariance_matrix = Xpd.cov()\n", - "print(covariance_matrix)" - ] - }, - { - "cell_type": "markdown", - "id": "03523168", - "metadata": { - "editable": true - }, - "source": [ - "We note here that the covariance is zero for the first rows and\n", - "columns since all matrix elements in the design matrix were set to one\n", - "(we are fitting the function in terms of a polynomial of degree $n$).\n", - "\n", - "This means that the variance for these elements will be zero and will\n", - "cause problems when we set up the correlation matrix. We can simply\n", - "drop these elements and construct a correlation\n", - "matrix without these elements." - ] - }, - { - "cell_type": "markdown", - "id": "ab0fcb46", + "id": "cab70377", "metadata": { "editable": true }, @@ -3944,7 +4190,7 @@ }, { "cell_type": "markdown", - "id": "9e11454d", + "id": "22ac4a6d", "metadata": { "editable": true }, @@ -3956,7 +4202,7 @@ }, { "cell_type": "markdown", - "id": "35eaf4ef", + "id": "cd0697a9", "metadata": { "editable": true }, @@ -3966,7 +4212,7 @@ }, { "cell_type": "markdown", - "id": "a426a48c", + "id": "b5043405", "metadata": { "editable": true }, @@ -3983,7 +4229,7 @@ }, { "cell_type": "markdown", - "id": "f0ac2bfa", + "id": "6a4edab6", "metadata": { "editable": true }, @@ -3993,7 +4239,7 @@ }, { "cell_type": "markdown", - "id": "c6d8c9b4", + "id": "e1b67496", "metadata": { "editable": true }, @@ -4008,7 +4254,7 @@ }, { "cell_type": "markdown", - "id": "a87b1d66", + "id": "47a041cb", "metadata": { "editable": true }, @@ -4018,7 +4264,7 @@ }, { "cell_type": "markdown", - "id": "cb950b85", + "id": "575264b7", "metadata": { "editable": true }, @@ -4032,7 +4278,7 @@ }, { "cell_type": "markdown", - "id": "c79e96c6", + "id": "7b049c58", "metadata": { "editable": true }, @@ -4044,7 +4290,7 @@ }, { "cell_type": "markdown", - "id": "1fceefdc", + "id": "9780789f", "metadata": { "editable": true }, @@ -4056,7 +4302,7 @@ }, { "cell_type": "markdown", - "id": "a8db1029", + "id": "da396c4a", "metadata": { "editable": true }, @@ -4068,7 +4314,7 @@ }, { "cell_type": "markdown", - "id": "03d9d476", + "id": "87a74417", "metadata": { "editable": true }, @@ -4078,7 +4324,7 @@ }, { "cell_type": "markdown", - "id": "9ccc8461", + "id": "c5ca565a", "metadata": { "editable": true }, @@ -4090,7 +4336,7 @@ }, { "cell_type": "markdown", - "id": "c6183bd4", + "id": "10944fc5", "metadata": { "editable": true }, @@ -4100,7 +4346,7 @@ }, { "cell_type": "markdown", - "id": "5cc84134", + "id": "7b985955", "metadata": { "editable": true }, @@ -4117,7 +4363,7 @@ }, { "cell_type": "markdown", - "id": "788d9c68", + "id": "1b748051", "metadata": { "editable": true }, @@ -4127,7 +4373,7 @@ }, { "cell_type": "markdown", - "id": "bd0e5aee", + "id": "40c1c1eb", "metadata": { "editable": true }, @@ -4139,7 +4385,7 @@ }, { "cell_type": "markdown", - "id": "dc52b90c", + "id": "e44c9e87", "metadata": { "editable": true }, @@ -4149,7 +4395,7 @@ }, { "cell_type": "markdown", - "id": "b593db6e", + "id": "28abe46c", "metadata": { "editable": true }, @@ -4161,7 +4407,7 @@ }, { "cell_type": "markdown", - "id": "8dcd6cc8", + "id": "178bc776", "metadata": { "editable": true }, @@ -4175,7 +4421,7 @@ }, { "cell_type": "markdown", - "id": "29312718", + "id": "1f0ead95", "metadata": { "editable": true }, @@ -4187,7 +4433,7 @@ }, { "cell_type": "markdown", - "id": "9818fcb1", + "id": "fcce3bb5", "metadata": { "editable": true }, @@ -4209,7 +4455,7 @@ }, { "cell_type": "markdown", - "id": "56fbca85", + "id": "5f9a7281", "metadata": { "editable": true }, @@ -4221,7 +4467,7 @@ }, { "cell_type": "markdown", - "id": "d9d152fe", + "id": "d3ea695c", "metadata": { "editable": true }, @@ -4236,7 +4482,7 @@ }, { "cell_type": "markdown", - "id": "b82dd67d", + "id": "edfff9e0", "metadata": { "editable": true }, @@ -4248,7 +4494,7 @@ }, { "cell_type": "markdown", - "id": "7dab117e", + "id": "c3c83244", "metadata": { "editable": true }, @@ -4260,7 +4506,7 @@ }, { "cell_type": "markdown", - "id": "3649c008", + "id": "00df5ee5", "metadata": { "editable": true }, @@ -4270,7 +4516,7 @@ }, { "cell_type": "markdown", - "id": "665e42ab", + "id": "4661cbd7", "metadata": { "editable": true }, @@ -4282,7 +4528,7 @@ }, { "cell_type": "markdown", - "id": "254d4fd4", + "id": "07b95265", "metadata": { "editable": true }, @@ -4292,7 +4538,7 @@ }, { "cell_type": "markdown", - "id": "4b84bb18", + "id": "8c9aeba4", "metadata": { "editable": true }, @@ -4304,7 +4550,7 @@ }, { "cell_type": "markdown", - "id": "d84538dd", + "id": "03690a72", "metadata": { "editable": true }, @@ -4314,7 +4560,7 @@ }, { "cell_type": "markdown", - "id": "f9a8f033", + "id": "478e2a9d", "metadata": { "editable": true }, @@ -4326,7 +4572,7 @@ }, { "cell_type": "markdown", - "id": "0c900c57", + "id": "e997196a", "metadata": { "editable": true }, @@ -4343,12 +4589,12 @@ }, { "cell_type": "markdown", - "id": "ef8aafcf", + "id": "770e587e", "metadata": { "editable": true }, "source": [ - "## Ridge and LASSO Regression\n", + "## Back to Ridge and LASSO Regression\n", "\n", "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", "our optimization problem is" @@ -4356,7 +4602,7 @@ }, { "cell_type": "markdown", - "id": "efc29850", + "id": "5cb701c2", "metadata": { "editable": true }, @@ -4368,7 +4614,7 @@ }, { "cell_type": "markdown", - "id": "586af0fa", + "id": "dee9d948", "metadata": { "editable": true }, @@ -4378,7 +4624,7 @@ }, { "cell_type": "markdown", - "id": "116d6e8b", + "id": "121cd014", "metadata": { "editable": true }, @@ -4391,7 +4637,7 @@ }, { "cell_type": "markdown", - "id": "90567753", + "id": "2ee745b4", "metadata": { "editable": true }, @@ -4401,7 +4647,7 @@ }, { "cell_type": "markdown", - "id": "aa3053f1", + "id": "94a15fb9", "metadata": { "editable": true }, @@ -4413,7 +4659,7 @@ }, { "cell_type": "markdown", - "id": "b2382074", + "id": "6341fe06", "metadata": { "editable": true }, @@ -4426,7 +4672,7 @@ }, { "cell_type": "markdown", - "id": "716a3c2a", + "id": "5bef6925", "metadata": { "editable": true }, @@ -4439,7 +4685,7 @@ }, { "cell_type": "markdown", - "id": "dfd4ebf1", + "id": "4539e82b", "metadata": { "editable": true }, @@ -4451,7 +4697,7 @@ }, { "cell_type": "markdown", - "id": "d782ab23", + "id": "1f496d16", "metadata": { "editable": true }, @@ -4463,7 +4709,7 @@ }, { "cell_type": "markdown", - "id": "f16d590d", + "id": "ee78adfa", "metadata": { "editable": true }, @@ -4473,7 +4719,7 @@ }, { "cell_type": "markdown", - "id": "a373fb74", + "id": "47a0c860", "metadata": { "editable": true }, @@ -4486,7 +4732,7 @@ }, { "cell_type": "markdown", - "id": "c4c2a6db", + "id": "6b452b5d", "metadata": { "editable": true }, @@ -4498,7 +4744,7 @@ }, { "cell_type": "markdown", - "id": "f14edde4", + "id": "a3331a79", "metadata": { "editable": true }, @@ -4510,134 +4756,12 @@ }, { "cell_type": "markdown", - "id": "899bc3ae", + "id": "9befd0e3", "metadata": { "editable": true }, "source": [ - "## Deriving the Ridge Regression Equations\n", - "\n", - "Using the matrix-vector expression for Ridge regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have" - ] - }, - { - "cell_type": "markdown", - "id": "0968804d", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\boldsymbol{\\theta}^T\\boldsymbol{\\theta},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "87643992", - "metadata": { - "editable": true - }, - "source": [ - "and \n", - "taking the derivatives with respect to $\\boldsymbol{\\theta}$ we obtain then\n", - "a slightly modified matrix inversion problem which for finite values\n", - "of $\\lambda$ does not suffer from singularity problems. We obtain\n", - "the optimal parameters" - ] - }, - { - "cell_type": "markdown", - "id": "900e9e55", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "6d3a5c46", - "metadata": { - "editable": true - }, - "source": [ - "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" - ] - }, - { - "cell_type": "markdown", - "id": "db3c68ed", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\sum_{i=0}^{p-1} \\theta_i^2 \\leq t,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "ab6d0410", - "metadata": { - "editable": true - }, - "source": [ - "with $t$ a finite positive number. \n", - "\n", - "If we keep the $1/n$ factor, the equation for the optimal $\\theta$ changes to" - ] - }, - { - "cell_type": "markdown", - "id": "ae454f20", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "2a508398", - "metadata": { - "editable": true - }, - "source": [ - "In many textbooks the $1/n$ term is often omitted. Note that a library like **Scikit-Learn** does not include the $1/n$ factor in the setup of the cost function.\n", - "\n", - "When we compare this with the ordinary least squares result we have" - ] - }, - { - "cell_type": "markdown", - "id": "6df6cf96", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "4ab38d70", - "metadata": { - "editable": true - }, - "source": [ - "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", - "\n", - "We see that Ridge regression is nothing but the standard OLS with a\n", + "Ridge regression, as discussed above, is nothing but the standard OLS with a\n", "modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n", "particular for our discussion of the bias-variance tradeoff are rather\n", "interesting. We will see that for specific values of $\\lambda$, we may\n", @@ -4649,7 +4773,7 @@ }, { "cell_type": "markdown", - "id": "cfa1a762", + "id": "9d8f2ba3", "metadata": { "editable": true }, @@ -4661,7 +4785,7 @@ }, { "cell_type": "markdown", - "id": "7394c1c2", + "id": "da712e9d", "metadata": { "editable": true }, @@ -4671,7 +4795,7 @@ }, { "cell_type": "markdown", - "id": "c67e8228", + "id": "d91cd5f7", "metadata": { "editable": true }, @@ -4683,7 +4807,7 @@ }, { "cell_type": "markdown", - "id": "95c698ae", + "id": "18940f72", "metadata": { "editable": true }, @@ -4693,7 +4817,7 @@ }, { "cell_type": "markdown", - "id": "372c4fae", + "id": "51fe3527", "metadata": { "editable": true }, @@ -4705,7 +4829,7 @@ }, { "cell_type": "markdown", - "id": "826e010b", + "id": "2e85d1eb", "metadata": { "editable": true }, @@ -4717,7 +4841,7 @@ }, { "cell_type": "markdown", - "id": "b3c2a178", + "id": "96c47485", "metadata": { "editable": true }, @@ -4733,7 +4857,7 @@ }, { "cell_type": "markdown", - "id": "2ec82017", + "id": "c1adcbbc", "metadata": { "editable": true }, @@ -4745,7 +4869,7 @@ }, { "cell_type": "markdown", - "id": "dacb08f8", + "id": "9693074c", "metadata": { "editable": true }, @@ -4757,7 +4881,7 @@ }, { "cell_type": "markdown", - "id": "af53eaa0", + "id": "46b05531", "metadata": { "editable": true }, @@ -4767,7 +4891,7 @@ }, { "cell_type": "markdown", - "id": "128f95fc", + "id": "2e4ce02b", "metadata": { "editable": true }, @@ -4779,7 +4903,7 @@ }, { "cell_type": "markdown", - "id": "bd5b4b5d", + "id": "dbd3f781", "metadata": { "editable": true }, @@ -4789,7 +4913,7 @@ }, { "cell_type": "markdown", - "id": "d316a1ae", + "id": "779e1ac5", "metadata": { "editable": true }, @@ -4801,7 +4925,7 @@ }, { "cell_type": "markdown", - "id": "1198b684", + "id": "ef53f2a5", "metadata": { "editable": true }, @@ -4818,7 +4942,7 @@ }, { "cell_type": "markdown", - "id": "bb24826b", + "id": "57adfaff", "metadata": { "editable": true }, @@ -4830,7 +4954,7 @@ }, { "cell_type": "markdown", - "id": "b9e921e5", + "id": "a8f94cba", "metadata": { "editable": true }, @@ -4842,7 +4966,7 @@ }, { "cell_type": "markdown", - "id": "06ad064c", + "id": "d60431ff", "metadata": { "editable": true }, @@ -4852,7 +4976,7 @@ }, { "cell_type": "markdown", - "id": "ab63b0d3", + "id": "f78d1dfd", "metadata": { "editable": true }, @@ -4864,7 +4988,7 @@ }, { "cell_type": "markdown", - "id": "49285a6a", + "id": "445800de", "metadata": { "editable": true }, @@ -4874,7 +4998,7 @@ }, { "cell_type": "markdown", - "id": "46be0f62", + "id": "3c54c49c", "metadata": { "editable": true }, @@ -4886,7 +5010,7 @@ }, { "cell_type": "markdown", - "id": "1ad7dd7c", + "id": "11366dc1", "metadata": { "editable": true }, @@ -4896,7 +5020,7 @@ }, { "cell_type": "markdown", - "id": "3d0b7ab5", + "id": "d68d3ac1", "metadata": { "editable": true }, @@ -4908,7 +5032,7 @@ }, { "cell_type": "markdown", - "id": "b360d642", + "id": "63fedbd6", "metadata": { "editable": true }, @@ -4918,7 +5042,7 @@ }, { "cell_type": "markdown", - "id": "8677a588", + "id": "470b20a8", "metadata": { "editable": true }, @@ -4930,12 +5054,12 @@ }, { "cell_type": "markdown", - "id": "d4f0eccb", + "id": "f3277c65", "metadata": { "editable": true }, "source": [ - "This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later." + "This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss how to code LASSO regression next week, when we have introduced gradient methods." ] } ], diff --git a/doc/LectureNotes/week35.ipynb b/doc/LectureNotes/week35.ipynb index 36dd263dd..35f134d71 100644 --- a/doc/LectureNotes/week35.ipynb +++ b/doc/LectureNotes/week35.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "3803481e", + "id": "82c2f725", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "90e0b055", + "id": "d2fc786b", "metadata": { "editable": true }, @@ -27,7 +27,7 @@ }, { "cell_type": "markdown", - "id": "0b94b0fb", + "id": "77512f33", "metadata": { "editable": true }, @@ -38,18 +38,18 @@ "\n", "1. Brief repetition from last week\n", "\n", - "2. Discussions of the equations for ordinary least squares\n", + "2. Discussions of the equations for ordinary least squares (**OLS**)\n", "\n", "3. Discussion on how to prepare data and examples of applications of linear regression\n", "\n", - "4. Material for the lecture on Monday: Mathematical interpretations of linear regression\n", + "4. Mathematical interpretations of OLS\n", "\n", - "5. Monday: Ridge and Lasso regression and Singular Value Decomposition" + "5. Introduction of Ridge and Lasso regression" ] }, { "cell_type": "markdown", - "id": "c8ad9a9b", + "id": "41516757", "metadata": { "editable": true }, @@ -60,7 +60,7 @@ "\n", "\n", "\n", - "2. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)\n", + "2. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra\n", "\n", "3. Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4.\n", "\n", @@ -69,7 +69,7 @@ }, { "cell_type": "markdown", - "id": "f5f74962", + "id": "d3703d2f", "metadata": { "editable": true }, @@ -102,7 +102,7 @@ }, { "cell_type": "markdown", - "id": "d61252cd", + "id": "f22b90af", "metadata": { "editable": true }, @@ -118,7 +118,7 @@ }, { "cell_type": "markdown", - "id": "3dd45959", + "id": "3fe305bb", "metadata": { "editable": true }, @@ -130,7 +130,7 @@ }, { "cell_type": "markdown", - "id": "0ab40f7c", + "id": "f2f38355", "metadata": { "editable": true }, @@ -140,7 +140,7 @@ }, { "cell_type": "markdown", - "id": "d219bba3", + "id": "c3d9a4b6", "metadata": { "editable": true }, @@ -152,7 +152,7 @@ }, { "cell_type": "markdown", - "id": "806e921f", + "id": "ff3ba043", "metadata": { "editable": true }, @@ -173,7 +173,7 @@ }, { "cell_type": "markdown", - "id": "5a10e185", + "id": "5de294d5", "metadata": { "editable": true }, @@ -185,7 +185,7 @@ }, { "cell_type": "markdown", - "id": "87b9af07", + "id": "5721942a", "metadata": { "editable": true }, @@ -198,7 +198,7 @@ }, { "cell_type": "markdown", - "id": "b5e5e170", + "id": "2b2aa843", "metadata": { "editable": true }, @@ -210,7 +210,7 @@ }, { "cell_type": "markdown", - "id": "bacf306d", + "id": "2b12c06f", "metadata": { "editable": true }, @@ -222,7 +222,7 @@ }, { "cell_type": "markdown", - "id": "971edd52", + "id": "276d4eba", "metadata": { "editable": true }, @@ -232,7 +232,7 @@ }, { "cell_type": "markdown", - "id": "1e764eaf", + "id": "4751c9b4", "metadata": { "editable": true }, @@ -244,7 +244,7 @@ }, { "cell_type": "markdown", - "id": "f2f1e2f4", + "id": "dd00bf4b", "metadata": { "editable": true }, @@ -257,7 +257,7 @@ }, { "cell_type": "markdown", - "id": "f00007b0", + "id": "bc9028f0", "metadata": { "editable": true }, @@ -269,7 +269,7 @@ }, { "cell_type": "markdown", - "id": "ad127894", + "id": "d6a27e54", "metadata": { "editable": true }, @@ -279,7 +279,7 @@ }, { "cell_type": "markdown", - "id": "87bea682", + "id": "82f1494a", "metadata": { "editable": true }, @@ -291,7 +291,7 @@ }, { "cell_type": "markdown", - "id": "592cb7cc", + "id": "d6427a35", "metadata": { "editable": true }, @@ -303,7 +303,7 @@ }, { "cell_type": "markdown", - "id": "8ad2c594", + "id": "38c78421", "metadata": { "editable": true }, @@ -314,7 +314,7 @@ }, { "cell_type": "markdown", - "id": "0d53b5e7", + "id": "f23ac12e", "metadata": { "editable": true }, @@ -326,39 +326,39 @@ }, { "cell_type": "markdown", - "id": "a3cbbda7", + "id": "88fd9c03", "metadata": { "editable": true }, "source": [ "where $\\langle y_i \\rangle$ is the mean value. Keep in mind also that\n", "till now we have treated $y_i$ as the exact value. Normally, the\n", - "response (dependent or outcome) variable $y_i$ is the outcome of a\n", + "output (response, target, dependent or outcome) variable $y_i$ is the outcome of a\n", "numerical experiment or another type of experiment and could thus be treated itself as an\n", "approximation to the true value. It is then always accompanied by an\n", "error estimate, often limited to a statistical error estimate given by\n", "the standard deviation discussed earlier. In the discussion here we\n", - "will treat $y_i$ as our exact value for the response variable.\n", + "will treat $y_i$ as our exact value for the output variable.\n", "\n", "In order to find the parameters $\\theta_i$ we will then minimize the spread of $C(\\boldsymbol{\\theta})$, that is we are going to solve the problem" ] }, { "cell_type": "markdown", - "id": "169c08ab", + "id": "a953b1a3", "metadata": { "editable": true }, "source": [ "$$\n", - "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "\\hat{\\boldsymbol{\\theta}}={\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "710a21eb", + "id": "19eb0838", "metadata": { "editable": true }, @@ -368,7 +368,7 @@ }, { "cell_type": "markdown", - "id": "f18b4513", + "id": "a4f910a4", "metadata": { "editable": true }, @@ -380,7 +380,7 @@ }, { "cell_type": "markdown", - "id": "20b56c08", + "id": "5060e089", "metadata": { "editable": true }, @@ -390,7 +390,7 @@ }, { "cell_type": "markdown", - "id": "7f6b7e59", + "id": "8542c5a3", "metadata": { "editable": true }, @@ -402,7 +402,7 @@ }, { "cell_type": "markdown", - "id": "36cdc402", + "id": "475e1436", "metadata": { "editable": true }, @@ -412,7 +412,7 @@ }, { "cell_type": "markdown", - "id": "cb8648c4", + "id": "88abf4ad", "metadata": { "editable": true }, @@ -424,7 +424,7 @@ }, { "cell_type": "markdown", - "id": "9d60b674", + "id": "62f558d9", "metadata": { "editable": true }, @@ -435,7 +435,7 @@ }, { "cell_type": "markdown", - "id": "475a553b", + "id": "99cc5104", "metadata": { "editable": true }, @@ -447,7 +447,7 @@ }, { "cell_type": "markdown", - "id": "69814fd7", + "id": "29043759", "metadata": { "editable": true }, @@ -457,7 +457,7 @@ }, { "cell_type": "markdown", - "id": "75c465dc", + "id": "e9d0f067", "metadata": { "editable": true }, @@ -469,7 +469,7 @@ }, { "cell_type": "markdown", - "id": "fb3f5818", + "id": "c34bfdd2", "metadata": { "editable": true }, @@ -479,19 +479,19 @@ }, { "cell_type": "markdown", - "id": "01c31fce", + "id": "3533f92e", "metadata": { "editable": true }, "source": [ "$$\n", - "\\boldsymbol{\\theta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "\\hat{\\boldsymbol{\\theta}} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "869d8825", + "id": "135631d4", "metadata": { "editable": true }, @@ -504,14 +504,14 @@ "supervised learning algorithms like classification with logistic\n", "regression or support vector machines, exhibit dimensionalities which\n", "allow for the usage of direct linear algebra methods such as **LU** decomposition or **Singular Value Decomposition** (SVD) for finding the inverse of the matrix\n", - "$\\boldsymbol{X}^T\\boldsymbol{X}$. This is discussed on Thursday this week.\n", + "$\\boldsymbol{X}^T\\boldsymbol{X}$.\n", "\n", - "**Small question**: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$? What kind of problems can we expect?" + "**Small question**: When inverting the matrix $\\boldsymbol{X}^T\\boldsymbol{X}, what kind of problems can we expect?" ] }, { "cell_type": "markdown", - "id": "86056e8c", + "id": "cf25f4a7", "metadata": { "editable": true }, @@ -538,7 +538,7 @@ }, { "cell_type": "markdown", - "id": "d36297d2", + "id": "5d82400a", "metadata": { "editable": true }, @@ -550,7 +550,7 @@ }, { "cell_type": "markdown", - "id": "42a61ab6", + "id": "00d8d123", "metadata": { "editable": true }, @@ -562,7 +562,7 @@ }, { "cell_type": "markdown", - "id": "6216fdfa", + "id": "cce17e14", "metadata": { "editable": true }, @@ -578,7 +578,7 @@ }, { "cell_type": "markdown", - "id": "1d8f89cb", + "id": "65efb775", "metadata": { "editable": true }, @@ -596,7 +596,7 @@ }, { "cell_type": "markdown", - "id": "4c3d1ea5", + "id": "2df9efd6", "metadata": { "editable": true }, @@ -608,7 +608,7 @@ }, { "cell_type": "markdown", - "id": "41d96182", + "id": "37b621d8", "metadata": { "editable": true }, @@ -620,7 +620,7 @@ }, { "cell_type": "markdown", - "id": "187e7d47", + "id": "297be9c9", "metadata": { "editable": true }, @@ -631,7 +631,7 @@ }, { "cell_type": "markdown", - "id": "fe831c6b", + "id": "fe288c99", "metadata": { "editable": true }, @@ -643,7 +643,7 @@ }, { "cell_type": "markdown", - "id": "5e9942fa", + "id": "d4333779", "metadata": { "editable": true }, @@ -653,7 +653,7 @@ }, { "cell_type": "markdown", - "id": "a1589dc4", + "id": "f7e72c3d", "metadata": { "editable": true }, @@ -665,7 +665,7 @@ }, { "cell_type": "markdown", - "id": "f4980fc7", + "id": "3423b702", "metadata": { "editable": true }, @@ -679,7 +679,7 @@ }, { "cell_type": "markdown", - "id": "5e5803ca", + "id": "fe66f713", "metadata": { "editable": true }, @@ -691,7 +691,7 @@ }, { "cell_type": "markdown", - "id": "8ff2d8a0", + "id": "43eed51b", "metadata": { "editable": true }, @@ -703,7 +703,7 @@ }, { "cell_type": "markdown", - "id": "a14f7d1e", + "id": "6f943ccb", "metadata": { "editable": true }, @@ -715,7 +715,7 @@ }, { "cell_type": "markdown", - "id": "8d0c272b", + "id": "5df09c55", "metadata": { "editable": true }, @@ -725,7 +725,7 @@ }, { "cell_type": "markdown", - "id": "9bfc54a6", + "id": "b92d9f2a", "metadata": { "editable": true }, @@ -737,7 +737,7 @@ }, { "cell_type": "markdown", - "id": "f68602df", + "id": "c6cf49ab", "metadata": { "editable": true }, @@ -749,7 +749,7 @@ }, { "cell_type": "markdown", - "id": "f3a9b46b", + "id": "3e000448", "metadata": { "editable": true }, @@ -761,7 +761,7 @@ }, { "cell_type": "markdown", - "id": "8b9c9796", + "id": "fbaa984b", "metadata": { "editable": true }, @@ -775,7 +775,7 @@ }, { "cell_type": "markdown", - "id": "e55c0131", + "id": "f4e444ea", "metadata": { "editable": true }, @@ -787,7 +787,7 @@ }, { "cell_type": "markdown", - "id": "0213690c", + "id": "454ba5ba", "metadata": { "editable": true }, @@ -799,7 +799,7 @@ }, { "cell_type": "markdown", - "id": "4bebb2ab", + "id": "c0434cd9", "metadata": { "editable": true }, @@ -811,7 +811,7 @@ }, { "cell_type": "markdown", - "id": "f90da960", + "id": "f2bcda9d", "metadata": { "editable": true }, @@ -821,7 +821,7 @@ }, { "cell_type": "markdown", - "id": "8178ffe4", + "id": "aea0ba73", "metadata": { "editable": true }, @@ -833,7 +833,7 @@ }, { "cell_type": "markdown", - "id": "d937141e", + "id": "5a58d6e8", "metadata": { "editable": true }, @@ -843,7 +843,7 @@ }, { "cell_type": "markdown", - "id": "a5a8213f", + "id": "927428aa", "metadata": { "editable": true }, @@ -855,7 +855,7 @@ }, { "cell_type": "markdown", - "id": "3bdb5bee", + "id": "372ad443", "metadata": { "editable": true }, @@ -865,7 +865,7 @@ }, { "cell_type": "markdown", - "id": "4a9ff5dc", + "id": "37a99489", "metadata": { "editable": true }, @@ -877,7 +877,7 @@ }, { "cell_type": "markdown", - "id": "97098140", + "id": "4872a1cd", "metadata": { "editable": true }, @@ -889,7 +889,7 @@ }, { "cell_type": "markdown", - "id": "76008e78", + "id": "b21e9cb2", "metadata": { "editable": true }, @@ -901,7 +901,7 @@ }, { "cell_type": "markdown", - "id": "7e07b9bf", + "id": "442375d1", "metadata": { "editable": true }, @@ -916,7 +916,7 @@ }, { "cell_type": "markdown", - "id": "a691ed11", + "id": "aabab237", "metadata": { "editable": true }, @@ -928,7 +928,7 @@ }, { "cell_type": "markdown", - "id": "73cfbba0", + "id": "106f09df", "metadata": { "editable": true }, @@ -938,7 +938,7 @@ }, { "cell_type": "markdown", - "id": "d78a9df5", + "id": "73d8f824", "metadata": { "editable": true }, @@ -950,7 +950,7 @@ }, { "cell_type": "markdown", - "id": "9e1e3a87", + "id": "f2626be7", "metadata": { "editable": true }, @@ -960,7 +960,7 @@ }, { "cell_type": "markdown", - "id": "aea5a5e4", + "id": "81d89d86", "metadata": { "editable": true }, @@ -972,7 +972,7 @@ }, { "cell_type": "markdown", - "id": "89e6fde8", + "id": "59c2576a", "metadata": { "editable": true }, @@ -982,7 +982,7 @@ }, { "cell_type": "markdown", - "id": "6c0f4449", + "id": "4b7bf3f9", "metadata": { "editable": true }, @@ -994,7 +994,7 @@ }, { "cell_type": "markdown", - "id": "a390ac6e", + "id": "ea1e8943", "metadata": { "editable": true }, @@ -1006,7 +1006,7 @@ }, { "cell_type": "markdown", - "id": "59d6ab1c", + "id": "d0425d72", "metadata": { "editable": true }, @@ -1018,7 +1018,7 @@ }, { "cell_type": "markdown", - "id": "8c99ee21", + "id": "38386834", "metadata": { "editable": true }, @@ -1028,7 +1028,7 @@ }, { "cell_type": "markdown", - "id": "7ea454e1", + "id": "bfbc4bd3", "metadata": { "editable": true }, @@ -1040,7 +1040,7 @@ }, { "cell_type": "markdown", - "id": "5a555af4", + "id": "3ced3cec", "metadata": { "editable": true }, @@ -1053,7 +1053,7 @@ }, { "cell_type": "markdown", - "id": "a39c02c1", + "id": "fc6f1dec", "metadata": { "editable": true }, @@ -1065,7 +1065,7 @@ }, { "cell_type": "markdown", - "id": "a9804433", + "id": "699662d2", "metadata": { "editable": true }, @@ -1075,7 +1075,7 @@ }, { "cell_type": "markdown", - "id": "8be8de93", + "id": "0991592c", "metadata": { "editable": true }, @@ -1087,7 +1087,7 @@ }, { "cell_type": "markdown", - "id": "4139dd13", + "id": "450dd961", "metadata": { "editable": true }, @@ -1097,7 +1097,7 @@ }, { "cell_type": "markdown", - "id": "6f1c778a", + "id": "a13de0fa", "metadata": { "editable": true }, @@ -1109,7 +1109,7 @@ }, { "cell_type": "markdown", - "id": "03ae847f", + "id": "c64bc916", "metadata": { "editable": true }, @@ -1119,7 +1119,7 @@ }, { "cell_type": "markdown", - "id": "054f8ba7", + "id": "ab1e18bc", "metadata": { "editable": true }, @@ -1131,7 +1131,7 @@ }, { "cell_type": "markdown", - "id": "61c62efa", + "id": "f1e74f34", "metadata": { "editable": true }, @@ -1141,7 +1141,7 @@ }, { "cell_type": "markdown", - "id": "ac84f4fd", + "id": "1ad3ebd9", "metadata": { "editable": true }, @@ -1153,7 +1153,7 @@ }, { "cell_type": "markdown", - "id": "b2f0a3d0", + "id": "37d1b4ca", "metadata": { "editable": true }, @@ -1163,7 +1163,7 @@ }, { "cell_type": "markdown", - "id": "9c6460f2", + "id": "3d5fc67a", "metadata": { "editable": true }, @@ -1175,67 +1175,7 @@ }, { "cell_type": "markdown", - "id": "0c4aebc4", - "metadata": { - "editable": true - }, - "source": [ - "## Other useful relations\n", - "\n", - "We list here some other useful relations we may encounter (recall that vectors are defined by boldfaced low-key letters)" - ] - }, - { - "cell_type": "markdown", - "id": "67aef36e", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{\\partial (\\boldsymbol{x}^T\\boldsymbol{a})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "30c88be7", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "fb84fcda", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{\\partial tr(\\boldsymbol{B}\\boldsymbol{A})}{\\partial \\boldsymbol{A}} = \\boldsymbol{B}^T,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "ebbeeb3b", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\frac{\\partial \\log{\\vert\\boldsymbol{A}\\vert}}{\\partial \\boldsymbol{A}} = (\\boldsymbol{A}^{-1})^T.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "fb88afbf", + "id": "d98472d9", "metadata": { "editable": true }, @@ -1251,7 +1191,7 @@ }, { "cell_type": "markdown", - "id": "89512329", + "id": "5d859f20", "metadata": { "editable": true }, @@ -1263,7 +1203,7 @@ }, { "cell_type": "markdown", - "id": "a651d788", + "id": "e7bb91ec", "metadata": { "editable": true }, @@ -1273,7 +1213,7 @@ }, { "cell_type": "markdown", - "id": "dca9d68f", + "id": "1553864b", "metadata": { "editable": true }, @@ -1285,25 +1225,25 @@ }, { "cell_type": "markdown", - "id": "2865ced8", + "id": "f8cc9c71", "metadata": { "editable": true }, "source": [ "For ordinary least squares, it is inversely proportional (derivation\n", "next week) with the variance of the optimal parameters\n", - "$\\hat{\\boldsymbol{\\theta}}$. Furthermore, we will see later this week that it is\n", + "$\\hat{\\boldsymbol{\\theta}}$. Furthermore, we will see next week that it is\n", "(aside the factor $1/n$) equal to the covariance matrix. It plays also a very\n", "important role in optmization algorithms and Principal Component\n", "Analysis as a way to reduce the dimensionality of a machine learning/data analysis\n", - "problem.\n", - "v\n", + "problem. We will discuss this in greater detail next week when we introduce gradient methods.\n", + "\n", "**Linear algebra question:** Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!)." ] }, { "cell_type": "markdown", - "id": "1a7a4045", + "id": "f1f01477", "metadata": { "editable": true }, @@ -1315,7 +1255,7 @@ }, { "cell_type": "markdown", - "id": "a483362e", + "id": "f2dea7e1", "metadata": { "editable": true }, @@ -1327,7 +1267,7 @@ }, { "cell_type": "markdown", - "id": "dcf896a3", + "id": "76b0f99e", "metadata": { "editable": true }, @@ -1337,7 +1277,7 @@ }, { "cell_type": "markdown", - "id": "13c80a7c", + "id": "cfa3ca25", "metadata": { "editable": true }, @@ -1349,7 +1289,7 @@ }, { "cell_type": "markdown", - "id": "582225c6", + "id": "9a9163d8", "metadata": { "editable": true }, @@ -1359,7 +1299,7 @@ }, { "cell_type": "markdown", - "id": "90476b18", + "id": "c6b54a1e", "metadata": { "editable": true }, @@ -1371,7 +1311,7 @@ }, { "cell_type": "markdown", - "id": "267ae1ff", + "id": "659c98d0", "metadata": { "editable": true }, @@ -1381,7 +1321,7 @@ }, { "cell_type": "markdown", - "id": "8c9cb8c5", + "id": "209b5b37", "metadata": { "editable": true }, @@ -1395,7 +1335,7 @@ }, { "cell_type": "markdown", - "id": "acbe4114", + "id": "b325d0c3", "metadata": { "editable": true }, @@ -1407,7 +1347,7 @@ }, { "cell_type": "markdown", - "id": "ba945f21", + "id": "c49a2fcd", "metadata": { "editable": true }, @@ -1418,7 +1358,7 @@ }, { "cell_type": "markdown", - "id": "dc442b90", + "id": "6978ef70", "metadata": { "editable": true }, @@ -1431,14 +1371,14 @@ { "cell_type": "code", "execution_count": 1, - "id": "883026f0", + "id": "01530c39", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ - "# matrix inversion to find beta\n", + "# matrix inversion to find theta\n", "# First we set up the data\n", "import numpy as np\n", "x = np.random.rand(100)\n", @@ -1451,14 +1391,14 @@ "X[:,2] = x**2\n", "X[:,3] = x**3\n", "X[:,4] = x**4\n", - "beta = (np.linalg.inv(X.T @ X) @ X.T ) @ y\n", + "theta = (np.linalg.inv(X.T @ X) @ X.T ) @ y\n", "# and then make the prediction\n", - "ytilde = X @ beta" + "ytilde = X @ theta" ] }, { "cell_type": "markdown", - "id": "826466d3", + "id": "90ff28d4", "metadata": { "editable": true }, @@ -1469,7 +1409,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "decea3b4", + "id": "1d052cbe", "metadata": { "collapsed": false, "editable": true @@ -1482,7 +1422,7 @@ }, { "cell_type": "markdown", - "id": "b52ed870", + "id": "3362efd4", "metadata": { "editable": true }, @@ -1496,7 +1436,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "ee4d5722", + "id": "3e9e681f", "metadata": { "collapsed": false, "editable": true @@ -1509,7 +1449,7 @@ }, { "cell_type": "markdown", - "id": "01892aaa", + "id": "78e4427f", "metadata": { "editable": true }, @@ -1520,7 +1460,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "d6ce54e2", + "id": "20aab659", "metadata": { "collapsed": false, "editable": true @@ -1532,7 +1472,7 @@ }, { "cell_type": "markdown", - "id": "91ed5956", + "id": "326f641d", "metadata": { "editable": true }, @@ -1543,7 +1483,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "7ffd63cb", + "id": "c8e80cf1", "metadata": { "collapsed": false, "editable": true @@ -1559,7 +1499,7 @@ }, { "cell_type": "markdown", - "id": "737370ed", + "id": "aba7b289", "metadata": { "editable": true }, @@ -1570,7 +1510,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "f1bbe22d", + "id": "7235bf1b", "metadata": { "collapsed": false, "editable": true @@ -1584,7 +1524,7 @@ }, { "cell_type": "markdown", - "id": "8e3a976b", + "id": "3eaf5f8d", "metadata": { "editable": true }, @@ -1605,7 +1545,7 @@ }, { "cell_type": "markdown", - "id": "8c2dee8c", + "id": "6fc5dc26", "metadata": { "editable": true }, @@ -1616,7 +1556,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "89d0750d", + "id": "98ca5627", "metadata": { "collapsed": false, "editable": true @@ -1651,16 +1591,16 @@ "X[:,4] = x**4\n", "# We split the data in test and training data\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n", - "# matrix inversion to find beta\n", - "beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", - "print(beta)\n", + "# matrix inversion to find theta\n", + "theta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train\n", + "print(theta)\n", "# and then make the prediction\n", - "ytilde = X_train @ beta\n", + "ytilde = X_train @ theta\n", "print(\"Training R2\")\n", "print(R2(y_train,ytilde))\n", "print(\"Training MSE\")\n", "print(MSE(y_train,ytilde))\n", - "ypredict = X_test @ beta\n", + "ypredict = X_test @ theta\n", "print(\"Test R2\")\n", "print(R2(y_test,ypredict))\n", "print(\"Test MSE\")\n", @@ -1669,7 +1609,7 @@ }, { "cell_type": "markdown", - "id": "12c44026", + "id": "09f0c1c6", "metadata": { "editable": true }, @@ -1680,7 +1620,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "6daa8ad9", + "id": "02220d97", "metadata": { "collapsed": false, "editable": true @@ -1705,7 +1645,7 @@ }, { "cell_type": "markdown", - "id": "9b4cbdbc", + "id": "5494af4d", "metadata": { "editable": true }, @@ -1717,7 +1657,7 @@ }, { "cell_type": "markdown", - "id": "7f685bde", + "id": "048560ee", "metadata": { "editable": true }, @@ -1746,7 +1686,7 @@ }, { "cell_type": "markdown", - "id": "a8e92a0b", + "id": "85b56c79", "metadata": { "editable": true }, @@ -1771,7 +1711,7 @@ }, { "cell_type": "markdown", - "id": "b0ac1e21", + "id": "60937483", "metadata": { "editable": true }, @@ -1791,7 +1731,7 @@ }, { "cell_type": "markdown", - "id": "3c8d9613", + "id": "5d0ea6a3", "metadata": { "editable": true }, @@ -1818,7 +1758,7 @@ }, { "cell_type": "markdown", - "id": "1d817e20", + "id": "8f6fc4e6", "metadata": { "editable": true }, @@ -1831,7 +1771,7 @@ }, { "cell_type": "markdown", - "id": "20673f07", + "id": "4bd392ed", "metadata": { "editable": true }, @@ -1843,7 +1783,7 @@ }, { "cell_type": "markdown", - "id": "4a36afc7", + "id": "4e33d3e3", "metadata": { "editable": true }, @@ -1854,7 +1794,7 @@ }, { "cell_type": "markdown", - "id": "4db40306", + "id": "7e1c73ac", "metadata": { "editable": true }, @@ -1870,7 +1810,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "ac782b4e", + "id": "2c315558", "metadata": { "collapsed": false, "editable": true @@ -1904,7 +1844,7 @@ }, { "cell_type": "markdown", - "id": "d60792bf", + "id": "aaa58775", "metadata": { "editable": true }, @@ -1914,7 +1854,7 @@ }, { "cell_type": "markdown", - "id": "6dbeeee3", + "id": "02b198ed", "metadata": { "editable": true }, @@ -1929,7 +1869,7 @@ }, { "cell_type": "markdown", - "id": "f1bd9e06", + "id": "4cef197d", "metadata": { "editable": true }, @@ -1941,7 +1881,7 @@ }, { "cell_type": "markdown", - "id": "c4e9c9cc", + "id": "23b34528", "metadata": { "editable": true }, @@ -1951,7 +1891,7 @@ }, { "cell_type": "markdown", - "id": "c682dfb2", + "id": "b2847956", "metadata": { "editable": true }, @@ -1967,7 +1907,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "f95e4477", + "id": "b8dacc33", "metadata": { "collapsed": false, "editable": true @@ -1984,7 +1924,7 @@ }, { "cell_type": "markdown", - "id": "3eaa6d2e", + "id": "cd89705f", "metadata": { "editable": true }, @@ -1997,7 +1937,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "fe340fe4", + "id": "d35541fd", "metadata": { "collapsed": false, "editable": true @@ -2044,7 +1984,7 @@ }, { "cell_type": "markdown", - "id": "e2f14a01", + "id": "36014ec9", "metadata": { "editable": true }, @@ -2058,7 +1998,7 @@ }, { "cell_type": "markdown", - "id": "ad741d5a", + "id": "16fb5f8f", "metadata": { "editable": true }, @@ -2070,7 +2010,7 @@ }, { "cell_type": "markdown", - "id": "05b080a8", + "id": "f1ec698e", "metadata": { "editable": true }, @@ -2082,7 +2022,7 @@ }, { "cell_type": "markdown", - "id": "5c64cd00", + "id": "787f58f5", "metadata": { "editable": true }, @@ -2094,7 +2034,7 @@ }, { "cell_type": "markdown", - "id": "66d8648a", + "id": "e23435fd", "metadata": { "editable": true }, @@ -2104,7 +2044,7 @@ }, { "cell_type": "markdown", - "id": "47178854", + "id": "ccaad283", "metadata": { "editable": true }, @@ -2116,7 +2056,7 @@ }, { "cell_type": "markdown", - "id": "9d89dcc7", + "id": "be8092da", "metadata": { "editable": true }, @@ -2126,7 +2066,7 @@ }, { "cell_type": "markdown", - "id": "d037dda7", + "id": "6e055e08", "metadata": { "editable": true }, @@ -2138,7 +2078,7 @@ }, { "cell_type": "markdown", - "id": "4d559b4d", + "id": "ff86c9bc", "metadata": { "editable": true }, @@ -2149,7 +2089,7 @@ }, { "cell_type": "markdown", - "id": "f8eec117", + "id": "b4f2e747", "metadata": { "editable": true }, @@ -2161,7 +2101,7 @@ }, { "cell_type": "markdown", - "id": "ecc2fed3", + "id": "5a9a9f87", "metadata": { "editable": true }, @@ -2173,7 +2113,7 @@ }, { "cell_type": "markdown", - "id": "8d3ae448", + "id": "1232e11a", "metadata": { "editable": true }, @@ -2183,7 +2123,7 @@ }, { "cell_type": "markdown", - "id": "0d7267fa", + "id": "a807825b", "metadata": { "editable": true }, @@ -2195,7 +2135,7 @@ }, { "cell_type": "markdown", - "id": "06e5716f", + "id": "4fa9855c", "metadata": { "editable": true }, @@ -2207,7 +2147,7 @@ }, { "cell_type": "markdown", - "id": "f8f001a6", + "id": "dcffefcf", "metadata": { "editable": true }, @@ -2217,7 +2157,7 @@ }, { "cell_type": "markdown", - "id": "761f6c51", + "id": "3c08ff3a", "metadata": { "editable": true }, @@ -2229,7 +2169,7 @@ }, { "cell_type": "markdown", - "id": "468bc321", + "id": "c2483a13", "metadata": { "editable": true }, @@ -2239,7 +2179,7 @@ }, { "cell_type": "markdown", - "id": "eb61e7ef", + "id": "9a82a8c5", "metadata": { "editable": true }, @@ -2251,7 +2191,7 @@ }, { "cell_type": "markdown", - "id": "be3586ce", + "id": "743f7309", "metadata": { "editable": true }, @@ -2261,7 +2201,7 @@ }, { "cell_type": "markdown", - "id": "f79b7c51", + "id": "e6d7b31a", "metadata": { "editable": true }, @@ -2278,7 +2218,7 @@ "As we will also see in the first project, \n", "this may\n", "however not the be case in general and a standard matrix inversion\n", - "algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.\n", + "algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below and in other examples.\n", "\n", "There is however a way to circumvent this problem and also\n", "gain some insights about the ordinary least squares approach, and\n", @@ -2301,7 +2241,7 @@ }, { "cell_type": "markdown", - "id": "ea84dc35", + "id": "518d6bb4", "metadata": { "editable": true }, @@ -2318,7 +2258,7 @@ }, { "cell_type": "markdown", - "id": "0ad4b3f9", + "id": "056673c8", "metadata": { "editable": true }, @@ -2341,7 +2281,7 @@ }, { "cell_type": "markdown", - "id": "671100e8", + "id": "63daba59", "metadata": { "editable": true }, @@ -2358,7 +2298,7 @@ }, { "cell_type": "markdown", - "id": "9d07ce4a", + "id": "89b74667", "metadata": { "editable": true }, @@ -2377,7 +2317,7 @@ }, { "cell_type": "markdown", - "id": "a26b13de", + "id": "588dd662", "metadata": { "editable": true }, @@ -2388,7 +2328,7 @@ }, { "cell_type": "markdown", - "id": "2352aaa2", + "id": "eb8348c6", "metadata": { "editable": true }, @@ -2400,7 +2340,7 @@ }, { "cell_type": "markdown", - "id": "5c201fe7", + "id": "62027bc1", "metadata": { "editable": true }, @@ -2418,7 +2358,7 @@ }, { "cell_type": "markdown", - "id": "a711f9aa", + "id": "0c73c4b7", "metadata": { "editable": true }, @@ -2434,7 +2374,7 @@ }, { "cell_type": "markdown", - "id": "816ee812", + "id": "a3a2d330", "metadata": { "editable": true }, @@ -2446,7 +2386,7 @@ }, { "cell_type": "markdown", - "id": "ca4f2d5d", + "id": "374b88cc", "metadata": { "editable": true }, @@ -2456,14 +2396,412 @@ }, { "cell_type": "markdown", - "id": "11b18875", + "id": "d5aa42e6", + "metadata": { + "editable": true + }, + "source": [ + "## Ridge and LASSO Regression\n", + "\n", + "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", + "our optimization problem is" + ] + }, + { + "cell_type": "markdown", + "id": "1c01a816", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f556eddc", + "metadata": { + "editable": true + }, + "source": [ + "or we can state it as" + ] + }, + { + "cell_type": "markdown", + "id": "dcf14a16", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "08c37095", + "metadata": { + "editable": true + }, + "source": [ + "where we have used the definition of a norm-2 vector, that is" + ] + }, + { + "cell_type": "markdown", + "id": "6815fa1b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "daf50946", + "metadata": { + "editable": true + }, + "source": [ + "By minimizing the above equation with respect to the parameters\n", + "$\\boldsymbol{\\theta}$ we could then obtain an analytical expression for the\n", + "parameters $\\boldsymbol{\\theta}$. We can add a regularization parameter $\\lambda$ by\n", + "defining a new cost function to be optimized, that is" + ] + }, + { + "cell_type": "markdown", + "id": "a7a60cb3", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "38105bfd", + "metadata": { + "editable": true + }, + "source": [ + "which leads to the Ridge regression minimization problem where we\n", + "require that $\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\\le t$, where $t$ is\n", + "a finite number larger than zero. By defining" + ] + }, + { + "cell_type": "markdown", + "id": "3d697464", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c298d7ba", + "metadata": { + "editable": true + }, + "source": [ + "we have a new optimization equation" + ] + }, + { + "cell_type": "markdown", + "id": "45265b59", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ab6ce408", + "metadata": { + "editable": true + }, + "source": [ + "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", + "\n", + "Here we have defined the norm-1 as" + ] + }, + { + "cell_type": "markdown", + "id": "ca0350a7", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e8146e6f", + "metadata": { + "editable": true + }, + "source": [ + "## Deriving the Ridge Regression Equations\n", + "\n", + "Using the matrix-vector expression for Ridge regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have" + ] + }, + { + "cell_type": "markdown", + "id": "05d97a86", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\boldsymbol{\\theta}^T\\boldsymbol{\\theta},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f84e7118", + "metadata": { + "editable": true + }, + "source": [ + "and \n", + "taking the derivatives with respect to $\\boldsymbol{\\theta}$ we obtain then\n", + "a slightly modified matrix inversion problem which for finite values\n", + "of $\\lambda$ does not suffer from singularity problems. We obtain\n", + "the optimal parameters" + ] + }, + { + "cell_type": "markdown", + "id": "28be9ec4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b9f5a5d8", + "metadata": { + "editable": true + }, + "source": [ + "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" + ] + }, + { + "cell_type": "markdown", + "id": "86b43b21", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\sum_{i=0}^{p-1} \\theta_i^2 \\leq t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "58744172", + "metadata": { + "editable": true + }, + "source": [ + "with $t$ a finite positive number. \n", + "\n", + "If we keep the $1/n$ factor, the equation for the optimal $\\theta$ changes to" + ] + }, + { + "cell_type": "markdown", + "id": "dd515e6f", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e8c98bfe", + "metadata": { + "editable": true + }, + "source": [ + "In many textbooks the $1/n$ term is often omitted. Note that a library like **Scikit-Learn** does not include the $1/n$ factor in the setup of the cost function.\n", + "\n", + "When we compare this with the ordinary least squares result we have" + ] + }, + { + "cell_type": "markdown", + "id": "08cc9626", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "725df93e", + "metadata": { + "editable": true + }, + "source": [ + "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", + "\n", + "We see that Ridge regression is nothing but the standard OLS with a\n", + "modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n", + "particular for our discussion of the bias-variance tradeoff are rather\n", + "interesting. We will see that for specific values of $\\lambda$, we may\n", + "even reduce the variance of the optimal parameters $\\boldsymbol{\\theta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n", + "\n", + "When we have discussed the singular value decomposition of the design\n", + "matrix $\\boldsymbol{X}$, we will in turn perform a more rigorous mathematical\n", + "discussion of Ridge regression.\n", + "\n", + "The code here is a simple demonstration of how to implement Ridge regression with our own code and compare this with scikit-learn." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "8d0397cd", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd\n", + "import matplotlib.pyplot as plt\n", + "from sklearn.model_selection import train_test_split\n", + "from sklearn import linear_model\n", + "\n", + "def MSE(y_data,y_model):\n", + " n = np.size(y_model)\n", + " return np.sum((y_data-y_model)**2)/n\n", + "\n", + "\n", + "# A seed just to ensure that the random numbers are the same for every run.\n", + "# Useful for eventual debugging.\n", + "np.random.seed(3155)\n", + "\n", + "n = 100\n", + "x = np.random.rand(n)\n", + "y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)\n", + "\n", + "Maxpolydegree = 20\n", + "X = np.zeros((n,Maxpolydegree))\n", + "#We include explicitely the intercept column\n", + "for degree in range(Maxpolydegree):\n", + " X[:,degree] = x**degree\n", + "# We split the data in test and training data\n", + "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n", + "\n", + "p = Maxpolydegree\n", + "I = np.eye(p,p)\n", + "# Decide which values of lambda to use\n", + "nlambdas = 6\n", + "MSEOwnRidgePredict = np.zeros(nlambdas)\n", + "MSERidgePredict = np.zeros(nlambdas)\n", + "lambdas = np.logspace(-4, 2, nlambdas)\n", + "for i in range(nlambdas):\n", + " lmb = lambdas[i]\n", + " OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n", + " # Note: we include the intercept column and no scaling\n", + " RegRidge = linear_model.Ridge(lmb,fit_intercept=False)\n", + " RegRidge.fit(X_train,y_train)\n", + " # and then make the prediction\n", + " ytildeOwnRidge = X_train @ OwnRidgeTheta\n", + " ypredictOwnRidge = X_test @ OwnRidgeTheta\n", + " ytildeRidge = RegRidge.predict(X_train)\n", + " ypredictRidge = RegRidge.predict(X_test)\n", + " MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)\n", + " MSERidgePredict[i] = MSE(y_test,ypredictRidge)\n", + " print(\"Theta values for own Ridge implementation\")\n", + " print(OwnRidgeTheta)\n", + " print(\"Theta values for Scikit-Learn Ridge implementation\")\n", + " print(RegRidge.coef_)\n", + " print(\"MSE values for own Ridge implementation\")\n", + " print(MSEOwnRidgePredict[i])\n", + " print(\"MSE values for Scikit-Learn Ridge implementation\")\n", + " print(MSERidgePredict[i])\n", + "\n", + "# Now plot the results\n", + "plt.figure()\n", + "plt.plot(np.log10(lambdas), MSEOwnRidgePredict, 'r', label = 'MSE own Ridge Test')\n", + "plt.plot(np.log10(lambdas), MSERidgePredict, 'g', label = 'MSE Ridge Test')\n", + "\n", + "plt.xlabel('log10(lambda)')\n", + "plt.ylabel('MSE')\n", + "plt.legend()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "dbc56c2b", + "metadata": { + "editable": true + }, + "source": [ + "The results here agree when we force **Scikit-Learn**'s Ridge function to include the first column in our design matrix.\n", + "We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.\n", + "What happens if we do not include the intercept in our fit? We will discuss this in more detail next week." + ] + }, + { + "cell_type": "markdown", + "id": "cfab477f", "metadata": { "editable": true }, "source": [ "## Basic math of the SVD\n", "\n", - "From standard linear algebra we know that a square matrix $\\boldsymbol{X}$ can be diagonalized if and only it is \n", + "From standard linear algebra we know that a square matrix $\\boldsymbol{X}$ can be diagonalized if and only if it is \n", "a so-called [normal matrix](https://en.wikipedia.org/wiki/Normal_matrix), that is if $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times n}$\n", "we have $\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{X}^T\\boldsymbol{X}$ or if $\\boldsymbol{X}\\in {\\mathbb{C}}^{n\\times n}$ we have $\\boldsymbol{X}\\boldsymbol{X}^{\\dagger}=\\boldsymbol{X}^{\\dagger}\\boldsymbol{X}$.\n", "The matrix has then a set of eigenpairs" @@ -2471,7 +2809,7 @@ }, { "cell_type": "markdown", - "id": "a6cebe96", + "id": "bc176ed4", "metadata": { "editable": true }, @@ -2483,7 +2821,7 @@ }, { "cell_type": "markdown", - "id": "4056c51d", + "id": "a9eef8e3", "metadata": { "editable": true }, @@ -2493,7 +2831,7 @@ }, { "cell_type": "markdown", - "id": "8c6fc251", + "id": "c33552ec", "metadata": { "editable": true }, @@ -2505,7 +2843,7 @@ }, { "cell_type": "markdown", - "id": "85d151ae", + "id": "9465d4f6", "metadata": { "editable": true }, @@ -2515,7 +2853,7 @@ }, { "cell_type": "markdown", - "id": "a7ed48ac", + "id": "3cd89a35", "metadata": { "editable": true }, @@ -2527,7 +2865,7 @@ }, { "cell_type": "markdown", - "id": "d61100de", + "id": "13a9f453", "metadata": { "editable": true }, @@ -2539,7 +2877,7 @@ }, { "cell_type": "markdown", - "id": "ae931201", + "id": "6c909f72", "metadata": { "editable": true }, @@ -2554,7 +2892,7 @@ }, { "cell_type": "markdown", - "id": "a1a55970", + "id": "b32ee709", "metadata": { "editable": true }, @@ -2565,7 +2903,7 @@ }, { "cell_type": "markdown", - "id": "7aff14ea", + "id": "e5d3f72c", "metadata": { "editable": true }, @@ -2585,7 +2923,7 @@ }, { "cell_type": "markdown", - "id": "c5bcd441", + "id": "18f488f0", "metadata": { "editable": true }, @@ -2597,7 +2935,7 @@ }, { "cell_type": "markdown", - "id": "ed33684a", + "id": "93618783", "metadata": { "editable": true }, @@ -2607,7 +2945,7 @@ }, { "cell_type": "markdown", - "id": "1d6ca687", + "id": "02f57528", "metadata": { "editable": true }, @@ -2619,7 +2957,7 @@ }, { "cell_type": "markdown", - "id": "c0535cd8", + "id": "762833f4", "metadata": { "editable": true }, @@ -2648,7 +2986,7 @@ }, { "cell_type": "markdown", - "id": "c6c08bf2", + "id": "9ab8ec4f", "metadata": { "editable": true }, @@ -2675,7 +3013,7 @@ }, { "cell_type": "markdown", - "id": "487a7ae7", + "id": "3e526d1f", "metadata": { "editable": true }, @@ -2685,8 +3023,8 @@ }, { "cell_type": "code", - "execution_count": 12, - "id": "3ff87df9", + "execution_count": 13, + "id": "f5d258ce", "metadata": { "collapsed": false, "editable": true @@ -2726,7 +3064,7 @@ }, { "cell_type": "markdown", - "id": "600f74f5", + "id": "4e4bf198", "metadata": { "editable": true }, @@ -2743,7 +3081,7 @@ }, { "cell_type": "markdown", - "id": "da923c99", + "id": "33a42ac4", "metadata": { "editable": true }, @@ -2766,7 +3104,7 @@ }, { "cell_type": "markdown", - "id": "989e9de3", + "id": "7b5d1bbb", "metadata": { "editable": true }, @@ -2780,7 +3118,7 @@ }, { "cell_type": "markdown", - "id": "b010904b", + "id": "1fee5f54", "metadata": { "editable": true }, @@ -2799,7 +3137,7 @@ }, { "cell_type": "markdown", - "id": "1385bdc7", + "id": "235f2975", "metadata": { "editable": true }, @@ -2809,7 +3147,7 @@ }, { "cell_type": "markdown", - "id": "1824f1de", + "id": "fd7415b7", "metadata": { "editable": true }, @@ -2821,7 +3159,7 @@ }, { "cell_type": "markdown", - "id": "50fb0869", + "id": "7c265b5d", "metadata": { "editable": true }, @@ -2835,7 +3173,7 @@ }, { "cell_type": "markdown", - "id": "242e0186", + "id": "976c31a2", "metadata": { "editable": true }, @@ -2847,7 +3185,7 @@ }, { "cell_type": "markdown", - "id": "b3d8ff35", + "id": "8dc8acd7", "metadata": { "editable": true }, @@ -2857,7 +3195,7 @@ }, { "cell_type": "markdown", - "id": "a8799cf8", + "id": "a1dc8e8e", "metadata": { "editable": true }, @@ -2869,7 +3207,7 @@ }, { "cell_type": "markdown", - "id": "69ecbf25", + "id": "eb688af0", "metadata": { "editable": true }, @@ -2886,7 +3224,7 @@ }, { "cell_type": "markdown", - "id": "accaad01", + "id": "eeadd85e", "metadata": { "editable": true }, @@ -2896,7 +3234,7 @@ }, { "cell_type": "markdown", - "id": "7c6d6458", + "id": "e9375ad7", "metadata": { "editable": true }, @@ -2912,7 +3250,7 @@ }, { "cell_type": "markdown", - "id": "92943a13", + "id": "025f1ca1", "metadata": { "editable": true }, @@ -2922,7 +3260,7 @@ }, { "cell_type": "markdown", - "id": "27a56b68", + "id": "d44951a1", "metadata": { "editable": true }, @@ -2938,7 +3276,7 @@ }, { "cell_type": "markdown", - "id": "7fab099a", + "id": "ecc9216b", "metadata": { "editable": true }, @@ -2948,7 +3286,7 @@ }, { "cell_type": "markdown", - "id": "86ef772d", + "id": "28e2ff44", "metadata": { "editable": true }, @@ -2964,7 +3302,7 @@ }, { "cell_type": "markdown", - "id": "a9fb58a3", + "id": "935324af", "metadata": { "editable": true }, @@ -2974,7 +3312,7 @@ }, { "cell_type": "markdown", - "id": "218d36d5", + "id": "a15e7526", "metadata": { "editable": true }, @@ -2991,7 +3329,7 @@ }, { "cell_type": "markdown", - "id": "3c358960", + "id": "5d3cef55", "metadata": { "editable": true }, @@ -3003,7 +3341,7 @@ }, { "cell_type": "markdown", - "id": "384a8da4", + "id": "0fab1783", "metadata": { "editable": true }, @@ -3015,7 +3353,7 @@ }, { "cell_type": "markdown", - "id": "506802df", + "id": "f9a8450e", "metadata": { "editable": true }, @@ -3027,7 +3365,7 @@ }, { "cell_type": "markdown", - "id": "d160546b", + "id": "485d1023", "metadata": { "editable": true }, @@ -3037,7 +3375,7 @@ }, { "cell_type": "markdown", - "id": "22fa787b", + "id": "a2da5dbe", "metadata": { "editable": true }, @@ -3049,7 +3387,7 @@ }, { "cell_type": "markdown", - "id": "4f9cc0fa", + "id": "13959099", "metadata": { "editable": true }, @@ -3061,7 +3399,7 @@ }, { "cell_type": "markdown", - "id": "5228fab9", + "id": "23f0b903", "metadata": { "editable": true }, @@ -3073,7 +3411,7 @@ }, { "cell_type": "markdown", - "id": "803c129e", + "id": "1c344e77", "metadata": { "editable": true }, @@ -3083,7 +3421,7 @@ }, { "cell_type": "markdown", - "id": "a0229d06", + "id": "d8064053", "metadata": { "editable": true }, @@ -3095,7 +3433,7 @@ }, { "cell_type": "markdown", - "id": "927c0677", + "id": "714ab01f", "metadata": { "editable": true }, @@ -3105,7 +3443,7 @@ }, { "cell_type": "markdown", - "id": "9b1faff4", + "id": "a3ff6988", "metadata": { "editable": true }, @@ -3117,7 +3455,7 @@ }, { "cell_type": "markdown", - "id": "2a9ba785", + "id": "5c204f79", "metadata": { "editable": true }, @@ -3133,7 +3471,7 @@ }, { "cell_type": "markdown", - "id": "13dd8452", + "id": "78c51f10", "metadata": { "editable": true }, @@ -3145,7 +3483,7 @@ }, { "cell_type": "markdown", - "id": "db18ae22", + "id": "2c432ed4", "metadata": { "editable": true }, @@ -3157,7 +3495,7 @@ }, { "cell_type": "markdown", - "id": "cc002ef4", + "id": "68a7b505", "metadata": { "editable": true }, @@ -3167,7 +3505,7 @@ }, { "cell_type": "markdown", - "id": "7808b2cd", + "id": "14268cd5", "metadata": { "editable": true }, @@ -3179,7 +3517,7 @@ }, { "cell_type": "markdown", - "id": "f6a8e671", + "id": "f3207844", "metadata": { "editable": true }, @@ -3190,7 +3528,7 @@ }, { "cell_type": "markdown", - "id": "e6298500", + "id": "4c7f5556", "metadata": { "editable": true }, @@ -3202,7 +3540,7 @@ }, { "cell_type": "markdown", - "id": "1a70e57a", + "id": "01bd8826", "metadata": { "editable": true }, @@ -3212,7 +3550,7 @@ }, { "cell_type": "markdown", - "id": "f2091227", + "id": "cb083248", "metadata": { "editable": true }, @@ -3224,7 +3562,7 @@ }, { "cell_type": "markdown", - "id": "029a7fce", + "id": "66bd1ec9", "metadata": { "editable": true }, @@ -3234,7 +3572,7 @@ }, { "cell_type": "markdown", - "id": "62dd4376", + "id": "b92a06bb", "metadata": { "editable": true }, @@ -3246,7 +3584,7 @@ }, { "cell_type": "markdown", - "id": "00865f17", + "id": "dad0b409", "metadata": { "editable": true }, @@ -3257,7 +3595,7 @@ }, { "cell_type": "markdown", - "id": "64bd2f73", + "id": "75d85fa8", "metadata": { "editable": true }, @@ -3269,7 +3607,7 @@ }, { "cell_type": "markdown", - "id": "ae49067e", + "id": "d0918773", "metadata": { "editable": true }, @@ -3287,7 +3625,7 @@ }, { "cell_type": "markdown", - "id": "9ea73c78", + "id": "65592b3b", "metadata": { "editable": true }, @@ -3303,7 +3641,7 @@ }, { "cell_type": "markdown", - "id": "6dcf8fff", + "id": "442404b8", "metadata": { "editable": true }, @@ -3315,7 +3653,7 @@ }, { "cell_type": "markdown", - "id": "8f12f6ca", + "id": "50eca816", "metadata": { "editable": true }, @@ -3327,7 +3665,7 @@ }, { "cell_type": "markdown", - "id": "03362376", + "id": "a27e487e", "metadata": { "editable": true }, @@ -3339,7 +3677,7 @@ }, { "cell_type": "markdown", - "id": "76d4b169", + "id": "49c74d48", "metadata": { "editable": true }, @@ -3352,7 +3690,7 @@ }, { "cell_type": "markdown", - "id": "6775daac", + "id": "214feab7", "metadata": { "editable": true }, @@ -3368,7 +3706,7 @@ }, { "cell_type": "markdown", - "id": "dc1392b1", + "id": "e61c0669", "metadata": { "editable": true }, @@ -3382,7 +3720,7 @@ }, { "cell_type": "markdown", - "id": "7faf1d79", + "id": "9f3db006", "metadata": { "editable": true }, @@ -3392,7 +3730,7 @@ }, { "cell_type": "markdown", - "id": "4c148d48", + "id": "6dfeb8ca", "metadata": { "editable": true }, @@ -3404,7 +3742,7 @@ }, { "cell_type": "markdown", - "id": "d2549195", + "id": "da47505d", "metadata": { "editable": true }, @@ -3414,7 +3752,7 @@ }, { "cell_type": "markdown", - "id": "4d7f14be", + "id": "87c1a756", "metadata": { "editable": true }, @@ -3426,7 +3764,7 @@ }, { "cell_type": "markdown", - "id": "3d795b82", + "id": "5028dae1", "metadata": { "editable": true }, @@ -3436,7 +3774,7 @@ }, { "cell_type": "markdown", - "id": "033e1b21", + "id": "98da161b", "metadata": { "editable": true }, @@ -3450,7 +3788,7 @@ }, { "cell_type": "markdown", - "id": "5d172bfd", + "id": "69065ef4", "metadata": { "editable": true }, @@ -3467,7 +3805,7 @@ }, { "cell_type": "markdown", - "id": "672de375", + "id": "3455143e", "metadata": { "editable": true }, @@ -3483,7 +3821,7 @@ }, { "cell_type": "markdown", - "id": "d0ed234f", + "id": "e322fdc8", "metadata": { "editable": true }, @@ -3495,7 +3833,7 @@ }, { "cell_type": "markdown", - "id": "7a2bde58", + "id": "f50b583d", "metadata": { "editable": true }, @@ -3508,7 +3846,7 @@ }, { "cell_type": "markdown", - "id": "88344390", + "id": "b12b62dc", "metadata": { "editable": true }, @@ -3522,7 +3860,7 @@ }, { "cell_type": "markdown", - "id": "4019cf50", + "id": "40355c87", "metadata": { "editable": true }, @@ -3532,7 +3870,7 @@ }, { "cell_type": "markdown", - "id": "95d6e396", + "id": "ee70666b", "metadata": { "editable": true }, @@ -3545,7 +3883,7 @@ }, { "cell_type": "markdown", - "id": "d5baa57d", + "id": "dc6d0cc7", "metadata": { "editable": true }, @@ -3564,7 +3902,7 @@ }, { "cell_type": "markdown", - "id": "f310ebda", + "id": "d6b6f792", "metadata": { "editable": true }, @@ -3576,7 +3914,7 @@ }, { "cell_type": "markdown", - "id": "a8366c66", + "id": "39c76bf5", "metadata": { "editable": true }, @@ -3588,7 +3926,7 @@ }, { "cell_type": "markdown", - "id": "9983b7fb", + "id": "f2d754e5", "metadata": { "editable": true }, @@ -3598,7 +3936,7 @@ }, { "cell_type": "markdown", - "id": "196e9dff", + "id": "77a984ab", "metadata": { "editable": true }, @@ -3610,7 +3948,7 @@ }, { "cell_type": "markdown", - "id": "fd64e6a2", + "id": "33060c1a", "metadata": { "editable": true }, @@ -3623,7 +3961,7 @@ }, { "cell_type": "markdown", - "id": "46a1e1da", + "id": "ed1e8279", "metadata": { "editable": true }, @@ -3642,7 +3980,7 @@ }, { "cell_type": "markdown", - "id": "6ea1ae21", + "id": "22b40263", "metadata": { "editable": true }, @@ -3652,7 +3990,7 @@ }, { "cell_type": "markdown", - "id": "fccc043c", + "id": "1b02469c", "metadata": { "editable": true }, @@ -3671,7 +4009,7 @@ }, { "cell_type": "markdown", - "id": "5d65cf6e", + "id": "0660e456", "metadata": { "editable": true }, @@ -3689,7 +4027,7 @@ }, { "cell_type": "markdown", - "id": "f6c00582", + "id": "0a40e5c0", "metadata": { "editable": true }, @@ -3703,7 +4041,7 @@ }, { "cell_type": "markdown", - "id": "fd88cfa1", + "id": "82d7e99d", "metadata": { "editable": true }, @@ -3717,8 +4055,8 @@ }, { "cell_type": "code", - "execution_count": 13, - "id": "c6da7058", + "execution_count": 14, + "id": "6f4b429b", "metadata": { "collapsed": false, "editable": true @@ -3739,7 +4077,7 @@ }, { "cell_type": "markdown", - "id": "bd31ca44", + "id": "a38f8430", "metadata": { "editable": true }, @@ -3755,8 +4093,8 @@ }, { "cell_type": "code", - "execution_count": 14, - "id": "d809cece", + "execution_count": 15, + "id": "87d44fdf", "metadata": { "collapsed": false, "editable": true @@ -3788,7 +4126,7 @@ }, { "cell_type": "markdown", - "id": "8253074c", + "id": "5f08f4dc", "metadata": { "editable": true }, @@ -3802,7 +4140,7 @@ }, { "cell_type": "markdown", - "id": "7a74656a", + "id": "a25b356f", "metadata": { "editable": true }, @@ -3814,8 +4152,8 @@ }, { "cell_type": "code", - "execution_count": 15, - "id": "804a8ee1", + "execution_count": 16, + "id": "928ae649", "metadata": { "collapsed": false, "editable": true @@ -3840,99 +4178,7 @@ }, { "cell_type": "markdown", - "id": "3871673a", - "metadata": { - "editable": true - }, - "source": [ - "We expand this model to the Franke function discussed above." - ] - }, - { - "cell_type": "markdown", - "id": "5e75414e", - "metadata": { - "editable": true - }, - "source": [ - "## Correlation Matrix with Pandas and the Franke function" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "b3efcdb9", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], - "source": [ - "# Common imports\n", - "import numpy as np\n", - "import pandas as pd\n", - "\n", - "\n", - "def FrankeFunction(x,y):\n", - "\tterm1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))\n", - "\tterm2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))\n", - "\tterm3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))\n", - "\tterm4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)\n", - "\treturn term1 + term2 + term3 + term4\n", - "\n", - "\n", - "def create_X(x, y, n ):\n", - "\tif len(x.shape) > 1:\n", - "\t\tx = np.ravel(x)\n", - "\t\ty = np.ravel(y)\n", - "\n", - "\tN = len(x)\n", - "\tl = int((n+1)*(n+2)/2)\t\t# Number of elements in beta\n", - "\tX = np.ones((N,l))\n", - "\n", - "\tfor i in range(1,n+1):\n", - "\t\tq = int((i)*(i+1)/2)\n", - "\t\tfor k in range(i+1):\n", - "\t\t\tX[:,q+k] = (x**(i-k))*(y**k)\n", - "\n", - "\treturn X\n", - "\n", - "\n", - "# Making meshgrid of datapoints and compute Franke's function\n", - "n = 4\n", - "N = 100\n", - "x = np.sort(np.random.uniform(0, 1, N))\n", - "y = np.sort(np.random.uniform(0, 1, N))\n", - "z = FrankeFunction(x, y)\n", - "X = create_X(x, y, n=n) \n", - "\n", - "Xpd = pd.DataFrame(X)\n", - "# subtract the mean values and set up the covariance matrix\n", - "Xpd = Xpd - Xpd.mean()\n", - "covariance_matrix = Xpd.cov()\n", - "print(covariance_matrix)" - ] - }, - { - "cell_type": "markdown", - "id": "03523168", - "metadata": { - "editable": true - }, - "source": [ - "We note here that the covariance is zero for the first rows and\n", - "columns since all matrix elements in the design matrix were set to one\n", - "(we are fitting the function in terms of a polynomial of degree $n$).\n", - "\n", - "This means that the variance for these elements will be zero and will\n", - "cause problems when we set up the correlation matrix. We can simply\n", - "drop these elements and construct a correlation\n", - "matrix without these elements." - ] - }, - { - "cell_type": "markdown", - "id": "ab0fcb46", + "id": "cab70377", "metadata": { "editable": true }, @@ -3944,7 +4190,7 @@ }, { "cell_type": "markdown", - "id": "9e11454d", + "id": "22ac4a6d", "metadata": { "editable": true }, @@ -3956,7 +4202,7 @@ }, { "cell_type": "markdown", - "id": "35eaf4ef", + "id": "cd0697a9", "metadata": { "editable": true }, @@ -3966,7 +4212,7 @@ }, { "cell_type": "markdown", - "id": "a426a48c", + "id": "b5043405", "metadata": { "editable": true }, @@ -3983,7 +4229,7 @@ }, { "cell_type": "markdown", - "id": "f0ac2bfa", + "id": "6a4edab6", "metadata": { "editable": true }, @@ -3993,7 +4239,7 @@ }, { "cell_type": "markdown", - "id": "c6d8c9b4", + "id": "e1b67496", "metadata": { "editable": true }, @@ -4008,7 +4254,7 @@ }, { "cell_type": "markdown", - "id": "a87b1d66", + "id": "47a041cb", "metadata": { "editable": true }, @@ -4018,7 +4264,7 @@ }, { "cell_type": "markdown", - "id": "cb950b85", + "id": "575264b7", "metadata": { "editable": true }, @@ -4032,7 +4278,7 @@ }, { "cell_type": "markdown", - "id": "c79e96c6", + "id": "7b049c58", "metadata": { "editable": true }, @@ -4044,7 +4290,7 @@ }, { "cell_type": "markdown", - "id": "1fceefdc", + "id": "9780789f", "metadata": { "editable": true }, @@ -4056,7 +4302,7 @@ }, { "cell_type": "markdown", - "id": "a8db1029", + "id": "da396c4a", "metadata": { "editable": true }, @@ -4068,7 +4314,7 @@ }, { "cell_type": "markdown", - "id": "03d9d476", + "id": "87a74417", "metadata": { "editable": true }, @@ -4078,7 +4324,7 @@ }, { "cell_type": "markdown", - "id": "9ccc8461", + "id": "c5ca565a", "metadata": { "editable": true }, @@ -4090,7 +4336,7 @@ }, { "cell_type": "markdown", - "id": "c6183bd4", + "id": "10944fc5", "metadata": { "editable": true }, @@ -4100,7 +4346,7 @@ }, { "cell_type": "markdown", - "id": "5cc84134", + "id": "7b985955", "metadata": { "editable": true }, @@ -4117,7 +4363,7 @@ }, { "cell_type": "markdown", - "id": "788d9c68", + "id": "1b748051", "metadata": { "editable": true }, @@ -4127,7 +4373,7 @@ }, { "cell_type": "markdown", - "id": "bd0e5aee", + "id": "40c1c1eb", "metadata": { "editable": true }, @@ -4139,7 +4385,7 @@ }, { "cell_type": "markdown", - "id": "dc52b90c", + "id": "e44c9e87", "metadata": { "editable": true }, @@ -4149,7 +4395,7 @@ }, { "cell_type": "markdown", - "id": "b593db6e", + "id": "28abe46c", "metadata": { "editable": true }, @@ -4161,7 +4407,7 @@ }, { "cell_type": "markdown", - "id": "8dcd6cc8", + "id": "178bc776", "metadata": { "editable": true }, @@ -4175,7 +4421,7 @@ }, { "cell_type": "markdown", - "id": "29312718", + "id": "1f0ead95", "metadata": { "editable": true }, @@ -4187,7 +4433,7 @@ }, { "cell_type": "markdown", - "id": "9818fcb1", + "id": "fcce3bb5", "metadata": { "editable": true }, @@ -4209,7 +4455,7 @@ }, { "cell_type": "markdown", - "id": "56fbca85", + "id": "5f9a7281", "metadata": { "editable": true }, @@ -4221,7 +4467,7 @@ }, { "cell_type": "markdown", - "id": "d9d152fe", + "id": "d3ea695c", "metadata": { "editable": true }, @@ -4236,7 +4482,7 @@ }, { "cell_type": "markdown", - "id": "b82dd67d", + "id": "edfff9e0", "metadata": { "editable": true }, @@ -4248,7 +4494,7 @@ }, { "cell_type": "markdown", - "id": "7dab117e", + "id": "c3c83244", "metadata": { "editable": true }, @@ -4260,7 +4506,7 @@ }, { "cell_type": "markdown", - "id": "3649c008", + "id": "00df5ee5", "metadata": { "editable": true }, @@ -4270,7 +4516,7 @@ }, { "cell_type": "markdown", - "id": "665e42ab", + "id": "4661cbd7", "metadata": { "editable": true }, @@ -4282,7 +4528,7 @@ }, { "cell_type": "markdown", - "id": "254d4fd4", + "id": "07b95265", "metadata": { "editable": true }, @@ -4292,7 +4538,7 @@ }, { "cell_type": "markdown", - "id": "4b84bb18", + "id": "8c9aeba4", "metadata": { "editable": true }, @@ -4304,7 +4550,7 @@ }, { "cell_type": "markdown", - "id": "d84538dd", + "id": "03690a72", "metadata": { "editable": true }, @@ -4314,7 +4560,7 @@ }, { "cell_type": "markdown", - "id": "f9a8f033", + "id": "478e2a9d", "metadata": { "editable": true }, @@ -4326,7 +4572,7 @@ }, { "cell_type": "markdown", - "id": "0c900c57", + "id": "e997196a", "metadata": { "editable": true }, @@ -4343,12 +4589,12 @@ }, { "cell_type": "markdown", - "id": "ef8aafcf", + "id": "770e587e", "metadata": { "editable": true }, "source": [ - "## Ridge and LASSO Regression\n", + "## Back to Ridge and LASSO Regression\n", "\n", "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", "our optimization problem is" @@ -4356,7 +4602,7 @@ }, { "cell_type": "markdown", - "id": "efc29850", + "id": "5cb701c2", "metadata": { "editable": true }, @@ -4368,7 +4614,7 @@ }, { "cell_type": "markdown", - "id": "586af0fa", + "id": "dee9d948", "metadata": { "editable": true }, @@ -4378,7 +4624,7 @@ }, { "cell_type": "markdown", - "id": "116d6e8b", + "id": "121cd014", "metadata": { "editable": true }, @@ -4391,7 +4637,7 @@ }, { "cell_type": "markdown", - "id": "90567753", + "id": "2ee745b4", "metadata": { "editable": true }, @@ -4401,7 +4647,7 @@ }, { "cell_type": "markdown", - "id": "aa3053f1", + "id": "94a15fb9", "metadata": { "editable": true }, @@ -4413,7 +4659,7 @@ }, { "cell_type": "markdown", - "id": "b2382074", + "id": "6341fe06", "metadata": { "editable": true }, @@ -4426,7 +4672,7 @@ }, { "cell_type": "markdown", - "id": "716a3c2a", + "id": "5bef6925", "metadata": { "editable": true }, @@ -4439,7 +4685,7 @@ }, { "cell_type": "markdown", - "id": "dfd4ebf1", + "id": "4539e82b", "metadata": { "editable": true }, @@ -4451,7 +4697,7 @@ }, { "cell_type": "markdown", - "id": "d782ab23", + "id": "1f496d16", "metadata": { "editable": true }, @@ -4463,7 +4709,7 @@ }, { "cell_type": "markdown", - "id": "f16d590d", + "id": "ee78adfa", "metadata": { "editable": true }, @@ -4473,7 +4719,7 @@ }, { "cell_type": "markdown", - "id": "a373fb74", + "id": "47a0c860", "metadata": { "editable": true }, @@ -4486,7 +4732,7 @@ }, { "cell_type": "markdown", - "id": "c4c2a6db", + "id": "6b452b5d", "metadata": { "editable": true }, @@ -4498,7 +4744,7 @@ }, { "cell_type": "markdown", - "id": "f14edde4", + "id": "a3331a79", "metadata": { "editable": true }, @@ -4510,134 +4756,12 @@ }, { "cell_type": "markdown", - "id": "899bc3ae", + "id": "9befd0e3", "metadata": { "editable": true }, "source": [ - "## Deriving the Ridge Regression Equations\n", - "\n", - "Using the matrix-vector expression for Ridge regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have" - ] - }, - { - "cell_type": "markdown", - "id": "0968804d", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\boldsymbol{\\theta}^T\\boldsymbol{\\theta},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "87643992", - "metadata": { - "editable": true - }, - "source": [ - "and \n", - "taking the derivatives with respect to $\\boldsymbol{\\theta}$ we obtain then\n", - "a slightly modified matrix inversion problem which for finite values\n", - "of $\\lambda$ does not suffer from singularity problems. We obtain\n", - "the optimal parameters" - ] - }, - { - "cell_type": "markdown", - "id": "900e9e55", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "6d3a5c46", - "metadata": { - "editable": true - }, - "source": [ - "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" - ] - }, - { - "cell_type": "markdown", - "id": "db3c68ed", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\sum_{i=0}^{p-1} \\theta_i^2 \\leq t,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "ab6d0410", - "metadata": { - "editable": true - }, - "source": [ - "with $t$ a finite positive number. \n", - "\n", - "If we keep the $1/n$ factor, the equation for the optimal $\\theta$ changes to" - ] - }, - { - "cell_type": "markdown", - "id": "ae454f20", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "2a508398", - "metadata": { - "editable": true - }, - "source": [ - "In many textbooks the $1/n$ term is often omitted. Note that a library like **Scikit-Learn** does not include the $1/n$ factor in the setup of the cost function.\n", - "\n", - "When we compare this with the ordinary least squares result we have" - ] - }, - { - "cell_type": "markdown", - "id": "6df6cf96", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "4ab38d70", - "metadata": { - "editable": true - }, - "source": [ - "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", - "\n", - "We see that Ridge regression is nothing but the standard OLS with a\n", + "Ridge regression, as discussed above, is nothing but the standard OLS with a\n", "modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n", "particular for our discussion of the bias-variance tradeoff are rather\n", "interesting. We will see that for specific values of $\\lambda$, we may\n", @@ -4649,7 +4773,7 @@ }, { "cell_type": "markdown", - "id": "cfa1a762", + "id": "9d8f2ba3", "metadata": { "editable": true }, @@ -4661,7 +4785,7 @@ }, { "cell_type": "markdown", - "id": "7394c1c2", + "id": "da712e9d", "metadata": { "editable": true }, @@ -4671,7 +4795,7 @@ }, { "cell_type": "markdown", - "id": "c67e8228", + "id": "d91cd5f7", "metadata": { "editable": true }, @@ -4683,7 +4807,7 @@ }, { "cell_type": "markdown", - "id": "95c698ae", + "id": "18940f72", "metadata": { "editable": true }, @@ -4693,7 +4817,7 @@ }, { "cell_type": "markdown", - "id": "372c4fae", + "id": "51fe3527", "metadata": { "editable": true }, @@ -4705,7 +4829,7 @@ }, { "cell_type": "markdown", - "id": "826e010b", + "id": "2e85d1eb", "metadata": { "editable": true }, @@ -4717,7 +4841,7 @@ }, { "cell_type": "markdown", - "id": "b3c2a178", + "id": "96c47485", "metadata": { "editable": true }, @@ -4733,7 +4857,7 @@ }, { "cell_type": "markdown", - "id": "2ec82017", + "id": "c1adcbbc", "metadata": { "editable": true }, @@ -4745,7 +4869,7 @@ }, { "cell_type": "markdown", - "id": "dacb08f8", + "id": "9693074c", "metadata": { "editable": true }, @@ -4757,7 +4881,7 @@ }, { "cell_type": "markdown", - "id": "af53eaa0", + "id": "46b05531", "metadata": { "editable": true }, @@ -4767,7 +4891,7 @@ }, { "cell_type": "markdown", - "id": "128f95fc", + "id": "2e4ce02b", "metadata": { "editable": true }, @@ -4779,7 +4903,7 @@ }, { "cell_type": "markdown", - "id": "bd5b4b5d", + "id": "dbd3f781", "metadata": { "editable": true }, @@ -4789,7 +4913,7 @@ }, { "cell_type": "markdown", - "id": "d316a1ae", + "id": "779e1ac5", "metadata": { "editable": true }, @@ -4801,7 +4925,7 @@ }, { "cell_type": "markdown", - "id": "1198b684", + "id": "ef53f2a5", "metadata": { "editable": true }, @@ -4818,7 +4942,7 @@ }, { "cell_type": "markdown", - "id": "bb24826b", + "id": "57adfaff", "metadata": { "editable": true }, @@ -4830,7 +4954,7 @@ }, { "cell_type": "markdown", - "id": "b9e921e5", + "id": "a8f94cba", "metadata": { "editable": true }, @@ -4842,7 +4966,7 @@ }, { "cell_type": "markdown", - "id": "06ad064c", + "id": "d60431ff", "metadata": { "editable": true }, @@ -4852,7 +4976,7 @@ }, { "cell_type": "markdown", - "id": "ab63b0d3", + "id": "f78d1dfd", "metadata": { "editable": true }, @@ -4864,7 +4988,7 @@ }, { "cell_type": "markdown", - "id": "49285a6a", + "id": "445800de", "metadata": { "editable": true }, @@ -4874,7 +4998,7 @@ }, { "cell_type": "markdown", - "id": "46be0f62", + "id": "3c54c49c", "metadata": { "editable": true }, @@ -4886,7 +5010,7 @@ }, { "cell_type": "markdown", - "id": "1ad7dd7c", + "id": "11366dc1", "metadata": { "editable": true }, @@ -4896,7 +5020,7 @@ }, { "cell_type": "markdown", - "id": "3d0b7ab5", + "id": "d68d3ac1", "metadata": { "editable": true }, @@ -4908,7 +5032,7 @@ }, { "cell_type": "markdown", - "id": "b360d642", + "id": "63fedbd6", "metadata": { "editable": true }, @@ -4918,7 +5042,7 @@ }, { "cell_type": "markdown", - "id": "8677a588", + "id": "470b20a8", "metadata": { "editable": true }, @@ -4930,12 +5054,12 @@ }, { "cell_type": "markdown", - "id": "d4f0eccb", + "id": "f3277c65", "metadata": { "editable": true }, "source": [ - "This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later." + "This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss how to code LASSO regression next week, when we have introduced gradient methods." ] } ],